I don’t know if I actually know enough to write this post. But I want to record what little I do know about this. <\/p>\n\n\n\n
The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most of the dynamic parts of the system. The root volume ‘\/’ is very small, about 20GB. Just enough to install the Centos code and keep a few little things. The great majority of the information needed to run the system is symlinked out of \/, which includes \/home, mail, databases, websites and their data, the repositories, certificates, local scripts etc. <\/p>\n\n\n\n
This is a 180GB disk, and it currently is running about 55% full, i.e. almost 100GB used. Among the information on this disk are snapshots of all the tarragondata, every night for 30 days. This isn’t disaster backup\/disk failure backup (which is elsewhere), this is “operator error” backup. <\/p>\n\n\n\n
A couple of weeks ago I began to experience a new kind of failure. In the middle of the night, suddenly this btrfs volume would report that it was out of space – usage 100%, although the amount of storage in use was, still the roughtly 100GB that it normally uses. It manifestly was not actually out of space. <\/p>\n\n\n\n\n\n\n\n
When this disk is out of space, everything dies pretty quickly: mysqld, dovecot, postfix, and apache all start throwing errors. Mail delivery stops, people can’t get authenticate into imap, wailing and gnashing of teeth ensues. <\/p>\n\n\n\n
I know a little more about this now, though by no means do I claim to have a full understanding. But this has to do with the way that btrfs manages space. <\/p>\n\n\n\n
Here is the ordinary system-level df: <\/p>\n\n\n\n
[root@tarragon ~]# df \/tarragon5data\nFilesystem 1K-blocks Used Available Use% Mounted on\n\/dev\/xvdf1 178256896 97931924 80189804 55% \/tarragon5data<\/code><\/pre>\n\n\n\nBut here is what btrfs’ private version of df says: <\/p>\n\n\n\n
[root@tarragon ~]# btrfs fi df \/tarragon5data\nData, single: total=164.99GiB, used=88.51GiB\nSystem, single: total=4.00MiB, used=48.00KiB\nMetadata, single: total=5.01GiB, used=4.56GiB\nGlobalReserve, single: total=326.34MiB, used=0.00B<\/code><\/pre>\n\n\n\nThe thing to note is that for the type Metadata, the used is almost all of the total. In addition, the btrfs total usage, shown by the fi show was: <\/p>\n\n\n\n
[root@tarragon ~]# btrfs fi show\nLabel: 'tarragondata' uuid: xxx\n\tTotal devices 1 FS bytes used 93.08GiB\n\tdevid 1 size 170.00GiB used 170.00GiB path \/dev\/xvdf1<\/code><\/pre>\n\n\n\nNotice in this display that btrfs has allocated all of the space on the device for use, and internally it seems that it initially allocated only 5.01GiB for Metadata, and all the rest has been (internally) already allocated to the Data spaces. When it suddenly needs more space for metadata, there is no way to allocate blocks for that, because all the blocks are already in use. <\/p>\n\n\n\n
In order to fix this problem, what one has to do is instruct btrfs to free up some of its blocks which are currently assigned to data, to be available for metadata. I encountered two different commands which can help with this – both are variants of the btrfs balance command. <\/p>\n\n\n\n
[root@tarragon]# btrfs fi balance start -dusage=5 \/tarragon5data\nDone, had to relocate 0 out of 172 chunks\n[root@tarragon]# btrfs fi balance start -dusage=10 \/tarragon5data\nDone, had to relocate 10 out of 172 chunks<\/code><\/pre>\n\n\n\nThis tells btrfs to move stuff around de-allocating blocks that are 5% of less full. It freed no blocks at 5% so I did it again at 10%, and it freed some up. This may have done the trick, but I looked at the btrfs fi df, expecting to see a change, and when I didn’t I read some more and found another articles which advised: <\/p>\n\n\n\n
[root@tarragon]# btrfs fi balance start -dlimit=3 \/tarragon5data\nDone, had to relocate 3 out of 163 chunks<\/code><\/pre>\n\n\n\nThe first may actually have done the job, but although the btrfs fi df didn’t change, had I thought to look at the btrfs fi show, and it says: <\/p>\n\n\n\n
[root@tarragon]# btrfs fi show\nLabel: 'tarragondata' uuid: xxx\n\tTotal devices 1 FS bytes used 93.07GiB\n\tdevid 1 size 170.00GiB used 158.02GiB path \/dev\/xvdf1<\/code><\/pre>\n\n\n\nNow btrfs has space available to allocate for metadata. <\/p>\n","protected":false},"excerpt":{"rendered":"
I don’t know if I actually know enough to write this post. But I want to record what little I do know about this. The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most … Continue reading Out of space on btrfs<\/span>