Category Archives: btrfs

backup, btrfs, linux, mysql, ubuntu

Rosemary Recovery 2020

There was a fail event reported on rosemary from one side of a pair of 60GB SSDs, which hold Rosemary_Data. Typical of my installations this mirror set holds the stuff the system needs beyond the os install: /home, databases, certificates, repositories, mail, samba, local bin, etc. Its a mirror set with an encrypted container, containing a btrfs filesystem. The older versions of these setups contain separate btrfs subvolumes for the different directories, newer ones have only one subvolume for that, and another for snaps. This is an old one.

Rosemary doesn’t have an extensive set of services – really only the /home and the databases. No real need for much of anything else. The local bin comes out of the repo anyway, there is no mail, no repository, no certs. However, without that volume the system won’t come up in a usable way. So lesson one learned here was when you get a fail event, attend to it. I let it go for a few days, because I knew I was going to have to pull the case out of the rack mount to get at the SSDs.

Continue reading Rosemary Recovery 2020 →
backup, btrfs, centos, linux

Out of space on btrfs

I don’t know if I actually know enough to write this post. But I want to record what little I do know about this.

The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most of the dynamic parts of the system. The root volume ‘/’ is very small, about 20GB. Just enough to install the Centos code and keep a few little things. The great majority of the information needed to run the system is symlinked out of /, which includes /home, mail, databases, websites and their data, the repositories, certificates, local scripts etc.

This is a 180GB disk, and it currently is running about 55% full, i.e. almost 100GB used. Among the information on this disk are snapshots of all the tarragondata, every night for 30 days. This isn’t disaster backup/disk failure backup (which is elsewhere), this is “operator error” backup.

A couple of weeks ago I began to experience a new kind of failure. In the middle of the night, suddenly this btrfs volume would report that it was out of space – usage 100%, although the amount of storage in use was, still the roughtly 100GB that it normally uses. It manifestly was not actually out of space.

Continue reading Out of space on btrfs →
apache, backup, btrfs, centos, cloud, fedora, linux, mail, mysql, networking, postfix, website building

Tarragon Rebuild 2019

This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.

The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.

Continue reading Tarragon Rebuild 2019 →
backup, boot issues, btrfs, fedora, linux

Adjusting the size of the tarragondata volume

At one point I was getting low on space on tarragondata, so I added an additional physical device to the btrfs filesystem containing tarragondata. 

[root@tarragon backup_scripts]# btrfs fi show
Label: 'tarragon_data' uuid: d6e4b6fc-8745-4e6e-b6b4-8548142b5154
 Total devices 2 FS bytes used 92.04GiB
 devid 1 size 120.00GiB used 120.00GiB path /dev/xvdf1
 devid 2 size 30.00GiB used 30.00GiB path /dev/xvdg

This is fine, but there are a couple of problems. The main one is that I can no longer use the EC2 snapshot capability on tarragondata, which meant that the nightly EC2 snapshot feature I was using had to be deimplemented.

But now I am about to create a new tarragon instance, and it would be really helpful to be able to snapshot tarragondata (Amazon snapshot, not btrfs snapshot) and then create a new Amazon volume with a consistent snapshot for testing. Continue reading Adjusting the size of the tarragondata volume →

btrfs, cloud, linux

New Server

The server for wmbuck.net (tarragon is its name) has been moved to Amazon EC2.

In part this was an experiment, motivated by curiosity about the ease or difficulty of maintaining a server in the cloud. But also in part it was motivated by dissatisfaction with the previous hosting environment, superb hosting. I needed to upgrade the server capacity. It had not been improved for 7 years, and it was ancient hardware when I got it. But at superb I ran into a lot of trouble, because I wanted to use btrfs (see earlier post  about using btrfs on the home computer, oregano). Superb offers old stable kernels, but I really  needed a kernel new enough to have a stable btrfs in it. After I lost all my work owing to btrfs driver problems in an old CentOS kernel, I was highly motivated to try another hosting arrangement where I could have my kernel of choice.

In mid May I set about creating a new tarragon. It took less than a week to set up, including all my learning curve, and all the migration issues, data transfer, etc. It has been running as wmbuck.net for over a month. The few hiccups were mistakes of transition on my part. Up to this moment I have had no problem which has turned out to be of any but my own making.
Continue reading New Server →

boot issues, btrfs, encryption, fedora, linux

New disk layout using btrfs

Preparatory to doing an upgrade of Fedora on one of my linux boxes, I decided it was time to revise the disk layouts. Specifically, I finally got around to building a separate partition for /home. After doing a little reading, it looked to me like a good way to do this was to use btrfs. Initially I also made a separate /var, but decided that was both unneeded and error prone, and went another way. The main objective of this work was to reduce the drag in upgrading Fedora.

Root is on a small (64GB) SSD, LUKS encrypted which has /boot and an LVM VG for / and swap. / is ext4. I built a 2GB mdadm raid-1 mirror with LUKS encryption on top of it, and then layered a btrfs file system on that called oreganodata. The root of oreganodata is not typically mounted, but has the following subvolumes automounted at boot time:

  1. /home -> oreganodata/home
  2. /mail->oreganodata/mail,  /var/lib/imap and /var/spool/imap symlink to the /mail/lib and /mail/spool directories.
  3. /var/lib/mysql->oreganodata/maria
  4. /usr/local/bin->oreganodata/bin
  5. /etc/pki/mycerts->oreganodata/certs
  6. /var/webdata->oreganodata/webdata (websites have symlinks to /var/webdata as needed)

Continue reading New disk layout using btrfs →