Category Archives: backup

Adjusting the size of the tarragondata volume

At one point I was getting low on space on tarragondata, so I added an additional physical device to the btrfs filesystem containing tarragondata.

[root@tarragon backup_scripts]# btrfs fi show
Label: 'tarragon_data' uuid: d6e4b6fc-8745-4e6e-b6b4-8548142b5154
 Total devices 2 FS bytes used 92.04GiB
 devid 1 size 120.00GiB used 120.00GiB path /dev/xvdf1
 devid 2 size 30.00GiB used 30.00GiB path /dev/xvdg

This is fine, but there are a couple of problems. The main one is that I can no longer use the EC2 snapshot capability on tarragondata, which meant that the nightly EC2 snapshot feature I was using had to be deimplemented.

But now I am about to create a new tarragon instance, and it would be really helpful to be able to snapshot tarragondata (Amazon snapshot, not btrfs snapshot) and then create a new Amazon volume with a consistent snapshot for testing.

So to do that, I am going to use the btrfs feature of adding and removing physical volumes to consolidate tarragondata on a single physical volume.

Steps are:

1) create a new 150GB physical volume in Amazon EC2 dashboard, and attach it to tarragon.

2) On tarragon, add the new physical volume to the tarragondata btrfs filesystem.

 btrfs device add /dev/xvdh /mnt/tarragondata

3) Remove the older smaller physical devices one at a time:

 btrfs device delete /dev/xvdg /mnt/tarragondata
 btrfs device delete /dev/xvdf1 /mnt/tarragondata

This successfully moved all the data and metadata to a new 150GB drive, which enabled me to snapshot it.

mdadm consistency checks

On ubuntu it seems there is an automatic mdadm array check provided in /etc/cron.d/mdadm, automatically installed with mdadm. This invokes a utility /usr/share/mdadm/checkarray and the cron is set to run this on the first Sunday of every month at 12:57am. And it is set to do this check on all arrays at one time.

This is horrible! So with 5 arrays, totalling 25TB, when this sucker fires up it quickly saturates the i/o capacity of cinnamon, slows to a crawl and settles in to run forever.

I’ve commented that out, and added my own /etc/cron.d/dee_mdadm which doesn’t do all the goofy shenanigans to try to ensure the thing runs on a Sunday (WHY?! Because the guy who wrote it doesn’t work on Sunday?). Instead, my version simply runs on the first of the month, at 12:57am, and on each month it starts the consistency check on a different array. I have 5 arrays, so 3 are checked twice a year, and 2 are checked thrice. Checking just one at a time means there is a good chance it will be done before morning, at least for the small arrays.

I don’t really think the whole consistency check idea is doing me much good, but at least this doesn’t unaccountably bring the system to its knees on the first Sunday of every month.

Updating certificates to “Let’s Encrypt” with ACME

I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.

Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.

But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME

S3cmd on ubuntu 15.04

After installing ubuntu 15.04 my backups to S3 stopped working.

I tried running them manually to see what was happening and I got errors – some goofy stuff about the url I was using net.wmbuck.backups….s3.amazonaws.com not being part of *s3.amazonaws.com. When I searched the net I found that there was a change in python 2.7.9 having to do with evaluating certificates, and some conflict with the wildcard cert being used by Amazon S3, with the result that there is an error which occurs whenever an S3 bucket happens to contain the “.” character in its title.

My buckets are all named net.wmbuck.x so I am vulnerable to this error.

There is a fix for this in S3cmd version 1.6.0 but the latest ubuntu as of this writing has only S3cmd 1.5.x and attempting to upgrade using apt-get doesn’t get anything new.

I did an apt-get remove of s3cmd, and then downloaded a tarball, and installed it into /usr/local/bin.

Ubuntu 15.10 will be coming out next month, and when I get around to installing that perhaps the version of s3cmd will have the fix.

top of page