Category Archives: centos

backup, btrfs, centos, linux

Out of space on btrfs

I don’t know if I actually know enough to write this post. But I want to record what little I do know about this.

The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most of the dynamic parts of the system. The root volume ‘/’ is very small, about 20GB. Just enough to install the Centos code and keep a few little things. The great majority of the information needed to run the system is symlinked out of /, which includes /home, mail, databases, websites and their data, the repositories, certificates, local scripts etc.

This is a 180GB disk, and it currently is running about 55% full, i.e. almost 100GB used. Among the information on this disk are snapshots of all the tarragondata, every night for 30 days. This isn’t disaster backup/disk failure backup (which is elsewhere), this is “operator error” backup.

A couple of weeks ago I began to experience a new kind of failure. In the middle of the night, suddenly this btrfs volume would report that it was out of space – usage 100%, although the amount of storage in use was, still the roughtly 100GB that it normally uses. It manifestly was not actually out of space.

Continue reading Out of space on btrfs →
apache, centos, cloud, encryption, linux, networking, security, website building

Apache certificate chains

When I switched my main server to CentOS, described in an earlier post, one of the big pains was that I had to use CentOS 7, and there was a lot of software which had come a long way since CentOS 7, and I had to upgrade a log of things from upstream to get functionality that I had grown reliant upon.

I didn’t realize that Apache itself was one of those things that was sufficiently backwards in CentOS 7 that I would have trouble.

Ever since I move the server to CentOSdid that “upgrade”, I’ve been struggling with problems with the certificates not being honored. For the last few days I have been working pretty diligently to try to figure out this nagging problem, and today I finally figured it out. It is owing to an old Apache.

Continue reading Apache certificate chains →
centos, eclipse, fedora, linux

Desktop Files

For a very goofy reason involving a bug with composer, and new warning messages in PHP, I decided I needed to reinstall eclipse on oregano. Reinstalling eclipse is always a test of my patience. During the entire course of my life I do not believe that the installation of eclipse along with the various components I need, especially subclipse, has ever “just worked”. There is always some issue. So, of course, I do not look forward to it with gleeful anticipation.

I think part of the problem for me – a problem of my own making I suppose – is that I mostly use distros (Fedora and Ubuntu) in which the packagers have already arranged for tools like eclipse to appear as “system level” rather than “user level” tools. By this I mean they appear (or are symlinked from) /usr/bin, their configuration files are located it /etc, their icons appear when you search for apps, etc. Whereas if you just use the upstream installers the installation will be carried out at the user level, they will ask you where to put the binary, and they will put configuration files in ~/.config,

Continue reading Desktop Files →
apache, backup, centos, cloud, fedora, linux, mail, postfix, security, website building

Clamd signatures and Apache memory

After implementing the new tarragon the biggest problem I had involved the clamav package, and its loading of signatures. If clamd doesn’t come up and open its socket, then amavisd (the daemon who is consulted by postfix to handle all the checking of each piece of mail on input and output) will fail (assuming he is configured to do virus checking), This results in various problems. Amavis will mark the mail as “unchecked”, but worse, it will report failure back to postfix who gets confused and very often the message is delivered two or three times.

Clamd, the clamav daemon, now has over 6 million signatures. There are a lot of bad boys out there. The signatures are loaded by clamd from its database (in /var/lib/clamav) on startup, into memory. As a result, clamd has a large memory footprint, almost 800Mb on my system. The first issue, discovered before going live, was that systemd’s default parameters expect any daemon he starts to load within 90 seconds. If it fails to check in within that time, systemd considers it broken and terminates it. Clamd takes at least 3 minutes to load. I had to set a special TimeoutStartSec value in the systemd service script for clamd@.service.

Whew! I thought, boy I’m glad I figured that out. Hah!

Continue reading Clamd signatures and Apache memory →
apache, backup, btrfs, centos, cloud, fedora, linux, mail, mysql, networking, postfix, website building

Tarragon Rebuild 2019

This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.

The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.

Continue reading Tarragon Rebuild 2019 →