Category Archives: linux

Out of space on btrfs

I don’t know if I actually know enough to write this post. But I want to record what little I do know about this.

The symptom is that my tarragondata volume on this system, tarragon, claims to be out of space. This is a btrfs volume, about which there are other posts. It contains most of the dynamic parts of the system. The root volume ‘/’ is very small, about 20GB. Just enough to install the Centos code and keep a few little things. The great majority of the information needed to run the system is symlinked out of /, which includes /home, mail, databases, websites and their data, the repositories, certificates, local scripts etc.

This is a 180GB disk, and it currently is running about 55% full, i.e. almost 100GB used. Among the information on this disk are snapshots of all the tarragondata, every night for 30 days. This isn’t disaster backup/disk failure backup (which is elsewhere), this is “operator error” backup.

A couple of weeks ago I began to experience a new kind of failure. In the middle of the night, suddenly this btrfs volume would report that it was out of space – usage 100%, although the amount of storage in use was, still the roughtly 100GB that it normally uses. It manifestly was not actually out of space.

Continue reading Out of space on btrfs

Goodbye fedora

I’ve commented in a number of previous posts that I have had an inordinate amount of trouble with Fedora, compared to other distributions I have used. There were also a few posts about my efforts with Arch. Now I am biting the bullet and declaring the move of my main daily work computer to Arch to have been completed. I have been running on Arch for a month or three, and there is no going back.

Continue reading Goodbye fedora

No Worthy mechs

I just built another little gateway pi, on Raspbian. It is a newer Raspbian than I used for the others. And newer is better, right? But when the little fellow came up, postfix claimed it couldn’t relay through tarragon, because it couldn’t accomplish login. Sasl authentication failed, no worthy mechs.

I remember good old “no worthy mechs” from way back when, always thought it a very cool error message. But why in the heck am I getting this? I’m not doing anything fancy.

I had a vague nagging feeling from some old Fedora problems, long ago. Could it be that I have to install some kind of sasl library, even to do plain authentication? Poked around a little. Eventually did an apt install libsasl2-modules and sasl2-bin. Sure enough, they actually installed.

And afterwords postfix came up and send the mail out of the queue. I’ll be dipped in … I am surprised. This was not something I had to do before. Is this an improvement in Raspbian. Don’t package any sasl mechs, make the poor sod figure out why sasl authentication won’t work.

This is to help me remember the next time this happens.

Re-signing Openssh Certificates

Seldom do I get to write a post where I am offering information which might not actually be out there in a lot of places. I could not find this information on the web, and had to figure it out myself, by reading the code, and doing experiments.

I talked in the last post about the need to re-issue all the openssh certificates, in order to update the hash algorithm used for the signatures. My way of maintaining the certificates, in my repository, would make it easy for the signing box to get all the existing certificates, but not (directly) the public keys that are inside those certificates.

Continue reading Re-signing Openssh Certificates

SSH Certificate signing

I’ve encountered a problem migrating from Fedora to Arch which ends up being about ssh and openssh certificates. I look back and discover that I never posted anything about my movement toward openssh certificates. Curious because I wrote a lengthy document about it (because of my leaky brain – not because I am any kind of authority on it).

I will probably go back and write a post about it, and back date it. But now a problem has arisen. Rather than explain, let the boys at openssh speak for themselves, in the release notes for openssh 8.2:

Continue reading SSH Certificate signing

Percona Toolkit

This is to help me remember how to get percona toolkit, so that I can get the grants information out of the mysql database – the database in mysql which keeps tracks of users, databases, and (in particular) grants, i.e. permissions.

I originally dug this up for use when I was migrating tarragon. But now I am working on migrating oregano from fedora to arch, and I will have to transfer the databases. I have finally learned (I admit I am slow) that one cannot expect to be able to move the database directory even from one release to another, let alone from one distribution to another. The only way to do this properly is to dump the databases, reinstall mariadb/mysql in the new place, and set it up, and then reimport the dumped databases.

Continue reading Percona Toolkit

Moving Oregano to arch

It is by no means certain that I will succeed with this effort, but I’m spending some time trying to get Oregano up on Arch.

The first step was just to get Arch booted up on oregano. My previous installation on a laptop didn’t involve an encrypted root, didn’t have raid arrays, didn’t have separate filesystems for things like /home and /var, didn’t run a web server, etc., so the first challenge is to get the system up with all that stuff.

Continue reading Moving Oregano to arch

Apache certificate chains

When I switched my main server to CentOS, described in an earlier post, one of the big pains was that I had to use CentOS 7, and there was a lot of software which had come a long way since CentOS 7, and I had to upgrade a log of things from upstream to get functionality that I had grown reliant upon.

I didn’t realize that Apache itself was one of those things that was sufficiently backwards in CentOS 7 that I would have trouble.

Ever since I move the server to CentOSdid that “upgrade”, I’ve been struggling with problems with the certificates not being honored. For the last few days I have been working pretty diligently to try to figure out this nagging problem, and today I finally figured it out. It is owing to an old Apache.

Continue reading Apache certificate chains

Odd VPN Problem

I have had trouble twice now with modifying a working vpn configuration, and then being unable to get it to start. Both times I never actually solved it, so much as eliminating the problem by switching to a different nordvpn config file.

There was a penetration at nordvpn in which some passwords and userinfo were leaked. I wanted to change my password, and did, and had to get into the vpn router and change it there. And after I did the vpn just would not start. Eventually, I switched to another vpn endpoint, put in a new .conf file in /etc/openvpn/client and it came right up.

I don’t know what this is about.