arch linux, encryption, fedora, gateways, linux, networking, security

SSH Certificate signing

I’ve encountered a problem migrating from Fedora to Arch which ends up being about ssh and openssh certificates. I look back and discover that I never posted anything about my movement toward openssh certificates. Curious because I wrote a lengthy document about it (because of my leaky brain – not because I am any kind of authority on it).

I will probably go back and write a post about it, and back date it. But now a problem has arisen. Rather than explain, let the boys at openssh speak for themselves, in the release notes for openssh 8.2:

Continue reading SSH Certificate signing →
arch linux, fedora, linux, mysql

Percona Toolkit

This is to help me remember how to get percona toolkit, so that I can get the grants information out of the mysql database – the database in mysql which keeps tracks of users, databases, and (in particular) grants, i.e. permissions.

I originally dug this up for use when I was migrating tarragon. But now I am working on migrating oregano from fedora to arch, and I will have to transfer the databases. I have finally learned (I admit I am slow) that one cannot expect to be able to move the database directory even from one release to another, let alone from one distribution to another. The only way to do this properly is to dump the databases, reinstall mariadb/mysql in the new place, and set it up, and then reimport the dumped databases.

Continue reading Percona Toolkit →
arch linux, boot issues, fedora, linux, mysql, networking

Moving Oregano to arch

It is by no means certain that I will succeed with this effort, but I’m spending some time trying to get Oregano up on Arch.

The first step was just to get Arch booted up on oregano. My previous installation on a laptop didn’t involve an encrypted root, didn’t have raid arrays, didn’t have separate filesystems for things like /home and /var, didn’t run a web server, etc., so the first challenge is to get the system up with all that stuff.

Continue reading Moving Oregano to arch →
apache, centos, cloud, encryption, linux, networking, security, website building

Apache certificate chains

When I switched my main server to CentOS, described in an earlier post, one of the big pains was that I had to use CentOS 7, and there was a lot of software which had come a long way since CentOS 7, and I had to upgrade a log of things from upstream to get functionality that I had grown reliant upon.

I didn’t realize that Apache itself was one of those things that was sufficiently backwards in CentOS 7 that I would have trouble.

Ever since I move the server to CentOSdid that “upgrade”, I’ve been struggling with problems with the certificates not being honored. For the last few days I have been working pretty diligently to try to figure out this nagging problem, and today I finally figured it out. It is owing to an old Apache.

Continue reading Apache certificate chains →
ipv6, linux, networking, security, vpn

Odd VPN Problem

I have had trouble twice now with modifying a working vpn configuration, and then being unable to get it to start. Both times I never actually solved it, so much as eliminating the problem by switching to a different nordvpn config file.

There was a penetration at nordvpn in which some passwords and userinfo were leaked. I wanted to change my password, and did, and had to get into the vpn router and change it there. And after I did the vpn just would not start. Eventually, I switched to another vpn endpoint, put in a new .conf file in /etc/openvpn/client and it came right up.

I don’t know what this is about.

March 26, 2021: I spent all morning on this again. I was changing scripts on Rosemary and also, double and triple checking that I did not allow IPv6 on obelisk/rosemary (if routable IPv6 addresses are available they will be used in preference to the IPv4 vpn, which is the whole point of obelisk). After a reboot of obelisk, it lost dns. I spent several hours trying to solve this, most of the time spent on obelisk. The vpn would come up, and I could ping raw addresses, but the dns wouldn’t work. I don’t even see what this has to do with the vpn tunnel.

Yet in the end, out of desperation, I tried bringing in a new nordvpn conf file (actually it comes in as an ovpn file, and once I put a password link in it it is saved as a conf file). A link in /etc/openvpn points to whichever of these is active. So I installed a new one, rebooted the router (again), and like magic it began to work.

centos, eclipse, fedora, linux

Desktop Files

For a very goofy reason involving a bug with composer, and new warning messages in PHP, I decided I needed to reinstall eclipse on oregano. Reinstalling eclipse is always a test of my patience. During the entire course of my life I do not believe that the installation of eclipse along with the various components I need, especially subclipse, has ever “just worked”. There is always some issue. So, of course, I do not look forward to it with gleeful anticipation.

I think part of the problem for me – a problem of my own making I suppose – is that I mostly use distros (Fedora and Ubuntu) in which the packagers have already arranged for tools like eclipse to appear as “system level” rather than “user level” tools. By this I mean they appear (or are symlinked from) /usr/bin, their configuration files are located it /etc, their icons appear when you search for apps, etc. Whereas if you just use the upstream installers the installation will be carried out at the user level, they will ask you where to put the binary, and they will put configuration files in ~/.config,

Continue reading Desktop Files →
apache, backup, centos, cloud, fedora, linux, mail, postfix, security, website building

Clamd signatures and Apache memory

After implementing the new tarragon the biggest problem I had involved the clamav package, and its loading of signatures. If clamd doesn’t come up and open its socket, then amavisd (the daemon who is consulted by postfix to handle all the checking of each piece of mail on input and output) will fail (assuming he is configured to do virus checking), This results in various problems. Amavis will mark the mail as “unchecked”, but worse, it will report failure back to postfix who gets confused and very often the message is delivered two or three times.

Clamd, the clamav daemon, now has over 6 million signatures. There are a lot of bad boys out there. The signatures are loaded by clamd from its database (in /var/lib/clamav) on startup, into memory. As a result, clamd has a large memory footprint, almost 800Mb on my system. The first issue, discovered before going live, was that systemd’s default parameters expect any daemon he starts to load within 90 seconds. If it fails to check in within that time, systemd considers it broken and terminates it. Clamd takes at least 3 minutes to load. I had to set a special TimeoutStartSec value in the systemd service script for clamd@.service.

Whew! I thought, boy I’m glad I figured that out. Hah!

Continue reading Clamd signatures and Apache memory →
apache, backup, btrfs, centos, cloud, fedora, linux, mail, mysql, networking, postfix, website building

Tarragon Rebuild 2019

This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.

The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.

Continue reading Tarragon Rebuild 2019 →
backup, gateways, linux, networking, raspberry pi, ubuntu

Memory on the Gateway Pi

I now have 8 of these gateway boxes out there. This morning as I was checking backups on one of them, I observed that it took quite a long time to respond. I ran a top on it and was horrified to see that its memory use was 100% and so was its swap. Holy @#$%!@ Batman!

Most of the memory was being used by the lxpanel. And (hangs head in embarrassment) there were actually two lxpanels running – one for the console and one in the vnc window I launch at startup.

It seems the lxpanels leak. I don’t know how badly, but it doesn’t matter. These boxes are meant to run forever so even a tiny leak is eventually fatal.

Well this was simple. I will seldom, if ever, need to get into a graphical environment remotely, and if I do I can always start vnc from the command line. So I took out the startvnc from the startup script. And I have even LESS need for a graphical console since there is not even a monitor on these things. So I set the default systemd target to multi-user.target.

Did this on all the gateways that are running on pi-zeros. Those few running on bigger ubuntu boxes I didn’t really have the problem anyway.

After rebooting them they come up with no lxpanels. I’ll watch the memory use, but I think this will fix the problem.