When I switched my main server to CentOS, described in an earlier post, one of the big pains was that I had to use CentOS 7, and there was a lot of software which had come a long way since CentOS 7, and I had to upgrade a log of things from upstream to get functionality that I had grown reliant upon.
I didn’t realize that Apache itself was one of those things that was sufficiently backwards in CentOS 7 that I would have trouble.
Ever since I move the server to CentOSdid that “upgrade”, I’ve been struggling with problems with the certificates not being honored. For the last few days I have been working pretty diligently to try to figure out this nagging problem, and today I finally figured it out. It is owing to an old Apache.
Continue reading Apache certificate chains
After implementing the new tarragon the biggest problem I had involved the clamav package, and its loading of signatures. If clamd doesn’t come up and open its socket, then amavisd (the daemon who is consulted by postfix to handle all the checking of each piece of mail on input and output) will fail (assuming he is configured to do virus checking), This results in various problems. Amavis will mark the mail as “unchecked”, but worse, it will report failure back to postfix who gets confused and very often the message is delivered two or three times.
Clamd, the clamav daemon, now has over 6 million signatures. There are a lot of bad boys out there. The signatures are loaded by clamd from its database (in /var/lib/clamav) on startup, into memory. As a result, clamd has a large memory footprint, almost 800Mb on my system. The first issue, discovered before going live, was that systemd’s default parameters expect any daemon he starts to load within 90 seconds. If it fails to check in within that time, systemd considers it broken and terminates it. Clamd takes at least 3 minutes to load. I had to set a special TimeoutStartSec value in the systemd service script for clamd@.service.
Whew! I thought, boy I’m glad I figured that out. Hah!
Continue reading Clamd signatures and Apache memory
This server, on Amazon, hosts my website and a dozen others, provides mail service for several people’s email including my own with postfix, dovecot, opendkim, amavis, spamassassin and clamd, provides contacts and calendar service using radicale, provides vpn service with openvpn, provides a tor relay, provides nextcloud service, and hosts my svn repository.
The server was last rebuilt in 2017. Long, long ago when I built the first version of it, I was most familiar with Red Hat/Fedora, and since then it has been easiest just to upgrade it with Fedora, always grumbling to myself that someday I’m going to change it. The problem with being on Fedora, of course, is that Fedora changes every 6 months, so I’m constantly behind. And after a year I’m at end of life. This is dumb for a server that I don’t want to be messing with all the time.
Continue reading Tarragon Rebuild 2019
An earlier post talked about switching my server tarragon (where this blog sits) to a wildcard certificate from letsencrypt. There were two reasons why I was using a wildcard certificate. One had to do with test versions of websites that run on this server, and the need that some of those sites have for wildcards, of the form: bob.websitename.com, sally.websitename.com, etc. The other reason was that I have a lot of hosts (oregano, cinnamon, paprika, lemongrass) in addition to tarragon that “need” to have a certificate, for https, for imap, and for smtp, and when I was having to pay for them, it was cheaper to get one wildcard for wmbuck.net. Continue reading Certificates Redux
In an earlier post I related how I had moved to letsencrypt for tarragon. In the process of doing some cleanup of the /etc/letsencypt directory, and my repository, I managed to stupidly get one wrong private key file into the batch of all the https vhosts, such that the http config file for xyz.com specified an SSLCertificateFile and SSLCertificateKeyFile which did not match.
It took me hours and hours to figure this out, because Apache simply fails to start and gives no indication whatever what has pissed him off. I wasn’t too stupid to figure out that I had been messing with the certs yesterday, and the problem might lie there somehow. But I have about 15 vhosts, so it was tedious. In the end I resorted to strace, and saw the problem.
Comodo is after me to renew, offering a free year. The last time I attempted to install a wildcard certificate from Lets Encrypt, shortly after they introduced the feature, I wasn’t able to figure it out. Now, 9 months later, there is a lot more information about how to do it. Before spending the money for a commercial cert, I thought I would give it a try.
I used the following on tarragon:
certbot certonly \
--server https://acme-v02.api.letsencrypt.org/directory \
-d wmbuck.net -d *.wmbuck.net \
It is important that the server url by v02, because v01 servers can’t issue wildcard certs. I had to put TXT records in the DNS for them to verify, and they created the cert into the /etc/letsencrypt/live directory where all the others are.
This was trivially easy. Goodbye Comodo.
I have been plagued by this error in subversion particularly when trying to commit from some of the boxes which I use less frequently:
svn: E175002: Unexpected HTTP status 200 ‘OK’ on ‘POST’ request to ‘/svn/!svn/me’
I have spent hours doing searches, reading posts, but have never found anyone whose issue was exactly like mine, not been able to figure it out based on other peoples issues. I resolved today to pay serious attention to figuring it out.
The solution turned out to be related to the url I used when I check something out of the svn repository. Long ago I set up a cname in dns for svn.wmbuck.net, and for a long time I used it. There is an apache config file for the servername svn.wmbuck.net, and it redirects http to https. Then at some point I began to just use https://wmbuck.net/svn/… to check things out. And that is where I went wrong, because that will work fine to do checkout, but when I try to commit from a box with that url (wmbuck.net) the http request is being routed to the default server, and the setup of the SSL session is failing.
I’m unsure exactly what is happening to cause the request to go to the default server. Perhaps the commit request does not specify SNI information.
What I do know is how to fix it. Do the checkout with https://svn.wmbuck.net/svn/ and commits work fine.
I had some trouble on the development box with permissions and decided it would be “easy” to just have apache run under user dee, that would just make everything so much easier. Right.
Two things have come up so far, and more likely to follow. One, I had to change the ownership on /var/lib/php/session from apache to dee. Second, I had to add dee into tlsusers so the media stuff can read the certificate.
This may have been a bad idea.
1/31/18: Went back to using apache user, when I moved to Fedora 27. Fedora now uses php-fpm service, and now apache needs to open a socket to it, and it just became complicated.
I have some problem using mod_xsendfile on tarragon. I’ve been working on getting this working for 2 days. I have had to get into the source code of the apache module to figure it out, and I want to turn on the debugging option to see what is going on.
So I have to recompile the c source file, with the define of _DEBUG, and install the it as a module. Had to figure out how to do this. It is very easy. But easy doesn’t mean I’ll remember it, thus this post.
I cloned the source:
git clone https://github.com/nmaier/mod_xsendfile (into my local git repo), and then cd into the directory, and
apxs -D_DEBUG -i -c mod_xsendfile.c
This creates the module with debugging defined, and puts it in /usr/lib64/httpd/modules/mod_xsendfile.so
It still needs to be loaded into apache. Instructions at his site say to use the -a flag, to activate, but while that would work on a simple site, it tries to put the LoadModule into /etc/httpd/conf/httpd.conf, and all my LoadModule statements are in files in the directory /etc/httpd/conf.modules.d so I need to create /etc/httpd/conf.modules.d/xsendfile.conf containing:
LoadModule xsendfile_module modules/mod_xsendfile.so
The module will log debug statements, but this still won’t actually get you any log records until you set LogLevel debug in the apache config file.
Then restart apache and Bob’s your uncle.
I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.
Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.
But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME