Category Archives: apache

Certificates Redux

An earlier post talked about switching my server tarragon (where this blog sits) to a wildcard certificate from letsencrypt. There were two reasons why I was using a wildcard certificate. One had to do with test versions of websites that run on this server, and the need that some of those sites have for wildcards, of the form: bob.websitename.com, sally.websitename.com, etc. The other reason was that I have a lot of hosts (oregano, cinnamon, paprika, lemongrass) in addition to tarragon that “need” to have a certificate, for https, for imap, and for smtp, and when I was having to pay for them, it was cheaper to get one wildcard for wmbuck.net. Continue reading Certificates Redux

Apache Quiet Failure on Certificate/Key mismatch

In an earlier post I related how I had moved to letsencrypt for tarragon. In the process of doing some cleanup of the /etc/letsencypt directory, and my repository, I managed to stupidly get one wrong private key file into the batch of all the https vhosts, such that the http config file for xyz.com specified an SSLCertificateFile and SSLCertificateKeyFile which did not match.

It took me hours and hours to figure this out, because Apache simply fails to start and gives no indication whatever what has pissed him off. I wasn’t too stupid to figure out that I had been messing with the certs yesterday, and the problem might lie there somehow. But I have about 15 vhosts, so it was tedious. In the end I resorted to strace, and saw the problem.

 

LetsEncrypt Wildcard Certificates

Comodo is after me to renew, offering a free year. The last time I attempted to install a wildcard certificate from Lets Encrypt, shortly after they introduced the feature, I wasn’t able to figure it out. Now, 9 months later, there is a lot more information about how to do it. Before spending the money for a commercial cert, I thought I would give it a try.

I used the following on tarragon:


certbot certonly \
--server https://acme-v02.api.letsencrypt.org/directory \
--manual
-d wmbuck.net -d *.wmbuck.net \
--preferred-challenges dns

It is important that the server url by v02, because v01 servers can’t issue wildcard certs. I had to put TXT records in the DNS for them to verify, and they created the cert into the /etc/letsencrypt/live directory where all the others are.

This was trivially easy. Goodbye Comodo.

SVN Error E175002

I have been plagued by this error in subversion particularly when trying to commit from some of the boxes which I use less frequently:

svn: E175002: Unexpected HTTP status 200 ‘OK’ on ‘POST’ request to ‘/svn/!svn/me’

I have spent hours doing searches, reading posts, but have never found anyone whose issue was exactly like mine, not been able to figure it out based on other peoples issues. I resolved today to pay serious attention to figuring it out.

The solution turned out to be related to the url I used when I check something out of the svn repository. Long ago I set up a cname in dns for svn.wmbuck.net, and for a long time I used it. There is an apache config file for the servername svn.wmbuck.net, and it redirects http to https. Then at some point I began to just use https://wmbuck.net/svn/… to check things out. And that is where I went wrong, because that will work fine to do checkout, but when I try to commit from a box with that url (wmbuck.net) the http request is being routed to the default server, and the setup of the SSL session is failing.

I’m unsure exactly what is happening to cause the request to go to the default server. Perhaps the commit request does not specify SNI information.

What I do know is how to fix it. Do the checkout with https://svn.wmbuck.net/svn/ and commits work fine.

Running Apache under my username

I had some trouble on the development box with permissions and decided it would be “easy” to just have apache run under user dee, that would just make everything so much easier. Right.

Two things have come up so far, and more likely to follow. One, I had to change the ownership on /var/lib/php/session from apache to dee. Second, I had to add dee into tlsusers so the media stuff can read the certificate.

This may have been a bad idea.

1/31/18: Went back to using apache user, when I moved to Fedora 27. Fedora now uses php-fpm service, and now apache needs to open a socket to it, and it just became complicated.

Apache module debugging

I have some problem using mod_xsendfile on tarragon. I’ve been working on getting this working for 2 days. I have had to get into the source code of the apache module to figure it out, and I want to turn on the debugging option to see what is going on.

So I have to recompile the c source file, with the define of _DEBUG, and install the it as a module. Had to figure out how to do this. It is very easy. But easy doesn’t mean I’ll remember it, thus this post.

I cloned the source:

git clone https://github.com/nmaier/mod_xsendfile (into my local git repo), and then cd into the directory, and

apxs -D_DEBUG -i -c mod_xsendfile.c

This creates the module with debugging defined, and puts it in /usr/lib64/httpd/modules/mod_xsendfile.so

It still needs to be loaded into apache. Instructions at his site say to use the -a flag, to activate, but while that would work on a simple site, it tries to put the LoadModule into /etc/httpd/conf/httpd.conf, and all my LoadModule statements are in files in the directory /etc/httpd/conf.modules.d so I need to create /etc/httpd/conf.modules.d/xsendfile.conf containing:

LoadModule xsendfile_module modules/mod_xsendfile.so

The module will log debug statements, but this still won’t actually get you any log records until you set LogLevel debug in the apache config file.

Then restart apache and Bob’s your uncle.

Updating certificates to “Let’s Encrypt” with ACME

I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.

Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.

But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME

Webalizer

I wanted to know what sort of traffic was being generated by the various websites I host.  So I set up webalizer. Every time I decide to use this, I have to figure it all out again.

Created directory /etc/webalizer.d/ with an entry for each website, e.g. /etc/webalizer.d/fred for fred.com, containing a copy of the /etc/webalizer.conf sample file, with the name of the log file changed and the output going into /var/www/usage/fred

Created backup_scripts/webalizer containing:

for W in /etc/webalizer.d/*.conf; do /usr/bin/webalizer -c $W; done

Started the script from crontab.

In apache, I have a 443 virtualhost for usage.wmbuck.net which gives me the statistics for website fred when invoked with https://usage.wmbuck.net/fred.

Ubuntu Javascript Fail

The “out of the box” apache on Ubuntu comes with a “feature” called “javascript-common” enabled. I haven’t got much idea what this wretched thing does, other than screw me up. I remember now that I had to struggle this some time back on Cinnamon. Now I am in Ohio trying to get something running on a box there, and tearing my hair out once again over the same issue.

This feature adds an Alias directive that takes the directory “/javascript” and sends it off to /usr/share/javascript. So if you are foolish enough to have a directory in your website called “/javascript” (and who would ever dream of putting their javascript files in a directory called javascript, after all) it will fail.

The directory /usr/share/javascript has some nice stuff in it, including jquery, and I guess it is a nice convenience feature for some people. But am I the only one who things it is crazy for a distribution to do something that breaks websites that have a commonly named directory like javascript!?

Authentication Tokens

I have two websites. The first (on the server tarragon) is readily available on the internet to the public (you are looking at it now), and also has a username/password based login capability. Some selected people are able to get into the back end of this website. Mostly these are people who get their mail on tarragon, or who have websites on tarragon, or both. The login capability allows them to manage their own accounts, change their password, etc.

The second website (on the server oregano) is inaccessible (or at least non-functional) except to authorized users. The authorized users are exactly those people having a login credential on tarragon. The only way to achieve a usable connection to oregano is to first log on to tarragon, and click a link there. This will create a redirect to oregano, passing a token which will allow the connection to succeed. The website on oregano will politely decline to function unless an appropriate token is received.

This article is about building that token. The properties of the token are as follows. First it must provide the identity (username) of the user (the login used on tarragon). It must be encrypted, so that all (or at least part) of its contents are protected. It must not be replayable, that is, it should not be possible for someone to capture the token used by an authorized person, and reuse it later. This includes the provision that it must be time limited, the token should expire after a short time, and subsequently be useless. It should be possible to include other information in the token if needed. For exampe, the same token machinery can be (and is) used for sending an email to handle forgotten passwords, presuming that if joe has forgotten his password, we can send a link to joe’s email address and only joe will receive it.

In the first implementation, I thought to use joe’s password for the encryption. While this can be done, it is really a flawed plan, because I don’t actually have joe’s cleartext password, I only have the hashed version of it. Using joe’s hashed password as a key is obviously vulnerable to capturing the file containing the hashed passwords. The reason they are hashed at all, of course, is that capture of files full of user information occurs all too frequently. If tarragon is available on the internet, I am obliged to assume that a sufficiently motivated and funded attacker could get his hands on the user database. Of course, it is highly unlikely that tarragon would be an interesting target for such an attack. But just because the server doesn’t have national security secrets, that is no excuse to be sloppy.

So I reimplemented it using the certificates on the two machines. Both of the servers have certificates, and use tls for their connections, i.e. they are https instead of http sites. After a little futzing around and reading, I discovered a fairly straightforward way for the php code in server ‘a’ (tarragon) to capture the certificate for remote server ‘b’ (oregano), and extract it’s public key. Then tarragon can encrypt the token using oregano’s public key, so that only oregano can decrypt the token. I could go further, and encrypt again (actually first) using tarragon’s private key, so that oregano could verify that only tarragon could have sent it.

Actually, public key encryption isn’t really used for the token. Instead, a random key is chosen for a symmetric cipher, and the key itself is then encrypted with oregano’s public key, and subsequently decrypted on oregano with his private key. The encrypted key is sent along with the encrypted message.

The function I used in php is part of the openssl library in php, and is called openssl_seal (and the other end is openssl_open. There are lower level functions that would allow one to accomplish the same things, but these seemed straightforward to use. One problem however, is that openssl_seal is written to use RC4 for the cipher. RC4 is frowned upon as insecure in a number of contexts. Openssl_seal allows passing an additional parameter, to select a different cipher, but strangely has no provision for passing an initialization vector so one can’t use any cipher that requires an initialization vector. Eventually I decided to use AES in ECB mode, despite the problems with ECB. This passed syntax but failed horribly at run time – meaning the apache worker just seemed to disappear! In debugging it an error_log call before the openssl_seal was present, and an error_log call afterwards was not, and a surrounding try catch block was not triggered. WTF? It took two days to figure this out. So I just went back to not specifying a cipher and letting it use RC4, and it worked. For the moment at least, I’m leaving it with RC4, since I am only encoding a small token.

The trick to getting a remote certificate in PHP was to use the stream facility, which opens a socket. and a stream context which is a set of parameters to the stream. The stream context is set to use ssl, the socket is established to port 443, and then the stream context will happily yield up the peer certificate that it received during the tls negotiation.