Category Archives: encryption

Setting Up Openvpn Server

The objective of this project was to install a vpn server on one of the boxes in the cloud (initially asafoetida, then moved to tarragon), in order to provide a VPN server service for a friend who was traveling. My friend uses the name Darrell for his client, so in what follows the vpn is called by this name.

A lot of the instructions, even from openvpn site, say to use the “easyrsa” package to generate the certificates for openvpn. This package seems to be put out by the openvpn boys, or at least with their cooperation. But I didn’t do that. I created a ca with raw openssl.

I built the ca on my laptop  in /etc/ssl/mycerts/ca. The root ca is in that directory, but a subdirectory called intermediate has an intermediate ca, and that is where the actual signing is done. I probably won’t leave it on the laptop. Not sure yet what I will do with the ca.

Once the ca is in place, we generate the certificates for server and client as follows:

First I generated 2048 private keys, one for server, one for client, with:
openssl genrsa -aes256 -out key.pem 2048

and generated the csr (one for each), using those keys with:
openssl req -new -sha256 -key key.pem -out csr.pem

There are directories within /etc/ssl/mycerts/ca/intermediate, certs/ for  the certs, and csr/ for the requests:

Now use the csr to generate the certs (one for each):
cd /etc/ssl/mycerts/ca/intermediate
openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions server_cert -in csr/darrell_server.csr -out certs/darrell_server2.cert.pem

openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions usr_cert -in csr/darrell_client.csr -out certs/darrell_client2.cert.pem

I had to reissue the certs because I failed to specify the “extensions” part, but you cannot reissue a cert with the same common_name (i.e. you can’t generate a new cert using the same csr) until you remove the old cert, which is a better practice anyway.
openssl ca -config openssl.cnf -revoke certs/darrell_server.cert.pem

Then I reissued the 2 certs, with the names server2 and client2.

I also created a diffie-hellman parameter for the link:
openssl dhparam -out dhparams.pem 4096

And I generated a ta key, which is apparently a shared key used to resist certain attacks:
openvpn --genkey --secret /etc/openvpn/ta.key

Ok, so on the server (asafoetida initially, later moved to tarragon), in /etc/openvpn/server, I have:
private/server.key ← this is the private key for darrell_server2.cert.pem

The file ca-chain.cert.pem has in it both the certificate for the intermediate ca and the certificate for the root ca which signed the intermediate. Both of these are needed on the server side, for sending to the client. Later, when we build the client side, it really only needs the root ca cert.

Initially, I started openvpn manually, with:
cd /etc/openvpn/server
openvpn server.conf

But see below discussion of automatic startup.

Client Side

On the client side (on lemongrass), in /etc/openvpn/client, I have these files:

And I start the client manually with:
cd /etc/openvpn/client
openvpn darrell.vpn.conf

The server is an amazon ec2 instance, and all ec2 instances have a firewall, called a “security group” provided by amazon. I used the same “security group” rules at the amazon level for asafoetida as for tarragon (http+mail+ssh, etc), but with an additional rule to admit udp 1194.

On asafoetida proper, there was not an additional iptables firewall – it had a default Accept-Accept-Accept filter table and no nat table. I added a couple of forward rules to filter, to accept forwarding of tun+ to eth0 and eth0 to tun+. Also I added a nat table postrouting rule to masquerade all output through eth0 to the vpn address.

On tarragon, the default redhat firewall sends forward traffic through the input chain (?), and I only had to accept udp 1194, and accept tun+., and add the nat table postrouting, so in /etc/sysconfig/iptables I added:
-A RH-Firewall-1-INPUT -i tun+ -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -p udp --dport 1194 -j ACCEPT

I had not previously had a nat table in the tarragon firewall, so I had to put one in:

The link gets made, and the routing table on both client and server seem to be adjusted properly, but traffic doesn’t flow out of the server. The problem turned out to be that I had failed to turn on ip forwarding on the box.
echo “1” > /proc/sys/net/ipv4/ip_forward

And to make that last, I put it in /etc/sysctl
net.sys.ipv4.ip_forward = 1

Here is the entire client config file /etc/openvpn/client/darrell.vpn.conf:
root@lemongrass:#cd /etc/openvpn/client
root@lemongrass:#cat darrell.vpn.conf
dev tun
proto udp
remote 1194
resolv-retry infinite
#tun-mtu 1500
#tun-mtu-extra 32
#mssfix 1450
ping 15
ping-restart 0
reneg-sec 0
#comp-lzo yes
explicit-exit-notify 3
verb 3
cipher AES-256-CBC
auth SHA512
cert darrell_client2.cert.pem
key client.key
remote-cert-tls server


key-direction 1
# 2048 bit OpenVPN static key
—–BEGIN OpenVPN Static key V1—–

—–END OpenVPN Static key V1—–

Starting Automatically

There are systemd instance scripts for starting a client (openvpn-client@) and for starting a server (openvpn-server@). So if you put a conf file in the /etc/openvpn/client directory named fred.conf, you can then do:
systemctl enable openvpn-client@fred

And similarly, for an openvpn server config named darrell.conf in /etc/openvpn/server:
systemctl enable openvpn-server@darrell

Certificates Redux

An earlier post talked about switching my server tarragon (where this blog sits) to a wildcard certificate from letsencrypt. There were two reasons why I was using a wildcard certificate. One had to do with test versions of websites that run on this server, and the need that some of those sites have for wildcards, of the form:,, etc. The other reason was that I have a lot of hosts (oregano, cinnamon, paprika, lemongrass) in addition to tarragon that “need” to have a certificate, for https, for imap, and for smtp, and when I was having to pay for them, it was cheaper to get one wildcard for Continue reading Certificates Redux

Apache Quiet Failure on Certificate/Key mismatch

In an earlier post I related how I had moved to letsencrypt for tarragon. In the process of doing some cleanup of the /etc/letsencypt directory, and my repository, I managed to stupidly get one wrong private key file into the batch of all the https vhosts, such that the http config file for specified an SSLCertificateFile and SSLCertificateKeyFile which did not match.

It took me hours and hours to figure this out, because Apache simply fails to start and gives no indication whatever what has pissed him off. I wasn’t too stupid to figure out that I had been messing with the certs yesterday, and the problem might lie there somehow. But I have about 15 vhosts, so it was tedious. In the end I resorted to strace, and saw the problem.


LetsEncrypt Wildcard Certificates

Comodo is after me to renew, offering a free year. The last time I attempted to install a wildcard certificate from Lets Encrypt, shortly after they introduced the feature, I wasn’t able to figure it out. Now, 9 months later, there is a lot more information about how to do it. Before spending the money for a commercial cert, I thought I would give it a try.

I used the following on tarragon:

certbot certonly \
--server \
-d -d * \
--preferred-challenges dns

It is important that the server url by v02, because v01 servers can’t issue wildcard certs. I had to put TXT records in the DNS for them to verify, and they created the cert into the /etc/letsencrypt/live directory where all the others are.

This was trivially easy. Goodbye Comodo.

Updating certificates to “Let’s Encrypt” with ACME

I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.

Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.

But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME

Fail: Resizing /boot on Cinnamon

The current 128GB (119.2GiB) Plextor M.2 disk on cinnamon was created with a very small (256MB/243MiB) /boot partition. Turns out this was a mistake, as it will only hold 3 kernels, so I’m constantly having to go in and remove kernel n-3 to make room.
The remainder of the drive is a single extended partition, containing a 119GiB LUKS container, which contains a 119GiB LVM volume group, which has a root of 107.3GiB and an 11.7GiB swap. The root is only 13% utilized, and while it may grow a little it is in no danger or running out of space. The swap is also far larger than needed. I want to steal a couple of GB from one of them to increase the /boot, but I want to do this while retaining the contents of both the / and the /boot.
Long story short, turns out I wasn’t able to do it. But the journey was a great learning experience for me, so I’ve kept the notes, for future reference.
Continue reading Fail: Resizing /boot on Cinnamon

Authentication Tokens

I have two websites. The first (on the server tarragon) is readily available on the internet to the public (you are looking at it now), and also has a username/password based login capability. Some selected people are able to get into the back end of this website. Mostly these are people who get their mail on tarragon, or who have websites on tarragon, or both. The login capability allows them to manage their own accounts, change their password, etc.

The second website (on the server oregano) is inaccessible (or at least non-functional) except to authorized users. The authorized users are exactly those people having a login credential on tarragon. The only way to achieve a usable connection to oregano is to first log on to tarragon, and click a link there. This will create a redirect to oregano, passing a token which will allow the connection to succeed. The website on oregano will politely decline to function unless an appropriate token is received.

This article is about building that token. The properties of the token are as follows. First it must provide the identity (username) of the user (the login used on tarragon). It must be encrypted, so that all (or at least part) of its contents are protected. It must not be replayable, that is, it should not be possible for someone to capture the token used by an authorized person, and reuse it later. This includes the provision that it must be time limited, the token should expire after a short time, and subsequently be useless. It should be possible to include other information in the token if needed. For exampe, the same token machinery can be (and is) used for sending an email to handle forgotten passwords, presuming that if joe has forgotten his password, we can send a link to joe’s email address and only joe will receive it.

In the first implementation, I thought to use joe’s password for the encryption. While this can be done, it is really a flawed plan, because I don’t actually have joe’s cleartext password, I only have the hashed version of it. Using joe’s hashed password as a key is obviously vulnerable to capturing the file containing the hashed passwords. The reason they are hashed at all, of course, is that capture of files full of user information occurs all too frequently. If tarragon is available on the internet, I am obliged to assume that a sufficiently motivated and funded attacker could get his hands on the user database. Of course, it is highly unlikely that tarragon would be an interesting target for such an attack. But just because the server doesn’t have national security secrets, that is no excuse to be sloppy.

So I reimplemented it using the certificates on the two machines. Both of the servers have certificates, and use tls for their connections, i.e. they are https instead of http sites. After a little futzing around and reading, I discovered a fairly straightforward way for the php code in server ‘a’ (tarragon) to capture the certificate for remote server ‘b’ (oregano), and extract it’s public key. Then tarragon can encrypt the token using oregano’s public key, so that only oregano can decrypt the token. I could go further, and encrypt again (actually first) using tarragon’s private key, so that oregano could verify that only tarragon could have sent it.

Actually, public key encryption isn’t really used for the token. Instead, a random key is chosen for a symmetric cipher, and the key itself is then encrypted with oregano’s public key, and subsequently decrypted on oregano with his private key. The encrypted key is sent along with the encrypted message.

The function I used in php is part of the openssl library in php, and is called openssl_seal (and the other end is openssl_open. There are lower level functions that would allow one to accomplish the same things, but these seemed straightforward to use. One problem however, is that openssl_seal is written to use RC4 for the cipher. RC4 is frowned upon as insecure in a number of contexts. Openssl_seal allows passing an additional parameter, to select a different cipher, but strangely has no provision for passing an initialization vector so one can’t use any cipher that requires an initialization vector. Eventually I decided to use AES in ECB mode, despite the problems with ECB. This passed syntax but failed horribly at run time – meaning the apache worker just seemed to disappear! In debugging it an error_log call before the openssl_seal was present, and an error_log call afterwards was not, and a surrounding try catch block was not triggered. WTF? It took two days to figure this out. So I just went back to not specifying a cipher and letting it use RC4, and it worked. For the moment at least, I’m leaving it with RC4, since I am only encoding a small token.

The trick to getting a remote certificate in PHP was to use the stream facility, which opens a socket. and a stream context which is a set of parameters to the stream. The stream context is set to use ssl, the socket is established to port 443, and then the stream context will happily yield up the peer certificate that it received during the tls negotiation.

Making MySQL serve UTF8 correctly

If the MySQL server decides that its default environment is UTF8, and that its client actually wants Latin1, it will translate the return values.

I’ve never before had to be careful of the distinction. Perhaps once in a blue moon I would have a record with a “real” quotation mark or a character with an accent, but if it didn’t work correctly, it was never much of a bother.

Once it became important, I had to understand what was happening. I have a table that has filenames in it, and some of those filenames contain characters (a acute, e grave, o umlaut, etc). The actual files on disk have the names encoded in utf8. The records in the database are also recorded in utf8. But the records were being translated by mysql from utf8 to latin1 as they came in. So “Mamá” was recorded on disk as Mam\xc3\xa1 in the directory, and in the database, but when I got the row into memory, the filename field said Mam\xe1. The difference between latin1 and utf8 for this purpose, is that all these many “western/latin special characters” were actually mapped in latin1 to values within the 256 characters available with 1 byte. So the first 128 in the latin1 codespace were ordinary ascii, and the high order 128 had as many of the western/latin diacriticals as possible crammed in there. And in latin1, e9 is a-acute.

But on the web these days, utf8 is much preferred. Latin1 is ok if all you want is the carefully selected subset of 128 characters that can be shoehorned into the high end of the code-point space. But utf8 is a far more general solution. Using a multibyte sequence to represent over a million characters and special symbols.

Turns out mysql has a bunch of variables to control character set and collating sequence. With phpmyadmin, one can look at database->Variables and see characters_set_database, character_set_filesystem, character_set_result, character_set_server, character_set_system, and a bunch more. Or in mysql client one can show varaiables like ‘%character_set%’;

My problem was that the server had come up believing that some of these were set to utf8 and some were set to latin1. I haven’t tried to figure out the logic of how it figures out its default – I don’t want it to default, I want to tell it what I want. So the solution was to add the directive: “character-set-server=utf8” to the mysql configuration file (on cinnamon it was /etc/mysql/mysql.conf.d/mysqld.conf. After restarting all of the relevant character_set_xxx variables come up as utf8.

Update: 8/9/17

I used these changes also on oregano and tarragon, but it results in a different problem for me on blogforacure data. The blogforacure database, built a long time ago, has lots of tables in latin1. There are not a lot of non-ascii characters, but there are a few. One frequent source is people typing double space after a period, which the ckeditor tries to preserve by creating a non-breaking space, which is hex A0 in latin 1. When the site reads this back, if mysql is told that the database is actually utf8, then it displays this as Â. So if I see a bunch of A circumflex in the output, it means I actually have latin 1 characters in the database, which I am interpreting as if they were utf-8.

Removing the specification of character-set-server=utf8 causes the negotiation to give the right result, and the latin1 non-breaking space appears correctly in the output.

Managing passwords on this server

This blog is running on my server, tarragon, in the Amazon cloud. This server, in addition to hosting this blog, hosts about 20-25 websites (for friends, most of them very low traffic), including my own. It also operates mail for myself and a few others, and provides some other services.
One of the weaknesses has been that most of the people who use the server aren’t really very unix literate, and they don’t really WANT to be. Perhaps they want a website, or they want to have a good place to manage their mail. But in general, the last thing they want is to learn how to ssh into the server to change their password.
So, for most of them, they just use whatever password I set up for them.
One of my friends, who just began using mail on the server, was surprised that it was not convenient to change his password. That spurred me to address the long standing problem. How to let people manage their password for access to services.
The blog now has a new menu on the left, for access to the backend, and for linking to the reset-password screen. There is also a reset password link on the login page
The same password is used for all the stuff: the password for access to mail, the password to get access to protected websites in apache, and the password for logging in to the backend website.
Continue reading Managing passwords on this server

Using pam_mysql for authentication

On the occasion of moving the server to amazon, I decided to stop using LDAP. I was making very little use of it, having started to keep my contacts elsewhere. All it was really doing was providing the authentication file for apache logins. And there were so few of those it was quite easy to manage with an htpasswd file.

But recently I’ve begun providing mail to some friends, and while I am happy to provide them with system accounts the problem is that they don’t really WANT system accounts – they just want mail. So the husband asked me, how do I change my password? And the only answer I have is, you have to log on. Worse than that of course, I can’t really even allow him to log on without making him set up for public key authentication.

I would like to enable people to use the server for mail, and to be able to authenticate with apache, and I would like them to manage their own passwords for this, without making them log on via ssh, which would require that they have a key pair registered with me. I looked into being able to change the system password via a webpage, but it looks very messy, and seriously – system passwords are supposed to require a human being – that is intentional.

So I looked into authenticating the mail, and apache, using a database. I found a pam module called pam_mysql which I can use with saslauthd. I set /etc/pam.d/imap and smtp to point to a new pam entry called mail, which uses pam.mysql to autheticate against the database (and still also authenticate against system accounts as well if there is no entry in the database).
Continue reading Using pam_mysql for authentication