Category Archives: security

Setting Up Openvpn Server

The objective of this project was to install a vpn server on one of the boxes in the cloud (initially asafoetida, then moved to tarragon), in order to provide a VPN server service for a friend who was traveling. My friend uses the name Darrell for his client, so in what follows the vpn is called by this name.

A lot of the instructions, even from openvpn site, say to use the “easyrsa” package to generate the certificates for openvpn. This package seems to be put out by the openvpn boys, or at least with their cooperation. But I didn’t do that. I created a ca with raw openssl.

I built the ca on my laptop  in /etc/ssl/mycerts/ca. The root ca is in that directory, but a subdirectory called intermediate has an intermediate ca, and that is where the actual signing is done. I probably won’t leave it on the laptop. Not sure yet what I will do with the ca.

Once the ca is in place, we generate the certificates for server and client as follows:

First I generated 2048 private keys, one for server, one for client, with:
openssl genrsa -aes256 -out key.pem 2048

and generated the csr (one for each), using those keys with:
openssl req -new -sha256 -key key.pem -out csr.pem

There are directories within /etc/ssl/mycerts/ca/intermediate, certs/ for  the certs, and csr/ for the requests:

Now use the csr to generate the certs (one for each):
cd /etc/ssl/mycerts/ca/intermediate
openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions server_cert -in csr/darrell_server.csr -out certs/darrell_server2.cert.pem

openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions usr_cert -in csr/darrell_client.csr -out certs/darrell_client2.cert.pem

I had to reissue the certs because I failed to specify the “extensions” part, but you cannot reissue a cert with the same common_name (i.e. you can’t generate a new cert using the same csr) until you remove the old cert, which is a better practice anyway.
openssl ca -config openssl.cnf -revoke certs/darrell_server.cert.pem

Then I reissued the 2 certs, with the names server2 and client2.

I also created a diffie-hellman parameter for the link:
openssl dhparam -out dhparams.pem 4096

And I generated a ta key, which is apparently a shared key used to resist certain attacks:
openvpn --genkey --secret /etc/openvpn/ta.key

Ok, so on the server (asafoetida initially, later moved to tarragon), in /etc/openvpn/server, I have:
private/server.key ← this is the private key for darrell_server2.cert.pem

The file ca-chain.cert.pem has in it both the certificate for the intermediate ca and the certificate for the root ca which signed the intermediate. Both of these are needed on the server side, for sending to the client. Later, when we build the client side, it really only needs the root ca cert.

Initially, I started openvpn manually, with:
cd /etc/openvpn/server
openvpn server.conf

But see below discussion of automatic startup.

Client Side

On the client side (on lemongrass), in /etc/openvpn/client, I have these files:

And I start the client manually with:
cd /etc/openvpn/client
openvpn darrell.vpn.conf

The server is an amazon ec2 instance, and all ec2 instances have a firewall, called a “security group” provided by amazon. I used the same “security group” rules at the amazon level for asafoetida as for tarragon (http+mail+ssh, etc), but with an additional rule to admit udp 1194.

On asafoetida proper, there was not an additional iptables firewall – it had a default Accept-Accept-Accept filter table and no nat table. I added a couple of forward rules to filter, to accept forwarding of tun+ to eth0 and eth0 to tun+. Also I added a nat table postrouting rule to masquerade all output through eth0 to the vpn address.

On tarragon, the default redhat firewall sends forward traffic through the input chain (?), and I only had to accept udp 1194, and accept tun+., and add the nat table postrouting, so in /etc/sysconfig/iptables I added:
-A RH-Firewall-1-INPUT -i tun+ -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -p udp --dport 1194 -j ACCEPT

I had not previously had a nat table in the tarragon firewall, so I had to put one in:

The link gets made, and the routing table on both client and server seem to be adjusted properly, but traffic doesn’t flow out of the server. The problem turned out to be that I had failed to turn on ip forwarding on the box.
echo “1” > /proc/sys/net/ipv4/ip_forward

And to make that last, I put it in /etc/sysctl
net.sys.ipv4.ip_forward = 1

Here is the entire client config file /etc/openvpn/client/darrell.vpn.conf:
root@lemongrass:#cd /etc/openvpn/client
root@lemongrass:#cat darrell.vpn.conf
dev tun
proto udp
remote 1194
resolv-retry infinite
#tun-mtu 1500
#tun-mtu-extra 32
#mssfix 1450
ping 15
ping-restart 0
reneg-sec 0
#comp-lzo yes
explicit-exit-notify 3
verb 3
cipher AES-256-CBC
auth SHA512
cert darrell_client2.cert.pem
key client.key
remote-cert-tls server


key-direction 1
# 2048 bit OpenVPN static key
—–BEGIN OpenVPN Static key V1—–

—–END OpenVPN Static key V1—–

Starting Automatically

There are systemd instance scripts for starting a client (openvpn-client@) and for starting a server (openvpn-server@). So if you put a conf file in the /etc/openvpn/client directory named fred.conf, you can then do:
systemctl enable openvpn-client@fred

And similarly, for an openvpn server config named darrell.conf in /etc/openvpn/server:
systemctl enable openvpn-server@darrell

Netfilter table order

II have a terrible time keeping this straight. This  is a memory aid. I think it is correct, but it has been assembled from internet sources and not from reading the code, so this is not authoritative.

  • Incoming packets:
      • raw table [PREROUTING]
      • contrack
      • mangle table [PREROUTING]
      • nat table [PREROUTING] (dnat)
      • route decision (either for us, or another host)
    • IF Destined for this host:
        • mangle table [INPUT]
        • filter table {INPUT]
        • security table [INPUT]
        • nat table [INPUT (rarely used)] (snat)
    • IF Destined for another host:
        • mangle table [FORWARD]
        • filter table [FORWARD]
        • security table [FORWARD]
      • (-> do output postrouting)
  • Packets generated on local host:
      • route decision (select interface, etc)
      • raw table [OUTPUT]
      • contrack
      • mangle table [OUTPUT (rarely used)]
      • nat table [OUTPUT (rarely used)] (dnat)
      • route decision
      • filter table [OUTPUT]
      • security table [OUTPUT}
    • (-> do output postrouting)
  • Output PostRouting
      • mangle table [POSTROUTING]
      • nat table [POSTROUTING] (snat/masquerade)

contrack means connection tracking,  unless the (always preceeding) raw table says not to track.

In nat table, PREROUTING chain can only use -i and do dnat; POSTROUTING chain can only use -o and do snat. What you can do with the nat INPUT and OUTPUT chains I don’t understand well, but I think INPUT can do snat and OUTPUT can do dnat.

Difference of snat vs masquerade on nat POSTROUTING: 1) for snat you set the ip address, for masquerade you give an interface, and it uses the address from that interface. Also Masquerade assumes the connection will change ip address when it goes down and comes back up, so throws away all connections (contrack) information.


Certificates Redux

An earlier post talked about switching my server tarragon (where this blog sits) to a wildcard certificate from letsencrypt. There were two reasons why I was using a wildcard certificate. One had to do with test versions of websites that run on this server, and the need that some of those sites have for wildcards, of the form:,, etc. The other reason was that I have a lot of hosts (oregano, cinnamon, paprika, lemongrass) in addition to tarragon that “need” to have a certificate, for https, for imap, and for smtp, and when I was having to pay for them, it was cheaper to get one wildcard for Continue reading Certificates Redux

Apache Quiet Failure on Certificate/Key mismatch

In an earlier post I related how I had moved to letsencrypt for tarragon. In the process of doing some cleanup of the /etc/letsencypt directory, and my repository, I managed to stupidly get one wrong private key file into the batch of all the https vhosts, such that the http config file for specified an SSLCertificateFile and SSLCertificateKeyFile which did not match.

It took me hours and hours to figure this out, because Apache simply fails to start and gives no indication whatever what has pissed him off. I wasn’t too stupid to figure out that I had been messing with the certs yesterday, and the problem might lie there somehow. But I have about 15 vhosts, so it was tedious. In the end I resorted to strace, and saw the problem.


LetsEncrypt Wildcard Certificates

Comodo is after me to renew, offering a free year. The last time I attempted to install a wildcard certificate from Lets Encrypt, shortly after they introduced the feature, I wasn’t able to figure it out. Now, 9 months later, there is a lot more information about how to do it. Before spending the money for a commercial cert, I thought I would give it a try.

I used the following on tarragon:

certbot certonly \
--server \
-d -d * \
--preferred-challenges dns

It is important that the server url by v02, because v01 servers can’t issue wildcard certs. I had to put TXT records in the DNS for them to verify, and they created the cert into the /etc/letsencrypt/live directory where all the others are.

This was trivially easy. Goodbye Comodo.

Recording last authentication

I wanted a way to be able to determine roughly how long it had been since a user had been active. I defined active to mean that the user had had to authenticate onto a system. This is so that a box on which the user had logged in has gone into screen lock, and the user has then authenticated again to the display manager.

I used the audit log of auditd to detect when a user has authenticated to a display manager. Auditd comes installed on Fedora, but I had to install it on the ubuntu boxes. Continue reading Recording last authentication

Running Apache under my username

I had some trouble on the development box with permissions and decided it would be “easy” to just have apache run under user dee, that would just make everything so much easier. Right.

Two things have come up so far, and more likely to follow. One, I had to change the ownership on /var/lib/php/session from apache to dee. Second, I had to add dee into tlsusers so the media stuff can read the certificate.

This may have been a bad idea.

1/31/18: Went back to using apache user, when I moved to Fedora 27. Fedora now uses php-fpm service, and now apache needs to open a socket to it, and it just became complicated.

Updating certificates to “Let’s Encrypt” with ACME

I’ve used a variety of certificate providers over the years, Thawte, CA-Cert, Verisign, Comodo, Startcom. Until about six months ago I was using Startcom, and had spent a fair amount of energy setting that up for my own site (this one) as well as all the other sites I manage.

Then Wo-Sign acquired Startcom, and browsers starting distrusting Startcom. I ended up buying a cert from Comodo for this site.

But then I found out about Let’s Encrypt. Not only are they free, but they have this whole ACME auto update thing worked out, using various ACME clients. I’ve been using Certbot from EFF. Continue reading Updating certificates to “Let’s Encrypt” with ACME

Disabling ipv6

I was really keen to get ipv6, and very pleased when my provider (wretched hateful Comcast) finally gave me an ipv6 address.

What I didn’t know was that most VPN providers, including the one I use, are only tunneling the ipv4 traffic. So if you make a connection to a location that supports ipv6 (has an AAAA record) the connection will completely bypass the vpn tunnel.

I have temporarily turned off ipv6 on the external interface by putting this into /etc/sysctl.conf:

net.ipv6.conf.enp4s1.disable_ipv6 = 1

Saslauth, mail and realms

This server (tarragon) runs a postfix instance which provides mail service for my own as well as for about a dozen other domains belonging to friends and clients. Postfix offers three different ways that a server can receive (be the final destination for) mail directed to a domain:

1) as what postfix calls the canonical destination (i.e. mail for where tarragon IS, and each mail recipient maps onto a user who has a login account on the server, and messages are delivered to that account;

2) as a virtual alias destination, where mail directed to is accepted, but for each such address there is a corresponding forward address to some other location or something, and the actual mail messages do not reside on the server; and finally

3) as a virtual mailbox destination, where mail directed to arrives and is stored in mailboxes on tarragon, awaiting pickup/reading by the user, but without requiring that there be a user z with an actual login account on tarragon. This requires that the mail store on tarragon be set up to maintain different sets of mailboxes for different domains. There can be a user and another user and the mail is not intermixed.

Tarragon uses cyrus-imap as the mail store, and it provides the ability to have different mailboxes for different domains. To support that, the mailboxes are actually constructed differently, so that cyrus-imap can have a mailbox fred, but can also have a mailbox

This requires, in turn, that the imap server be able to identify the correct mailbox when a mail client attaches, and be able to separately authenticate for each mailbox. When cyrus-imap is configured to support this separation, it requires that the username on login be, rather than simply fred.

Cyrus-imap uses the saslauthd daemon to authenticate, and saslauthd in turn calls upon pam, passing in the username,  password and realm (domain) received from imap (or postfix for smtp, or apache for website auth), who receives it in the login message from the user’s mail client. Pam’s authentication for mail is set to use a module called pam_mysql, which is able to match against credentials in a mysql database.

Here is where things get tricky. Take the mail account There is also a user dee with a system account (i.e a type 1 canonical mail account I can choose either to have a) only one entry in the database, for user dee, with a password. That same entry is consulted for access to either mailbox ( or but the are still separate mailboxes. Or alternatively, b) I can have different database entries for dee and, each with its own password. 

A digression: I could, and for many years did, choose to list as a canonical final destination in postfix. If I do that, then mail for goes into the mailbox for user dee on tarragon, just the same as mail for They go into the same mailbox. But when I began supporting virtual mailbox domains, I separated them – using the gray geek account as a test case for hosting virtual mailbox domains. 

Originally I set it up with the idea that there would be seapate database entries. With them separated, an imap login for user dee at host will attach to the mailbox for the user dee, while an imap login for user at host will attach to the mailbox for user who does not have an account on tarragon.

I set all this up over a year ago, and it seemed to be working. Then I set up a new account for a friend who had a new domain name. And I discovered that I had a problem. It so happened that in every case where I had created a mailbox of the form, I actually also had an account fred on, many of those never used and left over from the days when I was only doing canonical logins. I discovered that even though I had entries in the database for login as, the login process was actually using the database entry for fred. It so happened that all these accounts (fred and fred@) had the same password. As soon as I added an account bob@ which did NOT have a corresponding server login account bob, with the same password, it failed.

When I tried using testsaslauthd -u -p <pw> it would work, so the pam machinery and the pam_mysql plugin were working right. The problem occurred between cyrus-imap and saslauthd. I discovered that (a) cyrus imap takes an incoming username of a@b and separates it into “username” and “realm”, and passes those separately to saslauthd, and (b) saslauthd has a parameter ‘-r’ which I had previously failed to discover, which causes it to append the incoming ‘realm’ to the incoming ‘username’ when it attempts to authenticate. Without the ‘-r’ parameter, saslauthd was using only the incoming ‘username’ – fred or bob, in its call on pam. If there was such an account and the password matched, saslauthd would succeed, and the connection would be permitted. Note that cyrus-imapd would now actually connect to the correct mailbox – the issue was that it used the wrong database entry to authenticate against.

Once I turned on the ‘-r’ parameter to saslauthd, imap works for both dee and, but I get a problem in smtp.  On the call from smtp->saslauthd->pam_mysql when saslauthd has the -r parameter simply ‘dee’ as the username will not authenticate. The log record coming from pam_mysql indicates the username comes across as ‘dee’, but fails. If I create a database entry for, and set the username on smtp to, it works.

After a couple of weeks, I began to change my mind. The database is used for several kinds of logins. Going this way – with separate logins for and bob meant: I have to have 2 records in the database. Bob could get confused if he changed his password on one but not the other. For some users I could probably just delete the ‘bob’ entry, and force bob to type in his full email address in order to log in for other things. But I decided this was unappealing.

So I have turned off the ‘r’ parameter, and eliminated all the entries that say from the database. Bob must still use his full email address when connecting to imap but for plain logins in other places, he can use just bob. This also means I no longer have to have a separate database record for canonical mail users to do smtp.