Setting Up Openvpn Server

The objective of this project was to install a vpn server on one of the boxes in the cloud (initially asafoetida, then moved to tarragon), in order to provide a VPN server service for a friend who was traveling. My friend uses the name Darrell for his client, so in what follows the vpn is called by this name.

Create a Certificate Authority

A lot of the instructions, even from openvpn site, say to use the “easyrsa” package to generate the certificates for openvpn. This package seems to be put out by the openvpn boys, or at least with their cooperation. But I didn’t do that. I created a ca with raw openssl.

I built the ca on my laptop  in /etc/ssl/mycerts/ca. The root ca is in that directory, but a subdirectory called intermediate has an intermediate ca, and that is where the actual signing is done. I probably won’t leave it on the laptop. Not sure yet what I will do with the ca.

Generate the private keys and certificates

Once the ca is in place, we generate the certificates for server and client as follows:

First I generated 2048 private keys, one for server, one for client, with:
openssl genrsa -aes256 -out client.key 2048
openssl genrsa -aes256 -out server.key 2048

and generated the csr (one for each), using those keys with:
openssl req -new -sha256 -key client.key -out client.csr
openssl req -new -sha256 -key server.key -out server.csr

There are directories within /etc/ssl/mycerts/ca/intermediate, certs/ for the certs, and csr/ for the requests:

Now use the csr to generate the certs (one for each):
cd /etc/ssl/mycerts/ca/intermediate
openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions server_cert -in csr/server.csr -out certs/darrell_server.cert.pem

openssl ca -config openssl.cnf -days 365 -notext -md sha256 -extensions usr_cert -in csr/client.csr -out certs/darrell_client.cert.pem

I also created a diffie-hellman parameter for the link:
openssl dhparam -out dhparams.pem 4096

And I generated a “ta key”, which is apparently a shared key used to resist certain attacks:
openvpn --genkey --secret /etc/openvpn/ta.key

Also on both server and client sides, I had to build the config files for openvpn. They are called, respectively, darrell.server.conf and darrell.vpn.conf. Maybe I should have called the latter darrell.client.conf. The client side is shown later on. Here is the server side configuration file:

[root@tarragon server]# cat darrell.server.conf
port 1194
proto udp
dev tun
ca /etc/openvpn/server/ca-chain.cert.pem
cert /etc/openvpn/server/certs/darrell_server.cert.pem
key /etc/openvpn/server/private/server.key
dh /etc/openvpn/server/dhparams.pem
topology subnet
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS"
push "dhcp-option DNS"
keepalive 10 120
tls-auth /etc/openvpn/server/ta.key 0
cipher AES-256-CBC
compress lz4-v2
push "compress lz4-v2"
user nobody
group nobody
status /etc/openvpn/server/darrell-status.log
verb 3
explicit-exit-notify 1
tls-auth ta.key 0
auth SHA512

Ok, so on the server (asafoetida initially, later moved to tarragon), in /etc/openvpn/server, I have the following files:

The file ca-chain.cert.pem has in it both the certificate for the intermediate ca and the certificate for the root ca which signed the intermediate. Both of these are needed on the server side, for sending to the client. Later, when we build the client side, it really only needs the root ca cert.

Initially, I started the openvpn server manually, with:
cd /etc/openvpn/server
openvpn darrell.server.conf

But see below discussion of automatic startup.

Client Side

On the client side (on lemongrass), in /etc/openvpn/client, I have these files:

And on lemongrass also, I (initially) started the client manually with:
cd /etc/openvpn/client
openvpn darrell.vpn.conf

Here is the entire client config file /etc/openvpn/client/darrell.vpn.conf:
root@lemongrass:#cd /etc/openvpn/client
root@lemongrass:#cat darrell.vpn.conf
dev tun
proto udp
remote 1194
resolv-retry infinite
ping 15
ping-restart 0
reneg-sec 0
explicit-exit-notify 3
verb 3
cipher AES-256-CBC
auth SHA512
cert darrell_client.cert.pem
key client.key
remote-cert-tls server


key-direction 1
# 2048 bit OpenVPN static key
—–BEGIN OpenVPN Static key V1—–

—–END OpenVPN Static key V1—–

Firewall adjustments

The server is an amazon ec2 instance, and all ec2 instances have a firewall, called a “security group” provided by amazon. I used the same “security group” rules at the amazon level for asafoetida as for tarragon (http+mail+ssh, etc), but with an additional rule to admit udp 1194.

On asafoetida proper, there was not an additional iptables firewall – it had a default Accept-Accept-Accept filter table and no nat table. I added a couple of forward rules to filter, to accept forwarding of tun+ to eth0 and eth0 to tun+. Also I added a nat table postrouting rule to masquerade all output through eth0 to the vpn address.

On tarragon, the default redhat firewall sends forward traffic through the input chain (?), and I only had to accept udp 1194, and accept tun+., and add the nat table postrouting, so in /etc/sysconfig/iptables I added:
-A RH-Firewall-1-INPUT -i tun+ -j ACCEPT
-A RH-Firewall-1-INPUT -i eth0 -m state --state NEW -p udp --dport 1194 -j ACCEPT

I had not previously had a nat table in the tarragon firewall, so I had to put one in:

The link gets made, and the routing table on both client and server seem to be adjusted properly, but traffic doesn’t flow out of the server. The problem turned out to be that I had failed to turn on ip forwarding on the box.
echo “1” > /proc/sys/net/ipv4/ip_forward

And to make that last, I put it in /etc/sysctl
net.sys.ipv4.ip_forward = 1

Starting Automatically

There are systemd instance scripts for starting a client (openvpn-client@) and for starting a server (openvpn-server@). So if you put a conf file in the /etc/openvpn/client directory named fred.conf, you can then do:
systemctl enable openvpn-client@fred

And similarly, for an openvpn server config named darrell.conf in /etc/openvpn/server:
systemctl enable openvpn-server@darrell

Revoking Certificates

This is just a parenthetical note, to record one additional bit of experience, to help me remember. I had to reissue the certs because I failed the first time to specify the “extensions” part. However, you cannot reissue a cert with the same common_name (i.e. you can’t generate a new cert using the same csr) until you remove the old cert, which is a better practice anyway.
openssl ca -config openssl.cnf -revoke certs/darrell_server.cert.pem

Then I reissued the 2 certs, with the names server2 and client2. But that isn’t relavent to these instructions, and so in the instructions above I continued to reference the certs as darrell_server.cert.pem and darrell_client.cert.pem, even though on the box the names really had the “2” appended.

Automatic update of Fedora, not!

I tried doing an update of fedora the other day, with the dnf system upgrade business, to upgrade in place. I should have known better. The failure is almost certainly related to the ongoing frustration of the graphics card.

One is offered the choice of two nice poisons. One may elect to use the open source nouveau driver, in which case the graphics driver will spontaneously crash about once a week forcing a reboot. Or, alternatively, one may choose instead to install the proprietary nvidia driver, in which case every few months one will get a new kernel that invalidates the driver, and the machine will suddenly not boot at all, requiring that you get in a fool around with grub and intitramfs until you can get it up enough to download and rebuild a new nvidia driver.

The much ballyhooed DKMS which is supposed to solve this problem has never worked for me – that I know of. I suppose there could have been times when it magically worked and I didn’t have a hang, and didn’t know that I had dodged a bullet. But for sure I know when I do get the hangs.

And the worst time for all this is upon attempting to do the Fedora update. I knew I should not try it. It has always failed. I have specifically bought a little two drawer 2.5″ removable so that I can install onto a new SSD, and then switch back when it doesn’t work. But you never know – one of these days Fedora’s upgrade in place may actually work. Ubuntu seems to have their working reasonably well. And if I never try it on a new release of Fedora, I won’t know it has been fixed, if they ever fix it.

It took two days to get oregano back. In the meantime I had the problem that, for historical reasons (and stupidity on my part) oregano plays an essential part in the internal network. When oregano is down, dhcp is down, dns forwarding is down, the main firewall is down, etc. This has to change. See the next post.

Netfilter table order

II have a terrible time keeping this straight. This  is a memory aid. I think it is correct, but it has been assembled from internet sources and not from reading the code, so this is not authoritative.

  • Incoming packets:
      • raw table [PREROUTING]
      • contrack
      • mangle table [PREROUTING]
      • nat table [PREROUTING] (dnat)
      • route decision (either for us, or another host)
    • IF Destined for this host:
        • mangle table [INPUT]
        • filter table {INPUT]
        • security table [INPUT]
        • nat table [INPUT (rarely used)] (snat)
    • IF Destined for another host:
        • mangle table [FORWARD]
        • filter table [FORWARD]
        • security table [FORWARD]
      • (-> do output postrouting)
  • Packets generated on local host:
      • route decision (select interface, etc)
      • raw table [OUTPUT]
      • contrack
      • mangle table [OUTPUT (rarely used)]
      • nat table [OUTPUT (rarely used)] (dnat)
      • route decision
      • filter table [OUTPUT]
      • security table [OUTPUT}
    • (-> do output postrouting)
  • Output PostRouting
      • mangle table [POSTROUTING]
      • nat table [POSTROUTING] (snat/masquerade)

contrack means connection tracking,  unless the (always preceeding) raw table says not to track.

In nat table, PREROUTING chain can only use -i and do dnat; POSTROUTING chain can only use -o and do snat. What you can do with the nat INPUT and OUTPUT chains I don’t understand well, but I think INPUT can do snat and OUTPUT can do dnat.

Difference of snat vs masquerade on nat POSTROUTING: 1) for snat you set the ip address, for masquerade you give an interface, and it uses the address from that interface. Also Masquerade assumes the connection will change ip address when it goes down and comes back up, so throws away all connections (contrack) information.


Certificates Redux

An earlier post talked about switching my server tarragon (where this blog sits) to a wildcard certificate from letsencrypt. There were two reasons why I was using a wildcard certificate. One had to do with test versions of websites that run on this server, and the need that some of those sites have for wildcards, of the form:,, etc. The other reason was that I have a lot of hosts (oregano, cinnamon, paprika, lemongrass) in addition to tarragon that “need” to have a certificate, for https, for imap, and for smtp, and when I was having to pay for them, it was cheaper to get one wildcard for Continue reading Certificates Redux

Dynamic DNS

I have used dynamic dns for around 20 years, I think. But I have always used which these days seems to want to call themselves And some years back they were bought by Oracle, the kiss of death, and now they are impossible to deal with, arrogant, unsupportive, insular – all the things I expect of Oracle.

And why have I kept using them? Because that is what the routers supported. Dyndns was  there first, and the ubiquitous linksys and netgear routers usually have a feature to do automatic updates for dynamic dns, but (often) the router will only update dyndns: nobody else. And I’ve got routers installed in various people houses that are doing this.

But I recently added a new house that I support, and that person has a proprietary and ponderous comcast router, which will barely do anything useful, and has no facility to update dynamic dns.

Continue reading Dynamic DNS

Apache Quiet Failure on Certificate/Key mismatch

In an earlier post I related how I had moved to letsencrypt for tarragon. In the process of doing some cleanup of the /etc/letsencypt directory, and my repository, I managed to stupidly get one wrong private key file into the batch of all the https vhosts, such that the http config file for specified an SSLCertificateFile and SSLCertificateKeyFile which did not match.

It took me hours and hours to figure this out, because Apache simply fails to start and gives no indication whatever what has pissed him off. I wasn’t too stupid to figure out that I had been messing with the certs yesterday, and the problem might lie there somehow. But I have about 15 vhosts, so it was tedious. In the end I resorted to strace, and saw the problem.


LetsEncrypt Wildcard Certificates

Comodo is after me to renew, offering a free year. The last time I attempted to install a wildcard certificate from Lets Encrypt, shortly after they introduced the feature, I wasn’t able to figure it out. Now, 9 months later, there is a lot more information about how to do it. Before spending the money for a commercial cert, I thought I would give it a try.

I used the following on tarragon:

certbot certonly \
--server \
-d -d * \
--preferred-challenges dns

It is important that the server url by v02, because v01 servers can’t issue wildcard certs. I had to put TXT records in the DNS for them to verify, and they created the cert into the /etc/letsencrypt/live directory where all the others are.

This was trivially easy. Goodbye Comodo.

GDM fails to start

Experimenting with VMs on Ubuntu, I had the system running on a relatively small root disk. Was troubleshooting problems in the disk array, doing a lot of booting, and crashing, when suddenly the boot wouldn’t finish. Fortunately the boot wasn’t quiet and I could watch as it tried 20-30 times to start the gdm service.

I change the default to and got it up, and what do you know, the root filesystem is at 100%. Cleaned up some logs and crash logs and it came right up.

Interesting that it doesn’t have a way to at least alert you that is what its problem is.

Laptop backup and Lid events

I have some scripts which try to do backups on the laptop. It is a little more involved, since the laptop lid is closed most of the time, including at night. So the strategy is to have a cron job start periodically, determine if a backup had been done today and if not, attempt to do one.

Since the laptop could be in various locations, it tries to determine where it is – i.e. if it is in a known location (my house or my sister’s house, or one of my DC friends houses). If so, it can back up accordingly. This is done using SSIDs, which is adequate if not elegant.

One issue I’ve had is if the attempted backup starts right after the lid is opened, the wireless may not be ready yet, and so the backup may fail on account of no network. So I set out to try to figure out how to detect when the network is available, so the script can obtain that information. Continue reading Laptop backup and Lid events

Clamav and Amavisd

I find the whole clamav subsystem to be fragile. I think this is because it is written as a tool which stands on its own, but I’m only using it as a subsystem hung onto the side of amavisd. So there is some hand-waving and jiggery pokery with the sockets and the permissions to enable the two to communicate, which has to be done manually, and is not properly a part of either subsystem.

I have another article on setting up this subsystem here, which records some of the stuff being done. I think basically, amavisd has to know where the shared socket is, in order to send messages to clamav to check, and they have to agree on the ownership and permissions of the socket and its directory.

Once in a while that stuff gets crosswise, and since I only vaguely understood what was going on, and only did the hand-waving by rote, I got annoyed with it. I’ve grown used to being able to just have things slot in and work, without my having to actually dig in and understand them. The nerve of these people, to expect me to know what is going on in order to make it work! Irony intended. Continue reading Clamav and Amavisd