Protecting ssh

I have a dozen or so boxes, mostly little raspberry pis, out in people’s houses which let me do backups for them, and attach to their networks. I’ve documented this before in “Gateway pi”, “Memory on the Gateway Pi”, and “Timemachine on Gateway pi” for example.

Connection between these boxes and my house is with SSH, and I use openssh certificates as described in “Using openssh certificates” and “Re-signing Openssh Certificates”. However, there has always been a little nagging problem, which is that these boxes must (re-)establish their connection to me automatically upon reboot, without user intervention. This means that the private keys that accompany the certificates cannot be encrypted, for that would require human intervention.

So there is a risk. Those raspberry pis have upon them a certificate and a private key which would enable access to boxes in my house. Not completely unrestricted access, and not root access, but nevertheless.

I experimented a little with using the ChrootDirectory directive along with the Match directive to confine these boxes, but the problem was they actually did need the ability to transfer a backup with rsync.

My friend Mr. G. suggested that I might be able to do what I need with a Docker container. Based on his suggestion I have a new scheme which solves the problem. I actually separate the problem into two parts, as the requirement is in two parts. One, on reboot all these boxes open a persistent ssh connection to me, with reverse tunnels, which I use to get to their house through their firewall. Second, each night when they do backups to me, they perform an rsync through a separate ssh connection.

For the first, the establishment of an autossh connection to oregano from each gateway box, the sshd daemon on oregano will now use the match directive for user gateway (which is the userid they all use to connect, and the only userid which will work with the certificate they have), to (a) confine the connection to a chroot jail, and (b) forbid the creation of a tty connection. When the remote gateway opens the connection to oregano, it does so without a terminal, it’s only purpose is to open the reverse tunnels, and it is confined to a directory with virtually nothing in it. This is good for my protection, but they cannot then use this connection for the second purpose, the rsync. Thus the second part.

For the second, each night when the gateway box wishes to transfer its backup file, it now establishes a connection to cinnamon, rather than oregano. It connects to cinnamon on a special port, and that port is mapped to a Docker container, which contains a mapped drive for the backups that is specific to this specific gateway box. This solves a second problem which I haven’t mentioned, but which nagged a bit, which was that previously the contents of one home’s backup files would have been visible (to a sufficiently clever attacker) from another home. Now the docker container is confined to the specific directory for the current home’s backups. Also, it should be mentioned that the certificate issued to user gateway is denied access to cinnamon’s main ssh port (DenyUsers gateway); user gateway can only connect to its own special docker port on cinnamon.

So, the ssh connection is confined to the docker container. It is true that an evildoer in possession of the certificate and private key can open this connection, and can get a terminal window. But now they find themselves in a docker container with no access to anything.

16 February, 2022

I’m in the midst of rebuilding Cinnamon, and I realize that this post wasn’t very helpful in its primary objective, which is to remind ME of what I have done. So I’m updating to record what I did.

I have very little docker experience, and had to learn a little. I cloned a file from https://github.com/linuxserver/docker-openssh-server, which I found based on a recommendation from my friend Mr. G to check out https://docs.docker.com/samples/running_ssh_service/. I cloned the file and figured out how to add two additional files, rsync and emacs to it.

5 May, 2022

I find that these containers are being attacked and their logs are filling up with the (fruitless) attacks, and I want to implement the brute force protection described in Detecting SSH Brute Force on those containers.

As a start, I realize that I hadn’t put any of the brute force protection stuff into the repo, so that is the first thing I did: add sshdrop and ipt_recents to common localbin. I also changed the sshdrop script to parameterize the iptables chain to which these rules are added.

On tarragon the rules are added to the INPUT chain, but on cinnamon, where the docker stuff lives, when docker is fired up it creates a bunch of its own chains, including one called DOCKER-USER, which is otherwise empty, and seems intended precisely for this kind of situation — where the system administrator wants to add rules for things destined for docker containers. So in the case of cinnamon, I want the sshdrop script to add the rules to DOCKER-USER. I first have to flush the chain, as the current default (and only) first rule is unconditional accept.

Initially set the constraints to 3 attempts in 20 seconds, and was surprised at how little blocking I got. I increased it to 3 attempts in 60 seconds. This may prove ineffective.