Create a reverse SSH tunnel for remote access to a restricted Linux machine
Tags: ssh
Published: 2021-01-29 06:30:20

I did not find a clear modern description of this set-up so have written this post to help others.

Say you have a Linux machine to which you want to [ssh](https://www.openssh.com/) to but that machine is behind corporate or other firewalls etc which you have no control over, and thus you can not forward an external port for ssh etc. Let's call that the restricted machine and we assume you can make changes to it, e.g. locally at that machine. This post describes how you can set up a reverse ssh tunnel initiated and maintained from that restricted machine to a port on a Linux server elsewhere on the internet that you control. With the reverse tunnel in place, you can simply ssh to that port on your server to get a tunnelled ssh session back to the restricted machine through the restricted firewalls. Of course you should always check you have permission to do this with whoever controls the restricted network.

Note that [autossh](https://linux.die.net/man/1/autossh) was previously often used for this (and nearly all existing online tutorials about reverse ssh tunnels use [autossh](https://linux.die.net/man/1/autossh)) but [autossh](https://linux.die.net/man/1/autossh) is redundant nowadays since modern [openssh](https://www.openssh.com/) versions can monitor link health and exit if the link fails, and [systemd](https://systemd.io/) can then be used to restart the ssh tunnel. So [autossh](https://linux.die.net/man/1/autossh) is not required, we can implement an automatically maintained ssh tunnel using standard [ssh](https://www.openssh.com/) and [systemd](https://systemd.io/) alone.

In summary the steps are:

1. Set up the restricted machine.
2. Set up the server.
3. Final set-up and test.

### Set up the restricted machine

On the restricted machine, create an ssh key pair in `/etc/sshtunnel/` with an empty/no pass-phrase:

```
$ sudo mkdir -p /etc/sshtunnel
$ sudo ssh-keygen -qN "" -f /etc/sshtunnel/id_rsa
```

Create/paste the following `systemd` service file in `/etc/systemd/system/sshtunnel.service` and then edit it as needed for your specific server and ports:

```
[Unit]
Description=Service to maintain an ssh reverse tunnel
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
Type=simple
ExecStart=/usr/bin/ssh -qNn \
  -o ServerAliveInterval=30 \
  -o ServerAliveCountMax=3 \
  -o ExitOnForwardFailure=yes \
  -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -i /etc/sshtunnel/id_rsa \
  -R 9001:localhost:22 \
  sshtunnel@server.net -p 443
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target
```

You may likely want to change some of the unlabelled `ssh` parameters above so they are described below:

|Parameter|Description|
|----------|-----------|
|9001|Port on your server to which ssh clients will connect to.|
|22| Port you are running the ssh server on the restricted machine. Nearly always the default ssh port 22.|
|server.net | Internet hostname of your server. I actually use a dynamic DNS alias here (e.g. a hostname at [duckdns.org](http://www.duckdns.org/)) so I can easily re-select a different target server.|
|443        | Port on your server where the ssh server is running. Often the default 22 but 443 is recommended as outgoing connections are less likely to be blocked by the restricted firewalls. However, using 443 requires that your server does not have a https server running on that port. Of course, if you use 443 then you must change your sshd configuration to listen on that port.|

### Set up the server

On the server, create an account for the sshtunnel user but restrict it so that shell logins and remote commands are not allowed. The server will only allow the remote sshtunnel user to set up port forwarding:

```
$ sudo useradd -m -s /bin/true sshtunnel
$ sudo mkdir -p ~sshtunnel/.ssh
```

Copy the public ssh key you created above on the restricted machine (`/etc/sshtunnel/id_rsa.pub`) to the sshtunnel user's authorized keys file on the server (`~sshtunnel/.ssh/authorized_keys`).

Ensure the owner/permissions are strictly set as ssh requires them.
```
$ sudo chown -R sshtunnel:sshtunnel ~sshtunnel/.ssh
$ sudo chmod 700 ~sshtunnel/.ssh
$ sudo chmod 600 ~sshtunnel/.ssh/authorized_keys
```

### Final set-up and test

On the restricted machine, enable and start the service:

```
$ sudo systemctl enable --now sshtunnel
```

Check the status and logs with:

```
$ sudo systemctl status sshtunnel
$ sudo journalctl -u sshtunnel
```

Then on your server test the tunnel to the restricted machine with the following:

```
$ ssh -p 9001 localhost
```

To diagnose problems, check the `sshd` logs on both the server and the restricted machine:

```
$ sudo journalctl -u sshd
```

### External public access via server (optional)

For most users, the above configuration is completely sufficient. The tunnel is available by making an ssh connection to the port on the server localhost interface. Either you connect to this locally from within that server, or if you are connecting from another machine then you use the ssh [`ProxyJump`](https://man.openbsd.org/ssh_config#ProxyJump) configuration keyword to automatically redirect to the restricted machine via the server proxy, again via the localhost interface.

Some users may want to make the tunnel port also externally accessible on the server public network interfaces, e.g. for use by users who don't have ssh access to the server itself. Be aware you are opening a public port directly connecting to the restricted machine if you do this so this approach should be avoided, or at least disable password authentication and only allow [ssh key authentication](https://www.ssh.com/ssh/key/) on the restricted machine.

There are 2 extra steps to add this:

1. On the server, enable `GateWayPorts yes` in `/etc/ssh/sshd_config`. Then `sudo systemctl reload sshd` if you changed this.

2. On the restricted machine, prepend a colon (':') on the port number in the `/etc/systemd/system/sshtunnel.service` file, so the `ssh` option is `-R :9001:localhost:22` rather than `-R 9001:localhost:22`. Then do `sudo systemctl daemon-reload` and `sudo systemctl restart sshtunnel`.

### Removal

To remove all the changes described here do the following.

On the restricted machine:
```
sudo systemctl disable --now sshtunnel
sudo rm /etc/systemd/system/sshtunnel.service
sudo rm -rf /etc/sshtunnel
```

On the server:
```
$ sudo userdel -r sshtunnel
```

Disable `GateWayPorts yes` in `/etc/ssh/sshd_config` if you added it then `sudo systemctl reload sshd`.