HTTPS for your local endpoints for less than $1 a month
Tags: haproxy, letsencrypt, ec2, ssl
Published: 2020-02-04 00:13:13

*Originally published at https://pcarion.com/article/ssh-tunnel*

There are quite a few instances when you want a public URL to hit your development machine:

- you want to expose a webserver running on your local machine to the internet so that a colleague or a customer may have a look at it.
- you are using a service, like [twillio](https://www.twilio.com/), which allows you to setup webhooks URL: the service will call those URLs to notify you when *something* happens.
- you want to test an OAuth integration, with Facebook or Twitter, and you want to provide public https callback URLs.

An alternative option is to install your code on a public server and have those requests hit this server. During development, it is much more convenient to have those requests hit your local machine so that you can debug and see the logs in real time.

You can also use a service like [ngrok](https://ngrok.com/): this is a tool, developed by [Alan Shreve](https://twitter.com/inconshreveable), very easy to use and [perfect for webhooks](https://www.twilio.com/blog/2015/09/6-awesome-reasons-to-use-ngrok-when-testing-webhooks.html).

We'll describe here how you can setup your own solution on ec2.

Such a home made solution has several advantages:

- you can have persistent URLs - ngrok gives a different domain each time you use it unless you become a paying customer.
- you can automate the process (see the end of this post) meaning that you can quickly tear down your EC2 instance and restart it quickly when you need the URLs again.

At `$0.0069` per hour for a nano instance, the service will cost you $1 per month if you use it for about 144 hours. This is more than enough for a "night and weekend" project.

## Description of the solution

 The solution looks like this:

![reverse proxy setup](https://pcarion.com/static/general-a44f005457e576124d5987409e4a80e9.png)

A machine (C) on the public internet will be able to access multiple URLs, on the same public URL, to access one or more services on your local machine (A).

For instance:

- the URL `https://api.mydomain.com` will access `http://localhost:3000` on your dev machine
- the URL `https://api.mydomain.com` will access `http://localhost:3000` on your dev machine

Warning: a custom made solution is definitely more complex than using `ngrok`, so you should think twice before taking that route. You need to be comfortable with bash scripting and DNS setup to implement that solution.

This solution requires:

- a domain name: this domain, and subdomains, will be used to configure public URL(s) to access your local machine
- a public DNS: as you want to have public URL to access your server, a DNS is required.
- a machine on the public internet: this machine will act as a bridge between the public internet and your local machine. You can use Digital ocean or AWS EC2, with full root access
- a *sshd* daemon running on that server: SSH is the swiss army knife for that kind of setup, and you need to be able to fully configure the SSH server, especially to setup a reverse tunnel
- a SSL certificate. To protect your connection, you need to setup SSL certificates so that the public URL can be available only through https. We will use [Let's encrypt](https://letsencrypt.org/) for that.

The Gateway server (B) is the machine on the public internet and you need to configure a set of services on that server:

- configure the SSD service to allow TCP port forwarding
- install a reverse proxy - `HAProxy` - to forward different subdomain to different ssh tunnels
- configure let's encrypt to allow SSL traffic over https

In the rest of this document, we will use EC2 to install this gateway server.

## EC2 instance configuration

The setup is not intended for a production service and the traffic on that machine should be very low. For that reason, you can use a very small instance to run that server : I am using a `t2.nano` instance with an ubuntu OS.

![ubuntu instance](https://pcarion.com/static/Untitled-43234f546cc3fbc0bb6ab085f7c12de4.png)

### security group

The only setup to pay attention to is the network/security group definition.

By default, the setup would allow only the port 22 for your ssh access:

![aws default security group](https://pcarion.com/static/Untitled-43234f546cc3fbc0bb6ab085f7c12de4.png)

You need more inbound ports for your server:

- `22` for your SSH access
- `80` for your incoming http access
- `443` for your incoming https access
- different ssh tunnels ports, like `8080`, `8085`, `8090`, etc... The number of ports depend on the number of services you want to expose through the ssh tunnel

You can create a security group with those inbound ports:

![aws security group](https://pcarion.com/static/Untitled-466fa35b8fc40dc33b2defc8e999a832.png)

### private key file

To use the SSH connection, you need to create a `key pair` . Download the associated `.pem` file, copy it in a safe place and do a `chmod 400` on that file to avoid the error:

    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    Permissions 0644 for 'ssh_tunnel.pem' are too open.
    It is required that your private key files are NOT accessible by others.
    This private key will be ignored.
    Load key "ssh_tunnel.pem": bad permissions

## server configuration

At the end of this document, we will show how to automate entirely the setup process. 

The following description is useful if you want to have a better understanding of what the automated script will be doing.

Using your pem file, you can `ssh` to the server and proceed with the configuration.

First, make sure that all the packages are up to date

    sudo apt-get update

## **setup the sshd server for the reverse ssh tunnel**

You need to make sure that the sshd server is running and has the proper configuration.

This command will check the sshd configuration:

    $ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
    gatewayports no
    allowtcpforwarding yes

For the ssh tunnels to work, you need both parameters to be set to `yes`.

Open the file `/etc/ssh/sshd_config` and add or set those 2 lines:

    AllowTcpForwarding yes
    GatewayPorts yes

Then, you need to restart the ssh daemon for those parameters to be taken into account:

    service sshd restart

And then you can check that all is in order:

    $ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
    gatewayports yes
    allowtcpforwarding yes

## T**est of the reverse ssh tunnel setup**

Before moving forward with the server configuration, you can already check that the ssh setup is working properly.

In order to test that a local service can be reached from the public internet, you need to start some sort of local server on your development machine.

A simple HTTP server will do the trick: from a directory containing no sensitive data, you can start a simple python server:

    python -m SimpleHTTPServer 8000

You can check from your local browser from `http:127.0.0.1:8000` that the server is working, and ... not exposing sensitive data.

Now is the time to invoke the SSH port forwarding voodoo incantation, from your local machine:

    ssh -i "tunnel.pem" ubuntu@ec2-a-b-c-d.us-west-1.compute.amazonaws.com -N -R 8080:localhost:8000

This SSH command is much simpler than it looks like:

- `ubuntu@ec2-a-b-c-d.us-west-1.compute.amazonaws.com` this is the public DNS Address of your server and you can find the value in your aws/ec2 console
- `-N`: by default, ssh will create a shell on the remote machine. We don't need that here
- `-R`: with this option you are asking ssh to answer on the remote side (your gateway)
- `8080:localhost:8000`: any connection on port `8080` on the gateway will be tunneled to the the port `8000` on your local machine (where the webserver we started previously is listening on).

If all work as expected, you can open your browser at: `http://ec2-a-b-c-d.us-west-1.compute.amazonaws.com:8080`

and you should see the file served by your local server!

Your reverse ssh tunnel is working.

More information about reverse ssh tunneling can be found [here](https://unix.stackexchange.com/questions/46235/how-does-reverse-ssh-tunneling-work).

## DNS setup

The next step is that you want a *nicer* URL to access your service right?

You need to configure the DNS for your domain *yourdomain.com* and create a `A record` for *api.yourdomain.com* , with a TTL of 600, with a value of `a.b.c.d` which is the IP address of your gateway.

I may take time for the DNS configuration to propagate, but once it is done, you can then access your local web server through the URL: `http://api.yourdomain.com:8080`.

Better, but can still be improved: you may want to setup multiple subdomains which would allow you to host multiple local services, or have multiple machines using this tunnel (each using a specific subdomain).

To do that, you need a reverse proxy on your gateway.

## HAProxy setup

Let's install HAProxy:

    $ install haproxy
    $ sudo apt-get install -y haproxy
    $ haproxy -v
    HA-Proxy version 1.8.8-1ubuntu0.9 2019/12/02
    Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

You then need to configure its main configuration file: `/etc/haproxy/haproxy.cfg`

The setup is pretty basic: base on the domain being accessed (like `api.yourdomain.com`), you serve data from a local server (`127.0.0.1:8080`), which, through the ssh reverse tunnel will connect back to your local machine

Example of subdomain setting in `haproxy.cfg`

    frontend account
    	bind *:80
    	mode http
    	acl host_api hdr(host) -i api.yourdomain.com
    	use_backend account if host_api
    
    backend api
    	mode http
    	server node1 127.0.0.1:8080

With that configuration, you can then verify that the URL [`http://api.yourdomain.com`](http://api.yourdomain.com) serves also your local HTTP server data.

We won't go any further yet as the configuration of HAProxy is very dependent on the next step.

# SSL - letsencrypt

Nowadays, https is almost always required to access a server.

You can easily get a free SSL certificate using [Let's encrypt](https://letsencrypt.org/).

The installation of the required tool is easy:

    sudo add-apt-repository -y ppa:certbot/certbot
    sudo apt-get update
    sudo apt-get install -y certbot

There is a rate limiting with let's encrypt services so... you are limited in the number of trial and errors to configure your certificates.

There are 2 conditions to ensure before starting the let's encrypt setup:

- you need to make sure that the DNS setup is done and propagated for `http://yourdomain.com` domain name - the let's encrypt server relies on that to ensure that you are the rightful owner of the domain
- you must stop `haproxy` or any service using the port `80` as this port will be used by let's encrypt, using their own `cerbot` server to retrieve the certificate

Once those conditions are met, you can start the certificate retrieval process:

    sudo certbot certonly --standalone -d yourdomain.com -d api.yourdomain.com -d www.yourdomain.com --non-interactive --agree-tos --email you@email.com

You need to change the command line above with:

- the list of subdomains you want to have a certificate for in the `-d` arguments
- your email address as the `-email` argument

If all goes well, you should see this kind of output:

    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator standalone, Installer None
    Obtaining a new certificate
    Performing the following challenges:
    http-01 challenge for yourdomain.com
    http-01 challenge for api.yourdomain.com
    http-01 challenge for www.yourdomain.com
    Waiting for verification...
    Cleaning up challenges
    
    IMPORTANT NOTES:
    - Congratulations! Your certificate and chain have been saved at:
       /etc/letsencrypt/live/yourdomain.com/fullchain.pem
       Your key file has been saved at:
       /etc/letsencrypt/live/yourdomain.com/privkey.pem
       Your cert will expire on 2020-04-25. To obtain a new or tweaked
       version of this certificate in the future, simply run certbot
       again. To non-interactively renew *all* of your certificates, run
       "certbot renew"
     - If you like Certbot, please consider supporting our work by:
    
       Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
       Donating to EFF:                    https://eff.org/donate-le

## import certificates to HAProxy

The certificates generated above will be used by HAProxy and they need some massaging to be usable there:

    sudo mkdir /etc/haproxy/certs
    DOMAIN='yourdomain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'

The last step is to configure HAProxy, with all your domain and the https setup.

We will cover that in the next section

## Automation

As you can see, there are a lot of sets involved to set up a server but there is a way to automate the entire process.

### ec2 template

You can speed up the creation of the ec2 instance by using a template.

You can create a template from the instance you just created and make sure that:

- you are using the right security group with all the inbound ports you want to use
- you use the key-pair previously created

With that template, the creation of the instance becomes very easy with basically one click.

### automation steps

- create an instance using the ec2 instance template
- once the instance is booted, connect to it using your pem file to make sure your ssh setup is correct
- retrieve the IP address of this new instance and update your DNS setup. Confirm with a DNS lookup that the IP address has been updated
- you can confirm that the DNS is correct ifyou can connect to your instance with your domain name:

    `ssh -i "tunnel.pem" ubuntu@yourdomain.com`

- update the following script (see below) and run it with:

    `ssh -i "tunnel.pem" ubuntu@yourdomain.com 'bash -s' < setup_tunnel_host.sh`

### automation script - setup_tunnel_host.sh

Here is the script that you can run to automate all the steps describe above...

{% gist https://gist.github.com/pcarion/5b20af09c323a8214f6356d97d24d1ea %}

You need to set a couple of variables at the top of that script

- this script will setup a SSH tunnel for you domain: `BASE_DOMAIN_NAME`
- your email address for let's encrypt: `LETE_EMAIL`
- the script setup 2 subdomains with 2 ports (it's easy to update the script to use more ports/subdomains):
    - `SUBDOMAIN1`/ `REMOTE_REDIRECT_PORT1`
    - `SUBDOMAIN2`/ `REMOTE_REDIRECT_PORT2`

On your local machine, you can then initiates both tunnels with:

    ssh -i "tunnel.pem" [ubuntu@yourdomain.com](mailto:ubuntu@chirloute.com) -N -R 8080:localhost:8000 
    ssh -i "tunnel.pem" [ubuntu@yourdomain.com](mailto:ubuntu@chirloute.com) -N -R 8090:localhost:3000 

You must update the local port to match the server running on your local machine.

Enjoy!