Deploy an OpenVPN Instance to AWS with CloudFormation
Tags: aws, openvpn, showdev, tutorial
Published: 2019-11-09 22:56:41

Want to automatically deploy an OpenVPN EC2 instance, running on Amazon Linux 2, to AWS? One that auto-produces an OVPN configuration file that is compatible with OpenVPN Connect, on both Android and iOS? This article is a breakdown of the following CloudFormation template repo:

- [ScriptAutomate/openvpn-cfn: Deploy OpenVPN to AWS via CloudFormation and Amazon Linux 2](https://github.com/ScriptAutomate/openvpn-cfn)

Feel free to fork it, star it, test it, and comment on it! Want an in-depth look at how it works? This post helps breakdown how I patched up a non-functional CloudFormation template.

> ***NOTE:*** _This worked much more like a learning tool, and I'd rather recommend something like [AlgoVPN](https://github.com/trailofbits/algo) when it comes to deploying and managing your own VPN in AWS._

## Table of Contents

- [TL;DR](#TL;DR)
  - [Quick-Start](#Quick-Start)
- [Problems with The Original](#Problems-with-The-Original)
  - [Why Upgrade to Amazon Linux 2?](#Amazon-Linux-2-vs.-Amazon-Linux:-Why-Upgrade?)
  - [Let's Not Hard-Code AMI IDs for Base, Default AMIs](#Let's-Not-Hard-Code-AMI-IDs-for-Base,-Default-AMIs)
  - [Updating OpenVPN for Amazon Linux 2](#Updating-OpenVPN-for-Amazon-Linux-2)
    - [Extra Packages for Enterprise Linux (EPEL), OpenVPN, and Easy-RSA](#Extra-Packages-for-Enterprise-Linux-(EPEL),-OpenVPN,-and-Easy-RSA)
    - [Easy-RSA Changes](#Easy-RSA-Changes)
  - [One Last Fix for Mobile Clients](#One-Last-Fix-for-Mobile-Clients)
- [The Future](#The-Future)

## TL;DR

I gave an old template a major facelift, bringing about a new version of a template blogged about by [Linux Academy](https://linuxacademy.com/). If you want more information about Linux Academy, and how I found their original CFN template, scroll to the very bottom of the post. Otherwise, here is a mini quick-start from the bash shell.

### Quick-Start

Want to give this a test run? This is about what the repo README describes. It would be best to refer to the GitHub repository in the future, as it will be most up-to-date.

```bash
# First step is to download:
# git clone git@github.com:ScriptAutomate/openvpn-cfn.git
# cd openvpn-cfn

SSHKEYPAIR='labkeypair'
AWS_PROFILE='labcreds'

# Create SSH key pair, and save contents of SSH key
aws ec2 create-key-pair \
  --key-name $SSHKEYPAIR \
  --profile $AWS_PROFILE \
  --query 'KeyMaterial' \
  --output text > $SSHKEYPAIR.pem
chmod 400 $SSHKEYPAIR.pem # Prereq .pem file permissions required by ssh

# Deploy the CFN template
aws cloudformation deploy \
  --template-file cfn-openvpn.yaml \
  --stack-name openvpn-personal \
  --profile $AWS_PROFILE \
  --parameter-overrides SSHKeyName=$SSHKEYPAIR \
  --capabilities CAPABILITY_IAM

# If wanting to SSH into the OpenVPN instance
OPENVPNIP=`aws cloudformation describe-stacks \
  --stack-name openvpn-personal \
  --profile $AWS_PROFILE \
  --output text \
    | grep OpenVPNEIP \
    | sed s/^.*EIP// \
    | tr -d '[:blank:]'`
ssh -i $SSHKEYPAIR.pem ec2-user@$OPENVPNIP
```

- The `VPNClientsS3Bucket` created by the stack will have a client OVPN file as `client/openvpn_clientuser.ovpn`

That's it. Want to deep dive into more details? Keep on reading.

## Problems with The Original Template

First off, the original CloudFormation template from the Linux Academy blog _no longer works._ I wanted to get this up and working so that I could have a personal OpenVPN server that can be connected to by laptop or phone. Want to use that coffee shop wifi? This makes it happen.

The YAML produced by Linux Academy seemed like a good starting point.

### Amazon Linux 2 vs. Amazon Linux: Why Upgrade?

Why would one want to migrate to Amazon Linux 2, from Amazon Linux? Here is a breakdown quoted straight from the [Amazon Linux 2 FAQs](https://aws.amazon.com/amazon-linux-2/faqs/):

> 1. Amazon Linux 2 offers long-term support until June 30, 2023.
> 2. Amazon Linux 2 is available as virtual machine images for on-premises development and testing.
> 3. Amazon Linux 2 provides the **systemd service and systems manager as opposed to System V init system in Amazon Linux AMI.**
> 4. Amazon Linux 2 comes with an updated Linux kernel, C library, compiler, and tools.
> 5. Amazon Linux 2 provides the ability to install additional software packages through the extras mechanism.

The **bolded** section is what makes life a bit difficult, at a configuration level, because migrating System V init based service configurations over to Systemd can be painful. I won't go into the details as to why, because entire Linux communities have gone to war on this to the point that Linux distributions have forked off. Amazon Linux, along with most other major Linux distributions, have migrated over to Systemd.

OpenVPN tutorials, scripts, and configurations across the internet almost exclusively target non-systemd setups.

To get an EC2 instance of Amazon Linux 2, we need to target the right AMIs. What's the best path? I know what path I _didn't_ want to do.

### Let's Not Hard-Code AMI IDs for Base, Default AMIs

I have always been _annoyed_ with hard-coding AMI IDs.

A problem with past CFN templates, and still with many present ones, was needing to include hard-coded values for AMI IDs. This wasn't just the case for custom AMIs built internally, but also with Amazon-provided base AMIs.

Amazon knew this was a pain, and they released a solution in 2018 by allowing customers to easily [query for the latest Amazon Linux AMI IDs using AWS Systems Manager Parameter Store.](https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/)

> ***NOTE:*** _Hardcoding AMIs can have the added benefit in having logged what AMI(s) successfully worked with a CloudFormation template. It may also be a requirement in organizations to use a custom AMI that has base configurations and org-specific monitoring software installed. In the case of my approach, in using the parameter store, I just wanted to share something that was both easy to share and know would be using the latest AMI when playing with in a sandbox._

#### Setup Latest Amazon Linux 2 AMI IDs as Default in CFN

##### JSON
```json
"LatestAmiId": {
  "Type": "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>",
  "Default": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
}
```

##### YAML
```yaml
Parameters:
  # Parameter store makes finding latest AMI image id easy
  # Is region locked, by default. Wherever CFN is spun up,
  #  resulting image id is specific to source region!
  # Source: https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
```

No more region-specific mappings from 2017 that are woefully out-of-date! Any `RegionMap` references in the template now just need to be replaced by `!Ref LatestAmiId` and we're good. The following section is completely removed:

```yaml
Mappings:
  RegionMap:
    us-east-1:
      "AMAZONLINUXAMI" : "ami-8c1be5f6" # Amazon Linux AMI 2017.09
    us-east-2:
      "AMAZONLINUXAMI" : "ami-c5062ba0" # Amazon Linux AMI 2017.09
    us-west-1:
      "AMAZONLINUXAMI" : "ami-02eada62" # Amazon Linux AMI 2017.09
    us-west-2:
      "AMAZONLINUXAMI" : "ami-e689729e" # Amazon Linux AMI 2017.09
    ca-central-1:
      "AMAZONLINUXAMI" : "ami-fd55ec99" # Amazon Linux AMI 2017.09
    ...
    ...
    ...
```

Now it is future-proofed for brand-new deployments. I even included comments about what's going on, which is an extra bonus of YAML vs. JSON.

### Updating OpenVPN for Amazon Linux 2

We need to update the most painful part: the `UserData` and `AWS::CloudFormation::Init` sections.

This resulted in a ton of irritation of the kind that emphasizes the value in using a configuration management toolset like [Ansible](https://www.ansible.com/), [Chef](https://www.chef.io/), [SaltStack](https://www.saltstack.com/), or [Puppet](https://puppet.com/).

When you rely merely on `UserData` and `cfn-init` execution of shell commands, the portability of the configuration can have room for improvement. _Yes_, the code now moves along with the CFN template so it is portable in that sense. Though, it is permanently coupled to the template and has to be picked apart to properly decouple for separate versioning, updating, etc.

Of course, there is a cost (of both _time_ and _complexity_) when going with a configuration management tool solution so I won't be extending this template to use one. Maybe in the future?

#### Extra Packages for Enterprise Linux (EPEL), OpenVPN, and Easy-RSA

OpenVPN isn't available in the default `yum` repositories. This is also an issue with `easy-rsa`, another requirement for setup.

##### Before

```yaml
# Install the latest version of openvpn via the yum package manager
# Install easy-rsa via the EPEL repo
# Make a copy of the installed files to /opt/easy-rsa as our working directory
install_software:
  packages:
    yum:
      openvpn: []
  commands:
    01_install_software_install_easyrsa:
      command: "yum install easy-rsa -y --enablerepo=epel"
    02_install_software_copy_easyrsa:
      command: "cp -R /usr/share/easy-rsa/2.0 /opt/easy-rsa"
```

This doesn't work because, again, `openvpn` isn't in the default repositories. Not only that, but the `--enablerepo=epel` will not work because EPEL isn't installed on Amazon Linux 2 as a repository to enable. This means the `easy-rsa` install will fail, too.

How do we resolve these? We need to [enable access to that Extra Packages for Enterprise Linux (EPEL) repo.](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-enable-epel/)

##### After

###### New OpenVPNVersion Parameter
```yaml
Parameters:
  OpenVPNVersion:
    Type: "String"
    Default: "2.4.7"
```

###### Updated EC2OpenVPNInstance UserData

```yaml
# Install and auto-enable EPEL repository
# Update existing packages
# Run CFN configSets
# Source: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-enable-epel/
UserData:
    "Fn::Base64":
      !Sub |
        #!/bin/bash
        yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
        yum update -y
        /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource EC2OpenVPNInstance --configsets myCfnConfigSet --region ${AWS::Region}
        /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource EC2OpenVPNInstance --region ${AWS::Region}
```

###### Updated install_software `AWS::CloudFormation::Init` configSet

```yaml
install_software:
  packages:
    yum:
      openvpn: [ !Ref OpenVPNVersion ]
      easy-rsa: []
      yum-plugin-versionlock: []
  commands:
    01_install_software_copy_easyrsa:
      command: "cp -RL /usr/share/easy-rsa/3 /opt/easy-rsa"
    # Prevent future yum updates from updating openvpn
    02_lock_openvpn_version:
      command: "yum versionlock openvpn"
```

We added a little extra by also installing `yum-plugin-versionlock`, so that we can control what version of OpenVPN is installed. In this case, we added an additional parameter to the template called `OpenVPNVersion`.

#### Easy-RSA Changes

There is another change here: `easy-rsa` had a _major version_ update when it entered into v3.x territory from v2.x. All the helper scripts have drastically changed, from names to arguments.

This meant a heavy facelift for many of the configs in `myCfnConfigSet`:

```yaml
Metadata:
  AWS::CloudFormation::Init:
    # Our cfn-init config set rules, divided into logical sections to make reading it easier, hopefully :)
    configSets:
      myCfnConfigSet:
        - "configure_cfn"
        - "install_software"
        - "generate_easyrsa_vars"
        - "generate_secrets"
        - "generate_client"
        - "configure_server"
        - "upload_files"
```

You can see the differences by comparing the original template and the updated template on GitHub:

- [Old Template](https://gist.github.com/zugdud/b39eea02faa6926305f57fbde8d31b68)
- [New Template](https://github.com/ScriptAutomate/openvpn-cfn/blob/master/cfn-openvpn.yaml)

The commands all had to be updated for new file paths associated with `easy-rsa`, along with a huge difference in the commands used to produce the same result of v2.x commands.

If you've been wondering how to navigate an unattended/silent configuration of `openvpn` with `easy-rsa`, the commands featured in the template may be a great place to start, with the commands run as follows:

```bash
cd /opt/easy-rsa
/opt/easy-rsa/easyrsa init-pki
/opt/easy-rsa/easyrsa --batch build-ca nopass
/opt/easy-rsa/easyrsa gen-dh
/opt/easy-rsa/easyrsa build-server-full server nopass
/opt/easy-rsa/easyrsa gen-crl
openvpn --genkey --secret pki/statictlssecret.key
```

#### One Last OpenVPN Fix for Mobile Clients

Though I was able to connect to the OpenVPN server from a desktop client without any issues, I ran into this error on OpenVPN Connect (Android):

```
There was an error attempting to connect to the selected server.
Error message: mbed TLS: SSL read error: X509 - Certificate verification failed, e.g. CRL, CA or signature check failed.
```

The [OpenVPN Connect Android FAQ](https://openvpn.net/vpn-server-resources/faq-regarding-openvpn-connect-android/) had a response for this, which was of zero help:

> This is an error that tells you that the certificate could not be verified properly. This can occur for example if you are using an MD5 signed certificate. With such a type of certificate, the security level is so low, that the authenticity of the certificate simply cannot by any reasonable means be assured. In other words, it could very well be a fake certificate. The solution is to use a certificate not signed with MD5, but with SHA256 or better. You can find more information in the [MD5 signature algorithm support](https://openvpn.net/vpn-server-resources/faq-regarding-openvpn-connect-android/#MD5_signature_algorithm_support) section.

I had verified that my algo was of a higher level:

```bash
sudo openssl x509 -in /opt/easy-rsa/pki/ca.crt -noout -text | grep "Signature Algorithm"

# Output:
# Signature Algorithm: ecdsa-with-SHA256
```

*NOTE: I also went with the `ec` algorithm to see if that was the issue, due to the default being `rsa`. This is of reading through where someone else had struggled with Android and iOS connects. Supposedly that helped solve their issue, but that made zero difference for me. I have since found out this wasn't the issue, and `rsa` should work fine. I have added a parameter to the template for either algorithm to be used.*

I was banging my head against the wall until I came across someone on a forum with a similar problem from a year ago: [OVPN Profile Works on Windows but not on Android](https://forums.openvpn.net/viewtopic.php?f=33&t=26951)

In it, a person says `ns-cert-type` is getting **DEPRECATED** as a config setting! It turns out `ns-cert-type server` existed in my client OVPN config and was breaking when attempting to connect from Android, but my other non-mobile clients didn't seem to care (this will change in the future when OpenVPN releases hit v2.5.x). I changed the following in the `server.conf`:

```
ns-cert-type server
```

To:

```
remote-cert-tls server
```

And now it works without fail from the Android client!

![OpenVPN App Screenshot](https://www.icanteven.io/img/2019/11/openvpn-android-connected.png)
![OpenVPN App Screenshot Active](https://www.icanteven.io/img/2019/11/openvpn-android-connect.png)

### Replacing the Node.js 6.10 CustomResource

By default, CloudFormation stacks cannot delete an S3 bucket they created if there is any content within them.

The original template used a Node.js `CustomResource` in CloudFormation to help with the cleanup during a teardown. But it used the `nodejs6.10`, which is long gone in the AWS Lambda runtime world, [and `nodejs8.10` is approaching EOL](https://dev.to/scriptautomate/aws-lambda-node-js-v8-10-runtime-approaching-eol-end-of-life-gll).

I'm more experienced in Python, and I had seen a CFN template on GitHub that used the `python3.7` runtime to achieve the same goal of deleting an S3 bucket even if it isn't empty.

## The Future

What other problems still exist here? I'd like to approach all of the following:

- Certificate Management: Certificates are only found locally on the EC2 instance and with a single client OVPN dumped to an S3 bucket. This means I can't simply destroy/redeploy without starting over with certificates since the client certs wouldn't work with the a new root CA setup
  - What about issuing new certificates? Can we create a simplified way of doing this?
  - What about revoking certificates?
- Automated Patching: Update by way of replacing the instance with one running on the latest AMI. This should be achieved with an AWS Lambda monitoring for newer AMIs. Though, Amazon Linux AMIs don't often see updates so `yum-cron` should be setup until certificate management can be remedied
- The existing `AWS::IAM::Policy` permissions could be more fine-grained

I think it would be interesting to approach the certificate management and updating problems with AWS Lambda and AWS Secrets Manager. If on first deployment there aren't particular certificates stored in the secrets manager, it could create them. If they already exist, it could pull the existing certificates down instead of generating new secrets from scratch. This would mean a complete ability to redeploy the instance as part of an updating process.

I may use this project as a way of learning the AWS CDK by migrating it over in the future.

---

_[Linux Academy](https://linuxacademy.com/) is an excellent website for learning about Linux, containers, different cloud providers, configuration management tools, DevOps methodologies, certification prep, and more._

![LinuxAcademy Logo](https://thepracticaldev.s3.amazonaws.com/i/a8a186uulx5eycbdmobj.png)

***I am not associated with Linux Academy,*** _but my CloudFormation template is a modified version of one developed in the following blog posts by Linux Academy:_

- [How to Roll Your Own VPN with AWS CloudFormation - Part One](https://linuxacademy.com/blog/tutorials/roll-vpn-aws-cloudformation-part-one/)
- [How to Roll Your Own VPN with AWS CloudFormation - Part Two](https://linuxacademy.com/blog/tutorials/roll-vpn-aws-cloudformation-part-two/)
- [How to Roll Your Own VPN with AWS CloudFormation - Part Three](https://linuxacademy.com/blog/tutorials/how-to-roll-your-own-vpn-with-aws-cloudformation-part-three/)
- Original, Non-Functional GitHub Gist: [OpenVPNWalkthrough-Part3.yaml](https://gist.github.com/zugdud/b39eea02faa6926305f57fbde8d31b68)

---

_Originally published at [https://icanteven.io](https://www.icanteven.io/blog/aws-openvpn-cfn/) on November 8th, 2019_