Deploying Concourse with concourse-up
Tags: devops, aws, productivity, testing
Published: 2020-01-02 00:24:45

There are a lot of Continuous integration tools out there, and they all have minor pros and cons but are generally the same. You push some code, that code triggers a build, and that build could trigger others. Sometimes these are called pipelines, but the only actual implementation of a pipeline build system I have seen is [Concourse](https://concourse-ci.org/). There will be future posts on what a pipeline is and why it's such a powerful concept not that's not this post.  

For this post, you should be up and running with Concourse in AWS with SSL and have your first pipeline in about 45 minutes, and most of that time will be waiting for AWS.

With this guide, you will
* Prepare AWS
* Provision valid SSL Certificates for free
* Deploy Concourse to AWS using [concourse-up](https://github.com/EngineerBetter/concourse-up)

You will need:
* Docker
* An internet connection
* An AWS Account with some credit
* A Domain with access to the DNS

## Prepare AWS
You don't want to give [concourse-up](https://github.com/EngineerBetter/concourse-up) too many permissions so let's create some restricted keys and use those for our deployment.

First login to AWS and head to the `IAM` section. 

If you're not familiar with AWS' `IAM` rules I recommend reading up on them. There are ton's of guides out there that can help.

I like to use Groups to manage Permissions so first create a group by going to the Group's section and then `Add New Group`. After naming your group, you will be prompted to attach security policies.

Add the following:

* AmazonRDSFullAccess
* AmazonEC2FullAccess
* IAMFullAccess
* AmazonS3FullAccess
* AmazonVPCFullAccess
* IAMUserSSHKeys
* AmazonRoute53FullAccess

You should see your new group with 0 users in the group list.

Now we need to add a user. In the `Users` section select `Add User`, name the user, and check the box `Programmatic access`. On the next section add the User to the group you created above. You don't need any tags so finish building the user.

This next screen is essential When you see Success click `Download CSV` and put it in a secure place.

Now you're ready to get your SSL Certs ready

## Provision valid SSL Certificates

Concourse uses SSL for all of it's service to service communication. You also want to make sure that any content you serve is also SSL so let's do that next.

Certbot has a ton of options and can be really easy depending on your registrar and where you manage your DNS. 
The most basic way is to use Certbot's manual image to generate your certificates. 

```bash
sudo docker run -it --rm --name certbot \
	-v "$HOME/letsencrypt:/etc/letsencrypt" \
	-v "$HOME/letsencrypt:/var/lib/letsencrypt" \
	certbot/certbot certonly \
	--manual
```

Follow the prompts. You will be required to verify ownership of your domain. If you specify an exact domain like ci.focusedlabs.io you will need to serve a key at a specific domain. It will look something like this.

```bash
Create a file containing just this data:

6z1rbMF-R6XdSo-s8HrTMKEeLGIvszkIA1IJPqzr9iU.FUHXZSFi6r53NiLV9xzaHFovW4oXb9fjR0j9KNF36tw

And make it available on your web server at this URL:

http://ci.focusedlabs.io/.well-known/acme-challenge/6z1rbMF-R6XdSo-s8HrTMKEeLGIvszkIA1IJPqzr9iU
```

If you use a wildcard `*.focusedlabs.io` you will be required to add a TXT record to your DNS

```bash
Please deploy a DNS TXT record under the name
_acme-challenge.focusedlabs.io with the following value:

6gcFZfRzUVujnYkhMEF8po1i55e4NscuSFnUcnYR4CQ

Before continuing, verify the record is deployed.
```

After completing the verification, you will have the certificates in `~/letsencrypt/live`

## Deploy Concourse to AWS using [concourse-up](https://github.com/EngineerBetter/concourse-up)

Now we have certificates and AWS all ready to go so let's deploy Concourse. 

Download the latest release of `concourse-up` from github https://github.com/EngineerBetter/concourse-up/releases and put it in your path.

Now in one simple command our concourse deploy will begin

```bash
concourse-up \
	deploy \
	--domain ci.focusedlabs.io \
	--tls-cert "$(cat $HOME/letsencrypt/live/focusedlabs.io/fullchain.pem)" \
	--tls-key "$(cat $HOME/letsencrypt/live/focusedlabs.io/privkey.pem)" \
	--region "us-east-1" \
	focusedci
```

Now watch the magic happen!

## Next Steps

From here you can deploy a pipeline and start to experiment with all the [community resources](https://concourse-ci.org/community.html#community-resources). Or you could set up a pipeline that will automatically refresh the certs with certbot.

Have fun and happy deploying!