Automated SSL on our Internal Applications
Tags: automation, security, devops
Published: 2019-02-16 14:47:41

---
title: Automated SSL on our Internal Applications
published: true
description: 
tags: automation,security,devops
---

*This article was originally posted on [**Bizongo Engineering**](https://hustle.bizongo.in/automated-ssl-for-our-internal-applications-8bcf2ffc8d4a)*

-----

If you’re from the software engineering industry, I’m sure you must be aware of how important is Secure Sockets Layer or SSL as an extra layer of  security on any application, especially if you deal with sensitive  information such as user authentication, secret tokens and payments.

Since we at Bizongo deal with such critical information on a daily basis like all e-commerce companies, it becomes evident that all applications should communicate as securely as possible. This is where SSL enters the scenario as  illustrated in the below image or as an interactive web-app **[here](https://tls.ulfheim.net/)**.

![Image Courtesy](https://cheapsslsecurity.com/blog/wp-content/uploads/2017/10/ssl-tls-handshake-process-1024x670.png)


While our public facing storefronts, back-end APIs have it (*along with curated firewall rules*), our internal applications trailed behind in this regard. So we went about exploring possible solutions to discover [**LetsEncrypt**](https://letsencrypt.org/docs/faq/), a free Certificate Authority usable to generate fully compliant certificates trusted by browsers.


![Our Reaction on discovering LetsEncrypt](https://i.imgflip.com/1cvwdb.jpg)

###**Our Situation?**

You must have thought procuring a LetsEncrypt SSL would be all but there’s more to it. This is due to a few of our peculiar requirements listed  below:

1. Our internal Environments across a few domains are all level 2 wild-cards, i.e: `*.qaX.domain.com` where x is 1,2,3…n. Traditional wild-card SSL don’t cover these as they address only 1 level, e.g: `*.domain.com` — [reference1](https://community.letsencrypt.org/t/allow-multiple-level-wildcard-certificates/67242) and [reference2](https://community.letsencrypt.org/t/multi-level-subdomain-for-the-wildcard-certificate/57319)
2. We didn’t want to end up installing a bunch of software on all servers making it  tedious to maintain, upgrade and debug while subsequently becoming very  un-scalable.
3. SSL certificates are procurable for a limited duration *(LetsEncrypt has a validity of* [**90 days**](https://letsencrypt.org/2015/11/09/why-90-days.html))  and therefore need periodic renewal. We wished to automate this for the  ease of our entire engineering team avoiding any potential issues arising due to an expired one.
4. All relevant services like *nginx* should reload to reflect the renewed SSL seamlessly.
5. Finally, our solution should also have Email / Slack notifications for better observability.



###**The Solution:**

Before diving in-depth, do note a Google Search will showcase plenty of articles detailing how to setup **LetsEncrypt** for installing and automating SSL renewals. However, most of them cover only *bare* domains and/or *www* but not *wild-cards*  thereby not helping us en-route to solving this problem. So, we thought  of writing one ourselves to benefit anyone with a similar use-case.

Enough said, lets get cracking!

To cover all above points, we started to make a Shell Script in [**combination**](https://id-rsa.pub/post/certbot-auto-dns-validation-with-lexicon/) with  the AWS API that hosts all our infrastructure. As LetsEncrypt adheres to high compliance, anyone generating or renewing SSL for a domain should be able to prove its ownership as well. This is doable via [**methods**](https://letsencrypt.org/how-it-works/) like **HTTPS** validation through a standalone web-server or alternatively, **DNS** validation via publishing temporary resolving TXT records.

We found the latter simple as it didn't expose our internal Environment’s  ports 443 to the world or need an additional overhead of white-listing LetsEncrypt’s IP Addresses. Since the DNS records are temporarily  needed, they can be created & deleted post every renewal.

Below is how the records look like:

```
_acme-challenge.qa1.domain1.com 300 IN TXT “<random-hash-value-1>”
_acme-challenge.qa2.domain2.com 300 IN TXT “<random-hash-value-2>”
....
...
..
.
_acme-challenge.qaX.domainX.com 300 IN TXT “<random-hash-value-X>”
```

Considering that, our initial iteration of the script checked for the LetsEncrypt package i.e: *certbot* on the target server before proceeding to implement the remaining logic. Here’s where it became tricky as Certbot depends on Python libraries, either v2.7 or v3.4+ with [**sudo privileges**](https://certbot.eff.org/docs/install.html). For the uninitiated, fiddling with these can break many parts of a Unix System (*like the* **apt***,* **yum** *package managers*).

So we discovered this [**utility**](https://github.com/Neilpang/acme.sh) which is a Shell Script working as an [**ACME**](https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment) protocol client for generating or renewing SSL certificates. We then decided to run our script on a single host (*as opposed to all*), transfer them to requisite servers, reload any relevant services with a Slack status message.

This is how our final custom script logic looks like:

1. Do an `openssl` call on our internal domains to check for existing SSL certificates and get its dates.
2. Check if today’s date differs with the above by 7 days, then execute the remainder of the script, else exit.
3. If executed, generate all aforementioned wild-card SSL and `rsync` it to the target folders with correct permissions on our internal servers.
4. Do a configuration test and reload `nginx` **only** if the above succeeds.
5. Use `curl` to trigger a Slack web-hook and send an appropriate message to the configured channel.

###**What was the Result?**

We learned how LetsEncrypt works to provide an extra security layer and  leveraged it to protect our internal environments as a completely  scheduled automated solution — saving us the hassle of manual  certificate management. Our internal applications also communicate via  encrypted channels that is a crucial safeguard in today’s era of data-snooping, phishing and digital theft.

----------

While we’re sure this post will help you deploy a similar solution to make your infrastructure more secure, are there any other ideas or best practices you would like to add to make this even better? Please feel free to share in the comments below.