Track Certificate Expiration with Jenkins and Python 3!
Tags: continuousdelivery, designpatterns, fosslinux, jenkins
Published: 2022-10-23 20:27:23

---
title: Track Certificate Expiration with Jenkins and Python 3!
published: true
date: 2022-10-23 20:26:00 UTC
tags: ContinuousDelivery,DesignPatterns,FOSSLinux,Jenkins
canonical_url: https://blog.engyak.co/2022/10/track-certificate-expiration-with.html
---

CI/CD tools aren't just for automatically deploying apps! Jenkins excels at enabling an engineer to automatically execute and test code - **but,** it has a hidden super-power: **Automating boring and intensive IT tasks(**_ **removing toil** _**)****.**

Let's take a common and relatable IT problem - it doesn't matter if you're a DevOps engineer, a Agilista, or even a "normal" systems engineer. Tracking certificate expiration is not an enjoyable task, and can often involve either manual checking or (usually) _ **outages** _ to discover that a certificate has expired.

This solution will have several major elements:

- An inventory of TLS-issued hosts
- A Python 3 script (leveraging OpenSSL) to open up TLS connections and fetch certificates
- A Jenkins pipeline to execute that script against that inventory daily, emailing the results

#### Inventory

Full transparency, this example is executed in a **home lab**. It's naive to assume that this task is trivial for any enterprise, but here are some potential approaches to building an inventory at scale:

- Write a Python script to ingest DNS zone files, and loop curl to see if any listen on port 443
- Fetch a report for a vulnerability scanner (Retina, Qualys, Nexpose)
- Searching PKI issuance reports (if available)

We also want to write our inventory file in a way that's friendly to our execution approach. Python is **dynamically typed** , and most IT automation is fine with that - we're not doing any hardcore programming for most of it. The vast majority of IT automation involves sending and processing files and I/O.

Python will change a variable to any data type when you tell it to, so it's useful to map out what we want. Here are the relevant data types. I will also include the symbols JSON uses to signify them (if applicable):

- String (`""`): This is a type that encapsulates a series of text characters
- Integer (No wrapping): Whole number, and Signed. It can be positive or negative, but there's no decimal point **(decimal points are their own unique flavor of complexity in computer programming)**
- List (`[]`): To a dyed in the wool software developer, this will be similar to an `array`. Lists are indexed by an `int`eger, and can contain any data type below it

  - Python has a neat trick where a for loop can return a `list` item instead of the index, which saves a great deal of code
  - Python can sort a list by executing the function `.sort()` on that object
- Dictionary (`{}`): This is an advanced construct, and provides an engineer with a great deal of capability (at the expense of performance, and code simplicity in some cases)

  - `dict`s store entries as key-value pairs, and the `index` is usually a `str`ing
  - Python can add to a `dict`ionary by adding a new key, e.g. `dictionary["newkey"] = "b33f"`

When planning software functionality, we want to always use the right tool for the job. Downstream APIs (e.g. OpenSSL) want to see a particular format for a parameter(e.g. TCP port should be an integer), so documentation research is a must at this phase. I'll explain my logic for this file:

- I want to easily iterate through the `list`, without addressing indexes, and I want it to be fast. I **should** use a `list` for the top-level data in the inventory (`[]`)
- I want to ensure that I don't accidentally address the keys wrong, so each individual entry **should** be a dictionary (`{}`) with the following **keys: fqdn, port**

  - **fqdn** should store a `str`ing
  - **port** should store an `int`eger

Example:

`[{ "fqdn": "vcenter.engyak.co", "port": 443 }, { "fqdn": "nsx.engyak.co", "port": 443 }] `
#### Python Code

Here's a copy of my code. To execute it, the following pip packages need to be installed:

- `fqdn`
- `OpenSSL`
- `ruamel.yaml`

`datetime` in particular does quite a bit of heavy lifting here. The package providers in the Python community have managed to solve most of the truly difficult work, so interpreting expiration dates is a simple comparison operation.

I heavily rely on functions for this code to work in a maintainable format - this code is only 167 lines long, but most of the usage is for readability.

Another point of note - when writing Python to execute in a pipeline, it helps to be Perl levels of dramatic when crashing code. Jenkins doesn't evaluate output by default, and the easiest way to notify of a problem is by using sys.exit(""). This is why I placed a crash if errors exist at the end of the list.

<script src="https://gist.github.com/ngschmidt/ae78700d73d6767251a85dee5d78a6b0.js"></script>
#### Jenkins

This configuration example **should** provide some basic level of functionality. Jenkins has a **lot** of capability, so this tooling can be endlessly tweaked to your needs.

First, let's set up a SMTP server. With a default installation, the settings are under **Dashboard -> Manage Jenkins -> Configure System** :

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9K8GcvStRh02dwfgJq8CHSySg8FsaxrVrF-C1R3aA15Yiatbjj7B6YfbuGwKRgDf9gTgaJPSKvIQrTBB4c7kyNDPZCFU9RbLPYMx7yBYMeQHwbuoUN27aBHtC4ciF4Xx136Ga0ncdaRZah9QrYfAmZgKKcw77cvQ52Csme5GgdOshADboXZLyBamJ/w400-h326/2022-10-23%2011_53_43-Configure%20System%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9K8GcvStRh02dwfgJq8CHSySg8FsaxrVrF-C1R3aA15Yiatbjj7B6YfbuGwKRgDf9gTgaJPSKvIQrTBB4c7kyNDPZCFU9RbLPYMx7yBYMeQHwbuoUN27aBHtC4ciF4Xx136Ga0ncdaRZah9QrYfAmZgKKcw77cvQ52Csme5GgdOshADboXZLyBamJ/s432/2022-10-23%2011_53_43-Configure%20System%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

Advanced Settings will allow you to configure SMTP auth, port, if applicable. If you use Gmail, you can still[leverage MFA and app passwords](https://support.google.com/accounts/answer/185833?hl=en), preserving MFA and avoiding password proliferation.

Now, let's set up a freestyle pipeline:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgClGCfvu0OuswCjj8SGBrc3S-iB7Mx30UyYTq_xd7SoTCraumK-CX-GIfQt6umsZ_BAhLu9LDaAMysRc6HStRUfyRhdQ4A4q4PJk2kOdJ8yXjb0kCbP6Z8Uywvu5yxW_9LRtm00PfEYM90ZVg0bxsQFJAxOdzL9c-dnR3jqzZSoCXC4F-AzHCkpZzZ/w640-h136/2022-10-23%2011_57_58-New%20Item%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgClGCfvu0OuswCjj8SGBrc3S-iB7Mx30UyYTq_xd7SoTCraumK-CX-GIfQt6umsZ_BAhLu9LDaAMysRc6HStRUfyRhdQ4A4q4PJk2kOdJ8yXjb0kCbP6Z8Uywvu5yxW_9LRtm00PfEYM90ZVg0bxsQFJAxOdzL9c-dnR3jqzZSoCXC4F-AzHCkpZzZ/s1177/2022-10-23%2011_57_58-New%20Item%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

With Jenkins, all things should be executed from source code. **This is the way**.

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4AA-d7Bo_dxoyt8psOCH1jrwzWOTaa_6_sEvVKb_rjdzHS1_B32IcbmEnjQPMfn0o-U9HKH4fbp6rgvTTf-YwO2E0zCAT-F22CclPhplpcBvMZbl7c0Km5_1hbeicdDtI4tF_m1DI40UIo166bDeLcB271_JoI0tfF45Dg9OqiRlGXUcGlxK-Ma0z/w364-h640/2022-10-23%2012_10_23-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4AA-d7Bo_dxoyt8psOCH1jrwzWOTaa_6_sEvVKb_rjdzHS1_B32IcbmEnjQPMfn0o-U9HKH4fbp6rgvTTf-YwO2E0zCAT-F22CclPhplpcBvMZbl7c0Km5_1hbeicdDtI4tF_m1DI40UIo166bDeLcB271_JoI0tfF45Dg9OqiRlGXUcGlxK-Ma0z/s756/2022-10-23%2012_10_23-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

We want to run this **daily,** irrespective of source code changes. This requires a slight deviation from the usual **Poll SCM** approach:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdHvpXALsmAE0Sj-htWVTowgFFoAOkrHq3iyXEQFipXV2y6-2gv_b1elcDY1YRnVu_iqQN_6HetWkJMT3hLp33GkYJ0qdrCPXrBUOGVIxB04SYO-FUG7PeiRCn2Bf2v3Wrknzc8UmKqDI9Z75MLxKjY17XarUL_6_zE16CcnnZTkSfz29N2Jy1U1mx/s320/2022-10-23%2012_11_22-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdHvpXALsmAE0Sj-htWVTowgFFoAOkrHq3iyXEQFipXV2y6-2gv_b1elcDY1YRnVu_iqQN_6HetWkJMT3hLp33GkYJ0qdrCPXrBUOGVIxB04SYO-FUG7PeiRCn2Bf2v3Wrknzc8UmKqDI9Z75MLxKjY17XarUL_6_zE16CcnnZTkSfz29N2Jy1U1mx/s324/2022-10-23%2012_11_22-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

As always, amnesic workspaces are best:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsybGo7Xsx2l1UORKTTjO5hGUHGUZ7w9_w2iELouPM13qdZ5OgjypQVly_OPoQxkiTXtlEeVPm6oo2DQDsAsMxEsvwNPI1y48BUZ_i4prgOeRnVk9Y-eYbyXyN9qImMUagzay4l0eArGp-WJCxIC4aFBSARuQUQm7m1cpzz7qtkAG9FmJvi9CS5fvQ/s1600/2022-10-23%2012_16_27-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsybGo7Xsx2l1UORKTTjO5hGUHGUZ7w9_w2iELouPM13qdZ5OgjypQVly_OPoQxkiTXtlEeVPm6oo2DQDsAsMxEsvwNPI1y48BUZ_i4prgOeRnVk9Y-eYbyXyN9qImMUagzay4l0eArGp-WJCxIC4aFBSARuQUQm7m1cpzz7qtkAG9FmJvi9CS5fvQ/s281/2022-10-23%2012_16_27-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

Python and the inventory file simplify the Jenkins configuration as well. Just execute the script as-is:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgqjWODWJ9xZZAQIeHSB0hf2JE4zpP8UIpadth_RdExfOEV1wvFuEHRPXwshzwLtWE3KWFwEqAk1t2lShBmf-ui-X4dvKEUgOfRmf8UN6hvK3XYN1W66aSg5PRGowjvt1nhMssMtqK7rSmKYQwXEOS2okjgwYZ7nWTE74hZlTJlsYr6oU5SsO-S8O-/w400-h196/2022-10-23%2012_16_40-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgqjWODWJ9xZZAQIeHSB0hf2JE4zpP8UIpadth_RdExfOEV1wvFuEHRPXwshzwLtWE3KWFwEqAk1t2lShBmf-ui-X4dvKEUgOfRmf8UN6hvK3XYN1W66aSg5PRGowjvt1nhMssMtqK7rSmKYQwXEOS2okjgwYZ7nWTE74hZlTJlsYr6oU5SsO-S8O-/s461/2022-10-23%2012_16_40-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

The final step is to add a **Post-Build Action** to email if there is a failure:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijLdv3OWt5kC80WufmW97ZWZ0je88gpvICayH-N1LpHde9MkizL7L_xJsWwIqiPSC3_dMgtX9-LbbcUvQbbzEU5XJKDHKrZLcY5m1M8ZCOHuFSpgB-xh1FqkbpIZbWIEbb3juAeQzgoPlUoQpycYhKaPmfo-nRuB11kz9nXs0GIubZ_3x7k7Ynab7t/w640-h184/2022-10-23%2012_16_50-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijLdv3OWt5kC80WufmW97ZWZ0je88gpvICayH-N1LpHde9MkizL7L_xJsWwIqiPSC3_dMgtX9-LbbcUvQbbzEU5XJKDHKrZLcY5m1M8ZCOHuFSpgB-xh1FqkbpIZbWIEbb3juAeQzgoPlUoQpycYhKaPmfo-nRuB11kz9nXs0GIubZ_3x7k7Ynab7t/s1042/2022-10-23%2012_16_50-certificate-tracker%20Config%20%5BJenkins%5D%20%E2%80%94%20Mozilla%20Firefox.png)

It really is this simple. Jenkins will now execute daily and email you a list of expired, and soon to expire certificates!

#### Lessons Learned

I'm going to improve this code. Here are some of my ideas:

- Provide a set of rules for each warning level, e.g. 14 days for Critical
- JSON Schema validation will help make the ingest more intuitive
- [Write/Add a Jenkins parser to chart different severity issues, not unlike Jenkins' flake8 plugin](https://www.jenkins.io/doc/pipeline/steps/warnings-ng/) 

I'm continually amazed at what the open source community can achieve with this level of simplicity. Would you consider this approach out of reach or too challenging?