Verification of Self-Signed Certificates
Tags: security, python, webdev, devops
Published: 2021-01-05 18:53:38

When interfacing to third-party web services, one often has to deal with self-signed SSL certificates that trigger verification errors. One workaround is to suppress those errors. (For instance, the Curl tool has the 'insecure' flag for this purpose.) However, at Smarking, we found ways to verify such certificates and to safeguard data communication from Man-in-the-Middle attacks. 

Conventionally a web browser relies on a Public Key Infrastructure (PKI) to verify SSL certificates. Every certificate is signed by another (signing) certificate. That signing certificate must be signed by another, in a chain ending on a trusted certificate. This linkage allows a web server operator to switch to a new SSL certificate without requiring visitors to his website to update their web browsers. Alternatively, he could ask his users to trust his specific certificate so that the browser would not need to walk up the signature chain to verify it. 

An SSL certificate carries inside it a public key of the webserver. On a conceptual level, the authenticity of that public key is the thing that allows us to establish an authenticated Diffie-Hellman key exchange between the browser and the server. Thus, if we could verify that it is the correct public key,  we would have "verified" the certificate. However, instead of verifying a long public key, we could instead verify its checksum that is usually much shorter. A checksum of the entire certificate is called its *fingerprint*, and it's always formatted as a colon-separated list of hex codes. For instance, here is the SHA-256 fingerprint of the certificate served by https://google.com: 

```
14:71:16:87:6D:F6:76:8E:98:E5:66:62:70:64:F1:0F:F8:0F:87:39:B8:55:4C:47:26:22:DF:FA:7D:1D:A5:FE
```

To retrieve the details of a website's certificate, click on the lock icon in your browser's URL bar and then inspecting SSL certificate details. Alternatively, you could use the following shell script:

```bash
#!/bin/bash
HOST=example.com
PORT=443
PROXY=1.2.3.4:8888

# If your environment does not require a HTTP proxy, delete the '-proxy $PROXY' parameter below
echo quit | openssl s_client -showcerts -servername $HOST -connect $HOST:$PORT -proxy $PROXY > result.txt
```

The output file `result.txt` includes the certificate in PEM format and metadata. The PEM format consists of binary data encoded using Base64 into ASCII, enveloped with "begin" and "end" lines like so:

```
-----BEGIN CERTIFICATE-----
<certificate encoded in base64 encoding>
-----END CERTIFICATE-----
```

You may feed the `result.txt` file into the following command to compute the fingerprint of the certificate. (The `openssl` tool would use the first certificate it finds in the input file and ignores everything else.)

```bash
$ openssl x509 -noout -fingerprint -sha256 -inform pem -in result.txt
```

The above command uses a `-sha256` switch, which determines the length of the fingerprint to be 32 bytes. There are only a few widely-used variants, therefore the length of the fingerprint identifies the algorithm used to derive the fingerprint.

Here are three methods by which you can verify certificates by their fingerprints. 

## Method #1: Use Python

Verification of certificates by a fingerprint is supported out-of-the-box by the `urllib3` library using the `assert_fingerpint` parameter,

```python
import urllib3
from urllib.parse import urlparse

def http_get_request(url, fingerprint):
  parsed_url = urlparse(url)
  host = parsed_url.netloc
  path = parsed_url.path
  pool = urllib3.HTTPSConnectionPool(host, assert_fingerprint=fingerprint)
  response = pool.urlopen('GET', path)
  return response

response = http_get_request('https://example.com/a/b/c', '14:71:...')
print(response.data)
```

Notice that the fingerprint option configures an `HTTPSConnectionPool` object which could then be used to make a *series* of queries against a website, such that each of the queries would verify the fingerprint of the certificate.

The Python's `requests` library supports certificate fingerprint verification also because it builds upon the `urllib3` library. It is based on adapter objects that return the `HTTPSConnectionPool` objects discussed above, and it provides a method `Session::mount()` which allows setting a custom adapter for a particular base URL. Putting this together, we have this code:

```python
from urllib.parse import urlparse

def create_fingerprint_session(url, fingerprint):
  host = urlparse(url).netloc
  s = requests.Session()
  s.verify = False
  s.mount('https://{}/'.format(host), FingerprintAdapter(fingerprint))

session = create_fingerprint_session(('https://example.com/a/b/c', '14:71:...')
response = session.get(url)
print(response.text)
```

Note that the `verify` setting must be set to False, otherwise, the `requests` library would also try to verify the SSL certificate using the conventional way, by the signature chain and the domain name. 

(Note that the `verify` parameter may also be set to a location of a certificate file that contains a concatenated list of trusted certificates in PEM format. However, a self-signed certificate is signed by a custom Certificate Authority (CA), but the certificate of the CA is usually unknown to us. Thus, we do not use this option but set `verify` to `False`.)

All that remains now is to implement the `FingerprintAdapter`. The quickest way is to subclass `HTTPAdapter` class and to modify the methods that create an `HTTPPoolConnection` object to include the `assert_fingerprint` option: 

```python
from requests.adapters import HTTPAdapter

class FingerprintAdapter(HTTPAdapter):
  """
  A TransportAdapter that allows to verify certificates by fingerprint
  """
  def __init__(self, fingerprint, *args, **kwargs):
    self._fingerprint = fingerprint
    HTTPAdapter.__init__(self, *args, **kwargs)

  def init_poolmanager(self, *args, **kwargs):
    kwargs['assert_fingerprint'] = self._fingerprint
    return super().init_poolmanager(*args, **kwargs)

  def proxy_manager_for(self, *args, **kwargs):
    kwargs['assert_fingerprint'] = self._fingerprint
    return super().proxy_manager_for(*args, **kwargs)
```

In summary, set `verify=True` when working with certificates signed by a trusted CA, otherwise set `verify=False` and mount a `FingerprintAdapter` when verifying self-signed certificates by fingerprint. Test that the verification is working by altering the fingerprint value and observing a security error.



## Method #2: Site-wide

What if you wished to use other tools, besides Python, to query web sites signed with self-signed certificates? A site-wide solution is to add the self-signed certificate to a list of trusted certificates if it's there, no further signature checking will be made by the verifier.

However, the downside of this method is that a compromised trusted third party could now sign certificates for any domain which all programs on the machine would trust. This is a significant security risk for a long-lived server, but it may be tolerable if "site-wide" does not extend beyond a Docker container which runs a program that only connects to one endpoint.

The following instructions are for Ubuntu or Debian; for other distributions, make necessary adjustments. 

Look in directory `/usr/share/ca-certificates` and you will see the directory `mozilla`, with many certificate files inside it. Make your own subdirectory on the same nesting level, for instance, `/usr/share/ca-certficates/custom` and put in it self-signed certificates of interest in PEM format, stored as separate files with extension `.crt`. Next, edit `/var/ca-certificates.conf` and list the custom certificates after the `mozilla` certificates. For instance,

```
...
mozilla/USERTrust_RSA_Certification_Authority.crt
custom/example-com-self-signed.crt
custom/another-example-com-self-signed.crt
```

Next, run `update-ca-certificates` command. Once that's done, symlinks to your certificates would appear in `/etc/ssl/certs` directory. At this point, the `curl` tool would work to accept the self-signed certificate from `example.com`. 

However, the fingerprint method described previously, had the advantage that it worked even if there was a domain name mismatch. A mismatch would happen if you queried the target HTTPs server by an IP address (e.g.`https://1.2.3.4/a/b/`). If this is your situation, you can add an entry to `/etc/hosts` file to query the web server using the precise domain name that is listed inside the self-signed certificate.

The side-wide method works for all tools that rely on the `libopenssl` library, which includes `curl`. However, it is not sufficient for Python's `requests` library since it does some of its own checking of certificates.



## Method #3: Strip SSL

Another way to allow a variety of tools to access HTTPs websites signed by self-signed certificates is to access them through a trusted *proxy* server that would strip the SSL after verifying the legitimacy of the self-signed certificates using fingerprints. We could implement such a proxy server in Python using the techniques above. Alternatively, we could use a utility program called `stunnel` that stands for the "Universal SSL Tunnel."

First, prepare a `connection.conf` configuration file like this one:

```
pid = /var/run/stunnel1.pid
CApath = /etc/ssl/certs
foreground=yes

[connection1]
verifyChain=no
verifyPeer=yes
client=yes
accept=8081
connect=1.2.3.4:443
sni=example.com
```

Then, run `stunnel` with the configuration file as the argument to have `https://example.com` proxied as `http://localhost:8081`, 

```bash
$ stunnel connection.conf >& output.log &
$ curl 'http://localhost:8081/a/b/c'
```

An important thing to notice in the example configuration file is the `verifyChain` and `verifyPeer` options. They combine to verify the certificate by a fingerprint only and would ignore an incomplete signature chain. These options were added to `stunnel` in July 2016, in version 5.34. Another thing to notice that the domain name doesn't matter. The `sni` parameter is used only to instruct the webserver which virtual host you are interested in, but it plays no role in validating the certificate. 

To run `stunnel` site-wide make the following configuration changes: store the configuration as `/etc/stunnel/stunnel.conf`, remove the `foreground=yes` bit, set `pid` to `/var/run/stunnel.pid` and add additional connection sections as needed. 



## Summary

We have demonstrated three ways to work with self-signed certificates without compromising security. At Smarking, we are trusted by vendors of parking systems to protect their data, and we use such techniques to justify their trust.

To learn more about Smarking, visit [www.smarking.com] (https://www.smarking.com/).