If you’re not using SSH certificates you’re doing SSH wrong | Episode 1: Keys versus Certificates
Tags: devops, security, opensource, sre
Published: 2020-08-21 18:51:11

SSH is ubiquitous. It's the de-facto solution for remote administration of *nix systems. But SSH has some pretty gnarly issues when it comes to usability, operability, and security.

You're probably familiar with these issues:

* **SSH user experience is terrible.** SSH user on-boarding is slow and manual. Connecting to new hosts produces confusing security warnings. You're left with weird new credentials to manage with little guidance on how to do so.
* **Operating SSH at scale is a disaster.** Key approval & distribution is a silly waste of time. Host names can't be reused. Homegrown tools scatter key material across your fleet that must be cleaned up later to off-board users.
* **SSH encourages bad security practices.** Rekeying is hard, so it's not done. Users are exposed to key material and encouraged to reuse keys across devices. Keys are trusted permanently, so mistakes are fail-open.

The good news is this is all easy to fix.

None of these issues are actually inherent to SSH. They're actually problems with SSH _public key authentication_. The solution is to switch to _certificate authentication_.

**SSH certificate authentication makes SSH easier to use, easier to operate, and more secure.**


## Background

At smallstep, certificates are kind of our jam. We build open source software that lets you run your own private certificate authority and manage X.509 (TLS/HTTPS) certificates.

SSH certificates have been on our radar for a while. From our perspective they're all pros, no cons. But no one uses them. Why not? We've asked hundreds of people that question. Here's what we found out:

* People do use SSH certificates. In fact, pretty much _everyone_ operating at scale that knows what they're doing does ([Facebook](https://engineering.fb.com/security/scalable-and-secure-access-with-ssh/), [Uber](https://medium.com/uber-security-privacy/introducing-the-uber-ssh-certificate-authority-4f840839c5cc), Google, [Netflix](https://github.com/Netflix/bless), [Intercom](https://www.usenix.org/conference/srecon17europe/program/presentation/sheridan), [Lyft](https://eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d), etc), but...
* Certificates and public key infrastructure (PKI) are hard to grok. People don't immediately understand the benefits.
* There's a (small) tooling gap that exacerbates this knowledge gap. It's not hard to fill, but people are wary to do so themselves without a deeper understanding of PKI concepts.
* More than anything, SSH certificates haven't gotten the press they deserve. _Most_ people we asked hadn't heard of them at all.

We're convinced that SSH certificates are the right way to do SSH. They're not that hard to understand, and it's well worth the effort. SSH certificates deserve more press, and broader use.

![why don't more people know about ssh certificates?](https://smallstep.com/uploads/why-dont-more-people-use-ssh-certs.png)


## Public key authentication

Most SSH deployments use _public key authentication_, which uses asymmetric (public key) cryptography with a public / private key pair generated for each user & host to authenticate.

The magic of asymmetric cryptography is the special correspondence between a public and private key. You can _sign_ data with your private key and someone else can _verify_ your signature with the corresponding public key. Like a hash, it's computationally infeasible to forge a signature. Thus, if you can verify a signature, and you know who owns the private key, you know who generated the signature.

Simple authentication can be implemented by challenging someone to sign a big random number. If I open a socket to you and send a random number, and you respond with a valid signature over that number, I must be talking to you.

This is an oversimplification, but it's more or less how SSH public key authentication works. Certificate authentication works the same way, but with an important twist that we'll get to in a moment.

To SSH to a host using public key authentication the host needs to know your public key. By default, your public key must be added to `~/.ssh/authorized_keys`. Maintaining this file for every user across a fleet is operationally challenging and error prone.

![SSH Deployment Rube Goldberg Machine](https://smallstep.com/uploads/ssh-deploy-rgm.png)

SSH user onboarding with public key authentication usually starts with some baroque incantation of `ssh-keygen`, hopefully pulled from a runbook, but more likely cribbed from stack overflow. Next you'll be asked to submit your public key for _approval and distribution_. This process is typically manual and opaque. You might be asked to email an administrator or open a JIRA ticket. Then you wait. While you're doing that, some poor operator gets interrupted and told to add your key to a manifest in some repo and trigger a deploy. Once that's done you can SSH. Since key bindings are permanent, your SSH access will continue in perpetuity until someone reverses this process.

## Certificate authentication

Certificate authentication eliminates key approval and distribution. Instead of scattering public keys across static files, you bind a public key to a name with a _certificate_. A certificate is just a data structure that includes a public key, name, and ancillary data like an expiration date and permissions. The data structure is signed by a _certificate authority_ (CA).

    $ ssh-keygen -L -f id_ecdsa-cert.pub
    id_ecdsa-cert.pub:
            Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
            Public key: ECDSA-CERT SHA256:O6M6oIjDm5gPm1/aTY619BgC3KSpS4c3aHVWxYh/uGQ
            Signing CA: ECDSA SHA256:EY2EXJGoPv2LA6yEbjH+sf9JjG9Rd45FH1Wt/6H1k7Y
            Key ID: "mike@example.com"
            Serial: 4309995459650363134
            Valid: from 2019-09-11T14:50:01 to 2019-09-11T18:50:01
            Principals:
                    mike
            Critical Options: (none)
            Extensions:
                    permit-X11-forwarding
                    permit-agent-forwarding
                    permit-port-forwarding
                    permit-pty
                    permit-user-rc

To enable certificate authentication simply configure clients and hosts to trust any certificates signed by your CA's public key.

On each host, edit `/etc/ssh/sshd_config`, specifying the CA public key for verifying user certificates, the host's private key, and the host's certificate:

    # Path to the CA public key for verifying user certificates
    TrustedUserCAKeys /etc/ssh/ssh_user_key.pub
    
    # Path to this host's private key and certificate
    HostKey /etc/ssh/ssh_host_ecdsa_key
    HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub

On each client, add a line to `~/.ssh/known_hosts` specifying the CA public key for verifying host certificates:

    @cert-authority *.example.com ecdsa-sha2-nistp256 AAAAE...=

That's it. That's literally all that you need to do to start using certificate authentication. You can even use it alongside public key authentication to make transitioning easier.

![SSH Certificate Authentication - Example Architecture](https://smallstep.com/uploads/ssh-certificate-authentication.png)

Static keys in `~/.ssh/authorized_keys` are no longer needed. Instead, peers learn one another's public keys on demand, when connections are established, by exchanging certificates. Once certificates have been exchanged the protocol proceeds as it would with public key authentication.

To learn more about SSH certificates visit [the Smallstep website](https://smallstep.com/sso-ssh?utm_campaign=Dev.to&utm_source=dev.to). You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!