Am author.

Yea, fair... there's probably enough to cover for a whole follow-up on nuts & bolts and these sorts of considerations. This post was more about raising awareness and it was already super hard to keep it to 3000 words :P.

I think I can at least partially address all of these points though.

> Do all OS distributions have new enough openssh, is it ever built with missing support by default?

I haven't done an analysis but certificate authentication was added in OpenSSH 5.4 in March, 2010. So it's almost ten years old. So yea, I think all reasonably current distros have support at this point.

> what extensions are necessary in the certs to be trusted for ssh or the CA cert (is your average test CA cert going to be valid for ssh without tweaking)?

To clarify, SSH doesn't use X.509 like TLS/HTTPS. OpenSSH invented its own certificate format.

The docs for `ssh-keygen` have pretty good info on this.

* https://man.openbsd.org/ssh-keygen.1#CERTIFICATES has basic cert info
* https://man.openbsd.org/ssh-keygen.1#O has info on some cert options / extensions

If you have experience administering nix boxes & SSH the options / extensions should look pretty familiar. Basically, it's a lot of the same stuff you'd ordinarily put in config files on each host, but instead you're baking it into certificates.

> Do any (or all?) cloud server products regularly accept cert based configuration just as they accept public key lists?

As far as I know, none of the cloud providers work with SSH certificates out of the box. It's kind of weird, actually, because it seems like super low-hanging fruit and would be a better user experience. Instead, they use public key authentication and deliver pubkeys to an authorized keys file for you (via their metadata APIs).

That said, it's super easy to set this up yourself on a cloud VM. You could easily bake the necessary bootstrapping into an AMI. Alternatively, you could use a startup script. One of the things we're working on is streamlining this experience. For now, here's a gist demonstrating the concept (it assumes you already have `step-ca` running):

https://gist.github.com/mmalone/a5980799c9c6ec9d6530372d5b60...

Most of the work there applies to other tools but obviously you'd use a different client instead of `step` to get the certs.

> How will this interact with other features like using a pkcs mechanism for a token key?

Oh hrm. This is a good question but I'm not sure, mostly because I haven't put SSH keys in a PKCS module myself. I know it's possible. You can implement the `ssh-agent` API pretty easily and back it by whatever you want. This is on our roadmap but we haven't done much investigation. It's possible that it just works already with standard `ssh` and `ssh-agent`s.

For MFA, you can still use PAM authentication with something like Duo if you want. Certificate authentication basically works exactly like public key authentication, but it manages public key distribution differently (using certificates).

That said, personally, I think the right place for MFA is during certificate issuance rather than on each SSH connection. That way you're not constantly asking people to MFA. This decision is somewhat subjective and depends on your risk profile / threat model.

> revoking

Generally the best practice for certificates is to issue them for a short enough duration that you're comfortable not doing revocation. I think a work day is pretty safe and reasonable, but some people may not be ok with that and choose something shorter. Of course you'd have to reauthn more frequently then to get a new certificate. A thought to ponder: for any reasonably sized infrastructure public key removal won't be instantaneous either.

That said, SSH does have cert revocation capabilities. You create a "revoke file" and configure it using a `RevokeFile <filename>` directive in your `sshd_config`. So it's managed on each host, which means there are no connectivity issues, but also means you have more of a management problem (on par with managing authorized_keys :/). For that reason, if you really need revocation, my recommendation is to do revocation checks in a bastion host. Then you only need to maintain the revocation list in one place.

> Also questions like what does an eaves dropper see in the clear and how does it differ from what they see with raw public key?

Great question. I should know the answer to this, but I don't. I just spent like 20 minutes perusing the specs at:

* https://www.openssh.com/specs.html

and it's not obvious just from skimming. Also couldn't find a definitive answer from the google machine. I'll need to read the specs more closely.

That said, it looks like pubkeys/certificates are exchanged during diffie-hellman key exchange, before a symmetric key has been negotiated. The implication is that this information is sent in cleartext. TLS1.2 does the same thing, but 1.3 keeps certificates confidential. So there's possibly an infrastructure enumeration vector here -- a passive attacker could see whatever info is in your certs (usernames, public keys, permissions). It looks like normal public key authentication would also transmit a lot of this same data, but maybe not the permissions. Again, I'd have to look at the spec more closely. I think for most people this is not a big risk and the profile looks pretty similar to pubkey authn, but I'm saying that without a complete understanding so caveat emptor.

> I find it insane that ssh ultimately ends up with a weaker trust model than a pretty average website in a typical install due to less certainty and clarity about what a normal certificate based configuration should look like.

Lol yes. Me too.

To be fair though the trust on first use model with raw pubkeys made SSH way easier to deploy, which was one of the things that allowed SSH to displace telnet and, from a security perspective, it was obviously a huge improvement over telnet. So I don't fault the SSH folks for this. But it's 2019 now so we can/should do better!