GitHub Exposed A Private SSH Key: What You Need To Know
Published: 2023-04-10 14:10:11

Secret leakage is a growing problem affecting companies of all sizes: including GitHub.

[GitHub recently made an announcement on their blog](https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/) regarding their SSH keys:  "At approximately 05:00 UTC on March 24, out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com."  They went on to say that:  "This change only impacts Git operations over SSH using RSA. If you are using ECDSA or Ed25519 encryption, then you are not affected."

This is further evidence that secrets sprawl is not just being driven by inexperienced developers or new teams. In our [State of Secrets Sprawl 2023 report](https://www.gitguardian.com/state-of-secrets-sprawl-report-2023), we uncovered more than 10,000,000 new secrets that were pushed to public GitHub repos. This is a 67% increase in the number of secrets we detected over the previous year's report, while GitHub itself only saw a 27% increase in new accounts.

The increase in detected hardcoded credentials is being driven by many factors, but it is clear that the developers involved in the incidents range in seniority from novice to expert and from organizations of all maturity levels. From our report, we discovered that 1 in 10 committers exposed a secret in 2022. If you have exposed a secret publicly, you are certainly not alone.  GitHub serves as a good reminder we must stay vigilant in our security practices, no matter how large our team.

Let's take a look at what the risks are in this situation, how GitHub handled the remediation process, and some simple steps to avoid exposing your own private keys publicly. 

The Risks Of Leaked Private SSH Keys
------------------------------------

Later in their post, GitHub said: "We [replaced our RSA SSH host key] to protect our users from any chance of an adversary impersonating GitHub or eavesdropping on their Git operations over SSH. This key does not grant access to GitHub's infrastructure or customer data."

While we can take comfort in this last part, no customer data is exposed. It is also good that there is no known risk of a takeover of their infrastructure, especially given how many developers and so much of the internet relies on it day to day. 

But there is a serious risk from "adversary impersonating GitHub or eavesdropping on their Git operations."  This is what is referred to as a "[man-in-the-middle attack](https://en.wikipedia.org/wiki/Man-in-the-middle_attack)," where the end user can not tell the difference between the legitimate other party and the attacker. 

[![Man in the middle attack](https://lh6.googleusercontent.com/rSofPOVdmU1KcXQrPnskjhrehMed_YO11eEFoMf-CTBoAbHy1teaTVgH2LVR7bF6JxBFNUowLjPPU4j-XviYfyLADdw26y390b2aD6ptHFrvLxlvZLX_dpY-I3JOjrDxlIhxYBD9epeK16KS1lXHpA)](https://en.wikipedia.org/wiki/Man-in-the-middle_attack)

With so many developers and services relying on GitHub SSH communications to be truly secure, it made a lot of sense for the company to revoke and replace their SSH key. They also published, in that same article, [what you need to do](https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/#what-you-can-do) if you are affected and how to tell if you are affected.  If you get a warning when connecting to GitHub via SSH, `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!` you will need to remove the old key or manually update your `~/.ssh/known_hosts` file.

Aside from end users, your GitHub Actions might be affected. The rotation of their private SSH key will mean workflow runs will fail if they are using `actions/checkout` with the ssh-key option. This might feel familiar to [CirlceCI customers](https://blog.gitguardian.com/how-to-secure-your-ci-cd-pipeline/), who experienced a similar unexpected workflow interruption due to a leakage incident.

### Reactions And Risk From Mass Replacement

As expected, the developer community has had a wide range of public reactions to the event.  No one, it seems, is happy about the situation., but some people are helping [spread the word, without adding much commentary](https://twitter.com/eddiejaoude/status/1639175005157343232). Other conversations speculated on how this could have happened and further potential security issues that might come into play.

One concern that popped up [in several conversations](https://news.ycombinator.com/item?id=35285390#35285492) was the potential for a new man-in-the-middle-attack. Attackers know that keys are going to be replaced by many developers by using the command `ssh-keygen -R github.com` without manually verifying the new fingerprint is the expected value. It is entirely possible that a bad actor could insert their own fingerprint in certain situations. 

This is a great reminder that we should be embracing Zero Trust, moving away from 'trust, but verify' to a stance of 'verify, only then trust.'

## A Lesson In Incident Remediation

While it is an unfortunate situation that GitHub has needed to rotate this certificate, affected so many customers, and put out such a notice, we want to highlight some best practices they used to respond to the event. 

While we don't know how long the secret was exposed, we can assume it was fairly recently.

Acting quickly and deliberately, without panicking, is [essential in secrets remediation](https://youtu.be/QmY6pFJ0XVw). Given the timestamp they published, we can assume this incident caused a sleepless night for the team involved, and they had to weigh many factors. It seems they thought the security risks merited the key rotation, even if it would potentially affect a large swath of users.

We can also applaud them for communicating very quickly and publicly. While most people might not be subscribed to the GitHub blog, they pushed the news into [multiple channels](https://twitter.com/GitHubSecurity/status/1639141426159316993) as publicly as they could. While some in the community felt there was a little bit too much 'marketing spin' when using terms like "we have no reason to believe" or "an abundance of caution," the GitHub team has been straightforward in their communications throughout all of their security incidents. We will take a little soft peddling versus not getting an overview that includes remediation steps.

How To Prevent Leaking A Credential
-----------------------------------

Nobody is perfect, and we all make mistakes from time to time. Expecting humans always to deliver flawlessly only sets you up for disappointment, especially when dealing with something as conceptually so complex as Git. While we all love the world's favorite version control system, most developers can tell you it is a bit too easy to do something wrong, such as pushing to the wrong remote repository.

While we don't know exactly what caused this particular incident, we at GitGuardian have a lot of experience with similar leakage situations and want to use this opportunity to remind you of some best practices.

### Never Add Certificates To Version Control 

This one might seem obvious, but all of us touching code are sometimes guilty of this. There are times, especially when debugging something when you just need to test a credential works. This is true of passwords, API keys, and, relevant to this, certificates.

You might add a new cert directly in the project with the full intention of moving it someplace safe or modifying your `.gitignore` file, but then life happens, and you get distracted. One `git add -A` and `git commit` later, your cert is now in your git history. One 'git push' later, and it is now out in a shared repo.

This is where tools like git hooks and [ggshield](https://www.gitguardian.com/ggshield) can really come in handy. Setting up a [pre-commit hook](https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit) is very quick and simple. Once set up, every attempt at committing code will trigger a scan that will halt the operation if a secret is discovered in any of the tracked files. Catching a secret before it becomes part of your git history is the safest and cheapest place to remediate the situation.

### Double Check That Is The Right Remote

One of the strengths of Git is the ability to easily push all your changes to any remote repository you have permission to connect to. Unfortunately, this can get confusing rather quickly. Image your have two repos, one public and one private, with a remote name of  `proj-1-p` and `proj-1-pr`respectively. You are only one letter away from potentially pushing to the wrong place. Accidents happen. It is a very good idea to make those origin names more explicit.

Another factor a lot of developers deal with is their own shell aliases. For example, it is common to alias `gpo` as a shortcut for the often typed `git push origin.` It can be very easy to go into automatic mode and use a shortcut that might be sending the wrong branch to the wrong place.  When in doubt, type it out.

### Rotate Secrets Often

While rotating your SSH key might not be necessary for this event, do you know when the last time was that you rotated it? Was it this decade? If you don't know, then right now is a good time to go remedy that. The longer any valid credential lives, the more opportunities exist for that credential to be found and misused.\
We have found that teams that rotate keys more often are at the top of the pyramid, expert level when it comes to [secret management maturity](https://blog.gitguardian.com/a-maturity-model-for-secrets-management/). Going through the exercise of rotating credentials regularly when there is no emergency will prepare you to better handle times when there are those added pressures. 

Stay Vigilant And Safe
----------------------

Leaks happen to us all sometimes, even to massive platforms like GitHub. While there likely are many workflows that have been affected and many developers who need to update their `known_hosts` files, thanks to GitHub's clear communication on the incident, there is a clear remediation path, and we know the overall scope of the incident.

This is a good time to reflect on your own secrets management and detection strategy as well. GitGuardian is here to help.  You can help keep credentials, like certificates, out of your source control with ggshield. And if a secret does get pushed into a shared repo, the GitGuardian platform can let you know immediately, as well as help your team coordinate through the remediation process.

Stay safe out there; we are all on the internet together.