What a mysterious bug taught us about how Docker stores registry credentials
Tags: docker, devops, microservices
Published: 2020-06-23 14:44:29

We recently ran into a mysterious bug that required hours of digging into the
arcane details of Docker's registry credentials store to figure out.  Although
in the end the fix turned out to be easy, we learned a thing or two along the
way about the design of the credentials store and how, if you're not careful,
it can be configured insecurely.

[Blimp](http://kelda.io/blimp), sometimes needs to pull private images from a
Docker registry in order to boot those images in the cloud.  This typically
works fine, but unfortunately, when some users started
Blimp, they were getting the following error message:
```shell
Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials
```

At first, we were completely baffled by this cryptic message and had no clue it
was related to our handling of credentials.  To understand how we figured it
out, first you need to know a little about how modern Docker credentials are
handled.

## Docker's External Credentials Store

The recommended way to store your Docker credentials is in an external
credentials store. In your Docker config file, which is usually located at
`~/.docker/config.json`, there are two fields you can use to configure how
Docker gets and stores credentials: `credsStore` and `credHelpers`.


`credsStore` tells Docker which helper program to use to interact with the
credentials store. All helper programs have names that begin with
`docker-credential-` -- the value of `credsStore` is the suffix of the helper
program.

For example, if you work on a Mac laptop, you might decide to use the Mac OS
keychain. The name of the helper program to use the keychain is
`docker-credential-osxkeychain`. So your `config.json` would include the
following:

```json
{
  "credsStore": "osxkeychain"
}
```
 
If you want to see what credentials Docker currently has for you, you can use `list`. For example:
```
docker-credential-osxkeychain list
```

The result is a list of pairs of servers and usernames. For example:
```json
{
  "http://quay.io":"kklin",
  "https://index.docker.io":"kevinklin"
}
```

You may also notice `credHelpers` in your `config.json`. These helpers are similar
to `credsStore`, but are used to generate short lived credentials. For example,
if you use [gcr](http://gcr.io/), `gcloud` installs a `credHelper` that uses
your Google login to get tokens. This way, Docker never has your Google
credentials directly -- the `docker-credential-gcloud` acts as a middleman
between Docker and your Google credentials.

Once again, here's the error message our users were getting:
```
Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials
```

We were able to run the `docker-credential-osxkeychain` `list` and `get`
commands to see the credentials for `1234.dkr.ecr.us-east-1.amazonaws.com`, so
why were we getting an error that there weren't any credentials??

## In the Beginning: Docker Stores Your Registry Password In Your Config File

It turns out that external credentials stores weren't
[added](https://github.com/moby/moby/pull/20107) to Docker until version 1.11,
in 2016.  Before 1.11, Docker stored credentials via a config field called
`auths`. This field is stored in the same file as the `credStore`: `~/.docker/config.json`.

Whenever you logged into a registry, Docker would set the value of auths to
your password. For
[example](https://www.projectatomic.io/blog/2016/03/docker-credentials-store/),
your config file might contain the following:
```go
{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}
```

What we learned the hard way is that there's a quirk with Docker's `login`
command. When you log in using `docker login`, Docker adds an entry via the
`credsStore` **and** in `auths`, using slightly different server names. Your
credentials are properly stored in the credentials store, but the entry in
`auths` doesn't contain the username or password. The result looks something
like this:
```go
{
"auths": {
  "https://index.docker.io/v1/": {}
}
```

The problem is that Blimp grabs credentials from both `auths` **and**
`credsStore`. So it was passing two copies of the credentials to the Docker
image puller -- one with the correct username and password, and one without the
password at all.

Unfortunately, Docker preferred the `https://` version of the credential, and
attempt to pull the image with the empty credential. Thus, the `no basic auth
credentials` error.

Once we figured out that the problem was that an empty duplicate entry was
getting added to the insecure store, it was easy to [fix the
problem](https://github.com/kelda/blimp/blob/master/cli/up/up.go). All we
needed to do was add an `if` statement to skip empty credentials:
```go
	addCredentials := func(authConfigs map[string]clitypes.AuthConfig) {
		for host, cred := range authConfigs {
			// Don't add empty config sections.
			if cred.Username != "" ||
				cred.Password != "" ||
				cred.Auth != "" ||
				cred.Email != "" ||
				cred.IdentityToken != "" ||
				cred.RegistryToken != "" {
				creds[host] = types.AuthConfig{
					Username:      cred.Username,
					Password:      cred.Password,
					Auth:          cred.Auth,
					Email:         cred.Email,
					ServerAddress: cred.ServerAddress,
					IdentityToken: cred.IdentityToken,
					RegistryToken: cred.RegistryToken,
				}
			}
		}
	}
```

## A Potential Docker Credentials Security Risk

In the process of uncovering this bug, we noticed a potential security risk
that you may not be aware of.  As we learned, it's best practice to use an
external store to store your external registry credentials.  However, depending
on how and when you installed Docker it's possible you could still be using the
legacy `auths` method.  If you are, your `~/.docker/config.json` might look
something like this:
```go
{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}
```

This may look reasonable secure, the passwords appear to be a garbled bunch of
gibberish.  Surely those passwords are *encrypted*, right?

Guess again. All Docker did was encode the passwords using base64. And as David
Rieger pointed out on [Hacker
Noon](https://hackernoon.com/getting-rid-of-docker-plain-text-credentials-88309e07640d),
base64

> may look like encryption on first glance, but it's not. Base64 is a scheme
> for encoding, not encryption. You can simply copy the base64 string and
> convert it to ASCII in a matter of seconds.

That seemingly secure password of `aGVzdHVzZXI6dGVzdHBhc3N3b3Jk`? All you need to do to read the password is base64 decode it:
```
$ echo aGVzdHVzZXI6dGVzdHBhc3N3b3Jk| base64 -D
hestuser:testpassword
```

## The Moral of Our Story: Double Check Your Docker Credentials' Security

So that's the bad news: if Docker config file isn't properly set up, Docker is
storing your credentials password in plain text.

The good news is that it's easy to fix the problem.

All you and your team members need to do is take a quick look at
`~/.docker/config.json`. If it contains an `auths` password, get rid of it and
switch over to using a credentials store. To do so, just download the
appropriate `docker-credential-` helper for your system, and update the
`credsHelper` field in `~/.docker/config.json`.

Hope that helps!

Read the [Top 5 common mistakes when writing Docker Compose](https://kelda.io/blog/common-docker-compose-mistakes/)