Runner process unable to pull images (500: no basic auth credentials) from private ECR repo
Asked by Camilo Bernal:

I have a self-hosted runner configured on a Kubernetes cluster (EKS) and having some issues with the underlying runner process pulling images from ECR at runtime.

I've uploaded the auxiliary images from [Use your Docker images in self-hosted runners \| Bitbucket Cloud \| Atlassian Support](https://support.atlassian.com/bitbucket-cloud/docs/use-your-docker-images-in-self-hosted-runners/) into a private repo.

There is no issues pulling images from the cluster or even from within the containers (I've manually logged into the appropiate ECR repo), but it looks like the **runner has a java process that is triggered when a step is executed which pulls additional images at run time. This is where I get the following issue:**

**Status 500: {"message":"Head <https://XXX.dkr.ecr.eu-west-1.amazonaws.com/v2/XXX/tools/manifests/prod-stable>: no basic auth credentials"}**

It seems odd that the Java process doesn't use the docker config in the container.

Below is the redacted manifest I'm using to deploy the runner (note that I can deploy this runner successfully, the issue appears at job execution):

```
apiVersion: v1kind: Listitems:  - apiVersion: v1    kind: Secret    metadata:      name: runner-oauth-credentials      labels:        accountUuid: XXX        repositoryUuid: XXX        runnerUuid: XXXX    data:      oauthClientId: XXX=      oauthClientSecret: XXX==  - apiVersion: batch/v1    kind: Job    metadata:      name: runner    spec:      template:        metadata:          labels:            accountUuid: XXX            repositoryUuid: XXX            runnerUuid: XXX        spec:          containers:            - name: runner              image: XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/bitbucket/pipeline-runner:1.435              resources:                limits:                  memory: "8Gi"                  cpu: "2"              env:                - name: ACCOUNT_UUID                  value: "{XXX}"                - name: REPOSITORY_UUID                  value: "{XXXX}"                - name: RUNNER_UUID                  value: "{XXX}"                - name: OAUTH_CLIENT_ID                  valueFrom:                    secretKeyRef:                      name: runner-oauth-credentials                      key: oauthClientId                - name: OAUTH_CLIENT_SECRET                  valueFrom:                    secretKeyRef:                      name: runner-oauth-credentials                      key: oauthClientSecret                - name: WORKING_DIRECTORY                  value: "/tmp"                - name: PAUSE_IMAGE                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/bitbucket/pipeline-pause:latest"                - name: AUTH_PROXY_IMAGE                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/pipeline/XXX/pipeline-auth-proxy:prod-stable"                - name: CLONE_IMAGE                  value: "XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/pipeline-tools:prod-stable"              volumeMounts:                - name: tmp                  mountPath: /tmp                - name: docker-containers                  mountPath: /var/lib/docker/containers                  readOnly: true # the runner only needs to read these files never write to them                - name: var-run                  mountPath: /var/run            - name: docker-in-docker              image: XXX.dkr.ecr.eu-west-1.amazonaws.com/XXX/docker:20.10.7-dind              resources:                limits:                  memory: "8Gi"                  cpu: "2"              securityContext:                privileged: true # required to allow docker in docker to run and assumes the namespace your applying this to has a pod security policy that allows privilege escalation              volumeMounts:                - name: tmp                  mountPath: /tmp                - name: docker-containers                  mountPath: /var/lib/docker/containers                - name: var-run                  mountPath: /var/run          imagePullSecrets:            - name: ecr-credentials          restartPolicy: OnFailure # this allows the runner to restart locally if it was to crash          volumes:            - name: tmp # required to share a working directory between docker in docker and the runner            - name: docker-containers # required to share the containers directory between docker in docker and the runner            - name: var-run # required to share the docker socket between docker in docker and the runner        # backoffLimit: 6 # this is the default and means it will retry upto 6 times if it crashes before it considers itself a failure with an exponential backoff between        # completions: 1 # this is the default the job should ideally never complete as the runner never shuts down successfully        # parallelism: 1 # this is the default their should only be one instance of this particular runner
```


Answers:
author: Mindaugas Z
body: today i have same problem with self hosted runner ( yesterday everything worked )

also got error that runner is outdated. so upgraded them, and :

<br />

Unable to pull image: Head \\"[https://715509311748.dkr.ecr.us-west-2.amazonaws.com/v2/sox/atlassian/pipelines-auth-proxy/manifests/prod-stable\\](https://715509311748.dkr.ecr.us-west-2.amazonaws.com/v2/sox/atlassian/pipelines-auth-proxy/manifests/prod-stable\)": no basic auth credentials


author: kevin_flammang
body: This morning same problem for us (yesterday everything worked too)

We also upgraded the runner but it didn't help.

comments: author: Mindaugas Z
body: still happening <https://bitbucket.status.atlassian.com/incidents/19b4rq4rwfqw>


Tags: aws

bitbucket-cloud

bitbucket-runners

cloud

kubernetes

server