Pipeline broken after update to v25.0.2-multiarch-prod-stable
Asked by Sam Redway:

Docker pipeline now not able to build. It breaks at the step where docker service is defined. ie my pipeline looks like this:

```
steps:
  - step: &run_unit_tests
      runs-on:
        - self.hosted
        - linux
      name: Run Unit Tests
      oidc: true
      size: 8x
      services:
        - docker
      script:
        - ...
```

And I dont reach the script it breaks on build step. If I remove teh lines which do services: docker then it does run through (but obviously I dont have access to docker deamon in pipeline).

This broke for no reason this morning.   

I have this in the docker terminal:  

Runner warnings:  
3
[docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v25.0.2-multiarch-prod-stable:](http://docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v25.0.2-multiarch-prod-stable) Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.  
4  
5
/usr/local/bin/runit.sh: line 7: pipelines: not found  
6
ip: can't find device 'nf_tables'  
7
modprobe: can't change directory to '/lib/modules': No such file or directory  
8
ip: can't find device 'ip_tables'  
9
modprobe: can't change directory to '/lib/modules': No such file or directory  
10
iptables: Failed to initialize nft: Protocol not supported  
11

I checked and there is a new image when I do docker images called:

docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon: v25.0.2-multiarch-prod-stable

The old one had tag: v20.10.24-multiarch-prod-stable

I'm guessing this update (created date is 8 hours ago the same time roughly as my pipeline broke) is what caused the pipeline to break. Although this could be a red herring! Does anyone else have this issue? Is there a fix/ workaround?


Answers:
author: Theodora Boudale
body: Hi Sam and welcome to the community!

This issue seems to be related to the update of the Docker service. Our engineering team is looking into it.

In the meantime, since you are using a self-hosted runner, you can specify the image used by the docker service by adding the following in your yml file:

```
definitions: 
  services: 
    docker:
      image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.24-multiarch-runc-patch-prod-stable
```

If you are still experiencing issues, you can reach out via the support ticket that I see you created.

Kind regards,  
Theodora

comments: author: cmorris98
body: Thank you, this fixed our issue and our build pipelines are back up an running.


author: Theodora Boudale
body: That's good to hear [@cmorris98](/t5/user/viewprofilepage/user-id/5421316), thank you for the update!


author: Sam Redway
body: This helped me get it back up and running and actually found I could now get it work with latest image by creating this in my top level defintions  

```
definitions:
  services:
    docker-self-hosted:
      image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v25.0.2-multiarch-prod-stable
      type: docker
    docker-not-self-hosted:
      memory: 7128
      type: docker
```


author: Piotr Stuglik
body: How exactly is this solved? A workaround is not a solution.


author: Theodora Boudale
body: Hi everyone,

Our engineering team has published a community article with some of the limitations that the upgrade of the Docker Engine introduced:

* <https://community.atlassian.com/t5/Bitbucket-articles/Recent-docker-service-upgrade-in-Bitbucket-Pipelines/ba-p/2603987>

You can read this article about the different limitations and what action needs to be taken in each case.

If you are still experiencing issues after following the suggestions in the article:

* for workspaces on **Standard** or **Premium** plan you can create a support ticket via <https://support.atlassian.com/contact/#/>, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as product. When you are asked to provide the workspace URL, please make sure you enter the URL of the workspace that is on a paid billing plan to proceed with ticket creation.  

* for workspaces on the **Free** plan, please ++create a new question++ in community including relevant errors you see and we will look into it. Please create a new question instead of posting a reply here, as this question will become convoluted and difficult to follow if we try to troubleshoot multiple users' potentially different issues.

Kind regards,  
Theodora


author: DP Developer
body: Hi Team,  

Help me for this issue  

rpc error: code = Unknown desc = failed to pull and unpack image "docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable": failed to resolve reference "docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-auth-proxy:prod-stable": failed to do request: Head "<https://docker-public.packages.atlassian.com/v2/sox/atlassian/bitbucket-pipelines-auth-proxy/manifests/prod-stable>": read tcp 10.5.65.84:54922-\>204.246.191.16:443: read: connection reset by peer


author: Tiago Jesus
body:     Runner warnings:

        http://docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-docker-daemon:v20.10.24-multiarch-runc-patch-prod-stable Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.

Cannot fix it with the older version. How solve this?

comments: author: Piotr Stuglik
body: Is it still broken for you despite designating a specific dind image by any chance?


author: Tiago Jesus
body: Still broken with older version too, on my case.


author: Piotr Stuglik
body: New image produces the following log:  
![log01.PNG](https://community.atlassian.com/t5/image/serverpage/image-id/304988iFE3941F9D4F81AB5/image-size/large?v=v2&px=999 "log01.PNG")

The old image (when set in the pipeline config file) does not produce said error.

But both produces the following:  
![log02.png](https://community.atlassian.com/t5/image/serverpage/image-id/304989i3450E6584EF4A5D9/image-size/large?v=v2&px=999 "log02.png")

Is that your case, too?


author: Piotr Stuglik
body: Two runs with the Docker image tag pinned produces different SHA values in the logs:  
![log03.png](https://community.atlassian.com/t5/image/serverpage/image-id/304995i484E16C9281941BE/image-size/large?v=v2&px=999 "log03.png")

![log04.png](https://community.atlassian.com/t5/image/serverpage/image-id/304996iD37D9B24769CFEB2/image-size/large?v=v2&px=999 "log04.png")


author: Tiago Jesus
body: I have two machines:

- If i run with on Docker version 20.10.25, build b82b9f3 no problem.

- I have problems with Docker version 24+


author: Tiago Jesus
body: Docker version 20.10.25, build b82b9f3 can run self hosted runners.

Docker version 24+ i have problems, cannot run.


author: Piotr Stuglik
body: I've tried pinning the image tag to the previous one, and while the following error disappeared from the Docker logs, the steps leveraging Docker still fails on \`docker: command not found\`:  

```cat: can't open '/proc/net/arp_tables_names': No such file or directory```

I have discovered in the logs that the tag has two different SHA values between the failing and passing builds.


author: slowquery
body: I suddenly have a problem with the pipeline. I downgraded the image and temporarily solved it now, but I would appreciate it if you could reply as soon as the problem is resolved.

comments: author: Theodora Boudale
body: Hi [@slowquery](/t5/user/viewprofilepage/user-id/5423181), it's good to hear that the workaround solved the issue. I will post here when the issues are fixed.


author: cmorris98
body: Yes exact same issue!! Would love to know how to force it to use the old version. If anyone figures this out please please post here. Our entire build process for both test and production pipelines are totally down.


Tags: bitbucket-cloud

cloud