How to run anything that requires privileged flag in my self-hosted runner? Trying to run litmus tes
Asked by kalib:

I understand Bitbucket not allowing me to use the \`--privileged\` argument in the cloud for security reasons, but not even in my self-hosted runner? How am I supposed to run things that require that? Such as litmus puppet tests, for example?

I am trying to run:

```
docker run -d -it --privileged --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --tmpfs /tmp:exec -p 52243:22 --name docker-my-litmus-image-52243 my-litmus-service.services/litmusimage/debian:10
```

Is the solution really to run docker in docker and spin up a container inside my runner container?


Answers:
author: Theodora Boudale
body: Hi [@kalib](/t5/user/viewprofilepage/user-id/3485966),

It is possible to use the privileged flag in a runner by using an external dind image instead of the default Atlassian docker service.

An example yml file with such a definition is the following:

```
image: centos:7

definitions:
  services:
    docker:
      image: docker:dind

pipelines:
  default:
    - step:
        runs-on:
          - 'self.hosted'
        script: 
          - echo "this is a runner step"
          - docker run -dt --privileged centos:7 ls
        services:
          - docker
```

Is this something that works for you?

Kind regards,  
Theodora

comments: author: kalib
body: Hi [@Theodora Boudale](/t5/user/viewprofilepage/user-id/818086)

Have you tested this? I'm a bit confused, I am kind of new to Bitbucket pipelines.

I see you setting two images there, centos:7 at the top, and after that you set docker:dind.

I'm confused on where those go. For example, here's what I am testing.

Here is what I have at this moment:  

```
---
definitions:
  steps:
    - step: &docker-in-docker
        image: atlassian/default-image:3
        name: Setup docker in docker
        runs-on:
          - myrunner
        script:
          - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
      services:
        - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker
```

When I tried what you suggested, it didn't run saying I had a format error:  

```
---
image: ubuntu:latest
definitions:
  services:
    docker:
    image: docker:dind
  steps:
    - step: &docker-in-docker
         image: ubuntu
         name: Setup docker in docker
         runs-on:
           - my runner
         script:
           - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
        services:
          - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker
```


author: Theodora Boudale
body: Hi [@kalib](/t5/user/viewprofilepage/user-id/3485966),

Yes, I have tested this and it works. A few notes on the last yml file you posted at the end of your previous reply:

**1)** The **runs-on** parameter must contain a **self.hosted** label. You can include additional labels you have added to your runner, but the **self.hosted** label must always be included as well.

**2)** I don't think that runner labels can contain spaces (I see you have added 'my runner'). Please check the labels of your runner from **Repository settings \> Runners**

3) Another issue may be indentation. Please use the indentation as below:

```
image: ubuntu:latest
definitions:
  services:
    docker:
      image: docker:dind
  steps:
    - step: &docker-in-docker
        image: ubuntu
        name: Setup docker in docker
        runs-on:
          - self.hosted
          - myrunner
        script:
          - docker container run -t --privileged -v /opt/atlassian/pipelines/agent/build:/opt/atlassian/pipelines/agent/build --name buildtest ubuntu sh -c 'cd /opt/atlassian/pipelines/agent/build/ && apt update -qq && apt install wget apt-utils build-essential -y -qq && wget https://apt.puppet.com/puppet-tools-release-focal.deb && dpkg -i puppet-tools-release-focal.deb && apt update -qq && apt-cache policy pdk && apt install pdk && echo | pdk --version'
        services:
          - docker

pipelines:
  branches:
    self-hosted-runner:
      - step: *docker-in-docker

```

I have tested the above file in my runner and it works. Indentation is usually two spaces from the previous level, except for the keywords under **- step** (in this case the indentation should be 4 spaces).

<br />

To answer some of the questions also regarding the images:

Pipelines (and runners) builds run in a Docker container.  
For every step of your yml file, a Docker container starts, the repo is cloned, and then the commands of the step's script run in that Docker container.  
Then this Docker container gets destroyed.  
If there is a second step in your yml file, another Docker container starts, the repo is cloned, and the commands of the second step's script run there.

<br />

If you specify at the beginning of your yml file an image, e.g. *image: ubuntu:latest* or *image: centos:7*, every step of your build will use that image as a build container.

If you specify an image on a step level, e.g. the way you are doing here:

```
- step: &docker-in-docker
    image: atlassian/default-image:3
```

then this step will use *atlassian/default-image:3* as a build container, instead of what you defined at the top.

The image definition below:

```
definitions:
  services:
    docker:
      image: docker:dind
```

This specifies what image should be used for the Docker service only.

I hope this answers your questions. If you have any other questions or if you are still running into issues, please let me know and we can look into them.

Kind regards,  
Theodora


author: kalib
body: Oh wow, thank you very much for taking the time for this very detailed answer. It definitely helps a lot.

No, my runner doesn't have spaces in the name, I just used a random name in here to not give my runners name.

But yes, it looks like my error was in the yaml formatting, and after using your code it really worked fine with your  
docker:dind option.  
Thank you again for taking all the time to really explain it.  
There are so many things that are for some reason still not documented. For example, none of this I was able to find in bitbucket's documentation. :/


author: Theodora Boudale
body: Hi [@kalib](/t5/user/viewprofilepage/user-id/3485966),

Thank you for the update, it's good to hear that this worked and you are very welcome.

I will pass the feedback to our team so we can improve our documentation and also include information that is missing.

Please feel free to reach out if you ever need anything else!

Kind regards,  
Theodora


author: kalib
body: Is the solution really to run docker in docker and spin up a container inside my runner container?


Tags: bitbucket-pipelines

bitbucket-server

server