Why Job -> other -> "Clean working directory after each build" cleans ALL my docker mounted paths?
Asked by Mariusz Dullak:

TL;DR; How to I exclude directory (mounted by docker if this is of any relevance and not subdirectory of: ${bamboo.build.working.directory}) from being cleaned when "Clean working directory after each build" is checked.

Hi,

I have a big project (1,2 mln lines of C++ code) that I obviously require pull request in order to add code to develop and main branches.

One of (obvious) requirements is that code compiles. Compilation is happening inside docker container with following mounts (not my idea, the compilation script has some of those hardcoded and I can not do anything about it):

${bamboo.build.working.directory} -\> /opt/src

${bamboo.build.working.directory}/results -\> /opt/results

In order to speed up the compilation we tested usage of ccache and it works great. Compilation time down from around 40 minutes to around 4 minutes.

So the idea was to mount:

/home/ccache -\> /home/ccache

so that each docker will have access to that folder and cache will be shared.

Everything works OK as long as option "Clean working directory after each build" for the build job is unchecked.  
Now we want this option checked and ${bamboo.build.working.directory} cleaned after the build since we are talking 2+GB. Given all the other builds and branches ... there is a limited NVMe storage for the builders ...

My understanding was that "Clean working directory after each build" will clean only ${bamboo.build.working.directory} not ALL directories mounted by docker.

I really need to exclude /home/ccache from the clean.

How do I do that?


Answers:
author: Eduardo Alvarenga
body: Hello [@Mariusz Dullak](/t5/user/viewprofilepage/user-id/4697927)

Thank you for the detailed explanation.

We were able to reproduce this issue and for that have opened a new bug:

* [BAM-22118 - Bamboo Docker Runner Job will not substitute ${bamboo.working.directory} variable if that is customised as a mount volume option](https://jira.atlassian.com/browse/BAM-22118)

We recommend you watch it to be notified of any changes in the future.

There are a number of factors that determine how Atlassian prioritizes bug fixes. You can learn more about this by reading our [Atlassian Data Center and Server Bug Fix Policy](https://confluence.atlassian.com/support/atlassian-data-center-and-server-bug-fix-policy-201294573.html). And you can visit our [dashboard](https://jira.atlassian.com/secure/Dashboard.jspa) if you would like to search for any existing bugs in the future.

In the bug have included a ++workaround++ that you can use. We'll give more details below:

Do not use nested volumes such as:

1. **${bamboo.build.working.directory}** \>\> **${bamboo.build.working.directory}**
2. **${bamboo.build.working.directory}/results** \>\> **/opt/results**

Using nested volumes in Bamboo has the same effect as in Docker, when the nested source location doesn't exist (a fresh build directory), Docker will try to create it and assign it to the volume, but that doesn't work quite well as it will result in a broken git checkout as Git will consider the **${bamboo.build.working.directory}** as not empty, Bamboo will then try to delete its content to make Git happy and that will break any volume assignments.

In your case we recommend you use only:

* **${bamboo.build.working.directory}** \>\> **${bamboo.build.working.directory}**
* **${bamboo.tmp.directory}** \>\> **${bamboo.tmp.directory}**
* **/home/ccache** \>\> **/home/ccache**

Then add a Script task that will create symlinks to **/opt/src** and **/opt/results**:

```
$ ln -sf ${bamboo_build_working_directory} /opt/src
$ ln -sf ${bamboo_build_working_directory}/results /opt/results 
$ cd /opt/src && run-build-script
```

Cheers,

Eduardo Alvarenga  
Atlassian Support APAC

**--please don't forget to Accept the answer if the reply is helpful--**

comments: author: Mariusz Dullak
body: Hi [@Eduardo Alvarenga](/t5/user/viewprofilepage/user-id/4365410)

thank you for your help.

Unfortunately is seems that you focused on the issue that I am working around with simple

```
cd /opt/src
```

(please see my comments in the bug you mentioned) and is not the real issue.

Let me try to restate the issue.

**Steps to reproduce:**

use Docker in bamboo job.

Mount

```
${bamboo.build.working.directory} ->?/opt/src
/home/ccache ->?/home/ccache
```

Check option to "Clean working directory after each build" in the job other section.

In the job put some files to

```
/home/ccache
```

as simple test

```
touch /home/ccache/test
```

**Actual result:**

on bamboo agent host following folders are cleaned:

```
${bamboo.build.working.directory}
/home/ccache
```

**Expected result:**

on bamboo agent host **only** following folder is cleaned:

```
${bamboo.build.working.directory}
```

this is what the options says: working directory. Not all folders mounted in docker.

The file test in

```
/home/ccache/test
```

should remain on the host and next job in the plan (or other branches of the plan or even other plans) can access it mounting same folder in its docker.

I hope that it is much clearer now what the actual issue is. Please do note hesitate to contact me for more details


author: Eduardo Alvarenga
body: Hello [@Mariusz Dullak](/t5/user/viewprofilepage/user-id/4697927)

Thank you for the further explanation.

I cannot reproduce it locally. In my local instance **/home/ccache** content remains intact and any generated files are still there after the Plan is finished with an enabled *Clean working directory after each build* option.

Can you provide us with the exact version of Bamboo you are running along with an [exported Plan in Java Specs format](https://confluence.atlassian.com/bamboo/exporting-existing-plan-configurations-to-bamboo-specs-894743917.html)?


author: Mariusz Dullak
body: Hey [@Eduardo Alvarenga](/t5/user/viewprofilepage/user-id/4365410)

So I did some more testing and apparently that has something to do with me mounting

```
${bamboo.build.working.directory} ->?/opt/src
```

Because when I have this mounted the file disapears.

When I mount:

```
${bamboo.build.working.directory} ->?${bamboo.build.working.directory}
```

the file stays as expected.

So it looks like there is a problem with the docker mount.

Here are details of my "failing" plan (sorry for the formatting but apparently copy-paste does not work well here).

What I do is I run single scrip task in shell with content:

```
#!/bin/bash

touch /home/ccache/test
sleep 10
echo "test" >> /home/ccache/test
sleep 10

exit 0
```

bamboo version is: 9.2.1 build 90201. Here is JavaSpec

```
import com.atlassian.bamboo.specs.api.BambooSpec;
import com.atlassian.bamboo.specs.api.builders.BambooKey;
import com.atlassian.bamboo.specs.api.builders.BambooOid;
import com.atlassian.bamboo.specs.api.builders.Variable;
import com.atlassian.bamboo.specs.api.builders.docker.DockerConfiguration;
import com.atlassian.bamboo.specs.api.builders.permission.PermissionType;
import com.atlassian.bamboo.specs.api.builders.permission.Permissions;
import com.atlassian.bamboo.specs.api.builders.permission.PlanPermissions;
import com.atlassian.bamboo.specs.api.builders.plan.Job;
import com.atlassian.bamboo.specs.api.builders.plan.Plan;
import com.atlassian.bamboo.specs.api.builders.plan.PlanIdentifier;
import com.atlassian.bamboo.specs.api.builders.plan.Stage;
import com.atlassian.bamboo.specs.api.builders.plan.branches.BranchCleanup;
import com.atlassian.bamboo.specs.api.builders.plan.branches.PlanBranchManagement;
import com.atlassian.bamboo.specs.api.builders.plan.configuration.AllOtherPluginsConfiguration;
import com.atlassian.bamboo.specs.api.builders.plan.configuration.ConcurrentBuilds;
import com.atlassian.bamboo.specs.api.builders.project.Project;
import com.atlassian.bamboo.specs.builders.task.ScriptTask;
import com.atlassian.bamboo.specs.builders.trigger.BitbucketServerTrigger;
import com.atlassian.bamboo.specs.util.BambooServer;
import com.atlassian.bamboo.specs.util.MapBuilder;

@BambooSpec
public class PlanSpec {

public Plan plan() {
final Plan plan = new Plan(new Project()
.oid(new BambooOid("14g9zf7pyybcy"))
.key(new BambooKey("ATV"))
.name("Atlassian Toolchain Verification")
.description("Project for testing purposes. Lead Mariusz Dullak."),
"ccache test",
new BambooKey("CT"))
.oid(new BambooOid("14g0a7mcr4xz7"))
.pluginConfigurations(new ConcurrentBuilds())
.stages(new Stage("Default Stage")
.jobs(new Job("Default Job",
new BambooKey("JOB1"))
.pluginConfigurations(new AllOtherPluginsConfiguration()
.configuration(new MapBuilder()
.put("custom", new MapBuilder()
.put("auto", new MapBuilder()
.put("label", "")
.put("regex", "")
.build())
.put("clover.path", "")
.put("buildHangingConfig.enabled", "false")
.put("ncover.path", "")
.build())
.build()))
.tasks(new ScriptTask()
.description("Compile")
.inlineBody("#!/bin/bash\n\ntouch /home/ccache/test\nsleep 10\necho \"test\" >> /home/ccache/test\nsleep 10\n\nexit 0"))
.cleanWorkingDirectory(true)
.dockerConfiguration(new DockerConfiguration()
.image("debian:11-slim")
.withoutDefaultVolumes()
.volume("${bamboo.working.directory}", "/opt/src")
.volume("${bamboo.tmp.directory}", "${bamboo.tmp.directory}")
.volume("/home/ccache", "/home/ccache"))))
.linkedRepositories("Nova core U20.04")

.triggers(new BitbucketServerTrigger())
.variables(new Variable("workers",
"48"))
.planBranchManagement(new PlanBranchManagement()
.delete(new BranchCleanup())
.notificationForCommitters());
return plan;
}

public PlanPermissions planPermission() {
final PlanPermissions planPermission = new PlanPermissions(new PlanIdentifier("ATV", "CT"))
.permissions(new Permissions()
.userPermissions("dullak", PermissionType.EDIT, PermissionType.VIEW, PermissionType.ADMIN, PermissionType.CLONE, PermissionType.BUILD));
return planPermission;
}

public static void main(String... argv) {
//By default credentials are read from the '.credentials' file.
BambooServer bambooServer = new BambooServer("https://bamboo.espera.de");
final PlanSpec planSpec = new PlanSpec();

final Plan plan = planSpec.plan();
bambooServer.publish(plan);

final PlanPermissions planPermission = planSpec.planPermission();
bambooServer.publish(planPermission);
}
}
```


author: Eduardo Alvarenga
body: Hello [@Mariusz Dullak](/t5/user/viewprofilepage/user-id/4697927)

Thanks for the extra details. I appreciate that.

We have dug into it a little bit and come up with more findings! That ended up on another bug that is related to the first one:

* [BAM-22120 - Bamboo will delete the whole Docker Runner container including long-term mounted volumes data(!) if the ${bamboo.build.working.directory} doesn't exist or can't be accessed](https://jira.atlassian.com/browse/BAM-22120)

As usual, check the link, place comments and follow it to be notified of changes moving forward. The workaround is pretty much the same as the first bug, do not use a custom ${bamboo.build.working.directory} directory.

Thanks a lot for reporting that! We really appreciate the collaboration and understanding.

Sincerely,

Eduardo Alvarenga  
Atlassian Support APAC

**--please don't forget to Accept the answer if the reply is helpful--**


author: Mariusz Dullak
body: Hi [@Eduardo Alvarenga](/t5/user/viewprofilepage/user-id/4365410)

thank you for taking your time, confirming the issue, finding that the clean script is executed on / (root) and reporting all of this as a bug.

I believe that my question is now answered. This is a bug. Hopefully an easy one to fix and it will be fixed soon.

In the meantime, let me implement workarounds in all of my plans impacted.


Tags: c

cache

docker