How do I debug "no space left on device" in Bamboo OnDemand?
Asked by Ben:

I'm getting the same error quite often in various project builds in Bamboo OnDemand - it won't clone my repos. I have a number of projects, each with their own repo, but I can't imagine I'm actually filling up the space on the EC2 instance...

Job error summary:
> The build has failed to check out the source code: : No space left on device

From the "full build log"
>
> |----------------------|----------------------------------------------------------------------------------------------------------------------------------------|
> | 23-Jan-2014 21:00:42 | Build libjava_store - develop - Default Job #14 (STORE-DEV-JOB1-14) started building on agent Elastic Agent on i-c9a0e0e9              |
> | 23-Jan-2014 21:00:42 | Build working directory is /mnt/bamboo-ebs/bamboo-agent/build-dir/STORE-DEV-JOB1                                                       |
> | 23-Jan-2014 21:00:42 | Error occurred while executing the build for libjava_store - develop - Default Job #14 (STORE-DEV-JOB1-14) : : No space left on device |

I'm running on an Amazon Linux EBS x86_64, M1 LARGE, which should have:
> 7.5 GiB memory  
> 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)  
> 850 GB instance storage

I only do shallow clones, and not the entire repo (we use bitbucket) for the repositories. And even if I did get the whole repo and all history, my repos just aren't that big. So I can't figure out what's up.

I'm sure I've configured something wrong, but I have no idea how to figure out what...

Any ideas are welcome!


Answers:
author: Ben
body: We were unaware that the default bamboo-ebs image moved the default build-dir from the /media/ephemeral0/home (which seems to be \~400GB) to use the /mnt/bamboo-ebs (which for us is ony 1GB, since we use it just for settings.xml and to bootstrap the maven repo with all the basics).

Is there a rational as to why this is the default behaviour, when creating an EBS?

We are trying just to remove `bamboo-agent/bamboo-agent.cfg.xml `as per the [documentation](https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+instances+to+use+the+EBS), which will hopefully resolve the problem. It seems to me that the bamboo documentation around all of this is pretty poor. We had a hard time figuring out how to share settings.xml in the first place (which seems like it would be a pretty much required step for all maven users).


author: Malte H?bner
body: I think this is **exactly** what you might be searching for: "Use EC2 ephemeral storage as build directory even with an EBS volume attached": <https://confluence.atlassian.com/display/BAMKB/Use+EC2+ephemeral+storage+as+build+directory+even+with+an+EBS+volume+attached>

comments: author: Shai
body: I am having the same issue.

The script location describe in the article is wrong. i can't find it on my ec2 instance.

Any idea?

10x


author: Przemek Bruski
body: It looiks like you've customised your instance and set your build directory to your EBS volume - so, you're limited by the EBS size. The default build directory is on ephemeral storage ans has hundreds of GBs available.

You have to either increase the EBS size or stay with the default build directory location.

comments: author: Ben
body: We're running OnDemand, and I don't recall changing any such configuration.

In Bamboo Admin - System Information, it shows:

> Current running directory/sw/java/tomcat/apache-tomcat-6.0.32
>
> Configuration path/data/jirastudio/bamboo/home/xml-data/configuration
>
> Build path/data/jirastudio/bamboo/home/xml-data/builds
>
> Build working directory/data/jirastudio/bamboo/home/xml-data/build-dir
>
> Artifacts directory/data/jirastudio/bamboo/home/artifacts
>
> Bamboo home/data/jirastudio/bamboo/home
>
> Bamboo logs/data/jirastudio/bamboo/home/logs/atlassian-bamboo.log
>
> Temporary directory/data/service/j2ee_bamboo/catalina-base/temp
>
> User home /data/jirastudio/bamboo/home
>
which does not seem to indicate the /mnt/bamboo-ebs/

path listed above in the errors.


author: Przemek Bruski
body: Your build takes place on the build agent. All paths you've listed are from the server.


author: Ben
body: Ok, that makes sense - so how do I configure the build directory on elastic bamboo agents for OnDemand? I don't seen anything for that, but I'm sure I'm just not looking in the right place.


author: Ben
body: Digging further into the docs, I finally found [this](https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+instances+to+use+the+EBS):

* `bamboo-agent/bamboo-agent.cfg.xml`- This configuration file modifies the build working directory to point to build working directory on the EBS volume.

I didn't modify this, afaik. So I think you're tell me that the *default*behaviour in Bamboo when using an attached EBS (which we use to share maven settings.xml, etc) is to use the EBS as the working directory. So I gues all I have to do is modify or remove this file and create a a new EBS snapshot, and the problem should go away.


Tags: cloud