Troubleshooting AWS Cloud9 - RuntimeError: Container does not exist / No space left on device
Tags: aws, cloud, cloud9
Published: 2021-01-19 04:58:01

## **Issue description**

Recently I got a runtime error in AWS Cloud9 while trying to build my serverless application as a Docker container image. I used AWS SAM `build` command:

```YAML
sam build --use-container
```

As a result I got the following error issue:

`RuntimeError: Container does not exist. Cannot get logs for this container`

It turned out, AWS tried to create `amazon/aws-sam-cli-build-image-python3.8` Docker container image:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hvwklilt7jxridqtf09w.png)

but failed:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/fw1s2c6e9akjqmnw8yg1.png)

Then I tried to pull the image manually using docker command:

```YAML
docker pull amazon/aws-sam-cli-build-image-python3.8
```

but it looks like there is not enough disk space:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/5wlf5fdh4hb4fxpg2w5l.png)

You can use the following command to verify that the root volume mounted under "/" is full:

```YAML
df -h
```

Here is an output with 95% out of 10 GiB of EBS storage (/dev/xvda1) is used: 

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/42vusqblka31qcv1femx.png)

To check whether the EBS volume has a partition that must be extended, use the following command:

```YAML
lsblk
```

It displays information about the block devices attached to your instance.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ec8yuzh6zvm0rlz8xnzk.png)

## **Solution**

Once the problem is understood, improvement comes naturally. All I need is to increase the size of my EBS volume. 

:point_right: **Step 1.** Open a new file in Cloud9

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4eu5cry4sxuso7xqaia4.png)

:point_right: **Step 2.** Add the script below to the file:

```YAML
#!/bin/bash

# Specify the desired volume size in GiB as a command-line argument. If not specified, default to 20 GiB.
SIZE=${1:-20}

# Get the ID of the environment host Amazon EC2 instance.
INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)

# Get the ID of the Amazon EBS volume associated with the instance.
VOLUMEID=$(aws ec2 describe-instances \
  --instance-id $INSTANCEID \
  --query "Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId" \
  --output text)

# Resize the EBS volume.
aws ec2 modify-volume --volume-id $VOLUMEID --size $SIZE

# Wait for the resize to finish.
while [ \
  "$(aws ec2 describe-volumes-modifications \
    --volume-id $VOLUMEID \
    --filters Name=modification-state,Values="optimizing","completed" \
    --query "length(VolumesModifications)"\
    --output text)" != "1" ]; do
sleep 1
done

#Check if we're on an NVMe filesystem
if [ $(readlink -f /dev/xvda) = "/dev/xvda" ]
then
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/xvda 1

  # Expand the size of the file system.
  # Check if we are on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/xvda1
  fi

else
  # Rewrite the partition table so that the partition takes up all the space that it can.
  sudo growpart /dev/nvme0n1 1

  # Expand the size of the file system.
  # Check if we're on AL2
  STR=$(cat /etc/os-release)
  SUB="VERSION_ID=\"2\""
  if [[ "$STR" == *"$SUB"* ]]
  then
    sudo xfs_growfs -d /
  else
    sudo resize2fs /dev/nvme0n1p1
  fi
fi 
```

:point_right: **Step 3.** Save the file as `resize.sh` (press **Ctrl + S**)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/guh1kbh10kud84qw2and.png)

:point_right: **Step 4.** Run sh-file (shell script):

```YAML
sh resize.sh 20
```

You should get detailed information about data block changes:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/jysp1r8eie17vk72f1bz.png)

Here is the link to [AWS instuction](https://docs.aws.amazon.com/cloud9/latest/user-guide/move-environment.html#move-environment-resize).


## **Results**

To verify a new EBS volume size, run `df -h` command again:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/s529vv406lg6y97pluap.png)

Once there is enough disk space for Docker, my container issue is resolved:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/446rqx6849015xkjop0x.png)

Hopefully, this solution could save you both some time and headaches.