AWS CloudFormation UPDATE_ROLLBACK_FAILED fix in production
Tags: aws, python, cloud, cloudformation
Published: 2021-11-01 12:37:34

Recently, while trying to deploy a serverless application, the pipeline failed thus putting the cloud formation stack in "**UPDATE_ROLLBACK_FAILED**" state. This is due to an error in your pipeline. In my own case, I was trying to use a base layer - "**arn:aws:lambda:us-east-1:770693421928:layer:klayers-python38-pandas:35**" which at the point of writing this article does not exist.

If you are familiar with AWS SAM Cli or cloud formation generally, you will know that you wont be able to deploy a new update until this status changes to "**UPDATE_COMPLETE**".

Now, you have two options facing this situation (at least that I am aware of):
1. Attempt to complete the update rollback process.
2. Delete the stack and create another.

The second option is almost perfect and simple except for the fact that you might not want to do this in production has is basically shutting down your application.

The first option however is much more advisable. Now, how do you complete the update rollback.

## Using AWS CLI
You can make use of the [aws cloudformation continue-update-rollback](https://docs.aws.amazon.com/cli/latest/reference/cloudformation/continue-update-rollback.html) command to complete your update rollback.

```bash
aws cloudformation continue-update-rollback \
--stack-name STACK_NAME \
--resources-to-skip LIST_OF_RESOURCES
```
Where **STACK_NAME** is the name of your stack while **LIST_OF_RESOURCES** is the list of the logical IDs of resources you will like to skip. Please note that for **LIST_OF_RESOURCES** you have to specify resources that are in the **UPDATE_FAILED** state only.

Sample
```bash
aws cloudformation continue-update-rollback \
--stack-name eazido-app-stack \
--resources-to-skip CustomerApi PaymentApi 
```

## Using AWS Console
1. Visit your cloud formation area of the AWS console - [https://console.aws.amazon.com/cloudformation/](https://console.aws.amazon.com/cloudformation/)
2. Select the stack that requires rollback.
3. Under the "**Stack Action**" select "**Continue update rollback**"
**Note** If this update rollback still fails or you want to skip some resources, then select "**Advanced troubleshooting**" on the "**Continue update rollback**" dialog and tick the resources you will like to skip.

Once this is done, your stack should now carry the "** UPDATE_ROLLBACK_COMPLETE**" status. If you try to deploy your updates again, it should work just fine.

However, remember how the update failed because of an error, you will need to identify and fix this before you deploy. In my own case, I was using a layer whose version does not exist. I had to update the layer by updating the stack template (we will talk about this in a bit). You can tell why your stack failed by checking the **Status reason** column under the **Events** tab of the failed stack.

## Updating CloudFormation Template

- Select the stack you want to update its template and select "**View in designer**"" under the "**Template**" tab.

![Template tab of an AWS cloud formation stack](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o11onicnmx1n547z0h64.png)

Once the "**View in designer**"'s page loads, you should see a page similar to this.
![AWS Cloud formation designer page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nop13mkoe5g9tav3t0cd.png)


You can get the template for a particular stack using AWS CLI's (get-template)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/get-template.html] command.
```bash
aws cloudformation get-template --stack-name STACK_NAME
```

- You can either edit your template as json or Yaml file. Once you are done making changes, click on the file icon on the top left corner of the page and click "**save**". You have the option of saving on your laptop or in an s3 bucket. Once that is done, you can exit the page.

### Creating and executing a Change Set
1. Select the stack you want to update it's template.
2. Under "**Stack Actions**" select "**Create change set for current stack**". You should see a page like this
![AWS Create change set page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yg434s19zruy3sfznkmx.png)

3. Click on "**Replace current template**", select a template source (local or s3) based on where you saved the edited template.

4. Review the change set and **Execute**. This is will starts updating the AWS cloud formation stack and you can see the progress on the event tab.

You can update a particular stack using AWS CLI's (update-stack)[https://docs.aws.amazon.com/cli/latest/reference/cloudformation/update-stack.html] command.
```bash
aws cloudformation update-stack --stack-name STACK_NAME --template-url https://s3.amazonaws.com/sample/updated_template.template
```





**Note**: If you are getting AWS cli errors, you might want to take a look at the - [AWS cli troubleshooting guide.](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html)