How to detect messed up CloudFormation resources - part 1
Tags: aws, cloudformation, infrastructureascode
Published: 2021-05-03 16:26:35

If you use the principles of infrastructure as code to provision your cloud infrastructure, you get many benefits from it - repeatable, quick, and consistent deployments. 
You also need to apply discipline in how you update the infrastructure - it has to be through the same infrastructure as code workflow.
If anyone modifies the infrastructure through other means, like the AWS Management Console, then all bets are off and you may end up with a mess.

Fortunately, some tools can help to keep your cloud infrastructure tidy.
If you use AWS CloudFormation, then you have a feature called **Drift Detection** which may help you detect when _someone_ changes the infrastructure through other means than CloudFormation.
Read further to see how you can use **Drift Detection** for your benefit.

<!-- more -->

This is the first article in a series. 
We are going to look at _drift detection_ in the **AWS Management Console** in part 1.
In [part 2](https://cloudgnosis.com/articles/how-to-detect-messed-up-cfn-2/) we then continue with some simple scripts to enable some level of automation.
In part 3 we are going to continue further to deploy our drift detection using infrastructure as code and notifications.

I am going to assume that you are familiar with AWS Management Console and you know a bit about **AWS CloudFormation** and how that is used to provision AWS infrastructure resources.
I will also assume that you have familiarity with concepts such as **CloudFormation stacks, resources, and templates**.
If this is new to you, one starting point to get a bit more familiar is to look at AWS own information about the [AWS Cloudformation service](https://aws.amazon.com/cloudformation/).

### Check drift status in AWS Management Console

If you in the AWS Management Console select the service **CloudFormation**, you will get a list of all _CloudFormation stacks_ that are currently provisioned in the current AWS account and region.

In my example here I will use a CloudFormation stack called **demo-stack**, which includes many resources, including some EC2 servers that sit behind an application load balancer.
However, the specific details of what is in the stack are not that important.

In the overview of the stack information, two items are of interest to us right now, **Drift status** and **Last drift check time**.

![Drift status](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nhmjo6gno79v56to7kzb.png)

You will notice that the **drift status** says _NOT\_CHECKED_.
This means that there have not been any drift detection checks on this stack at all, so its state regarding any uncontrolled changes is unknown.
Hence the **last drift check time** has no value - it has never been checked.

_So let us change that_. 
In the **Stack actions** menu we can select the option _Detect drift_. 
This will start a drift detection check on this stack. 
This is a small stack and the check is done right away - larger stacks with many resources may take a little bit of time.

![Select Detect drift](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/t13t522t1u1hl6xzqe5e.png)

You may need to refresh the page to see the update, both **Drift status** and **Last drift check time** will have changed.
The **drift status** is now _IN\_SYNC_ and **last drift check time** is set to the time you ran the drift detection.

![Updated drift status](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q0ggcpgavvupw9eogn3i.png)

So now we know the last time the state of this CloudFormation stack was in good shape and no uncontrolled changes had been made to its resources.

### Let us cause some drift!

Now it is time to put the _drift detection_ to a test!
We browse through the different resources available in the stack and pick one that should be an easy and noticeable change in this case.
The load balancer that the stack sets up in front of the servers uses a security group towards its targets, the servers.

![Our drift victim - security group](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0r89c1w8zswl4hh84r7v.png)

This security group allows traffic to _port 8080_ and this we are going to change.
We can simply go to this security group and change the _ingress rule_ it has for this port to use _port 8081 instead_.
This is of course bad, our solution might not work any longer.
Even a small change like this might not be obvious to detect when things start to break down and people might start to complain the solution has stopped working properly.

![The change](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dup469tztlhmzd13arpy.png)

This might not have been a malicious change - someone just tried to make a quick fix for something else, or just made a typo.
Either way, we have introduced a discrepancy that may or may not be easy to detect, and which could affect the users of the solution in a bad way.

_Next, we will look at detecting a change like this_.

### Detecting our change

To detect this change we start a drift detection again, from the **Stack actions** menu and by selecting _Detect drift_.
This will now change the status of our stack.
The **Drift status** is now _DRIFTED_ and the **Last drift check time** has been updated to this last time we ran the drift detection.

![Updated drift status](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/esx7f1xzf91is6ej9ct2.png)

So now we have detected that we have a drift in the stack.
We cannot see from the status itself though _what has changed_.
For this, we need to choose to view what changes have been detected.
This we can do in the **Stack actions** menu by choosing _View drift results_.

![Select View drift results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bn874litpyc7joqizaeq.png)

We will see the list of resources in the stack, and in this list, we can see that we have a **modified security group** - the status shows _MODIFIED_.

![Securitty group drift](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n9zogeg1668lbbxk63p4.png)

We can expand this modification entry to see more details what has been changed - what are the current values and what are the expected values, i.e. what is in our stack definition.

Here the user interface shows clearly what the change is and what is the expected value and what the actual value is.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ma29vxabj5gswl6ao3mw.png)

### What to do about it?

So the next question here is, what to do next?
Do we change the modified value back to the expected value, or do we change the CloudFormation stack to match the actual value?

In most cases, the right answer would be to change the value back to its original state. 
One of the points with using infrastructure-as-code is that you have a version-controlled and readable description of the desired state of the infrastructure.
Any changes should be done through a workflow involving this version control state description and then provisioned.
So detecting an uncontrolled change like this should be fixed **by reverting the change to its original expected value**, then _if_ the value was intended to change, do that by **updating the version-controlled description of the infrastructure and provision these changes**.

Now, in the case where it is just a simple change like this and we are ok with making a manual change to revert it, then that is easy.
But if there are changes that we cannot revert that easily (e.g. someone deleted a resource) or we want some automated remediation, then this becomes more complex, unfortunately.

There is no unified and simple _fix my stack state_ operation, besides deleting the stack and recreating it, which in some cases is **not** what you want to do, especially if the stack contains stateful resources (e.g. databases, persistent disks).

There are multi-step approaches that can be applied with [_CloudFormation import functionality_](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-resolve-drift.html), which is out of scope for this article.

Other infrastructure-as-code solutions, like **Terraform**, [have different and sometimes easier approaches to drift remediation](https://www.hashicorp.com/blog/detecting-and-managing-drift-with-terraform).

In the end though, regardless of the toolset you use, **you do not want drift to happen**.
Some AWS services have remediation features built-in that will repair some state issues automatically, and using [**immutable infrastructure**](https://medium.com/the-cloud-architect/immutable-infrastructure-21f6613e7a23) will also make it less scary to handle such issues.

Even though remediation may be hard sometimes, **the alternative of not detecting uncontrolled changes will be worse**.
In addition to detecting drift, you should have _policies and procedures in place to avoid that such changes will be made in the first place_.
This may be a cultural and organizational shift, but an important one.
Technology and tools alone will not stop things from getting messed up.

### Final thoughts

This was in introduction to detecting (uncontrolled) changes to your CloudFormation stacks, which may mess up your infrastructure-as-code deployments.
It is a very useful feature, but in practice, it is not that practical to do this via AWS Management Console manually.
We want to script and automate these features of course.

In [part 2](https://cloudgnosis.com/articles/how-to-detect-messed-up-cfn-2/) I will show an approach to write scripts to trigger drift detection and retrieve the results, which may be a bit better than a pure manual check.
This will also allow us to check on multiple stacks in one operation, as well as multiple regions.
That is relatively simple to do but still requires an explicit action to perform this operation.

In part 3 we will therefore continue to build something that performs a more continuous check and reporting of the drift and which will be deployed as a CloudFormation stack in itself.
This can be done in multiple ways - via **AWS Lambda**, which can be in combination with **AWS Config**.

I thank you for your time reading this article to the end, and I hope you got some value out of it!