Fixing AWS CodeDeploy issue where Auto Scaling Group is not attached to Target Group
Tags: aws, codedeploy, issue, fix
Published: 2019-05-06 08:32:23

---
title: Fixing AWS CodeDeploy issue where Auto Scaling Group is not attached to Target Group
published: true
description: AWS CodeDeploy Blue/green deployments can automatically copy the previous Auto Scaling Group's configuration. But there's a very important piece of information that is not copied and it can cause unexpected outages.
tags: aws, codedeploy, issue, fix
---

AWS CodeDeploy Blue/green deployments can automatically copy the previous _Auto Scaling Group_ configuration. One thing that we discovered the hard way was: after a deployment the newly created _Auto Scaling Group_, that had a _Target Group_ attached, no longer has it.

![Auto Scaling Group missing Target Group](https://thepracticaldev.s3.amazonaws.com/i/nvparwq0vqly9nbil3ct.png)

It's is a [known issue](https://forums.aws.amazon.com/thread.jspa?messageID=812431&tstart=0#812431) that hasn't been fixed yet. You could have noticed it incidentally or you could have noticed, as in our case, due a major outage.

## How could that lead to an outage?

_Target Groups_ are responsible for checking the health of instances and remove them from the _Load Balancer_, when they are unhealthy. An _Auto Scaling Group_ configured to maintain a fixed number of instances takes care of launching new instances to replace unhealthy ones and terminate the unhealthy instances after the replacements report as healthy.

Since the _Auto Scaling Group_ created by CodeDeploy is not attached to the existing _Target Group_ used by the original _Auto Scaling Group_ from which CodeDeploy copied information, the unhealthy instances will be removed from the _Load Balancer_ by the _Target Group_ but no new instances will be launched by the new _Auto Scaling Group_.

If your unhealthy instances can't recover by themselves that means your application will be completely down. In our situation it turned out that the we were using a version of the Erlang VM that would suddenly crash and make our instances unhealthy.

## How could we fix it?

We imagined that we could attach the _Target Group_ into to newly created _Auto Scaling Group_ as part of the deployment process. As seen in the [documentation](https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#reference-appspec-file-structure-hooks-run-order) the last hook for a Blue/green deployment is the `AfterAllowTraffic` hook, so we decided to attach them in it.

We've created a simple bash script that uses the AWS Command Line Interface (CLI) to find the necessary information and then attach the _Target Group_ to the _Auto Scaling Group_, again using CLI. You can see the Environment Variables available for hooks to fetch information in the [official documentation](https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-hooks.html#reference-appspec-file-structure-environment-variable-availability).

```sh
targetGroupName=$(aws deploy get-deployment --deployment-id $DEPLOYMENT_ID --query "deploymentInfo.loadBalancerInfo.targetGroupInfoList[*].name" --output text)
targetGroupArn=$(aws elbv2 describe-target-groups --names $targetGroupName --query "TargetGroups[*].TargetGroupArn" --output text)
autoscalingGroupName=$(aws deploy get-deployment --deployment-id $DEPLOYMENT_ID --query "deploymentInfo.targetInstances.autoScalingGroups" --output text)
aws autoscaling attach-load-balancer-target-groups --auto-scaling-group-name "$autoscalingGroupName" --target-group-arns "$targetGroupArn"
```

To attach the _Target Group_ to the _Auto Scaling Group_ we need the _Auto Scaling Group's_ name and the _Target Group's_ Amazon Resource Name (ARN). We find the _Target Group_'s name using the `$DEPLOYMENT_ID` of the current deploy and then use that to find the _Target Group_'s ARN. We find the _Auto Scaling Group_'s name with the same call we used to find the _Target Group_'s name but with a different query. Finally we attach them using the `attach-load-balancer-target-groups` command.

It's worth mentioning that to be able to run those commands your instances would need permission to the following actions:

* `codedeploy:GetDeployment`
* `elasticloadbalancing:DescribeTargetGroups`
* `autoscaling:AttachLoadBalancerTargetGroups`

That fixes the problem of having the _Auto Scaling Group_ not being attached to the _Target Group_ after a deploy. But what happens when the _Auto Scaling Group_ decides to launch a new instance?

## Auto Scaling Group decides to launch a new instance

When the _Auto Scaling Group_ decides to launch a new instance, a new deploy will be triggered for the that instance after it is started. But that deploy has a different **Initiating Event**. A normal Blue/green deployment would have a **Initiating Event** as `user` while a deploy created by an _Auto Scaling Group_ action would have a **Initiating Event** as `autoScaling`.

Important to us, though, is that the information used by our naive script in the last section would be different. More specifically, since the deployment is not copying the _Auto Scaling Group_ configuration this time, there's no information about the _Auto Scaling Group_. So our command to find the `autoscalingGroupName` would return `"None"`.  That leads to a infinite loop of instances failing the deployment and the _Auto Scaling Group_ launching new instances and trying to deploy to them again.

Since the _Target Group_ would already been attached to the _Auto Scaling Group_ when the script ran in the `AfterAllowTraffic` during the Blue/green deployment, we can change our script to only attach them if the `autoscalingGroupName` is different from `"None"`.

```sh
targetGroupName=$(aws deploy get-deployment --deployment-id $DEPLOYMENT_ID --query "deploymentInfo.loadBalancerInfo.targetGroupInfoList[*].name" --output text)
targetGroupArn=$(aws elbv2 describe-target-groups --names $targetGroupName --query "TargetGroups[*].TargetGroupArn" --output text)
autoscalingGroupName=$(aws deploy get-deployment --deployment-id $DEPLOYMENT_ID --query "deploymentInfo.targetInstances.autoScalingGroups" --output text)

if [ "$autoscalingGroupName" != "None" ]
then
  aws autoscaling attach-load-balancer-target-groups --auto-scaling-group-name "$autoscalingGroupName" --target-group-arns "$targetGroupArn"
fi
```

## Conclusion

Running this script as part of our deployment process helped us to mitigate an outage. Bear in mind that we see the use of the aforementioned bash script as a temporary fix while we wait for the issue to be fixed in the AWS CodeDeploy side.

After further testing and simulation of the case that caused the outage, we're now confident that the replacement of unhealthy instances are working as expected. That brought us immense peace of mind.