The hidden Git backdoor in AWS CodePipeline and AWS CodeBuild
Tags: aws, git, devops
Published: 2020-11-28 08:43:33

Personally, I think cutting corners is inevitably linked to how we function as human beings. It's all about following the [desired path](https://en.wikipedia.org/wiki/Desire_path). **In my head, I often go over a problem and its solutions and in the meantime, I consider the tradeoffs of all options. Usually, this process is about finding the right balance between value and effort.** For example, for a one-time job it's often not worth to put tremendous effort into automation. 

However, from time to time, you stumble upon something which you thought you'd well considered. A few weeks later, giving your solution a second thought, you could feel like you're being hit by lightning.

## Lets come to the point
One of the first things you learn on AWS is [to follow the standard security advice of granting least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege). It's a simple rule, but that doesn't mean it's easy to implement.  When we started creating AWS CodePipelines a while ago we came up with the following CloudFormation code describing our infrastructure:

```
  DeliveryPipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtefactStores:
        - Region: eu-west-1
          ArtifactStore:
            Location: !Ref ArtefactBucketName
            Type: S3
      Name: some-project-pipeline
      RoleArn: !GetAtt CodePipelineServicenRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: GitSource
              ActionTypeId:
                Category: Source
                Owner: ThirdParty
                Provider: GitHub
                Version: "1"
              Configuration:
                Owner: !Ref GitHubOwner
                Repo: !Ref GitRepo
                Branch: !Ref GitBranch
                PollForSourceChanges: False
                OAuthToken: !Ref GitHubOAuthToken
              OutputArtifacts:
                - Name: SourceZip
        ...
        - Name: Deploy
          Actions:
            - Name: DeployPersistantStack
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: "1"
              Configuration:
                ActionMode: CREATE_UPDATE
                Capabilities: CAPABILITY_NAMED_IAM
                RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                StackName: some-stack-cfn
                TemplatePath: BuildArtifactAsZip::cfn-template.yaml
                TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json
              InputArtifacts:
                - Name: BuildArtifactAsZip
              RunOrder: 1
   		...
  	
  CloudFormationExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          Action:
            - sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
              - cloudformation.amazonaws.com
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess
```

I think the corner that has been cut in the code above is easy to find. A `CloudFormationExecutionRole` having `AdministratorAccess` isn't compliant the least privilege rule. The reason to take a shortcut -in this case- was the fact that CloudFormation stacks often do a lot of different things. Applying the least privilege rule in this context would mean a long and well-thought-out policy. So for simplicity and speed, we decided to cut a corner. The trade-off? Allowing the pipeline a tad more access then it should, no big deal, I could live with that.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/csz3gljgs1dwhbe6a860.jpg)

## Defence lines

When it comes to security, I think we made a pretty good fortress of our AWS environment: a good IDP, AWS WAF, AWS Security Hub, AWS Inspector and an endless list of other tools to close the gates. We made AWS a safe place to go to. A Git repository, however, often lives outside the AWS ecosystem. GitHub, GitLab and Bitbucket are the most common places to put your code. Next question: is your Git repository set up like a fortress as well? If the answer is no, then allow me to create some horror. 
**If someone could gain access to your Git repository, he could do one of the following if a pipeline has been granted `AdministratorAccess` for CloudFormation:**
- Change a DNS Name to re-route traffic (to a phishing site)
- Change your [DeletionPolicy](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html) and throw away all your resources 
- Change an instance profile or a security group
- *I think this already showed you enough horror*

If I now repeat my question: can I live with that? The answer is the complete opposite. It's a big deal, no way I can accept this!

So this is also a note to myself. Always think twice when cutting corners, especially if security might be at risk.  Sometimes it's not easy to understand the blast radius of decisions at first. To make things even worse: in many cases, it will be very hard or even impossible for AWS Security tools to define the correct severity level. In this particular case, AWS will give you a warning but it cannot describe all the underlying risks.

I'm feeling lucky that I got this epiphany before this could turn into a real issue.

### Extra warning: Self-maintaining Pipelines
Although I have never been a fan of pipelines that maintain themselves, in this case, I need to give an extra warning. For clarity: by self-maintaining pipelines I mean pipelines that have a stage to update themselves. Having such a pipeline listening to Git means that the Pipeline policy itself can be changed by pushing a commit to Git. So if your self-maintaining pipeline doesn't have `AdministratorAccess`, this is easily changed by having access to its Git repository.
For that reason, I would remove self-updating logic from a pipeline. We use makefiles that need a client to authenticate in order to make Pipelines changes.

## The solution
In the end, the best way to close this security hole is to apply the least privilege rule to your Pipeline. The downside of this approach is the introduction of extra work. Whenever someone will make changes to the CloudFormation stack that incur security context changes he will need to update the Pipeline's policy first. Be prepared for people who get really upset by this.

If you are in a situation where your current pipeline policy is too relaxed, you could already consider starting with an intermediate solution. You could begin by denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is already more secure, this approach has some downsides. So it's just to buy you a bit more time before applying the least privilege rule.

Sharing is caring ;-) 

Until next time.

*Note: the same security hole can be created on AWS CodeBuild.*