Integrate GitHub monorepo with AWS CodePipeline to run project-specific CI/CD pipelines.
Published: 2023-06-25 13:51:11

## Introduction

CI/CD (Continuous Integration and Continuous Deployment/Delivery) is an approach to software development that emphasizes automation and frequent integration of code changes. It involves regularly merging code from multiple developers into a shared repository and automatically building and testing the code to ensure its quality. With CI/CD, the focus is on automating the entire process of delivering software, from development to deployment, enabling faster and more reliable releases. By adopting CI/CD practices, organizations can achieve improved code quality, accelerated delivery cycles, and streamlined software deployment. There are different [CICD Tools](https://katalon.com/resources-center/blog/ci-cd-tools) that organization use, however my focus is on AWS CodePipeline. 
[AWS CodePipeline](https://aws.amazon.com/codepipeline/) is a fully managed continuous delivery service in aws that helps you automate your release pipelines for fast and reliable application and infrastructure updates.

## What we want to achieve

Creating a CICD Pipeline is easy especially when you are working on a monolithic applications or Microservice based applications with seperate Github repositories.
This becomes a problem when working with Monorepo (Microservice based applications in which all the services are in one Github repository)because there are as many pipelines as the number of Microservices so any change in the repository will trigger all the pipelines.This is not what we want to achieve.

We want a service pipeline to be triggered only when there a change in the corresponding directory in the GitHub repository. We can achieve this with Jenkins pipeline (using `changeset`), Github Action using `Path Filter`, etc but We want to achieve this AWS CodePipeline.

It is difficult at the moment to achieve this directly with AWS CodePipeline unless you use an external environment like Github Action, Jenkins to trigger the AWS CodePipeline but we want everything to be done in AWS.

One way to achieve this is using **AWS API-Gateway** and **Lambda Function** to trigger the pipeline.
[click here to read up more](https://aws.amazon.com/blogs/devops/integrate-github-monorepo-with-aws-codepipeline-to-run-project-specific-ci-cd-pipelines/)

## Method

To Achieve this we will follow 6 major steps

- [Set up AWS Pipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/getting-started-codepipeline.html). I will assume you already have AWS CodePipeline. I will not write about this.
- [Set up a Lambda Function](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html)
- Add StartExecution Permission in the Lambda Roles Created
- [Setup api Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html)
- [Set up Webhook in Github repository](https://hookdeck.com/webhooks/platforms/getting-started-github-webhooks#what-are-github-webhooks)
- Configure the Lambda function with logic to trigger the AWS CodePipeline



![Architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jq41g56tqks7w7nctxho.png)

## Setting up AWS Lambda
Log on to your aws console and search for Lambda.

![Click on `Create Function`](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/klgx5qe7ntuxhguz1k0u.png)

- Put the function name as you like and select python 3.9 as the runtime.

![Runtime setting](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ts2gkrdguckvbfuv8u4r.png)

- Click on advanced setting, Enable `function URL` and select `None` as the Auth type.
- Click on Create

![advanced setting](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/88ylqlf35huh4xtvkmjf.png)

- Hit the function URL on the Lambda function page and you will get this response (`"Hello from Lambda!"`) on the browser

![Lambda function page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8scbhcqu45wg6no5bkr7.png)

![Response from the Function URL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zrxpmnvh5ifzioie0ksj.png)

## Add StartExecution Permission
The lambda Function will use StartExecution API to trigger the AWS Codepipeline so we need to add permission to the role we created while setting up the Lambda Function.
Go to IAM and Select Role
search for the role you created

![Modify the role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d9rg7a5o8irpco3oi6fb.png)

Click on the role

![Create Inline Policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6x312i7an3wtpftekrm4.png)
Click on add permission
select inline policy
click on json and add below policy then save.

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowStartPipelineExecution",
            "Effect": "Allow",
            "Action": "codepipeline:StartPipelineExecution",
            "Resource": "*"
        }
    ]
}
```

## Setting up AWS api gateway
On the aws console, search for api gateway. We are using `HTTP api`

![Select the `HTTP api`](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g2lg07cue65ocz6tdzt1.png)

- click on the `Build`

![configure the api gateway](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dbilshx04geq9o7bbjhv.png)

- click on integrations and select the lambda.
- select the Lambda function you created before in the correct region.
- Put the name of the api.
- click Next.

![Configure Routes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/avlwpsmaj2tknv0rh8dz.png)

- Select POST as the Method.
- Put context path you want resource path. (This will be used when making a request from GitHub).
- Select the target (The Lambda Function You created).
- Click on Next.
- Leave other things as default and create.

![api gateway details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/catg7vcoo8nt7gtbbqpy.png)
Copy the invoke URL.
The webhook URL will be `invoke URL/route`. Something like this (	https://lbxy6ibso5.execute-api.eu-west-2.amazonaws.com/github-webhook)

![Routes](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/idgor2houdy8butetk8z.png)

## Setting up the webhook URL in the Github Repository

Go to the Github and select the GitHub repository that has the mono-repo.

![Configure the webhook](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bz5p0r2wlpwj1gzsb0zt.png)
Click on setting and select webhook.
Click on add webhook.

![Webhook Configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zwe574rlrmv5y3bbs5kw.png)

Copy the wehbook URL you created in api gateway and set in the payload URL.
Change the content type to `application/json`
Click on `add webhook`

![successful ping to api gateway](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u1wmmvnln0raa1033a21.png)


You can check the logs to be sure the GitHub gets to the lambda function.

![check the logs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/en2ug03osp1g8dfswt9p.png)

![Request from the Github to AWS Lambda](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5qmxu3ce14r0yvrp6nmv.png)

## Configure the Lambda function with logic to trigger the AWS CodePipeline.
Let assume that we are using 2 AWS code-pipelines named Codepipeline-A and Codepipeline-B.
Update below code in the Lambda function and click on deploy,

```
import boto3
import json

def lambda_handler(event, context):
    print(json.dumps(event))
    payload = json.loads(event['body'])

    # Extract necessary information from the GitHub webhook payload
    repository = payload.get('repository', {}).get('full_name')
    branch = payload.get('ref')
    commits = payload.get('commits')

    # Define the target folders for each microservice
    capture_widget_folder = 'apps/capture-widget'
    login_service_folder = 'apps/login-service'
    

    # Initialize the AWS CodePipeline client
    codepipeline_client = boto3.client('codepipeline')
    
    print(
      'repository is:', repository,
      'branch is:', branch,
      'commits is:', commits
    )

    # logic for codepipeline triggering  
    try:
        # Check if the changes include the capture-widget folder
        if repository and commits and any(file.startswith(capture_widget_folder) for commit in commits for file in commit.get('added', []) + commit.get('modified', [])):
            # Trigger the capture-widget CodePipeline
            response = codepipeline_client.start_pipeline_execution(
                name='Codepipeline-A'
            )
            print('Triggered CodePipeline-A:', response)

        # Check if the changes include the login-service folder
        if repository and commits and any(file.startswith(login_service_folder) for commit in commits for file in commit.get('added', []) + commit.get('modified', [])):
            # Trigger the login-service CodePipeline
            response = codepipeline_client.start_pipeline_execution(
                name='Codepipeline-B'
            )
            print('Triggered Codepipeline-B:', response)
        else:
             print('wahala dey oo ...')

        return {
            'statusCode': 200,
            'body': json.dumps('CodePipeline triggered successfully')
        }
    except Exception as e:
        print('Error:', e)
        raise e
```

![Update Lambda Function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wnafdx3is7qb2a49zhej.png)

You can now make changes in the GitHub and it will trigger the pipeline depending on the folder you made changes to and the the one you configured in the lambda function.

Remember to disable `Change Detection Option` in AWS CodePipeline.


![Disable Change Detection Option](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7f2nqs5auwkqere7mkx8.png)

## Conclusion

This can be achieved in several ways with other CICD tools and you can check the logs in the CloudWatch incase you are having issues or error.

I also know that most people manage their infrastructure with Terraform. This can be also be implemented with Terraform.