metadata

title: My Gradio App Mnist Classifier
emoji: 🚀
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.7.1
app_file: app.py
pinned: false

aws_ec2_automation

Here’s a detailed explanation of the GitHub Actions (GHA) pipeline in raw Markdown format:

GitHub Actions Pipeline Documentation

Name: Deploy PyTorch Training with EC2 Runner and Docker Compose

This pipeline automates the following tasks:

Starts an EC2 instance as a self-hosted GitHub runner.
Deploys a PyTorch training pipeline using Docker Compose.
Builds, tags, and pushes Docker images to Amazon ECR.
Stops the EC2 instance after the job is completed.

Workflow Triggers

on:
  push:
    branches:
      - main

Trigger: This workflow runs whenever a push is made to the main branch.

Jobs Overview

1. start-runner

Starts a self-hosted EC2 runner using the GitHub Actions Runner.

Steps:

Configure AWS Credentials:

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ secrets.AWS_REGION }}

Authenticates with AWS using access keys and the region specified in the secrets.
Required for creating and managing the EC2 instance.

Start EC2 Runner:

- name: Start EC2 runner
  id: start-ec2-runner
  uses: machulav/ec2-github-runner@v2
  with:
    mode: start
    github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
    ec2-image-id: ami-044b0717aadbc9dfa
    ec2-instance-type: t2.xlarge
    subnet-id: subnet-024811dee81325f1c
    security-group-id: sg-0646c2a337a355a31

Starts an EC2 instance with the specified AMI, instance type, subnet, and security group.
Outputs:
- label: A unique label for the EC2 runner.
- ec2-instance-id: The ID of the created EC2 instance.

2. deploy

Deploys the PyTorch training pipeline using the EC2 runner started in the previous step.

Dependencies:

needs: start-runner
runs-on: ${{ needs.start-runner.outputs.label }}

Depends on the start-runner job and runs on the newly created EC2 instance.

Steps:

Checkout Repository:

- name: Checkout repository
  uses: actions/checkout@v4

Clones the current repository to the runner.

Set Up Docker Buildx:
```
- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3
```
- Configures Docker Buildx for building multi-platform Docker images.

Configure AWS Credentials:

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ secrets.AWS_REGION }}

Reconfigures AWS credentials for Docker ECR authentication and resource management.

Log in to Amazon ECR:

- name: Log in to Amazon ECR
  id: login-ecr
  uses: aws-actions/amazon-ecr-login@v2

Logs into Amazon ECR for pushing and pulling Docker images.

Create .env File:

- name: Create .env file
  run: |
    echo "AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}" >> .env
    echo "AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}" >> .env
    echo "AWS_REGION=${{ secrets.AWS_REGION }}" >> .env

Generates a .env file for the application with AWS credentials and region.

Run Docker Compose for Train and Eval Services:
```
- name: Run Docker Compose for train and eval service
  run: |
    docker-compose build
    docker-compose up --build
    docker-compose logs --follow
    docker-compose down --remove-orphans
```
- Build: Builds all services defined in the docker-compose.yml file.
- Up: Runs all services, including training and evaluation.
- Logs: Outputs logs for debugging purposes.
- Down: Stops all services and removes orphaned containers.

Build, Tag, and Push Docker Image to Amazon ECR:

- name: Build, tag, and push Docker image to Amazon ECR
  env:
    REGISTRY: ${{ steps.login-ecr.outputs.registry }}
    REPOSITORY: soutrik71/mnist
    IMAGE_TAG: ${{ github.sha }}
  run: |
    docker build -t $REGISTRY/$REPOSITORY:$IMAGE_TAG .
    docker push $REGISTRY/$REPOSITORY:$IMAGE_TAG
    docker tag $REGISTRY/$REPOSITORY:$IMAGE_TAG $REGISTRY/$REPOSITORY:latest
    docker push $REGISTRY/$REPOSITORY:latest

Build: Creates a Docker image with the repository and tag.
Push: Pushes the image to Amazon ECR.
Tag: Updates the latest tag.

Pull and Verify Docker Image from ECR:

- name: Pull Docker image from ECR and verify
  env:
    REGISTRY: ${{ steps.login-ecr.outputs.registry }}
    REPOSITORY: soutrik71/mnist
    IMAGE_TAG: ${{ github.sha }}
  run: |
    docker pull $REGISTRY/$REPOSITORY:$IMAGE_TAG
    docker images | grep "$REGISTRY/$REPOSITORY"

Pull: Pulls the built image from ECR.
Verify: Ensures the image exists locally.

Clean Up Environment:

- name: Clean up environment
  run: |
    rm -f .env
    docker system prune -af

Deletes the .env file and removes unused Docker resources.

3. stop-runner

Stops and terminates the EC2 runner created in the start-runner job.

Dependencies:

needs:
  - start-runner
  - deploy

Steps:

Configure AWS Credentials:

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: ${{ secrets.AWS_REGION }}

Stop EC2 Runner:

- name: Stop EC2 runner
  uses: machulav/ec2-github-runner@v2
  with:
    mode: stop
    github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
    label: ${{ needs.start-runner.outputs.label }}
    ec2-instance-id: ${{ needs.start-runner.outputs.ec2-instance-id }}

Stops the EC2 runner instance created in the first job.

Validate EC2 Termination:

- name: Validate EC2 termination
  run: aws ec2 describe-instances --instance-ids ${{ needs.start-runner.outputs.ec2-instance-id }}

Ensures the EC2 instance has been properly terminated.

Key Highlights

Sequential Execution:
- The start-runner, deploy, and stop-runner jobs are executed sequentially.
Error Handling:
- The stop-runner job runs even if previous jobs fail (if: ${{ always() }}).
Efficiency:
- Docker layer caching speeds up builds.
- Cleanup steps maintain a clean environment.
Security:
- Secrets are masked and removed after use.
- Proper resource cleanup ensures cost efficiency.

This pipeline ensures robust deployment with error handling, logging, and cleanup mechanisms. So far we have discussed the GitHub Actions pipeline , the basic structure of the pipeline, and the steps involved in the pipeline. Next we will have an interdependent pipeline where the output of one job will be used as input for the next job.

Advanced Pipeline with

Sequential Flow: Each job has clear dependencies, ensuring no step runs out of order.
Code Checkout: Explicit repository checkout in each job ensures consistent source code.
Secure Credential Handling: Sensitive credentials are masked and stored securely.
Resource Cleanup: Includes Docker clean-up and EC2 instance termination validation.
Logging: Added detailed logs to improve debugging and monitoring.

Step 1: Start EC2 Runner Purpose: Initializes a self-hosted EC2 runner for running subsequent jobs. Key Actions: Configures AWS credentials. Launches an EC2 instance using specified AMI, instance type, and networking configurations. Outputs the runner label and instance ID for downstream jobs. Step 2: Test PyTorch Code Using Docker Compose Purpose: Tests the PyTorch training and evaluation services. Key Actions: Checks out the repository. Sets up Docker Buildx for advanced build capabilities. Configures AWS credentials and creates a masked .env file for secure credential sharing. Runs all services (train, eval) using Docker Compose, monitors logs, and cleans up containers. Step 3: Build, Tag, and Push Docker Image Purpose: Builds a Docker image, tags it, and pushes it to Amazon ECR after successful tests. Key Actions: Checks out the repository again to ensure consistency. Logs into Amazon ECR using AWS credentials. Builds and tags the Docker image with latest and SHA-based tags. Pushes the image to Amazon ECR and verifies by pulling it back. Step 4: Stop and Delete EC2 Runner Purpose: Stops and terminates the EC2 instance to ensure cost efficiency and cleanup. Key Actions: Configures AWS credentials. Stops the EC2 instance using the label and instance ID from start-runner. Validates the termination state of the EC2 instance to ensure proper cleanup.