AWS GitHub & S3 Backup
Tags: aws
Published: 2024-03-10 17:05:51

Introduction:
Data backup and disaster recovery services are critical aspects of protecting a business’s most asset — its data.
Losing this data can result in severe consequences, including financial loss, reputational damage, and operational disruptions. Therefore, it’s essential to understand the importance of data backup and disaster recovery planning and implement effective strategies to safeguard your business’s assets.

AWS S3 BACKUPS STORAGE:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qthmgn48zro8t4xn2wbj.png)

Always consider both folders:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ska0joixaf3bq12h4yo0.png)


Backup procedures:
Github:




![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0mz0xuddpt51cgmy5w1m.png)



Docker file:

FROM debian:12-slim

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y \
    awscli

COPY script.sh /usr/local/bin/script.sh
RUN chmod +x /usr/local/bin/script.sh

CMD ["/usr/local/bin/script.sh"]

Script:

#!/bin/bash

# Directory paths
SOURCE_DIR="/app/storage"
DEST_DIR="/app/backup"
TIMESTAMP=$(date "+%Y-%m-%d")

# Ensure destination directory exists
mkdir -p "$DEST_DIR"

# Count total directories, excluding lost+found
TOTAL_DIRS=$(find "$SOURCE_DIR" -mindepth 1 -maxdepth 1 -type d ! -name 'lost+found' | wc -l)

# Counter for processed directories
PROCESSED_DIRS=0

RETENTION_PERIOD=13

# List directories in the source directory
for dir in "$SOURCE_DIR"/*; do
    if [ -d "$dir" ]; then
        # Get directory name
        dir_name=$(basename "$dir")

        # Skip the lost+found directory
        if [ "$dir_name" = "lost+found" ]; then
            continue
        fi

        echo "Processing directory: $dir"
        
        # Copy directory to destination
        cp -r "$dir" "$DEST_DIR/$dir_name"
        
        # Compress the copied directory
        tar -czf "$DEST_DIR/$dir_name.tar.gz" -C "$DEST_DIR" "$dir_name"
        
        # Upload to AWS S3
        aws s3 cp /app/backup/${dir_name}.tar.gz s3://${BUCKET}/files/${TIMESTAMP}/${NAMESPACE}/${dir_name}/${dir_name}.tar.gz
        
        # Clean up
        rm -rf "$DEST_DIR/$dir_name" "$DEST_DIR/$dir_name.tar.gz"

        # Increment processed directories counter
        PROCESSED_DIRS=$((PROCESSED_DIRS+1))

        # Calculate and display progress
        PROGRESS=$(( (PROCESSED_DIRS * 100) / TOTAL_DIRS))
        echo "Progress: $PROGRESS% ($PROCESSED_DIRS/$TOTAL_DIRS directories processed)"

        # Deleting folders older than retention period
        RETENTION_DATE=$(date -d "${TIMESTAMP} -${RETENTION_PERIOD} days" "+%Y-%m-%d")
        
        # List all date folders
        FOLDER_LIST=$(aws s3 ls s3://${BUCKET}/files/ | awk '$0 ~ /PRE/ {print $2}' | grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}/' | sed 's/\/$//')
        
        # Loop through each folder and delete if older than retention period
        for folder in $FOLDER_LIST; do
            FOLDER_TIMESTAMP=$(date -d "${folder}" "+%s")
            RETENTION_TIMESTAMP=$(date -d "${RETENTION_DATE}" "+%s")
            if [ $FOLDER_TIMESTAMP -lt $RETENTION_TIMESTAMP ]; then
                aws s3 rm s3://${BUCKET}/files/${folder}/ --recursive --quiet
            fi
        done 
    fi
done


The backup retention period is 14 days, you can restore back any day you want.

YAML file:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-files
  namespaces: backup
spec:
  schedule: "0 3 * * *"
  jobTemplate: 
    spec:
      template:
        spec:
          containers: 
          - name: backup-container
            image: ghcr.io/backup/backup-files:latest
            env:
            - name: NAMESPACE 
              value: "backup"
            - name: BUCKET 
              value: "nombre del bucket"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-secret
                  key: AWS_ACCESS_KEY_ID
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-secret
                  key: AWS_SECRET_ACCESS_KEY
            volumeMounts:
            - name: my-pvc
              mounthPath: /app/storage
        restartPolicy: OnFailure
        imagePullSecrets:
          - name: registry-credentials-back
        volumes:
        - name: my-pvc
          persistentVolumeClaim:
            claimName: backend-upload-storage-pvc
      backoffLimit: 4

Postgres YAML file:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-files
  namespaces: backup
spec:
  schedule: "0 3 * * *"
  jobTemplate: 
    spec:
      template:
        spec:
          containers: 
          - name: backup-container
            image: ghcr.io/backup/backup-files:latest
            env:
            - name: NAMESPACE 
              value: "backup"
            - name: BUCKET 
              valueFrom:
                secretKeyRef:
                  name: postgres-pguser-backup
                  key: host
            - name: PG_PORT
              valueFrom:
                secretKeyRef:
                  name: postgres-pguser-backup
                  key: host
            - name: PG_USER
              valueFrom:
                secretKeyRef:
                  name: postgres-pguser-backup
                  key: user
            - name: PG_PASS
              valueFrom:
                secretKeyRef:
                  name: postgres-pguser-backup
                  key: password      
            - name: BUCKET
              value: "nombre del bucket"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-secret
                  key: AWS_ACCESS_KEY_ID
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-secret
                  key: AWS_SECRET_ACCESS_KEY
        restartPolicy: OnFailure
        imagePullSecrets:
          - name: registry-credentials-back
    backoffLimit: 4

Thank you for your time.