| # Lessons Learned | |
| ## Table of Contents | |
| 1. [Docker](#1-docker) | |
| 2. [Dev Containers](#2-dev-containers) | |
| 3. [Redis](#3-redis) | |
| 4. [Postgres](#4-postgres) | |
| 5. [Unit Tests](#5-unit-tests) | |
| 6. [Locust](#6-locust) | |
| 7. [AWS](#7-aws) | |
| 8. [GitHub Actions](#8-github-actions) | |
| 9. [Streamlit](#9-streamlit) | |
| ## 1. Docker | |
| - Install [Docker](https://docs.docker.com/engine/install/) | |
| - Some basic Docker commands: | |
| ```bash | |
| # List all containers | |
| docker ps | |
| # List all images | |
| docker images | |
| # Remove resources | |
| docker rmi $(docker images -q) | |
| docker rm $(docker ps -aq) | |
| # Network | |
| docker network ls | |
| docker network rm <network_id> | |
| # Volumes | |
| docker volume ls | |
| docker volume rm <volume_id> | |
| # Build an image | |
| docker build -t <image_name> . | |
| # Build a test image (target) | |
| docker build -t ui_test --progress=plain --target test . | |
| # Run a container | |
| docker run -p <port>:<port> -d <image_name> | |
| ``` | |
| ### 1.1 Dockerfile | |
| - A basic example of a Dockerfile | |
| ```Dockerfile | |
| # Base image | |
| FROM python:3.9 | |
| # Set working directory | |
| WORKDIR /app | |
| # Copy files | |
| COPY . /app | |
| # Install dependencies (during build time) | |
| RUN pip install -r requirements.txt | |
| # Run the app (after installing dependencies) | |
| CMD ["python", "main.py"] | |
| ``` | |
| ### 1.2 Docker Compose | |
| - A basic example of a Docker Compose file running microservices: | |
| ```yml | |
| services: | |
| api: # Name of the service | |
| build: # Build the image | |
| context: ./ui | |
| target: build # Target | |
| ports: # Ports | |
| - "8000:8000" | |
| environment: # Env variables | |
| POSTGRES_DB: $POSTGRES_DB | |
| POSTGRES_USER: $POSTGRES_USER | |
| POSTGRES_PASSWORD: $POSTGRES_PASSWORD | |
| DATABASE_HOST: $DATABASE_HOST | |
| depends_on: # Dependencies | |
| - redis | |
| - db | |
| networks: # Network | |
| - shared_network | |
| volumes: # Volumes | |
| - ./uploads:/src/uploads | |
| db: # Another service | |
| image: postgres:13-alpine | |
| volumes: # Volumes | |
| - postgres_data:/var/lib/postgresql/data | |
| redis: # Another service | |
| image: redis:6.2.6 | |
| networks: | |
| - shared_network | |
| networks: | |
| shared_network: | |
| volumes: | |
| postgres_data: | |
| ``` | |
| ## 2. Dev Containers | |
| - Install [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension | |
| - Create a `.devcontainer` folder | |
| - Create folders for each service you want to develop (api, model, ui, etc.) | |
| - Create a `devcontainer.json` file | |
| - Optionally you can create Dockerfiles or Docker Compose files for each service | |
| ```json | |
| { | |
| "name": "ML Project - API", | |
| "dockerComposeFile": "../../docker-compose-dev.yml", | |
| "service": "api", | |
| "workspaceFolder": "/src", | |
| "customizations": { | |
| "vscode": { | |
| "extensions": ["ms-python.python"] | |
| } | |
| }, | |
| "shutdownAction": "none" | |
| } | |
| ``` | |
| - Note: Don't use `COPY` command inside your Dockerfile, use `volumes` in your Docker Compose file to edit files inside the container. (for development stage) | |
| ```yml | |
| # ... | |
| volumes: | |
| - ./ui:/src:cached | |
| # ... | |
| ``` | |
| - Run the dev container by selecting `devcontainer: Reopen in Container` option (`ctrl + shift + P`) | |
| ## 3. Redis | |
| - To connect to a Redis instance by terminal you can use the following command: | |
| ```bash | |
| redis-cli | |
| ``` | |
| - Some basics commands: | |
| ```bash | |
| # Set a key | |
| SET key value | |
| # Get a key | |
| GET key | |
| # Delete a key | |
| DEL key | |
| # List all keys | |
| KEYS * | |
| ``` | |
| Review [Redis Commands Cheat Sheet](https://redis.io/learn/howtos/quick-start/cheat-sheet) for more commands. | |
| - Monitor the Redis instance with the command: | |
| ```bash | |
| redis-cli monitor | |
| ``` | |
| ## 4. Postgres | |
| - To connect to a Postgres instance by terminal you can use the following command: | |
| ```bash | |
| psql -U <username> -p 5432 -d <database_name> | |
| ``` | |
| - Can also use a Postgres GUI client, like DBeaver. So you can use the connection values from the Docker Compose file. | |
| - Some basics commands are: | |
| ```bash | |
| # List all databases | |
| \l | |
| # Create a database | |
| CREATE DATABASE <database_name>; | |
| # List all tables | |
| \dt | |
| # Exit | |
| \q | |
| ``` | |
| ## 5. Unit Tests | |
| ### 5.1 Unittest | |
| - To run unit tests you can use the following commands: | |
| ```bash | |
| # Run test file | |
| python3 -m unittest -vvv tests.test_model | |
| # Run an individual test | |
| python -m unittest -vvv tests.test_image_classifier_app.TestMLService.test_login_failure | |
| ``` | |
| - Additionally can run test from the python module: | |
| ```py | |
| if __name__ == "__main__": | |
| unittest.main(verbosity=2) | |
| ``` | |
| And then run the module | |
| ```bash | |
| python tests/test_image_classifier_app.py | |
| ``` | |
| ### 5.2 Pytest | |
| - To run unit tests you can use the following commands: | |
| ```bash | |
| # Run test file | |
| pytest -v -s tests/test_model.py | |
| # Run an individual test | |
| pytest -v -s tests/test_image_classifier_app.py::TestMLService::test_login_failure | |
| ``` | |
| ## 6. Locust | |
| - Install [locust](https://docs.locust.io/en/stable/quickstart.html#locust-s-web-interface) | |
| ```bash | |
| uv add locust | |
| ``` | |
| - Create a folder `stress_test` and `locustfile.py` file | |
| ```py | |
| # Basic example | |
| from locust import HttpUser, task | |
| class HelloWorldUser(HttpUser): | |
| @task | |
| def hello_world(self): | |
| self.client.get("/hello") | |
| self.client.get("/world") | |
| ``` | |
| - Run `locust -f stress_test/locustfile.py` | |
| - Open `http://127.0.0.1:8089` | |
| - Start a load test and fill the number of users/ramp up | |
| - Add the host, for example `http://localhost:8000` | |
| - Start the load test | |
| - Review the results, stats and charts | |
| ## 7. AWS | |
| - Download the `epm` file from your AWS account | |
| - Give read permissions to the file with: | |
| ```bash | |
| chmod 400 file.epm | |
| ``` | |
| - Connect it via ssh | |
| ```bash | |
| ssh -i file.epm <ec2_user>@<public_ip> | |
| ``` | |
| - To copy files from local host to remote host: | |
| ```bash | |
| scp -i file.epm -r <local_path> <ec2_user>@<public_ip>: # Default home | |
| scp -i file.epm -r <local_path> <ec2_user>@<public_ip>:<remote_path> | |
| ``` | |
| - Create a tunnel to the remote host: | |
| ```bash | |
| ssh -L <local_port>:<remote_host>:<remote_port> -i file.epm <ec2_user>@<public_ip> | |
| ``` | |
| ## 8. GitHub Actions | |
| GitHub Actions is a **CI/CD (Continuous Integration and Continuous Deployment)** tool built into GitHub that allows you to **automate workflows** directly from your repository. With Actions, you can run tests, build your code, deploy applications, and perform other automated tasks whenever certain events occur (like pushing code, creating pull requests, or publishing releases). | |
| Key points: | |
| - **Event-driven**: Workflows run based on triggers like `push`, `pull_request`, or scheduled cron jobs. | |
| - **YAML-based**: Workflows are defined in `.github/workflows/` using YAML syntax. | |
| - **Cross-platform**: Supports Linux, Windows, and macOS runners. | |
| - **Marketplace**: Offers reusable actions to speed up development. | |
| --- | |
| ### **Basic Example: CI Workflow for Python Project** | |
| ```yaml | |
| # File: .github/workflows/python-ci.yml | |
| name: Python CI | |
| on: | |
| push: | |
| branches: ["main"] | |
| pull_request: | |
| branches: ["main"] | |
| jobs: | |
| build: | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout code | |
| uses: actions/checkout@v3 | |
| - name: Set up Python | |
| uses: actions/setup-python@v4 | |
| with: | |
| python-version: "3.10" | |
| - name: Install dependencies | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install -r requirements.txt | |
| - name: Run tests | |
| run: | | |
| pytest | |
| ``` | |
| **What this does:** | |
| - Runs when you **push or create a pull request** to the `main` branch. | |
| - Uses **Ubuntu** as the environment. | |
| - Sets up **Python 3.10**, installs dependencies, and runs **pytest** for tests. | |
| ## 9. Streamlit | |
| To fix **telemetry issues** and **403 errors on file uploads** during deployment (Docker/Hugging Face Spaces)., configure Streamlit with a `config.toml` in `/app/.streamlit/`: | |
| ```toml | |
| [browser] | |
| gatherUsageStats = false | |
| [server] | |
| enableCORS = false | |
| enableXsrfProtection = false | |
| ``` | |
| In Dockerfile: | |
| ```dockerfile | |
| RUN mkdir -p /app/.streamlit /app/tmp | |
| COPY .streamlit/ /app/.streamlit/ | |
| ``` | |
| Or generate the same configuration in build stage: | |
| ```dockerfile | |
| RUN mkdir -p /app/.streamlit \ | |
| && echo "[browser]\n" \ | |
| "gatherUsageStats = false\n\n" \ | |
| "[server]\n" \ | |
| "enableCORS = false\n" \ | |
| "enableXsrfProtection = false\n" \ | |
| > /app/.streamlit/config.toml | |
| ``` | |