Day 3 of My 90 Days of DevOps Projects: Monitoring with Prometheus and Grafana
Tags: devops, monitoring, beginners, day3
Published: 2024-06-21 01:43:17





Hi guys,

I am a bit late today to Day 3 of my 90 days of DevOps project building. I am one of the participants in the ongoing Zero to Merge open source projects organized by CNCF, and it's my first time juggling that with other life issues. It's quite interesting, but we move! Now, I am here, and today my focus was on Prometheus for monitoring and Grafana for visualization and alerting.

### What Do Prometheus and Grafana Do?

Prometheus and Grafana are like having a health monitor for your applications. Prometheus collects data about how well your applications are running, such as response times and error rates. Grafana then takes this data and displays it in beautiful, easy-to-read graphs and charts. This helps you see at a glance if your applications are running smoothly or if something needs attention.

### What You'll Gain from This Task

By the end of this task, you’ll learn how to:
- Set up a Node.js application that Prometheus can monitor.
- Use Docker to run Prometheus and Grafana.
- Create a Grafana dashboard to visualize metrics collected by Prometheus.
- Understand the basics of monitoring and why it's crucial in DevOps.
- Use GitHub Actions for CI/CD to build and deploy Docker images.

### What You Need

Before we dive in, ensure you have the following:
- Docker installed on your computer.
- A GitHub account for repository management.
- Basic knowledge of Node.js and Docker.
- Access to Docker Hub.

### Step-by-Step Guide

#### Step 1: Set Up a Sample Node.js Application

 **1. Create a Simple Node.js Application**:
   - Initialize a new Node.js project and install the necessary dependencies.
   - Create a server that exposes some metrics for Prometheus.

   ```javascript
   const express = require('express');
   const app = express();
   const port = 4000;

   app.get('/', (req, res) => res.send('Hello World!'));

   app.listen(port, () => console.log(`Example app listening at http://localhost:${port}`));
   ```

**2. Add Prometheus Metrics**:
   - Use the `prom-client` package to collect and expose metrics.

   ```javascript
   const client = require('prom-client');
   const collectDefaultMetrics = client.collectDefaultMetrics;

   collectDefaultMetrics();

   app.get('/metrics', (req, res) => {
     res.set('Content-Type', client.register.contentType);
     res.end(client.register.metrics());
   });
   ```

**3. Dockerize Your Application**:
   - Create a `Dockerfile` to containerize your Node.js application.

   ```dockerfile
   FROM node:14
   WORKDIR /app
   COPY package*.json ./
   RUN npm install
   COPY . .
   EXPOSE 4000
   CMD ["node", "index.js"]
   ```

**4. Build and Push Docker Image**:
   - Use Docker commands to build and push your Docker image to Docker Hub.

   ```bash
   docker build -t your_dockerhub_username/my-monitoring-app:latest .
   docker push your_dockerhub_username/my-monitoring-app:latest
   ```

#### Step 2: Set Up GitHub Secrets for Docker Login

1. **Create GitHub Secrets**:
   - Go to your GitHub repository, click on **Settings**, then **Secrets** and **Actions**.
   - Add the following secrets:
     - `DOCKER_USERNAME`: Your Docker Hub username.
     - `DOCKER_PASSWORD`: Your Docker Hub password.

#### Step 3: Configure GitHub Actions for CI/CD

1. **Create a GitHub Actions Workflow**:
   - Create a file `.github/workflows/docker-deploy.yml` with the following content:

   ```yaml
   name: Docker Build & Deploy

   on:
     push:
       branches:
         - main

   jobs:
     build-and-deploy:
       runs-on: ubuntu-latest
       steps:
         - name: Checkout Repository
           uses: actions/checkout@v2

         - name: Login to Docker Hub
           uses: docker/login-action@v2
           with:
             username: ${{ secrets.DOCKER_USERNAME }}
             password: ${{ secrets.DOCKER_PASSWORD }}

         - name: Build and Push Docker Image
           run: |
             docker build -t ${{ secrets.DOCKER_USERNAME }}/my-monitoring-app:latest .
             docker push ${{ secrets.DOCKER_USERNAME }}/my-monitoring-app:latest

         - name: Deploy Docker Container
           run: |
             docker-compose up -d
   ```

#### Step 4: Set Up Prometheus and Grafana with Docker Compose

**1. Prometheus Configuration**:
   - Create a `prometheus.yml` file to define how Prometheus will scrape metrics from your Node.js application.

   ```yaml
   global:
     scrape_interval: 15s

   scrape_configs:
     - job_name: 'node-app'
       static_configs:
         - targets: ['host.docker.internal:4000']
   ```

**2. Create a `docker-compose.yml` File**:
   - Define services for Prometheus and Grafana.

   ```yaml
   version: '3'

   services:
     prometheus:
       image: prom/prometheus
       volumes:
         - ./prometheus.yml:/etc/prometheus/prometheus.yml
       ports:
         - "9090:9090"

     grafana:
       image: grafana/grafana
       ports:
         - "3000:3000"
   ```

**3. Run Docker Compose**:
   - Use Docker Compose to start Prometheus and Grafana.

   ```bash
   docker-compose up -d
   ```

#### Step 5: Create a Dashboard in Grafana

1. **Access Grafana**:
   - Open your browser and navigate to [http://localhost:3000](http://localhost:3000).
   - Log in with the default credentials (Username: `admin`, Password: `admin`).

2. **Add Prometheus as a Data Source**:
   - Go to **Configuration > Data Sources > Add data source > Prometheus**.
   - Set the URL to `http://prometheus:9090` and click **Save & Test**.

3. **Create a New Dashboard**:
   - Click on the **+ (plus sign)** in the left sidebar and select **Dashboard**.
   - Click **Add new panel** to create a new panel.

4. **Configure the Panel**:
   - Set the **Panel title** to something like "HTTP Request Duration".
   - In the **Query** section, select your Prometheus data source and enter a query like `http_request_duration_ms_bucket`.
   - Customize the visualization settings to your preference and click **Apply**.

5. **Save the Dashboard**:
   - After configuring the panel, click **Save dashboard** at the top right, give it a name, and save it.

### Challenges and Solutions

**Challenge 1: Docker Daemon Not Running**
- **Problem**: Received an error indicating that the Docker daemon is not running.
- **Solution**: Ensure Docker Desktop is running and restart the Docker service.

**Challenge 2: Port Already Allocated**
- **Problem**: Encountered an error that the port 3000 is already allocated.
- **Solution**: Stop any service using the port 3000 and changed the Grafana port in the `docker-compose.yml` file to 3100.

**Challenge 3: Grafana Panel Configuration Confusion**
- **Problem**: Confusion about how to add a panel in Grafana.
- **Solution**: Detailed step-by-step instructions were provided to create and configure a panel in Grafana.

**Challenge 4: Setting Up GitHub Secrets for Docker Login**
- **Problem**: Difficulty in setting up GitHub secrets for Docker login.
- **Solution**: Provided a detailed guide on adding Docker Hub credentials as GitHub secrets.

### How to Add This Project to GitHub

1. **Initialize a Git Repository**:
   - Navigate to your project directory and initialize a git repository.

   ```bash
   git init
   ```

2. **Commit Your Changes**:
   - Add and commit your project files.

   ```bash
   git add .
   git commit -m "Initial commit"
   ```

3. **Push to GitHub**:
   - Create a new repository on GitHub and push your local repository.

   ```bash
   git remote add origin https://github.com/your_username/your_repository_name.git
   git push -u origin main
   ```

### Conclusion

It’s not easy, but I will push through till the end. Thanks for stopping by, and see you on Day 4 in a bit!

Thanks to my amazing community that goes over and beyond @shecodeAfrica and my amazing mentor @cornelia for her great support.

In this project, I set up a monitoring system using Prometheus and Grafana for a Node.js application. This task involved Dockerizing the application, running Prometheus and Grafana using Docker Compose, creating a Grafana dashboard to visualize metrics, and configuring GitHub Actions for CI/CD. Through these steps, I learned the importance of monitoring in DevOps and how to set up and configure essential monitoring tools.

By following this guide, you should now have a working monitoring setup for your own applications, and you can build on this foundation to add more sophisticated monitoring and alerting as needed. Happy monitoring!

---