The Data Engineering Docker-Compose Starter Kit
Tags: docker, tutorial, development, dataengineering
Published: 2023-09-22 09:38:28


We’ve all been there. Starting a new data engineering project and, instead of diving right into the fun stuff, we’re stuck setting up configurations. Again. 

![Kermit is worrying](https://cdn-images-1.medium.com/max/2000/0*830vZHtPpoeV4nZt.gif)

So, I’ve put together a Docker Compose config with a bunch of services you’ll often see in data engineering projects. Just pull it up on your computer and you’re all set to kick off your own thing. Now, instead of building from scratch, you can jump straight into the action.

Hint: If you don’t know what [Docker](https://docs.docker.com/get-started/) and [Docker Compose](https://docs.docker.com/compose/) are, now’s the perfect time to go find out.

## Why We’re Here and What’s the Deal

[Here](https://github.com/olgazju/blog_de_starter_kit) you can find my starter kit that includes the following services/tools:

 1. [RedPanda](https://docs.redpanda.com/current/home/): Think of it as your data’s personal post office. It’s similar to Kafka, but many say it’s cooler. You send data in, and RedPanda makes sure it gets to where it’s supposed to go, all in real-time. Ideal for managing loads of data and crafting event-driven applications.

 2. Python Consumer Service: This is a tool that’s always active, monitoring specific topic from RedPanda. It grabs incoming data and takes action. For now, I’ve set it to store messages in a table in MySQL, but you can customize it to do much more complex tasks.

 3. [MySQL](https://dev.mysql.com/doc/): You’ve probably heard of it, super popular. Great for when we need to keep things organized and query data quickly.

 4. [S3 (Minio)](https://min.io/docs/minio/container/index.html): It’s our digital storage room. Just like Amazon’s S3, but we can set it up on our local machines for free.

 5. [Mage.ai](https://docs.mage.ai/introduction/overview): Consider it the coordinator for all our data activities. It’s similar to Airflow but sports a fresher look and added flair. It sets the when and how for data tasks, making organization simple and stylish.

 6. [Jupyter Notebook](https://docs.jupyter.org/en/latest/): It’s like our digital lab notebook. We can write code, run it, and see the results all in one place. Makes experimenting and playing with data super fun and easy.

So, with all these tools combined, we’re basically setting up a state-of-the-art digital workshop to handle, play with, and make the most out of our data. Fun times!

## Data’s Dance Across the Stage

Now, let’s transition and focus our attention on Docker.

Docker lets you create, ship, and run applications inside containers. These containers pack everything an application needs from libraries to dependencies. So, when you move your application to a different machine with Docker, it just works. No more “it works on my machine” headaches.

So, what happens when our application has a bunch of containers? Enter Docker Compose.

Docker Compose is like a coordinator for a multi-container orchestra. Instead of manually creating and managing individual containers, developers can define their whole setup in one file: docker-compose.yml. This file lists all the services, networks, and storage spaces our app needs. And the best part? One command, docker-compose up, brings the whole setup to life.

![Me, diving into Docker](https://cdn-images-1.medium.com/max/2000/1*an8dsf_QLgdc6PWKl82HfQ.png)

Now that we’ve talked about the power of Docker and the organizing genius of Docker Compose, you’re probably wondering how to get all this magic working on your machine. Good news: I’ve made it super easy for you. The next steps will guide you through running my ready-to-use docker-compose.yml file.

First, let’s create a file for passwords and secrets in the root folder of the project. This file has already been added to the .gitignore file, so it won't go public.

    touch .secrets

Inside the .secrets file, we'll store the password for the MySQL database and save it. 

    MYSQL_ROOT_PASSWORD=very_strong_password

Time to get things rolling. Execute the command docker-compose up from terminal and see it all come to life.

In just a few moments, here’s what you’ll have ready and waiting:

* RedPanda UI: Accessible at localhost:8080. You'll find a pre-created topic named "my-topic".

* Mage: Head to localhost:6789 to see "my_project" already set up for you.

* Jupyter: Launch it at localhost:8888 and you'll find a notebook titled "my-notebook.ipynb".

* Minio: Visit localhost:9001 and inside you'll discover a bucket named "my-bucket" with a "data.csv" file.

* MySQL: Available at localhost:3306, there's a database named "my_database" waiting for you.

* Python Consumer Service: This is working behind the scenes and will create a “sessions” table in the “my_database”.

For illustration purposes I’ve set up two data flows to highlight how these services interact. You’re free to configure them as you wish.

 1. **Event-Driven App Flow**: Stream events into the Python service and record them in MySQL.

 2. **ETL Flow**: This involves a data pipeline in Mage. Here’s what it does:

* Reads files from s3 (using Minio)

* Aggregates data with pandas

* Saves results to MySQL

* And as a bonus, you can explore the data in Jupyter, ideal for data analytics and scientists.

### First flow: RedPanda -> Python Consumer Service -> MySQL table

Head over to RedPanda at [http://localhost:8080/topics](http://localhost:8080/topics). You’ll notice the “my-topic” is already set up for you.

![precreated my-topic topic](https://cdn-images-1.medium.com/max/4752/1*_3KDG033dEY22s55gZLWag.png)

Click on “my-topic”, then select “Publish message”.

![Publish message button](https://cdn-images-1.medium.com/max/4660/1*fAIRX5at4kzSSboecPoknw.png)

When you publish a message, our Python consumer service will record it in the table defined in the code block below. You don’t need to provide an ‘id’ since it auto-increments. Also, ‘session_id’ is automatically generated. So, all you need to send is a JSON with the ‘user_id’ like this: {"user_id": <your_int_value>}.

    class Sessions(Base):
        __tablename__ = 'sessions'
    
        id = Column(Integer, primary_key=True, autoincrement=True)
        session_id = Column(String(255), default=uuid_gen, nullable=False)
        user_id = Column(Integer, nullable=False)

Go ahead and send a test message.

![sending a test message](https://cdn-images-1.medium.com/max/4012/1*i0xVmJqhtasKpzXIo38w_g.png)

Great, as you can see the message is published.

![the message is published](https://cdn-images-1.medium.com/max/4732/1*-a1Qr0FXoFtQr7BjeUnPAg.png)

If you have Docker Desktop installed, check the Python consumer service log. You should spot our recently published message there.

![published message in Docker Desktop](https://cdn-images-1.medium.com/max/3892/1*Xya67Xh1XXMFRxjfk7KRpQ.png)

Alternatively, you can achieve the same using the command line with docker ps and docker logs commands.

![published message in docker logs](https://cdn-images-1.medium.com/max/4940/1*xTt37XjQUlhbRikp5q-WnQ.png)

Let’s inspect the database.

Install DBeaver community [version](https://dbeaver.io/download/) on your computer and launch it. Your new connection settings should resemble the following:

    Host: localhost
    Port: 3306
    Database: my_database
    Username: root
    Password: very_strong_password

![DBeaver connection](https://cdn-images-1.medium.com/max/2804/1*mJjbkARPBhltayYnoA4gKg.png)

Additionally, navigate to “Driver Settings” -> “Driver Properties” and input the settings as shown in the screenshot below:

![DBeaver properties](https://cdn-images-1.medium.com/max/2000/1*tTMFaiI-imicGOwNOAuBBA.png)

If you’ve set up the connection correctly, this is what you should see:

![DBeaver database structure](https://cdn-images-1.medium.com/max/2000/1*jovc2eC3GDv3a3p8CHqq2w.png)

You’ll likely observe that the “sessions” table already exists. Let’s run a query on it.

    SELECT *
    FROM my_database.sessions;

And the result:

![the test messages appears in the table](https://cdn-images-1.medium.com/max/2156/1*z-gqtC4KX_ruBTFMTfn_Zg.png)

Done! We can see our test message.

### Second flow: S3(minio) -> Mage -> MySQL -> Jupyter Notebook

Access Minio at [http://localhost:9001/login](http://localhost:9001/login). Use both the username and password as “minioadmin”. Once logged in, you’ll find our pre-created bucket containing a file.

![minio bucket screnshot](https://cdn-images-1.medium.com/max/5276/1*TLhgXNpKRYN4BnAiUH0LNg.png)

The “data.csv” file contains historical weather data from Tel Aviv, sourced from another [article](https://dev.to/olgabraginskaya/the-golang-saga-a-coders-journey-there-and-back-again-part-1-leaving-the-shire-4ago) I wrote. If you wish, you can swap it with a public dataset, for example, from [here](https://github.com/awesomedata/awesome-public-datasets).

Next, navigate to Mage at [localhost:6789](http://localhost:6789/). Inside, you’ll discover a pre-established “my_project” containing a single pipeline named “my_pipeline.”

![Mage pipeline screenshot](https://cdn-images-1.medium.com/max/3380/1*w3MROPdSTk5vMjQ45ryrKg.png)

Upon inspecting the pipeline, you’ll notice its workflow: it goes from S3 load to a transformer (pandas) and finally to MySQL export.

![Mage pipeline screenshot](https://cdn-images-1.medium.com/max/2000/1*UXUKyzTn6-7bW2iMRQJwvw.png)

By clicking “Edit” on “my_pipeline”, you’ll be presented with all the associated files and the code for each step.

![Mage pipeline files screenshot](https://cdn-images-1.medium.com/max/3884/1*xvEIP0MaqwZjQGU-Cw_yNw.png)

You might wonder how Mage recognizes connections to S3 or MySQL. In the list of files on the left, there’s an io_config.yaml file detailing these connection settings. Within it, there's a MySQL section pointing to my_database and an AWS section for Minio. 

    version: 0.1.1
    default:
      # Default profile created for data IO access.
      # Add your credentials for the source you use, and delete the rest.
      # AWS
      AWS_ACCESS_KEY_ID: ""
      AWS_SECRET_ACCESS_KEY: ""
      AWS_REGION: us-east-1
      AWS_ENDPOINT: http://minio:9000
      
      # MySQL
      MYSQL_DATABASE: my_database
      MYSQL_HOST: mysql-db
      MYSQL_PASSWORD: very_strong_password
      MYSQL_PORT: 3306
      MYSQL_USER: root

Before you run this pipeline, ensure you’ve set up your own AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in Minio.

 1. Navigate to [http://localhost:9001/access-keys](http://localhost:9001/access-keys) and select “Create access key”.

 2. Copy the provided values into your io_config.yaml under AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

 3. Click the “Create” button and ensure you save the io_config.yaml file.

![set up your own AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY](https://cdn-images-1.medium.com/max/3280/1*9verEW5eYNvWQoBxK9Ad_Q.png)

Next, let’s set up a new table in MySQL. Open DBeaver and execute the following query. While Mage can auto-generate tables, I want to ensure this table remains consistent for all readers.

    CREATE TABLE `mytable` (
      `station` text,
      `_year` text,
      `tmin` decimal(10,2) DEFAULT NULL,
      `tmax` decimal(10,2) DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci

Start the pipeline by visiting [http://localhost:6789/pipelines/my_pipeline/triggers](http://localhost:6789/pipelines/my_pipeline/triggers) and click on “Run pipeline now.” You should see a green indicator, signaling success.

![The pipeline finished successfully](https://cdn-images-1.medium.com/max/3836/1*CWmOTIU2LPbkYLW6NNQHPg.png)

While you can look at the pipeline code for details, here’s a summary: It first fetched our data.csv from s3 during the s3_load step. Subsequently, it performed some transformations using pandas in the transformer step.

      # Filter out rows where TMIN or TMAX is NaN
      df = data.dropna(subset=['TMIN', 'TMAX'])
    
      # Create a 'YEAR' column
      df['YEAR'] = df['DATE'].str[:4]
    
      # Group by year and compute average TMIN and TMAX
      data = df.groupby(['STATION','YEAR']).agg({'TMIN': 'mean', 'TMAX': 'mean'}).round(2).reset_index()
    
      return data

Quick heads-up: What I’m doing in the code above from transformation step is setting up data for a graph we’ll create in a Jupyter notebook. Specifically, I’m calculating the yearly average for both minimum and maximum temperatures, grouped by station and year. 

Finally, in the third step, these results saved to a MySQL table via the mysql_export step, using Mage's connector which you can observe in the code block below.

      with MySQL.with_config(ConfigFileLoader(config_path, config_profile)) as loader:
          loader.export(
              df,
              None,
              table_name,
              index=False,  # Specifies whether to include index in exported table
              if_exists='append',  # Specify resolution policy if table name already exists
          )

Now, let’s verify the data in MySQL. In DBeaver, refresh my_database and then run a query on mytable.

    SELECT *
    FROM my_database.mytable;

This is our aggregation.

![aggregated data after mage pipeline](https://cdn-images-1.medium.com/max/2000/1*ZPmbnKsoA7ezmia_Ffa_8g.png)

Now, let’s explore this data further. Launch Jupyter Notebook localhost:8888. Inside, you’ll find a pre-established notebook. Go ahead and click on it.

![jupyter screenshot](https://cdn-images-1.medium.com/max/4884/1*EvM4ZOix_XDnsnyqFXScuQ.png)

Inside the notebook, you’ll find some code. It essentially connects to the mytable table in MySQL, loads the data into a pandas dataframe, and then utilizes the plotly module to render an impressive graph.

![jupyter code screenshot](https://cdn-images-1.medium.com/max/3844/1*y8FbHq5cp8bvvSM73f2b6A.png)

Execute each cell sequentially, and by the end, you’ll see the result. If you’re curious about the graph’s details, check out the explanation [here](https://medium.com/dev-genius/the-golang-saga-a-coders-journey-there-and-back-again-part-3-the-graphing-conundrum-7f8e0c3f76df)

![graph](https://cdn-images-1.medium.com/max/2948/1*E4BDKl9KFRAXA_ou93crIw.png)

Amazing, right?

![Kermit excited gif](https://cdn-images-1.medium.com/max/2000/0*2PbDF3EhXJs5qe29.gif)

## Dressing Up Your Docker Config for a Perfect Fit

Ready for more? Here are additional ways you can leverage these services:

Alerting and Notification System:

* RedPanda captures real-time metrics or logs from applications.

* The Python listener checks for specific conditions or thresholds. If something looks off, it triggers an alert.

* Mage could then be used to manage these alerts, maybe aggregating them, or pushing them to other systems, or even initiating automatic remedial tasks.

User Behavior Analytics:

* RedPanda streams user activities from a website or application.

* The Python listener processes and enriches this stream with additional user info from MySQL.

* The enriched data lands in S3 (Minio), where analysts can use Jupyter to analyze user behaviors, build funnels, or segment users.

Real-Time Analytics Dashboard:

* RedPanda captures real-time events from different sources, like user activities or sensor data.

* Mage listens to this data, perhaps aggregating data or filtering specific events.

* Processed data is stored in MySQL for structured, quick access.

* A Jupyter notebook fetches this data for analysis and visualization, providing a real-time analytics dashboard for end-users.

Consider adding a Python producer service or producing messages directly from Mage. With Mage, you can even set a trigger to launch the pipeline for every new file on s3. Interested in running a dbt [project](https://docs.getdbt.com/)? You can do it in the Mage pipeline on top of the MySQL database. If MySQL isn’t your favorite, swap it out for Postgres or Mongo. Essentially, there’s a wide range of options for adjusting and enhancing these services.

Also, consider exploring my [article](https://dev.to/olgabraginskaya/unleash-your-pipeline-creativity-local-development-with-argo-workflows-and-minio-on-minikube-2764) about running Argo Workflows locally and another [one](https://dev.to/olgabraginskaya/from-kafka-to-amazon-s3-partitioning-outputs-624) about partitioning outputs from RedPanda on s3.

## Behind Docker’s Doors

Let’s take a look at my repository and look at each service individually.

![repo screenshot](https://cdn-images-1.medium.com/max/3600/1*QGZkY5kLBterZJobyk6S9A.png)

In the repository, you’ll spot two primary files: docker-compose.yml and docker-compose-infra-services.yml. Executing docker-compose up typically looks for the default docker-compose.yml to kick things off. However, to distinguish third-party services (like Kafka or MySQL) from the actual development environment, I've placed those services in docker-compose-infra-services.yml. This structure lets you run external services independently and develop using pure Python, Go, etc., connecting directly via localhost.

    docker-compose -f docker-compose-infra-services.yml up --build

Let’s look at our services one by one.

### RedPanda

In the docker-compose-infra-services.yml file, RedPanda is represented by three interlinked services:

* redpanda-0: This is the RedPanda broker itself. When communicating within the same network, you’d use the host redpanda-0:9092.

* console: This serves as RedPanda’s UI and is accessible at [http://localhost:8080/](http://localhost:8080/).

* redpanda-init: This component waits for redpanda-0 to be ready and then initializes the 'my-topic' topic for us on redpanda-0:9092 

         redpanda-init:
            image: docker.redpanda.com/redpandadata/redpanda:v23.2.6
            depends_on:
              redpanda-0:
                condition: service_healthy
            networks:
              - redpanda_network
            command: topic create -r 1 -p 1 my-topic --brokers redpanda-0:9092

### MySQL

In the docker-compose file, MySQL is represented as a single service. However, it has its nuances. One of them is that it uses the .secrets file to pull the MYSQL_ROOT_PASSWORD variable.

    mysql-db:
        build:
          context: .
          dockerfile: Dockerfile.mysql
        container_name: mysql-db
        healthcheck:
          test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
          timeout: 20s
          retries: 10
        ports:
          - "3306:3306"
        environment:
          - MYSQL_DATABASE=my_database
        env_file:
           - '.secrets'
        volumes:
          - mysql-data:/var/lib/mysql
        networks:
          - redpanda_network

You may also observe that it utilizes a custom Dockerfile, which is shown in the following code block.

    # Dockerfile.mysql
    FROM mysql:8.1
    
    COPY ./init.sql /docker-entrypoint-initdb.d/

The reason behind this is that I wanted to have a precreated database ready from the get-go. This way, you can set up anything you want in MySQL before launching your services, all within the init.sql file in the form of MySQL queries. Feel free to ingest data or make any necessary configurations. Let's take a look at the sql file:

    -- init.sql
    CREATE DATABASE IF NOT EXISTS my_database;
    
    USE my_database;

### Minio/s3

It comprises two linked services. The aws service takes care of creating a bucket and uploading the data.csv file from the repository on our behalf. However, you also have the flexibility to create your own buckets and upload data.

    minio:
        image: minio/minio:RELEASE.2023-08-29T23-07-35Z
        container_name: s3
        ports:
          - "9000:9000"
          - "9001:9001"
        command: server --console-address ":9001" /data
        networks:
          - redpanda_network
    
      aws:
        image: amazon/aws-cli
        container_name: aws-cli
        command: -c "
                sleep 2
                && aws --endpoint-url http://minio:9000 s3 mb s3://my-bucket --region us-east-1
                && aws --endpoint-url http://minio:9000 s3 cp data.csv  s3://my-bucket/data.csv
                && exit 0"
        entrypoint: [/bin/bash]
        volumes:
          - "./local_data:/aws"
        environment:
          AWS_ACCESS_KEY_ID: "minioadmin"
          AWS_SECRET_ACCESS_KEY: "minioadmin"
        depends_on:
          - minio
        networks:
          - redpanda_network

### Mage

This service is quite straightforward. It loads Python sources from mage_src as my_project. Any modifications you make locally in this folder  will be instantly reflected in Mage.

     mage-ai:
        image: mageai/mageai:latest
        ports:
          - 6789:6789
        command:
          - /app/run_app.sh
          - mage
          - start
          - my_project
        volumes:
          - ./mage_src:/home/src
        networks:
          - redpanda_network

### Jupyter

This service loads the precreated my-notebook.ipynb file into Jupyter notebooks. You won't be prompted for a password due to the empty NotebookApp.token and NotebookApp.password.

    jupyter:
        image: jupyter/minimal-notebook:latest
        ports:
          - 8888:8888
        volumes:
          - ./jupyter_workspace:/home/jovyan
        # disable password for jupyter
        command:
          - jupyter
          - notebook
          - --no-browser
          - --NotebookApp.token=''
          - --NotebookApp.password=''
        networks:
          - redpanda_network

### Python consumer service

    my-service:
        build:
          context: .
          dockerfile: python_app/Dockerfile
        depends_on:
          mysql-db:
            condition: service_healthy 
          redpanda-init:
            condition: service_started
        networks:
          - redpanda_network
        environment:
          - BROKER_URL=redpanda-0:9092
          - BROKER_TOPIC=my-topic
          - MYSQL_DATABASE=my_database
          - MYSQL_URL=mysql-db:3306
        env_file:
           - '.secrets'
        #volumes:
        #  - .:/home/user/app
        restart: on-failure 

The code for this service is located in the python_app folder. It utilizes confluent_kafka for handling RedPanda messages and sqlalchemy for writing to the MySQL table. Essentially, it continuously polls messages from Kafka and persists them in MySQL.

{% gist https://gist.github.com/olgazju/47e273e02def0e85ffc1ad975a3a020f.js %}

The service waits for both MySQL and RedPanda to be up and running. It then uses environment variables for establishing connections within the code. Additionally, it retrieves the MYSQL_ROOT_PASSWORD from the .secret file.


## The Show’s Over, But the Journey’s Just Begun

That’s all there is to it. As you work on your data projects, keep this kit in your toolbox — it’s here to simplify your journey and make your data work smoother.