How I built a real-time Machine Learning system with Kafka, Elasticsearch, Kibana, and Docker
Tags: streaming, machinelearning, dataengineering, datascience
Published: 2022-12-28 16:49:20

---
title: How I built a real-time Machine Learning system with Kafka, Elasticsearch, Kibana, and Docker

published: true

date: 2022-12-04 04:00:14 UTC

cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xo1ixrbne1vpbuzvhwj1.png

tags: streaming, machinelearning, dataengineering, datascience

canonical_url: https://dipankarmedhi.hashnode.dev/how-i-built-a-real-time-machine-learning-system-with-kafka-elasticsearch-kibana-and-docker
---

We will design and build a real-time sentiment analysis and hate detection system.

This is a project that I made in the **Turn Language into Action, Natural Language Hackathon by** [**Expert.ai**](http://Expert.ai) **.**

I have always been interested in real-time systems and have always wondered how things work under the hood.

**HOW**? 🤔

![](https://media.giphy.com/media/10yIEN8cMn4i9W/giphy.gif)

So, I found this hackathon to be a perfect opportunity for me to learn and build something new.

Well then, Lets **ROLL!!!**

![](https://media.giphy.com/media/qWSB4rUykts9SWayCW/giphy.gif)

# Project Architecture

This is what the complete pipeline looks like. Dont worry I will cover everything in detail.

![Project Architecture](https://miro.medium.com/max/875/0*hVyJLo7cFRfiQkwp.png)

But before we move on with the tools and architecture, let me talk about our data sources.

I have used Twitter API for real-time tweets, specifically pythons [tweepy](https://docs.tweepy.org/en/stable/) library for streaming tweets. In addition to that, I have used [NewsAPI](https://newsapi.org/) for daily news articles.

I have used **docker** to set up all the necessary tools as containers for this project.

Now lets talk about each component.

# Apache Kafka

For ingesting the real-time data, I have used Apache Kafka.

Now, what is **Apache Kafka?** Well

> Apache Kafka (Kafka) is an open source, distributed streaming platform that enables (among other things) the development of real-time, event-driven applications. IBM

Since I have used Python, there is a python client [**kafka-python**](https://github.com/dpkp/kafka-python) available that makes working with Kafka relatively easy.

Using the **KafkaProducer** , Ive sent the messages (Twitter and NewsAPI) via 2 Kafka topics to the **KafkaConsumer**. One for the tweets and the other one for the news articles respectively.

KafkaConsumer then calls the Machine Learning service to classify the sentiments of the news media articles and detect hate in the tweets.

# Machine Learning service

> [Expert.ai](http://Expert.ai) turns language into data so teams can make better decisions.

Since I built this project as a part of the [**Expert.ai**](http://Expert.ai) hackathon, I have used their API for sentiment analysis/classification and hate detection.

However, you can always use your own **Tensorflow** or **PyTorch** model. Also, **Huggingface** has some very relevant models for sentiment classifications and they are straightforward to set up. You should check them out!

I am using the [**Sentiment Analysis**](https://docs.expert.ai/nlapi/latest/guide/sentiment-analysis/) and [**Hate speech detection**](https://docs.expert.ai/nlapi/latest/guide/detection/hate-speech/) APIs from [Expert.ai](http://Expert.ai) NL API.

# Elasticsearch

Okay, we have the classified data. Now What?

We have to store that data somewhere to use it for further analytics. I have used Elasticsearch and Kibana to visualize the stored data.

You might ask, why Kibana?

Let me introduce you to the **ELK stack**.

> ELK is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a serverside data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a stash like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch. [Elastic.co](http://Elastic.co)

Elasticsearch, Logstash and Kibana go hand in hand in most data engineering or data ingestion use cases. But I have omitted Logstash to keep the pipeline simple and clear to its goal.

But, you can always add Logstash and scale the pipeline further as needed.

That is enough about the ELK stack. Lets jump into the Elasticsearch design.

### Elasticsearch: The Official Distributed Search & Analytics Engine

Like databases, Elasticsearch has " **Indexes"**. These indexes store data defined with certain mappings type. Mapping is more like a schema in other databases.

The mapping describes the fields in the JSON documents along with their data type, as well as how they should be indexed in the indexes.

![](https://miro.medium.com/max/875/1*Uy1onwyELiEMK_DqHY8nPw.png)

`Databases ~ Indexes`

The above image will give you a better idea about Elasticsearch indexes compared to MySQL or PostgreSQL.

# Kibana

Done with storing the messages/data in the Elasticsearch indexes? Okay, Great! We can finally use that resultant data to visualize and get more insights about the data.

We use Kibana for that.

## Kibana: Explore, Visualize, Discover Data | Elastic

### Your window into the Elastic Stack Kibana is a free and open user interface that lets you visualize your Elasticsearch

[www.elastic.co](http://www.elastic.co)

> Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack.

### Kibana Dashboard

![](https://miro.medium.com/max/875/1*oIRtrvCI1c-m6ebjdn_DvQ.gif)

This is what my final Kibana dashboard looks like. You can check out the code at my GitHub [repo](https://github.com/Dipankar-Medhi/sense_media).

Feel free to leave a star if you like the project.

This part covers only the idea or the overview of the project along with the project architecture. Ill soon add the coding section in a separate part so stay tuned for that

* * *

Thats all folks. See you soon 👋

Happy coding.