To study Apache Kafka Architecture in details, and how to install, deploy configure Apache kafka.
Tags: bigdata, apache, kafka, manual
Published: 2022-11-17 20:46:27

Apache Kafka is an open-source, distributed event streaming platform developed by the Apache Software Foundation. This is written in Scala and Java programming languages. You can install Kafka on any platform that supports Java programming language.

This tutorial provides you with step-by-step instructions to install Apache Kafka on Ubuntu 22.04 LTS Linux system. You will also learn to create topics in Kafka and run producer and consumer nodes.

**Prerequisites**
You must have sudo privileged account access to the Ubuntu 22.04 Linux system.

**Step 1 – Installing Java**
We can run the Apache Kafka server on systems that support Java. So make sure, you have Java installed on your Ubuntu system.

Use the following commands to install OpenJDK on your Ubuntu system from the official repositories.

```
sudo apt update  
sudo apt install default-jdk 
```
Verify the current active Java version.

```
java --version 
```
**Output:**

> openjdk version "11.0.15" 2022-04-19
> OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1)
> OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.22.04.1, mixed mode, sharing)


**Step 2 – Download Latest Apache Kafka**
You can download the latest Apache Kafka binary files from its [official download page](https://kafka.apache.org/downloads). Alternativaly you can download Kafka 3.2.0 with the below command.

```
wget https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz 
```
Then extract the downloaded archive file and place them under /usr/local/kafka directory.

```

tar xzf kafka_2.13-3.2.0.tgz 
sudo mv kafka_2.13-3.2.0 /usr/local/kafka 

```

**Step 3 – Create Systemd Startup Scripts**
Now, create systemd unit files for the Zookeeper and Kafka services. That will help you to start/stop the Kafka service in an easy way.

First, create a systemd unit file for Zookeeper:

```
sudo nano /etc/systemd/system/zookeeper.service 
```
And add the following content:

```
[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties
ExecStop=/usr/local/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
```
Save the file and close it.

Next, create a systemd unit file for the Kafka service:

```
sudo nano /etc/systemd/system/kafka.service 
```
Add the below content. Make sure to set the correct JAVA_HOME path as per the Java installed on your system.

find `/usr/lib/jvm/java-1.x.x-openjdk`

```
sudo vim /etc/profile
```

```
export JAVA_HOME="path that you found"
export PATH=$JAVA_HOME/bin:$PATH
```
 logout and login again, reboot, or use `source /etc/profile` to apply changes immediately in your current shell
```
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties
ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target
```
Save the file and close.

Reload the systemd daemon to apply new changes.

```
sudo systemctl daemon-reload 
```

This will reload all the systemd files in the system environment.


Step 4 – Start Zookeeper and Kafka Services

Let’s start both services one by one. First, you need to start the ZooKeeper service and then start Kafka. Use the systemctl command to start a single-node ZooKeeper instance.


```
sudo systemctl start zookeeper 
sudo systemctl start kafka 
```

Verify both of the services status:


```
sudo systemctl status zookeeper 
sudo systemctl status kafka 
```

That’s it. You have successfully installed the Apache Kafka server on Ubuntu 22.04 system. Next, we will create topics in the Kafka server.