Multi Node Kafka Cluster Installation, Configuration and Test (Java, zookeeper, kafka, cmak) on Ubuntu20
Tags: kafka, zookeeper, java, cmak
Published: 2021-12-08 22:24:37

**Hosts** dosyasını **edit**‘leyin.

```javascript
10.10.10.181 kafka01
10.10.10.51  kafka02
10.10.10.128 kafka03
```

**1- Java** install
```javascript
sudo apt install default-jre default-jdk
```

**Zookeeper** ve **Kafka** servisleri için kullanıcı tanımlayalım
```javascript
sudo useradd -m kafka -s /usr/sbin/nologin
sudo useradd -m zookeeper -s /usr/sbin/nologin
```

**2- Zookeeper** kurulum ve yapılandırılması **Tüm node**‘ler aynı adımlar
```javascript
cd /opt/

sudo wget https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz

sudo tar xzf apache-zookeeper-3.6.3-bin.tar.gz

sudo rm apache-zookeeper-3.6.3-bin.tar.gz

sudo mv apache-zookeeper-3.6.3-bin/conf/zoo_sample.cfg apache-zookeeper-3.6.3-bin/conf/zoo.cfg
```
```javascript
sudo vi apache-zookeeper-3.6.3-bin/conf/zoo.cfg

dataDir=/var/zookeeper
clientPort=2181
server.1=10.10.10.181:2888:3888
server.2=10.10.10.51:2888:3888
server.3=10.10.10.128:2888:3888
```
```javascript
sudo mkdir /var/zookeeper/ -p
```

**Herbir node**‘de farklı **echo** satırı yapılabilir.
```javascript
#kafka01 node
sudo echo '1' >> /var/zookeeper/myid

#kafka02 node
sudo echo '2' >> /var/zookeeper/myid

#kafka03 node
sudo echo '3' >> /var/zookeeper/myid
```

Servisin düzgün çalışıp çalışmadığını **manual** olarak aşağıda ki komutlarla test edebilirsiniz, biz ayrıca **systemd** servisi haline getireceğiz
```javascript
sudo /opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh start /opt/apache-zookeeper-3.6.3-bin/conf/zoo.cfg
sudo /opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh stop /opt/apache-zookeeper-3.6.3-bin/conf/zoo.cfg
```

Şimdi ilgili klasörlere **zookeeper** kullanıcısı için sahiplik yetkisi verelim.
```javascript
sudo chown -R zookeeper:zookeeper /opt/apache-zookeeper-3.6.3-bin/
sudo chown -R zookeeper:zookeeper /var/zookeeper/
```

**Zookeeper systemd servis** dosyasını **create** edelim
```javascript
sudo vi /lib/systemd/system/zookeeper.service


[Unit]
Description=Zookeeper Daemon
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target

[Service]
Type=forking
WorkingDirectory=/opt/apache-zookeeper-3.6.3-bin
User=zookeeper
Group=zookeeper
ExecStart=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh start /opt/apache-zookeeper-3.6.3-bin/conf/zoo.cfg
ExecStop=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh stop /opt/apache-zookeeper-3.6.3-bin/conf/zoo.cfg
ExecReload=/opt/apache-zookeeper-3.6.3-bin/bin/zkServer.sh restart /opt/apache-zookeeper-3.6.3-bin/conf/zoo.cfg
TimeoutSec=30
Restart=on-failure

[Install]
WantedBy=default.target
```
```javascript
sudo systemctl daemon-reload
sudo systemctl start zookeeper.service
sudo systemctl enable zookeeper.service
sudo systemctl status zookeeper.service
```

**3- Kafka** kurulum ve yapılandırılması **Tüm node**‘ler aynı adımlar
```javascript
cd /opt/

sudo wget https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz

sudo tar xzf kafka_2.13-3.0.0.tgz

sudo rm kafka_2.13-3.0.0.tgz
```

Herbir **node**‘de farklı **kafka config** ayarı yapılır
```javascript
sudo vi kafka-3.0.0-src/config/server.properties
```

**kafka01 node**
```javascript
broker.id=1
advertised.host.name=kafka01
advertised.listeners=PLAINTEXT://kafka01:9092
zookeeper.connect=10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181
default.replication.factor=3
#delete.topic.enable = true
```

**kafka02 node**
```javascript
broker.id=2
advertised.host.name=kafka02
advertised.listeners=PLAINTEXT://kafka02:9092
zookeeper.connect=10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181
default.replication.factor=3
#delete.topic.enable = true
```

kafka03 node
```javascript
broker.id=3
advertised.host.name=kafka03
advertised.listeners=PLAINTEXT://kafka03:9092
zookeeper.connect=10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181
default.replication.factor=3
#delete.topic.enable = true
```

Servisin düzgün çalışıp çalışmadığını **manual** olarak aşağıda ki komutlarla test edebilirsiniz, biz ayrıca **systemd** servisi haline getireceğiz.
```javascript
sudo /opt/kafka_2.13-3.0.0/bin/kafka-server-start.sh /opt/kafka_2.13-3.0.0/config/server.properties
sudo /opt/kafka_2.13-3.0.0/bin/kafka-server-stop.sh /opt/kafka_2.13-3.0.0/config/server.properties
```

**Kafka systemd servis** dosyasını **create** edelim
```javascript
sudo vi /lib/systemd/system/kafka.service


[Unit]
Description=Kafka Daemon
Documentation=https://kafka.apache.org
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
WorkingDirectory=/opt/kafka_2.13-3.0.0
User=kafka
Group=kafka
Environment="JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64"
ExecStart=/opt/kafka_2.13-3.0.0/bin/kafka-server-start.sh /opt/kafka_2.13-3.0.0/config/server.properties
ExecStop=/opt/kafka_2.13-3.0.0/bin/kafka-server-stop.sh /opt/kafka_2.13-3.0.0/config/server.properties
TimeoutSec=30
Restart=on-failure

[Install]
WantedBy=default.target
```
```javascript
sudo systemctl daemon-reload
sudo systemctl start kafka.service
sudo systemctl enable kafka.service
sudo systemctl status kafka.service
```

**TEST CLI and UI**

**Kafka Cluster**‘ı komut satırı ile **test** etmek
```javascript
#Testi başlatmak, topic açmak için;
/opt/kafka_2.13-3.0.0/bin/kafka-console-producer.sh --topic <fatihaslan> --bootstrap-server kafka01:9092,kafka02:9092,kafka03:9092


#Topic'leri dinlemek için(baştan sona);
/opt/kafka_2.13-3.0.0/bin/kafka-console-consumer.sh --topic <fatihaslan> --bootstrap-server kafka01:9092,kafka02:9092,kafka03:9092 --from-beginning


#Topic detaylarını görmek için lider, replica vs;
/opt/kafka_2.13-3.0.0/bin/kafka-topics.sh --describe --bootstrap-server kafka01:9092,kafka02:9092,kafka03:9092 --topic <fatihaslan>
Topic: fatihaslan       TopicId: qA4gvDiUS0eDH9_V-JngPw PartitionCount: 1       ReplicationFactor: 3    Configs: segment.bytes=1073741824
        Topic: fatihaslan       Partition: 0    Leader: 2       Replicas: 1,3,2 Isr: 2,3,1
```

**4- CMAK** kurulum ve yapılandırılması **Tek host**‘ta yapılması yeterlidir
```javascript
sudo git clone https://github.com/yahoo/CMAK.git

cd CMAK/
```
```javascript
vi /opt/CMAK/conf/application.conf

kafka-manager.zkhosts="10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181"
cmak.zkhosts="10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181"
```
```javascript
./sbt clean dist

sudo cp /home/ubuntu/.sbt/1.0/staging/9fe122a9540185ff93da/cmak/target/universal/cmak-3.0.0.5.zip .

sudo unzip cmak-3.0.0.5.zip

cd cmak-3.0.0.5/

sudo bin/cmak -Dconfig.file=/opt/CMAK/conf/application.conf -Dhttp.port=9000
```
```javascript
Open GUI http://10.10.10.181:9000/

add cluster

Cluster Name enter <3NodesKafkaBroker>

Cluster Zookeeper Hosts <10.10.10.181:2181,10.10.10.51:2181,10.10.10.128:2181>

SAVE
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/r9mdh9kcc6z7qz5mcv7j.png)