Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD
Tags: apachekafka, apachenifi, cloudera, streamsmessagingma
Published: 2019-12-03 21:48:56

---
title: Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD
published: true
date: 2019-08-22 20:57:00 UTC
tags: apache-kafka,apache-nifi,cloudera,streams messaging ma
canonical_url: 
---

#### Using Cloudera Streams Messaging Manager for Apache Kafka Monitoring, Management, Analytics and CRUD

SMM is powerful tool to work with Apache Kafka and provide you with monitoring, management, analytics and creating Kafka topics.   You will be able to monitor servers, brokers, consumers, producers, topics and messages.   You will also be able to easily build alerts based on various events that can occur with those entities.

  

From Cloudera Manager, we can now install and manage Kafka, SMM, NiFi and Hadoop services.

  

[![](https://1.bp.blogspot.com/-SFyNviPo3lI/XV7tjwZ7OFI/AAAAAAAAYGI/2c0K4MqA1KQQ1intN5Dwl0dtnEjhF3kIwCLcBGAs/s640/cemCM63Services.png)](https://1.bp.blogspot.com/-SFyNviPo3lI/XV7tjwZ7OFI/AAAAAAAAYGI/2c0K4MqA1KQQ1intN5Dwl0dtnEjhF3kIwCLcBGAs/s1600/cemCM63Services.png)

  

Let's create a Kafka topic, no command-line!

  

[![](https://1.bp.blogspot.com/-ae9ZHnlJ5qs/XV7tj-buhqI/AAAAAAAAYGE/uBvzBdqSFQo_F_jUos92cT2CVgAcPvsKACLcBGAs/s640/addTopicHigh.png)](https://1.bp.blogspot.com/-ae9ZHnlJ5qs/XV7tj-buhqI/AAAAAAAAYGE/uBvzBdqSFQo_F_jUos92cT2CVgAcPvsKACLcBGAs/s1600/addTopicHigh.png)

  

  

  

  

For a simple topic, we select Low size for **replication factor** of one and **replica** count of one.  We also set a cleanup policy of **delete**.
  

[![](https://1.bp.blogspot.com/-0pQz-U08-sA/XV7tjyAPIrI/AAAAAAAAYGA/CEZkStnbiaQ2vpRcz0CdBm8gEiNthwkZwCLcBGAs/s640/smmAddTopic.png)](https://1.bp.blogspot.com/-0pQz-U08-sA/XV7tjyAPIrI/AAAAAAAAYGA/CEZkStnbiaQ2vpRcz0CdBm8gEiNthwkZwCLcBGAs/s1600/smmAddTopic.png)

  

  

Let's create an **alert.**
  

[![](https://1.bp.blogspot.com/-kqVY7Og1VtQ/XV7tkkjSxLI/AAAAAAAAYGM/e8rbMSPOknE3JVvTok5f6ywGqmNuWKVNwCLcBGAs/s640/smmAlertPolicy.png)](https://1.bp.blogspot.com/-kqVY7Og1VtQ/XV7tkkjSxLI/AAAAAAAAYGM/e8rbMSPOknE3JVvTok5f6ywGqmNuWKVNwCLcBGAs/s1600/smmAlertPolicy.png)

  

For this one if the **nifi-reader** consumer group has a lag then send an email to me.

  

  

Let's browse our Kafka infrastructure in our AWS Cloudera Kafka cluster, so easy to navigate.

  

[![](https://1.bp.blogspot.com/-kJ3SvunVEIo/XV7tlp5KMrI/AAAAAAAAYGU/6wP_9bomEt4TN4eOCS3VRi3ZGhBoEGyOgCLcBGAs/s640/smmMetrics.png)](https://1.bp.blogspot.com/-kJ3SvunVEIo/XV7tlp5KMrI/AAAAAAAAYGU/6wP_9bomEt4TN4eOCS3VRi3ZGhBoEGyOgCLcBGAs/s1600/smmMetrics.png)

  

[![](https://1.bp.blogspot.com/-JHRF3B76yhE/XV7tmiiPgPI/AAAAAAAAYGc/gIDdlLIT-QcejxXnjJdBKznk3FOp9QjuACLcBGAs/s640/smmTopicsMetrics.png)](https://1.bp.blogspot.com/-JHRF3B76yhE/XV7tmiiPgPI/AAAAAAAAYGc/gIDdlLIT-QcejxXnjJdBKznk3FOp9QjuACLcBGAs/s1600/smmTopicsMetrics.png)

  

You can dive into a topic and see individual messages, see offsets, keys, values, timestamps and more.
  

[![](https://1.bp.blogspot.com/-4tpKFAErs0k/XV7tlP_vrMI/AAAAAAAAYGQ/NRfPfTwS_b44KxEda401RspHNvHCfSU8wCLcBGAs/s640/smmDataExplorer.png)](https://1.bp.blogspot.com/-4tpKFAErs0k/XV7tlP_vrMI/AAAAAAAAYGQ/NRfPfTwS_b44KxEda401RspHNvHCfSU8wCLcBGAs/s1600/smmDataExplorer.png)

  

Zoom into one message in a topic.
  

[![](https://1.bp.blogspot.com/-Ia0aa-Vj1xs/XV7tnGMHPYI/AAAAAAAAYGg/BzYpUZ0vfZkTk5u7ab4vn1VCUBYLKG9qACLcBGAs/s640/smmmessageDetails.png)](https://1.bp.blogspot.com/-Ia0aa-Vj1xs/XV7tnGMHPYI/AAAAAAAAYGg/BzYpUZ0vfZkTk5u7ab4vn1VCUBYLKG9qACLcBGAs/s1600/smmmessageDetails.png)
  

Let's analyze a topic's configuration.
  

[![](https://1.bp.blogspot.com/-MQzwNvGCWVU/XV7tmMZLDxI/AAAAAAAAYGY/oRA_AE5QOzE-zkUKnFd9bfDamJwoXP4VQCLcBGAs/s640/smmTopicDetails.png)](https://1.bp.blogspot.com/-MQzwNvGCWVU/XV7tmMZLDxI/AAAAAAAAYGY/oRA_AE5QOzE-zkUKnFd9bfDamJwoXP4VQCLcBGAs/s1600/smmTopicDetails.png)
  

  

The result of the **alert** we built is an email sent to me with this data:  
  
  
**Example Alert Sent**  
  
Notification id: 56d35dcc-8fc0-4c59-b70a-<wbr style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: small;"></wbr>ccbd1bb35681,  
Root resource name: nifi-reader,  
Root resource type: CONSUMER,  
Created timestamp: Thu Aug 22 18:42:41 UTC 2019 : 1566499361366,  
Last updated timestamp: Thu Aug 22 18:42:41 UTC 2019 : 1566499361366,   
State: RAISED,  
  
**Message:**  
Alert policy : "ALERT IF ( CONSUMER (name="nifi-reader") CONSUMER\_GROUP\_LAG \>= 100 )" has been evaluated to true Condition : "CONSUMER\_GROUP\_LAG\>=100" has been evaluated to true for following CONSUMERS - CONSUMER = "nifi-reader" had following attribute values \* CONSUMER\_GROUP\_LAG = 139  
  
  
  

#### Software
  

  

- CSP 2.1 
- CDH 6.3.0
- Cloudera Schema Registry 0.80
- CFM 1.0.1.0
- Apache NiFi Registry 0.3.0
- Apache NiFi 1.9.0.1.0
- JDK 1.8

  

  

  

#### **Resources**
  

- [https://www.cloudera.com/products/cdf/csp.html](https://www.cloudera.com/products/cdf/csp.html)
- [https://docs.hortonworks.com/HDPDocuments/CSP/CSP-1.0.0/index.html](https://docs.hortonworks.com/HDPDocuments/CSP/CSP-1.0.0/index.html)
- [https://docs.hortonworks.com/HDPDocuments/SMM/SMM-1.2.1/monitoring-kafka-clusters/content/smm-monitoring-clusters.html](https://docs.hortonworks.com/HDPDocuments/SMM/SMM-1.2.1/monitoring-kafka-clusters/content/smm-monitoring-clusters.html)