FLANK Stack:   NiFi Processor for Kafka Consumption on Demand - REST Proxy Example
Tags: 2020, apacheflink, apachekafka, apachenifi
Published: 2020-01-22 23:03:08

---
title: FLANK Stack:   NiFi Processor for Kafka Consumption on Demand - REST Proxy Example
published: true
date: 2020-01-18 21:57:00 UTC
tags: 2020,apache-flink,apache-kafka,apache-nifi
canonical_url: 
---

### Writing a Custom Kafka Rest Proxy in 4 Hours
  
A custom processor for using NiFi as a REST Proxy to Kafka is very easy.  So I made one in NiFi 1.10.   It's a simple Kafka smart client that accepts POSTs, GETs or whatever HTTP request and returns a message from a Kafka topic, topic can be set via variables, HTTP request or your choice.   To get on of the query parameters, you do so like this:    **${http.query.param.topic}.   ** I am using the plain old KafkaConsumer class.    
  

[![](https://1.bp.blogspot.com/-TfaQjJN_tao/XiN48ZnJt9I/AAAAAAAAZRw/f5RIJRH3xI4TFfLhKjj5O0hHP3bcpAaxgCLcBGAsYHQ/s640/kafkarestide.png)](https://1.bp.blogspot.com/-TfaQjJN_tao/XiN48ZnJt9I/AAAAAAAAZRw/f5RIJRH3xI4TFfLhKjj5O0hHP3bcpAaxgCLcBGAsYHQ/s1600/kafkarestide.png)
  
  
If you need data, just CURL it or use your HTTP/REST client controls in Java, Python, Go, Scala, Ruby, C#, VB.NET or whatever.  
  
<style type="text/css">p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 18.0px Menlo; color: #ffffff; background-color: #224fbc} span.s1 {font-variant-ligatures: no-common-ligatures} </style>  

curl http://localhost:9089?topic=bme680
  
Download a pre-built NAR and install.   Note:   this is a pre-release Alpha that I quickly built and tested on a few clusters.   This is not an official project or product.   This is a POC for myself to see how hard it could it be.   It's not!   Roll your own or join me in building out an open source project for one.   What requirements do you have?   This worked for me, send a curl, get a message.  
  
  

[![](https://1.bp.blogspot.com/-AGhzh6ZLHSs/XiNpFwFNQ8I/AAAAAAAAZP4/WcxJK9-Y4x46fFulI4jXhHtUqmL8_WBeQCLcBGAsYHQ/s640/addKafkaRestProcessor.png)](https://1.bp.blogspot.com/-AGhzh6ZLHSs/XiNpFwFNQ8I/AAAAAAAAZP4/WcxJK9-Y4x46fFulI4jXhHtUqmL8_WBeQCLcBGAsYHQ/s1600/addKafkaRestProcessor.png)
  
Build a topic for Kafka with SMM in seconds  

[![](https://1.bp.blogspot.com/-FQtM2bcdmx4/XiNpFzA3EvI/AAAAAAAAZP0/6zIVX1RTcXMT5aQdngg8KH-0hDVBV4byQCLcBGAsYHQ/s640/addTopicGassSensorSMM.png)](https://1.bp.blogspot.com/-FQtM2bcdmx4/XiNpFzA3EvI/AAAAAAAAZP0/6zIVX1RTcXMT5aQdngg8KH-0hDVBV4byQCLcBGAsYHQ/s1600/addTopicGassSensorSMM.png)
  

  

 Here's an entire Kafka REST Proxy in a few steps.  
  
HandleHttpRequest (We could have hundreds of options here.  
RouteOnAttribute  
  

[![](https://1.bp.blogspot.com/-F90ryUWOPGI/XiNpGt9ohEI/AAAAAAAAZQE/ULqSDUbrRjIwGRMPSYX-thhJTZJRdfZMQCLcBGAsYHQ/s640/exampleKafkaRest.png)](https://1.bp.blogspot.com/-F90ryUWOPGI/XiNpGt9ohEI/AAAAAAAAZQE/ULqSDUbrRjIwGRMPSYX-thhJTZJRdfZMQCLcBGAsYHQ/s1600/exampleKafkaRest.png)
  
**Provenance For An Example Kafka REST Call**  

[![](https://1.bp.blogspot.com/-VncOdEZ0vJs/XiNpHVO-0UI/AAAAAAAAZQM/k6acNIMkHFU5-LGU7pjxxFIx6Xpv8H6wwCLcBGAsYHQ/s640/kafkaattributes.png)](https://1.bp.blogspot.com/-VncOdEZ0vJs/XiNpHVO-0UI/AAAAAAAAZQM/k6acNIMkHFU5-LGU7pjxxFIx6Xpv8H6wwCLcBGAsYHQ/s1600/kafkaattributes.png)
  
To use the processor, we need to set some variables for Kafka Broker, Topic, Offset Reset, # of Records to grab at a time, Client Id, Group Id - important for keeping your offset, auto commit, deserializer for your key and value types - String is usually good, maybe Byte.  
  

[![](https://1.bp.blogspot.com/-2_tXomsBNQA/XiNpHJjibJI/AAAAAAAAZQI/9EUHxLRMuLAb3JzGxHRllBs5MMidcNAAwCLcBGAsYHQ/s640/kafkaRestProcessor.png)](https://1.bp.blogspot.com/-2_tXomsBNQA/XiNpHJjibJI/AAAAAAAAZQI/9EUHxLRMuLAb3JzGxHRllBs5MMidcNAAwCLcBGAsYHQ/s1600/kafkaRestProcessor.png)
  
**Kafka to HBase is Easy.**  

[![](https://1.bp.blogspot.com/-jLMWl3RWXv4/XiNpIGfOcQI/AAAAAAAAZQQ/u57bChs5MeAbbJ65lrm77IFYk8sNXLrLgCLcBGAsYHQ/s640/kafkatoHbaseClouderaNow.png)](https://1.bp.blogspot.com/-jLMWl3RWXv4/XiNpIGfOcQI/AAAAAAAAZQQ/u57bChs5MeAbbJ65lrm77IFYk8sNXLrLgCLcBGAsYHQ/s1600/kafkatoHbaseClouderaNow.png)
  
**Kafka to Kudu is Easy.**  

[![](https://1.bp.blogspot.com/-aedghB2UK9U/XiNpF-XBruI/AAAAAAAAZP8/HCEaFGYM28QNuW16ZEmj57T_T4OuGcingCLcBGAsYHQ/s640/KafkaToKudu.png)](https://1.bp.blogspot.com/-aedghB2UK9U/XiNpF-XBruI/AAAAAAAAZP8/HCEaFGYM28QNuW16ZEmj57T_T4OuGcingCLcBGAsYHQ/s1600/KafkaToKudu.png)
  

[![](https://1.bp.blogspot.com/-5b66NEW_neY/XiNpNMMubSI/AAAAAAAAZRE/av70LZL15PAAY_HW_WAO7C8gLBc_wlEdACLcBGAsYHQ/s640/totalMessagesKafka.png)](https://1.bp.blogspot.com/-5b66NEW_neY/XiNpNMMubSI/AAAAAAAAZRE/av70LZL15PAAY_HW_WAO7C8gLBc_wlEdACLcBGAsYHQ/s1600/totalMessagesKafka.png)
  
  
  

### Kafka Proxy Processor For Message Consumption Source
  
[https://github.com/tspannhw/kafkarest-processor](https://github.com/tspannhw/kafkarest-processor)  
  
**Pre-Built NAR**  
  
[https://github.com/tspannhw/kafkarest-processor/releases/download/0.1/nifi-kafkarest-nar-1.0.nar](https://github.com/tspannhw/kafkarest-processor/releases/download/0.1/nifi-kafkarest-nar-1.0.nar)  

### Other Kafka Articles
  

- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257)
- [https://dev.to/tspannhw/using-cloudera-streams-messaging-manager-for-apache-kafka-monitoring-management-analytics-and-crud-41fb](https://dev.to/tspannhw/using-cloudera-streams-messaging-manager-for-apache-kafka-monitoring-management-analytics-and-crud-41fb)
- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-55gj](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-55gj)
- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-1npb](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-1npb)
- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-31d1](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-31d1)
- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-syslog-to-kafka-3a1k](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-syslog-to-kafka-3a1k)
- [https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-http-rest-sink-and-http-rest-source-to-kafka-sink-1257)
- [https://dev.to/tspannhw/using-grovepi-with-raspberry-pi-and-minifi-agents-for-data-ingest-to-parquet-kudu-orc-kafka-hive-and-impala-2h9k](https://dev.to/tspannhw/using-grovepi-with-raspberry-pi-and-minifi-agents-for-data-ingest-to-parquet-kudu-orc-kafka-hive-and-impala-2h9k)
- [https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221](https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221)
- [https://github.com/tspannhw/kafkaadmin-processor](https://github.com/tspannhw/kafkaadmin-processor)
- [https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html ](https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html%C2%A0)
  
It's so easy, didn't wake the cat.  
  

[![](https://1.bp.blogspot.com/-fRwiv5icSZQ/XiN1nk2K1iI/AAAAAAAAZRk/ZRpvA1YDxmgLIdRJTa4FDm0pkT94oA_iwCLcBGAsYHQ/s640/IMG-7191.jpg)](https://1.bp.blogspot.com/-fRwiv5icSZQ/XiN1nk2K1iI/AAAAAAAAZRk/ZRpvA1YDxmgLIdRJTa4FDm0pkT94oA_iwCLcBGAsYHQ/s1600/IMG-7191.jpg)