Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink
Tags: apachekafka, apachenifi, apachenifi19, cloudera
Published: 2019-12-03 21:48:37

---
title: Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink
published: true
date: 2019-10-08 20:49:00 UTC
tags: apache-kafka,apache-nifi,apache-nifi-1.9,cloudera
canonical_url: 
---

### Migrating Apache Flume Flows to Apache NiFi:  Kafka Source to HTTP REST Sink and HTTP REST Source to Kafka Sink

  
**Article 6 - This.**  
**Article 5 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_35.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-syslog-to-kafka-2g4-temp-slug-8252770)  
**Article 4 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_8.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-2f5o-temp-slug-2494436)

**Article 3 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_7.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-19o3-temp-slug-1137778)

**Article 2 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-3iec-temp-slug-5337136)

**Article 1**  - [https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html](https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html)

**Source Code:  ** [https://github.com/tspannhw/flume-to-nifi](https://github.com/tspannhw/flume-to-nifi)

  

This is a simple use case of being a gateway between REST API and Kafka.   We can do a lot more than that in NiFi.  We can be a Kafka Consumer and Producer as well as POST REST calls and receive any REST calls on configurable ports.  All with No Code.  
  
NiFi can act as a listener for HTTP Requests and provide HTTP Responses in a scriptable full Web Server mechanism with JETTY.   Or it can listen for HTTP REST calls on a port and route those files anywhere. [https://community.cloudera.com/t5/Community-Articles/Parsing-Web-Pages-for-Images-with-Apache-NiFi/ta-p/248415](https://community.cloudera.com/t5/Community-Articles/Parsing-Web-Pages-for-Images-with-Apache-NiFi/ta-p/248415) .  We can also do websockets [https://community.cloudera.com/t5/Community-Articles/An-Example-WebSocket-Application-in-Apache-NiFi-1-1/ta-p/248598](https://community.cloudera.com/t5/Community-Articles/An-Example-WebSocket-Application-in-Apache-NiFi-1-1/ta-p/248598).  [https://community.cloudera.com/t5/Community-Articles/Accessing-Feeds-from-EtherDelta-on-Trades-Funds-Buys-and/ta-p/248316](https://community.cloudera.com/t5/Community-Articles/Accessing-Feeds-from-EtherDelta-on-Trades-Funds-Buys-and/ta-p/248316)  
  
It is extremely easy to do this in NiFi.  
  

[![](https://1.bp.blogspot.com/-hDjYboTL2uA/XZz2Lcuj2oI/AAAAAAAAYiM/FV2NdVVVqLQVhKs7n-v3qpNgOP7Oow0UQCLcBGAsYHQ/s640/kafkaToRest.png)](https://1.bp.blogspot.com/-hDjYboTL2uA/XZz2Lcuj2oI/AAAAAAAAYiM/FV2NdVVVqLQVhKs7n-v3qpNgOP7Oow0UQCLcBGAsYHQ/s1600/kafkaToRest.png)
  

[![](https://1.bp.blogspot.com/-FpbHPmeFf9E/XZz2LWi7eZI/AAAAAAAAYiI/nveKhl806LUKHFcVX6TmztswEriubsSSQCLcBGAsYHQ/s640/resttokafka.png)](https://1.bp.blogspot.com/-FpbHPmeFf9E/XZz2LWi7eZI/AAAAAAAAYiI/nveKhl806LUKHFcVX6TmztswEriubsSSQCLcBGAsYHQ/s1600/resttokafka.png)
  
  

### Kafka Consumer to REST POST
[![](https://1.bp.blogspot.com/-34frfWJh-lc/XZzsXSqW_NI/AAAAAAAAYhc/DoplzBBHLgUAnDyzc-MNlDzGOlk7SjrBQCLcBGAsYHQ/s640/kafkatorest.png)](https://1.bp.blogspot.com/-34frfWJh-lc/XZzsXSqW_NI/AAAAAAAAYhc/DoplzBBHLgUAnDyzc-MNlDzGOlk7SjrBQCLcBGAsYHQ/s1600/kafkatorest.png)  
  

### HTTP REST to Kafka Producer
[![](https://1.bp.blogspot.com/-duh9xG2ia0I/XZzsV_RtdGI/AAAAAAAAYhM/GC8e8T5rvDoj-v7kQAP9xFBmnR4AVbHrQCLcBGAsYHQ/s640/HTTPtoKafka.png)](https://1.bp.blogspot.com/-duh9xG2ia0I/XZzsV_RtdGI/AAAAAAAAYhM/GC8e8T5rvDoj-v7kQAP9xFBmnR4AVbHrQCLcBGAsYHQ/s1600/HTTPtoKafka.png)  
  
  

### Full Monitoring on Apache NiFi
  

[![](https://1.bp.blogspot.com/-Fj5VBrO6qB4/XZzsVmIEb0I/AAAAAAAAYhI/Pqkc_ZNQ2HEn7JlxU-mYLnryI5UG-OgigCLcBGAsYHQ/s640/cmcfmdiskthroughput.png)](https://1.bp.blogspot.com/-Fj5VBrO6qB4/XZzsVmIEb0I/AAAAAAAAYhI/Pqkc_ZNQ2HEn7JlxU-mYLnryI5UG-OgigCLcBGAsYHQ/s1600/cmcfmdiskthroughput.png)
  

[![](https://1.bp.blogspot.com/-qUsPcx9C74s/XZzsV9qKPRI/AAAAAAAAYhQ/7BHZvZBSkRoJij8MVzDav9xrisDV8LwXQCLcBGAsYHQ/s640/cmCFMProcesses.png)](https://1.bp.blogspot.com/-qUsPcx9C74s/XZzsV9qKPRI/AAAAAAAAYhQ/7BHZvZBSkRoJij8MVzDav9xrisDV8LwXQCLcBGAsYHQ/s1600/cmCFMProcesses.png)
  

[![](https://1.bp.blogspot.com/-jXuaGrHLqfs/XZzsWEHZCgI/AAAAAAAAYhU/jwEILDu04zM2xC5kD22um_zhoFZieEJ2gCLcBGAsYHQ/s640/cmhostcpuusage.png)](https://1.bp.blogspot.com/-jXuaGrHLqfs/XZzsWEHZCgI/AAAAAAAAYhU/jwEILDu04zM2xC5kD22um_zhoFZieEJ2gCLcBGAsYHQ/s1600/cmhostcpuusage.png)
  

[![](https://1.bp.blogspot.com/-ZM9upIrgE_8/XZzsWjcBc6I/AAAAAAAAYhY/Qt7YAOcOetUm-doZrODN_siH7uIVJOaQgCLcBGAsYHQ/s640/cmnifilogs.png)](https://1.bp.blogspot.com/-ZM9upIrgE_8/XZzsWjcBc6I/AAAAAAAAYhY/Qt7YAOcOetUm-doZrODN_siH7uIVJOaQgCLcBGAsYHQ/s1600/cmnifilogs.png)
  

### A Very Common Use Case:  Ingesting Stock Feeds From REST to Kafka

  

[https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221](https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221)

  

### References

- [https://www.datainmotion.dev/2019/08/creating-apache-kafka-topics.html](https://www.datainmotion.dev/2019/08/creating-apache-kafka-topics.html)
- [https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221](https://community.cloudera.com/t5/Community-Articles/Real-Time-Stock-Processing-With-Apache-NiFi-and-Apache-Kafka/ta-p/249221)
- [https://community.cloudera.com/t5/Community-Articles/IoT-Edge-Use-Cases-with-Apache-Kafka-and-Apache-NiFi-MiniFi/ta-p/249232](https://community.cloudera.com/t5/Community-Articles/IoT-Edge-Use-Cases-with-Apache-Kafka-and-Apache-NiFi-MiniFi/ta-p/249232)
- [https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/](https://dataworkssummit.com/nosql-day-2019/session/tracking-crime-as-it-occurs-with-apache-phoenix-apache-hbase-and-apache-nifi/)
- [https://community.cloudera.com/t5/Community-Articles/NiFi-REST-API-FlowFile-Count-Monitoring/ta-p/244208](https://community.cloudera.com/t5/Community-Articles/NiFi-REST-API-FlowFile-Count-Monitoring/ta-p/244208)
- [https://dzone.com/articles/simple-apache-nifi-operations-dashboard-part-2-spr](https://dzone.com/articles/simple-apache-nifi-operations-dashboard-part-2-spr)
- [https://dzone.com/articles/building-a-custom-apache-nifi-operations-dashboard](https://dzone.com/articles/building-a-custom-apache-nifi-operations-dashboard)
- [https://www.progress.com/tutorials/jdbc/connect-and-import-data-from-any-rest-api-in-apache-nifi](https://www.progress.com/tutorials/jdbc/connect-and-import-data-from-any-rest-api-in-apache-nifi)