Migrating Apache Flume Flows to Apache NiFi: Twitter Source to Kafka Sink
Published: 2019-12-03 21:49:21

---
title: Migrating Apache Flume Flows to Apache NiFi:  Twitter Source to Kafka Sink
published: true
date: 2019-10-08 19:27:00 UTC
tags: 
canonical_url: 
---

### Migrating Apache Flume Flows to Apache NiFi:  Twitter Source to Kafka Sink

  
**Article 4 -  This.**

**Article 3 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_7.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-19o3-temp-slug-1137778)

**Article 2 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-3iec-temp-slug-5337136)

**Article 1**  - [https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html](https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html)

**Source Code:  ** [https://github.com/tspannhw/flume-to-nifi](https://github.com/tspannhw/flume-to-nifi)

  

This is a simple use case of pushing Tweets to Kafka.   We can do a lot more than that in NiFi.   If you see the referenced article I can do Deep Learning on Tweet Images, Run Sentiment Analysis, Query the Tweets in Stream, Send messages to email / slack based on certain criteria and retweet automagically.  All with No Code.  
  
It is extremely easy to do this in NiFi.  
  

[![](https://1.bp.blogspot.com/-1P04NDUSD-w/XZziSrNaYtI/AAAAAAAAYgs/Sjpf2NxmqFQvrfWtRZuSX6G9xGL59YFNgCLcBGAsYHQ/s640/twittertokafkadiagram2.png)](https://1.bp.blogspot.com/-1P04NDUSD-w/XZziSrNaYtI/AAAAAAAAYgs/Sjpf2NxmqFQvrfWtRZuSX6G9xGL59YFNgCLcBGAsYHQ/s1600/twittertokafkadiagram2.png)
  

[![](https://1.bp.blogspot.com/-E8N26-E4rJc/XZziSmvmR1I/AAAAAAAAYgo/JwMCP0PZbV0ja16azXgnMNM-UH3nseuwACLcBGAsYHQ/s640/twittertokafkadiagram.png)](https://1.bp.blogspot.com/-E8N26-E4rJc/XZziSmvmR1I/AAAAAAAAYgo/JwMCP0PZbV0ja16azXgnMNM-UH3nseuwACLcBGAsYHQ/s1600/twittertokafkadiagram.png)
  

  

  

### Create a Kafka Topic To Send Tweets To

[![](https://1.bp.blogspot.com/-k9g8AF8hjOw/XZzbLlaG4aI/AAAAAAAAYgA/81B3wS5tgrACDb_XAictZBIYN8Riypk9ACLcBGAsYHQ/s640/createTopicForTweets.png)](https://1.bp.blogspot.com/-k9g8AF8hjOw/XZzbLlaG4aI/AAAAAAAAYgA/81B3wS5tgrACDb_XAictZBIYN8Riypk9ACLcBGAsYHQ/s1600/createTopicForTweets.png)

  

### Example Tweet

[![](https://1.bp.blogspot.com/-06uOFVi_4LQ/XZzbMJWihMI/AAAAAAAAYgE/knmAZSJOdoMZ8Xp7XILj_4n3zeerVoThACLcBGAsYHQ/s640/exampleTweetinKafkaTopic.png)](https://1.bp.blogspot.com/-06uOFVi_4LQ/XZzbMJWihMI/AAAAAAAAYgE/knmAZSJOdoMZ8Xp7XILj_4n3zeerVoThACLcBGAsYHQ/s1600/exampleTweetinKafkaTopic.png)

  

  

### Configure Your Connection to Twitter

[![](https://1.bp.blogspot.com/-jlp_i-ZcKiY/XZziT1RAVjI/AAAAAAAAYgw/CtxF_xfYwMEuv74WDFi96Ekb_GFmzpGxACEwYBhgL/s640/TwitterSettings.png)](https://1.bp.blogspot.com/-jlp_i-ZcKiY/XZziT1RAVjI/AAAAAAAAYgw/CtxF_xfYwMEuv74WDFi96Ekb_GFmzpGxACEwYBhgL/s1600/TwitterSettings.png)

  

  

### Configure Kafka Producer

[![](https://1.bp.blogspot.com/-dManY7XaEzs/XZziT55LRDI/AAAAAAAAYg0/3xMrUAU4Nege3uQ8z736sUdjfOTPhLWigCEwYBhgL/s640/KafkaSettings.png)](https://1.bp.blogspot.com/-dManY7XaEzs/XZziT55LRDI/AAAAAAAAYg0/3xMrUAU4Nege3uQ8z736sUdjfOTPhLWigCEwYBhgL/s1600/KafkaSettings.png)
  

  

### NiFi Flow to Send Two Twitter Sources To Kafka

[![](https://1.bp.blogspot.com/-CYpS-TgtZdg/XZzbLiF5ZwI/AAAAAAAAYf8/jCCNPBBbiRg5eCeHx4V63VZ18oZKjA2tgCLcBGAsYHQ/s640/TwitterToKafkaFlow.png)](https://1.bp.blogspot.com/-CYpS-TgtZdg/XZzbLiF5ZwI/AAAAAAAAYf8/jCCNPBBbiRg5eCeHx4V63VZ18oZKjA2tgCLcBGAsYHQ/s1600/TwitterToKafkaFlow.png)

  

### Additional Apache NiFi Flow Details For Other Processing

[![](https://1.bp.blogspot.com/-uK-pgw6pd0Y/XZzbKpjgg5I/AAAAAAAAYfw/ZcMoGc7QVPY6DhZ8wFOjVgyz2Xfd2D3CwCLcBGAsYHQ/s640/RouteOnMediaVsText.png)](https://1.bp.blogspot.com/-uK-pgw6pd0Y/XZzbKpjgg5I/AAAAAAAAYfw/ZcMoGc7QVPY6DhZ8wFOjVgyz2Xfd2D3CwCLcBGAsYHQ/s1600/RouteOnMediaVsText.png)
  

[![](https://1.bp.blogspot.com/-LwnYY2q-NhM/XZzbKlYvg4I/AAAAAAAAYfo/zM9udSxaKHAkA6olfK9CEUSRM9b0IcplQCLcBGAsYHQ/s640/RunSentiment.png)](https://1.bp.blogspot.com/-LwnYY2q-NhM/XZzbKlYvg4I/AAAAAAAAYfo/zM9udSxaKHAkA6olfK9CEUSRM9b0IcplQCLcBGAsYHQ/s1600/RunSentiment.png)
  

[![](https://1.bp.blogspot.com/--Au5jAJa06Y/XZzbKhDn8LI/AAAAAAAAYfs/1LuDDpllZj4294f8Vme7wUBcwk3t_jzMQCLcBGAsYHQ/s640/SendTweets.png)](https://1.bp.blogspot.com/--Au5jAJa06Y/XZzbKhDn8LI/AAAAAAAAYfs/1LuDDpllZj4294f8Vme7wUBcwk3t_jzMQCLcBGAsYHQ/s1600/SendTweets.png)
  

[![](https://1.bp.blogspot.com/-LGxLiTwwM1M/XZzbLCfeaJI/AAAAAAAAYf0/k0HCRdg5h5IJVoJtihEEny7VmgvDmrF1gCLcBGAsYHQ/s640/TwitterNotifications.png)](https://1.bp.blogspot.com/-LGxLiTwwM1M/XZzbLCfeaJI/AAAAAAAAYf0/k0HCRdg5h5IJVoJtihEEny7VmgvDmrF1gCLcBGAsYHQ/s1600/TwitterNotifications.png)
  

[![](https://1.bp.blogspot.com/-IhoDDYeb6t4/XZzbLZ8u4NI/AAAAAAAAYf4/NVw1SQHLdVgPMABMhaT00hLUDJPs4Q43QCLcBGAsYHQ/s640/TwitterOverview.png)](https://1.bp.blogspot.com/-IhoDDYeb6t4/XZzbLZ8u4NI/AAAAAAAAYf4/NVw1SQHLdVgPMABMhaT00hLUDJPs4Q43QCLcBGAsYHQ/s1600/TwitterOverview.png)
  
  

### Kafka Messages in Topic Sent From NiFi Producer 

[![](https://1.bp.blogspot.com/-TulQmIj-Oes/XZzbMfJ8E3I/AAAAAAAAYgI/I_OwPQkEe4w4bViAQ95hP_7WcsEmsuB7wCLcBGAsYHQ/s640/viewKafkaMessagesSMM.png)](https://1.bp.blogspot.com/-TulQmIj-Oes/XZzbMfJ8E3I/AAAAAAAAYgI/I_OwPQkEe4w4bViAQ95hP_7WcsEmsuB7wCLcBGAsYHQ/s1600/viewKafkaMessagesSMM.png)
  

  

### Source Code

[https://github.com/tspannhw/flume-to-nifi](https://github.com/tspannhw/flume-to-nifi)

  

### References

- [https://community.cloudera.com/t5/Community-Articles/Automating-Social-Media-Sending-Tweets-with-Apache-NiFi/ta-p/248274](https://community.cloudera.com/t5/Community-Articles/Automating-Social-Media-Sending-Tweets-with-Apache-NiFi/ta-p/248274)