Migrating Apache Flume Flows to Apache NiFi: SYSLOG to KAFKA
Published: 2019-12-03 21:49:12

---
title: Migrating Apache Flume Flows to Apache NiFi:  SYSLOG  to KAFKA
published: true
date: 2019-10-08 20:42:00 UTC
tags: 
canonical_url: 
---

### Migrating Apache Flume Flows to Apache NiFi:  SYSLOG  to KAFKA

  
**Article 5 - This**  
**Article 4 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_8.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-twitter-source-to-kafka-sink-2f5o-temp-slug-2494436)

**Article 3 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_7.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-apache-parquet-on-hdfs-19o3-temp-slug-1137778)

**Article 2 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-3iec-temp-slug-5337136)

**Article 1**  - [https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html](https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html)

**Source Code:  ** [https://github.com/tspannhw/flume-to-nifi](https://github.com/tspannhw/flume-to-nifi)  
  

[![](https://1.bp.blogspot.com/-uwZ7i5vlLMw/XZz1jTGytyI/AAAAAAAAYiA/cTRxkMWqaIQa0-sr0TdbrM3ZOG0y97LvgCLcBGAsYHQ/s640/flowkafka.png)](https://1.bp.blogspot.com/-uwZ7i5vlLMw/XZz1jTGytyI/AAAAAAAAYiA/cTRxkMWqaIQa0-sr0TdbrM3ZOG0y97LvgCLcBGAsYHQ/s1600/flowkafka.png)
  

  

[![](https://1.bp.blogspot.com/-y05_dozlSkA/XZz1InwM_tI/AAAAAAAAYh4/bRKmLgJKS1APccqbTjvpN95RQ6ltbhilgCLcBGAsYHQ/s640/listensyslog.png)](https://1.bp.blogspot.com/-y05_dozlSkA/XZz1InwM_tI/AAAAAAAAYh4/bRKmLgJKS1APccqbTjvpN95RQ6ltbhilgCLcBGAsYHQ/s1600/listensyslog.png)
  
  

This is a simple use case of being a smart gateway/proxy between SYSLOGand Kafka.   We can do a lot more than that in NiFi.  We can be a Kafka Consumer and Producer as well as read and parse all types of logs including SYSLOG.  We have a GrokReader for converting semistructured logs into manageable tabular style data with schemas.   Log-\>JSON/CSV/AVRO/PARQUET.  All with No Code.  
  
It is extremely easy to do this in NiFi.  
  
See:  Log Parsing:   [https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html](https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html)  
  

### Detailed Tutorials
[https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/](https://blog.davidvassallo.me/2018/09/19/apache-nifi-from-syslog-to-elasticsearch/)  
[https://community.cloudera.com/t5/Community-Articles/NiFi-Send-to-syslog/ta-p/248638](https://community.cloudera.com/t5/Community-Articles/NiFi-Send-to-syslog/ta-p/248638)  
[http://www.youritgoeslinux.com/impl/bigdata/nifi/syslog](http://www.youritgoeslinux.com/impl/bigdata/nifi/syslog)  

### Processors
[https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListenSyslog/](https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ListenSyslog/)  
  
[https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ParseSyslog/index.html](https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.ParseSyslog/index.html)  

  

  

  

### References

- [https://community.cloudera.com/t5/Community-Articles/Su-Su-Sussudio-Sudoers-Log-Parsing-with-Apache-NiFi/ta-p/249461](https://community.cloudera.com/t5/Community-Articles/Su-Su-Sussudio-Sudoers-Log-Parsing-with-Apache-NiFi/ta-p/249461)