Migrating Apache Flume Flows to Apache NiFi: Kafka Source to Apache Parquet on HDFS
Published: 2019-12-03 21:49:29

---
title: Migrating Apache Flume Flows to Apache NiFi: Kafka Source to Apache Parquet on HDFS
published: true
date: 2019-10-08 00:03:00 UTC
tags: 
canonical_url: 
---

### Migrating Apache Flume Flows to Apache NiFi: Kafka Source to Apache Parquet on HDFS

  

**Article 3 - This**

**Article 2 - ** [https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html](https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-kafka-source-to-hdfs-kudu-file-hive-3iec-temp-slug-5337136)

**Article 1**  - [https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html](https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html)

**Source Code:  ** [https://github.com/tspannhw/flume-to-nifi](https://github.com/tspannhw/flume-to-nifi)

  

This is one possible simple, fast replacement for "[Flafka](https://blog.cloudera.com/flafka-apache-flume-meets-apache-kafka-for-event-processing/)".   I can read any/all Kafka topics, route and transform them with SQL and store them in Apache ORC, Apache Avro, Apache Parquet, Apache Kudu, Apache HBase, JSON, CSV, XML or compressed files of many types in S3, Apache HDFS, File Systems or anywhere you want to stream this data in Real-time.   Also with a fast easy to use Web UI.   Everything you liked doing in Flume but now easier and with more Source and Sink options.

  

  

  

[![](https://1.bp.blogspot.com/-OpFBDcIOMoA/XZe-_p1-FGI/AAAAAAAAYco/7T9Ip16ijfo1Me7BrVJjgrd8-FdqUckRgCLcBGAsYHQ/s640/flumereplacementgraph.png)](https://1.bp.blogspot.com/-OpFBDcIOMoA/XZe-_p1-FGI/AAAAAAAAYco/7T9Ip16ijfo1Me7BrVJjgrd8-FdqUckRgCLcBGAsYHQ/s1600/flumereplacementgraph.png)

  

  

### [  
](https://1.bp.blogspot.com/-rrzR-xonOAg/XZvQp8wsUDI/AAAAAAAAYew/7LUVLuqb5hY1pkAPV9l8pU_vPOvHf640gCLcBGAsYHQ/s1600/createParquetTable.png)[  
](https://1.bp.blogspot.com/-rrzR-xonOAg/XZvQp8wsUDI/AAAAAAAAYew/7LUVLuqb5hY1pkAPV9l8pU_vPOvHf640gCLcBGAsYHQ/s1600/createParquetTable.png)Consume Kafka And Store to Apache Parquet

[![](https://1.bp.blogspot.com/-rO2s3zQqDiM/XZvQowvw28I/AAAAAAAAYek/bs00LIuyFRUcXp6bbRiDZyt57JCVi6fLQCLcBGAsYHQ/s640/KafkaToParquet.png)](https://1.bp.blogspot.com/-rO2s3zQqDiM/XZvQowvw28I/AAAAAAAAYek/bs00LIuyFRUcXp6bbRiDZyt57JCVi6fLQCLcBGAsYHQ/s1600/KafkaToParquet.png)
  

### Kafka to Kudu, ORC, AVRO and Parquet 

[![](https://1.bp.blogspot.com/-7mZOT9f6tzw/XZvQoxF_YNI/AAAAAAAAYeo/o3zqPVBD6C8m7_v0uh0GzzNBBP-EBRq8wCLcBGAsYHQ/s640/KafkaToParquetNiFiFlow.png)](https://1.bp.blogspot.com/-7mZOT9f6tzw/XZvQoxF_YNI/AAAAAAAAYeo/o3zqPVBD6C8m7_v0uh0GzzNBBP-EBRq8wCLcBGAsYHQ/s1600/KafkaToParquetNiFiFlow.png)

  

**With Apache 1.10 I can send those Parquet files anywhere not only HDFS.**
  

[![](https://1.bp.blogspot.com/-0WfUd0jcvaI/XZvQp54XyTI/AAAAAAAAYe0/l2bAEyojtekPBIBfAtxcQBZgJVfNh605gCLcBGAsYHQ/s640/kafkaToParquetRecords.png)](https://1.bp.blogspot.com/-0WfUd0jcvaI/XZvQp54XyTI/AAAAAAAAYe0/l2bAEyojtekPBIBfAtxcQBZgJVfNh605gCLcBGAsYHQ/s1600/kafkaToParquetRecords.png)
  

**JSON (or CSV or AVRO or ...) and Parquet Out**

[![](https://1.bp.blogspot.com/-JXLu8zBb6Rg/XZvQox7nM9I/AAAAAAAAYeg/dekugiN_MYYFXBcCtB7BkwRRzKOx7oFdwCLcBGAsYHQ/s640/MergeRecordJsontoParquet.png)](https://1.bp.blogspot.com/-JXLu8zBb6Rg/XZvQox7nM9I/AAAAAAAAYeg/dekugiN_MYYFXBcCtB7BkwRRzKOx7oFdwCLcBGAsYHQ/s1600/MergeRecordJsontoParquet.png)

  

**In Apache 1.10, Parquet has a dedicated reader and writer**
  

[![](https://1.bp.blogspot.com/-UajUJWs7hCI/XZvQqmnjh3I/AAAAAAAAYe4/Tsj3HBxPYmoK-8RJ3WJLQwWO0hAD-YwLACLcBGAsYHQ/s640/parquetReader.png)](https://1.bp.blogspot.com/-UajUJWs7hCI/XZvQqmnjh3I/AAAAAAAAYe4/Tsj3HBxPYmoK-8RJ3WJLQwWO0hAD-YwLACLcBGAsYHQ/s1600/parquetReader.png)

  

**Or I can use PutParquet**

  

[![](https://1.bp.blogspot.com/-USD7z9W5pyY/XZvQrtPuOfI/AAAAAAAAYfA/I0WIqD0QA90ape0Z0117TXdqq1xdIZecQCLcBGAsYHQ/s640/putparquet.png)](https://1.bp.blogspot.com/-USD7z9W5pyY/XZvQrtPuOfI/AAAAAAAAYfA/I0WIqD0QA90ape0Z0117TXdqq1xdIZecQCLcBGAsYHQ/s1600/putparquet.png)

  

  

**Create A Parquet Table and Query It**

### [![](https://1.bp.blogspot.com/-SdO_GQWzQQE/XZvQpSzqtnI/AAAAAAAAYes/K16KhYLrqDQbQoEe6CSFJmXeyWgH4-uGwCLcBGAsYHQ/s640/createParqueHueImpala.png)](https://1.bp.blogspot.com/-SdO_GQWzQQE/XZvQpSzqtnI/AAAAAAAAYes/K16KhYLrqDQbQoEe6CSFJmXeyWgH4-uGwCLcBGAsYHQ/s1600/createParqueHueImpala.png)[![](https://1.bp.blogspot.com/-i3pB2ImEoY0/XZvQsEv4pzI/AAAAAAAAYfE/HO8Sa8_daio7iK5h1hWRHHWUFJK-cXdDgCLcBGAsYHQ/s640/queryParquetFromImpalaHue.png)](https://1.bp.blogspot.com/-i3pB2ImEoY0/XZvQsEv4pzI/AAAAAAAAYfE/HO8Sa8_daio7iK5h1hWRHHWUFJK-cXdDgCLcBGAsYHQ/s1600/queryParquetFromImpalaHue.png)
  

[![](https://1.bp.blogspot.com/-sadDWPRuzus/XZvQrFpsVOI/AAAAAAAAYe8/mJYoGU1nN9U4fhadm5mx-wK9mfvLUKKIgCLcBGAsYHQ/s640/parquetSqlImpalaHue.png)](https://1.bp.blogspot.com/-sadDWPRuzus/XZvQrFpsVOI/AAAAAAAAYe8/mJYoGU1nN9U4fhadm5mx-wK9mfvLUKKIgCLcBGAsYHQ/s1600/parquetSqlImpalaHue.png)
  

  

  

  

  

  

  

### References

- [https://www.progress.com/tutorials/jdbc/ingest-salesforce-data-incrementally-into-hive-using-apache-nifi](https://www.progress.com/tutorials/jdbc/ingest-salesforce-data-incrementally-into-hive-using-apache-nifi)
- [https://community.cloudera.com/t5/Community-Articles/RDBMS-to-Hive-using-NiFi-small-medium-tables/ta-p/244677](https://community.cloudera.com/t5/Community-Articles/RDBMS-to-Hive-using-NiFi-small-medium-tables/ta-p/244677)
- [https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2018/ta-p/249363](https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2018/ta-p/249363)
- [https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2017/ta-p/247541](https://community.cloudera.com/t5/Community-Articles/My-Year-in-Review-2017/ta-p/247541)