Sizing Your Apache NiFi Cluster For Production Workloads
Tags: 2020, apachenifi, apachenifi111
Published: 2020-07-21 18:06:31

---
title: Sizing Your Apache NiFi Cluster For Production Workloads
published: true
date: 2020-07-21 17:31:00 UTC
tags: 2020,apachenifi,apachenifi,apachenifi111
canonical_url: 
---

# Sizing Your Apache NiFi Cluster For Production Workloads

Cloudera Flow Management provides an enterprise edition of support Apache NiFi managed by Cloudera Manager.    The official documentation provides a great guide for sizing your cluster.

  

[https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-nifi-sizing.html](https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-nifi-sizing.html)
  

If the use case fits, NiFi Stateless Engine may fit and perform better utilizing no disk.

  

Check out that heap usage and utilization, you may need to increase.    24-32 Gigabytes of RAM is a nice sweet spot for most instances.

  

[![](https://1.bp.blogspot.com/-7EMiW641Qi0/Xxcgfa3KDsI/AAAAAAAAbZY/16AI1e2o5qUBHwC-fHEvJsgDNoj2A0nHACLcBGAsYHQ/w781-h126/NiFiCluster1.png)](https://1.bp.blogspot.com/-7EMiW641Qi0/Xxcgfa3KDsI/AAAAAAAAbZY/16AI1e2o5qUBHwC-fHEvJsgDNoj2A0nHACLcBGAsYHQ/s2723/NiFiCluster1.png)

  

  

Check out how your nodes, threads and queues are doing.   If queue is not processing fast or thread count is high, you may need more cores, RAM or nodes.
  

[![](https://1.bp.blogspot.com/-BlQMyCREPX8/XxcgfDmqnyI/AAAAAAAAbZQ/pOi9l7HAHGYR_k0N0YjVlvr1mpcxozEmACLcBGAsYHQ/w781-h116/nificluster2.png)](https://1.bp.blogspot.com/-BlQMyCREPX8/XxcgfDmqnyI/AAAAAAAAbZQ/pOi9l7HAHGYR_k0N0YjVlvr1mpcxozEmACLcBGAsYHQ/s2902/nificluster2.png)

  

  

When you are managing your cluster in Cloudera Manager, make sure you increase the default JVM memory for Apache NiFi.  512MB is not going to cut it for anything but single user development.
  

[![](https://1.bp.blogspot.com/-qJr00j2DlIw/XxcgfADdthI/AAAAAAAAbZU/wP5gaMmbf183AdYLeM_GCub8AZlH7LRYACLcBGAsYHQ/w625-h249/nificmdefault.png)](https://1.bp.blogspot.com/-qJr00j2DlIw/XxcgfADdthI/AAAAAAAAbZU/wP5gaMmbf183AdYLeM_GCub8AZlH7LRYACLcBGAsYHQ/s2131/nificmdefault.png)

  

  

Do this correctly and process a billion events!!!   [https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/](https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/).  - Notice the **hardware** and **performance** sections of that article

  

  

**General tips:**

  

Make sure you use SSD for Provenance and other repositories.  Faster disk, happier user. [https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-disk-configuration.html](https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-disk-configuration.html)

  

Monitor your flows to see how much resources you need:  [https://www.datainmotion.dev/2020/07/report-on-this-apache-nifi-1114-monitor.html](https://dev.to/tspannhw/report-on-this-apache-nifi-1-11-4-monitor-all-the-things-hn7-temp-slug-1298248).

  

Three or more nodes plus external Zookeepers.   [https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-cluster-layout.html](https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-cluster-layout.html)

  

Use Records, if it's semistructured GrokReader can help.   [https://www.nifi.rocks/record-path-cheat-sheet/](https://www.nifi.rocks/record-path-cheat-sheet/)  If it's CSV, JSON, XML, Parquet, Logs then use Readers and writers.   They are much faster, easier and cleaner.

  

Good clean code is better than spaghetti.   [https://docs.cloudera.com/cfm/2.0.1/nifi-tuning/topics/cfm-tuning-your-data-flow.html](https://docs.cloudera.com/cfm/2.0.1/nifi-tuning/topics/cfm-tuning-your-data-flow.html).  Saw no to bad coding:   [https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html](https://dev.to/tspannhw/no-more-spaghetti-flows-2emd). 

  

Recommendations:   [https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-recommendations.html](https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-recommendations.html)

  

Minimize use of CPU or Memory intensive processors (or make a not of them during sizing):   [https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-resource-intensive-processors.html](https://docs.cloudera.com/cfm/2.0.1/nifi-sizing/topics/cfm-sizing-resource-intensive-processors.html)

  

There are a few decisions to make on repositories, talk to your Cloudera friends.    [https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/configuration-best-practices.html](https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/configuration-best-practices.html)