Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway
Published: 2021-02-13 21:07:39

---
title: Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway
published: true
date: 2021-02-13 21:01:00 UTC
tags: 
canonical_url: 
---

## Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway

**What does it look like?**

[![](https://1.bp.blogspot.com/-H7hac9Z96JM/XIwavQPoUfI/AAAAAAAAXCk/-BEZxb1IPgEd5kyBDNmeavHvcGTXL6gZgCPcBGAYYCw/w640-h466/postimageprocessor.png)](https://1.bp.blogspot.com/-H7hac9Z96JM/XIwavQPoUfI/AAAAAAAAXCk/-BEZxb1IPgEd5kyBDNmeavHvcGTXL6gZgCPcBGAYYCw/s1661/postimageprocessor.png)
  
  

**Where Can I Run This Magic Engine:**

Private Cloud, Public Cloud, Hybrid Cloud, VM, Bare Metal, Single Node, Laptop, Raspberry Pi or anywhere you have a 1GB of RAM and some CPU is a good place to run a powerful graphical integration and dataflow engine.   You can also run MiNiFi C++ or Java agents if you want it even smaller.

**Sounds Too Powerful and Expensive:**

Apache NiFi is Open Source and can be run freely anywhere.

**For What Use Cases:**

Microservices, Images, Deep Learning and Machine Learning Models, Structured Data, Unstructured Data, NLP, Sentiment Analysis, Semistructured Data, Hive, Hadoop, MongoDB, ElasticSearch, SOLR, ETL/ELT, MySQL CDC, MySQL Insert/Update/Delete/Query, Hosting Unlimited REST Services, Interactive with Websockets, Ingesting Any REST API, Natively Converting JSON/XML/CSV/TSV/Logs/Avro/Parquet, Excel, PDF, Word Documents, Syslog, Kafka, JMS, MQTT, TCP/IP, UDP, FTP, sFTP, Files, Directories, Google Forms, Object Stores, NoSQL, Lookups, Hosting Web sites, Updates and live SQL on data streams.

[![](https://1.bp.blogspot.com/-NOebp98Dcgc/Xe7HsE3W0SI/AAAAAAAAY-E/aL5QSfzKoPgXRKQeWgnfnNCa620Pd2YMACPcBGAYYCw/w640-h498/hbaseProcessNiFiList.png)](https://1.bp.blogspot.com/-NOebp98Dcgc/Xe7HsE3W0SI/AAAAAAAAY-E/aL5QSfzKoPgXRKQeWgnfnNCa620Pd2YMACPcBGAYYCw/s1602/hbaseProcessNiFiList.png)
  

 MySQL/REST/MQTT/JMS/REST/Files/S3/Object Stores.   You also have an expert available on NiFi here.   https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache\_9.html. 

This makes these tasks much easier to develop, deploy, manage and control.  A single Data Engineer can now build, deploy and manage thousands of data streams in batch, microbatch and streams.

- [https://www.datainmotion.dev/2020/12/simple-change-data-capture-cdc-with-sql.html](https://dev.to/tspannhw/simple-change-data-capture-cdc-with-sql-selects-via-apache-nifi-flank-19m4)
- [https://www.datainmotion.dev/2021/01/flank-real-time-transit-information-for.html](https://dev.to/tspannhw/flank-real-time-transit-information-for-ny-nj-ct-transcom-5f7k)
- [https://www.datainmotion.dev/2021/01/flank-using-apache-kudu-as-cache-for.html](https://dev.to/tspannhw/flank-using-apache-kudu-as-a-cache-for-fda-updates-4knj)
- [https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html](https://dev.to/tspannhw/ingesting-websocket-data-for-live-stock-streams-with-cloudera-flow-management-powered-by-apache-nifi-4oom)
- [https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html?es\_id=1fb9486166](https://www.datainmotion.dev/2020/12/smart-stocks-with-flank-nifi-kafka.html?es_id=1fb9486166)
- [https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse](https://dzone.com/articles/lets-build-a-simple-ingest-to-cloud-data-warehouse)
- [https://dzone.com/articles/real-time-streaming-deep-learning-pipelines-with-d](https://dzone.com/articles/real-time-streaming-deep-learning-pipelines-with-d)
- [https://www.datainmotion.dev/2020/10/automating-building-migration-backup.html](https://dev.to/tspannhw/automating-the-building-migration-backup-restore-and-testing-of-streaming-applications-30d8)
- [https://www.datainmotion.dev/2020/10/tracking-satellites-with-apache-nifi.html](https://dev.to/tspannhw/tracking-satellites-with-apache-nifi-44o7)
- [https://www.datainmotion.dev/2020/07/ingesting-all-weather-data-with-apache.html](https://dev.to/tspannhw/ingesting-all-the-weather-data-with-apache-nifi-2ho4)
- [https://dev.to/tspannhw/ingesting-all-the-weather-data-with-apache-nifi-2ho4](https://dev.to/tspannhw/ingesting-all-the-weather-data-with-apache-nifi-2ho4)

[![](https://1.bp.blogspot.com/-FpbHPmeFf9E/XZz2LWi7eZI/AAAAAAAAYiQ/qKlSqZwrIEwHuMDZ02HQPZuilrjbkZ11wCPcBGAYYCw/w640-h330/resttokafka.png)](https://1.bp.blogspot.com/-FpbHPmeFf9E/XZz2LWi7eZI/AAAAAAAAYiQ/qKlSqZwrIEwHuMDZ02HQPZuilrjbkZ11wCPcBGAYYCw/s2048/resttokafka.png)
  

**How about version control?**

NiFi Registry provides easy to integrate version control with full REST API and can export your flows to a Git repository like Github.

- [https://www.datainmotion.dev/2019/11/nifi-toolkit-cli-for-nifi-110.html](https://dev.to/tspannhw/nifi-toolkit-cli-for-nifi-1-10-213h)
- [https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/](https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/)
- [https://dzone.com/articles/devops-for-apache-nifi-17-and-more](https://dzone.com/articles/devops-for-apache-nifi-17-and-more)
- [https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-Flow-Versioning-and-Automation/ta-p/247976](https://community.cloudera.com/t5/Community-Articles/Big-Data-DevOps-Apache-NiFi-Flow-Versioning-and-Automation/ta-p/247976)

[![](https://1.bp.blogspot.com/-vH1W_YgCs-E/XJ1RMopYQpI/AAAAAAAAXHk/Tu-9V4lAZjYOoHtdKETIxHDNpGdtpbOFACPcBGAYYCw/w640-h372/gardennififlowoverview.png)](https://1.bp.blogspot.com/-vH1W_YgCs-E/XJ1RMopYQpI/AAAAAAAAXHk/Tu-9V4lAZjYOoHtdKETIxHDNpGdtpbOFACPcBGAYYCw/s2048/gardennififlowoverview.png)
  
  

**DevOps?**

- [https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html](https://dev.to/tspannhw/automating-starting-services-in-apache-nifi-and-applying-parameters-5h4n)
- [https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html](https://dev.to/tspannhw/devops-working-with-parameter-contexts-in-apache-nifi-1-11-4-12pk) 
- [https://www.datainmotion.dev/2019/11/nifi-toolkit-cli-for-nifi-110.html](https://dev.to/tspannhw/nifi-toolkit-cli-for-nifi-1-10-213h)

**How about deployment?**

Apache NiFi can run anywhere!

You can run Apache NiFi on a single VM or localhost or laptop:

[https://nifi.apache.org/download.html](https://nifi.apache.org/download.html)

On **OpenShift**

[https://catalog.redhat.com/software/containers/cdt-common-rns/nifi/6026bb6c2937380b51711b73](https://catalog.redhat.com/software/containers/cdt-common-rns/nifi/6026bb6c2937380b51711b73)

[https://github.com/rromannissen/nifi-openshift](https://github.com/rromannissen/nifi-openshift)

Apache NiFi **Stateless** can run all in RAM, one event at a time like a Job or Function as a Service:

[https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment](https://github.com/SamHjelmfelt/OpenWhisk-YarnDeployment)

[https://github.com/apache/nifi/blob/7d76bcd5202a8680c952d3a19072087a971d0b69/nifi-nar-bundles/nifi-framework-bundle/nifi-stateless-bundle/README.md](https://github.com/apache/nifi/blob/7d76bcd5202a8680c952d3a19072087a971d0b69/nifi-nar-bundles/nifi-framework-bundle/nifi-stateless-bundle/README.md)

**Docker**

[https://hub.docker.com/r/apache/nifi](https://hub.docker.com/r/apache/nifi)

<iframe frameborder="0" height="270" src="https://youtube.com/embed/HxU73SAKlOI" width="480"></iframe>

**What If I don't like easy to use Web UIs?**

You can code everything with REST calls:

[https://nifi.apache.org/docs/nifi-docs/rest-api/index.html](https://nifi.apache.org/docs/nifi-docs/rest-api/index.html)

Okay, maybe not that low-level, what about a CLI?

You can run and install it here:  [https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi\_CLI](https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI). 

<iframe frameborder="0" height="270" src="https://youtube.com/embed/VYNRo0pbFtg" width="480"></iframe>

**Can I get more information:**

- [https://github.com/tspannhw/EverythingApacheNiFi](https://github.com/tspannhw/EverythingApacheNiFi)
- [https://nifi.apache.org/docs.html](https://nifi.apache.org/docs.html)
- [https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html](https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html)
- [https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html](https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html)
- [https://www.youtube.com/watch?v=RjWstt7nRVY](https://www.youtube.com/watch?v=RjWstt7nRVY)
- [https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html](https://nifi.apache.org/docs/nifi-docs/html/walkthroughs.html)
- 
- [https://nifi.apache.org/docs/nifi-docs/html/overview.html](https://nifi.apache.org/docs/nifi-docs/html/overview.html)
- [https://nifi.apache.org/docs/nifi-docs/html/getting-started.html](https://nifi.apache.org/docs/nifi-docs/html/getting-started.html)
- [https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html](https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html)
- [https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html](https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html)
- [https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html](https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html)
- [https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html](https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html)
- [https://nifi.apache.org/docs/nifi-docs/rest-api/index.html](https://nifi.apache.org/docs/nifi-docs/rest-api/index.html)
- [https://nifi.apache.org/docs/nifi-docs/html/user-guide.html](https://nifi.apache.org/docs/nifi-docs/html/user-guide.html)
- [https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html](https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html)
- [https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html](https://dev.to/tspannhw/no-more-spaghetti-flows-2emd)

**I need support:**

A partner of Amazon, Google, Microsoft, Oracle, IBM and thousands more, you can trust Cloudera for enterprise cloud hosting, support and development.  Cloudera has a majority of developers of Apache NiFi working on Open Source.

[https://www.cloudera.com/products/cdf.html](https://www.cloudera.com/products/cdf.html)

[![](https://lh3.googleusercontent.com/-PgX4fDeyvPk/YCg7iGYU0xI/AAAAAAAAczs/n3kDL1ZqPYYIhbrBXl0pnAhE4QFeLRodQCLcBGAsYHQ/w640-h607/image.png)](https://lh3.googleusercontent.com/-PgX4fDeyvPk/YCg7iGYU0xI/AAAAAAAAczs/n3kDL1ZqPYYIhbrBXl0pnAhE4QFeLRodQCLcBGAsYHQ/image.png)