Connecting Apache NiFi to Apache Atlas For Data Governance At Scale in Streaming
Tags: 2020, apacheatlas, apachekafka, apachenifi
Published: 2020-02-22 20:27:55

---
title: Connecting Apache NiFi to Apache Atlas For Data Governance At Scale in Streaming
published: true
date: 2020-02-21 19:32:00 UTC
tags: 2020,apache-atlas,apache-kafka,apache-nifi
canonical_url: 
---

##   
Connecting Apache NiFi to Apache Atlas For Data Governance At Scale in Streaming
  
Once connected you can see NiFi and Kafka flowing to Atlas.  
  
You must add **Atlas**  **Report** to **NiFi** cluster.  
  

[![](https://1.bp.blogspot.com/-yi0GgMz44XE/XlAuqLQkM_I/AAAAAAAAZcI/mLVCpAiT8IYpZizSKAu8j0vIT1cmosmZACLcBGAsYHQ/s640/ReportLineageToAtlasRunning.png)](https://1.bp.blogspot.com/-yi0GgMz44XE/XlAuqLQkM_I/AAAAAAAAZcI/mLVCpAiT8IYpZizSKAu8j0vIT1cmosmZACLcBGAsYHQ/s1600/ReportLineageToAtlasRunning.png)
  
  

**Add a ReportLineageToAtlas under Controller Settings / Reporting Tasks**

**You must add URL for Atlas, Authentication method and if basic, username/password.**
  
[![](https://1.bp.blogspot.com/-UqmPMIzfIyk/XlAuvPzohEI/AAAAAAAAZdI/LdwHu006F4M0josYlUGAUKcnOEgEk0ivgCLcBGAsYHQ/s640/reportLineageAtlasConfig1.png)](https://1.bp.blogspot.com/-UqmPMIzfIyk/XlAuvPzohEI/AAAAAAAAZdI/LdwHu006F4M0josYlUGAUKcnOEgEk0ivgCLcBGAsYHQ/s1600/reportLineageAtlasConfig1.png)  
  
  
  

**You need to set Atlas Configuration directory, NiFi URL to use, Lineage Strategy - Complete Path**
 ![](https://1.bp.blogspot.com/--d5_Q86j5aQ/XlAuvvkZ5dI/AAAAAAAAZdM/_56KILwjmiQjpj60jC9NQ-wE9DzBmfKWACLcBGAsYHQ/s640/reportLineageAtlasConfig2.png)  
  

**Another example with an AWS hosted NiFi and Atlas:**
  

[![](https://1.bp.blogspot.com/-noTNtzczH40/XlAv2n0J03I/AAAAAAAAZds/2hkPBiG5Ao0IDyXOLv1zE4ZpHgQEP-h5ACLcBGAsYHQ/s640/reportlineageatlasAWS.png)](https://1.bp.blogspot.com/-noTNtzczH40/XlAv2n0J03I/AAAAAAAAZds/2hkPBiG5Ao0IDyXOLv1zE4ZpHgQEP-h5ACLcBGAsYHQ/s1600/reportlineageatlasAWS.png)
  

[![](https://1.bp.blogspot.com/-pxfR63pqpKQ/XlAv2nGO0aI/AAAAAAAAZdw/jWwEAVvTeKAHNOkBAgJcXFZd4vPD5g37ACLcBGAsYHQ/s640/reportlineageAws2.png)](https://1.bp.blogspot.com/-pxfR63pqpKQ/XlAv2nGO0aI/AAAAAAAAZdw/jWwEAVvTeKAHNOkBAgJcXFZd4vPD5g37ACLcBGAsYHQ/s1600/reportlineageAws2.png)
  
  
**You can now see the lineage state:**  

  

[![](https://1.bp.blogspot.com/-GsxFMWBH388/XlAurMc9I1I/AAAAAAAAZcY/NYG5pBvsS_Q09qIQNzJ65aiszAzkDVN0QCLcBGAsYHQ/s640/atlaslineagestate.png)](https://1.bp.blogspot.com/-GsxFMWBH388/XlAurMc9I1I/AAAAAAAAZcY/NYG5pBvsS_Q09qIQNzJ65aiszAzkDVN0QCLcBGAsYHQ/s1600/atlaslineagestate.png)

  

  

**Configure Atlas to Be Enabled and Have Kafka**
  

[![](https://1.bp.blogspot.com/-AtVIxul_WvE/XlAurYVkhUI/AAAAAAAAZcc/3h1RjnT-Zl8qKhxXBPrmeOReNbOwa5kKQCLcBGAsYHQ/s640/cmAtlasConfig.png)](https://1.bp.blogspot.com/-AtVIxul_WvE/XlAurYVkhUI/AAAAAAAAZcc/3h1RjnT-Zl8qKhxXBPrmeOReNbOwa5kKQCLcBGAsYHQ/s1600/cmAtlasConfig.png)
  

**Have Atlas Service enabled in NiFi configuration**

  

[![](https://1.bp.blogspot.com/-JFi15UAh2Ew/XlAurgbuyiI/AAAAAAAAZcg/8UloJWGFIyYEz0tvnzUYi4haBuROZXYjwCLcBGAsYHQ/s640/cmNiFiAtlasConfiguration.png)](https://1.bp.blogspot.com/-JFi15UAh2Ew/XlAurgbuyiI/AAAAAAAAZcg/8UloJWGFIyYEz0tvnzUYi4haBuROZXYjwCLcBGAsYHQ/s1600/cmNiFiAtlasConfiguration.png)
  
  
**Example Configuration**  
  
You must have access to Atlas Application Properties.  
  
**/etc/atlas/conf**  
  

atlas-application.properties 

  

#Generated by Apache NiFi ReportLineageToAtlas ReportingTask at 2020-02-21T17:18:28.493Z

#Fri Feb 21 17:18:28 UTC 2020

atlas.kafka.bootstrap.servers=princeton0.field.hortonworks.com\:9092

atlas.enableTLS=false

atlas.kafka.client.id=ReportLineageToAtlas.687a48e2-0170-1000-0000-00000a0de4ea

atlas.cluster.name=Princeton0
<style type="text/css">p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 18.0px Menlo; color: #ffffff; background-color: #224fbc} span.s1 {font-variant-ligatures: no-common-ligatures} </style>  

atlas.kafka.security.protocol=PLAINTEXT
  
  

atlas-server.properties 

  

princeton0.field.hortonworks.com:atlas.authentication.method.kerberos=false

princeton0.field.hortonworks.com:atlas.enableTLS=false

princeton0.field.hortonworks.com:atlas.kafka.zookeeper.connection.timeout.ms=30000

princeton0.field.hortonworks.com:atlas.kafka.zookeeper.session.timeout.ms=60000

princeton0.field.hortonworks.com:atlas.kafka.zookeeper.sync.time.ms=20

princeton0.field.hortonworks.com:atlas.server.bind.address=0.0.0.0

princeton0.field.hortonworks.com:atlas.server.http.port=31000
<style type="text/css">p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 18.0px Menlo; color: #ffffff; background-color: #224fbc} span.s1 {font-variant-ligatures: no-common-ligatures} </style>  

princeton0.field.hortonworks.com:atlas.server.https.port=31443
  
  
**Running Atlas**  
  
[![](https://1.bp.blogspot.com/-4AMYt3v_vSk/XlAupjiPtZI/AAAAAAAAZcE/-zlwg1V54IUYQehxBohr-CaoNGgbiL2swCLcBGAsYHQ/s640/89d039dd-6a5a-1e18-bb78-321dc894022e%2540Princeton0.png)](https://1.bp.blogspot.com/-4AMYt3v_vSk/XlAupjiPtZI/AAAAAAAAZcE/-zlwg1V54IUYQehxBohr-CaoNGgbiL2swCLcBGAsYHQ/s1600/89d039dd-6a5a-1e18-bb78-321dc894022e%2540Princeton0.png)  
  
[![](https://1.bp.blogspot.com/-uA_NHspAThg/XlAur17LzjI/AAAAAAAAZck/HoN7aqjsiuI5b3JIa-KLM0TBTn1uhbJyACLcBGAsYHQ/s640/coral%2540Princeton0.png)](https://1.bp.blogspot.com/-uA_NHspAThg/XlAur17LzjI/AAAAAAAAZck/HoN7aqjsiuI5b3JIa-KLM0TBTn1uhbJyACLcBGAsYHQ/s1600/coral%2540Princeton0.png)  
  
 ![](https://1.bp.blogspot.com/-dMU5xK2pDFc/Xks20bHoHdI/AAAAAAAAZXw/NPDdIBtjevscTnvRredxNLwr7oA2IjarQCLcBGAsYHQ/s640/atlasCemInputport.png)  
  
 ![](https://1.bp.blogspot.com/-Tf7oFSUsnxQ/Xks20dS4ULI/AAAAAAAAZXs/VRN8ea3WWRselkBvW8mkzKcjAVdKzciEgCLcBGAsYHQ/s640/atlasKafkaNiFi.png)  
  
 ![](https://1.bp.blogspot.com/-YDx8-DfdTPc/Xks4XCixRDI/AAAAAAAAZYA/g1DnxPoKYf8t-TJ48czfIRc-j3H-y4WbQCLcBGAsYHQ/s640/atlasCemInputport.png)  
  
 ![](https://1.bp.blogspot.com/-RMZQKQ1T6w4/Xks4XHs3-TI/AAAAAAAAZYI/C5NhnCDRkAQYZJhZUrzRwwFNv3QdFzLNgCLcBGAsYHQ/s640/atlasConsumeKafkaRecord.png)  
  
 ![](https://1.bp.blogspot.com/-Mr1AB-pff44/Xks4XP57krI/AAAAAAAAZYE/C1XIToBAiRIwHxHh1_z4v3P-mnPZAJmBwCLcBGAsYHQ/s640/atlasKafkaNiFi.png)  
  
 ![](https://1.bp.blogspot.com/-GvNPtZ2WMzo/Xks4XrSAwEI/AAAAAAAAZYM/W2wnSENFZIQ_L0m9k1_4qkZ9OObA-CoyACLcBGAsYHQ/s640/atlasNiFiKafkaFlow.png)  
  
  
  
  
  

### See:   

- [https://atlas.apache.org/#/](https://atlas.apache.org/#/)
- [https://medium.com/@kandalkarbhushan/integrating-nifi-with-atlas-in-secured-environment-c1b0f3cb7318](https://medium.com/@kandalkarbhushan/integrating-nifi-with-atlas-in-secured-environment-c1b0f3cb7318)
- [https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.1/bk\_installing-hdf-and-hdp/content/nifi-atlas.html](https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-and-hdp/content/nifi-atlas.html)
- [https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf-and-hdp-ppc/content/configure\_nifi\_for\_atlas\_integration.html](https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/installing-hdf-and-hdp-ppc/content/configure_nifi_for_atlas_integration.html)
- [https://docs.cloudera.com/runtime/7.0.2/cdp-governance-overview/topics/atlas-overview.html](https://docs.cloudera.com/runtime/7.0.2/cdp-governance-overview/topics/atlas-overview.html)
- [https://www.datainmotion.dev/2020/02/apache-atlas-for-monitoring-edge2ai-iot.html](https://dev.to/tspannhw/apache-atlas-for-monitoring-edge2ai-iot-flows-41pe-temp-slug-6402251)
- [https://docs.cloudera.com/cloudera-manager/7.0.3/reference/topics/cm\_props\_cr703\_atlas.html](https://docs.cloudera.com/cloudera-manager/7.0.3/reference/topics/cm_props_cr703_atlas.html)