Building Search Indexes with Apache NiFi Streams
Tags: apachenifi, apachesolr, clouderasearch
Published: 2020-04-27 12:42:01

---
title: Building Search Indexes with Apache NiFi Streams
published: true
date: 2020-04-22 14:48:00 UTC
tags: apache-nifi,apache-solr,cloudera-search
canonical_url: 
---

## Building Search Indexes with Apache NiFi Streams

  

  

I can go to my Apache Solr dashboard hosted on Cloudera Data Platform - Data Center via port :8983/solr/#/

  

For those who have done the CFM or CDF Hands on Labs, we extend that lab to store to SOLR.

  

**Login to Solr**

  

[![](https://1.bp.blogspot.com/-ydvOFIoMCRI/XqBU57641TI/AAAAAAAAZtU/E1WQukHtLIQA1195VWCpsIhPd1-Pqom1ACLcBGAsYHQ/s640/solrstats.png)](https://1.bp.blogspot.com/-ydvOFIoMCRI/XqBU57641TI/AAAAAAAAZtU/E1WQukHtLIQA1195VWCpsIhPd1-Pqom1ACLcBGAsYHQ/s1600/solrstats.png)

  

[![](https://1.bp.blogspot.com/-zyCVNM9izac/XqBU5nz9mGI/AAAAAAAAZtQ/WoQwDUaAl7A3B0HYZFyDz7ZXIxvRkrU7ACLcBGAsYHQ/s640/solrdashboard.png)](https://1.bp.blogspot.com/-zyCVNM9izac/XqBU5nz9mGI/AAAAAAAAZtQ/WoQwDUaAl7A3B0HYZFyDz7ZXIxvRkrU7ACLcBGAsYHQ/s1600/solrdashboard.png)

  

  

**Create a Collection for Your Data (sensors)**

  

[![](https://1.bp.blogspot.com/-fyl8Xjs1Or0/XqBU5DVHJWI/AAAAAAAAZtE/HhaPZurKH3kRbrxQU9JU_qzFHs5ZbTRbwCLcBGAsYHQ/s640/addSolrCollection.png)](https://1.bp.blogspot.com/-fyl8Xjs1Or0/XqBU5DVHJWI/AAAAAAAAZtE/HhaPZurKH3kRbrxQU9JU_qzFHs5ZbTRbwCLcBGAsYHQ/s1600/addSolrCollection.png)

  

[![](https://1.bp.blogspot.com/-CcIQlVLGHCI/XqBU5NbZYUI/AAAAAAAAZtI/NTNyj6z1iiQnZ9GGEBUNOx1_KYAJYm4bACLcBGAsYHQ/s640/solrcollections.png)](https://1.bp.blogspot.com/-CcIQlVLGHCI/XqBU5NbZYUI/AAAAAAAAZtI/NTNyj6z1iiQnZ9GGEBUNOx1_KYAJYm4bACLcBGAsYHQ/s1600/solrcollections.png)

  

  

**NiFi Flow (Data in, Output Records to Solr)**

  

[![](https://1.bp.blogspot.com/-fJIv0VSr6CM/XqBY_C6ubxI/AAAAAAAAZts/DKktyTPD29cWpa8KIGuEabCS9i95tatEwCLcBGAsYHQ/s640/putsolrrecordnififlow.png)](https://1.bp.blogspot.com/-fJIv0VSr6CM/XqBY_C6ubxI/AAAAAAAAZts/DKktyTPD29cWpa8KIGuEabCS9i95tatEwCLcBGAsYHQ/s1600/putsolrrecordnififlow.png)

  

  

We need to configure the processor.   Just set **Solr Type** to **Cloud,** set **Solr Location** to **URL:2181/solr**  and then pick your record reader ours is JSON.   You also need to select **Fields to Index**  in the data.  In our JSON file I pick the timestamp and Id.

  

[![](https://1.bp.blogspot.com/-35FY3tlHF3g/XqBY_G8IZLI/AAAAAAAAZtw/4FIBEKFWnQoog0rVuFpXa0SUpcsxa_FJwCLcBGAsYHQ/s640/configuresolr.png)](https://1.bp.blogspot.com/-35FY3tlHF3g/XqBY_G8IZLI/AAAAAAAAZtw/4FIBEKFWnQoog0rVuFpXa0SUpcsxa_FJwCLcBGAsYHQ/s1600/configuresolr.png)

  

  

**View The Collection To See Data Sent From NiFi**
  

[![](https://1.bp.blogspot.com/-Ka02qJVvsic/XqBU5NZrsMI/AAAAAAAAZtM/NO6ihVBhrng8w2JgcaK4QOSmvkK88rfnQCLcBGAsYHQ/s640/PushedToSolr.png)](https://1.bp.blogspot.com/-Ka02qJVvsic/XqBU5NZrsMI/AAAAAAAAZtM/NO6ihVBhrng8w2JgcaK4QOSmvkK88rfnQCLcBGAsYHQ/s1600/PushedToSolr.png)
  

  

  

  

** To Query Via REST API**  
  
[http://YOURURL:8983/solr/sensors/select?q=\*%3A\*](http://ec2-3-87-238-1.compute-1.amazonaws.com:8983/solr/sensors/select?q=*%3A*)  
  

  

**Example Collection Data**  
  
{ "responseHeader":{ "zkConnected":true, "status":0, "QTime":2, "params":{ "q":"\*:\*", "\_":"1587566383627"}}, "response":{"numFound":153,"start":0,"docs":[{ "sensor\_id":[73], "sensor\_ts":[1587566202642550], "id":"e8652da3-dddf-4735-a187-799f95d2e362", "\_version\_":1664683988343586816}, { "sensor\_id":[22], "sensor\_ts":[1587566203806855], "id":"9bae2dff-e5ec-4751-b20e-f4d3a9c6e894", "\_version\_":1664683988399161344}, { "sensor\_id":[10], "sensor\_ts":[1587566204972747], "id":"4cc9dd26-e559-4368-b0c2-47095a0f78ab", "\_version\_":1664683988407549952}, { "sensor\_id":[48], "sensor\_ts":[1587566206142079], "id":"ba73b5d7-4c06-4202-8145-0f3437f37759", "\_version\_":1664683988415938560}, { "sensor\_id":[9], "sensor\_ts":[1587566207304793], "id":"be85793a-1c3c-46ab-96d8-14057e6a2e27", "\_version\_":1664683988422230016}, { "sensor\_id":[77], "sensor\_ts":[1587566208471159], "id":"94a1a150-2836-4b60-aafd-9b09fe2ac62a", "\_version\_":1664683988428521472}, { "sensor\_id":[75], "sensor\_ts":[1587566209635840], "id":"729b488c-7a75-49a8-9121-3e192bd84a81", "\_version\_":1664683988434812928}, { "sensor\_id":[64], "sensor\_ts":[1587566210807450], "id":"e6f76296-f70a-430d-a532-8040fbf4139e", "\_version\_":1664683988440055808}, { "sensor\_id":[1], "sensor\_ts":[1587566211973630], "id":"a3cb1028-76d1-4dba-a2ef-03b267d6714b", "\_version\_":1664683988447395840}, { "sensor\_id":[33], "sensor\_ts":[1587566213135150], "id":"dea6ea44-28ab-4d6e-9fad-b0ffaef69dcb", "\_version\_":1664683988454735872}] }}  
  
**Example Code**  
  
[https://github.com/tspannhw/nifi-solr-example](https://github.com/tspannhw/nifi-solr-example)  
  
  
**References**  
  
[https://lucene.apache.org/solr/](https://lucene.apache.org/solr/)  
  
[https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search\_introducing.html](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search_introducing.html)  
  
[https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cm\_mc\_solr\_service.html](https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cm_mc_solr_service.html)  
  
[https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search\_tutorial.html](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/search_tutorial.html)  
  
[https://www.cloudera.com/campaign/how-tos-for-gurus/chapters/chapter-8/cloudera-search--solr--overview.html](https://www.cloudera.com/campaign/how-tos-for-gurus/chapters/chapter-8/cloudera-search--solr--overview.html)