Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi
Tags: agent, apachekafka, apachenifi, apachenifi19
Published: 2019-12-23 14:15:52

---
title: Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi
published: true
date: 2019-08-23 19:23:00 UTC
tags: agent,apache-kafka,apache-nifi,apache-nifi-1.9
canonical_url: 
---

#### Updating Machine Learning Models At The Edge With Apache NiFi and MiNiFi

Yes, we have bidirectional communication with MiNiFi agents from Apache NiFi via Site-to-Site (S2S) over HTTPS.   This means I can push in anything I want to the agent, including commands, files and updates.

  

I can also transmit data to edge agents via MQTT, REST and Kafka amongst other options.

  

[![](https://1.bp.blogspot.com/-BWOud_4dNQI/XWAuIxFJ09I/AAAAAAAAYH0/4JeMBAGVbBAIsPCdKK8UhvF7sff7UnWRwCLcBGAs/s640/cemMiNiFiJavaBiDirectional.png)](https://1.bp.blogspot.com/-BWOud_4dNQI/XWAuIxFJ09I/AAAAAAAAYH0/4JeMBAGVbBAIsPCdKK8UhvF7sff7UnWRwCLcBGAs/s1600/cemMiNiFiJavaBiDirectional.png)

  

NiFi Ready To Send and Receive Messages From Other NiFi Nodes, Clusters and MiNiFi Agents

  

[![](https://1.bp.blogspot.com/-1ldX2dghWEE/XWAuJ_Gx9oI/AAAAAAAAYIA/iH8ANjS-BgwNnWNCT1EgVnybZGNM5ye3ACLcBGAs/s640/nifibidirectional.png)](https://1.bp.blogspot.com/-1ldX2dghWEE/XWAuJ_Gx9oI/AAAAAAAAYIA/iH8ANjS-BgwNnWNCT1EgVnybZGNM5ye3ACLcBGAs/s1600/nifibidirectional.png)

  

Our NiFi flow is consuming Kafka and MQTT Messages, as well as reading updated model files and generating integration test sensor data.

  

[![](https://1.bp.blogspot.com/-2A1NNMQ9DTI/XWAuIy9DGGI/AAAAAAAAYH4/WAGcczYdbn0O74PT9ZvJtBQEm_tSGPtqgCLcBGAs/s640/bidirectionalnififlow.png)](https://1.bp.blogspot.com/-2A1NNMQ9DTI/XWAuIy9DGGI/AAAAAAAAYH4/WAGcczYdbn0O74PT9ZvJtBQEm_tSGPtqgCLcBGAs/s1600/bidirectionalnififlow.png)
  
  

MiNiFi Agents Have Downloaded The Model and Anything Else We Send to It
  

[![](https://1.bp.blogspot.com/-UkAtwfCV7TE/XWAuJilWiTI/AAAAAAAAYIE/5nOAoe4bHNoX1Dbzkf14H-t6cvq1WP2PACLcBGAs/s640/filescem2.png)](https://1.bp.blogspot.com/-UkAtwfCV7TE/XWAuJilWiTI/AAAAAAAAYIE/5nOAoe4bHNoX1Dbzkf14H-t6cvq1WP2PACLcBGAs/s1600/filescem2.png)
  

[![](https://1.bp.blogspot.com/-zmt1YJ2CeAE/XWAuJzmWQfI/AAAAAAAAYII/EDqlGUl8a-cEEHjjrZAys140ZyEH2KlQwCLcBGAs/s640/fileswrittencem.png)](https://1.bp.blogspot.com/-zmt1YJ2CeAE/XWAuJzmWQfI/AAAAAAAAYII/EDqlGUl8a-cEEHjjrZAys140ZyEH2KlQwCLcBGAs/s1600/fileswrittencem.png)
  

It's Easy to Configure MQTT Message Consumption in CEM, we just need the broker (with port) and a topic to filter on if you wish.
  

[![](https://1.bp.blogspot.com/-afbEldLSS5g/XWA3oNGUMlI/AAAAAAAAYI4/Ou-LqLcgdJoUhcWiatcbK32dfBU1K3vSgCLcBGAs/s640/cemConsumeMQTT1.png)](https://1.bp.blogspot.com/-afbEldLSS5g/XWA3oNGUMlI/AAAAAAAAYI4/Ou-LqLcgdJoUhcWiatcbK32dfBU1K3vSgCLcBGAs/s1600/cemConsumeMQTT1.png)
  

[![](https://1.bp.blogspot.com/-5IO0Ek8OIFQ/XWA3oJKIQpI/AAAAAAAAYIw/nGLjZROoiO8vv5jcX3diQdA53KBZha0GQCLcBGAs/s640/cemConsumeMQTT.png)](https://1.bp.blogspot.com/-5IO0Ek8OIFQ/XWA3oJKIQpI/AAAAAAAAYIw/nGLjZROoiO8vv5jcX3diQdA53KBZha0GQCLcBGAs/s1600/cemConsumeMQTT.png)

  

  

To Listen For Files/Models You can easily add a REST End Point to Proxy in Data of Your Choice with or without SSL
  

[![](https://1.bp.blogspot.com/-7ahlPwQ12LA/XWA3ooki9KI/AAAAAAAAYI0/yipYgzksCPIxzrCtUrJiKJ3FSQFO8UROQCLcBGAs/s640/cemListenHTTP.png)](https://1.bp.blogspot.com/-7ahlPwQ12LA/XWA3ooki9KI/AAAAAAAAYI0/yipYgzksCPIxzrCtUrJiKJ3FSQFO8UROQCLcBGAs/s1600/cemListenHTTP.png)

  

Here's an example CURL script to test that REST API:

  

<style type="text/css">p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000} span.s1 {font-variant-ligatures: no-common-ligatures} </style>

curl -d '{"key1":"value1", "key2":"value2"}' -H "Content-Type: application/json" -X POST http://ec2-3-85-54-189.compute-1.amazonaws.com:8899/upload

  

  
We can generate JSON IoT Style Data for Integration Tests with ease using GenerateFlowFile:  

[![](https://1.bp.blogspot.com/-K7dhGJXvQKo/XWA3nvbLTmI/AAAAAAAAYIo/x3hG1Hu6-zM5C8rqFXpKmYVZ0c8Zbf7AACLcBGAs/s640/GenerateFlowFileJSON.png)](https://1.bp.blogspot.com/-K7dhGJXvQKo/XWA3nvbLTmI/AAAAAAAAYIo/x3hG1Hu6-zM5C8rqFXpKmYVZ0c8Zbf7AACLcBGAs/s1600/GenerateFlowFileJSON.png)

  

  

Let's grab updated models when they change from my Data Science server:
  

[![](https://1.bp.blogspot.com/-ZSRvh3dKbpY/XWA3nv3qtsI/AAAAAAAAYIk/NiIFildtHhcbCGn_UqOoOejInDlUmePQQCLcBGAs/s640/GrabModelsNiFi.png)](https://1.bp.blogspot.com/-ZSRvh3dKbpY/XWA3nv3qtsI/AAAAAAAAYIk/NiIFildtHhcbCGn_UqOoOejInDlUmePQQCLcBGAs/s1600/GrabModelsNiFi.png)

  

I can read Kafka messages and send them to MiNiFi agents as well.
  

[![](https://1.bp.blogspot.com/-svmRMndbPM4/XWA3pTZiabI/AAAAAAAAYI8/5H9tvBZ5hXUYRO-0-bze6gFaajSLyVxoACLcBGAs/s640/publishKafkaRec2.png)](https://1.bp.blogspot.com/-svmRMndbPM4/XWA3pTZiabI/AAAAAAAAYI8/5H9tvBZ5hXUYRO-0-bze6gFaajSLyVxoACLcBGAs/s1600/publishKafkaRec2.png)

  

  

  

So I pushed a TFLITE model, but ONNX, PMML, Docker or Pickle are all options.