Easy Deep Learning in Apache NiFi with DJL
Tags: ai, apachenifi, apachenifi110, cloudera
Published: 2019-12-19 15:44:33

---
title: Easy Deep Learning in Apache NiFi with DJL
published: true
date: 2019-12-17 19:44:00 UTC
tags: AI,apache-nifi,apache-nifi-1.10,cloudera
canonical_url: 
---

  

## **Custom Processor for Deep Learning**
  
  

[![](https://1.bp.blogspot.com/-9lqodzhzEZA/XfkvwaDjMBI/AAAAAAAAZB0/4l6xN6sKuCI_cxESVivk2MdzA8WqmJRLgCLcBGAsYHQ/s640/e9a0cc85-1725-47a2-a2d0-e183139ee1e7.person%2B%25281%2529.png)](https://1.bp.blogspot.com/-9lqodzhzEZA/XfkvwaDjMBI/AAAAAAAAZB0/4l6xN6sKuCI_cxESVivk2MdzA8WqmJRLgCLcBGAsYHQ/s1600/e9a0cc85-1725-47a2-a2d0-e183139ee1e7.person%2B%25281%2529.png)

##   **Happy Mm.. FLaNK Day!**
  
I have been experimenting with the awesome new Apache 2.0 licensed Java Deep Learning Library, DJL.   In NiFi I was trying to figure out a quick use case and demo.   So I use my Web Camera processor to grab a still shot from my Powerbook webcam and send it to the processor.   The results are sent to slack.  
  
Since it's the holidays I think of my favorite holiday movies:   The Matrix and Blade Runner.   So I thought a Voight-Kampf test would be fun.   Since I don't have a Deep Learning QA piece built yet, let's start by seeing if you look human.  We'll call them 'person'.   I am testing to see if I am a replicant.  Sometimes hard to tell.   Let's see if DJL thinks I am human.  
  
**See** :   [http://nautil.us/blog/-the-science-behind-blade-runners-voight\_kampff-test](http://nautil.us/blog/-the-science-behind-blade-runners-voight_kampff-test)  
  
[![](https://1.bp.blogspot.com/-X7AP6UPXTzM/XfkSXZ3SfFI/AAAAAAAAZAc/H3VvsTrMMzwuSJ_0FRP1xG638iywqhp6wCLcBGAsYHQ/s640/slackr1.png)](https://1.bp.blogspot.com/-X7AP6UPXTzM/XfkSXZ3SfFI/AAAAAAAAZAc/H3VvsTrMMzwuSJ_0FRP1xG638iywqhp6wCLcBGAsYHQ/s1600/slackr1.png)  
  
Okay, so it least it thinks I am a person.   The classification of a Christmas tree is vaguely accurate.  

  

  

[![](https://1.bp.blogspot.com/-RNNwP9x4xks/Xfkr4DW68vI/AAAAAAAAZBc/Xwuvt4FUebIbwqr62PVZdyjjNqgS0hMLgCLcBGAsYHQ/s640/76aaeb83-654a-4d33-a2df-84632f759093.person.png)](https://1.bp.blogspot.com/-RNNwP9x4xks/Xfkr4DW68vI/AAAAAAAAZBc/Xwuvt4FUebIbwqr62PVZdyjjNqgS0hMLgCLcBGAsYHQ/s1600/76aaeb83-654a-4d33-a2df-84632f759093.person.png)
  
  

[![](https://1.bp.blogspot.com/-fgMhulSDmB4/XfkSVa2I8hI/AAAAAAAAY_8/7ThVjcLj2jMNzfyeMSk09Ja6ZSFN2CjCACLcBGAsYHQ/s640/55fd709d-d886-4908-b4e6-76fecabc90f1.pottedplant.png)](https://1.bp.blogspot.com/-fgMhulSDmB4/XfkSVa2I8hI/AAAAAAAAY_8/7ThVjcLj2jMNzfyeMSk09Ja6ZSFN2CjCACLcBGAsYHQ/s1600/55fd709d-d886-4908-b4e6-76fecabc90f1.pottedplant.png)
  

[![](https://1.bp.blogspot.com/-zYDjeN2wYlI/XfkSVXemW9I/AAAAAAAAZAE/WAV4qP11F3w8kUCaaUzEj0cXUSsl2U8iwCLcBGAsYHQ/s640/4599c32c-3540-4f2b-a272-38c8626b0b6d.pottedplant.png)](https://1.bp.blogspot.com/-zYDjeN2wYlI/XfkSVXemW9I/AAAAAAAAZAE/WAV4qP11F3w8kUCaaUzEj0cXUSsl2U8iwCLcBGAsYHQ/s1600/4599c32c-3540-4f2b-a272-38c8626b0b6d.pottedplant.png)

  

It did not identify my giant french bread.

  

**Building A New Custom Processor for Deep Learning**

  

[![](https://1.bp.blogspot.com/-kGFDxY9mRSU/XfkSWWi_UvI/AAAAAAAAZAQ/x97fJeagYbEuTECnwfdq34_NhH4YLbQyACLcBGAsYHQ/s640/djlNiFiTest.png)](https://1.bp.blogspot.com/-kGFDxY9mRSU/XfkSWWi_UvI/AAAAAAAAZAQ/x97fJeagYbEuTECnwfdq34_NhH4YLbQyACLcBGAsYHQ/s1600/djlNiFiTest.png)
  

[![](https://1.bp.blogspot.com/-iwo2z5VPc5Y/XfkSW_d0WqI/AAAAAAAAZAU/qXyyZq3TNyIHw8ZtLocL4TQLcLfsCZH5wCLcBGAsYHQ/s640/idepom.png)](https://1.bp.blogspot.com/-iwo2z5VPc5Y/XfkSW_d0WqI/AAAAAAAAZAU/qXyyZq3TNyIHw8ZtLocL4TQLcLfsCZH5wCLcBGAsYHQ/s1600/idepom.png)
  
  
The hardest part of was a good NiFi Integration test.   The DJL team provide some great examples and it's really easy to plug into their models.  
  
ZooModel<BufferedImage, DetectedObjects> model =  
                     MxModelZoo.SSD.loadModel(criteria, new ProgressBar())  
Predictor<BufferedImage, DetectedObjects> predictor = model.newPredictor()  
DetectedObjects detection = predictor.predict(img);  
  
All the source is in github and references the below DJL sites and repos.  
  

**Using a New Custom Processor as part of a Real-time Holiday Flow**

  

We first add the DeepLearningProcessor to our canvas.
  

[![](https://1.bp.blogspot.com/-hmWuyrBMlEA/XfkSVkQavXI/AAAAAAAAZAA/YEKfsQBS-LkUqQsJhqdYsYL6zs36FRK8QCLcBGAsYHQ/s640/addDeepLearningProcessor.png)](https://1.bp.blogspot.com/-hmWuyrBMlEA/XfkSVkQavXI/AAAAAAAAZAA/YEKfsQBS-LkUqQsJhqdYsYL6zs36FRK8QCLcBGAsYHQ/s1600/addDeepLearningProcessor.png)

  

  

An example flow:

- **GetWebCameraProcessor** :  grab an image from an attached webcamera
- **UpdateAttribute** :  Add media type for image
- **DeepLearningProcessor** :   Run our DJL deep learning model from a zoo
- **PutSlack** :   Put DJL results in a text window in slack
- **PostSlack** :  Send our DJL altered image to slack
- **Funnel** :   Send all failures to Valhalla
  

[![](https://1.bp.blogspot.com/-Vk-ukwGZkRs/XfkSXOvZWHI/AAAAAAAAZAY/riTuGMxZKiEi2uDacGC5jrdTvOTLaP-JwCLcBGAsYHQ/s640/nifi10Flow.png)](https://1.bp.blogspot.com/-Vk-ukwGZkRs/XfkSXOvZWHI/AAAAAAAAZAY/riTuGMxZKiEi2uDacGC5jrdTvOTLaP-JwCLcBGAsYHQ/s1600/nifi10Flow.png)

  

  

If we example the provenance we can see how long it took to run and some other interesting attributes.
  

[![](https://1.bp.blogspot.com/-sCrz7Cl3CRE/XfkSVxrMQ9I/AAAAAAAAZAI/2OwDL0Sw3oormZbAg0DVBoWlecYepDT2wCLcBGAsYHQ/s640/deepLearningLineage.png)](https://1.bp.blogspot.com/-sCrz7Cl3CRE/XfkSVxrMQ9I/AAAAAAAAZAI/2OwDL0Sw3oormZbAg0DVBoWlecYepDT2wCLcBGAsYHQ/s1600/deepLearningLineage.png)

  

We place the results of our image analysis in attributes while we return a new image that has a bounding box on the found object(s).
  

[![](https://1.bp.blogspot.com/-TGTLGABCh7c/XfkSWEuY02I/AAAAAAAAZAM/vrZCi-yhE_4YcFKQSsYtGzrmWoJDzM_CACLcBGAsYHQ/s640/deepLearningProvenance.png)](https://1.bp.blogspot.com/-TGTLGABCh7c/XfkSWEuY02I/AAAAAAAAZAM/vrZCi-yhE_4YcFKQSsYtGzrmWoJDzM_CACLcBGAsYHQ/s1600/deepLearningProvenance.png)
  

  

  

 We now a full classification workflow for real-time deep learning analysis on images, could be used for Santa watching, Security, Memes and other important business purposes.  
  
  
The initial release is available here:   [https://github.com/tspannhw/nifi-djl-processor/releases/tag/v1.0](https://github.com/tspannhw/nifi-djl-processor/releases/tag/v1.0)  
Using library and example code from the  **Deep Java Library** ([https://djl.ai/](https://djl.ai/)).  
  

#### 
- [https://github.com/awslabs/djl#getting-started](https://github.com/awslabs/djl#getting-started)

- [https://djl.ai/mxnet/mxnet-model-zoo/](https://djl.ai/mxnet/mxnet-model-zoo/)

- [https://github.com/awslabs/djl/blob/master/examples/pom.xml](https://github.com/awslabs/djl/blob/master/examples/pom.xml)

- [https://github.com/tspannhw/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/ObjectDetection.java](https://github.com/tspannhw/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/ObjectDetection.java)

- [https://github.com/aws-samples/djl-demo](https://github.com/aws-samples/djl-demo)

- [https://github.com/awslabs/djl/blob/master/jupyter/BERTQA.ipynb](https://github.com/awslabs/djl/blob/master/jupyter/BERTQA.ipynb)

- [https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/BertQaInference.java](https://github.com/awslabs/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/BertQaInference.java)
  
  
**Source Code: **  [https://github.com/tspannhw/nifi-djl-processor/](https://github.com/tspannhw/nifi-djl-processor/releases/tag/v1.0)  
  
And now for something completely different, Christmas Cats:  
  

[![](https://1.bp.blogspot.com/-8BAn7DS657w/XfkrzsaBJSI/AAAAAAAAZBA/KA5NqvnispcEFOkHTAcXprytfhZJBOH9wCLcBGAsYHQ/s640/IMG-6757.jpg)](https://1.bp.blogspot.com/-8BAn7DS657w/XfkrzsaBJSI/AAAAAAAAZBA/KA5NqvnispcEFOkHTAcXprytfhZJBOH9wCLcBGAsYHQ/s1600/IMG-6757.jpg)
  

[![](https://1.bp.blogspot.com/-RsZTam5vw4o/Xfkrz4Prg4I/AAAAAAAAZBE/tB5DBkaqeJM6KCRRBNABqKqq6b9nj9aBQCLcBGAsYHQ/s640/IMG-6766.jpg)](https://1.bp.blogspot.com/-RsZTam5vw4o/Xfkrz4Prg4I/AAAAAAAAZBE/tB5DBkaqeJM6KCRRBNABqKqq6b9nj9aBQCLcBGAsYHQ/s1600/IMG-6766.jpg)
  

[![](https://1.bp.blogspot.com/-jRzVTBN1kfs/XfkrzzT4ssI/AAAAAAAAZBI/Yes74BdCK741PhrS2kVWGvg_EcdU-dW8gCLcBGAsYHQ/s640/IMG-6767.jpg)](https://1.bp.blogspot.com/-jRzVTBN1kfs/XfkrzzT4ssI/AAAAAAAAZBI/Yes74BdCK741PhrS2kVWGvg_EcdU-dW8gCLcBGAsYHQ/s1600/IMG-6767.jpg)
  

[![](https://1.bp.blogspot.com/-RhCQNTPwiMI/Xfkr0L1WY9I/AAAAAAAAZBM/XHGb9u6ZOoQO-FKpwIPxNbaPBmLIJDJqQCLcBGAsYHQ/s640/IMG-6769.jpg)](https://1.bp.blogspot.com/-RhCQNTPwiMI/Xfkr0L1WY9I/AAAAAAAAZBM/XHGb9u6ZOoQO-FKpwIPxNbaPBmLIJDJqQCLcBGAsYHQ/s1600/IMG-6769.jpg)
  

[![](https://1.bp.blogspot.com/-KQksSF_wByM/Xfkr0XdExvI/AAAAAAAAZBQ/vAszJVSQb3EoX0ePX70bPLqJzEg8wOvdgCLcBGAsYHQ/s640/IMG-6770.jpg)](https://1.bp.blogspot.com/-KQksSF_wByM/Xfkr0XdExvI/AAAAAAAAZBQ/vAszJVSQb3EoX0ePX70bPLqJzEg8wOvdgCLcBGAsYHQ/s1600/IMG-6770.jpg)
  

[![](https://1.bp.blogspot.com/-_8g2RzWKHCs/Xfkr09jKz-I/AAAAAAAAZBY/E7c9IZ2lt6II_RSvhQdaSv0lH7JQGo49gCLcBGAsYHQ/s640/IMG-6771.JPG)](https://1.bp.blogspot.com/-_8g2RzWKHCs/Xfkr09jKz-I/AAAAAAAAZBY/E7c9IZ2lt6II_RSvhQdaSv0lH7JQGo49gCLcBGAsYHQ/s1600/IMG-6771.JPG)
  

[![](https://1.bp.blogspot.com/-5gOj0WjO2xc/Xfkr02vMD5I/AAAAAAAAZBU/lj_J-ra-iscTgTFQ-qKFW-SEz7oVhm6NQCLcBGAsYHQ/s640/IMG-6772.jpg)](https://1.bp.blogspot.com/-5gOj0WjO2xc/Xfkr02vMD5I/AAAAAAAAZBU/lj_J-ra-iscTgTFQ-qKFW-SEz7oVhm6NQCLcBGAsYHQ/s1600/IMG-6772.jpg)