Seamless Integration: Unleashing the Power of Real-Time Groceries with NiFi, Kafka, Flink and JQuery
Published: 2023-06-05 14:22:45

---
title: Seamless Integration: Unleashing the Power of Real-Time Groceries with NiFi, Kafka, Flink and JQuery
published: true
date: 2023-06-01 14:29:00 UTC
tags: 
canonical_url: 
---

 

<section style="box-sizing: inherit;"><div style="box-sizing: inherit;"><div style="box-sizing: inherit; overflow-wrap: break-word; word-break: break-word;">
<div style="box-sizing: inherit; display: flex; justify-content: center;"><div style="box-sizing: inherit; margin: 0px 24px; max-width: 680px; min-width: 0px; width: 680px;">
<h1 data-selectable-paragraph="" id="85cc" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>FLaNK-Store</h1>
<p data-selectable-paragraph="" id="1984" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Real-Time Retail Grocery Store</span> with Apache MiNiFi, Apache NiFi, Apache Kafka, Apache Flink, Apache Kudu, Apache Ozone, Apache Iceberg, HTML, JQuery, DataTables.</p>
<p data-selectable-paragraph="" id="5b0a" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Source</span>: <a href="https://github.com/tspannhw/FLaNK-Store" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/FLaNK-Store</a></p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 666px;"><img alt="" height="500" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:666/0*38xwlLgKJwjw_Ahj.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 666px;" width="666"></div>
<p data-selectable-paragraph="" id="ecb3" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>In today’s example, I need to ingest grocery items for some analytics, so let’s read these via a secured REST API. To follow along, you will need to sign up for your own free key to see this interesting data. Who doesn’t want to ingest bananas with NiFi. Maybe I just really need to know about eggs and sugar. A friend of mine Brent wanted me to build this during lockdown to determine the current price of a basket of household goods. I had built it and it sat on the shelf as most of the APIs I wanted were not available. Thanks to the innovative retail work at Kroger, we can get the data at speed with their API. Let’s explore. This is the first in the series, I will cover storage to Ozone (on S3), Iceberg and Kudu. I will also cover In-Store Data Collection, Updating Shelf Pricing in Real-Time (Raspberry Pi with E-Ink) and more retail use cases.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2082px;"><img alt="" height="251" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*-D49lvBNABe2jutE" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<h1 data-selectable-paragraph="" id="afea" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>We are ingesting data from Kroger</h1>
<p data-selectable-paragraph="" id="2b21" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>You will need to sign-up to get your credentials to get this cool data.</p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="4f85" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2em; padding-left: 0px;'><a href="https://developer.kroger.com/reference/#operation/productGet" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/reference/#operation/productGet</a></li>
<li data-selectable-paragraph="" id="38c1" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://developer.kroger.com/documentation" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/documentation</a></li>
<li data-selectable-paragraph="" id="bea7" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://developer.kroger.com/documentation/public/getting-started/quick-start" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/documentation/public/getting-started/quick-start</a></li>
<li data-selectable-paragraph="" id="60d6" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://developer.kroger.com/documentation/api-products/public/products/overview" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/documentation/api-products/public/products/overview</a></li>
</ul>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="362" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*JK1MwS8p9MH7rC9Z.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="8522" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>We picked a basket of items to watch from the stores.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 472px;"><img alt="" height="499" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:472/1*fVUrE4FRM-IIXBuw9rIUAA.png" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 472px;" width="472"></div>
<p data-selectable-paragraph="" id="2624" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Some of my basket items: Organic Banana,<br style="box-sizing: inherit;">Velveeta Original Cheese, Imperial Cane Sugar, Oreo Team USA Chocolate Sandwich Cookies, Kroger® Fat Free Skim Milk,<br style="box-sizing: inherit;">Land O Lakes® Salted Butter Sticks,<br style="box-sizing: inherit;">Kroger® 1 lb. Lean Ground Beef Chuck Roll 80/20,<br style="box-sizing: inherit;">Stouffer’s® Macaroni &amp; Cheese Frozen Meal and<br style="box-sizing: inherit;">Eggo Homestyle Frozen Waffles.</p>
<h1 data-selectable-paragraph="" id="e559" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>Apache NiFi Flow Walk Through</h1>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="270" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*yJGvYIn73mnzKHjc.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1194px;"><img alt="" height="543" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*7DKwdG-YCC_7omHi.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="d226" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>We have a few interesting flows for working with Retail data. The first one is ingesting product information from Kroger.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1465px;"><img alt="" height="787" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*bGFfj09dhh1DPyv9.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Scheduled every 10 seconds, pick a “random” item from our basket, call the OAuth authentication and use it’s token.</figcaption>
</div></div>
<div style="box-sizing: inherit; clear: both;"><div style="box-sizing: inherit; display: flex; justify-content: center;"><div style="box-sizing: inherit; margin: 0px 64px; max-width: 1192px; min-width: 0px; width: 1192px;">
<p style="box-sizing: inherit; clear: both; margin: 56px 0px 0px; padding-bottom: 5px; padding-top: 5px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 1192px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2434px;"><img alt="" height="671" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:1000/0*b2UeTUesZk64zq43.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 1192px;" width="1000"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>We call that items product detail, split out the record to send different images to a separate topic, for the main flow we build a new pricing object.</figcaption>
</div></div></div>
<div style="box-sizing: inherit; display: flex; justify-content: center;"><div style="box-sizing: inherit; margin: 0px 24px; max-width: 680px; min-width: 0px; width: 680px;">
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="404" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*Fvk5ZHIDdRMRx8Q4.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Send images URLs to their own <span style="box-sizing: inherit; font-weight: 600;">itemimage</span> Kafka topic.</figcaption>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="449" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*ZIU2RAaoaGdwc7__.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Add a schema, validate our record and push it to the <span style="box-sizing: inherit; font-weight: 600;">item</span> topic.</figcaption>
<h1 data-selectable-paragraph="" id="c92f" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>Kafka Topics Shown in Streams Messaging Manager</h1>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="338" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*27HB0v42ltP9I6Z7.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1209px;"><img alt="" height="994" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*_ppUUmC0I4m8E_yO.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="276" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*n6wPZWaZfGu7ywRO.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="360" loading="eager" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*R0t3fpJi6vwcUWLv" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<h1 data-selectable-paragraph="" id="407d" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>Apache Kafka Item Image</h1>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1205px;"><img alt="" height="261" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*xgCmkiLLyuDserv4.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<h1 data-selectable-paragraph="" id="3d3a" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>Querying Kafka Topic for Items</h1>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="283" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*DnBd2NbKnfBCxnsQ.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="195" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*THb7pGxZYfi-cEND.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="373" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*EQEGcgoQALhu0ypf.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="332" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*gbfTi0OPYq-2D4Hy.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<h1 data-selectable-paragraph="" id="6893" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>NiFi Calcite SQL — To Transform and Enrich Item Price Stream</h1>
<pre style='background: rgb(249, 249, 249); border-radius: 4px; border: 1px solid rgb(229, 229, 229); box-sizing: inherit; color: #242424; font-family: source-code-pro, Menlo, Monaco, "Courier New", Courier, monospace; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 32px;'><span data-selectable-paragraph="" id="97f3" style="box-sizing: inherit; display: block; font-size: 14px; letter-spacing: -0.022em; line-height: 1.4; margin-bottom: -0.2em; margin-top: -0.2em; min-width: fit-content;"><span style="box-sizing: inherit; color: #aa0d91;">SELECT</span> brandname,category,countryorigin,<span style="box-sizing: inherit; color: #c41a16;">'${date}'</span> <span style="box-sizing: inherit; color: #aa0d91;">as</span> itemdate,displayimage,<br style="box-sizing: inherit;"> images,item,itemId,itemdescription,itemheatsen,itemsize,longdescrption,<br style="box-sizing: inherit;"> msrp,originstore,<span style="box-sizing: inherit; color: #5c2699;">cast</span>(<span style="box-sizing: inherit; color: #5c2699;">COALESCE</span>(price, <span style="box-sizing: inherit; color: #1c00cf;">0.00</span>) <span style="box-sizing: inherit; color: #aa0d91;">as</span> <span style="box-sizing: inherit; color: #5c2699;">float</span>) <span style="box-sizing: inherit; color: #aa0d91;">as</span> price,<br style="box-sizing: inherit;"> productid,tmpind,tpr,ts,upc,uuid,updatedate<br style="box-sizing: inherit;"><span style="box-sizing: inherit; color: #aa0d91;">FROM</span> FLOWFILE</span></pre>
<h1 data-selectable-paragraph="" id="e10d" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>Flink SQL to Browse Data</h1>
<p data-selectable-paragraph="" id="f02d" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>An example of querying Apache Kafka topics with Apache Flink SQL via Schema Registry catalog.</p>
<pre style='background: rgb(249, 249, 249); border-radius: 4px; border: 1px solid rgb(229, 229, 229); box-sizing: inherit; color: #242424; font-family: source-code-pro, Menlo, Monaco, "Courier New", Courier, monospace; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 32px;'><span data-selectable-paragraph="" id="d5a7" style="box-sizing: inherit; display: block; font-size: 14px; letter-spacing: -0.022em; line-height: 1.4; margin-bottom: -0.2em; margin-top: -0.2em; min-width: fit-content;"><span style="box-sizing: inherit; color: #aa0d91;">select</span> brandname, item, itemdescription, itemsize, <br style="box-sizing: inherit;"> price, category, <br style="box-sizing: inherit;"> updatedate, longdescrption, displayimage<br style="box-sizing: inherit;"><span style="box-sizing: inherit; color: #aa0d91;">from</span> `sr1`.`default_database`.`item`</span></pre>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="525" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*yTHqOF6TLy_bqcY2.jpg" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="accd" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>I wish to make this data available for Jupyter notebooks and also HTML pages. I will add some notebooks and <a href="https://docs.cloudera.com/data-visualization/7/visuals/topics/viz-visual-builder.html" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Cloudera Data Visualization</a>.</p>
<p data-selectable-paragraph="" id="b888" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The easiest way to do this is create a materialized view in SQL Stream Builder to make my query results available as JSON over REST.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2416px;"><img alt="" height="469" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*2R-qqJD-uFvUzvGN" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1712px;"><img alt="" height="514" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*7jX59vTdmW6At_4s" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="86e3" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Once I see the results of my query, I am good to go. Let’s build an HTML view of this data.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="357" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*bzdKlNt-kEUSPN3u" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="d505" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s make sure our materialized view is loaded in the raw REST feed.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="192" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*0S24F8dewtrqz0d0" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="da00" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s use <a href="https://datatables.net/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">DataTables</a> and <a href="https://jquery.com/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">JQuery</a> to build a dynamic HTML table view.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="293" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*52t2t0AylmjLhad6" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="5486" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>All of the code is available in github for you to use with your own data or your own basket to load from Kroger.</p>
<p data-selectable-paragraph="" id="6b8e" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The final results here:</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 680px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2560px;"><img alt="" height="353" loading="lazy" role="presentation" src="https://miro.medium.com/v2/resize:fit:700/0*pBSEwcL2BPCAlaUV" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 680px;" width="700"></div></div>
<p data-selectable-paragraph="" id="8165" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">ChatGPT gave me some good ideas.</span></p>
<p data-selectable-paragraph="" id="1fe2" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Using the Kroger REST API for groceries with Apache NiFi is a great idea for several reasons. Let’s explore the benefits of combining these two technologies:</p>
<p data-selectable-paragraph="" id="5d01" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Data Integration:</span> Apache NiFi is a powerful data integration tool that enables the seamless flow and transformation of data between various systems. By integrating the Kroger REST API with NiFi, you can easily fetch, process, and distribute grocery-related data from Kroger’s services to your desired destinations.</p>
<p data-selectable-paragraph="" id="7296" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Real-Time Data:</span> The Kroger REST API provides real-time access to a wide range of grocery-related information, including product details, prices, availability, and more. By leveraging NiFi’s capabilities, you can constantly monitor and retrieve the latest data from Kroger’s API, ensuring that you have up-to-date information at all times.</p>
<p data-selectable-paragraph="" id="c3c4" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Scalability:</span> NiFi is designed to handle high volumes of data and can scale horizontally to accommodate increased workloads. This scalability makes it well-suited for processing large quantities of grocery data fetched from the Kroger API. You can configure NiFi to handle parallel processing, data partitioning, and load balancing, ensuring efficient data flow even during peak times.</p>
<p data-selectable-paragraph="" id="0b58" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Data Transformation and Enrichment: </span>NiFi provides a wide range of processors and functions that facilitate data transformation and enrichment. You can use NiFi’s processors to extract specific data from the Kroger API responses, apply transformations, perform calculations, and enrich the data with additional information. This capability allows you to tailor the data from the Kroger API to suit your specific requirements.</p>
<p data-selectable-paragraph="" id="67c4" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Data Quality and Reliability:</span> NiFi offers extensive data quality monitoring and control capabilities. You can implement data validation rules, perform data cleansing operations, and apply data governance practices to ensure the accuracy and reliability of the grocery data received from the Kroger API. NiFi also provides features like provenance tracking, error handling, and data lineage, which help you maintain data integrity throughout the data flow.</p>
<p data-selectable-paragraph="" id="d389" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Integration with Other Systems:</span> NiFi supports integration with a wide range of systems and platforms, including databases, data lakes, messaging systems, and cloud services. By combining the Kroger REST API with NiFi, you can seamlessly integrate the fetched grocery data with your existing data infrastructure, enabling further analysis, reporting, and integration with downstream systems.</p>
<p data-selectable-paragraph="" id="dc16" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Workflow Orchestration:</span> NiFi allows you to design complex data workflows through its intuitive graphical user interface. You can create workflows that fetch data from the Kroger API, apply various transformations, perform validations, and route the data to different destinations based on predefined rules. This workflow orchestration capability simplifies the data integration process and provides better control over data flow.</p>
<p data-selectable-paragraph="" id="ec01" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>By utilizing the Kroger REST API with Apache NiFi, you can leverage the strengths of both technologies to build a robust and scalable grocery data integration pipeline. This combination empowers you to access real-time grocery data, apply transformations, ensure data quality, and integrate with other systems effectively.</p>
<h1 data-selectable-paragraph="" id="c24d" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 24px; letter-spacing: -0.016em; line-height: 30px; margin: 1.95em 0px -0.28em;'>References</h1>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="8cf7" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 0.86em; padding-left: 0px;'><a href="https://developer.kroger.com/documentation/api-products/public/products/tutorial" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/documentation/api-products/public/products/tutorial</a></li>
<li data-selectable-paragraph="" id="c5dc" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://github.com/tspannhw/retail-dynamic-shelf-pricing" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/retail-dynamic-shelf-pricing</a></li>
<li data-selectable-paragraph="" id="b72e" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://github.com/tspannhw/FLaNK-AllTheStreams/tree/main/schemas" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/FLaNK-AllTheStreams/tree/main/schemas</a></li>
<li data-selectable-paragraph="" id="fe87" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://developer.kroger.com/documentation/partner/getting-started/apis" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://developer.kroger.com/documentation/partner/getting-started/apis</a></li>
<li data-selectable-paragraph="" id="b7f8" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://documenter.getpostman.com/view/4833726/TVReeWJm" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://documenter.getpostman.com/view/4833726/TVReeWJm</a></li>
<li data-selectable-paragraph="" id="33ea" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://github.com/tspannhw/FLaNK-AllTheStreams" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/FLaNK-AllTheStreams</a></li>
<li data-selectable-paragraph="" id="6bd9" style='box-sizing: inherit; color: #292929; font-family: source-serif-pro, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 21px; letter-spacing: -0.003em; line-height: 32px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.05em; padding-left: 0px;'><a href="https://medium.com/@tspann/building-a-real-time-data-pipeline-a-comprehensive-tutorial-on-minifi-nifi-kafka-and-flink-ee03ee6722cb" rel="noopener" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;">https://medium.com/@tspann/building-a-real-time-data-pipeline-a-comprehensive-tutorial-on-minifi-nifi-kafka-and-flink-ee03ee6722cb</a></li>
</ul>
</div></div>
</div></div></section>

[
Apache Flink
](https://medium.com/tag/apache-flink?source=post_page-----32d659fe0903---------------apache_flink-----------------)

[
Apache Nifi
](https://medium.com/tag/apache-nifi?source=post_page-----32d659fe0903---------------apache_nifi-----------------)

[
Apache Kafka
](https://medium.com/tag/apache-kafka?source=post_page-----32d659fe0903---------------apache_kafka-----------------)

[
Jquery
](https://medium.com/tag/jquery?source=post_page-----32d659fe0903---------------jquery-----------------)

[
Open Source
](https://medium.com/tag/open-source?source=post_page-----32d659fe0903---------------open_source-----------------)