Tracking Aircraft in Real-Time With Open Source
Tags: apachepulsar, apacheflink, realtime, aircraft
Published: 2022-09-01 18:03:33

---
title: Tracking Aircraft in Real-Time With Open Source
published: true
date: 2022-09-01 14:48:00 UTC
tags: apachepulsar, apacheflink, realtime, aircraft
canonical_url: 
---

 

<section style="box-sizing: inherit;"><div style="box-sizing: inherit;">
<div style="box-sizing: inherit; overflow-wrap: break-word; word-break: break-word;">
<div style="box-sizing: inherit;"><h1 data-selectable-paragraph="" id="36de" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 32px; letter-spacing: -0.016em; line-height: 40px; margin: 0.6em 0px -0.27em;'>Tracking Aircraft in Real-Time With Open Source</h1></div>
<p data-selectable-paragraph="" id="a974" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Use Case: Automatic Dependent Surveillance-Broadcast (ADS-B Analytics)</span></p>
<div style="box-shadow: rgb(230, 230, 230) 0px 0px 0px 1px inset; box-sizing: inherit; margin-top: 32px;"><a href="https://en.wikipedia.org/wiki/Automatic_Dependent_Surveillance%E2%80%93Broadcast" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit; text-decoration-line: none;" target="_blank"><div style="box-sizing: inherit; display: flex; flex: 0 0 auto; padding: 0px;">
<div style="box-sizing: inherit; display: flex; flex-direction: column; flex: 1 1 auto; justify-content: center; padding: 16px 20px;">
<h2 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #292929; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>Automatic Dependent Surveillance-Broadcast - Wikipedia</h2>
<div style="box-sizing: inherit; margin-top: 8px;"><h3 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; font-weight: 400; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>Automatic Dependent Surveillance-Broadcast ( ADS-B) is a surveillance technology in which an aircraft determines its…</h3></div>
<div style="box-sizing: inherit; margin-top: 12px;"><p style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>en.wikipedia.org</p></div>
</div>
<div style="box-sizing: inherit; width: 160px;"><div style='background-image: url("https://miro.medium.com/max/320/0*gGR3Pmc0BMK-vOPC"); background-origin: border-box; background-position: 50% 50%; background-size: cover; box-shadow: rgb(230, 230, 230) 0px 0px 0px 1px inset; box-sizing: inherit; height: 167px; max-width: 100%; width: 160px;'></div></div>
</div></a></div>
<p data-selectable-paragraph="" id="0205" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Software Stack:</span></p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="a8c0" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2.14em; padding-left: 0px;'><a href="https://www.python.org/downloads/release/python-3100/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Python 3.10</a></li>
<li data-selectable-paragraph="" id="fb9b" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://pulsar.apache.org/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Apache Pulsar 2.10.1</a></li>
<li data-selectable-paragraph="" id="6fa8" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://pulsar.apache.org/docs/next/client-libraries-python" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Apache Pulsar Python Library</a></li>
<li data-selectable-paragraph="" id="4049" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://spark.apache.org/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Apache Spark</a></li>
<li data-selectable-paragraph="" id="2651" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://flink.apache.org/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Apache Flink</a></li>
<li data-selectable-paragraph="" id="c4d8" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://openjdk.org/projects/jdk/17/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Java JDK 17</a></li>
<li data-selectable-paragraph="" id="d7b1" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://maven.apache.org/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Apache Maven</a></li>
<li data-selectable-paragraph="" id="4eba" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://sdkman.io/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">SDKMan</a></li>
<li data-selectable-paragraph="" id="9de8" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://www.raspberrypi.com/software/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Raspian Linux</a></li>
</ul>
<div style="box-shadow: rgb(230, 230, 230) 0px 0px 0px 1px inset; box-sizing: inherit; margin-top: 32px;"><a href="https://www.faa.gov/nextgen/programs/adsb/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit; text-decoration-line: none;" target="_blank"><div style="box-sizing: inherit; display: flex; flex: 0 0 auto; padding: 0px;"><div style="box-sizing: inherit; display: flex; flex-direction: column; flex: 1 1 auto; justify-content: center; padding: 16px 20px;">
<h2 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #292929; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>Next Generation Air Transportation System (NextGen)</h2>
<div style="box-sizing: inherit; margin-top: 8px;"><h3 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; font-weight: 400; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>An official website of the United States government</h3></div>
<div style="box-sizing: inherit; margin-top: 12px;"><p style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>www.faa.gov</p></div>
</div></div></a></div>
<p data-selectable-paragraph="" id="bcc1" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Hardware Stack:</span></p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="b405" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2.14em; padding-left: 0px;'><a href="https://flightaware.store/products/pro-stick-plus" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Flightaware Pro Plus Stick (Blue)</a></li>
<li data-selectable-paragraph="" id="7a41" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://flightaware.store/products/antenna-1090mhz" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">1090 MHz Antenna and Cable</a></li>
<li data-selectable-paragraph="" id="0311" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://rpilocator.com/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Raspberry Pi 4 with 2GB RAM</a></li>
<li data-selectable-paragraph="" id="d4b1" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://www.raspberrypi.com/products/type-c-power-supply/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">USB-C Power Supply</a></li>
</ul>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 4032px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*saK0gPy0A79LLgX8iy6vTA.jpeg" srcset="https://miro.medium.com/max/300/1*saK0gPy0A79LLgX8iy6vTA.jpeg 300w, https://miro.medium.com/max/640/1*saK0gPy0A79LLgX8iy6vTA.jpeg 640w, https://miro.medium.com/max/720/1*saK0gPy0A79LLgX8iy6vTA.jpeg 720w, https://miro.medium.com/max/750/1*saK0gPy0A79LLgX8iy6vTA.jpeg 750w, https://miro.medium.com/max/786/1*saK0gPy0A79LLgX8iy6vTA.jpeg 786w, https://miro.medium.com/max/828/1*saK0gPy0A79LLgX8iy6vTA.jpeg 828w, https://miro.medium.com/max/1400/1*saK0gPy0A79LLgX8iy6vTA.jpeg 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<p data-selectable-paragraph="" id="5ec4" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">References:</span></p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="3339" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">What is ADS-B?</span> <a href="https://www.faa.gov/nextgen/programs/adsb/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://www.faa.gov/nextgen/programs/adsb/</a>
</li>
<li data-selectable-paragraph="" id="fe75" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">Function Source</span>: <a href="https://github.com/tspannhw/pulsar-adsb-function" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/pulsar-adsb-function</a>
</li>
<li data-selectable-paragraph="" id="2eeb" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">Analytics Source: </span><a href="https://github.com/tspannhw/FLiP-Py-ADS-B" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/FLiP-Py-ADS-B</a>
</li>
</ul>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 450px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 450px" src="https://miro.medium.com/max/900/1*g-qbB-LZR2Js9b17glOy9g.png" srcset="https://miro.medium.com/max/300/1*g-qbB-LZR2Js9b17glOy9g.png 300w, https://miro.medium.com/max/640/1*g-qbB-LZR2Js9b17glOy9g.png 640w, https://miro.medium.com/max/720/1*g-qbB-LZR2Js9b17glOy9g.png 720w, https://miro.medium.com/max/750/1*g-qbB-LZR2Js9b17glOy9g.png 750w, https://miro.medium.com/max/786/1*g-qbB-LZR2Js9b17glOy9g.png 786w, https://miro.medium.com/max/828/1*g-qbB-LZR2Js9b17glOy9g.png 828w, https://miro.medium.com/max/900/1*g-qbB-LZR2Js9b17glOy9g.png 900w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 450px;"></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Main Source Code</figcaption>
<p data-selectable-paragraph="" id="50a5" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Utilizing the Open Source FLiP Stack we can track aircraft overhead with ease! It does require a little bit of hardware and some Python magic. You can build your own at home with a few bits of hardware and some easy open-source software.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 3024px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg" srcset="https://miro.medium.com/max/300/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 300w, https://miro.medium.com/max/640/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 640w, https://miro.medium.com/max/720/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 720w, https://miro.medium.com/max/750/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 750w, https://miro.medium.com/max/786/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 786w, https://miro.medium.com/max/828/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 828w, https://miro.medium.com/max/1400/1*s0RkMoc8qBIDsy2hG7wC7w.jpeg 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<p data-selectable-paragraph="" id="9cfc" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Not only are we capturing, transmitting, enriching, and storing this data but we are also contributing back to the world. You can check out my feed at the FlightAware website.</p>
<div style="box-shadow: rgb(230, 230, 230) 0px 0px 0px 1px inset; box-sizing: inherit; margin-top: 32px;"><a href="https://flightaware.com/adsb/stats/user/TimothySpann" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit; text-decoration-line: none;" target="_blank"><div style="box-sizing: inherit; display: flex; flex: 0 0 auto; padding: 0px;">
<div style="box-sizing: inherit; display: flex; flex-direction: column; flex: 1 1 auto; justify-content: center; padding: 16px 20px;">
<h2 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #292929; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; letter-spacing: 0px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>Timothy Spann ADS-B Feeder Statistics - FlightAware</h2>
<div style="box-sizing: inherit; margin-top: 8px;"><h3 style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px; font-weight: 400; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>You can help us keep FlightAware free by allowing ads from FlightAware.com. We work hard to keep our advertising…</h3></div>
<div style="box-sizing: inherit; margin-top: 12px;"><p style='-webkit-box-orient: vertical; -webkit-line-clamp: 2; box-sizing: inherit; color: #757575; display: -webkit-box; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 13px; line-height: 20px; margin: 0px; max-height: 40px; overflow: hidden; text-overflow: ellipsis;'>flightaware.com</p></div>
</div>
<div style="box-sizing: inherit; width: 160px;"><div style='background-image: url("https://miro.medium.com/max/320/0*3AXhXXoU8FSp_KN5"); background-origin: border-box; background-position: 50% 50%; background-size: cover; box-shadow: rgb(230, 230, 230) 0px 0px 0px 1px inset; box-sizing: inherit; height: 167px; max-width: 100%; width: 160px;'></div></div>
</div></a></div>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 3430px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*VI3hza4vtQONIB_4Tx-fAA.png" srcset="https://miro.medium.com/max/300/1*VI3hza4vtQONIB_4Tx-fAA.png 300w, https://miro.medium.com/max/640/1*VI3hza4vtQONIB_4Tx-fAA.png 640w, https://miro.medium.com/max/720/1*VI3hza4vtQONIB_4Tx-fAA.png 720w, https://miro.medium.com/max/750/1*VI3hza4vtQONIB_4Tx-fAA.png 750w, https://miro.medium.com/max/786/1*VI3hza4vtQONIB_4Tx-fAA.png 786w, https://miro.medium.com/max/828/1*VI3hza4vtQONIB_4Tx-fAA.png 828w, https://miro.medium.com/max/1400/1*VI3hza4vtQONIB_4Tx-fAA.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<p data-selectable-paragraph="" id="b76b" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>If you think that this looks like some data that you want for your Lakehouse, let’s start ingesting it now. Once you have your hardware configured, see the directions from <a href="https://flightaware.com/adsb/piaware/build" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Piaware</a>. Once you’ve rebooted and everything is running you should be able to see the status of everything running.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1454px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/0*PgNVYDuCVROmCLek" srcset="https://miro.medium.com/max/300/0*PgNVYDuCVROmCLek 300w, https://miro.medium.com/max/640/0*PgNVYDuCVROmCLek 640w, https://miro.medium.com/max/720/0*PgNVYDuCVROmCLek 720w, https://miro.medium.com/max/750/0*PgNVYDuCVROmCLek 750w, https://miro.medium.com/max/786/0*PgNVYDuCVROmCLek 786w, https://miro.medium.com/max/828/0*PgNVYDuCVROmCLek 828w, https://miro.medium.com/max/1400/0*PgNVYDuCVROmCLek 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Raspberry Pi 4 Console</figcaption>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1458px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/0*4-fF4c9JsIMXG7K4.png" srcset="https://miro.medium.com/max/300/0*4-fF4c9JsIMXG7K4.png 300w, https://miro.medium.com/max/640/0*4-fF4c9JsIMXG7K4.png 640w, https://miro.medium.com/max/720/0*4-fF4c9JsIMXG7K4.png 720w, https://miro.medium.com/max/750/0*4-fF4c9JsIMXG7K4.png 750w, https://miro.medium.com/max/786/0*4-fF4c9JsIMXG7K4.png 786w, https://miro.medium.com/max/828/0*4-fF4c9JsIMXG7K4.png 828w, https://miro.medium.com/max/1400/0*4-fF4c9JsIMXG7K4.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Online Dashboards Of Data Published</figcaption>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2554px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/0*MN-r0SUT954xqODB.png" srcset="https://miro.medium.com/max/300/0*MN-r0SUT954xqODB.png 300w, https://miro.medium.com/max/640/0*MN-r0SUT954xqODB.png 640w, https://miro.medium.com/max/720/0*MN-r0SUT954xqODB.png 720w, https://miro.medium.com/max/750/0*MN-r0SUT954xqODB.png 750w, https://miro.medium.com/max/786/0*MN-r0SUT954xqODB.png 786w, https://miro.medium.com/max/828/0*MN-r0SUT954xqODB.png 828w, https://miro.medium.com/max/1400/0*MN-r0SUT954xqODB.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>Local Flight Feed Running on Raspberry Pi 4</figcaption>
</div>
<div role="separator" style="box-sizing: inherit; display: flex; justify-content: center; margin-bottom: 14px; margin-top: 32px; padding-bottom: 10px; padding-top: 24px;">
<span style="background-color: #080808; border-radius: 50%; box-sizing: inherit; display: inline-block; height: 3px; margin-right: 20px; width: 3px;"></span><span style="background-color: #080808; border-radius: 50%; box-sizing: inherit; display: inline-block; height: 3px; margin-right: 20px; width: 3px;"></span><span style="background-color: #080808; border-radius: 50%; box-sizing: inherit; display: inline-block; height: 3px; width: 3px;"></span>
</div>
<div style="box-sizing: inherit; overflow-wrap: break-word; word-break: break-word;">
<p data-selectable-paragraph="" id="a5c6" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>If you want to skip to a video demoing what I did, you can do that.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px 0px 0px;"></p>
<div style="box-sizing: inherit; margin: auto; position: relative;"><div style="box-sizing: inherit; height: 0px; padding-bottom: 519px;"><iframe allowfullscreen="" frameborder="0" height="480" scrolling="auto" src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0LMOglFN8ZA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0LMOglFN8ZA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0LMOglFN8ZA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" style="box-sizing: inherit; height: 519px; left: 0px; position: absolute; top: 0px; width: 692px;" title="Tracking Aircraft in Real-Time With Open Source - Apache Pulsar" width="640"></iframe></div></div>
<h1 data-selectable-paragraph="" id="6031" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 1</h1>
<p data-selectable-paragraph="" id="cd08" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Let’s build out the topics we are going to need for all this raw and processed flight data.<code style='background-color: #f2f2f2; box-sizing: inherit; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 15px; padding: 2px 4px;'>bin/pulsar-admin topics create persistent://public/default/adsbraw</code></p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="ff5e" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin topics create persistent://public/default/aircraft</span><span data-selectable-paragraph="" id="3c3a" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>bin/pulsar-admin topics create persistent://public/default/adsblog</span><span data-selectable-paragraph="" id="3f2a" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>bin/pulsar-admin topics create persistent://public/default/adsbdead</span></pre>
<p data-selectable-paragraph="" id="d781" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The first topic is our raw JSON ADS-B data. We may want to use that later so we can let this topic store the data forever, perhaps at some point we will turn on tiered storage and have it automagically stored to object storage on AWS, Google Cloud, or Azure. We also have a topic for clean data, aircraft. Finally, we have a topic for logs output from our processing and one for messages that can be processed. They aren’t dead, since we can raise them like zombies and process them again. Never give up on your data!</p>
<p data-selectable-paragraph="" id="af6d" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The first thing I did was examine the data that was displayed on that pretty local map. By running Chrome in Developer Mode, I saw all the REST Calls made. I see JSON data from simple REST calls. This is where I grabbed what I needed and put it into that handy Python script.</p>
<p data-selectable-paragraph="" id="8394" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Since we are going to be transmitting, storing, analyzing, querying, and generally sharing this data with different developers, analysts, data scientists, and other people we should make sure we know what this data is.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="2b98" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'> hex (String optional)<br style="box-sizing: inherit;"> flight (String optional - name of plane)<br style="box-sizing: inherit;"> alt_baro (int optional - altitude)<br style="box-sizing: inherit;"> alt_geom (int optional) <br style="box-sizing: inherit;"> track (int optional)<br style="box-sizing: inherit;"> baro_rate (int optional)<br style="box-sizing: inherit;"> category (string optional)<br style="box-sizing: inherit;"> nav_qnh (float optional)<br style="box-sizing: inherit;"> nav_altitude_mcp (int optional)<br style="box-sizing: inherit;"> nav_heading (float optional)<br style="box-sizing: inherit;"> nic (int optional)<br style="box-sizing: inherit;"> rc (int optional)<br style="box-sizing: inherit;"> seen_pos (float optional)<br style="box-sizing: inherit;"> version (int optional)<br style="box-sizing: inherit;"> nic_baro (int optional)<br style="box-sizing: inherit;"> nac_p (int optional)<br style="box-sizing: inherit;"> nac_v (int optional)<br style="box-sizing: inherit;"> sil (int optional) <br style="box-sizing: inherit;"> sil_type (string optional)<br style="box-sizing: inherit;"> mlat (array optional)<br style="box-sizing: inherit;"> tisb (array optional)<br style="box-sizing: inherit;"> messages (int optional)<br style="box-sizing: inherit;"> seen (float optional)<br style="box-sizing: inherit;"> rssi (float optional) <br style="box-sizing: inherit;"> squawk (optional) - look at # conversion 7600, 7700, 4000, 5000, 7777, 6100, 5400, 4399, 4478, ...)<br style="box-sizing: inherit;"> speed (optional)<br style="box-sizing: inherit;"> mach (optional speed, mac to mph *767)<br style="box-sizing: inherit;"> emergency (optional string)<br style="box-sizing: inherit;"> lat (long optional)<br style="box-sizing: inherit;"> lon (long optional)</span></pre>
<p data-selectable-paragraph="" id="5d3d" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>We need to define a schema with names, types, and optionality. Once we have done this we can build a JSON or Python schema for it and utilize that in Apache Pulsar, Pulsar SQL (Presto/Trino), Apache Spark SQL, Apache Flink SQL, and any consumer that can read a schema from the Pulsar Schema Registry. Data without a contract is just bytes.</p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="858c" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: hex is ICAO identifier</li>
<li data-selectable-paragraph="" id="4c77" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: flight is IDENT identifier</li>
<li data-selectable-paragraph="" id="19f2" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: altBaro is the altitude in feet (barometric)</li>
<li data-selectable-paragraph="" id="8077" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: lat, lon is latitude and longitude</li>
<li data-selectable-paragraph="" id="4374" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: gs is the ground speed in knots</li>
<li data-selectable-paragraph="" id="63bf" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'>
<span style="box-sizing: inherit; font-weight: 700;">field</span>: altGeom is altitude (geometric)</li>
</ul>
<p data-selectable-paragraph="" id="51f7" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>I looked up what the fields were and made some notes. Squawk values are interesting and that may be of interest to people running SQL later.</p>
<p data-selectable-paragraph="" id="072c" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Once you get data into a Java class and start sending messages to your aircraft topic, you can pull out the autogenerated <a href="https://github.com/tspannhw/pulsar-adsb-function/blob/main/aircraft.schema" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">schema</a>.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="68f5" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin schemas get persistent://public/default/aircraft</span></pre>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 4032px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*M25zPkm5VydP-zqj4bQiAA.jpeg" srcset="https://miro.medium.com/max/300/1*M25zPkm5VydP-zqj4bQiAA.jpeg 300w, https://miro.medium.com/max/640/1*M25zPkm5VydP-zqj4bQiAA.jpeg 640w, https://miro.medium.com/max/720/1*M25zPkm5VydP-zqj4bQiAA.jpeg 720w, https://miro.medium.com/max/750/1*M25zPkm5VydP-zqj4bQiAA.jpeg 750w, https://miro.medium.com/max/786/1*M25zPkm5VydP-zqj4bQiAA.jpeg 786w, https://miro.medium.com/max/828/1*M25zPkm5VydP-zqj4bQiAA.jpeg 828w, https://miro.medium.com/max/1400/1*M25zPkm5VydP-zqj4bQiAA.jpeg 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<h1 data-selectable-paragraph="" id="b2ee" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 2</h1>
<p data-selectable-paragraph="" id="ba70" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Let’s build our Python application to acquire data and publish it to Pulsar! We could choose to use many different libraries as Pulsar supports lots of protocols like Websockets, Kafka, MQTT, AMQP, and RocketMQ. To keep things simple and vanilla I am going to use the tried and true Pulsar protocol and the standard Python Pulsar library. I installed the latest version with <span style="box-sizing: inherit; font-weight: 700;"><em style="box-sizing: inherit;">pip3.10 install pulsar-client[all]</em></span>. I did it all since I wanted the FastAvr library, GRPC, Schemas, and other fancy stuff. You can install what you need.</p>
<p data-selectable-paragraph="" id="be86" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The full Python code is <a href="https://github.com/tspannhw/FLiP-Py-ADS-B/blob/main/cloudadsb.py" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">here</a>. I will show you the important bits.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="49bf" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>import pulsar<br style="box-sizing: inherit;">from pulsar import Client, AuthenticationOauth2</span><span data-selectable-paragraph="" id="bab7" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>client = pulsar.Client(service_url, authentication = AuthenticationOauth2(auth_params))</span><span data-selectable-paragraph="" id="8a0d" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>producer = client.create_producer(topic=topic ,properties={"producer-name": "adsb-rest","producer-id": "adsb-py" })</span><span data-selectable-paragraph="" id="0e5d" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>uniqueid = 'thrml_{0}_{1}'.format(randomword(3),strftime("%Y%m%d%H%M%S",gmtime()))</span><span data-selectable-paragraph="" id="2517" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>uuid2 = '{0}_{1}'.format(strftime("%Y%m%d%H%M%S",gmtime()),uuid.uuid4())</span><span data-selectable-paragraph="" id="7e79" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>url_data = "<a href="http://localhost:8080/data/aircraft.json?_=" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">http://localhost:8080/data/aircraft.json?_=</a>" + str(uuid.uuid4())</span><span data-selectable-paragraph="" id="6e45" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>response = json.dumps(requests.get(url_data).json())</span><span data-selectable-paragraph="" id="0ff6" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: 1.91em; white-space: pre-wrap;'>producer.send(response.encode('utf-8'),partition_key=uniqueid)</span></pre>
<p data-selectable-paragraph="" id="59bc" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Then we can run it and start producing raw ADS-B JSON data to our Pulsar topic.</p>
<h1 data-selectable-paragraph="" id="0be2" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 3</h1>
<p data-selectable-paragraph="" id="73bf" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Let’s do a simple check to see if data is coming in.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="9e10" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-client consume "persistent://public/default/adsbraw" -s adsbrawreader -n 0</span></pre>
<p data-selectable-paragraph="" id="ec6c" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>What does this raw data look like?</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="b289" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>{'now': 1659471117.0, 'messages': 7381380, 'aircraft': [{'hex': 'ae6d7a', 'alt_baro': 25000, 'mlat': [], 'tisb': [], 'messages': 177, 'seen': 0.1, 'rssi': -22.7}, {'hex': 'a66174', 'alt_baro': 23000, 'mlat': [], 'tisb': [], 'messages': 5, 'seen': 23.6, 'rssi': -27.8}, .. }</span></pre>
<p data-selectable-paragraph="" id="bb64" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Well, that is a lot of that. I abbreviated it for the sake of scrolling.</p>
<p data-selectable-paragraph="" id="41fc" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Thanks for staying tuned so far, here’s a cat.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 3024px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg" srcset="https://miro.medium.com/max/300/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 300w, https://miro.medium.com/max/640/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 640w, https://miro.medium.com/max/720/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 720w, https://miro.medium.com/max/750/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 750w, https://miro.medium.com/max/786/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 786w, https://miro.medium.com/max/828/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 828w, https://miro.medium.com/max/1400/1*cx4oLJOI0P9OWLxRjNGp6A.jpeg 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'>This cat is mine, this cat is fat. She is okay with that.</figcaption>
<p data-selectable-paragraph="" id="e097" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Raw data is nice and all, I can surely have Apache Spark, Flink, Python or others clean it up. I recommend you set up a Delta Lake, Apache Hudi, or Apache Iceberg sink to store this raw data in your lakehouse if you wish.</p>
<p data-selectable-paragraph="" id="e42e" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Architect Note:</span> You could just keep it in Apache Pulsar forever or in Apache Pulsar-controlled tiered storage.</p>
<h1 data-selectable-paragraph="" id="70b3" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 4</h1>
<p data-selectable-paragraph="" id="7a01" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>I wanted to do this quickly and automagically inside the Pulsar environment so I wrote a quick <span style="box-sizing: inherit; font-weight: 700;">Java</span> Pulsar Function to split, parse, enrich, cleanup, and route data to a new topic. This will be the cleaned data topic. We could have built this function in <span style="box-sizing: inherit; font-weight: 700;">Python</span> or <span style="box-sizing: inherit; font-weight: 700;">Golang</span> as well. I chose Java this time.</p>
<p data-selectable-paragraph="" id="a774" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Yes, we had to build our own function before we could deploy it in Step 4. Let’s take a quick look at our <a href="https://github.com/tspannhw/pulsar-adsb-function" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Java function</a>:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="6572" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>public class ADSBFunction implements Function&lt;byte[], Void&gt; {</span></pre>
<p data-selectable-paragraph="" id="edf8" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>This means our function takes raw bytes but does not return anything. We don’t have an output specified here since I will dynamically decide where to send the output. We could send to any number of topics on the fly.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="46bd" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>context.newOutputMessage(PERSISTENT_PUBLIC_DEFAULT, JSONSchema.of(Aircraft.class)) .key(UUID.randomUUID().toString()) .property(LANGUAGE, JAVA) .value(aircraft).send();</span></pre>
<p data-selectable-paragraph="" id="b12c" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>At the end of my function, I am going to send the data to a topic (could create these on the fly if we want to send them to different topics). We might want to send all of Elon Musk’s flights to a special topic. We could do those lookups with something like <a href="https://github.com/tspannhw/airquality-datastore" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">Scylla</a>, I did that for my Air Quality application. I add a key, add a property, set the schema, and send the data. What is nice here is that I don’t have to use formal language to define the schema. I can just build a plain old Java Bean. Keeping it simple and old school, that works for me.</p>
<p data-selectable-paragraph="" id="cc78" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>In between that code, I have a helper <a href="https://github.com/tspannhw/pulsar-adsb-function/blob/main/src/main/java/dev/pulsarfunction/adsb/AirCraftParserService.java" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">service</a> that parses that jumble of JSON and pulls out the good bits one aircraft event at a time. It’s pretty simple but nice to keep that code inside a simple function that runs on every event or message that enters the ADSBRAW topic. You will need the Java JDK (11 or 17) and Maven. I recommend you utilize <a href="https://sdkman.io/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">SDKMan</a> so you can run multiple JVMS and tools.</p>
<p data-selectable-paragraph="" id="d741" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>To build the function we just need to type:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="12a5" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>mvn package</span></pre>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2325px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/0*Z9a4fNeP6u5PgQve.png" srcset="https://miro.medium.com/max/300/0*Z9a4fNeP6u5PgQve.png 300w, https://miro.medium.com/max/640/0*Z9a4fNeP6u5PgQve.png 640w, https://miro.medium.com/max/720/0*Z9a4fNeP6u5PgQve.png 720w, https://miro.medium.com/max/750/0*Z9a4fNeP6u5PgQve.png 750w, https://miro.medium.com/max/786/0*Z9a4fNeP6u5PgQve.png 786w, https://miro.medium.com/max/828/0*Z9a4fNeP6u5PgQve.png 828w, https://miro.medium.com/max/1400/0*Z9a4fNeP6u5PgQve.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<h1 data-selectable-paragraph="" id="6f44" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 5</h1>
<p data-selectable-paragraph="" id="d34a" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Pulsar makes it easy to add event processing on each topic, so I created a simple one in Java. It’s very easy to deploy, monitor, start, stop, and delete these.</p>
<p data-selectable-paragraph="" id="baba" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s deploy our <a href="https://pulsar.apache.org/docs/functions-develop/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">function</a>:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="0893" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin functions create --auto-ack true --jar /opt/demo/java/pulsar-adsb-function/target/adsb-1.0.jar --<span style="box-sizing: inherit; font-weight: 700;">classname</span> "dev.pulsarfunction.adsb.ADSBFunction" --dead-letter-topic "persistent://public/default/adsbdead" --inputs "persistent://public/default/adsbraw" --log-topic "persistent://public/default/adsblog" --name ADSB --<span style="box-sizing: inherit; font-weight: 700;">namespace</span> default --<span style="box-sizing: inherit; font-weight: 700;">tenant</span> public --max-message-retries 5</span></pre>
<p data-selectable-paragraph="" id="b23a" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>For people new to Apache Pulsar, please note that we have to specify a namespace and tenant for where this will live. This is for discoverability, multitenancy, and just plain cleanliness. We can have as many input topics as we want. Log and Dead letter topics are for special outputs. In this instance, for Java, we have our application stored in a JAR file.</p>
<p data-selectable-paragraph="" id="d189" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Once deployed, let’s check the status:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="46c3" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin functions status --name ADSB</span></pre>
<p data-selectable-paragraph="" id="0852" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The results are JSON, which lends itself to DevOps automation. If we wanted we could administrate this with REST or a DevOps tool.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="c633" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>{<br style="box-sizing: inherit;"> "numInstances" : 1,<br style="box-sizing: inherit;"> "numRunning" : 1,<br style="box-sizing: inherit;"> "instances" : [{<br style="box-sizing: inherit;"> "instanceId" : 0,<br style="box-sizing: inherit;"> "status" : {<br style="box-sizing: inherit;"> "running" : true,<br style="box-sizing: inherit;"> "error" : "",<br style="box-sizing: inherit;"> "numRestarts" : 0,<br style="box-sizing: inherit;"> "numReceived" : 28,<br style="box-sizing: inherit;"> "numSuccessfullyProcessed" : 28,<br style="box-sizing: inherit;"> "numUserExceptions" : 0,<br style="box-sizing: inherit;"> "latestUserExceptions" : [],<br style="box-sizing: inherit;"> "numSystemExceptions" : 0,<br style="box-sizing: inherit;"> "latestSystemExceptions" : [],<br style="box-sizing: inherit;"> "averageLatency" : 144.23374035714286,<br style="box-sizing: inherit;"> "lastInvocationTime" : 1659725881406,<br style="box-sizing: inherit;"> "workerId" : "c-standalone-fw-127.0.0.1-8080"<br style="box-sizing: inherit;"> }<br style="box-sizing: inherit;"> } ]<br style="box-sizing: inherit;">}</span></pre>
<p data-selectable-paragraph="" id="9041" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>If we wanted to stop it:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="c816" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin functions stop --name ADSB --namespace default --tenant public</span></pre>
<p data-selectable-paragraph="" id="2640" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>If we needed to delete it:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="cb2f" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-admin functions delete --name ADSB --namespace default --tenant public</span></pre>
<h1 data-selectable-paragraph="" id="f51b" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 6</h1>
<p data-selectable-paragraph="" id="0177" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Now that the function has processed the data, let’s do a quick check of that clean data.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="67fa" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>bin/pulsar-client consume "persistent://public/default/aircraft" -s "aircraftconsumer" -n 0</span></pre>
<p data-selectable-paragraph="" id="3553" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>An example of a JSON row returned:</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="4b1e" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>----- got message -----<br style="box-sizing: inherit;">key:[c480cd8e-a803-47fe-81b4-aafdec0f6b68], properties:[language=Java], content:{"flight":"N86HZ","category":"A7","emergency":"none","squawk":1200,"hex":"abcd45","gs":52.2,"track":106.7,"lat":40.219757,"lon":-74.580566,"nic":9,"rc":75,"version":2,"sil":3,"gva":2,"sda":2,"mlat":[],"tisb":[],"messages":2259,"seen":1.1,"rssi":-19.9}</span></pre>
<p data-selectable-paragraph="" id="1ef3" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">Architect Note:</span> Always set a key when you produce messages.</p>
<h1 data-selectable-paragraph="" id="c81e" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 7</h1>
<p data-selectable-paragraph="" id="f57a" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>We now have clean data!</p>
<p data-selectable-paragraph="" id="fe36" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s check that data stream with Apache Spark Structured Streaming!</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="5628" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>val dfPulsar = spark.readStream.format("pulsar").option("service.url", "pulsar://pulsar1:6650").option("admin.url", "http://pulsar1:8080").option("topic", "persistent://public/default/aircraft").load()<br style="box-sizing: inherit;"><br style="box-sizing: inherit;">dfPulsar.printSchema()<br style="box-sizing: inherit;">root<br style="box-sizing: inherit;"> |-- altBaro: integer (nullable = true)<br style="box-sizing: inherit;"> |-- altGeom: integer (nullable = true)<br style="box-sizing: inherit;"> |-- baroRate: integer (nullable = true)<br style="box-sizing: inherit;"> |-- category: string (nullable = true)<br style="box-sizing: inherit;"> |-- emergency: string (nullable = true)<br style="box-sizing: inherit;"> |-- flight: string (nullable = true)<br style="box-sizing: inherit;"> |-- gs: double (nullable = true)<br style="box-sizing: inherit;"> |-- gva: integer (nullable = true)<br style="box-sizing: inherit;"> |-- hex: string (nullable = true)<br style="box-sizing: inherit;"> |-- lat: double (nullable = true)<br style="box-sizing: inherit;"> |-- lon: double (nullable = true)<br style="box-sizing: inherit;"> |-- mach: double (nullable = true)<br style="box-sizing: inherit;"> |-- messages: integer (nullable = true)<br style="box-sizing: inherit;"> |-- mlat: array (nullable = true)<br style="box-sizing: inherit;"> | |-- element: struct (containsNull = false)<br style="box-sizing: inherit;"> |-- nacP: integer (nullable = true)<br style="box-sizing: inherit;"> |-- nacV: integer (nullable = true)<br style="box-sizing: inherit;"> |-- navAltitudeMcp: integer (nullable = true)<br style="box-sizing: inherit;"> |-- navHeading: double (nullable = true)<br style="box-sizing: inherit;"> |-- navQnh: double (nullable = true)<br style="box-sizing: inherit;"> |-- nic: integer (nullable = true)<br style="box-sizing: inherit;"> |-- nicBaro: integer (nullable = true)<br style="box-sizing: inherit;"> |-- rc: integer (nullable = true)<br style="box-sizing: inherit;"> |-- rssi: double (nullable = true)<br style="box-sizing: inherit;"> |-- sda: integer (nullable = true)<br style="box-sizing: inherit;"> |-- seen: double (nullable = true)<br style="box-sizing: inherit;"> |-- seenPos: double (nullable = true)<br style="box-sizing: inherit;"> |-- sil: integer (nullable = true)<br style="box-sizing: inherit;"> |-- silType: string (nullable = true)<br style="box-sizing: inherit;"> |-- speed: double (nullable = true)<br style="box-sizing: inherit;"> |-- squawk: integer (nullable = true)<br style="box-sizing: inherit;"> |-- tisb: array (nullable = true)<br style="box-sizing: inherit;"> | |-- element: struct (containsNull = false)<br style="box-sizing: inherit;"> |-- track: double (nullable = true)<br style="box-sizing: inherit;"> |-- version: integer (nullable = true)<br style="box-sizing: inherit;"> |-- __key: binary (nullable = true)<br style="box-sizing: inherit;"> |--__topic: string (nullable = true)<br style="box-sizing: inherit;"> |-- __messageId: binary (nullable = true)<br style="box-sizing: inherit;"> |--__publishTime: timestamp (nullable = true)<br style="box-sizing: inherit;"> |-- __eventTime: timestamp (nullable = true)<br style="box-sizing: inherit;"> |--__messageProperties: map (nullable = true)<br style="box-sizing: inherit;"> | |-- key: string<br style="box-sizing: inherit;"> | |-- value: string (valueContainsNull = true)<br style="box-sizing: inherit;"><br style="box-sizing: inherit;"><br style="box-sizing: inherit;">val pQuery = dfPulsar.selectExpr("*").writeStream.format("console")<br style="box-sizing: inherit;">.option("truncate", false).start()</span></pre>
<p data-selectable-paragraph="" id="27ea" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>The above Spark code connected to the Pulsar cluster grabbed data from our Pulsar topic and had a table built. As you see setting that schema is a great idea. We can then easily query it as a micro-batch and in this case output it to the console for debugging. We could have also sent that stream somewhere else like S3.</p>
<h1 data-selectable-paragraph="" id="148a" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 8</h1>
<p data-selectable-paragraph="" id="8808" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>Let’s run a continuous SQL query with Apache Flink.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="6d13" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>CREATE CATALOG pulsar WITH (<br style="box-sizing: inherit;"> 'type' = 'pulsar',<br style="box-sizing: inherit;"> 'service-url' = 'pulsar://pulsar1:6650',<br style="box-sizing: inherit;"> 'admin-url' = 'http://pulsar1:8080',<br style="box-sizing: inherit;"> 'format' = 'json'<br style="box-sizing: inherit;">);<br style="box-sizing: inherit;"><br style="box-sizing: inherit;">USE CATALOG pulsar;</span></pre>
<p data-selectable-paragraph="" id="a308" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>We create a catalog to connect from Flink to Pulsar.</p>
<p data-selectable-paragraph="" id="8059" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s take a look at our table.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="0bf0" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>describe aircraft;<br style="box-sizing: inherit;">+------------------+-----------------------+------+-----+--------+-----------+<br style="box-sizing: inherit;">| name | type | null | key | extras | watermark |<br style="box-sizing: inherit;">+------------------+-----------------------+------+-----+--------+-----------+<br style="box-sizing: inherit;">| alt_baro | INT | true | | | |<br style="box-sizing: inherit;">| alt_geom | INT | true | | | |<br style="box-sizing: inherit;">| baro_rate | INT | true | | | |<br style="box-sizing: inherit;">| category | STRING | true | | | |<br style="box-sizing: inherit;">| emergency | STRING | true | | | |<br style="box-sizing: inherit;">| flight | STRING | true | | | |<br style="box-sizing: inherit;">| gs | DOUBLE | true | | | |<br style="box-sizing: inherit;">| gva | INT | true | | | |<br style="box-sizing: inherit;">| hex | STRING | true | | | |<br style="box-sizing: inherit;">| lat | DOUBLE | true | | | |<br style="box-sizing: inherit;">| lon | DOUBLE | true | | | |<br style="box-sizing: inherit;">| mach | DOUBLE | true | | | |<br style="box-sizing: inherit;">| messages | INT | true | | | |<br style="box-sizing: inherit;">| mlat | ARRAY&lt;ROW&lt;&gt; NOT NULL&gt; | true | | | |<br style="box-sizing: inherit;">| nac_p | INT | true | | | |<br style="box-sizing: inherit;">| nac_v | INT | true | | | |<br style="box-sizing: inherit;">| nav_altitude_mcp | INT | true | | | |<br style="box-sizing: inherit;">| nav_heading | DOUBLE | true | | | |<br style="box-sizing: inherit;">| nav_qnh | DOUBLE | true | | | |<br style="box-sizing: inherit;">| nic | INT | true | | | |<br style="box-sizing: inherit;">| nic_baro | INT | true | | | |<br style="box-sizing: inherit;">| rc | INT | true | | | |<br style="box-sizing: inherit;">| rssi | DOUBLE | true | | | |<br style="box-sizing: inherit;">| sda | INT | true | | | |<br style="box-sizing: inherit;">| seen | DOUBLE | true | | | |<br style="box-sizing: inherit;">| seen_post | DOUBLE | true | | | |<br style="box-sizing: inherit;">| sil | INT | true | | | |<br style="box-sizing: inherit;">| sil_type | STRING | true | | | |<br style="box-sizing: inherit;">| speed | DOUBLE | true | | | |<br style="box-sizing: inherit;">| squawk | INT | true | | | |<br style="box-sizing: inherit;">| tisb | ARRAY&lt;ROW&lt;&gt; NOT NULL&gt; | true | | | |<br style="box-sizing: inherit;">| track | DOUBLE | true | | | |<br style="box-sizing: inherit;">| version | INT | true | | | |<br style="box-sizing: inherit;">+------------------+-----------------------+------+-----+--------+-----------+<br style="box-sizing: inherit;">33 rows in set</span></pre>
<p data-selectable-paragraph="" id="193b" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Let’s run some simple queries.</p>
<pre style="background: rgb(242, 242, 242); box-sizing: inherit; margin-bottom: 0px; margin-top: 56px; overflow-x: auto; padding: 20px;"><span data-selectable-paragraph="" id="9c13" style='box-sizing: inherit; color: #292929; display: block; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; letter-spacing: -0.022em; line-height: 1.18; margin-bottom: -0.09em; margin-top: -0.09em; white-space: pre-wrap;'>select alt_baro,<br style="box-sizing: inherit;"> gs,<br style="box-sizing: inherit;"> alt_geom,<br style="box-sizing: inherit;"> baro_rate,<br style="box-sizing: inherit;"> mach, <br style="box-sizing: inherit;"> hex, flight, lat, lon<br style="box-sizing: inherit;">from aircraft;<br style="box-sizing: inherit;"><br style="box-sizing: inherit;">select max(alt_baro) as MaxAltitudeFeet, min(alt_baro) as MinAltitudeFeet, avg(alt_baro) as AvgAltitudeFeet,<br style="box-sizing: inherit;"> max(alt_geom) as MaxGAltitudeFeet, min(alt_geom) as MinGAltitudeFeet, avg(alt_geom) as AvgGAltitudeFeet,<br style="box-sizing: inherit;"> max(gs) as MaxGroundSpeed, min(gs) as MinGroundSpeed, avg(gs) as AvgGroundSpeed, <br style="box-sizing: inherit;"> count(alt_baro) as RowCount, <br style="box-sizing: inherit;"> hex as ICAO, flight as IDENT<br style="box-sizing: inherit;">from aircraft <br style="box-sizing: inherit;">group by flight, hex;</span></pre>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0"><div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 2590px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/1*1nPZvJbBIrBChjUHTMj7qA.png" srcset="https://miro.medium.com/max/300/1*1nPZvJbBIrBChjUHTMj7qA.png 300w, https://miro.medium.com/max/640/1*1nPZvJbBIrBChjUHTMj7qA.png 640w, https://miro.medium.com/max/720/1*1nPZvJbBIrBChjUHTMj7qA.png 720w, https://miro.medium.com/max/750/1*1nPZvJbBIrBChjUHTMj7qA.png 750w, https://miro.medium.com/max/786/1*1nPZvJbBIrBChjUHTMj7qA.png 786w, https://miro.medium.com/max/828/1*1nPZvJbBIrBChjUHTMj7qA.png 828w, https://miro.medium.com/max/1400/1*1nPZvJbBIrBChjUHTMj7qA.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div></div>
<p data-selectable-paragraph="" id="6af9" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>We can do basic queries that will return every row when it arrives or aggregate them.</p>
<h1 data-selectable-paragraph="" id="70b8" style='box-sizing: inherit; color: #292929; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 22px; letter-spacing: 0px; line-height: 28px; margin: 3.14em 0px -0.37em;'>Step 9</h1>
<p data-selectable-paragraph="" id="a307" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 0.86em 0px -0.46em; word-break: break-word;'>We did it. Let’s start streaming our little dream stream.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 500px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 500px" src="https://miro.medium.com/max/1000/1*MCPHnm77GaRjQHIURw9c4A.jpeg" srcset="https://miro.medium.com/max/300/1*MCPHnm77GaRjQHIURw9c4A.jpeg 300w, https://miro.medium.com/max/640/1*MCPHnm77GaRjQHIURw9c4A.jpeg 640w, https://miro.medium.com/max/720/1*MCPHnm77GaRjQHIURw9c4A.jpeg 720w, https://miro.medium.com/max/750/1*MCPHnm77GaRjQHIURw9c4A.jpeg 750w, https://miro.medium.com/max/786/1*MCPHnm77GaRjQHIURw9c4A.jpeg 786w, https://miro.medium.com/max/828/1*MCPHnm77GaRjQHIURw9c4A.jpeg 828w, https://miro.medium.com/max/1000/1*MCPHnm77GaRjQHIURw9c4A.jpeg 1000w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 500px;"></div>
<p data-selectable-paragraph="" id="8c4f" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'>Thanks for staying for the entire app building. I hope to see you soon at a meetup or event. Contact <a href="https://twitter.com/paasdev" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">me</a> if you have any questions or are looking for other apps built powered by FLiPN+.</p>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 450px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 450px" src="https://miro.medium.com/max/900/1*fH8xMPjqHxNknPLP4n6YMA.png" srcset="https://miro.medium.com/max/300/1*fH8xMPjqHxNknPLP4n6YMA.png 300w, https://miro.medium.com/max/640/1*fH8xMPjqHxNknPLP4n6YMA.png 640w, https://miro.medium.com/max/720/1*fH8xMPjqHxNknPLP4n6YMA.png 720w, https://miro.medium.com/max/750/1*fH8xMPjqHxNknPLP4n6YMA.png 750w, https://miro.medium.com/max/786/1*fH8xMPjqHxNknPLP4n6YMA.png 786w, https://miro.medium.com/max/828/1*fH8xMPjqHxNknPLP4n6YMA.png 828w, https://miro.medium.com/max/900/1*fH8xMPjqHxNknPLP4n6YMA.png 900w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 450px;"></div>
<figcaption data-selectable-paragraph="" style='box-sizing: inherit; color: #757575; font-family: sohne, "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: auto; margin-right: auto; margin-top: 10px; max-width: 728px; text-align: center;'><a href="https://github.com/tspannhw/SpeakerProfile" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/SpeakerProfile</a></figcaption>
<p data-selectable-paragraph="" id="8b0b" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 32px; margin: 2em 0px -0.46em; word-break: break-word;'><span style="box-sizing: inherit; font-weight: 700;">For additional resources to build more advanced apps:</span></p>
<ul style="box-sizing: inherit; list-style: none none; margin: 0px; padding: 0px;">
<li data-selectable-paragraph="" id="38ac" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 2.14em; padding-left: 0px;'><a href="https://flightaware.com/adsb/piaware/build/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://flightaware.com/adsb/piaware/build/</a></li>
<li data-selectable-paragraph="" id="c2af" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/adsbxchange/dump1090-fa" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/adsbxchange/dump1090-fa</a></li>
<li data-selectable-paragraph="" id="b5bb" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://flightaware.com/adsb/piaware/build" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://flightaware.com/adsb/piaware/build</a></li>
<li data-selectable-paragraph="" id="1c75" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://flightaware.com/adsb/piaware/build/optional#piawarecommands" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://flightaware.com/adsb/piaware/build/optional#piawarecommands</a></li>
<li data-selectable-paragraph="" id="3539" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://dzone.com/articles/ingesting-flight-data-ads-b-usb-receiver-with-apac" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://dzone.com/articles/ingesting-flight-data-ads-b-usb-receiver-with-apac</a></li>
<li data-selectable-paragraph="" id="386e" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://globe.adsbexchange.com/" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://globe.adsbexchange.com/</a></li>
<li data-selectable-paragraph="" id="7d06" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940</a></li>
<li data-selectable-paragraph="" id="efad" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://elmwoodelectronics.ca/blogs/news/tracking-and-logging-flights-with-ads-b-flight-aware-and-raspberry-pi" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://elmwoodelectronics.ca/blogs/news/tracking-and-logging-flights-with-ads-b-flight-aware-and-raspberry-pi</a></li>
<li data-selectable-paragraph="" id="6c31" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/flightaware/piaware" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/flightaware/piaware</a></li>
<li data-selectable-paragraph="" id="c94b" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/erikbeebe/flink_adsb_stream" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/erikbeebe/flink_adsb_stream</a></li>
<li data-selectable-paragraph="" id="ba2f" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/erikbeebe/flink_adsb_querystate" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/erikbeebe/flink_adsb_querystate</a></li>
<li data-selectable-paragraph="" id="a549" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/openskynetwork/java-adsb" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/openskynetwork/java-adsb</a></li>
<li data-selectable-paragraph="" id="45c8" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://openskynetwork.github.io/opensky-api/rest.html" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://openskynetwork.github.io/opensky-api/rest.html</a></li>
<li data-selectable-paragraph="" id="68ef" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/openskynetwork/opensky-api" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/openskynetwork/opensky-api</a></li>
<li data-selectable-paragraph="" id="1138" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://opensky-network.org/my-opensky" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://opensky-network.org/my-opensky</a></li>
<li data-selectable-paragraph="" id="e603" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://dev.to/tspannhw/tracking-air-quality-with-apache-nifi-cloudera-data-science-workbench-pyspark-and-parquet-28c" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://dev.to/tspannhw/tracking-air-quality-with-apache-nifi-cloudera-data-science-workbench-pyspark-and-parquet-28c</a></li>
<li data-selectable-paragraph="" id="0e69" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://openskynetwork.github.io/opensky-api/rest.html" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://openskynetwork.github.io/opensky-api/rest.html</a></li>
<li data-selectable-paragraph="" id="a5a2" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/tspannhw/airquality" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/tspannhw/airquality</a></li>
<li data-selectable-paragraph="" id="9046" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://www.faa.gov/air_traffic/publications/atpubs/atc_html/chap5_section_2.html" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://www.faa.gov/air_traffic/publications/atpubs/atc_html/chap5_section_2.html</a></li>
<li data-selectable-paragraph="" id="34fc" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://aviation.stackexchange.com/questions/60747/what-are-all-the-squawk-codes" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://aviation.stackexchange.com/questions/60747/what-are-all-the-squawk-codes</a></li>
<li data-selectable-paragraph="" id="1d38" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://en.wikipedia.org/wiki/List_of_transponder_codes" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://en.wikipedia.org/wiki/List_of_transponder_codes</a></li>
<li data-selectable-paragraph="" id="0540" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://github.com/briantwalter/spark1090" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://github.com/briantwalter/spark1090</a></li>
<li data-selectable-paragraph="" id="ecfe" style='box-sizing: inherit; color: #292929; font-family: charter, Georgia, Cambria, "Times New Roman", Times, serif; font-size: 20px; letter-spacing: -0.003em; line-height: 28px; list-style-type: disc; margin-bottom: -0.46em; margin-left: 30px; margin-top: 1.14em; padding-left: 0px;'><a href="https://youtu.be/0LMOglFN8ZA" rel="noopener ugc nofollow" style="-webkit-tap-highlight-color: transparent; box-sizing: inherit;" target="_blank">https://youtu.be/0LMOglFN8ZA</a></li>
</ul>
<p style="box-sizing: inherit; clear: both; margin: 56px auto 0px;"></p>
<div role="button" style="box-sizing: inherit; cursor: zoom-in; position: relative; transition: transform 300ms cubic-bezier(0.2, 0, 0.2, 1) 0s; width: 692px; z-index: auto;" tabindex="0">
<div style="box-sizing: inherit; margin-left: auto; margin-right: auto; max-width: 1441px;"><img alt="" role="presentation" sizes="(max-width: 700px) 100vw, 700px" src="https://miro.medium.com/max/1400/0*2nnEumz4fhjSCw4M.png" srcset="https://miro.medium.com/max/300/0*2nnEumz4fhjSCw4M.png 300w, https://miro.medium.com/max/640/0*2nnEumz4fhjSCw4M.png 640w, https://miro.medium.com/max/720/0*2nnEumz4fhjSCw4M.png 720w, https://miro.medium.com/max/750/0*2nnEumz4fhjSCw4M.png 750w, https://miro.medium.com/max/786/0*2nnEumz4fhjSCw4M.png 786w, https://miro.medium.com/max/828/0*2nnEumz4fhjSCw4M.png 828w, https://miro.medium.com/max/1400/0*2nnEumz4fhjSCw4M.png 1400w" style="box-sizing: inherit; height: auto; max-width: 100%; vertical-align: middle; width: 692px;"></div>
<div><br></div>
</div>
</div>
</div></section>