Real-Time Irish Transit Analytics
Tags: apachekafka, apachenifi, json, realtime
Published: 2024-03-28 13:09:49

---
title: Real-Time Irish Transit Analytics
published: true
date: 2024-03-28 13:03:00 UTC
tags: apachekafka,apachenifi,json,realtime
canonical_url: 
---

 

<section name="880f"><div><hr></div>
<div><div>
<h3 name="f958">Real-Time Irish Transit Analytics</h3>
<p name="f932">Apache NiFi, Postgresql, GenAI, Apache Kafka, Apache Flink, JSON, GTFS</p>
<p name="f0a2"><img data-height="1024" data-image-id="0*UT2QFs9KXMEpwbCy" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/0*UT2QFs9KXMEpwbCy"></p>
<p name="0882">Let’s hop on a bus in Ireland!</p>
<p name="67de">We need to load static (rarely changing lookup data). We can do this with NiFi very easily. We build and insert these into new Postgresql tables.</p>
<p name="742f">See me here:</p>
<p name="2896"><img data-height="615" data-image-id="1*7bKSJRW0jBD4is4PzI4fJw.png" data-width="861" src="https://cdn-images-1.medium.com/max/2000/1*7bKSJRW0jBD4is4PzI4fJw.png"></p>
<h3 name="df87">ChatGPT Authored Introduction:</h3>
<p name="fc35"><strong>Unlocking the Future of Transportation: Real-Time Irish Transit Analytics</strong></p>
<p name="e6a4">In the bustling landscape of modern transportation, the ability to harness real-time data is not just a competitive advantage; it’s a necessity. In Ireland, where efficient transit systems are the lifeblood of daily commutes and city connectivity, the fusion of cutting-edge technologies is revolutionizing how we understand and optimize public transportation. This article delves into the world of Real-Time Irish Transit Analytics, where Apache NiFi, PostgreSQL, GenAI, Apache Kafka, Apache Flink, JSON, and GTFS converge to create a dynamic and responsive ecosystem.</p>
<p name="c3fb">Every day, thousands of passengers rely on Ireland’s public transit systems to navigate cities, reach work, or simply explore the beauty of the countryside. Yet, behind the scenes of this seemingly seamless operation lies a complex network of data streams, from vehicle locations to passenger counts, schedules to service updates. Here, Apache NiFi emerges as a pivotal tool, seamlessly orchestrating the flow of data from various sources into a unified pipeline.</p>
<p name="c783">PostgreSQL steps in as the reliable database backbone, providing a robust foundation for storing and querying vast amounts of transit data. With the power of GenAI, machine learning algorithms sift through this data trove, uncovering valuable insights into passenger behaviors, traffic patterns, and optimal routes.</p>
<p name="c598">But data is only as valuable as its timeliness, and this is where Apache Kafka and Apache Flink shine. Kafka acts as the real-time messaging hub, ensuring that updates from buses, trains, and stations are instantly propagated through the system. Flink’s stream processing capabilities then come into play, analyzing incoming data on the fly to generate actionable intelligence.</p>
<p name="ac29">In the realm of data interchange, JSON (JavaScript Object Notation) emerges as the lingua franca, facilitating seamless communication between different components of the analytics ecosystem. And anchoring it all is the General Transit Feed Specification (GTFS), a standardized format for public transit schedules and geographic information, ensuring interoperability and accuracy across the board.</p>
<p name="2178">Join us on a journey through the intricacies of Real-Time Irish Transit Analytics, where these technologies converge to enhance efficiency, improve passenger experiences, and pave the way for the future of smart transportation.</p>
<p name="d1d5">An important source of data is the static GTFS lookup tables provided a zip file of CSV. We can download and parse this automagically in NiFi. No need to know and precreate tables. NiFi will determine the fields for you.</p>
<p name="b63e"><a data-href="https://www.transportforireland.ie/transitData/Data/GTFS_Realtime.zip" href="https://www.transportforireland.ie/transitData/Data/GTFS_Realtime.zip" rel="nofollow noopener noopener" target="_blank">https://www.transportforireland.ie/transitData/Data/GTFS_Realtime.zip</a></p>
<h3 name="e51f">GTFS Static Data Load</h3>
<p name="4754"><img data-height="1661" data-image-id="1*etzpO4YvzDBqMOwb2tbJbQ.png" data-width="1606" src="https://cdn-images-1.medium.com/max/2000/1*etzpO4YvzDBqMOwb2tbJbQ.png"></p>
<h3 name="b659">Skip shapes.txt as we aren’t loading those</h3>
<p name="e44d"><img data-height="553" data-image-id="1*HjO235JbmyettBamzvkt_A.png" data-width="1285" src="https://cdn-images-1.medium.com/max/2000/1*HjO235JbmyettBamzvkt_A.png"></p>
<h3 name="e5ca">Set a Default Primary Key</h3>
<p name="46a3"><img data-height="1120" data-image-id="1*gp0ix_aNJBruyghwQoFBWQ.jpeg" data-width="1725" src="https://cdn-images-1.medium.com/max/2000/1*gp0ix_aNJBruyghwQoFBWQ.jpeg"></p>
<h3 name="b184">Setting All the Correct Primary Keys for all the Static Files/Tables</h3>
<p name="95c0"><img data-height="1789" data-image-id="1*LdsevuwPPwP3nXvDC3yMOQ.png" data-width="3344" src="https://cdn-images-1.medium.com/max/2000/1*LdsevuwPPwP3nXvDC3yMOQ.png"></p>
<p name="88c5"><img data-height="1841" data-image-id="1*AduwqZCsDI9eSx9nd23R_A.png" data-width="3372" src="https://cdn-images-1.medium.com/max/2000/1*AduwqZCsDI9eSx9nd23R_A.png"></p>
<p name="ea83"><img data-height="592" data-image-id="1*0GWd98wlJTDV8_VC_49N8g.png" data-width="3331" src="https://cdn-images-1.medium.com/max/2000/1*0GWd98wlJTDV8_VC_49N8g.png"></p>
<h3 name="f5ef">Split Up Tables into 1,000 Row Chunks to Make it Easier for Postgresql</h3>
<p name="740c"><img data-height="565" data-image-id="1*nVYNUkkTbuvTf7Flmuh3ew.png" data-width="1193" src="https://cdn-images-1.medium.com/max/2000/1*nVYNUkkTbuvTf7Flmuh3ew.png"><figcaption>We converted CSV to JSON and split up in 1 step</figcaption></p>
<p name="67f9"><img data-height="1649" data-image-id="1*MtSTW_IlsCh78ToIt8_FWw.png" data-width="3297" src="https://cdn-images-1.medium.com/max/2000/1*MtSTW_IlsCh78ToIt8_FWw.png"><figcaption>Loaded Results</figcaption></p>
<h3 name="cd09">Update the SQL Automagically</h3>
<p name="b369"><img data-height="1044" data-image-id="1*dOwKPzW7Rln-yhHF8elcUQ.png" data-width="1405" src="https://cdn-images-1.medium.com/max/2000/1*dOwKPzW7Rln-yhHF8elcUQ.png"><figcaption>we do not manually set field names, no SQL injection here</figcaption></p>
<h3 name="94eb">Send this SQL to the Database</h3>
<p name="4a65"><img data-height="1122" data-image-id="1*PoBX8SECiN9672ifM4Dqlw.png" data-width="1632" src="https://cdn-images-1.medium.com/max/2000/1*PoBX8SECiN9672ifM4Dqlw.png"></p>
<h3 name="ccdc">A list of Ireland Lookup Trips loaded from trips.txt</h3>
<p name="62c5"><img data-height="1900" data-image-id="1*rkdMZg2nsir0dvoWn-l5Iw.png" data-width="2638" src="https://cdn-images-1.medium.com/max/2000/1*rkdMZg2nsir0dvoWn-l5Iw.png"></p>
<p name="9a5a"><img data-height="680" data-image-id="1*1al7HBA3k66M7LeNegSGBQ.png" data-width="2545" src="https://cdn-images-1.medium.com/max/2000/1*1al7HBA3k66M7LeNegSGBQ.png"></p>
<p name="2f47"><img data-height="1726" data-image-id="1*z53aKzp1p0kOXaWLdAUQEQ.png" data-width="2370" src="https://cdn-images-1.medium.com/max/2000/1*z53aKzp1p0kOXaWLdAUQEQ.png"></p>
<p name="22fd"><img data-height="1738" data-image-id="1*QVqEZtX9b4kca3nDUx25Yw.png" data-width="2657" src="https://cdn-images-1.medium.com/max/2000/1*QVqEZtX9b4kca3nDUx25Yw.png"></p>
<p name="05b0"><img data-height="1922" data-image-id="1*K2M8MtyfdomLR6tBvOFxsA.png" data-width="2654" src="https://cdn-images-1.medium.com/max/2000/1*K2M8MtyfdomLR6tBvOFxsA.png"></p>
<p name="086c">Let’s parse the real time transit information for Ireland.</p>
<h3 name="0656">GTFS Real-Time</h3>
<p name="d094">Vehicle Positions is the primary API to get where the buses are.</p>
<p name="c862"><img data-height="1717" data-image-id="1*_9z1G6q7K-C1sUKpjeYNPg.png" data-width="2279" src="https://cdn-images-1.medium.com/max/2000/1*_9z1G6q7K-C1sUKpjeYNPg.png"></p>
<p name="4ec5"><img data-height="951" data-image-id="1*8OvdSapdc6zeqPsBxI2xdA.png" data-width="712" src="https://cdn-images-1.medium.com/max/2000/1*8OvdSapdc6zeqPsBxI2xdA.png"></p>
<p name="7f30"><img data-height="1664" data-image-id="1*2-sFYi07YYGvRoWRTg_j9Q.png" data-width="1554" src="https://cdn-images-1.medium.com/max/2000/1*2-sFYi07YYGvRoWRTg_j9Q.png"></p>
<p name="dbf7"><img data-height="1539" data-image-id="1*49y7FltNDD_gl01XpAIPNw.png" data-width="2714" src="https://cdn-images-1.medium.com/max/2000/1*49y7FltNDD_gl01XpAIPNw.png"></p>
<p name="26ba"><img data-height="1755" data-image-id="1*CEb9eyE_L-Reej5FDe3GOQ.png" data-width="2833" src="https://cdn-images-1.medium.com/max/2000/1*CEb9eyE_L-Reej5FDe3GOQ.png"></p>
<h3 name="311d">API REST TEST</h3>
<p name="98ab"><img data-height="1268" data-image-id="1*9K_NU0DRO5bL4rErbONneQ.png" data-width="2525" src="https://cdn-images-1.medium.com/max/2000/1*9K_NU0DRO5bL4rErbONneQ.png"></p>
<div name="d828">
<a data-href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=gtfsr-v2" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=gtfsr-v2" title="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=gtfsr-v2"><strong>APIs: Details</strong><br><em>Discover APIs, learn how to use them, try them out interactively, and sign up to acquire keys.</em>developer.nationaltransport.ie</a><a data-media-id="86bad833c3c36201d38ce7e405c50c82" data-thumbnail-img-id="0*8XfI-llry_ruNASn" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=gtfsr-v2" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*8XfI-llry_ruNASn);"></a>
</div>
<pre data-code-block-lang="bash" data-code-block-mode="2" name="63df" spellcheck="false"><span>GET https://api.nationaltransport.ie/gtfsr/v2/gtfsr?format=json HTTP/1.1<br><br>Cache-Control: no-cache<br><br>x-api-key: dddddd</span></pre>
<p name="e723">As opposed to most transit systems we have seen in GTFS and GTFS-R feeds they don’t have three types, just the two. They are missing alerts.</p>
<p name="8bf3"><strong>[Trip Updates, Vehicle Positions]</strong></p>
<div name="481e">
<a data-href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=TripUpdates" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=TripUpdates" title="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=TripUpdates"><strong>APIs: Details</strong><br><em>Discover APIs, learn how to use them, try them out interactively, and sign up to acquire keys.</em>developer.nationaltransport.ie</a><a data-media-id="7624b0a1df42245dbd5b4b9c577f82f9" data-thumbnail-img-id="0*IRy_V9K8t6_Mgtd2" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=TripUpdates" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*IRy_V9K8t6_Mgtd2);"></a>
</div>
<div name="1879">
<a data-href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=vehicles" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=vehicles" title="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=vehicles"><strong>APIs: Details</strong><br><em>Discover APIs, learn how to use them, try them out interactively, and sign up to acquire keys.</em>developer.nationaltransport.ie</a><a data-media-id="6abe13e07c0dc6b8822b4a32e301a8dc" data-thumbnail-img-id="0*TxG-5Stgb6JUsImJ" href="https://developer.nationaltransport.ie/api-details#api=gtfsr&amp;operation=vehicles" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*TxG-5Stgb6JUsImJ);"></a>
</div>
<blockquote name="7dc5">The GTFS-R API contains real-time updates for services provided by Dublin Bus, Bus Éireann, and Go-Ahead Ireland.</blockquote>
<p name="7d0b">You have to sign up and subscribe to the API to use this.</p>
<p name="1b69"><img data-height="646" data-image-id="1*lQNIn1rS_zsB1iwZmh5u4w.png" data-width="1367" src="https://cdn-images-1.medium.com/max/2000/1*lQNIn1rS_zsB1iwZmh5u4w.png"></p>
<p name="3ed8"><img data-height="1004" data-image-id="1*i32Zi5r62V9oYwWYH9U8Fg.png" data-width="1145" src="https://cdn-images-1.medium.com/max/2000/1*i32Zi5r62V9oYwWYH9U8Fg.png"><figcaption>x-api-key is the header for our private key</figcaption></p>
<h3 name="1dbc">Example Vehicle Position as JSON</h3>
<pre data-code-block-lang="json" data-code-block-mode="1" name="64df" spellcheck="false"><span><span>[</span> <span>{</span><br> <span>"recordid"</span> <span>:</span> <span>"V56"</span><span>,</span><br> <span>"route_id"</span> <span>:</span> <span>"3924_62692"</span><span>,</span><br> <span>"directionid"</span> <span>:</span> <span>"0"</span><span>,</span><br> <span>"latitude"</span> <span>:</span> <span>"53.3537788"</span><span>,</span><br> <span>"tripid"</span> <span>:</span> <span>"3924_16321"</span><span>,</span><br> <span>"starttime"</span> <span>:</span> <span>"22:50:00"</span><span>,</span><br> <span>"vehicleid"</span> <span>:</span> <span>"274"</span><span>,</span><br> <span>"startdate"</span> <span>:</span> <span>"20240322"</span><span>,</span><br> <span>"uuid"</span> <span>:</span> <span>"8a50c084-0aea-496e-b4c3-dbed373e812e"</span><span>,</span><br> <span>"longitude"</span> <span>:</span> <span>"-6.40118694"</span><span>,</span><br> <span>"timestamp"</span> <span>:</span> <span>"1711150967"</span><span>,</span><br> <span>"ts"</span> <span>:</span> <span>"1711167213555"</span><br><span>}</span> <span>]</span></span></pre>
<h3 name="858b">Vehicle Position Slack Message</h3>
<pre data-code-block-lang="bash" data-code-block-mode="1" name="eb14" spellcheck="false"><span>Irish Transit Tracking<br>Direction <span>${directionid}</span><br>Request <span>${invokehttp.request.url}</span> <span>${invokehttp.status.message}</span> <span>${invokehttp.tx.id}</span><br>Lat/Long <span>${latitude}</span>/<span>${longitude}</span><br>Vehicle <span>${vehicleid}</span><br>Route <span>${route_id}</span><br>Scheduled? <span>${scheduled}</span><br>Start Date/Time/TS <span>${startdate}</span> / <span>${starttime}</span> / <span>${timestamp}</span><br>IDs <span>${uuid}</span> <span>${recordid}</span> TripID <span>${tripid}</span><br>Scheduled: <span>${scheduled}</span></span></pre>
</div></div></section><section name="9cd1"><div><hr></div>
<div><div>
<h3 name="06f8">Trip Updates</h3>
<p name="4797"><img data-height="1720" data-image-id="1*c9q8jnqskiF9PQ44fhBaGw.png" data-width="2264" src="https://cdn-images-1.medium.com/max/2000/1*c9q8jnqskiF9PQ44fhBaGw.png"></p>
<p name="c8ef"><img data-height="1750" data-image-id="1*HCFnAgGTjoIdCuOO4ah-Xw.png" data-width="3240" src="https://cdn-images-1.medium.com/max/2000/1*HCFnAgGTjoIdCuOO4ah-Xw.png"></p>
<p name="e7dd"><img data-height="1685" data-image-id="1*nKrv9W7xoZphEjl9RKMF2g.png" data-width="2081" src="https://cdn-images-1.medium.com/max/2000/1*nKrv9W7xoZphEjl9RKMF2g.png"></p>
<p name="eaee"><img data-height="1454" data-image-id="1*4_S5tsmxjVCKLeYE-9LbyA.png" data-width="2441" src="https://cdn-images-1.medium.com/max/2000/1*4_S5tsmxjVCKLeYE-9LbyA.png"></p>
<h3 name="1a9d">Example Trip Update as JSON</h3>
<pre data-code-block-lang="json" data-code-block-mode="1" name="fb24" spellcheck="false"><span><span>{</span><br> <span>"triptimestamp"</span> <span>:</span> <span>"1711415067"</span><span>,</span><br> <span>"stopsequence"</span> <span>:</span> <span>"10"</span><span>,</span><br> <span>"schedulerelationship"</span> <span>:</span> <span>"SCHEDULED"</span><span>,</span><br> <span>"tripstarttime"</span> <span>:</span> <span>"21:30:00"</span><span>,</span><br> <span>"stopid"</span> <span>:</span> <span>"8530B1520901"</span><span>,</span><br> <span>"departuredelay"</span> <span>:</span> <span>"-104"</span><span>,</span><br> <span>"tripid"</span> <span>:</span> <span>"3950_45558"</span><span>,</span><br> <span>"tripschedulerelationship"</span> <span>:</span> <span>"SCHEDULED"</span><span>,</span><br> <span>"tripstartdate"</span> <span>:</span> <span>"20240325"</span><span>,</span><br> <span>"uuid"</span> <span>:</span> <span>"46595e37-4fdd-48db-8431-216bcabe4dd7"</span><span>,</span><br> <span>"departuretime"</span> <span>:</span> <span>""</span><span>,</span><br> <span>"tripdirectionid"</span> <span>:</span> <span>"0"</span><span>,</span><br> <span>"arrivaltime"</span> <span>:</span> <span>""</span><span>,</span><br> <span>"arrivaldelay"</span> <span>:</span> <span>"-104"</span><span>,</span><br> <span>"triprouteid"</span> <span>:</span> <span>"3950_62756"</span><span>,</span><br> <span>"ts"</span> <span>:</span> <span>"1711476673867"</span><span>,</span><br> <span>"route_long_name"</span> <span>:</span> <span>"Dublin - Airport - Cavan - Donegal"</span><span>,</span><br> <span>"stop_name"</span> <span>:</span> <span>"Topaz Belleek"</span><br><span>}</span></span></pre>
<h3 name="c871">Trip Update Slack Message</h3>
<pre data-code-block-lang="bash" data-code-block-mode="1" name="5203" spellcheck="false"><span>Irish Transit Tracking Trip Updates<br>Request <span>${invokehttp.request.url}</span> <span>${invokehttp.status.message}</span> <span>${invokehttp.tx.id}</span><br>IDs <span>${uuid}</span> <br>Arrival Delay / Time: <span>${arrivaldelay}</span> / <span>${arrivaltime}</span><br>Departure Delay / Time: <span>${departuredelay}</span> / <span>${departuretime}</span><br>Schedule: <span>${schedulerelationship}</span> <span>${tripschedulerelationship}</span><br>Stop ID/Sequence: <span>${stopid}</span> / <span>${stopsequence}</span><br>Trip Direction: <span>${tripdirectionid}</span> <span>${tripid}</span><br>Trip Route: <span>${triprouteid}</span><br>Trip Start Date / Time / TS: <span>${tripstartdate}</span> / <span>${tripstarttime}</span> / <span>${triptimestamp}</span></span></pre>
<h3 name="600a">Create Table in Flink</h3>
<p name="20cf"><img data-height="1044" data-image-id="1*B0D7V6uw7cyBrg3HF1DR3g.png" data-width="1794" src="https://cdn-images-1.medium.com/max/2000/1*B0D7V6uw7cyBrg3HF1DR3g.png"></p>
<p name="4580"><strong>Query Kafka Topic — Flink SQL Table in SSB</strong></p>
<p name="658e"><img data-height="919" data-image-id="1*jNHSaphzF7sXJWCkez1VUA.png" data-width="2296" src="https://cdn-images-1.medium.com/max/2000/1*jNHSaphzF7sXJWCkez1VUA.png"></p>
<div name="dfe9">
<a data-href="https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html" href="https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html" title="https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html"><strong>Installing CSP Community Edition</strong><br><em>You need to access the Downloads Page of Cloudera Stream Processing (CSP) to download the Community Edition version of…</em>docs.cloudera.com</a><a data-media-id="261a500b7d919599df4acba55e37906f" href="https://docs.cloudera.com/csp-ce/latest/installation/topics/csp-ce-installing-ce.html"></a>
</div>
<p name="e2af"><img data-height="712" data-image-id="1*jTdeB0xSpMzg9xCjlQQx8w.png" data-width="2067" src="https://cdn-images-1.medium.com/max/2000/1*jTdeB0xSpMzg9xCjlQQx8w.png"></p>
<p name="8f89"><img data-height="510" data-image-id="1*EjUVIjFJoyGE36tw0-qvSA.png" data-width="2061" src="https://cdn-images-1.medium.com/max/2000/1*EjUVIjFJoyGE36tw0-qvSA.png"></p>
<h3 name="9480">Send Messages</h3>
<p name="b229"><img data-height="783" data-image-id="1*3jS9-Jn3Gsr7bs1v8F52TQ.png" data-width="1430" src="https://cdn-images-1.medium.com/max/2000/1*3jS9-Jn3Gsr7bs1v8F52TQ.png"></p>
<h3 name="faf0">Lookups From Postgresql Table</h3>
<p name="618a"><img data-height="333" data-image-id="1*fWrBVfGmJvtqgAb9o4Ea0w.png" data-width="2263" src="https://cdn-images-1.medium.com/max/2000/1*fWrBVfGmJvtqgAb9o4Ea0w.png"></p>
<h3 name="5d99">Finally Send Messages to Slack</h3>
<p name="021f"><img data-height="826" data-image-id="1*CrBTKuIzwGl6STcgm0kqLg.png" data-width="2967" src="https://cdn-images-1.medium.com/max/2000/1*CrBTKuIzwGl6STcgm0kqLg.png"></p>
<p name="bd6e"><img data-height="800" data-image-id="1*aYRawMZHGPRAV00VGYZJrw.png" data-width="2916" src="https://cdn-images-1.medium.com/max/2000/1*aYRawMZHGPRAV00VGYZJrw.png"></p>
<p name="2dd0"></p>
<div>
<div style="padding-bottom: 56.2%;"></div>
<div><iframe allowfullscreen="" data-height="480" data-media-id="f2b5b9a652ea761ad1f9d8cfc6a9c250" data-thumbnail="https://i.embed.ly/1/image?url=http%3A%2F%2Fi.ytimg.com%2Fvi%2FEIpH7RPO2Yo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07" data-width="854" frameborder="0" height="393" src="/media/f2b5b9a652ea761ad1f9d8cfc6a9c250" width="700"></iframe></div>
</div>
<p name="3c2e"><img data-height="1024" data-image-id="0*aLLwOQ15mdxr8bAn" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/0*aLLwOQ15mdxr8bAn"></p>
<h3 name="cf9f">NATIONAL ROADS WEATHER STATION</h3>
<div name="7be4">
<a data-href="https://data.gov.ie/dataset/national-roads-weather-station-data/resource/97c486e0-c1e5-4308-a30a-a93bc33ca0be" href="https://data.gov.ie/dataset/national-roads-weather-station-data/resource/97c486e0-c1e5-4308-a30a-a93bc33ca0be" title="https://data.gov.ie/dataset/national-roads-weather-station-data/resource/97c486e0-c1e5-4308-a30a-a93bc33ca0be"><strong>National Roads Weather Station Data - Weather Data - metadata - data.gov.ie</strong><br><em>Real-time data from TII's national network of 80+ weather stations. Includes air temperature, precipitation, wind speed…</em>data.gov.ie</a><a data-media-id="4f213058524ca70a22bcccc07d472a36" data-thumbnail-img-id="0*mtCykVZBlRoqUjmp" href="https://data.gov.ie/dataset/national-roads-weather-station-data/resource/97c486e0-c1e5-4308-a30a-a93bc33ca0be" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*mtCykVZBlRoqUjmp);"></a>
</div>
<p name="ae29"><img data-height="1024" data-image-id="0*p0i6kVzmhJNW_Khi" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/0*p0i6kVzmhJNW_Khi"></p>
<h3 name="72c5">PUBLIC TRANSPORT DATA</h3>
<div name="e269">
<a data-href="https://www.transportforireland.ie/transitData/PT_Data.html" href="https://www.transportforireland.ie/transitData/PT_Data.html" title="https://www.transportforireland.ie/transitData/PT_Data.html"><strong>GTFS | Transport for Ireland</strong><br><em>Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user…</em>www.transportforireland.ie</a><a data-media-id="c4bfe6bbc898e5144d065933ef21fefa" href="https://www.transportforireland.ie/transitData/PT_Data.html"></a>
</div>
<h3 name="f4b8">LOOKUP DATA FROM GTFS</h3>
<div name="f761">
<a data-href="https://gtfs.org/schedule/reference/#stopstxt" href="https://gtfs.org/schedule/reference/#stopstxt" title="https://gtfs.org/schedule/reference/#stopstxt"><strong>Reference - General Transit Feed Specification</strong><br><em>This document defines the format and structure of the files that comprise a GTFS dataset. The key words "MUST", "MUST…</em>gtfs.org</a><a data-media-id="b2d11783722f41ba72fc2a13fb6140da" href="https://gtfs.org/schedule/reference/#stopstxt"></a>
</div>
<p name="04c8"><code>stop_id</code>Unique ID</p>
<div name="f80c">
<a data-href="https://gtfs.org/schedule/reference/#stop_timestxt" href="https://gtfs.org/schedule/reference/#stop_timestxt" title="https://gtfs.org/schedule/reference/#stop_timestxt"><strong>Reference - General Transit Feed Specification</strong><br><em>This document defines the format and structure of the files that comprise a GTFS dataset. The key words "MUST", "MUST…</em>gtfs.org</a><a data-media-id="3a24e4a2547dec698c9700444514ef0f" href="https://gtfs.org/schedule/reference/#stop_timestxt"></a>
</div>
<p name="57c3">Primary key (<code>trip_id</code>, <code>stop_sequence</code>)</p>
<div name="e0aa">
<a data-href="https://gtfs.org/schedule/reference/#feed_infotxt" href="https://gtfs.org/schedule/reference/#feed_infotxt" title="https://gtfs.org/schedule/reference/#feed_infotxt"><strong>Reference - General Transit Feed Specification</strong><br><em>This document defines the format and structure of the files that comprise a GTFS dataset. The key words "MUST", "MUST…</em>gtfs.org</a><a data-media-id="a5ed68fabe7d0fb727c516eb15b19cdf" href="https://gtfs.org/schedule/reference/#feed_infotxt"></a>
</div>
<h3 name="7329">DUBLIN BIKES</h3>
<div name="0ce8">
<a data-href="https://data.smartdublin.ie/dublinbikes-api" href="https://data.smartdublin.ie/dublinbikes-api" title="https://data.smartdublin.ie/dublinbikes-api"><strong>data.smartdublin.ie</strong><br><em>Edit description</em>data.smartdublin.ie</a><a data-media-id="52b374efbceb160e7ff269cbda768e92" href="https://data.smartdublin.ie/dublinbikes-api"></a>
</div>
<h3 name="8b08">RAILROAD</h3>
<p name="7f99"><img data-height="1024" data-image-id="1*Fe_JkDcJ5dv0JfsvZxzzvw.jpeg" data-is-featured="true" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/1*Fe_JkDcJ5dv0JfsvZxzzvw.jpeg"></p>
<p name="6c60"><img data-height="1679" data-image-id="1*r5L8UB7YSqFwhbC8b25KZA.png" data-width="2536" src="https://cdn-images-1.medium.com/max/2000/1*r5L8UB7YSqFwhbC8b25KZA.png"></p>
<p name="6727"><img data-height="1651" data-image-id="1*_X12QljW9tl-CJaprpbAFA.png" data-width="2311" src="https://cdn-images-1.medium.com/max/2000/1*_X12QljW9tl-CJaprpbAFA.png"></p>
<p name="ff8e"><img data-height="1678" data-image-id="1*_E-SsoYC94d2cTzLHaEhpw.png" data-width="2397" src="https://cdn-images-1.medium.com/max/2000/1*_E-SsoYC94d2cTzLHaEhpw.png"></p>
<p name="0558"><img data-height="1024" data-image-id="1*M3KX9A8zJvTnQ3O6i3pHBQ.png" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/1*M3KX9A8zJvTnQ3O6i3pHBQ.png"></p>
<h3 name="55d6">IRISH STATIONS</h3>
<pre data-code-block-lang="json" data-code-block-mode="2" name="d6e5" spellcheck="false"><span><span>{</span><span>"StationDesc"</span><span>:</span><span>"Millstreet"</span><span>,</span><span>"StationAlias"</span><span>:</span><span><span>null</span></span><span>,</span><br><span>"StationLatitude"</span><span>:</span><span>52.0776</span><span>,</span><span>"StationLongitude"</span><span>:</span><span>-9.06973</span><span>,</span><br><span>"StationCode"</span><span>:</span><span>"MLSRT"</span><span>,</span><span>"StationId"</span><span>:</span><span>24</span><span>,</span><span>"ts"</span><span>:</span><span>"1711496919762"</span><span>,</span><br><span>"uuid"</span><span>:</span><span>"f6e71a76-41cc-4a8e-8795-323c3b43d62f"</span><span>}</span></span></pre>
<h3 name="bfde">IRISH TRAIN RECORD</h3>
<pre data-code-block-lang="json" data-code-block-mode="2" name="a483" spellcheck="false"><span><span>{</span><br><span>"TrainStatus"</span><span>:</span><span>"R"</span><span>,</span><span>"TrainLatitude"</span><span>:</span><span>53.4169</span><span>,</span><br><span>"TrainLongitude"</span><span>:</span><span>-6.1512</span><span>,</span><span>"TrainCode"</span><span>:</span><span>"P617"</span><span>,</span><br><span>"TrainDate"</span><span>:</span><span>"27 Mar 2024"</span><span>,</span><br><span>"PublicMessage"</span><span>:</span><span>"P617\\n16:02 - Drogheda to Dublin Pearse (1 mins late)\\nDeparted Portmarnock next stop Dublin Connolly"</span><span>,</span><br><span>"Direction"</span><span>:</span><span>"Southbound"</span><span>,</span><span>"ts"</span><span>:</span><span>"1711557932947"</span><span>,</span><br><span>"uuid"</span><span>:</span><span>"b485cefb-67e8-482d-86ba-1ca43e0b523a"</span><br><span>}</span></span></pre>
<h3 name="f618">IRISH STATION RECORD</h3>
<pre data-code-block-lang="json" data-code-block-mode="2" name="4e71" spellcheck="false"><span><span>{</span><br><span>"StationDesc"</span><span>:</span><span>"Midleton"</span><span>,</span><br><span>"StationAlias"</span><span>:</span><span><span>null</span></span><span>,</span><br><span>"StationLatitude"</span><span>:</span><span>51.9212</span><span>,</span><br><span>"StationLongitude"</span><span>:</span><span>-8.17579</span><span>,</span><br><span>"StationCode"</span><span>:</span><span>"MDLTN"</span><span>,</span><br><span>"StationId"</span><span>:</span><span>68</span><span>,</span><br><span>"ts"</span><span>:</span><span>"1711558009615"</span><span>,</span><br><span>"uuid"</span><span>:</span><span>"1f5ae394-4726-4f3e-8e53-7f50f95ae05e"</span><br><span>}</span></span></pre>
<p name="b7b1"><img data-height="1645" data-image-id="1*eDYMbPJgFwHulU7tKlC0Iw.png" data-width="3418" src="https://cdn-images-1.medium.com/max/2000/1*eDYMbPJgFwHulU7tKlC0Iw.png"></p>
<p name="09c0"><img data-height="1644" data-image-id="1*-_U9Bs7PyuDBk5Gg9Bnmog.png" data-width="3412" src="https://cdn-images-1.medium.com/max/2000/1*-_U9Bs7PyuDBk5Gg9Bnmog.png"></p>
<h3 name="5fde">SOURCE CODE</h3>
<div name="b47e">
<a data-href="https://github.com/tspannhw/FLaNK-IrelandTransit/tree/main" href="https://github.com/tspannhw/FLaNK-IrelandTransit/tree/main" title="https://github.com/tspannhw/FLaNK-IrelandTransit/tree/main"><strong>GitHub - tspannhw/FLaNK-IrelandTransit: Transit in Ireland</strong><br><em>Transit in Ireland. Contribute to tspannhw/FLaNK-IrelandTransit development by creating an account on GitHub.</em>github.com</a><a data-media-id="1261f5a98e3f7009436429468e3eeed0" data-thumbnail-img-id="0*IWwwcmwf_afNQAcd" href="https://github.com/tspannhw/FLaNK-IrelandTransit/tree/main" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*IWwwcmwf_afNQAcd);"></a>
</div>
<p name="5589"><img data-height="1024" data-image-id="1*rU2GEgAQQMji_E4aN9Pa6A.png" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/1*rU2GEgAQQMji_E4aN9Pa6A.png"></p>
<h3 name="76a6">FLINK SQL KAFKA TABLE</h3>
<pre data-code-block-lang="sql" data-code-block-mode="2" name="6493" spellcheck="false"><span><span>CREATE</span> <span>TABLE</span> `ssb`.`Meetups`.`irelandvehicle` (<br> `recordid` <span>VARCHAR</span>(<span>2147483647</span>),<br> `route_id` <span>VARCHAR</span>(<span>2147483647</span>),<br> `directionid` <span>VARCHAR</span>(<span>2147483647</span>),<br> `latitude` <span>VARCHAR</span>(<span>2147483647</span>),<br> `tripid` <span>VARCHAR</span>(<span>2147483647</span>),<br> `starttime` <span>VARCHAR</span>(<span>2147483647</span>),<br> `vehicleid` <span>VARCHAR</span>(<span>2147483647</span>),<br> `startdate` <span>VARCHAR</span>(<span>2147483647</span>),<br> `uuid` <span>VARCHAR</span>(<span>2147483647</span>),<br> `longitude` <span>VARCHAR</span>(<span>2147483647</span>),<br> `<span>timestamp</span>` <span>VARCHAR</span>(<span>2147483647</span>),<br> `ts` <span>VARCHAR</span>(<span>2147483647</span>),<br> `route_long_name` <span>VARCHAR</span>(<span>2147483647</span>),<br> `trip_short_name` <span>VARCHAR</span>(<span>2147483647</span>),<br> `trip_headsign` <span>VARCHAR</span>(<span>2147483647</span>),<br> `eventTimeStamp` <span>TIMESTAMP</span>(<span>3</span>) <span>WITH</span> <span>LOCAL</span> <span>TIME</span> ZONE METADATA <span>FROM</span> <span>'timestamp'</span>,<br> WATERMARK <span>FOR</span> `eventTimeStamp` <span>AS</span> `eventTimeStamp` <span>-</span> <span>INTERVAL</span> <span>'3'</span> <span>SECOND</span><br>) <span>WITH</span> (<br> <span>'scan.startup.mode'</span> <span>=</span> <span>'group-offsets'</span>,<br> <span>'deserialization.failure.policy'</span> <span>=</span> <span>'ignore_and_log'</span>,<br> <span>'properties.request.timeout.ms'</span> <span>=</span> <span>'120000'</span>,<br> <span>'properties.auto.offset.reset'</span> <span>=</span> <span>'earliest'</span>,<br> <span>'format'</span> <span>=</span> <span>'json'</span>,<br> <span>'properties.bootstrap.servers'</span> <span>=</span> <span>'kafka:9092'</span>,<br> <span>'connector'</span> <span>=</span> <span>'kafka'</span>,<br> <span>'properties.transaction.timeout.ms'</span> <span>=</span> <span>'900000'</span>,<br> <span>'topic'</span> <span>=</span> <span>'irelandvehicle'</span>,<br> <span>'properties.group.id'</span> <span>=</span> <span>'irelandconsumersbb1'</span><br>)</span></pre>
<p name="53d8"><img data-height="1024" data-image-id="1*3xJ0rjW5EtVUhOLTwhAHGA.png" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/1*3xJ0rjW5EtVUhOLTwhAHGA.png"></p>
<h3 name="4417">RESOURCES</h3>
<div name="377b">
<a data-href="https://blog.cloudera.com/cloudera-recognized-as-a-great-place-to-work-in-ireland-and-costa-rica/" href="https://blog.cloudera.com/cloudera-recognized-as-a-great-place-to-work-in-ireland-and-costa-rica/" title="https://blog.cloudera.com/cloudera-recognized-as-a-great-place-to-work-in-ireland-and-costa-rica/"><strong>Cloudera Recognized as a Great Place to Work in Ireland and Costa Rica - Cloudera Blog</strong><br><em>We're excited to announce that Cloudera has been named the Best Medium Workplace in Ireland™, one of the Best…</em>blog.cloudera.com</a><a data-media-id="f5512dcff7fac2441f32a04a8e6d131c" data-thumbnail-img-id="0*IDTjvoXXMdWl9BIg" href="https://blog.cloudera.com/cloudera-recognized-as-a-great-place-to-work-in-ireland-and-costa-rica/" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*IDTjvoXXMdWl9BIg);"></a>
</div>
<div name="0804">
<a data-href="https://medium.com/@sriraaman.v/xml-data-processing-using-apache-nifi-d6469826e09b" href="https://medium.com/@sriraaman.v/xml-data-processing-using-apache-nifi-d6469826e09b" title="https://medium.com/@sriraaman.v/xml-data-processing-using-apache-nifi-d6469826e09b"><strong>XML data processing- using Apache NiFi</strong><br><em>Intr</em>medium.com</a><a data-media-id="940ac14dd90b310363e13649c6ddb73a" data-thumbnail-img-id="1*sf3UlKE7Q9FT2OZQy1rQTA.png" href="https://medium.com/@sriraaman.v/xml-data-processing-using-apache-nifi-d6469826e09b" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*sf3UlKE7Q9FT2OZQy1rQTA.png);"></a>
</div>
<div name="d151">
<a data-href="https://www.linkedin.com/pulse/parsing-xml-logs-nifi-part-1-3-chris-gambino/" href="https://www.linkedin.com/pulse/parsing-xml-logs-nifi-part-1-3-chris-gambino/" title="https://www.linkedin.com/pulse/parsing-xml-logs-nifi-part-1-3-chris-gambino/"><strong>Parsing XML Logs With Nifi - Part 1 of 3</strong><br><em>I have a plan to write a 3 part "intro" series as to how to handle your XML files.</em>www.linkedin.com</a><a data-media-id="d759d332c0a9591c3bcdacb3ee0fd9e3" data-thumbnail-img-id="0*r7N5ZtzS6PRzce7f" href="https://www.linkedin.com/pulse/parsing-xml-logs-nifi-part-1-3-chris-gambino/" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*r7N5ZtzS6PRzce7f);"></a>
</div>
<div name="27fa">
<a data-href="https://www.cloudera.com/campaign/apache-nifi-for-dummies.html?i" href="https://www.cloudera.com/campaign/apache-nifi-for-dummies.html?i" title="https://www.cloudera.com/campaign/apache-nifi-for-dummies.html?i"><strong>Apache NiFi for Dummies</strong><br><em>Get actionable tips and insights about Apache NiFi, an open-source tool with a drag-and-drop interface for building…</em>www.cloudera.com</a><a data-media-id="c79b26c741bfccf1ae4f6b284c7adad9" data-thumbnail-img-id="0*jeIFALd0z_DgPran" href="https://www.cloudera.com/campaign/apache-nifi-for-dummies.html?i" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*jeIFALd0z_DgPran);"></a>
</div>
<div name="7c86">
<a data-href="https://www.projectpro.io/recipes/extract-values-from-xml-data-nifi#mcetoc_1g4h9crqde" href="https://www.projectpro.io/recipes/extract-values-from-xml-data-nifi#mcetoc_1g4h9crqde" title="https://www.projectpro.io/recipes/extract-values-from-xml-data-nifi#mcetoc_1g4h9crqde"><strong>How to extract values from XML data in NiFi -</strong><br><em>This recipe helps you extract values from XML data in NiFi</em>www.projectpro.io</a><a data-media-id="c1a45417a8cbd3c4204fe7e8864c1d56" data-thumbnail-img-id="0*IIjncgjnqrDQimVx" href="https://www.projectpro.io/recipes/extract-values-from-xml-data-nifi#mcetoc_1g4h9crqde" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*IIjncgjnqrDQimVx);"></a>
</div>
<div name="4cb6">
<a data-href="http://www.silvercloudcomputing.com/nifi-tips.html" href="http://www.silvercloudcomputing.com/nifi-tips.html" title="http://www.silvercloudcomputing.com/nifi-tips.html"><strong>NiFi Tips</strong><br><em>I created an install script (linux machines) that will install a local secure version of NiFi. It will also generate…</em>www.silvercloudcomputing.com</a><a data-media-id="35283c8fdf374c58521cdfa07391c620" data-thumbnail-img-id="0*qMMFeWZXjPg2TK2e" href="http://www.silvercloudcomputing.com/nifi-tips.html" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*qMMFeWZXjPg2TK2e);"></a>
</div>
<div name="c155">
<a data-href="https://github.com/tspannhw/EverythingApacheNiFi" href="https://github.com/tspannhw/EverythingApacheNiFi" title="https://github.com/tspannhw/EverythingApacheNiFi"><strong>GitHub - tspannhw/EverythingApacheNiFi: EverythingApacheNiFi</strong><br><em>EverythingApacheNiFi. Contribute to tspannhw/EverythingApacheNiFi development by creating an account on GitHub.</em>github.com</a><a data-media-id="79a6f31857fe9948a5b972a09818511c" data-thumbnail-img-id="0*_9EJwVm4K1yJR1yl" href="https://github.com/tspannhw/EverythingApacheNiFi" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*_9EJwVm4K1yJR1yl);"></a>
</div>
<div name="055a">
<a data-href="https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350" href="https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350" title="https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350"><strong>Jolt quick reference for Nifi Jolt Processors</strong><br><em>OBJECTIVE: Provide a quick-start guide for using the Jolt language within a NiFi JoltTransform (JoltTransformJSON or…</em>community.cloudera.com</a><a data-media-id="e47e92b7afa9da773a0bc53c9114f63c" data-thumbnail-img-id="0*kc5W4ZKo2eQJr_2y" href="https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*kc5W4ZKo2eQJr_2y);"></a>
</div>
<div name="361d">
<a data-href="https://gtfs.org/schedule/reference/" href="https://gtfs.org/schedule/reference/" title="https://gtfs.org/schedule/reference/"><strong>Reference - General Transit Feed Specification</strong><br><em>This document defines the format and structure of the files that comprise a GTFS dataset. The key words "MUST", "MUST…</em>gtfs.org</a><a data-media-id="f3e36317bce07bda799eae39c4fc1cc4" href="https://gtfs.org/schedule/reference/"></a>
</div>
<div name="9606">
<a data-href="https://community.cloudera.com/t5/Support-Questions/IF-statement-for-NiFi-attribute-result-NiFi/td-p/333464" href="https://community.cloudera.com/t5/Support-Questions/IF-statement-for-NiFi-attribute-result-NiFi/td-p/333464" title="https://community.cloudera.com/t5/Support-Questions/IF-statement-for-NiFi-attribute-result-NiFi/td-p/333464"><strong>IF statement for NiFi attribute result - NiFi</strong><br><em>Hello all Within NiFi, updateAttribute processor I am trying to change an attribute called 'hive_database' based on the…</em>community.cloudera.com</a><a data-media-id="2f87e35990627ebb5fee389db3e2628d" data-thumbnail-img-id="0*7pEBvjkI20dlVAQ0" href="https://community.cloudera.com/t5/Support-Questions/IF-statement-for-NiFi-attribute-result-NiFi/td-p/333464" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*7pEBvjkI20dlVAQ0);"></a>
</div>
<div name="32ea">
<a data-href="https://nifi.apache.org/documentation/nifi-2.0.0-M2/components/org.apache.nifi/nifi-update-attribute-nar/2.0.0-M2/org.apache.nifi.processors.attributes.UpdateAttribute/index.html" href="https://nifi.apache.org/documentation/nifi-2.0.0-M2/components/org.apache.nifi/nifi-update-attribute-nar/2.0.0-M2/org.apache.nifi.processors.attributes.UpdateAttribute/index.html" title="https://nifi.apache.org/documentation/nifi-2.0.0-M2/components/org.apache.nifi/nifi-update-attribute-nar/2.0.0-M2/org.apache.nifi.processors.attributes.UpdateAttribute/index.html"><strong>UpdateAttribute</strong><br><em>Updates the Attributes for a FlowFile by using the Attribute Expression Language and/or deletes the attributes based on…</em>nifi.apache.org</a><a data-media-id="8930ed1417f0e0d9097947c6c1f21e01" href="https://nifi.apache.org/documentation/nifi-2.0.0-M2/components/org.apache.nifi/nifi-update-attribute-nar/2.0.0-M2/org.apache.nifi.processors.attributes.UpdateAttribute/index.html"></a>
</div>
<div name="2c27">
<a data-href="https://medium.com/mpharma-tech-blog/dealing-with-conditionals-in-apache-nifi-e24fbc01037b" href="https://medium.com/mpharma-tech-blog/dealing-with-conditionals-in-apache-nifi-e24fbc01037b" title="https://medium.com/mpharma-tech-blog/dealing-with-conditionals-in-apache-nifi-e24fbc01037b"><strong>Dealing with conditionals in Apache NiFi</strong><br><em>When it comes to creating resilient data pipelines, one of the tools that comes to mind is Apache NiFi. At the heart of…</em>medium.com</a><a data-media-id="2c90c9203d186ef3f3b9d31bc625070c" data-thumbnail-img-id="1*4Zh1rRaHzwRWQI7_x6twGg.jpeg" href="https://medium.com/mpharma-tech-blog/dealing-with-conditionals-in-apache-nifi-e24fbc01037b" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*4Zh1rRaHzwRWQI7_x6twGg.jpeg);"></a>
</div>
<div name="3d81">
<a data-href="https://medium.com/@sriraaman.v/exploring-nifi-updateattribute-processor-15288677e2b8" href="https://medium.com/@sriraaman.v/exploring-nifi-updateattribute-processor-15288677e2b8" title="https://medium.com/@sriraaman.v/exploring-nifi-updateattribute-processor-15288677e2b8"><strong>Exploring NiFi- UpdateAttribute processor</strong><br><em>I</em>medium.com</a><a data-media-id="198eb0c527d39f5923e283b23af8b615" data-thumbnail-img-id="1*sf3UlKE7Q9FT2OZQy1rQTA.png" href="https://medium.com/@sriraaman.v/exploring-nifi-updateattribute-processor-15288677e2b8" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*sf3UlKE7Q9FT2OZQy1rQTA.png);"></a>
</div>
<div name="f327">
<a data-href="https://pratikbarjatya.medium.com/building-data-ingestion-system-using-apache-nifi-76e90765ac43" href="https://pratikbarjatya.medium.com/building-data-ingestion-system-using-apache-nifi-76e90765ac43" title="https://pratikbarjatya.medium.com/building-data-ingestion-system-using-apache-nifi-76e90765ac43"><strong>Building a Data Ingestion System Using Apache NiFi</strong><br><em>Collecting data from SQL and NoSQL systems and building a data ingestion pipeline can be a complex process, but it can…</em>pratikbarjatya.medium.com</a><a data-media-id="b8580913aae109b4631c43bd175aad05" data-thumbnail-img-id="0*OQ5ngfwBAfBY8khK" href="https://pratikbarjatya.medium.com/building-data-ingestion-system-using-apache-nifi-76e90765ac43" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*OQ5ngfwBAfBY8khK);"></a>
</div>
<div name="874c">
<a data-href="https://docs.griddb.net/tutorial/nifi/#build-the-nifi-flow" href="https://docs.griddb.net/tutorial/nifi/#build-the-nifi-flow" title="https://docs.griddb.net/tutorial/nifi/#build-the-nifi-flow"><strong>ETL With Apache Nifi | GridDB | Docs</strong><br><em>GridDB Documentation</em>docs.griddb.net</a><a data-media-id="9e66f3d10ed61d2ed65075e07f47ad95" href="https://docs.griddb.net/tutorial/nifi/#build-the-nifi-flow"></a>
</div>
<div name="c18a">
<a data-href="https://github.com/mobilitydata/gtfs-validator" href="https://github.com/mobilitydata/gtfs-validator" title="https://github.com/mobilitydata/gtfs-validator"><strong>GitHub - MobilityData/gtfs-validator: Canonical GTFS Validator project for schedule (static) files.</strong><br><em>Canonical GTFS Validator project for schedule (static) files. - MobilityData/gtfs-validator</em>github.com</a><a data-media-id="1a292ad4226a5007b6ff4dec7cda7d1d" data-thumbnail-img-id="0*wE0EgbyIgRv6c0Qi" href="https://github.com/mobilitydata/gtfs-validator" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*wE0EgbyIgRv6c0Qi);"></a>
</div>
<div name="449c">
<a data-href="https://developers.google.com/transit/gtfs-realtime#what_is_live_transit_updates_for_google_maps" href="https://developers.google.com/transit/gtfs-realtime#what_is_live_transit_updates_for_google_maps" title="https://developers.google.com/transit/gtfs-realtime#what_is_live_transit_updates_for_google_maps"><strong>GTFS Realtime Overview | Realtime Transit | Google for Developers</strong><br><em>Format for exchanging realtime public transit information.</em>developers.google.com</a><a data-media-id="80a47c4372e74cb7cf2aff3aa2b1a57b" data-thumbnail-img-id="0*LnoWOlclBJBghmHJ" href="https://developers.google.com/transit/gtfs-realtime#what_is_live_transit_updates_for_google_maps" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*LnoWOlclBJBghmHJ);"></a>
</div>
<div name="934d">
<a data-href="https://github.com/MobilityData/awesome-transit" href="https://github.com/MobilityData/awesome-transit" title="https://github.com/MobilityData/awesome-transit"><strong>GitHub - MobilityData/awesome-transit: Community list of transit APIs, apps, datasets, research…</strong><br><em>Community list of transit APIs, apps, datasets, research, and software :bus::star2::train::star2::steam_locomotive: …</em>github.com</a><a data-media-id="332b9b3bf85e3a447c510186c8e590d8" data-thumbnail-img-id="0*Ls839OPySqWJJQQX" href="https://github.com/MobilityData/awesome-transit" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*Ls839OPySqWJJQQX);"></a>
</div>
<div name="79fe">
<a data-href="https://medium.com/cloudera-inc/nyc-traffic-are-you-kidding-me-6d3fa853903b" href="https://medium.com/cloudera-inc/nyc-traffic-are-you-kidding-me-6d3fa853903b" title="https://medium.com/cloudera-inc/nyc-traffic-are-you-kidding-me-6d3fa853903b"><strong>NYC Traffic!?!??! Are You Kidding Me?</strong><br><em>Apache NiFi, Python, Traffic, JSON, Web Camera, REST, XML, RSS, JSON</em>medium.com</a><a data-media-id="b8c09607b7c9e7faec300a6d9c5dd860" data-thumbnail-img-id="0*Pw3nE7gQDbnpShPa" href="https://medium.com/cloudera-inc/nyc-traffic-are-you-kidding-me-6d3fa853903b" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*Pw3nE7gQDbnpShPa);"></a>
</div>
<div name="1930">
<a data-href="https://medium.com/cloudera-inc/iteration-2-building-a-system-to-consume-all-the-unsecured-real-time-transit-data-in-the-world-5f365c0cf240" href="https://medium.com/cloudera-inc/iteration-2-building-a-system-to-consume-all-the-unsecured-real-time-transit-data-in-the-world-5f365c0cf240" title="https://medium.com/cloudera-inc/iteration-2-building-a-system-to-consume-all-the-unsecured-real-time-transit-data-in-the-world-5f365c0cf240"><strong>Iteration 2: Building a System to Consume All the (Unsecured) Real-Time Transit Data in the World</strong><br><em>This is the remix.</em>medium.com</a><a data-media-id="e0775e0495f1ac85c1007d86917294c9" data-thumbnail-img-id="1*iNxd8Xy8L1-owvuVDCd_7Q.png" href="https://medium.com/cloudera-inc/iteration-2-building-a-system-to-consume-all-the-unsecured-real-time-transit-data-in-the-world-5f365c0cf240" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*iNxd8Xy8L1-owvuVDCd_7Q.png);"></a>
</div>
<div name="665f">
<a data-href="https://medium.com/cloudera-inc/never-get-lost-in-the-stream-2ade0af99a73" href="https://medium.com/cloudera-inc/never-get-lost-in-the-stream-2ade0af99a73" title="https://medium.com/cloudera-inc/never-get-lost-in-the-stream-2ade0af99a73"><strong>Never Get Lost in the Stream</strong><br><em>NiFi-Kafka-Flink for getting to work. Can’t we just work remote?</em>medium.com</a><a data-media-id="44845d4fa5763f70aafa5cb3342b71ce" data-thumbnail-img-id="1*xvfaus-Hll2RUUqVW-ovXg.png" href="https://medium.com/cloudera-inc/never-get-lost-in-the-stream-2ade0af99a73" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*xvfaus-Hll2RUUqVW-ovXg.png);"></a>
</div>
<div name="0e1b">
<a data-href="https://medium.com/cloudera-inc/flank-for-halifax-canada-transit-2d4dc5b8ad64" href="https://medium.com/cloudera-inc/flank-for-halifax-canada-transit-2d4dc5b8ad64" title="https://medium.com/cloudera-inc/flank-for-halifax-canada-transit-2d4dc5b8ad64"><strong>FLaNK for Halifax Canada Transit</strong><br><em>Event Streaming in Canada with NiFi, Kafka, Flink, PostgreSQL</em>medium.com</a><a data-media-id="688fed82daf85c4907d2be0cc405b58f" data-thumbnail-img-id="1*EXYmnRvYXVC3ExTvXOA1_g.png" href="https://medium.com/cloudera-inc/flank-for-halifax-canada-transit-2d4dc5b8ad64" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*EXYmnRvYXVC3ExTvXOA1_g.png);"></a>
</div>
<div name="211c">
<a data-href="https://medium.com/cloudera-inc/transit-in-sao-paulo-brasil-flank-style-eaec6753cc63" href="https://medium.com/cloudera-inc/transit-in-sao-paulo-brasil-flank-style-eaec6753cc63" title="https://medium.com/cloudera-inc/transit-in-sao-paulo-brasil-flank-style-eaec6753cc63"><strong>Transit in Sao Paulo, Brasil — FLaNK Style</strong><br><em>Streaming with NiFi, Kafka, Flink</em>medium.com</a><a data-media-id="4e8285761427a873bda7423458666398" data-thumbnail-img-id="0*eNHiuiYuqDA8NYgW" href="https://medium.com/cloudera-inc/transit-in-sao-paulo-brasil-flank-style-eaec6753cc63" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*eNHiuiYuqDA8NYgW);"></a>
</div>
<div name="098f">
<a data-href="https://medium.com/cloudera-inc/iteration-1-building-a-system-to-consume-all-the-real-time-transit-data-in-the-world-at-once-4322b160df9d" href="https://medium.com/cloudera-inc/iteration-1-building-a-system-to-consume-all-the-real-time-transit-data-in-the-world-at-once-4322b160df9d" title="https://medium.com/cloudera-inc/iteration-1-building-a-system-to-consume-all-the-real-time-transit-data-in-the-world-at-once-4322b160df9d"><strong>Iteration 1: Building a System to Consume All the Real-Time Transit Data in the World At Once</strong><br><em>Source Code: https://github.com/tspannhw/FLaNK-EveryTransitSystem</em>medium.com</a><a data-media-id="d24333373a5353ffbf9561a72f5b5033" data-thumbnail-img-id="1*e9BHhP0C7zY-V9X_hg6ebw.png" href="https://medium.com/cloudera-inc/iteration-1-building-a-system-to-consume-all-the-real-time-transit-data-in-the-world-at-once-4322b160df9d" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/1*e9BHhP0C7zY-V9X_hg6ebw.png);"></a>
</div>
<div name="3ceb">
<a data-href="https://github.com/google/transit/tree/master/gtfs-realtime/spec/en" href="https://github.com/google/transit/tree/master/gtfs-realtime/spec/en" title="https://github.com/google/transit/tree/master/gtfs-realtime/spec/en"><strong>transit/gtfs-realtime/spec/en at master · google/transit</strong><br><em>Contribute to google/transit development by creating an account on GitHub.</em>github.com</a><a data-media-id="e0a6374f9f102fa71e632e1076b5589e" data-thumbnail-img-id="0*VpE-_FzhpPC9m2Oy" href="https://github.com/google/transit/tree/master/gtfs-realtime/spec/en" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*VpE-_FzhpPC9m2Oy);"></a>
</div>
<div name="bb2f">
<a data-href="https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/vehicle-positions.md" href="https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/vehicle-positions.md" title="https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/vehicle-positions.md"><strong>transit/gtfs-realtime/spec/en/vehicle-positions.md at master · google/transit</strong><br><em>Contribute to google/transit development by creating an account on GitHub.</em>github.com</a><a data-media-id="2b670e888798a668d8a6118126b891c4" data-thumbnail-img-id="0*YAMTiH3WWGqwRTz2" href="https://github.com/google/transit/blob/master/gtfs-realtime/spec/en/vehicle-positions.md" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*YAMTiH3WWGqwRTz2);"></a>
</div>
<div name="6cbb">
<a data-href="https://gtfs.org/realtime/reference/" href="https://gtfs.org/realtime/reference/" title="https://gtfs.org/realtime/reference/"><strong>Reference - General Transit Feed Specification</strong><br><em>A GTFS Realtime feed lets transit agencies provide consumers with realtime information about disruptions to their…</em>gtfs.org</a><a data-media-id="76a9b39c16a488b15e84bbed089ebf48" href="https://gtfs.org/realtime/reference/"></a>
</div>
<div name="5261">
<a data-href="https://www.transportforireland.ie/plan-a-journey/live-departures/" href="https://www.transportforireland.ie/plan-a-journey/live-departures/" title="https://www.transportforireland.ie/plan-a-journey/live-departures/"><strong>Live departures</strong><br><em>View live departures. Search for and select a stop to view available departure times, with map and satellite views…</em>www.transportforireland.ie</a><a data-media-id="9bb3c100e5ed1e6fa9fcf1e0dba2e8b2" href="https://www.transportforireland.ie/plan-a-journey/live-departures/"></a>
</div>
<div name="a8d3">
<a data-href="https://github.com/graphhopper/graphhopper/#public-transit" href="https://github.com/graphhopper/graphhopper/#public-transit" title="https://github.com/graphhopper/graphhopper/#public-transit"><strong>GitHub - graphhopper/graphhopper: Open source routing engine for OpenStreetMap. Use it as Java…</strong><br><em>Open source routing engine for OpenStreetMap. Use it as Java library or standalone web server. …</em>github.com</a><a data-media-id="3ce0c0442f2d9d2abec299e84c78a847" data-thumbnail-img-id="0*S3m-17Bb2IHuB2UA" href="https://github.com/graphhopper/graphhopper/#public-transit" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*S3m-17Bb2IHuB2UA);"></a>
</div>
<div name="ad50">
<a data-href="https://github.com/graphhopper/graphhopper/blob/master/README.md#OpenStreetMap-Support" href="https://github.com/graphhopper/graphhopper/blob/master/README.md#OpenStreetMap-Support" title="https://github.com/graphhopper/graphhopper/blob/master/README.md#OpenStreetMap-Support"><strong>graphhopper/README.md at master · graphhopper/graphhopper</strong><br><em>Open source routing engine for OpenStreetMap. Use it as Java library or standalone web server. - graphhopper/README.md…</em>github.com</a><a data-media-id="370f587d40a40d781b1deb42b4d1caa9" data-thumbnail-img-id="0*F19kqjHbeHblleNU" href="https://github.com/graphhopper/graphhopper/blob/master/README.md#OpenStreetMap-Support" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*F19kqjHbeHblleNU);"></a>
</div>
<div name="ff0b">
<a data-href="https://developer.onebusaway.org/" href="https://developer.onebusaway.org/" title="https://developer.onebusaway.org/"><strong>OneBusAway Developer Documentation</strong><br><em>OneBusAway is a suite of open source transit information tools that enable transit agencies to provide real-time…</em>developer.onebusaway.org</a><a data-media-id="2d9908f63273708302ef9c7f9ec13e56" data-thumbnail-img-id="0*7rBHZRG3AWpXjiS2" href="https://developer.onebusaway.org/" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*7rBHZRG3AWpXjiS2);"></a>
</div>
<div name="002e">
<a data-href="https://thetransitclock.github.io/" href="https://thetransitclock.github.io/" title="https://thetransitclock.github.io/"><strong><em>TheTransitClock</em>.github.io</strong><br>TheTransitClockthetransitclock.github.io</a><a data-media-id="30d55fe0547869c292ba02fcc74f0712" data-thumbnail-img-id="0*L9Yi-8J6Vc-PxsAI" href="https://thetransitclock.github.io/" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*L9Yi-8J6Vc-PxsAI);"></a>
</div>
<div name="4aec">
<a data-href="https://github.com/TheTransitClock/transitclockDocker" href="https://github.com/TheTransitClock/transitclockDocker" title="https://github.com/TheTransitClock/transitclockDocker"><strong>GitHub - TheTransitClock/transitclockDocker: Docker installer for TheTransitClock</strong><br><em>Docker installer for TheTransitClock. Contribute to TheTransitClock/transitclockDocker development by creating an…</em>github.com</a><a data-media-id="9403e5f6cea224bbf8824e688fc63a69" data-thumbnail-img-id="0*xibxQVwikBmCgJel" href="https://github.com/TheTransitClock/transitclockDocker" style="background-image: url(https://cdn-images-1.medium.com/fit/c/400/400/0*xibxQVwikBmCgJel);"></a>
</div>
<div name="3870">
<a data-href="https://github.com/TheTransitClock/transitime" href="https://github.com/TheTransitClock/transitime" title="https://github.com/TheTransitClock/transitime"><strong>GitHub - TheTransitClock/transitime: TheTransitClock real-time transit information system</strong><br><em>TheTransitClock real-time transit information system - TheTransitClock/transitime</em>github.com</a><a data-media-id="4ad1f399432dc26c0ae281c9785fce0b" href="https://github.com/TheTransitClock/transitime"></a>
</div>
<p name="02a9"><img data-height="1024" data-image-id="1*9pjAY2qsUxpPpqJDK9obXw.jpeg" data-width="1024" src="https://cdn-images-1.medium.com/max/2000/1*9pjAY2qsUxpPpqJDK9obXw.jpeg"></p>
<p name="c490"><br></p>
</div></div></section>