FLaNK:  Low Code Streaming:  Populating Kafka Topics with FlinkSQL Joins in Real-Time
Tags: 2020, apachekafka, apachenifi, apacheflink
Published: 2020-05-13 18:15:33

---
title: FLaNK:  Low Code Streaming:  Populating Kafka Topics with FlinkSQL Joins in Real-Time
published: true
date: 2020-05-12 21:21:00 UTC
tags: 2020,apache kafka,apache nifi,apache-flink
canonical_url: 
---

 **<font size="6"><br></font>**

**<font size="6">FLaNK:  Low Code Streaming:  Populating Kafka Topics with FlinkSQL Joins in Real-Time </font>**

  

  

[![](https://1.bp.blogspot.com/-7bs5rJqhchM/Xrr69TZpAOI/AAAAAAAAaSA/ABySdUNdzuQcC-Qwv3t7ncG382eicK0AgCK4BGAsYHg/w640-h640/flinksqlclient.png)](https://1.bp.blogspot.com/-7bs5rJqhchM/Xrr69TZpAOI/AAAAAAAAaSA/ABySdUNdzuQcC-Qwv3t7ncG382eicK0AgCK4BGAsYHg/flinksqlclient.png)

  

  

  

  

  

[![FLaNK](https://1.bp.blogspot.com/-NUJ_YqnAAhI/XrsRdHVOtpI/AAAAAAAAaU0/7rxJJiN5zRwunoeLTBZqoZurcDITFDnPQCK4BGAsYHg/w480-h640/mlx90640-2020-01-05-20-48-26.gif "FLaNK")](https://1.bp.blogspot.com/-NUJ_YqnAAhI/XrsRdHVOtpI/AAAAAAAAaU0/7rxJJiN5zRwunoeLTBZqoZurcDITFDnPQCK4BGAsYHg/mlx90640-2020-01-05-20-48-26.gif)

  

[![](https://1.bp.blogspot.com/-0rHKKjMAvTY/XrsRehEwYcI/AAAAAAAAaVE/zMFxQsjgTyMAYWEoag8oVbjpFOkFS_OngCK4BGAsYHg/w640-h360/CSA%2BFlinkSQL%2BDemo%2BApril%2B2020%2B%25282%2529.jpg)](https://1.bp.blogspot.com/-0rHKKjMAvTY/XrsRehEwYcI/AAAAAAAAaVE/zMFxQsjgTyMAYWEoag8oVbjpFOkFS_OngCK4BGAsYHg/CSA%2BFlinkSQL%2BDemo%2BApril%2B2020%2B%25282%2529.jpg)
  

[![](https://1.bp.blogspot.com/-SMjvkYmKPl4/XrsRfObIbCI/AAAAAAAAaVI/y5TPqHmyiGAjySmVrLGQZoQJ8ZuDv5frgCK4BGAsYHg/w640-h360/CSA%2BFlinkSQL%2BDemo%2BApril%2B2020%2B%25281%2529.jpg)](https://1.bp.blogspot.com/-SMjvkYmKPl4/XrsRfObIbCI/AAAAAAAAaVI/y5TPqHmyiGAjySmVrLGQZoQJ8ZuDv5frgCK4BGAsYHg/CSA%2BFlinkSQL%2BDemo%2BApril%2B2020%2B%25281%2529.jpg)
  

  

Then I can create my 3 tables.   Two are the source ones to join and the third is the destination for my insert.

  

 ![](https://1.bp.blogspot.com/-45YamgYhlmg/XrCDiuVjbAI/AAAAAAAAZ9s/L1jhscluCxMgc2PaTYGrnGoEfnYQpxxgwCLcBGAsYHQ/s640/flinkjoin.png)

  

  

INSERT INTO global\_sensor\_events 

SELECT 

 scada.uuid, 

 scada.systemtime ,  

scada.temperaturef , 

scada.pressure , 

scada.humidity , 

scada.lux , 

scada.proximity , 

scada.oxidising , 

scada.reducing , 

scada.nh3 , 

scada.gasko,

energy.`current`, 

energy.voltage ,

energy.`power` ,

energy.`total`,

energy.fanstatus

  

FROM energy,

     scada

WHERE

    scada.systemtime = energy.systemtime;

  

**Examples**

[https://github.com/tspannhw/meetup-sensors/blob/master/flink-sql/](https://github.com/tspannhw/meetup-sensors/blob/master/flink-sql/demo.sh)

  

**Assets / Scripts / DDL / SQL**

[https://github.com/tspannhw/FlinkSQLDemo](https://github.com/tspannhw/FlinkSQLDemo)

  

**Flink Guide to SQL Joins**
[https://www.youtube.com/watch?v=5AuBlVRKQuo](https://www.youtube.com/watch?v=5AuBlVRKQuo)  
  

**Slides**

[https://www.slideshare.net/bunkertor/time-series-analysis-dataflow](https://www.slideshare.net/bunkertor/time-series-analysis-dataflow)

  

**Article on Joins**

[https://www.datainmotion.dev/2020/05/flink-sql-preview.html](https://dev.to/tspannhw/flink-sql-preview-22l)

  

**Resources**

- [https://github.com/cloudera/flink-tutorials](https://github.com/cloudera/flink-tutorials)
- [https://docs.cloudera.com/csa/1.1.0/job-lifecycle/topics/csa-run-job.html](https://docs.cloudera.com/csa/1.1.0/job-lifecycle/topics/csa-run-job.html)
- [https://github.com/asdaraujo/edge2ai-workshop/tree/master/flink](https://github.com/asdaraujo/edge2ai-workshop/tree/master/flink)
- [https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sqlClient.html](https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sqlClient.html)
- [https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sqlClient.html ](https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sqlClient.html%20)

  

[![](https://1.bp.blogspot.com/-S1XVjr3pNB4/XrsRc9A8-uI/AAAAAAAAaUw/3L5i_P_iOEYrqOsuXdyiUBgszGwJiY9OwCK4BGAsYHg/s320/unnamed.png)](https://1.bp.blogspot.com/-S1XVjr3pNB4/XrsRc9A8-uI/AAAAAAAAaUw/3L5i_P_iOEYrqOsuXdyiUBgszGwJiY9OwCK4BGAsYHg/unnamed.png)