Ingesting All The Weather Data With Apache NiFi
Tags: apachenifi
Published: 2020-07-11 03:11:59

---
title: Ingesting All The Weather Data With Apache NiFi
published: true
date: 2020-07-10 19:02:00 UTC
tags: apachenifi
canonical_url: 
---

<font size="5"><br></font>
<font size="5"><b>Ingesting All The Weather Data With Apache NiFi</b></font>

  

  

[![](https://1.bp.blogspot.com/-xQPASF1FTc0/Xwi62A2SF_I/AAAAAAAAbP8/shkg_HH9hIUrCySv2bjYajF34rgvRBhCgCLcBGAsYHQ/w650-h781/ingestusweather.png)](https://1.bp.blogspot.com/-xQPASF1FTc0/Xwi62A2SF_I/AAAAAAAAbP8/shkg_HH9hIUrCySv2bjYajF34rgvRBhCgCLcBGAsYHQ/s1566/ingestusweather.png)

  

**Step By Step NiFi Flow**

  

1. **GenerateFlowFile -** build a schedule matching when NOAA updates weather
2. **InvokeHTTP -** download all weather ZIP
3. **CompressContent -** decompress ZIP
4. **UnpackContent -** extract files from ZIP
5. **\*RouteOnAttribute -** just give us ones that are airports (${filename:startsWith('K')}). **optional**.
6. **\*QueryRecord** - XMLReader to JsonRecordSetWriter.    **Query** :  SELECT \* FROM FLOWFILE WHERE NOT location LIKE '%Unknown%'.  This is to remove some locations that are not identified.  **optional**.
7. Send it somewhere for storage.   Could put PutKudu, PutORC, PutHDFS, PutHiveStreaming, PutHbaseRecord, PutDatabaseRecord, PublishKafkaRecord2\* or others.

[![](https://1.bp.blogspot.com/-rOUfWVVPToo/Xwi610zm-EI/AAAAAAAAbP4/hV5Zq9t6SAEyx_YELQ5Jzd7_nPGhge1ZgCLcBGAsYHQ/w625-h335/invokehttp.png)](https://1.bp.blogspot.com/-rOUfWVVPToo/Xwi610zm-EI/AAAAAAAAbP4/hV5Zq9t6SAEyx_YELQ5Jzd7_nPGhge1ZgCLcBGAsYHQ/s1275/invokehttp.png)

  

[![](https://1.bp.blogspot.com/-V8aDi477gew/Xwi61-oW4-I/AAAAAAAAbP0/hStSdAICq0ED7pH0Sff-9Mz-esCwC0CawCLcBGAsYHQ/w625-h186/decompress.png)](https://1.bp.blogspot.com/-V8aDi477gew/Xwi61-oW4-I/AAAAAAAAbP0/hStSdAICq0ED7pH0Sff-9Mz-esCwC0CawCLcBGAsYHQ/s1041/decompress.png)
  

[![](https://1.bp.blogspot.com/-FXUcBKH2MP8/Xwi62h7e_ZI/AAAAAAAAbQA/ppKQ82HwKrwF9IlrAxsF5AP3TJk5w9-iACLcBGAsYHQ/w625-h163/queryrecord.png)](https://1.bp.blogspot.com/-FXUcBKH2MP8/Xwi62h7e_ZI/AAAAAAAAbQA/ppKQ82HwKrwF9IlrAxsF5AP3TJk5w9-iACLcBGAsYHQ/s1276/queryrecord.png)
  

[![](https://1.bp.blogspot.com/-ZEzTmow4UkQ/Xwi62zZnnsI/AAAAAAAAbQE/m0dCPx1Egnsvp8Xv8VVhS64F0ShG6Km8gCLcBGAsYHQ/w625-h126/route.png)](https://1.bp.blogspot.com/-ZEzTmow4UkQ/Xwi62zZnnsI/AAAAAAAAbQE/m0dCPx1Egnsvp8Xv8VVhS64F0ShG6Km8gCLcBGAsYHQ/s1068/route.png)
  

[![](https://1.bp.blogspot.com/-fV6i5utm0LY/Xwi63L2AZ8I/AAAAAAAAbQI/If_UfoRN0g43B5zi7AWsb4O_rYdgRLZMACLcBGAsYHQ/w625-h125/unpack.png)](https://1.bp.blogspot.com/-fV6i5utm0LY/Xwi63L2AZ8I/AAAAAAAAbQI/If_UfoRN0g43B5zi7AWsb4O_rYdgRLZMACLcBGAsYHQ/s1043/unpack.png)
  

[![](https://1.bp.blogspot.com/-_xNTuz4lTbM/Xwi63SpwMxI/AAAAAAAAbQM/oXiP-SViT1wnB4eFmFGutwGj5dZHUQd2gCLcBGAsYHQ/w625-h525/xmlreader.png)](https://1.bp.blogspot.com/-_xNTuz4lTbM/Xwi63SpwMxI/AAAAAAAAbQM/oXiP-SViT1wnB4eFmFGutwGj5dZHUQd2gCLcBGAsYHQ/s829/xmlreader.png)

  

  

  

**URL For All US Data**

  

invokehttp.request.url

https://w1.weather.gov/xml/current\_obs/all\_xml.zip

  

  

  

**Example Record As Converted JSON**

  

[ {

  "credit" : "NOAA's National Weather Service",

  "credit\_URL" : "http://weather.gov/",

  "image" : {

    "url" : "http://weather.gov/images/xml\_logo.gif",

    "title" : "NOAA's National Weather Service",

    "link" : "http://weather.gov"

  },

  "suggested\_pickup" : "15 minutes after the hour",

  "suggested\_pickup\_period" : 60,

  "location" : "Stanley Municipal Airport, ND",

  "station\_id" : "K08D",

  "latitude" : 48.3008,

  "longitude" : -102.4064,

  "observation\_time" : "Last Updated on Jul 10 2020, 9:55 am CDT",

  "observation\_time\_rfc822" : "Fri, 10 Jul 2020 09:55:00 -0500",

  "weather" : "Fair",

  "temperature\_string" : "66.0 F (19.0 C)",

  "temp\_f" : 66.0,

  "temp\_c" : 19.0,

  "relative\_humidity" : 83,

  "wind\_string" : "South at 6.9 MPH (6 KT)",

  "wind\_dir" : "South",

  "wind\_degrees" : 180,

  "wind\_mph" : 6.9,

  "wind\_kt" : 6,

  "pressure\_in" : 30.03,

  "dewpoint\_string" : "60.8 F (16.0 C)",

  "dewpoint\_f" : 60.8,

  "dewpoint\_c" : 16.0,

  "visibility\_mi" : 10.0,

  "icon\_url\_base" : "http://forecast.weather.gov/images/wtf/small/",

  "two\_day\_history\_url" : "http://www.weather.gov/data/obhistory/K08D.html",

  "icon\_url\_name" : "skc.png",

  "ob\_url" : "http://www.weather.gov/data/METAR/K08D.1.txt",

  "disclaimer\_url" : "http://weather.gov/disclaimer.html",

  "copyright\_url" : "http://weather.gov/disclaimer.html",

  "privacy\_policy\_url" : "http://weather.gov/notice.html"

} ]

  

  

**<font size="5">Source Code</font>**

[https://github.com/tspannhw/ClouderaFlowManagementWorkshop/tree/main/flows](https://github.com/tspannhw/ClouderaFlowManagementWorkshop/tree/main/flows)

  

**<font size="5">Resources</font>**

  

- [https://www.datainmotion.dev/2020/05/cloudera-flow-management-101-lets-build.html](https://dev.to/tspannhw/cloudera-flow-management-101-let-s-build-a-simple-rest-ingest-to-cloud-datawarehouse-with-lowcode-powered-by-apache-nifi-2275)
- [https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html](https://www.datainmotion.dev/2019/03/advanced-xml-processing-with-apache.html)
- [https://www.datainmotion.dev/2020/01/analyzing-wood-burning-stoves-with\_23.html](https://dev.to/tspannhw/analyzing-wood-burning-stoves-with-flank-stack-part-2-analytics-497h)
- [https://www.datainmotion.dev/2020/01/cloudera-edge2ai-minifi-java-agent-with.html](https://dev.to/tspannhw/cloudera-edge2ai-minifi-java-agent-with-raspberry-pi-and-thermal-camera-and-air-quality-sensor-part-1-3oo9)
- [https://community.cloudera.com/t5/Community-Articles/Tracking-Air-Quality-with-HDP-and-HDF-Part-1-Apache-NiFi/ta-p/248265](https://community.cloudera.com/t5/Community-Articles/Tracking-Air-Quality-with-HDP-and-HDF-Part-1-Apache-NiFi/ta-p/248265)
- [https://community.cloudera.com/t5/Community-Articles/Part-2-IoT-Augmenting-GPS-Data-with-Weather/ta-p/245685](https://community.cloudera.com/t5/Community-Articles/Part-2-IoT-Augmenting-GPS-Data-with-Weather/ta-p/245685)