Tracking Air Quality with Apache NiFi, Cloudera Data Science Workbench, Pyspark and Parquet
Published: 2019-12-03 21:49:47

---
title: Tracking Air Quality with Apache NiFi, Cloudera Data Science Workbench, Pyspark and Parquet
published: true
date: 2019-10-02 00:29:00 UTC
tags: 
canonical_url: 
---

# Tracking Air Quality 

Indoor vs Outdoor

  

Using a few sensors on a MiniFi node we are able to generate some air quality sensor readings.

Source: **[https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.py](https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.py)**

**Data** :

| row['bme680\_tempc'] = '{0:.2f}'.format(sensor.data.temperature) |
| row['bme680\_tempf'] = '{0:.2f}'.format((sensor.data.temperature \* 1.8) + 32) |
| row['bme680\_pressure'] = '{0:.2f}'.format(sensor.data.pressure) |
| row['bme680\_gas'] = '{0:.2f}'.format(gas) |
| row['bme680\_humidity'] = '{0:.2f}'.format(hum) |
| row['bme680\_air\_quality\_score'] = '{0:.2f}'.format(air\_quality\_score) |
| row['bme680\_gas\_baseline'] = '{0:.2f}'.format(gas\_baseline) |
| row['bme680\_hum\_baseline'] = '{0:.2f}'.format(hum\_baseline) |

**See Part 1:**

[https://community.hortonworks.com/articles/189630/tracking-air-quality-with-hdp-and-hdfi-part-1-apac.html](https://community.hortonworks.com/articles/189630/tracking-air-quality-with-hdp-and-hdfi-part-1-apac.html)

Newark / NYC Hazecam

[https://hazecam.net/images/main/newark.jpg](https://hazecam.net/images/main/newark.jpg)

Example

{"bme680\_air\_quality\_score": "82.45", "uuid": "20190131191921\_59c5441c-47b4-4f6f-a6d6-b3943bc9cf2b", "ipaddress": "192.168.1.166", "bme680\_gas\_baseline": 367283.28, "bme680\_pressure": "1024.51", "bme680\_hum\_baseline": 40.0, "memory": 11.7, "end": "1548962361.4146328", "cputemp": 47, "host": "piups", "diskusage": "9992.7", "bme680\_tempf": "87.53", "te": "761.2184100151062", "starttime": "01/31/2019 14:06:40", "systemtime": "01/31/2019 14:19:21", "bme680\_humidity": "13.22", "bme680\_tempc": "30.85", "bme680\_gas": "363274.92"}

{

"end" : "1548967753.7064438",

"host" : "piups",

"diskusage" : "9990.4",

"cputemp" : 47,

"starttime" : "01/31/2019 15:44:11",

"bme680\_hum\_baseline" : "40.00",

"bme680\_humidity" : "13.23",

"ipaddress" : "192.168.1.166",

"bme680\_tempc" : "30.93",

"te" : "301.96490716934204",

"bme680\_air\_quality\_score" : "83.27",

"systemtime" : "01/31/2019 15:49:13",

"bme680\_tempf" : "87.67",

"bme680\_gas\_baseline" : "334942.60",

"uuid" : "20190131204913\_4984a635-8dcd-408a-ba23-c0d225ba2d86",

"bme680\_pressure" : "1024.69",

"memory" : 12.6,

"bme680\_gas" : "336547.19"

}

  
[https://www.nps.gov/subjects/air/webcams.htm?site=wash](https://www.nps.gov/subjects/air/webcams.htm?site=wash)  
  

Outdoor air quality
[https://community.cloudera.com/t5/Community-Articles/Tracking-Air-Quality-with-HDP-and-HDF-Part-1-Apache-NiFi/ta-p/248265](https://community.cloudera.com/t5/Community-Articles/Tracking-Air-Quality-with-HDP-and-HDF-Part-1-Apache-NiFi/ta-p/248265)  
  
https://openweathermap.org/api/pollution/co  
  
https://airquality.weather.gov/probe\_aq\_data.php?city=hightstown&state=NJ&Submit=Get+Guidance  
  
http://feeds.enviroflash.info/rss/realtime/445.xml  
  
http://feeds.enviroflash.info/cap/aggregate.xml  
  
http://www.airnowapi.org/aq/forecast/zipCode/?format=application/json&zipCode=08520&date=2019-09-05&distance=25&API\_KEY=code  
  
https://docs.airnowapi.org/webservices  
  
http://www.airnowapi.org/aq/observation/zipCode/current/?format=application/json&zipCode=08520&distance=50&API\_KEY=  

# code
  
https://api.openaq.org/v1/measurements?country=US&date\_from=2018-05-04  
  
https://api.openaq.org/v1/latest?country=US  
  
http://www.airnowapi.org/aq/observation/zipCode/current/?format=application/json&zipCode=08520&distance=25&API\_KEY=code  
  

#   

  

  

  

  

  

  

  
**Flight Data**  
  
[https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940](https://community.cloudera.com/t5/Community-Articles/Ingesting-Flight-Data-ADS-B-USB-Receiver-with-Apache-NiFi-1/ta-p/247940)  
  
**Air Traffic Overhead**  
  
[https://opensky-network.org/api/states/all?lamin=40.270599&lomin=-74.522430&lamax=40.270599&lomax=-74.522430](https://opensky-network.org/api/states/all?lamin=40.270599&lomin=-74.522430&lamax=40.270599&lomax=-74.522430)  
  
[http://scorecard.goodguide.com/about/txt/data.html](http://scorecard.goodguide.com/about/txt/data.html)  
  
[https://www.epa.gov/visibility](https://www.epa.gov/visibility)  
  
[https://www.airnow.gov/](https://www.airnow.gov/)  
  
[https://www.state.nj.us/dep/daq/](https://www.state.nj.us/dep/daq/)  
  
[http://www.nynjpollen.com/](http://www.nynjpollen.com/)  
  
[http://www.njaqinow.net/](http://www.njaqinow.net/)  
  
[https://www.fsvisimages.com/descriptions.aspx](https://www.fsvisimages.com/descriptions.aspx)  
  
[https://www.datainmotion.dev/2019/03/iot-series-sensors-utilizing-breakout\_74.html](https://www.datainmotion.dev/2019/03/iot-series-sensors-utilizing-breakout_74.html)  
  
[https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.py](https://github.com/tspannhw/minifi-breakoutgarden/blob/master/aqminifi.py)