Using Apache Airflow ETL to fetch and analyze BTC data
Tags: etl, apacheairflow, dataengineering, datapipeline
Published: 2021-10-08 14:05:38

---
title: Using Apache Airflow ETL to fetch and analyze BTC data
published: true
date: 2021-10-05 14:51:38 UTC
tags: etl,apacheairflow,dataengineering,datapipeline
canonical_url: https://itnext.io/using-apache-airflow-etl-to-fetch-and-analyze-btc-data-1adab96c410c
---

![](https://cdn-images-1.medium.com/max/1024/0*MiiUoIfEgyTolSx1.png)

I am taking a short break from [Blockchain-related](https://blog.adnansiddiqi.me/tag/blockchain-programming) posts.

In past, I have covered Apache Airflow posts [here](http://blog.adnansiddiqi.me/tag/airflow/). In this post, I am discussing how to use the [CCXT](https://github.com/ccxt/ccxt) library to grab BTC/USD data from exchanges and create an ETL for data analysis and visualization. I am using the [dockerized](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html) version of Airflow. It is easy to set up and using proper different images to run different components instead of a one-machine setup. The docker-compose.yml file is available [here](https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml) but I have made a few changes to install custom libraries. Therefore, it is advised you use the file I have provided in the Github repo. Before I continue, the demo of the work is given here:

{% youtube 72scllBBgVo %}

### The DAG

The entire flow looks like the below:

![](https://cdn-images-1.medium.com/max/1024/0*rNYzjh-Cg7kO3vhM.png)

I am fetching BTC/USD data from three different exchanges: Binance, ByBit, and FTX. Once the data is available, I am sending all three sources into the next stage where I am only pulling _close_ and _timestamp_ data because these are fields that I need for the next stages. From here, I am branching out two tasks: one to create text files and storing data for later use and the other is storing into the DB. The 3 different files are then sent to the server via FTP. The data from load\_data section is then used to generate reports and then FTPing the file to the server. Notice that some borders are dark green and some are light green in color. Dark green means these tasks are executed successfully while light green means they are still _running_. The task, _ftp\_pdf\_file_ is yet to run. In Airflow terms, the DAG flow has been set like below:

![](https://cdn-images-1.medium.com/max/1024/0*Zb2m1B9uBsXsdZHc.png)

In first stage all 3 tasks run in parallel and output results to transform\_data therefore I put all 3 tasks in a Python list. The **>>** sign is telling that it is an upstream. The transformed data is being to create\_text\_file and load\_data tasks who themselves are _upstream_ for ftp\_file and generate\_pdf\_reports tasks. Notice I am using << here to define upstream for ftp\_pdf\_file and generate\_pdf\_reports tasks. Again these tasks were first added in a Python list and then branched out for the next stage.

In order to pass data from one task to another task, Airflow provides xcom\_pull and xcom\_push methods. **XCom** means _Cross Communication_. When you use xcom\_pull, it means you are fetching data from the task whose Ids have been passed. For instance:

![](https://cdn-images-1.medium.com/max/1024/0*IbGXPgIzUNgqgMgT.png)

Here I passed the task\_id of three tasks and assigned in three different variables. Wondering where those task\_ids are defined? they are defined as PythonOperator.

![](https://cdn-images-1.medium.com/max/1024/0*nadvUaDgPkhmt8K2.png)

Check the log of the task fetch\_binance\_ohlcv:

![](https://cdn-images-1.medium.com/max/1024/0*eioIyvWVEZgRhuft.png)

I highlighted the returned value of the task. This and the other two tasks values are _xcom\_pulled_ in the transform\_data task.

I am also using two external components here: FTP and Database. You may use the traditional python code by importing the required libraries but Airflow also provides the options of _hooks_ and _connections_. For instance, pgsql connection looks like below:

![](https://cdn-images-1.medium.com/max/1024/0*ZlnnFK7nxEktzduP.png)

Once the connection is defined, you may use PG Hook to perform Postgres-based operations.

![](https://cdn-images-1.medium.com/max/1024/0*n6KdYNrH-SwJRDCk.png)

Cool, isn’t it? No need to saving credentials in the code. The same thing has been done with FTP.

![](https://cdn-images-1.medium.com/max/1024/0*c5dVsWu-iFQ7r0ey.png)

In the end, data is analyzed and generates a two-page PDF for executives.

![](https://cdn-images-1.medium.com/max/1024/0*Gufi9pOpU9KMjzJd.png)

the graph is being generated in _matplotlib_ and saved as an embedded image in the PDF file.

![](https://cdn-images-1.medium.com/max/1024/0*lPr0Itb431V-TpUF.png)

Sweet, No? I was supposed to add data and charts of both FTX and ByBit too but I leave it for you ;). Also, this is not a heavy analysis of the data. I have provided you all data need. You may play with ta-lib or other libraries and find insights. All up to you 😉

I am using _fpdf_ python library to generate PDF files. This is how files are being sent via FTP.

![](https://cdn-images-1.medium.com/max/1024/0*7JVeHmhZPiG1jh_4.png)

### Conclusion

In this post, you learned how you can make complex flows for ETLs and use connections and hooks to connect 3rd party tools like FTP, DB, AWS, etc. Airflow could be a pretty powerful tool if used correctly. Like always, the code is available on [Github](https://github.com/kadnan/AirFlowCryptoETL).

_Originally published at_ [_http://blog.adnansiddiqi.me_](http://blog.adnansiddiqi.me/using-apache-airflow-etl-to-fetch-and-analyze-btc-data/) _on October 5, 2021._

* * *