How to create a DIY Inexpensive Cloud Data Lake
Tags: python, datascience, design, bigdata
Published: 2022-03-26 05:04:00

**Introduction**

The recent surge interest in data-driven application has led me to attempt to build a Data-Lake on the cloud.  I have always been someone to do things myself and a fan of opensource aka DIY & Opensource guy. Data science is all about how people can use this collected data and make some “learning out of it”. I will cover **supervised. unsupervised & Reinforce Learning** in another post. There are 5 stages company actively use the data (Analytics Continuum): 
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d8y0d0qecx5ilotdeukf.png)  

Don’t fret and lose sleep over this, data scientists are paid to do magic to bring use to the cognitive nirvana stage. Cognitive means machine behaving & thinking like human.


**Data-Lake**

This is what data-lake in a nutshell:

Zaloni describe it as:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vd2kdva2hpjy04r5hnhd.png)

or put it in simpler term:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jtmswy8zr8p18dx6l0d7.png) 
 
Data lifecycle: **Ingestion, ETL, Modelling, & Data Publishing, Housekeeping & Retraining.**


1. **Ingestion**
Install an **_Apache NIFI_** cluster in a Kubernetes cluster to ingest data from MQTT, pull data from API Endpoints. Additionally, an **_API Gateway_** is required for others to sink data. For **cold stream**, data will be saved in **_STAGING_** zone implemented using database, **HDFS, ElasticSearch**. For **real-time stream**, data goes to the modelling stage (can push data onto **AWS Kinesis Stream** if you are using **AWS Kinesis Analysis**).

2. **ETL**
You can install and use **_Apache Airflow_** to execute pre-populated python scripts before moving the data to the input of the Models. Again, Here we can store processed data in Document Store like **Mongo**or Graph-store **_Neo4J_**.

3. **Model**
Model is a bunch of code taking inputs, apply mathematics and output the findings. Models can be implemented using **_Python Scikit-Learn_** and real-time streaming **_Apache Spark Graph_** (**AWS Kinesis Analytics** is an alternative here). Output from these models can be classified into **_Predictive Analytics, Regression Model, Pattern Analysis_**. All these are basically mathematics. Mathematics are effectively: **_arithmetic, calculus and geometry (2d/3d)_**. Run these modelling codes inside the Kubernetes Cluster.

Here are the various modelling techniques:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l1acs9d0jfg3v5rtr5vc.png)

Here are the real time streaming platforms:
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aampwgfsh58t2xjej4uw.png)
  
4. **Data Publishing**
Effectively, the data output from the models can be used for other vendors or microservices. Data is published to **_Apache Kafka_** & persisted in a self-managed **_Mongo_**.
Insight Data can be subscribed from endpoints using API Gateway. Each customer will be given a legitimate access key to access the endpoints. Data also can be displayed in dashboard like **_Kibana_**, make available to customer.  We can setup a **_ELK_** stack in Kubernetes Cluster or you can subscribed to service like **_AWS OpenSearch_**. Data output can be made to ran through workflows for checks, for instance sending email alerts to stakeholder if abnormality is found. It makes sense to use native services (**_AWS SES & AWS Lambda_**).

5. **Housekeeping**
Data>Retaining period have to be cleared with scripts running in  **_cron-jobs or  AWS Lambda_**.

6. **Retraining**
Raw data coming in have to be stores somewhere for retraining the models. We can use storage like **_EBS, EFS, Blob Storage_** in this context. For retraining and redeployment (**_CI/CD_**) of models, we can use tools like **_Apache MLFlow_**.


**What’s next**

1. **Data Governance**
After finishing the base pipeline, the next phase is to implement data governance. We can use Opensource **_Apache Altas_**. A more enterprise one is  **_Talend Data Governance Platform_**. 
 
2. **Data Warehouse**
Output from data lake can then be store in structured **_data warehouse (AWS Redshift)_** and create **_data-mart_**. **_Snowflake Cloud Data Platform_** is a popular choice here.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jmmmihoaklla3akel88a.png)