Data Engineering Series #2: Cloud Services and FOSS in Data Engineer's world
Tags: aws, opensource, datascience
Published: 2020-06-24 05:25:02


<br>**"<i>Open Source (OSS)</i>** frameworks have improved the quality of Big Data processing with its diverse set of tools addressing numerous use cases 

In fact, if you are a part of a team working on building a modern data architecture, chances are high you are using an open-source stack. 
<br>
Similarly,  **<i>Cloud Computing</i>** has been enabling Big Data Solutions in yielding scalable and cost-effective solutions in analytics space.

------------

<h3><center><b>Open Source and Cloud : The Correlation</b></center></h3>


In the <i>cloud ecosystem</i>, many of the commercially available **cloud services** are either  

**Similar to an OSS** ➡ Similar in Features <i>(**Eg:** AWS Step Functions and Apache Airflow )</i>
<center>OR</center>
**Modeled after an OSS** ➡ Follows/ Inherits the design principles of an existing Open Source framework. <i>(**Eg:** AWS Kinesis and Apache Kafka)</i>
<center>OR</center>
**Managed service of an OSS** ➡ Takes care of deployment & maintenance of the OSS framework and making it ready to use. <i>(**Eg:**  AWS RDS Postgres and PostgresDB)</i>

To understand more, Let's touch upon the basics...


------------

<h3><center><b>Getting to know the cloud</b></center></h3>

The first step that many of us go through while getting to know about cloud services is to start wondering where to start from the plethora of services available out there. 

So, For the ease of understanding, Irrespective of the cloud provider (AWS, Azure, GCP, etc). let's group the **big data related cloud services** into these stages.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/di5xj6ds1bxu9iid7o7s.png)


------------
 


Now, Let's try to understand the cloud ecosystem by comparing  **AWS** cloud services with its equivalent open source frameworks. (Similar comparison can be drawn with Azure and GCP as well)


<h4><b>:round_pushpin: Data Ingestion:</b></h4>

| AWS Service 	| What it does  	| Relation with OSS| OSS Alternative|
|-------------	|----------------	|-----------------------	|------------------	|
| [Kinesis](https://aws.amazon.com/kinesis/)     	| Stream Processing 	| **Modelled After**     	| [Apache Kafka](https://kafka.apache.org/)     	|
| [SQS](https://aws.amazon.com/sqs/)         	| Message Queue  	| **Similar to**            	| [RabbitMQ](https://www.rabbitmq.com/)         	|
| [Managed Streaming for Kafka (MSK)](https://aws.amazon.com/msk/)         	| Stream Processing 	| **Managed Service of**  	| [Apache Kafka](https://kafka.apache.org/)     	|


<h4><b>:round_pushpin: Data Storage:</b></h4>

| AWS   Service        	| What it does          	| Relation with OSS| OSS   Alternative       	|
|----------------------	|-----------------------	|-----------------------	|---------------------------	|
| [S3](https://aws.amazon.com/s3/)                   	| Object store          	| **Similar to**            	| [Minio](https://min.io/), [Swift](https://launchpad.net/swift), [Ceph](https://ceph.io/),   ... 	|
| [RDS](https://aws.amazon.com/rds/)                  	| Relational database   	| **Managed Service of**  	| [MariaDB](https://mariadb.org/), [MySQL](https://www.mysql.com/), [Postgres](https://www.postgresql.org/)  	|
| [DynamoDB](https://aws.amazon.com/dynamodb/)             	| NoSQL database        	| **Similar to**            	| [Apache Cassandra](https://cassandra.apache.org/ "Apache Cassandra")          	|
| [ElastiCache](https://aws.amazon.com/elasticache/)          	| In-memory cache       	| **Managed Service of**                      	| [Memcached](https://www.memcached.org/ "Memcached"), [Redis](https://redis.io/ "Redis")          	|
| [Neptune](https://aws.amazon.com/neptune/)              	| Graph database        	| **Similar to**                      	| [Neo4j](https://neo4j.com/ "Neo4j")                     	|
| [Amazon QLDB](https://aws.amazon.com/qldb/)          	| Ledger database       	| **Modelled After**     	| [Hyperledger](https://www.hyperledger.org/ "Hyperledger")               	|
| [Amazon DocumentDB](https://aws.amazon.com/documentdb/)  	| Document database     	|**Similar to**            	| [MongoDB](https://www.mongodb.com/ "MongoDB")                   	|
| [AWS Lake Formation](https://aws.amazon.com/lake-formation/) 	| Data lake             	| **Similar to**            	| [HDFS](http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html "HDFS")                      	|
| [EC2 EBS](https://aws.amazon.com/ebs/)              	| Block storage for EC2 	| **Similar to**                      	| [OpenEBS](https://www.openebs.io/ "OpenEBS"), [Portworx](https://github.com/portworx/px-dev "Portworx")         	|

<h4><b>:round_pushpin: Data Processing:</b></h4>

| AWS   Service 	|  What it does  	|       Relation with OSS|   OSS   Alternative  	|
|:-------------:	|:--------------:	|:---------------------:	|:---------------------:	|
|      [Elastic Map Reduce](https://aws.amazon.com/emr/)      	| Hadoop  	| **Managed Service of**  	|     [Hadoop](https://hadoop.apache.org/),      	|
| [Step Functions](https://aws.amazon.com/step-functions/) 	|       Worflow Orchestrator      	| **Similar to**                       	|     [Apache Airflow](https://airflow.apache.org/) ,  [Flyte](https://flyte.org/)   	|
|    [AWS Glue](https://aws.amazon.com/glue/)   	|       ETL      	|    **Managed Service of**    	|      [Apache Spark](https://spark.apache.org/)     	|
|     [Lambda](https://aws.amazon.com/lambda/)    	|   Serverless   	|      **Similar to**       	| [Knative](https://knative.dev/ "Knative"), [OpenFaaS](https://www.openfaas.com/ "OpenFaaS"), [Fn](https://fnproject.io/ "Fn") 	|
|     [Batch](https://aws.amazon.com/batch/)     	|    Batch Job Computing   	|      **Similar to**       	|     [Apache Airflow on Kubernetes](https://airflow.apache.org/docs/stable/kubernetes.html)    	|

<h4><b>:round_pushpin: Data Analysis & Visualization:</b></h4>

|      AWS   Service      	|    What it does    	|  Relation with OSS|        OSS   Alternative        	|
|:-----------------------:	|:------------------:	|:-----------:	|:--------------------------------:	|
| [Amazon   Redshift](https://aws.amazon.com/redshift)       	| Data warehousing   	| **Similar to** 	| [Spark SQL](https://spark.apache.org/sql/), [Apache Hive](https://hive.apache.org/), [Presto](https://prestodb.io/) 	|
| [Athena](https://aws.amazon.com/athena)                  	| Data warehousing   	| **Similar to**            	| [Spark SQL](https://spark.apache.org/sql/), [Apache Hive](https://hive.apache.org/), [Presto](https://prestodb.io/) 	|
| [CloudSearch](https://aws.amazon.com/cloudsearch)	| Search             	|**Similar to**              	| [Elasticsearch](https://www.elastic.co/)                    	|
| [Elasticsearch Service](https://aws.amazon.com/elasticsearch-service) 	| Search      	| **Managed Service of**          	| [Elasticsearch](https://www.elastic.co/)                    	|
| [QuickSight](https://aws.amazon.com/quicksight)              	| Business analytics 	| **Similar to** 	| [PowerBI](https://powerbi.microsoft.com/en-us/)                 	|

<h4><b>:round_pushpin: Deployment:</b></h4>

| AWS Service     	| What it does             	| Relation with OSS| OSS Alternative      	|
|-----------------	|--------------------------	|---------------------	|-----------------------	|
| [Elastic Container Registry (ECR)](https://aws.amazon.com/ecr)             	| Container registry       	| **Managed Service of**  	| [Docker Registry](https://www.docker.com/), [Quay](https://quay.io/) 	|
| [Elastic Container Service (ECS)](https://aws.amazon.com/ecs)          	| Container orchestration  	| **Managed Service of**  	| [Kubernetes](https://kubernetes.io/), [Marathon](https://mesosphere.github.io/marathon/)  	|
| [Elastic Kubernetes Services (EKS)](https://aws.amazon.com/eks)           	| Container orchestration  	| **Managed Service of**  	| [Kubernetes](https://kubernetes.io/)           	|
| [Cloud Formation](https://aws.amazon.com/cloudformation) 	| Infrastructure as a code 	| **Similar to**          	| [Terraform](https://www.terraform.io/)             	|

<h4>Some of the notable cloud adoptions with respect to Big Data.</h4>

<blockquote>- Till now, AWS users have launched more than 15 million Hadoop clusters. (EMR / Containerized versions) <br>
- "container-as-a-service" (EKS, ECS) and "Database-as-a-service" (RDS, DynamoDB) are the most commonly used managed services in 2020.<br>
- Database services usage up 127% year over year.
</blockquote>

Next Steps...
1. You can understand how these services are put to use in real-world use cases in this [article](https://aws.amazon.com/big-data/use-cases/)
2. This [Whitepaper](http://d0.awsstatic.com/whitepapers/Big_Data_Analytics_Options_on_AWS.pdf) from AWS on Big Data will be a good place to understand its Services.
3. And start getting hands-on following this [repo](https://github.com/manuparra/starting-bigdata-aws)


Going forward, I'll publish detailed posts on tools and frameworks used by Data Engineers day in and day out.

<img src="https://media.giphy.com/media/J4zA6LplubvC5weDyo/giphy.gif" width="120" height="120"/>

Follow for updates.