Data Engineering Projects for Beginners
Tags: database, dataengineering, python
Published: 2022-06-15 23:40:58

Hi everyone,

I am a little bit obsessed with data engineering and lately I have been working on several open source projects about this topic, here is a list of repositories and technologies used in each one, if you decide to go deeper into this funny world then these repositories could help you as a guide.

❤ means "I like this one"


### ❤ **[Tracking your Uber Rides and Uber Eats expenses through a data engineering process](https://github.com/Wittline/uber-expenses-tracking)**
#### **Technologies and skills**:
_Python, Docker, Apache Airflow, AWS Redshift, Power BI, data modelling, Task schedulling, ETL and ELT processes, Data warehousing, Cloud_

### ❤ **[Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag](https://github.com/Wittline/pyDag)**
#### **Technologies and skills**:
_Python, Docker, Big Data, Cloud, BigQuery, Workflow Engines, GCP, Task scheduler, Google Cloud Platform, Dataproc cluster, GCS, Google Cloud Storage, Redis, DAG, Parallel Processing, Apache Spark_

### ❤ **[Building Big Data Pipelines in the Cloud with AWS EMR](https://github.com/Wittline/pyspark-on-aws-emr)**
#### **Technologies and skills**:
_Python, PySpark, AWS EMR, Task Schedulling, IAC, EC2 Instances, Apache Spark, Cloud_

### ❤ **[Building a Lossless Data Compression and Data Decompression Pipeline](https://github.com/Wittline/wbz)**
#### **Technologies and skills**:
_Python, Data compression, BZIP2, Parallel programming_

### **[Learn how to dockerize an Apache Spark Standalone Cluster](https://github.com/Wittline/apache-spark-docker)**
#### **Technologies and skills**:
_Python, Jupyter Notebook, Apache Spark, Docker, docker-compose, Hive_

### ❤ **[Dockerizing and Consuming an Apache Livy environment](https://github.com/Wittline/docker-livy)**
#### **Technologies and skills**:
_Python, Big Data, Docker, docker-compose, Apache Livy, Apache Spark, PostgreSQL, PySpark, Jupyter Notebook_

### ❤ **[Design, Development and Deployment of a simple Data Pipeline](https://github.com/Wittline/data-engineer-challenge)**
#### **Technologies and skills**:
_Python, data Modelling, Docker, docker-compose, PostgreSQL, data pipeline, FastApi_

### **[Dockerizing a Python Script for Faster Web Scraping](https://github.com/Wittline/data-engineering-challenge-th)**
#### **Technologies and skills**:
_Python, Docker, Sqlite, Dockerfile, Web scraping, Data pipeline,  FastApi_

### **[Understanding Similarity Measures for Text Analysis](https://github.com/Wittline/distance-metrics)**
#### **Technologies and skills**:
_Python, Machine Learning, Similarity measures, Distance metrics, Text Analysis_

### ❤ **[Learn how to build a content-based Movie Recommender System](https://github.com/Wittline/recommendation-system)**
#### **Technologies and skills**:
_Python, Machine Learning, TF-IDF, Cosine similarity, BM25, BERT, NLP, word2vec, Text Analysis, recsys_

### **[A Text Analysis of Speeches](https://github.com/Wittline/text-analysis-speeches-amlo)**
#### **Technologies and skills**:
_Python, Machine Learning, NLP, word2vec, Text Analysis, Sentiment Analysis, PCA, t-SNE, Word Embeddings, Text Preprocessing, Web scraping, Data Visualization, Mexico_

### ❤ **[Dropout Students Prediction](https://github.com/Wittline/Dropout-Students-Prediction)**
#### **Technologies and skills**:
_R, Genetic algorithm, Neural Networks, K-Means, Clustering, Machine Learning_



## I will be working on more complex projects in the next months using modern tech data stacks.