Position: Data Engineer
Category: Data Engineer
English level: fluent
Years of experience: 2


Professional Experience

Data Engineer & Mentor at Making Science • Sweeft, July 2021-Present

- Developed an analytical and visualization platform for automatically ingesting data to and from multiple marketing sources.
- Designed, developed, built, deployed and migrated existing numerous and reusable data pipelines using serverless and serverful technologies on Google Cloud Platform, resulting in a 50% reduction in data processing time and costs.
- Developed and maintained Data Lakes and Warehouses on Google Cloud Platform.
- Utilized best practices to support continuous process automation for data ingestion and data pipeline workflows.
- Developed reusable CI/CD workflows and IAC for data pipelines using Terraform and Cloud Build.
- Continually explored new technologies in an effort to always grow and uncover better ways to solve problems.
- Worked closely with stakeholders across departments to design, build and deploy various initiatives within the data platform
- Mentored interns aiming to become Data Engineers.

Technologies Used:
- Apache Spark(PySpark) on Dataproc
- Apache Beam on Dataflow
- Docker
- Cloud Build
- Git
- Python
- Bash
- BigQuery
- Firestore
- PostgreSQL
- Jupyter Notebooks
- pytest
- Terraform
- Airflow(Composer)
- FastAPI
- Cloud Pub/Sub
- Cloud Functions
- Cloud Run
- Cloud Workflows
- Cloud Storage


Personal Projects

ISS Satelite

- Built an on-premise ELT pipeline from a REST API providing real-time data about the ISS satellite.
- Designed PSA, Data Warehouse and Data Marts in PostgreSQL with read replicas using cross-database real-time writing.
- Used Apache Airflow for batch processes along with Redis for caching the latest data for the satellite.

Air Quality, Weather and Mortality Data Insights

-Collected continuously updated air quality, weather and mortality data from public OpenAQ, NCEI CDO and CDC sources
- Designed a Data Warehousing(Data Vault 2.0) solution and deployed the project on Google Cloud Platform entirely using Terraform and Cloud Build CI/CD.
- Developed ETL pipelines in Apache Spark(Dataproc) and Apache Beam(Dataflow), orchestrated by Airflow(Composer).
- Enriched the data and filled missing spots using various algorithms and interpolation techniques such as IDW and RBF.
- Built a presentation layer with visualizations and analysis.