Position: Data Engineer
Category: Data Engineer
English level: upper
Years of experience: 3


Collected and cleansed large datasets, applying data wrangling techniques to ensure data quality and consistency. Implemented data validation and cleansing processes to identify and rectify anomalies and discrepancies in the data.
Utilized Spark to implement distributed data processing, enabling faster and more efficient analysis of big data. Leveraged Spark's capabilities for parallel processing and in-memory computing to handle large-scale data sets and improve data processing speed.
Developed robust ETL pipelines using Airflow, orchestrating the end-to-end data flow from various sources to the target database. Created and maintained DAGs (Directed Acyclic Graphs) to schedule and monitor data extraction, transformation, and loading tasks.
Collaborated closely with project managers and analysts to understand their data requirements and deliver optimized data pipelines that supported their KPI tracking needs. Translated business requirements into technical specifications, ensuring the data pipelines met the desired outcomes.
Actively participated in project documentation design and maintenance, ensuring that all processes and procedures were well-documented and easily accessible to team members. Created comprehensive documentation for data pipelines, including data flow diagrams, data mappings, and transformation rules.
Worked with a range of tools and technologies including Python, ETL, PySpark, Postgres, Redshift, Alembic, Airflow, AWS Services, Jenkins, and BitBucket. Leveraged these technologies to design, develop, and deploy data solutions that met business needs.