Position: Data Engineer
Category: Data Engineer
English level: intermediate
Years of experience: 11

Spark Jobs with Python, I would like to work with AWS / Python
MySQL (SQL) - 3 years
Oracle (SQL, pl-sql) - 8 years
PostgreSQL (SQL, PL/pgSQL) 3 years
Teradata - 2 years
AWS Redshift, EMR, S3, RDS - 3 years 
Apache Airflow - 3 years 

- maintenance and creation database objects;
- development and design of database/DWH, optimisation database structures;
- build queries based upon analytical functions, creating hierarchical queries;
- using collection, dynamic SQL;
- queries optimisation;
- creating SQL scripts for ETL processes organised in the Apache Airflow; 
- creating and maintenance Airflow DAGs
- creating PL/pgSQL functions for parsing json/jsonb PostgreSQL data type with following data transformation and loading to DWH in compliance with required SCD types. 
- creating ETL processes in the Core banking platform based upon different sources (text, xml, csv files; database tables; SQL queries) with using different approaches for data parsing (XQuery, Shell, Python scripting). Building reports based on loaded data;
- AWS: S3, EC2, RDS, Amazon Redshift, Lambda, IAM (basic level), QuickSight
- IBM InfoSphere DataStage (version 8.5)
- Python: pySpark, pandas, BeautifulSoup libraries, creating airflow DAGs
- creating Spark Jobs on Python for data aggregation in ORC, Parquet, AVRO, CSV formats.
- creating a pipelines/DAGs in Apache Airflow to process data from different sources like RDS, s3, kinesis with the using a pyspark jobs which run in EMR and upload final aggregated result to the Redshift tables;
- cost and performance optimisation EMR Clusters;
- cost optimisation of s3 resources.
interesting to work with the following tech stack: Data Engineer with Python, Spark, AWS Services.