Position: Data Engineer
Category: Data Engineer
English level: intermediate
Years of experience: 8


Mar-2021 - Till now Tasks performed: • Developed ETL batch Spark job, covered with unit tests • Gained experience with AWS services(Glue, SNS, CloudWatch, Cloudformation, SecretsManager, IAM, S3). Environment: • AWS services, Github, Maven, Git, Jenkins, Jira • Spark, Scala, Java, AWS Services Oct-2020 - Mar-2021 Project: Aim of the project is to gain experience in production like tasks that included simulation of ETL process that included implementing and orchestrating multiple Spark jobs. Multiple batch and streaming jobs were implemented to clean, transform and load raw data into datawarehouse. As datawarehouse solution Hive was used. Hortonwork Data Platform was used as run all needed software. Gained experience with AWS services(EC2, EMR, S3) Tasks performed: • Developed ETL batch and streaming Spark job, covered with unit tests • Working with Hive managed and external tables • Implemneted solution for tracking already processed files • Gained experience working with Kafka API and managing offsets • Gained experience with AWS services(EMR, EC2, IAM, S3). Environment: • Hive, MongoDB • Intellij IDEA, Virtualbox, HDP, Data analytics studio, Maven, • Apache Hive, Apache Hadoop, Kafka, Spark, Spark Streaming, Java 8, MongoDB May-2020 - Oct-2020 Project: Project to track RD-related activities and education costs in UA Office, to separate it from other GDOs and mid-term to replace huge and old EPM-FARM. Large amount of tables for store information about test, users, topics, groups of testers, liquibase information, test sessions for testers, permissions for test manager and so on. PostgeSQL is using for RD Portal, it contains information about students, locations, time slots of testing and so on Tasks performed: • I worked with DB: searching, retrieving, sorting, updating and etc. Wrote REST end-points, covered code with JUnit-test, bugfixing, change/add configurstions, work with email- notifications, create/improve/change scheduled tasks, refactor code. • In case of difficulties with the work process, I contacted the team lead or project manager. • Worked in SCRUM team, completed tasks according to SCRUM sprint. Completed tasks was checked by Team Lead. Environment: • MySQL, PostgeSQL. MySQL is using for Test Portal. • Intellij IDEA, MYSQL Workbench, Git, JIRA
Looking for an interesting project with progressive technologies, infrastructure and professional team