Position: Data Software Engineer (Java/Scala/Groovy)
Category: Data Engineer
English level: upper
Years of experience: 2

Main achievement is prepared generic solution of Delta table monitoring mechanism, that is currently accepted for all other similar features on project. I'm enthusiastic in building architecture of data pipeline and few my ideas was approved by customer and implemented.
On last project I was working on development ETL data pipeline using Apache Spark, Scala, SQL on Databricks.
General description of project activity:
1. Creating monitoring mechanism using CDF (Change Data Feed) tool
2. Loading batch data using Apache NiFi and store data on AWS S3 storage
3. Loading data from AWS S3 bucket into Databricks bronze layer table as Delta table with location on AWS S3.
4. Extraction and transformation data from bronze layer table based on business requirements and save result on silver layer
5. Creation of streaming flow for INSERT/UPDATE/DELETE both layer tables
6. Introduction of logging and metrices collection
7. Dashboard creation based on collected metrics
8. Creation of Databricks data flow using notebooks created earlier.
9. Promotion into production environment.
10. Working with Databricks APIs (Secrets API, Jobs API etc.) and CLI

On previous projects I worked with Apache NiFi, where we built data pipeline using Groovy. Also my previous experience include Apache Kafka, ELK stack, Cassandra etc.
I have some educational experience with Google Cloud Platform (GCP) and Azure Cloud.
Other hands-on tools: DBeaver, Postman, Kibana, DataDog, Terraform