Position: Data Engineer
Category: Java
English level: intermediate
Years of experience: 0.5


Bike Rental Service 
Implemented data pipelines such as:
1) Spark streaming from MinIO into Kafka topic
2) Spark batch pre-processing data for further computation
3) Spark batch processing two datasets from MinIO and Cassandra with further saving to Elastic and visualization in Kibana
4) Kafka alert producer via Kafka Streams API
Additionally, I was responsible for: Spark Stateful Streaming operations; building uber-jars and creating common Utils module using Maven; Airflow with Spark containerization and deployment using custom image.
As a team member, I was involved in writing docker-compose files and entrypoint scripts for setting up the environment, configurating spark to execute multiple jobs in FAIR mode, creating DAG for batch jobs orchestration via airflow and bug-fixing on every development stage as well.