Position: Data Engineer
Category: Data Engineer
English level: intermediate
Years of experience: 0


Configured Amazon Kinesis Data Firehose to extract streaming data from the source
and then upload it to S3. Created delivery stream in which configured partition of
extracted data.
Created and deployed a CloudFormation template for an S3 bucket in which
configured the name of the bucket and access rights.
Scripting: Written python script for reading the parquet data from the corresponding
S3 location using the boto3 client for S3. Extracted partition attribute value
separately from the respective partition prefix. Created class that corresponds to the
events structure and mapped the events to the class instance. Collected instances in
the list and output to stdout.
Created and configured Glue Job with the Spark script editor, which uses the
concatenate and aggregate functions, added a new column to the created dataframe,
which is calculated ecpm according to a formula in which contained data from other
columns. Uploaded data to S3 storage.
Created dashboard in Grafana in which added 4 panels in which displayed
information by device for some metrics. Performed query using the Athena query
string to query data in AWS Glue data table created before.