Learnings from my last assignment 
Tags: learnings, avro, kafka, presto
Published: 2020-01-03 06:28:22


---
title: Learnings from my last assignment 
published: true
description: Revisiting the year 2019, I also wanted to take time to document all the learnings from my last assignment. Here I am jotting down the learnings from my last assignment. While I learnt many things, one of the most important lesson to me goes in sync with a great man&#8217;s words &#8220;All I know is, Nothing&#8221;.
tags: Learnings, Avro, Kafka, Presto
canonical_url: https://anilkulkarni.com/2019/12/learnings-from-my-last-assignment/
cover_image: https://i1.wp.com/anilkulkarni.com/wp-content/uploads/2019/12/Collage-of-Logos.jpg?fit=1280%2C1143&ssl=1
---

![](https://i1.wp.com/anilkulkarni.com/wp-content/uploads/2019/12/Collage-of-Logos.jpg?w=665)Learnings from my last assignment

The year 2019 as well as my previous work assignment, has been very interesting, challenging and educating. I happen to work on different technologies and domains. 
Here I am jotting down the learnings from my last assignment. While I learnt many things, one of the most important lesson to me goes in sync with a great man’s words “**All I know is, Nothing**“.

Standards
---------

- Importance of Coding and Logging standards for an organization or at-least a team. By defining those standards upfront, the effort saved as the application grows bigger was quite evident to me.
- Importance of defining metrics and having a clear distinction between application and business metrics.
- Identifying standards for publishing events and consuming them.
- Identifying data storage formats, I worked mainly on Apache Avro. The advantages it brings when there is a Schema Evolution.
- Importance Encryption and Tokenization standards for the data we deal with. You know privacy is no more a wishlist, but a law. It also made me understand difference between the two terms used here.

Apache Avro
-----------

- Importance of defining a schema/model for the data we deal with and how Avro enforces certain data quality checks in the pipeline.
- I also understood what schema evolution is and how its needs to be a planned move. Enforcing schema compatibility checks before changing them.
- Avro IDL makes it super easy to define/design schemas for the data we deal with.
- Challenges involved with Union and complex union types.
- Impacts of breaking schema changes on production systems and probable solutions to handle that.
- Defining our custom Logical type as part of Avro. Example : Encryption and Tokenization.

Kafka
-----

- I learnt a lot about how Kafka can fit into some of the applications, especially when its event based.
- Difference between System and Business events.
- Leveraging Schema registry to enforce checks while producing and consuming messages from topic.
- Kafka headers and how that can be leveraged in the pipeline.
- Producing data with multiple schemas versus single schema to a single topic.
- Importance of metadata (Data about data).
- Kafka Connect and use cases around it.
- A little bit about Kafka Streams.

Spark
-----

- Learnt Scala (just enough for Spark), build and execute jobs by leveraging Livy.
- Understand about partitioning, re-partitioning, data shuffles.
- Consuming messages from Kafka in batch mode.
- Learnt few things about the executor, executor cores and memory management in Spark.
- [Spark History](https://anilkulkarni.com/2019/06/how-to-view-spark-history-logs-locally/), Zeppelin notebooks for Spark.
- Unit testing in Spark and its importance.

Presto/Hive
-----------

- Presto is nothing but SQL, but all the processing in-memory.
- [Writing UDF for Presto](https://anilkulkarni.com/2019/03/create-udf-presto-1/)
- [Hive QL](https://gist.github.com/anilkulkarni87/2e234f3fcae39ef89ed4dbe15c4dd72b).

Airflow
-------

- Writing workflow as code and understanding of how Airflow works.
- Creating custom Operators.

Docker and Kubernetes
---------------------

- How docker can help us have $0 infrastructure cost during development and Unit testing.
- Kubernetes is still a partially known area. I learnt about accessing the pods, managing secrets, YAML files.

Design Patterns – Java
----------------------

- From reading about Design patterns to actually to see it being used.
- Importance of Unit Testing and Code Reviews.

Domains
-------

I have listed some of the domains that I have worked on. If one cannot traverse across domains for a specific Customer, then we are hardly making use of the richness of the data.

- **Customer** – I was able to understand the different challenges a company will have in dealing with Customer data.
- **Sales** – All the different attributes related to a transaction and why its critical to have them available in the system Near Real time.
- **Preference** – For people who work under marketing, the preference of customer plays a great role and with more laws around them, its important to have them updated and be available to the marketing teams.
- **Loyalty** – The success of this program can only be measured when the company can leverage this data to its benefit.

This is a brain dump of the previous assignment I worked on but it will also serve as a reminder of all the learnings as well as the unknowns. I also plan to write other posts related to some of the topics listed here as I truly believe in “**To teach is to learn twice**“.

The post [Learnings from my last assignment](https://anilkulkarni.com/2019/12/learnings-from-my-last-assignment/) appeared first on [Anil Kulkarni | Blog](https://anilkulkarni.com).