Spark Journey begins...
Tags: spark, python, bigdata
Published: 2020-09-28 10:14:44

As an engineer with several years of experience in Backend and Frontend projects it feels like the next natural step is big data challenges. 
In the big data world I expect to find computing, IO and scaling challenges not usually found in ordinary/plain/textbook architectures.

I decided that Spark is the best way to get started. Specifically - the Databricks certification, which is focused on Spark programming and architecture. 

My game plan to pass the [Databricks spark certification](https://academy.databricks.com/exam/databricks-certified-associate-developer) is to: 

 1. Read "Learning Spark Lightning fast big data analysis" book and work through all the examples + summarising important insights and lessons so I can repeat those later.
 2. Go over the skeletons of Databricks Developer course that I found on GitHub from 15 months ago. Should be pretty updated - https://github.com/vivek-bombatkar/spark-training + https://github.com/vivek-bombatkar/Spark-with-Python---My-learning-notes-
 3. Going through example questions.

Please, If you can advice on any source of preparation - write in the comments it will help me.

I will update as I go for others (and myself).

##Learning Schedule

### Theory

Reading throughly the book ["Learning Spark Lightning-fast..."](https://laptrinhx.com/learning-spark-lightning-fast-data-analytics-2nd-edition-436517903/)
I think it's reasonable to go through 2 chapters per week.
this means: reading, summarizing and running important code snippets on my own.

*Week 1*
Chapter 3 
Chapter 4 

*Week 2*
Chapter 5 
Chapter 6 

*Week 3*
Chapter 7 
Chapter 8 

*Week 4*
Chapter 9 
Chapter 10 

*Week 5*
Chapter 11 - Quick read it's not that important

### Hands on coding 

Basics (4 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-pyspark

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-from-pandas-to-spark

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-weather-df

https://github.com/vivek-bombatkar/spark-training/blob/master/spark-python/jupyter-weather-df/Weather%20Analysis%20Exercise.ipynb

Advanced topics (10 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced

Windows (4 notebook)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-windows

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-windows

UDF (3 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-udf

Spark execution(1 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-execution

Caching (3 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-caching

Pivoting (1 notebook)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-pivoting

total 26 notebooks
I hope to do 3-4 notebooks per week (some will be easy some harder, so taking the average). This will result in 8 weeks of going through the notebooks. Learning what I'm missing etc. 

Everything should take 3 months until I'm ready for the exam. 


### Books PDFs 

Learning Spark: Lightning-Fast Big Data Analysis
First Edition
https://b-ok.asia/book/2493162/9b8d4f?dsource=recommend
Second Edition
https://laptrinhx.com/learning-spark-lightning-fast-data-analytics-2nd-edition-436517903/

Spark: The Definitive Guide: Big Data Processing Made Simple
https://b-ok.asia/book/3505368/f04c83?regionChanged

Spark in Action
https://b-ok.asia/book/3502170/d3383b