Spark & AI summit and a glimpse of Spark 3.0
Tags: news, apachespark
Published: 2020-06-02 15:35:19

If there is a framework that super excites me, it's Apache Spark.
If there is a conference that excites me, it's the Spark & AI summit.

This year, with the current COVID-19 pandemic, the North America version of the Spark & AI summit is online and [free](https://databricks.com/sparkaisummit). No need to travel, buy an expensive flight ticket, pay for accommodation and conference fees. It's all free and online.

One caveat, it is in pacific timezone(PDT) friendly hours. I kind of wish the organizers would take a more global approach.

Having said that, the agenda and content look promising!


To get ready for the conference and learn about Spark 3.0 
I decided to spin a Spark 3.0 cluster with [Azure Databricks](https://docs.microsoft.com/azure/azure-databricks/quickstart-create-databricks-workspace-portal?WT.mc_id=blog-spark3-adpolak), you can do the same or use the [Databricks Community Edition](https://community.cloud.databricks.com/login.html). 
Please note that with community edition, there are no workers nodes.


## The Workspace:
 - [Azure Databricks](https://docs.microsoft.com/azure/azure-databricks/quickstart-create-databricks-workspace-portal?WT.mc_id=blog-spark3-adpolak)
 - [Databricks Community Edition](https://community.cloud.databricks.com/login.html)

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/8cmg8nwturk4etohfxhv.png)


## MANY Exciting Features, let's briefly look at 2 of them

- **Pandas UDFs and Python Type Hints**
    Probably going to be mostly used by the DataScience and Python developers communities. This feature allows us to create a more readable code and support code static analysis by IDEs such as [PyCharm](https://www.jetbrains.com/help/pycharm/quick-start-guide.html).
    Read about it [here](https://databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html).

- **SQL Join hints**
  Before this change, we had broadcast hash join hints.
  Meaning, if there is a join operation and one of the tables can fit in memory, Spark will broadcast it to execute a faster join. The class in charge of it was named `ResolveBroadcastHints`. It was replaced with `ResolveJoinStrategyHints`.
  To learn more, check out the JIRA ticket :  [SPARK-27225](https://issues.apache.org/jira/browse/SPARK-27225).


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0fs5ol9l9cne6nh5d2o6.png)

List of available hints:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/lnr46j8o3cqi1sv7lg18.png)

To better understand how they work, I recommend checking out the Apache Spark open source code, specificly, this file:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/hints.scala `

If you are interested in learning more about the Spark SQL optimization engine - the Catalyst, I wrote a deep dive on it, [here](https://medium.com/microsoftazure/apache-spark-deep-dive-4e01022afc32).




## Top 4 recommended sessions

#### -1- End-to-End Deep Learning with Horovod on Apache Spark
For the last months, I have been working on various Autonomous Cars scenarios that involve a high load of data. One of the challenges I faced is enabling the DataScience to run Deep Learning at scale. After digging in, I discovered Horovod's framework and the HorovodEstimator. I am excited to attend this session and learn more about it!
Are you curious about it? read about it more [here](https://docs.microsoft.com/en-us/azure/databricks/applications/deep-learning/distributed-training/horovod-estimator?WT.mc_id=blog-spark3-adpolak).

[Session link](https://databricks.com/session_na20/end-to-end-deep-learning-with-horovod-on-apache-spark).



#### -2- Building Reliable ML Pipelines with MLflow
If you follow me for a while now, you know I'm deep into how to build machine learning pipelines at scale. 
Here is a [GitHub repo](https://github.com/adipola/ms-build-e2e-ml-bigdata) describing an End-to-End platform I built for Microsoft Build 2020 session. The platform includes MLFlow, Azure Databricks, Azure Machine Learning, and social media text classification with Scikit learn. The repository include data flow, architecture, tutorials, and code.

[Session link](https://databricks.com/session_na20/building-reliable-ml-pipelines-with-mlflow).

* Please note that this session is long (~1 hour) and is running multiple times during the online conference.


#### -3- An Approach to Data Quality for Netflix Personalization Systems
If you watched my sessions on Big Data and ML, I always mention that:

> **You are only as Good as your Data**

I am referring here to the Machine Learning models of course. We see many biased machine learning models due to unbalanced data and misuse/lack of tools for assessing Data Quality.  Many times during the Data Quality process, we need to filter out the data; this is where having a large set of Data can help. However, it brings challenges, as well. 

This is why I am excited to hear from Netflix how they tackle these challenges.

BTW, if you would like to get familiar with Data Bias challenges, I recommend this [short read](https://www.microsoft.com/en-us/research/blog/what-are-the-biases-in-my-data?WT.mc_id=blog-spark3-adpolak) from Microsoft Research Blog.

[Session link](https://databricks.com/session_na20/an-approach-to-data-quality-for-netflix-personalization-systems).


#### -4- The Apache Spark File Format Ecosystem
The **Veraset** Software developers team is closely involved with open source Spark initiatives such as 
Datasource V2 and the External Shuffle Service, and it's interesting to hear from them how using the right file format can improve performance. As well as permit [Predicate Pushdown](https://medium.com/microsoftazure/data-at-scale-learn-how-predicate-pushdown-will-save-you-money-7063b80878d7).

[Session link](https://databricks.com/session_na20/the-apache-spark-file-format-ecosystem).


## That's it for now !

Thank you for reading so far.
![](https://media.giphy.com/media/26DOMeaD2gdGE44LK/giphy.gif)


These are **my personal opinions** about the summit. 
If you enjoy reading, please follow me here on [dev.to](https://dev.to/adipolak
) , [Twitter](https://twitter.com/AdiPolak) and [LinkedIn](https://www.linkedin.com/in/adi-polak-68548365/). 

Always happy to take your thoughts and opinions.






### 💡 Which session can't you wait to attend? What excites you about Apache Spark?