Getting started with Apache Spark and TensorFlow on Azure Databricks
Tags: tensorflow, azure, apachespark, tutorial
Published: 2019-03-12 12:45:32


---
title: Getting started with Apache Spark and TensorFlow on Azure Databricks
published: true
description: 
tags: Tensor Flow , Azure, Apache Spark, tutorial
---

<h4>TensorFlow is now available on Apache Spark framework, but how do you get started? Read this !</h4><h3>TL;DR</h3><p>Have you ever wanted to write an app that uses AI ? Follow along with me and you'll be there by the end of this short article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*ODRJsCDhVbhIBXrASNWY3w.png" /></figure><p>Big Data is a huge topic that consists of many domains and expertise. All the way from DevOps, Data Engineers to Data Scientist, AI, Machine Learning, algorithm developers and many more. We all struggle with massive amounts of data. When we deal with a massive amount of data, we need the best minds and tools. This is where the <strong><em>magical</em></strong> combination of <strong>Apache Spark</strong> and <strong>Tensor Flow</strong> takes place and we call it <strong>TensorFrame</strong>.</p><p>Apache Spark took over the Big Data world, giving answers and supporting Data Engineers to be a more successful while, Data Scientist had to figure their way around the limitation of the machine learning library that Spark provides, the Spark MLlib.</p><p>But no more, now, there is TensorFlow support for Apache Spark users. These tools combined makes the work of Data Scientists more productive, more accurate and faster. And taking outcome from Research to Develop to Production, faster than ever.</p>

![Alt Text](https://media.giphy.com/media/lRnUWhmllPI9a/giphy.gif)


<p>Before we start, let&#39;s align the terms:</p><ul><li><strong>Tensor Flow</strong> is an open source machine learning framework for high-performance numerical computations created by Google. It comes with strong support for AI: machine learning and deep learning.</li><li><strong>Azure Databricks</strong> is an Apache Spark-based analytics platform optimized for the <strong>Microsoft Azure </strong>cloud services platform. Azure Databricks also acts as Software as a Service( SaaS) / Big Data as a Service (BDaaS).</li><li>TensorFrames is an Apache Spark component that enables us to create our own scalable TensorFlow learning algorithms on Spark Clusters.</li></ul><h3>-1- the workspace:</h3><p>First, we need to create the workspace, we are using Databricks workspace and here is a <a href="https://docs.microsoft.com/azure/azure-databricks/quickstart-create-databricks-workspace-portal?WT.mc_id=devto-blog-adpolak">tutorial</a> for creating it.</p><h3>-2- the cluster:</h3><p>After we have the workspace, we need to create the cluster itself. Let’s create our spark cluster using<a href="https://docs.microsoft.com/azure/azure-databricks/quickstart-create-databricks-workspace-portal?WT.mc_id=devto-blog-adpolak"> this</a> tutorial, make sure you have the next configurations in your cluster:</p><ul><li>A working version of Apache Spark (2.4 or greater)</li><li>Java 8+</li><li>(Optional) python 2.7+/3.6+ if you want to use the python interface.</li><li>(Optional) the python TensorFlow package if you want to use the python interface. See the <a href="https://www.tensorflow.org/install/">official instructions</a> on how to get the latest release of TensorFlow.</li><li>(Optional) pandas &gt;= 0.19.1 if you want to use the python interface</li></ul><p>The configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*wr20JKB-RJpZxT-FzmAlQw.png" /></figure><p>with Databricks runtime versions or above :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*iPalUgtj4vgp3qWm-Sfmlg.png" /></figure><p>Press start to launch the cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/1*-GNxbubA5d-6GR5-BmEK8A.png" /></figure><h3>-3- import the library:</h3><p>Under <strong>Azure Databricks</strong>, go to <strong>Common Tasks</strong> and click <strong>Import Library</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*lS7tnxQN0fAlTJnNhVElgQ.png" /></figure><p>TensorFrame can be found on <a href="https://mvnrepository.com/artifact/databricks/tensorframes/0.2.9-s_2.11">maven repository</a>, so choose the Maven tag. Under Coordinates, insert the library of your choice, for now, it will be:</p><pre>databricks:tensorframes:0.6.0-s_2.11</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*-OT9abkysm_ijRNivrOigA.png" /></figure><p>Click the <strong>Create</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*ACFlxkDte7lo7Cs220RXtA.png" /></figure><p>Click <strong>Install</strong>.</p><p>You will see this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*Z8tbAdxoaLnZ2996CgOtuQ.png" /></figure><p>BOOM. you have TensorFrame on your Databricks Cluster.</p>





![Alt Text](https://media.giphy.com/media/ZtMkorgeyRu5q/giphy.gif)



<h3>-4- the notebook:</h3><p>We use the notebook as our code notebook where we can write the code and run it directly on our Spark Cluster.</p><p>Now that we have a running cluster, let’s run a notebook:</p><p>Click the <strong>New Notebook</strong> and choose the programming language of your choice ( here we chose Scala)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*5LL_FIGaTvGJ1ZKk7_9lXg.png" /></figure><p>This is how it looks like with scala code on the notebook portal with TensorFrames:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ygxKvXVbAwbXA_9yvylgAQ.jpeg" /></figure><p>The code example is from <a href="https://github.com/databricks/tensorframes/blob/master/README.md">Databricks git repository</a>.</p>

<h3>-The End-</h3><p>Now that you have everything up and running, what they will do with tensorflow?</p>



# Please comment !

<p> Comment and let me know your thoughts.  Your comments enable me to improve and get better every day. Thank you :heart: </p>