The way to launch Apache Spark + Apache Zeppelin + InterSystems IRIS
Tags: beginners, practice, apache, machinelearning
Published: 2021-05-28 18:15:05

Hi all. Yesterday I tried to connect Apache Spark, Apache Zeppelin, and InterSystems IRIS. During the process, I experienced troubles connecting it all together and I did not find a useful guide. So, I decided to write my own.

### **Introduction**

What is Apache Spark and Apache Zeppelin and find out how it works together. Apache Spark is an open-source cluster-computing framework. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. So, it is very useful when you need to work with Big Data. And Apache Zeppelin is a notebook, that provides cool UI to work with analytics and machine learning. Together, it works like this: IRIS provides data, Spark reads provided data, and in a notebook we work with the data.

Note: I have done the following on Windows 10.

### **Apache Zeppelin**

Now, we will install all the necessary programs. First of all, download apache zeppelin from [the official site of apache zeppelin](https://zeppelin.apache.org/download.html). I have used **zeppelin-0.8.0-bin-all.tgz**. It includes **Apache**** Spark**, **Scala,** and **Python. **Unzip it to any folder. After that you can launch zeppelin by calling \bin\zeppelin.cmd from the root of your Zeppelin folder. Wait until the **Done, zeppelin server started** string appears and open http://localhost:8080 in your browser. If everything is okay, you will see **Welcome to Zeppelin! **message.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e9k6q1loe9ov56bs5srg.png)

Note: I assume, that InterSystems IRIS already installed. If not, download and install it before the next step.

### **Apache Spark**

So, we have the browser's open window with Zeppelin notebook. In the upper-right corner click on **anonymous** and after, click on **Interpreter**. Scroll down and find **spark**.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5irxqiciire3bvyw1ftm.jpg)

Next to the spark find **edit **button and click on it. Scroll down and add dependencies to **intersystems-spark-1.0.0.jar** and to  **intersystems-jdbc-3.0.0.jar**. I installed InterSystems IRIS to the C:\InterSystems\IRIS\ directory, so artifacts I need to add are at:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbhtthm01kee6yo51wyb.jpg)

My files are here:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1vrayu5hpc98pybnedkq.jpg)

And save it.

### **Check that it works**

Let us check it. Create a new note, and in a paragraph paste the following code:

> <p style="margin:0cm 0cm 0.0001pt">
>   <span style="font-size:12pt"><span style="font-family:Cambria,serif">var dataFrame=spark.read.format("com.intersystems.spark").option("url", "IRIS://localhost:51773/NAMESPACE").option("user", "UserLogin").option("password", "UserPassword").option("dbtable", "Sample.Person").load()</span></span>
> </p>
> 
> <p style="margin:0cm 0cm 0.0001pt">
>   <span style="font-size:12pt"><span style="font-family:Cambria,serif">// dbtable - name of your table</span></span>
> </p>

<p style="margin: 0cm 0cm 0.0001pt;">
  URL - IRIS address. It is formed as follows IRIS://ipAddress:superserverPort/namespace:
</p>

<li style="margin: 0cm 0cm 0.0001pt;">
  protocol IRIS is a JDBC connection over TCP/IP that offers Java shared memory connection;
</li>
<li style="margin: 0cm 0cm 0.0001pt;">
  ipAddress — The IP address of the InterSystems IRIS instance. If you are connecting locally, use 127.0.0.1 instead of localhost;
</li>
<li style="margin: 0cm 0cm 0.0001pt;">
  superserverPort — The superserver port number for the IRIS instance, which is not the same as the webserver port number. To find the superserver port number, in the Management Portal, go to System Administration > Configuration > System Configuration > Memory and Startup; namespace — An existing namespace in the InterSystems IRIS instance. In this demo, we connect to the USER namespace.
</li>

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/40oix996bw7unusafse3.png)

<p style="margin:0cm 0cm 0.0001pt">
   
</p>

<p style="margin:0cm 0cm 0.0001pt">
  Run the paragraph. If everything is okay, you will see <strong>FINISHED</strong>.
</p>

<p style="margin:0cm 0cm 0.0001pt">
  My notebook:
</p>

  ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7jg2vhr00vcuy0sdhg7s.png)

<p style="margin:0cm 0cm 0.0001pt">
   
</p>

<h3 style="margin: 0cm 0cm 0.0001pt;">
  <strong>Conclusion</strong>
</h3>

<p style="margin:0cm 0cm 0.0001pt">
   
</p>

<p style="margin:0cm 0cm 0.0001pt">
  In conclusion, we found out how Apache Spark, Apache Zeppelin, and InterSystems IRIS can work together. In my next articles, I will write about data analysis.
</p>

<p style="margin:0cm 0cm 0.0001pt">
   
</p>

<h3 style="margin: 0cm 0cm 0.0001pt;">
  <strong>Links</strong>
</h3>

  * [The official site of Apache Spark](https://spark.apache.org)
  * [Apache Spark documentation](https://spark.apache.org/documentation.html)
  * [IRIS Protocol](https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=AFL_jdbc)
  * [Using the InterSystems Spark Connector](https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=BSPK)