The way to launch Jupyter Notebook + Apache Spark + InterSystems IRIS
Tags: ai, tutorial, python, api
Published: 2024-01-28 16:02:30

<p>Hi all. Today we are going to install Jupyter Notebook and connect it to Apache Spark and InterSystems IRIS.</p>
<p>Note: I have done the following on Ubuntu 18.04,&nbsp; Python 3.6.5.</p>
<h3><strong>Introduction</strong></h3>
<p>If you are looking for&nbsp;well-known, widely-spread and mainly popular among Python users notebook instead of <strong>Apache Zeppelin</strong>, you should choose <strong>J</strong><strong>upyter</strong><strong> notebook. Jupyter notebook</strong>&nbsp;is a very powerful and&nbsp;great data science tool. it has a big community and a lot of additional software and integrations. Jupyter notebook allows you to create and share documents that contain live code, equations, visualizations&nbsp;and narrative text. Uses include&nbsp;data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more. And&nbsp;most importantly, it is a big community that will help you solve the&nbsp;problems you face.</p>
<h3><strong>Check requirements</strong></h3>
<p>If something doesn't work, look at the "Possible problems and solutions" paragraph in the bottom.</p>
<p>First of all, ensure that you have Java&nbsp;8 (<strong>java -version</strong> returns "1.8.x"). Next, download <a href="https://spark.apache.org/downloads.html">apache spark</a>&nbsp;and unzip it. After, run the following in the terminal:</p>
<blockquote>
<p>pip3 install jupyter</p>
<p>pip3 install toree</p>
<p>jupyter toree install --spark_home=/path_to_spark/spark-2.3.1-bin-hadoop2.7 --interpreters=PySpark --user</p></blockquote>
<p>Now, open the terminal and run <strong>vim ~/.bashrc</strong> .&nbsp;Paste in the bottom the following code (this is environment variables):</p>
<blockquote>
<p>&nbsp;export JAVA_HOME=/usr/lib/jvm/<strong>installed java 8</strong><br>&nbsp;export PATH="$PATH:$JAVA_HOME/bin"<br>&nbsp;export SPARK_HOME=/<strong>path to spark</strong>/spark-2.3.1-bin-hadoop2.7<br>&nbsp;export PATH="$PATH:$SPARK_HOME/bin"<br>&nbsp;export PYSPARK_DRIVER_PYTHON=jupyter<br>&nbsp;export PYSPARK_DRIVER_PYTHON_OPTS="notebook"</p></blockquote>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dyau9pzfngmxr6n6vclf.png)
<p>And run <strong>source ~/.bashrc</strong>.</p>
<h3><strong>Check that it works</strong></h3>
<p>Now, let us launch Jupyter notebook. Run <strong>pyspark</strong>&nbsp;in the terminal.</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/55ebiuq0kyvptkg7fphk.png)
<p>Open in your browser the returned URL.&nbsp; Should be something like the image below:</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p8qwo5pqv937lj00lwbx.png)
<p>Click on&nbsp;<strong>new</strong>,&nbsp;choose <b>Python 3</b>,&nbsp;paste the following code into a paragraph:</p>
<blockquote>
<p>import sys<br>print(sys.version)<br>sc</p></blockquote>
<p>Your output should look like this:</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q0f2c1wllsfj73pnn90d.png)
<p>Stop jupyter using ctrl-c in the terminal.</p>
<p>Note: To add custom jars just move desired jars into $SPARK_HOME/jars.</p>
<p>So, we want to work with intersystems-jdbc and intersystems-spark (we will also need a jpmml&nbsp;library). Let us copy required jars into spark. Run the following in the terminal:</p>
<blockquote>
<p>sudo cp /path to intersystems iris/dev/java/lib/JDK18/intersystems-jdbc-3.0.0.jar /path to spark/spark-2.3.1-bin-hadoop2.7/jars</p>
<p>sudo cp /path to intersystems iris/dev/java/lib/JDK18/intersystems-spark-1.0.0.jar /path to spark/spark-2.3.1-bin-hadoop2.7/jars</p>
<p>sudo cp /path to jpmml/jpmml-sparkml-executable-version.jar /path to spark/spark-2.3.1-bin-hadoop2.7/jars</p></blockquote>
<p>Ensure that is works.&nbsp;Run <strong>pyspark</strong>&nbsp;in the terminal again and run the following code (from the previous <a href="https://community.intersystems.com/post/load-ml-model-intersystems-iris">article</a>):</p>
<blockquote>
<p>from pyspark.ml.linalg import Vectors<br>from pyspark.ml.feature import VectorAssembler<br>from pyspark.ml.clustering import KMeans<br>from pyspark.ml import Pipeline<br>from pyspark.ml.feature import RFormula<br>from pyspark2pmml import PMMLBuilder</p>
<p>dataFrame=spark.read.format("com.intersystems.spark").\<br>option("url", "IRIS://localhost:51773/NAMESPACE").option("user", "dev").\<br>option("password", "123").\<br>option("dbtable", "DataMining.IrisDataset").load() # load iris dataset</p>
<p>(trainingData, testData) = dataFrame.randomSplit([0.7, 0.3]) # split the data into two sets<br>assembler = VectorAssembler(inputCols = ["PetalLength", "PetalWidth", "SepalLength", "SepalWidth"], outputCol="features") # add a new column with features</p>
<p>kmeans = KMeans().setK(3).setSeed(2000) # clustering algorithm that we use</p>
<p>pipeline = Pipeline(stages=[assembler, kmeans]) # First, passed data will run against assembler and after will run against kmeans.<br>modelKMeans = pipeline.fit(trainingData) # pass training data</p>
<p>pmmlBuilder = PMMLBuilder(sc, dataFrame, modelKMeans)<br>pmmlBuilder.buildFile("KMeans.pmml") # create pmml model</p></blockquote>
<p>My output:</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/51mz2jrmvcsxh32r4gcv.png)
<p>The output file is a jpmml kmeans model. Everything works!</p>
<p><strong style="color: inherit; font-family: inherit; font-size: 24px;">Possible problems and solutions</strong></p>
<ul>
<li><strong>command not found: '</strong><strong>jupyter':</strong></li>
</ul>
<ol>
<li>vim ~/bashrc;</li>
<li>add in the bottom&nbsp;<strong>export PATH="$PATH:~/.local/bin"</strong>;</li>
<li>in terminal source ~/.bashrc.</li>
<li>If it doesn't help, reinstall pip3 and jupyter.</li>
</ol>
<ul>
<li><strong>env: '</strong><strong>jupyter</strong><strong>': No such file or directory:</strong></li>
</ul>
<ol>
<li>In ~/.bashrc export PYSPARK_DRIVER_PYTHON=/home/.../.local/bin/jupyter.</li>
</ol>
<ul>
<li><strong>TypeError: 'JavaPackage' object is not callable:</strong></li>
</ul>
<ol>
<li>Check that the required .jar file in /.../spark-2.3.1-bin-hadoop2.7/jars;</li>
<li>Restart notebook.</li>
</ol>
<ul>
<li><strong>Java gateway process exited before sending the driver its port number:</strong></li>
</ul>
<ol>
<li>Your Java version should be 8 (probably works with Java 6/7 too, but I didn't check it);</li>
<li><strong>echo $JAVA_HOME</strong> should return to you Java 8 version. If not, change the path in ~/.bashrc;</li>
<li>Paste&nbsp;<strong>sudo</strong><strong> update-alternatives --config java&nbsp;</strong>in the terminal and choose a proper java version;</li>
<li>Paste&nbsp;<strong>sudo</strong> <strong> update-alternatives --config javac&nbsp;</strong>in the terminal and choose a proper java version.</li>
</ol>
<ul>
<li><strong>PermissionError: [Errno 13] Permission denied: '/usr/local/share/jupyter'</strong></li>
</ul>
<ol>
<li>Add --user at the end of your command in the terminal</li>
</ol>
<ul>
<li><strong>Error executing Jupyter command '</strong><strong>toree</strong><strong>': [Errno 2] No such file or directory</strong></li>
</ul>
<ol>
<li>Run the command without sudo.</li>
</ol>
<ul>
<li>A specific error may appear if you use system variables&nbsp;like&nbsp;<strong>PYSPARK_SUBMIT_ARGS&nbsp;</strong>and other spark/pyspark&nbsp;variables or because of /.../spark-2.3.1-bin-hadoop2.7/conf/spark-env.sh changes.</li>
</ul>
<ol>
<li>Delete these variables and check spark-env.sh.</li>
</ol>
<h3><strong>Links</strong></h3>
<ul>
<li><a href="http://jupyter.org/"><strong>Jupyter</strong></a></li>
<li><strong><a href="https://toree.incubator.apache.org/">Apache Toree</a></strong></li>
<li><a href="https://spark.apache.org/"><strong>Apache Spark</strong></a></li>
<li><a href="https://community.intersystems.com/post/load-ml-model-intersystems-iris"><strong>Load a ML model into InterSystems IRIS</strong></a></li>
<li><a href="https://community.intersystems.com/post/k-means-clustering-iris-dataset"><strong>K-Means clustering of the Iris Dataset</strong></a></li>
<li><a href="https://community.intersystems.com/post/way-launch-apache-spark-apache-zeppelin-intersystems-iris"><strong>The way to launch Apache Spark + Apache Zeppelin + InterSystems IRIS</strong></a></li>
</ul>