A Containerized Machine Learning Playground with InterSystems IRIS Community Edition, Spark, and Zeppelin
Tags: beginners, containers, machinelearning, ai
Published: 2023-07-31 18:59:55

<p style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">The last time that I created a playground for experimenting with machine learning using Apache Spark and an InterSystems data platform,  see <a href="https://community.intersystems.com/post/machine-learning-spark-and-cach%C3%A9">Machine Learning with Spark and Caché</a>, I installed and configured everything directly on my laptop: Caché, Python, Apache Spark, Java, some Hadoop libraries, to name a few. It required some effort, but eventually it worked. Paradise. But, I worried. Would I ever be able to reproduce all those steps? Maybe. Would it be possible for a random Windows or Java update to wreck the whole thing in an instant? Almost certainly.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Now, thanks to the increasingly widespread availability of containers and the increasingly usable Docker for Windows, I have my choice of pre-configured machine learning and data science environments . See, for example, <a href="https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html">Jupyter Docker Stacks</a>  and  <a href="https://hub.docker.com/r/apache/zeppelin">Zeppelin on Docker Hub</a>.  With InterSystems making the community edition of the IRIS Data Platform available via container (<a href="https://community.intersystems.com/post/intersystems-iris-community-edition-now-available-docker-store">InterSystems IRIS now Available on the Docker Store</a>), I have easy access to a data platform supporting both machine learning and analytics among a host of other features. By using containers, I do not need to worry about any automatic updates wrecking my playground. If my office floods and my laptop is destroyed, I can easily recreate the playground with a single text file, which I have of course placed in source control ;-)</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">In the following, I will share a Docker compose file that I used to create a container-based machine learning and data science playground. The playground involves two containers: one with a Zeppelin and Spark environment, the other with the InterSystems IRIS Data Platform community edition. Both use images available on Docker hub. I’ll then show how to configure the InterSystems Spark Connector to connect the two. I will end by loading some data into InterSystems IRIS and using Spark to do some data exploration, visualization, and some very basic machine learning . Of course, my example will barely scratch the surface of the capabilities of both Spark and InterSystems IRIS. However, I hope the article will be useful to help others get started doing more complex and useful work.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Note: I created and tested everything that follows on my Windows 10 laptop, using Docker for Windows. </span></span></span><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">For information on configuring Docker for Windows for use with InterSystems IRIS please see the following. The second of the two articles also discusses the basics of using compose files to configure Docker containers.</span></span></span>
</p>

<li style="margin-bottom: 11px;">
  <a href="https://community.intersystems.com/post/using-intersystems-iris-containers-docker-windows">Using InterSystems IRIS Containers with Docker for Windows</a>
</li>
  1. [Docker for Windows and the InterSystems IRIS Data Platform](https://community.intersystems.com/post/docker-windows-and-intersystems-iris-data-platform)

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Compose File for the Two-Container Playground</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Hopefully, the comments in the following compose file do a reasonably adequate job of explaining the environment, but in case they do not, here are the highlights. The compose file defines:</span></span></span>
</p>

  1. <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Two containers: One containing the InterSystems IRIS Community Edition and the other containing both the Zeppelin notebook environment and Apache Spark. Both containers are based on images pulled from the Docker store.</span></span></span>
  2. <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">A network for communication between the two containers. With this technique, we can use the container names as host names when setting up communication between the containers.</span></span></span>
  3. <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Local directories mounted in each container. We can use these directories to make jar files available to the Spark environment and some data files available to the IRIS environment.</span></span></span>
  4. <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">A named volume for the durable %SYS feature needed by InterSystems IRIS. Named volumes are necessary for InterSystems IRIS when running in containers on Docker for Windows. For more about this see below for links to other community articles.</span></span></span>
<li style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Map some networking ports inside the containers to ports available outside the containers to provide easy access.</span></span></span>
</li>

<pre style="margin-bottom: 11px;">version: '3.2'
services:
  #container 1 with InterSystems IRIS
  iris:
    # iris community edition image to pull from docker store.
    image: store/intersystems/iris:2019.1.0.510.0-community
    
    container_name: iris-community
    
    ports:
    # 51773 is the superserver default port 
    - "51773:51773"
    # 52773 is the webserver/management portal default port
    - "52773:52773"
    volumes:
    # Sets up a named volume durable_data that will keep the durable %SYS data
    - durable:/durable
    # Maps a /local directory into the container to allow for easily passing files and test scripts
    - ./local/samples:/samples/local
    environment:
    # Set the variable ISC_DATA_DIRECTORY to the durable_data volume that we defined above to use durable %SYS
    - ISC_DATA_DIRECTORY=/durable/irissys   
    
    # Adds the IRIS container to the network defined below.
    networks:
      - mynet
  
  #container 2 with Zeppelin and Spark
  zeppelin:
    # zeppelin notebook with spark image to pull from docker store.
    image: apache/zeppelin:0.8.1
    
    container_name: spark-zeppelin
    
    #Ports for accessing Zeppelin environment
    ports: 
    #Port for Zeppelin notebook
    - "8080:8080"
    #Port for Spark jobs page
    - "4040:4040"
    
    #Maps /local directories for saving notebooks and accessing jar files.
    volumes:
    - ./local/notebooks:/zeppelin/notebook
    - ./local/jars:/home/zeppelin/jars
    
    #Adds the Spark and Zeppelin container to the network defined below.
    networks:
    - mynet
    
   
   #Declares the named volume for the IRIS durable %SYS
volumes:
  durable:
  
  # Defines a network for communication between the two containers.
networks:
  mynet:
    ipam:
      config:
        - subnet: 172.179.0.0/16</pre>

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Launching the Containers</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Place the compose file in a directory on your system. Note that the directory name becomes the Docker project name. You will need to create sub-directories matching those mentioned in the compose file. So, my directory structure looks like this</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">iris_spark_zeppelin </span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">              local</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">                    jars</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">                    notebooks</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">                    samples</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">              docker-compose.yml</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">To launch the containers, execute the following Docker command from inside your project directory:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>C:\iris_spark_zeppelin>docker-compose up –d</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Note that the –d flag causes the containers in detached mode. You will not see them logging any information to the command line.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">You can inspect the log files for the containers using the docker logs command. For example, to see the log file for the iris-community container, execute the following:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>C:\>docker logs iris-community</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">To inspect the status of the containers, execute the following command:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>C:\>docker container ls</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">When the iris-community container is ready, you can access the IRIS Management Portal with this url:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <a href="http://localhost:52773/csp/sys/UtilHome.csp"><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">http://localhost:52773/csp/sys/UtilHome.csp </span></span></span></a>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Note: The first time you login to IRIS use the username/password: SuperUser/SYS. You will be re-directed to a password change page.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">You can access the Zeppelin notebook with this url:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <a href="http://localhost:8080"><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">http://localhost:8080</span></span></span></a>
</p>

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Copying Some Jar Files</u><br />In order to use the InterSystems Spark Connector, the Spark environment needs access to two jar files:</span></span></span>
</h2>

<li style="list-style-type: none;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">1. intersystems-jdbc-3.0.0.jar</span></span></span>
</li>
<li style="list-style-type: none;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">2. intersystems-spark-1.0.0.jar</span></span></span>
</li>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Currently, these jar files are with IRIS in the iris-community container. We need to copy them out into the locally mapped directory so that the spark-zeppelin container can access them.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">To do this, we can use the Docker cp command to copy all the JDK 1.8 version files from inside the iris-community container into one of the local directories visible to the spark-zeppelin container. Open a CMD prompt in the project directory and execute the following command:</span></span></span><br /><br /><kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>C:\iris_spark_zeppelin>docker cp iris-community:/usr/irissys/dev/java/lib/JDK18 local/jars</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11.0pt"><span style="line-height:107%"><span style='font-family:"Calibri",sans-serif'>This will add a JDK18 directory containing the above jar files along with a few others to &lt;project-directory>/local/jars.</span></span></span>
</p>

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Adding Some Data</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">No data, no machine learning. We can use the local directories mounted by the iris-community container to add some data to the data platform. I used the Iris data set (no relation to InterSystems IRIS Data Platform). The Iris data set contains data about flowers. It has long served as the “hello world” example for machine learning (<a href="https://en.wikipedia.org/wiki/Iris_flower_data_set">Iris flower data set</a>). You can download or pull an InterSystems class definition for generating the data, along with code for several related examples, from GitHub (<a href="https://github.com/intersystems/Samples-Data-Mining">Samples-Data-Mining</a>). We are interested in only one file from this set: DataMining.IrisDataset.cls.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Copy DataMining.IrisDataset.cls into your &lt;project-directory>/local/samples directory. Next, open a bash shell inside the iris-community container by executing the following from a command prompt on your local system:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>C:\>docker exec –it iris-community bash</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">From the bash shell, launch an IRIS terminal session:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>/# iris session iris</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">IRIS asks for a username/password.  If this is the first time that you are logging into IRIS in this container, use SuperUser/SYS. You will then be asked to change the password. If you have logged in before, for example through the Management Portal, then you changed the password already. Use your updated password now.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Execute the following command to load the file into IRIS:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>USER>Do $System.OBJ.Load(“/samples/local/IrisDataset.cls”,”ck”)</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">You should see output about the above class file compiling and loading successfully. Once this code is loaded, execute the following commands to generate the data for the Iris dataset</span></span></span>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>USER>Set status = ##class(DataMining.IrisDataset).load()</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <kbd><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>USER>Write status</span></span></span></span></kbd>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">The output from the second command should be 1. The database now contains data for 150 examples of Iris flowers. </span></span></span>
</p>

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Launching Zeppelin and Configuring Our Notebook</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">First, download the Zeppelin notebook note available here: <a href="https://github.com/denelson/DevCommunity">https://github.com/denelson/DevCommunity</a>. The name of the note is “Machine Learning Hello World”.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">You can open the Zeppelin notebook in your web browser using the following url:</span></span></span>
</p>

<p style="margin-bottom:11px">
  <a href="http://localhost:8080"><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">http://localhost:8080</span></span></span></a>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">It looks something like this. </span></span></span>
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5mnbisq7kum0vsm9w2nr.png)

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Click the “Import note” link and import “Machine Learning Hello World.json”. </span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">The first code paragraph contains code that will load the InterSystems JDBC driver and Spark Connector. By default, Zeppelin notebooks provide the z variable for accessing Zeppelin context.  See <a href="https://zeppelin.apache.org/docs/0.8.0/usage/other_features/zeppelin_context.html">Zeppelin Context </a>in the Zeppelin documentation.</span></span></span>
</p>

<pre style="margin-bottom: 11px;">%spark.dep
//z supplies Zeppelin context
z.reset() 
z.load("/home/zeppelin/jars/JDK18/intersystems-jdbc-3.0.0.jar")
z.load("/home/zeppelin/jars/JDK18/intersystems-spark-1.0.0.jar")</pre>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Before running the paragraph, click the down arrow next to the word “anonymous” and then select “Interpreter”.</span></span></span>
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j086wmx6fb2o0f8wl9w1.png)

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">On the Interpreters page, search for spark, then click the restart button on the right-hand-side and then ok on the ensuing pop-up.</span></span></span>
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7pi4bc5rlzkc3r7yc5z3.png)

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Now return to the Machine Learning Hello World notebook and run the paragraph by clicking the little arrow all the way at the right. You should see output similar to that in the following screen capture:</span></span></span>
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2audjftb29jtjpjot0ed.png)

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>Connecting to IRIS and Exploring the Data</u></span></span></span>
</h2>

<p style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Everything is all configured. Now we can connect code running in the spark-zeppelin container to InterSystems IRIS, running in our iris-community container, and begin exploring the data we added earlier. The following Python code connects to InterSystems IRIS and reads the table of data that we loaded in an earlier step (DataMining.IrisDataset) and then displays the first ten rows. </span></span></span>
</p>

<p style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Here are a couple of notes about the following code:</span></span></span>
</p>

  * <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">We need to supply a username and password to IRIS. Use the password that you provided in an earlier step when you logged into IRIS and were forced to change your password. I used SuperUser/SYS1</span></span></span>
  * <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"> “iris” in the spark.read.format(“iris”) snippet is an alias for the com.intersystems.spark class, the spark connector.</span></span></span>
  * <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"> The connection url, including “IRIS” at the start, specifies the location of the InterSystems IRIS default Spark master server</span></span></span>
<li style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">The spark variable points to the Spark session supplied by the Zeppelin Spark interpreter.</span></span></span>
</li>

<pre style="margin-bottom: 11px;">%pyspark
uname = "SuperUser"
pwd = "SYS1"
#spark session available by default through spark variable.
#URL uses the name of the container, iris-community, as the host name.
iris = spark.read.format("iris").option("url","IRIS://iris-community:51773/USER").option("dbtable","DataMining.IrisDataset").option("user",uname).option("password",pwd).load()
iris.show(10)</pre>

<p style="margin-bottom:11px">
  <span style="font-size:11.0pt"><span style="line-height:107%"><span style='font-family:"Calibri",sans-serif'>Note: For more information on configuring the Spark connection to InterSystems IRIS, see <a href="https://irisdocs.intersystems.com/iris20191/csp/docbook/DocBook.UI.Page.cls?KEY=BSPK_intro">Using the InterSystems Spark Connector</a> in the InterSystems IRIS documentation. For more information on the spark session and other context variables provided by Zeppelin, see </span></span></span><a href="http://SparkContext,%20SQLContext,%20SparkSession,%20ZeppelinContext">SparkContext, SQLContext, SparkSession, ZeppelinContext</a> in the Zeppelin documentation.
</p>

<p style="margin-bottom:11px">
  Running the above paragraph results in the following output:
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m9bgwq15w4dumq9lhvj4.png)

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Each row represents an individual flower and records its petal length and width, its sepal length and width, and the Iris species it belongs to.</span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Here is some SQL-esque code for further exploration:</span></span></span>
</p>

<pre style="margin-bottom: 11px;">%pyspark
iris.groupBy("Species").count().show()</pre>

<span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><span style='font-family:"Courier New"'>Running the paragraph produces the following output:</span></span></span></span>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a0ziqc7cqsqssvug7xgy.png) 

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">So there are three different Iris species represented in the data. The data represents each species equally. </span></span></span>
</p>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Using Python’s matplotlib library, we can even draw some graphs.  Here is code to plot Petal Length vs. Petal Width:</span></span></span>
</p>

<pre><span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">%pyspark
%matplotlib inline
import matplotlib.pyplot as plt
#Retrieve an array of row objects from the DataFrame
items = iris.collect()
petal_length = []
petal_width = []
for item in items:
    petal_length.append(item['PetalLength'])
    petal_width.append(item['PetalWidth'])</span></span></span>
<span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">plt.scatter(petal_width,petal_length)
plt.xlabel("Petal Width")
plt.ylabel("Petal Length")
plt.show()</span></span></span></pre>

<span style="font-size:11.0pt"><span style="line-height:107%"><span style='font-family:"Courier New"'>Running the paragraph creates the following scatter plot:</span></span></span>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zoygw4ll4uzvauv3r6dh.png)

<span style="font-size:11.0pt"><span style="line-height:107%"><span style='font-family:"Calibri",sans-serif'>Even to the untrained eye, it looks like there is a pretty strong correlation between Petal Width and Petal Length. We should be able to reliably predict petal length based on petal width.</span></span></span>

<u>A Little Machine Learning</u>

Note: I copied the following code from my earlier playground article, cited above.

In order to predict petal length based on petal width, we need a model of the relationship between the two. We can create such a model very easily using Spark. Here is some code that uses Spark's linear regression API to train a regression model. The code does the following:

  1. Creates a new Spark DataFrame containing the petal length and petal width columns. The petal width column represents the "features" and the petal length column represents the "labels". We use the features to predict the labels.
  2. Randomly divides the data into training (70%) and test (30%) sets.
  3. Uses the training dat to fit the linear regression model.
  4. Runs the test data through the model and then displays the petal length, petal width, features, and predictions.

<pre>%pyspark
from pyspark.ml.regression import LinearRegression
from pyspark.ml.feature import VectorAssembler
# Transform the "Features" column(s) into the correct vector format
df = iris.select('PetalLength','PetalWidth')
vectorAssembler = VectorAssembler(inputCols=["PetalWidth"],
                                  outputCol="features")
data=vectorAssembler.transform(df)
# Split the data into training and test sets.
trainingData,testData = data.randomSplit([0.7, 0.3], 0.0)
# Configure the model.
lr = LinearRegression().setFeaturesCol("features").setLabelCol("PetalLength").setMaxIter(10)
# Train the model using the training data.
lrm = lr.fit(trainingData)
# Run the test data through the model and display its predictions for PetalLength.
predictions = lrm.transform(testData)
predictions.show(10)</pre>

<span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Running the paragraph results in the following output:</span></span></span>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ynhko0yiszbfzq7nynzu.png)

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>The Regression Line</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">The “model” is really just a regression line through the data. It would be nice to have the slope and y-intercept of that line. It would also be nice to be able to visualize that line superimposed on our scatter plot. The following code retrieves the slope and y-intercept from the trained model and then uses them to add a regression line to the scatter plot of the petal length and width data.</span></span></span>
</p>

<pre>%pyspark
%matplotlib inline
import matplotlib.pyplot as plt
# retrieve the slope and y-intercepts of the regression line from the model.
slope = lrm.coefficients[0]
intercept = lrm.intercept
print("slope of regression line: %s" % str(slope))
print("y-intercept of regression line: %s" % str(intercept))
items = iris.collect()
petal_length = []
petal_width = []
petal_features = []
for item in items:
    petal_length.append(item['PetalLength'])
    petal_width.append(item['PetalWidth'])
fig, ax = plt.subplots()
ax.scatter(petal_width,petal_length)
plt.xlabel("Petal Width")
plt.ylabel("Petal Length")
y = [slope*x+intercept for x in petal_width]
ax.plot(petal_width, y, color='red')
plt.show()</pre>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">Running the paragraph results in the following output:</span></span></span>
</p>


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c6lq14t78yeljlmr9png.png) 

<h2 style="margin-bottom: 11px;">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif"><u>What’s Next?</u></span></span></span>
</h2>

<p style="margin-bottom:11px">
  <span style="font-size:11pt"><span style="line-height:107%"><span style="font-family:Calibri,sans-serif">There is much, much more we can do. Obviously, we can load much larger and much more interesting datasets into IRIS. See, for example the Kaggle datasets (<a href="https://www.kaggle.com/datasets">https://www.kaggle.com/datasets</a>) With a fully licensed IRIS we could configure sharding and see how Spark running through the InterSystems Spark Connector takes advantage of the parallelism sharding offers. Spark, of course, provides many more machine learning and data analysis algorithms. It supports several different languages, including Scala and R. </span></span></span>
</p>