Creating and running Spark Jobs in Scala on Cloud Dataproc !!!
Tags: scala, googlecloud, spark, bigdata
Published: 2021-12-22 23:20:12

Deploy and Run Jobs Spark on Scala in GCP is easy!!

This is a simple tutorial with examples of using Google Cloud to run Spark jobs done in Scala easily! :)

Install the Java 8 JDK or Java 11 JDK

To check if Java is installed on your operating system, use the command below:


`java -version` 

Depending on the version of Java, this command can change … :)

If Java has not been installed yet, install from these links: Oracle Java 8, Oracle Java 11, or AdoptOpenJDK 8/11; always checking the compatibility between the versions of JDK and Scala following the guidelines of this link: JDK Compatibility.

Install Scala

* Ubuntu:

`sudo apt install scala`

* macOS:


```Scala
brew update
brew install scala
``` 

Set the SCALA_HOME environment variable and add it to the path, as shown in the Scala installation instructions. Example:

`export SCALA_HOME=/usr/local/share/scala export PATH=$PATH:$SCALA_HOME/`

if you use Windows:

```
%SCALA_HOME%c:\Progra~1\Scala 
%PATH%%PATH%;%SCALA_HOME%\bin
```

Start Scala.

```
$ scala 
Welcome to Scala 2.11.12 (OpenJDK 64-Bit Server VM, Java 11.0.9.1).
Type in expressions for evaluation. Or try :help.
```

Depending on your installation, the version of Scala and Java may change …

Copy and paste the HelloWorld code into Scala REPL (interactive mode via terminal)

```
object HelloWorld {
 def main(args: Array[String]): Unit = {
 println(“Hello, world!”)
 }
```

Save HelloWorld.scala and exit REPL.

```
scala> :save HelloWorld.scala 
scala> :q
```

Compile with: scalac.

`$ scalac HelloWorld.scala`

List the compiled .class files.

```
$ ls -al 
HelloWorld*.class 
HelloWorld$.class 
HelloWorld.class
```

Copying the Scala code

Let’s put an order in the house!

```
mkdir hello
$ cd hello
$ echo \
'object HelloWorld {def main(args: Array[String]) = println("Hello, world!")}' > \
HelloWorld.scala
```

Create a sbt.build configuration file to define artifactName (the name of the jar file you will generate below) as “HelloWorld.jar”

```
echo \
'artifactName := { (sv: ScalaVersion, module: ModuleID, artifact: Artifact) =>
"HelloWorld.jar" }' > \
build.sbt
```

Start SBT (Scala Package and Dependency Manager) and run the code:

`$ sbt`

If SBT has not been installed yet, it will be downloaded and installed; it may take some time….

```
sbt:hello> run
[info] running HelloWorld 
Hello, world!
[success] Total time: 0 s, completed 4 de fev de 2021 00:20:26
```

Package your project to create a .jar file

```
sbt:hello> package
[success] Total time: 0 s, completed 4 de fev de 2021 00:20:35
sbt:hello> exit
[info] shutting down sbt server
```
The compiled file is at “./target/scala-version_xpto/HelloWorld.jar”

Google cloud platform (GCP)

To use the Google cloud service, it is necessary to enable its billing; in this example, we will use cloud storage to store the jar file with our Scala code and Cloud Dataproc to execute the used Spark file.

Both services have a free usage fee, and once that quota ends, you will be charged to the credit card linked to the account!

Copy the Jar file to a bucket on cloud storage:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1639920619534/9c2Hckabh.png)

With the link to the bucket where the jar file is, submit a new job in the Cloud Dataproc:


![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1639920648605/9FRvzaJPT.png)

Field references:

* Cluster: Select the specific cluster from the list.

* Job type: select “Spark”

* Main class or jar: paste the link to the bucket where the jar file is located.

Before submitting, check if the Spark cluster has been created :)

After filling out the form, we have to select “submit,” the job will be executed and will appear in the list of jobs:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1639920705894/yUn496Jje.png)

By selecting the Job ID, it will be possible to see its output:

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1639920729112/a3wh2-BONN.png)

This was a brief example of deploying a Spark routine done in Scala in the Google environment, there is the possibility to interact with the Spark cluster via spark-shell accessing via ssh, and instead of doing the submit via form, we could use the Google CLI CLoud.

**References:**

https://cloud.google.com/dataproc/docs/tutorials/spark-scala

<a href="https://www.buymeacoffee.com/jlgjosue" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>

Follow me on [Medium](https://jlgjosue.medium.com/)  :)