Installing and Running Hadoop and Spark on Ubuntu 18
Tags: hadoop, spark, java, scala
Published: 2019-12-13 14:19:35

# Installing and Running Hadoop and Spark on Ubuntu 18

This is a short guide (updated from my previous guides) on how to install Hadoop and Spark on Ubuntu Linux. Roughly this same procedure should work on most Debian-based Linux distros, at least, though I've only tested it on Ubuntu. No prior knowledge of Hadoop, Spark, or Java is assumed.

I'll be setting all of this up on a virtual machine (VM) using [Oracle's VirtualBox](https://www.virtualbox.org/), so I first need to get an [ISO file](https://en.wikipedia.org/wiki/ISO_image) to install the Ubuntu operating system (OS). I'll download [the most recent Long-Term Support (LTS) version of Ubuntu](https://ubuntu.com/download/desktop) from their website (as of this writing, that's 18.04.3). Setting up a virtual machine [is fairly straightforward](https://askubuntu.com/a/153098/360539) and since it's not directly relevant, I won't be replicating those linked instructions here. Instead, let's just start with a clean Ubuntu installation...

## Installing Java

Hadoop requires Java to be installed, and my minimal-installation Ubuntu doesn't have Java by default. You can check this with the command:

```bash
$ java -version

Command 'java' not found, but can be installed with:

sudo apt install default-jre
sudo apt install openjdk-11-jre-headless
sudo apt install openjdk-8-jre-headless
```

> __Note:__ we're going to ignore these suggestions and install Java in a different way.

Hadoop runs smoothly with Java 8, but may encounter bugs with newer versions of Java. So I'd like to install Java 8 specifically. To manage multiple Java versions, I [install SDKMAN!](https://sdkman.io/install) (but first I need to install curl):

```bash
$ sudo apt install curl -y
```

...enter your password, and then install SDKMAN! with

```bash
$ curl -s "https://get.sdkman.io" | bash
```

SDKMAN! is a great piece of software that allows you to install multiple versions of all sorts of different packages, languages, and more. You can see a huge list of available software with:

```bash
$ sdk ls # or sdk list
```

![sshot_04.PNG](https://thepracticaldev.s3.amazonaws.com/i/fd97yo2wqk7d1ryu9ugu.PNG)

To make sure you can use SDKMAN! in every new terminal, run the following command to append a line which sources the SDKMAN! initialisation script whenever a new terminal is opened:

```bash
$ echo "source ~/.sdkman/bin/sdkman-init.sh" >> ~/.bashrc
```

We're only going to use SDKMAN! to install one thing -- Java. You can list all available versions of a particular installation candidate with:

```bash
$ sdk list <software>
```

So in our case, that's

```bash
$ sdk list java
```

...we can see all of the different available Java versions:

![sshot_05.PNG](https://thepracticaldev.s3.amazonaws.com/i/q85ur3ecf09gjk408zve.PNG)

To install a specific version, we use the `Identifier` in the column all the way on the right with:

```bash
$ sdk install <software> <Identifier>
```

I'm going to install AdoptOpenJDK's Java 8.0.232 (HotSpot), so this command, for me, is:

```bash
$ sdk install java 8.0.232.hs-adpt
```

SDKMAN! candidates are installed, by default, at `~/.sdkman/candidates`:

```bash
$ ls ~/.sdkman/candidates/java
8.0.232.hs-adpt  current
```

The `current` symlink always points to whichever Java version SDKMAN! thinks is the version you're currently using, and this is reflected in the `java -version` command. After the last step, this command returns:

```bash
$ java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
```

If you install multiple Java versions, you can easily switch between them with `sdk use`:

```bash
$ sdk install java 13.0.1.hs-adpt

...

$ sdk use java 13.0.1.hs-adpt

Using java version 13.0.1.hs-adpt in this shell.
```

```
$ java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (AdoptOpenJDK)(build 13.0.1+9)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 13.0.1+9, mixed mode, sharing)
```

We also need to explicitly define the `JAVA_HOME` environment variable by adding it to the `~/.bashrc` file:

```bash
$ echo "export JAVA_HOME=\$(readlink -f \$(which java) | sed 's:bin/java::')" >> ~/.bashrc
```

`echo`-ing `JAVA_HOME` should now give us the path to the SDKMAN! directory:

```bash
$ echo $JAVA_HOME
/home/andrew/.sdkman/candidates/java/13.0.1.hs-adpt
```

Make sure you switch back to Java 8 before continuing with this tutorial:

```bash
$ sdk use java 8.0.232.hs-adpt

Using java version 8.0.232.hs-adpt in this shell.
```

## Installing Hadoop

With Java installed, the next step is to install Hadoop. You can get [the most recent version of Hadoop from Apache's website](https://hadoop.apache.org/releases.html). As of this writing, that version is Hadoop 3.2.1 (released 22 Sep 2019). If you click on the link on that webpage, it may redirect you. Click until a `*.tar.gz` file is downloaded. The link I ended up using was

```
http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
```

You can download that in the browser, or by using `wget` in the terminal:

```bash
$ wget http://mirrors.whoishostingthis.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
```

Unpack the archive with `tar`, and redirect the output to the `/opt/` directory:

```bash
$ sudo tar -xvf hadoop-3.2.1.tar.gz -C /opt/
```

Remove the archive file and move to the `/opt/` directory:

```bash
$ rm hadoop-3.2.1.tar.gz && cd /opt
```

Rename the Hadoop directory and change its permissions so that its owned by you (my username is `andrew`) and not `root` or `1001`:

```bash
$ sudo mv hadoop-3.2.1 hadoop && sudo chown andrew:andrew -R hadoop
```

Finally, define the `HADOOP_HOME` environment variable and add the correct Hadoop binaries to your `PATH` by echoing the following lines and concatenating them to your `~/.bashrc` file:

```bash
$ echo "export HADOOP_HOME=/opt/hadoop" >> ~/.bashrc
$ echo "export PATH=\$PATH:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin" >> ~/.bashrc
```

Now, when you `source` your `~/.bashrc` (or open any new shell), you should be able to check that Hadoop has been installed correctly:

```bash
$ hadoop version
Hadoop 3.2.1
Source code repository...
Compiled by ...
...
```

In order for HDFS to run correctly later, we also need to define `JAVA_HOME` in the file `/opt/hadoop/etc/hadoop/hadoop-env.sh`. Find the line in that file which begins with:

```bash
# export JAVA_HOME=
```

and edit it to match the `JAVA_HOME` variable we defined earlier:

```bash
export JAVA_HOME=/home/<username>/.sdkman/candidates/java/8.0.232.hs-adpt
```

Make sure you change the `<username>` above to the appropriate username for your setup. In my case, I replace `<username>` with `andrew`.

## Installing Spark

The last bit of software we want to install is [Apache Spark](https://spark.apache.org/). We'll install this in a similar manner to how we installed Hadoop, above. First, get the most recent `*.tgz` file from [Spark's website](https://spark.apache.org/downloads.html). I downloaded the Spark 3.0.0-preview (6 Nov 2019) pre-built for Apache Hadoop 3.2 and later with the command:

```bash
$ wget http://mirrors.whoishostingthis.com/apache/spark/spark-3.0.0-preview/spark-3.0.0-preview-bin-hadoop3.2.tgz
```

As with Hadoop, unpack the archive with `tar`, and redirect the output to the `/opt/` directory:

```bash
$ sudo tar -xvf spark-3.0.0-preview-bin-hadoop3.2.tgz -C /opt/
```

Remove the archive file and move to the `/opt/` directory:

```bash
$ rm spark-3.0.0-preview-bin-hadoop3.2.tgz && cd /opt
```

Rename the Spark directory and change its permissions so that its owned by you (my username is `andrew`) and not `root` or `1001`:

```bash
$ sudo mv spark-3.0.0-preview-bin-hadoop3.2 spark && sudo chown andrew:andrew -R spark
```

Finally, define the `SPARK_HOME` environment variable and add the correct Spark binaries to your `PATH` by echoing the following lines and concatenating them to your `~/.bashrc` file:

```bash
$ echo "export SPARK_HOME=/opt/spark" >> ~/.bashrc
$ echo "export PATH=\$PATH:\$SPARK_HOME/bin" >> ~/.bashrc
```

Now, when you `source` your `~/.bashrc` (or open any new shell), you should be able to check that Spark has been installed correctly:

```bash
$ spark-shell --version
...
...version 3.0.0-preview
...
```

## Configuring HDFS

At this point, Hadoop and Spark are installed and running correctly, but we haven't yet set up the Hadoop Distributed File System (HDFS). As its name suggests, HDFS is usually _distributed_ across many machines. If you want to build a Hadoop Cluster, I've [previously written instructions for doing that across a small cluster of Raspberry Pis](https://dev.to/awwsmm/building-a-raspberry-pi-hadoop-spark-cluster-8b2). But for simplicity's sake, we'll just set up a standalone, local installation here.

To configure HDFS, we need to edit several files located at `/opt/hadoop/etc/hadoop/`. The first such file is `core-site.xml`. Edit that file so it has the following XML structure:

```xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
```

The second file is `hdfs-site.xml`, which gives the locations of the the namenode and datanode directories. Edit that file so it looks like:

```xml
<configuration>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/datanode</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///opt/hadoop_tmp/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration> 
```

We set `dfs.replication` to 1 because this is a one-machine cluster -- we can't replicate files any more than once here.

> [Read more about data replication in HDFS here.](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html#Data+Replication)

The directories given above (`/opt/hadoop_tmp/hdfs/datanode` and `/opt/hadoop_tmp/hdfs/namenode`) must exist and be read/write-able by the current user. So create them now, and adjust their permissions, with:

```bash
$ sudo mkdir -p /opt/hadoop_tmp/hdfs/datanode
$ sudo mkdir -p /opt/hadoop_tmp/hdfs/namenode
$ sudo chown andrew:andrew -R /opt/hadoop_tmp
```

The next configuration file is `mapred-site.xml`, which you should edit to look like:

```xml
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
```

...and finally `yarn-site.xml`, which you should edit to look like:

```xml
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration> 
```

## Configuring SSH

If you started with a _minimal_ Ubuntu installation like I did, you may need to first set up your `ssh` connection (as HDFS connects to `localhost:9000`). To check if the SSH server is running, enter the command

```bash
$ which sshd
```

If nothing is returned, then the SSH server is not installed (this is the case with the minimal Ubuntu installation). To get this up and running, install `openssh-server`, which will start the SSH service automatically:

```bash
$ sudo apt install openssh-server
```

```bash
$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
  Loaded: loaded ...
  Actve: active...
  ...
```

To check that this worked, try `ssh`-ing into `localhost`:

```bash
$ ssh localhost
...
Are you sure you want to continue connecting (yes/no)? yes
...
Welcome to Ubuntu 18.04.3 LTS...
...
```

> You can `exit` to escape this superfluous self-connection.

Then, create a public-private keypair (if you haven't already):

```bash
$ ssh-keygen
Generating public/private rsa key pair.
...
```

Hit 'enter' / 'return' over and over to create a key in the default location with no passphrase. When you're back to the normal shell prompt, append the public key to your `~/.ssh/authorized_keys` file:

```bash
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
```

You should now be able to boot HDFS. Continue to the next section.

## Formatting and Booting HDFS

At this point, we can format the distributed filesystem. __BE CAREFUL__ and do not run the following command unless you are sure there is no important data currently stored in the HDFS because __IT WILL BE LOST__. But if you're setting up HDFS for the first time on this computer, you've got nothing to worry about:

Format the HDFS with

```bash
$ hdfs namenode -format -force
```

You should get a bunch of output and then a `SHUTDOWN_MSG`:

![sshot_06.PNG](https://thepracticaldev.s3.amazonaws.com/i/40bq7pnshvdnu75zxkgh.PNG)

We can then boot the HDFS with the following two commands:

```bash
$ start-dfs.sh && start-yarn.sh
```

> __Note:__ if you performed a _minimal_ installation, you may need to [install `openssh-server` by following the instructions given here.](https://linuxize.com/post/how-to-enable-ssh-on-ubuntu-18-04/)

You can check that HDFS is running correctly with the command `jps`:

```bash
$ jps
10384 DataNode
11009 NodeManager
4113 ResourceManager
11143 Jps
10218 NameNode
10620 SecondaryNameNode
```

You should see a `NameNode` and a `DataNode`, at minimum, in that list. Check that HDFS is behaving correctly by trying to create a directory, then listing the contents of the HDFS:

```bash
$ hdfs dfs -mkdir /test
$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - andrew supergroup          0 2019-12-13 13:56 /test
```

If you can see your directory, you've correctly configured the HDFS!

## Monitoring

Hadoop and Spark come with built-in web-based monitors that you can access by going to `http://localhost:8088`:

![](https://thepracticaldev.s3.amazonaws.com/i/wn5hk8yh0mzhttcwzarf.PNG)

...and `http://localhost:9870` in your browser:

![](https://thepracticaldev.s3.amazonaws.com/i/wxnisj2jyzj769rz26io.PNG)

## Working with Spark and HDFS

One of the benefits of working with Spark and Hadoop is that they're both Apache products, so they work very nicely with each other. It's easy to read a file from HDFS into Spark to analyse it. To test this, let's copy a small file to HDFS and analyse it with Spark.

Spark comes with some example resource files. With the above configuration, they can be found at `/opt/spark/examples/src/main/resources`. Let's copy the file `users.parquet` to HDFS:

```bash
$ hdfs dfs -put /opt/spark/examples/src/main/resources/users.parquet /users.parquet
```

> _Parquet_ files are [another Apache creation](http://parquet.apache.org/), designed for fast data access and analysis.

Next, open the Spark shell and read in the file with `read.parquet`:

```bash
$ spark-shell
...
Welcome to
... version-3.0.0-preview
...
```

```scala
scala> val df = spark.read.parquet("hdfs://localhost:9000/users.parquet")
df: org.apache.spark.sql.DataFrame = [name: string, favorite_color: string ... 1 more field]

scala> df.collect.foreach(println)
[Alyssa,null,WrappedArray(3,9,15,20)]
[Ben,red,WrappedArray()]
```

This is just a small example, but it shows how Spark and HDFS can work closely together. You can easily read files from HDFS and analyse them with Spark!

__If you want to stop the HDFS__, you can run the commands:

```bash
$ stop-dfs.sh
```

and

```
$ stop-yarn.sh
```

## Conclusion

I hope this guide will be useful for anyone trying to set up a small Hadoop / Spark installation for testing or education. If you're interested in learning more about Hadoop and Spark, please [check out my other articles in this series on Dev](https://dev.to/awwsmm/installing-and-running-hadoop-and-spark-on-windows-33kc)! Thanks for reading!