Installing and Running Hadoop and Spark on Windows
Tags: hadoop, spark, windows, tutorial
Published: 2018-11-05 15:36:43

# Installing and Running Hadoop and Spark on Windows

We recently got a **big** new server at work to run Hadoop and Spark (H/S) on for a proof-of-concept test of some software we're writing for the biopharmaceutical industry and I hit a few snags while trying to get H/S up and running on Windows Server 2016 / Windows 10. I've documented here, step-by-step, how I managed to install and run this pair of Apache products directly in the Windows `cmd` prompt, without any need for Linux emulation.

> _**Update 16 Dec 2019:** Software version numbers have been updated and the text has been clarified._

# Get the Software

The first step is to download Java, Hadoop, and Spark. Spark seems to have trouble working with newer versions of Java, so I'm sticking with Java 8 for now:

- [Java (using version: 8u230+)](https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
- [Hadoop (using version: 3.1.3)](https://hadoop.apache.org/releases.html)
- [Spark (using version: 3.0.0 preview)](https://spark.apache.org/downloads.html)

> I can't guarantee that this guide works with newer versions of Java. Please try with Java 8 if you're having issues. Also, with the new Oracle licensing structure (2019+), you may need to create an Oracle account to download Java 8. To avoid this, simply [download from AdoptOpenJDK instead](https://adoptopenjdk.net/).

For Java, I download the "Windows x64" version of the AdoptOpenJDK HotSpot JVM (`jdk8u232-b09`); for Hadoop, the binary of v3.1.3 (`hadoop-3.1.3.tar.gz`); for Spark, v3.0.0 "Pre-built for Apache Hadoop 2.7 and later" (`spark-3.0.0-preview-bin-hadoop2.7.tgz`). From this point on, I'll refer generally to these versions as `hadoop-<version>` and `spark-<version>`; please replace these with your version number throughout the rest of this tutorial.

> Even though newer versions of Hadoop and Spark are currently available, [there is a bug with Hadoop 3.2.1 on Windows](https://issues.apache.org/jira/browse/HDFS-14890) that causes installation to fail. Until that patched version is available (3.3.0 or 3.1.4 or 3.2.2), you must use an earlier version of Hadoop on Windows.

Next, [download 7-Zip](https://www.7-zip.org/) to extract the `*gz` archives. Note that you may need to extract twice (once to move from `*gz` to `*.tar` files, then a second time to "untar"). Once they're extracted (Hadoop takes a while), you can delete all of the `*.tar` and `*gz` files. You should now have two directories and the JDK installer in your Downloads directory:

![](https://thepracticaldev.s3.amazonaws.com/i/sn274lipt5r1dq60q4y2.PNG)

Note that -- as shown above -- the "Hadoop" directory and "Spark" directory each contain a `LICENSE`, `NOTICE`, and `README` file. With particular versions of Hadoop, you may extract and get a directory structure like

```dos
C:\Users\<username>\Downloads\hadoop-<version>\hadoop-<version>\...
```

...if this is the case, move the contents of the _inner_ `hadoop-<version>` directory to the outer `hadoop-<version>` directory by copying-and-pasting, then delete the _inner_ `hadoop-<version>` directory. The path to the `LICENSE` file, for example, should then be:

```dos
C:\Users\<username>\Downloads\hadoop-<version>\LICENSE
```

...and similar for the "Spark" directory.

> _**WARNING**: If you see a message like "Can not create symbolic link : A required privilege is not held by the client" in 7-Zip, [you **MUST** run 7-Zip in Administrator Mode](https://superuser.com/a/1076531/728488), **then** unzip the directories. If you skip these files, you may end up with a broken Hadoop installation._

Move the Spark and Hadoop directories into the `C:\` directory (you may need administrator privileges on your machine to do this). Then, run the Java installer but change the destination folder from the default `C:\Program Files\AdoptOpenJDK\jdk-<version>\` to just `C:\Java`. (H/S can have trouble with directories with spaces in their names.)

![](https://thepracticaldev.s3.amazonaws.com/i/9gcxr91eej5cuc5kay8f.PNG)

Once the installation is finished, you can delete the Java `*.msi` installer. Make two new directories called `C:\Hadoop` and `C:\Spark` and copy the `hadoop-<version>` and `spark-<version>` directories into those directories, respectively:

![](https://thepracticaldev.s3.amazonaws.com/i/3jarnzcyy5nczmfzwx7o.PNG)

If you get "name too long"-type warnings, skip those files. These are only `*.html` files and aren't critical to running H/S.

# Set Up Your Environment Variables

Next, we need to set some environment variables. Go to `Control Panel > System and Security > System > Advanced System Settings > Environment Variables...`:

![](https://thepracticaldev.s3.amazonaws.com/i/gydxky0nm5t3pyyn6ls6.PNG)

...and add new System variables (bottom box) called:

- `JAVA_HOME` --> `C:\Java`
- `HADOOP_HOME` --> `C:\Hadoop\hadoop-<version>`
- `SPARK_HOME` --> `C:\Spark\spark-<version>`

(Adjust according to the versions of Hadoop and Spark that you've downloaded.)

Then, edit the `Path` (again, in the System variables box at the bottom) and add those variables with `\bin` appended (also `\sbin` for Hadoop):

![](https://thepracticaldev.s3.amazonaws.com/i/p5gj8qs2sf8qq99beq57.PNG)

If you `echo %PATH%` in `cmd` you should now see these three directories somewhere in the middle of the path, because the User Path is appended to the System Path for the `%PATH` variable. You should check now that `java -version`, `hdfs -version`, and `spark-shell --version` return version numbers, as shown below. This means that they were correctly installed and added to your `%PATH%`:

![](https://thepracticaldev.s3.amazonaws.com/i/ohe2pkcd4o85odi4jk0l.PNG)

__Please note__ that if you try to run the above commands from a location _with any spaces in the path_, the commands may fail. For example, if your username is "Firstname Lastname" and you try to check the Hadoop version, you may see an error message like:

```
C:\Users\Firstname Lastname>hdfs -version
Error: Could not find or load main class Lastname
```

To fix this, simply move to a working directory without any spaces in the path (as I did in the screenshot above):

```
C:\Users\Firstname Lastname>cd ..

C:\Users>hdfs -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode)
```

# Configure Hadoop

Next, go to `%HADOOP_HOME%\etc\hadoop` and edit (or create) the file `core-site.xml` so it looks like the following:

## core-site.xml

```xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
```

In the same directory, edit (or create) `mapred-site.xml` with the following contents:

## mapred-site.xml

```xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>
```

Next, edit (or create) `hdfs-site.xml`:

## hdfs-site.xml

```xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///C:/Hadoop/hadoop-<version>/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///C:/Hadoop/hadoop-<version>/datanode</value>
  </property>
</configuration>
```

...yes, they should be forward slashes, even though Windows uses backslashes. This is due to the way that Hadoop interprets these file paths. Also, be sure to replace `<version>` with the appropriate Hadoop version number. Finally, edit `yarn-site.xml` so it reads:

## yarn-site.xml

```xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>
```

The last thing we need to do is create the directories that we referenced in `hdfs-site.xml`:

![](https://thepracticaldev.s3.amazonaws.com/i/4zv6rhjo808tomijw4s4.PNG)

# Patch Hadoop

Now, you need to apply [a patch created by and posted to GitHub by user cdarlint](https://github.com/cdarlint/winutils). (_Note that this patch is specific to the version of Hadoop that you're installing,_ but if the exact version isn't available, try to use the one just before the desired version... that works sometimes.)

Make a backup of your `%HADOOP_HOME%\bin` directory (copy it to `\bin.old` or similar), then copy the patched files (specific to your Hadoop version, downloaded from the above git repo) to the old `%HADOOP_HOME%\bin` directory, replacing the old files with the new ones.

Now, if you run `hdfs namenode -format` in `cmd`, you should see:

![](https://thepracticaldev.s3.amazonaws.com/i/o1r4vh0ky329o5jg6dp1.PNG)

One more thing to do: copy `hadoop-yarn-server-timelineservice-<version>` from `C:\Hadoop\hadoop-<version>\share\hadoop\yarn\timelineservice` to `C:\Hadoop\hadoop-<version>\share\hadoop\yarn` (the parent directory). (These are short version numbers, like `3.1.3`, and may not match between the `JAR` file name and the directory name.)

# Boot HDFS

Finally, you can boot HDFS by running `start-dfs.cmd` and `start-yarn.cmd` in `cmd`:

![](https://thepracticaldev.s3.amazonaws.com/i/hxs3insfi8dozupt41z3.PNG)

You should verify that the `namenode`, `datanode`, `resourcemanager`, and `nodemanager` are all running using the `jps` command:

![](https://thepracticaldev.s3.amazonaws.com/i/ar9irvm9sikggucetzov.PNG)

You can also open `localhost:8088` and `localhost:9870` in your browser to monitor your shiny, new Hadoop Distributed File System:

![](https://thepracticaldev.s3.amazonaws.com/i/b3d4dz32ynzeciu8sn71.PNG)

Finally, test that you can edit the filesystem by running `hadoop fs -mkdir /test`, which will make a directory called `test` in the root directory:

![](https://thepracticaldev.s3.amazonaws.com/i/mh4cw1ga0ev5lgyk8re7.PNG)

# Testing Hadoop and Spark

We know now how to create directories (`fs -mkdir`) and list their contents (`fs -ls`) in HDFS, what about creating and editing files? Well, files can be copied from the local file system to HDFS with `fs -put`. We can then read files in the `spark-shell` with `sc.textFile(...)`:

![](https://thepracticaldev.s3.amazonaws.com/i/nhowj0hgwqdw6edi8965.PNG)

Note that you read a file from HDFS on `hdfs://localhost:9000/` and not just `hdfs://`. This is because this is the `defaultFS` we defined in `core-site.xml`.

__If you want to stop the HDFS__, you can run the commands:

```dos
C:\Users> stop-dfs.cmd
```

and

```dos
C:\Users> stop-yarn.cmd
```

---

So there you have it! Spark running on Windows, reading files stored in HDFS. This took a bit of work to get going and I owe a lot to people who previously encountered the same bugs as me, or previously wrote tutorials which I used as a framework for this walkthrough. Here are the blogs, GitHub repos, and SO posts I used to build this tutorial:

- [Muhammad Bilal Yar's Hadoop 2.8.0 walkthrough](https://github.com/MuhammadBilalYar/Hadoop-On-Window/wiki/Step-by-step-Hadoop-2.8.0-installation-on-Window-10)
- [java.net.URISyntaxException](https://stackoverflow.com/questions/38784134/hadoop-installation-issue-on-windows)
- [java.lang.UnsatisfiedLinkError](https://stackoverflow.com/questions/33211599/hadoop-error-on-windows-java-lang-unsatisfiedlinkerror)
- [FATAL resourcemanager.ResourceManager](https://stackoverflow.com/questions/51118358/noclassdeffounderror-org-apache-hadoop-yarn-server-timelineservice-collector-tim)
- [localhost:50070 error](https://stackoverflow.com/questions/19641326/http-localhost50070-does-not-work-hadoop)
- [Kuldeep Singh's walkthrough and troubleshooting guide](https://dzone.com/articles/working-on-apache-spark-on-windows)
- [Jacek Laskowski's GitBook](https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-tips-and-tricks-running-spark-windows.html)
- [java.io.IOException: Incompatible clusterIDs](https://stackoverflow.com/questions/22316187/datanode-not-starts-correctly)
- [HDFS basic commands](https://www.tutorialspoint.com/hadoop/hadoop_hdfs_operations.htm)
- [Spark basic commands](https://spark.apache.org/examples.html)