Install Hadoop on Ubuntu
Tags: bigdata, hadoop, ubuntu, learning
Published: 2023-11-04 18:58:55

Apache Hadoop is a collection of utilities that allows you to manage the processing of large datasets across clusters of computers.

Also, it is tolerant to cluster failures. If one of your clusters crashes, Hadoop can be used to recover data from other nodes.

And in this guide, I will walk you through the installation process.

---

## **How to install Hadoop on Ubuntu**

To install Hadoop, you will have to go through various steps, which include:

- Installing Java and configuring environment variables
- Creating user and configuring SSH
- Installation and configuration of Hadoop

---

So let's start with the first step:

## **Step 1: Installing Java on Ubuntu**

To install java on Ubuntu, all you have to do is execute the following command:

```
sudo apt install default-jdk default-jre -y
```

To verify the installation, check the java version on your system:

```
java -version
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pjngfdqz2758pmavloxk.jpg)

---

## **Step 2: Create a user for Hadoop and configure SSH**

First, create a new user named `hadoop`:

```
sudo adduser hadoop
```

To enable superuser privileges to the new user, add it to the `sudo` group:

```
sudo usermod -aG sudo hadoop
```

Once done, switch to the user `hadoop`:

```
sudo su - hadoop
```

Next, install the OpenSSH server and client:

```
sudo apt install openssh-server openssh-client -y
```

Now, use the following command to generate private and public keys:

```
ssh-keygen -t rsa
```

Here, it will ask you:

- Where to save the key (hit enter to save it inside your home directory)
- Create passphrase for keys (leave blank for no passphrase)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2nz8p4aa783an4yst980.jpg)

Now, add the public key to `authorized_keys`:

```
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
```

Use the chmod command to change the file permissions of `authorized_keys`:

```
sudo chmod 640 ~/.ssh/authorized_keys
```

Finally, verify the SSH configuration:

```
ssh localhost
```

If you have not configured the password, all you have to do is type `yes` and hit enter if you added a passphrase for the keys, it will ask you to enter here:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/20d96xlbd1lh67j6f2mf.jpg)

---

## **Step 3: Download and install Apache Hadoop on Ubuntu**

If you have created a user for Hadoop, first, log in as the `hadoop` user:

```
sudo su - hadoop
```

Now, visit the download page for Apache Hadoop and copy the link for the most recent stable release.

While writing, its `3.3.6` so I will be using the wget command to download this release:

```
wget https://downloads.apache.org/hadoop/common/stable/hadoop-3.3.6.tar.gz
```

Once you are done with the download, extract the file using the following command:

```
tar -xvzf hadoop-3.3.6.tar.gz
```

Next, move the extracted file to the `/usr/local/hadoop` using the following command:

```
sudo mv hadoop-3.3.6 /usr/local/hadoop
```

Now, create a directory using mkdir command to store logs:

```
sudo mkdir /usr/local/hadoop/logs
```

Finally, change the ownership of the `/usr/local/hadoop` to the user `hadoop`:

```
sudo chown -R hadoop:hadoop /usr/local/hadoop
```

---

## **Step 4: Configure Hadoop on Ubuntu**

Here, I will walk you through the configuration of the Hadoop environment variable.

First, open the `.bashrc` file using the following command:

```
sudo nano ~/.bashrc
```

Jump to the end of the line in the nano text editor by pressing `Alt + /` and paste the following lines:

```
export HADOOP_HOME=/usr/local/hadoop

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/budlru594i9ea61qt3gg.jpg)

Save changes and exit from the nano text editor.

To enable the changes, source the `.bashrc` file:

```
source ~/.bashrc
```

---

## **Step 5: Configure java environment variables**

To use Hadoop, you are required to enable its core functions which include YARN, HDFS, MapReduce, and Hadoop-related project settings.

To do that, you will have to define java environment variables in `hadoop-env.sh` file.

### **Edit the hadoop-env.sh file**

First, open the `hadoop-env.sh` file:

```
sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
```

Press `Alt + /` to jump to the end of the file and paste the following lines in the file to add the path of the Java:

```
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

export HADOOP_CLASSPATH+=" $HADOOP_HOME/lib/*.jar"
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1grd0s4m4lsbw6yot88q.jpg)


Save changes and exit from the text editor.

Next, change your current working directory to `/usr/local/hadoop/lib`:

```
cd /usr/local/hadoop/lib
```

Here, download the javax activation file:

```
sudo wget https://jcenter.bintray.com/javax/activation/javax.activation-api/1.2.0/javax.activation-api-1.2.0.jar
```

Once done, check the Hadoop version in Ubuntu:

```
hadoop version
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jcirkkn3q6t8vdlfxlha.jpg)

Next, you will have to edit the `core-site.xml` file to specify the URL for the name node.

### **Edit the `core-site.xml` file**

First, open the `core-site.xml` file using the following command:

```
sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml
```

And add the following lines in between `<configuration>  </configuration>`:

```
<property>

     <name>fs.default.name</name>

          <value>hdfs://0.0.0.0:9000</value>

          <description>The default file system URI</description>

</property>
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kpff73b7e1yjdyz8eyr8.jpg)

Save the changes and exit from the text editor.

Next, create a directory to store node metadata using the following command:

```
sudo mkdir -p /home/hadoop/hdfs/{namenode,datanode}
```

And change the ownership of the created directory to the `hadoop` user:

```
sudo chown -R hadoop:hadoop /home/hadoop/hdfs
```

### **Edit the `hdfs-site.xml` configuration file**

By configuring the `hdfs-site.xml` file, you will define the location for storing node metadata, fs-image file.

So first open the configuration file:

```
sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml
```

And paste the following line in between `<configuration> ... </configuration>`:

```
<property>

     <name>dfs.replication</name>

     <value>3</value>

</property>

<property>

     <name>dfs.name.dir</name>

     <value>file:///home/hadoop/hdfs/namenode</value>

</property>

<property>

     <name>dfs.data.dir</name>

     <value>file:///home/hadoop/hdfs/datanode</value>

</property>
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6rkviih4g18078kubva.jpg)

Save changes and exit from the `hdfs-site.xml` file.

### **Edit the `mapred-site.xml` file**

By editing the `mapred-site.xml` file, you can define the MapReduce values.

To do that, first, open the configuration file using the following command:

```
sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml
```

And paste the following line in between `<configuration> ... </configuration>`:

```
<property>

      <name>mapreduce.framework.name</name>

      <value>yarn</value>

   </property>

<property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rb9dzxk858hyk2acs5fq.jpg)

Save and exit from the nano text editor.

### **Edit the `yarn-site.xml` file**

This is the last configuration file that needs to be edited to use the Hadoop service.

The purpose of editing this file is to define the YARN settings.

First, open the configuration file:

```
sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
```

Paste the following in between `<configuration> ... </configuration>`:

```
<property>

      <name>yarn.nodemanager.aux-services</name>

      <value>mapreduce_shuffle</value>

   </property>
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qyqbqqa5mqstzyw2cxkr.jpg)

Save changes and exit from the config file.

Finally, use the following command to validate the Hadoop configuration and to format the HDFS NameNode:

```
hdfs namenode -format
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dktrmu77pd5g4ximj67j.jpg)

---

## **Step 6: Start the Hadoop cluster**

To start the Hadoop cluster, you will have to start the previously configured nodes.

So let's start with starting the NameNode and DataNode:

```
start-dfs.sh
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vocnhtsn62cmw7z64str.jpg)

Next, start the node manager and resource manager:

```
start-yarn.sh
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/erzmpo8a4xxf69f00wnv.jpg)

To verify whether the services are running as intended, use the following command:


```
jps
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w27m3b8fvq7i8ol8viqh.jpg)

---

## **Step 7: Access the Hadoop web interface**

To access the Hadoop web interface, you will have to know your IP and append the port no `9870` in your address bar:

```
http://server-IP:9870
```

My IP is `10.0.2.15` so I will be entering the following:

```
http://10.0.2.15:98705
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u0c21sbh1tu8zgoe2kau.jpg)

And there you have it!

---

## **Wrapping Up**
While the tutorial was lengthy for sure, the major part was just copying and pasting lines to your terminal.

I hope you will find this guide helpful.

But if you encounter any errors while executing shown guide, let me know in the comments and I will try my best to come up with the best solution possible.