Hadoop configuration
Tags: hadoop, bigdata
Published: 2024-03-03 08:42:45


Video version of this article: https://www.youtube.com/watch?v=Slbi-uzPtnw
Credits:@codewitharjun 

(Video uses cli text editor to edit config files, this tutorial will use normal text editor.)

First install java-jdk-8

`sudo apt install openjdk-8-jdk`

(Optional) To check it’s there 

`cd /usr/lib/jvm`

Now ensure that you are at the root of the terminal if not run
`cd ~`

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8vgh2wfe7t3i9gldh3re.png)


Open .bashrc file 

`sudo gedit .bashrc`

And paste in the following block

```
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 
export PATH=$PATH:/usr/lib/jvm/java-8-openjdk-amd64/bin 
export HADOOP_HOME=~/hadoop-3.3.6/ 
export PATH=$PATH:$HADOOP_HOME/bin 
export PATH=$PATH:$HADOOP_HOME/sbin 
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export YARN_HOME=$HADOOP_HOME 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native 
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native" 
export HADOOP_STREAMING=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar
export HADOOP_LOG_DIR=$HADOOP_HOME/logs 
export PDSH_RCMD_TYPE=ssh
```

`sudo apt-get install ssh`

For following commands check your hadoop version number at the time of writing this it is 3.3.6

Now go to hadoop.apache.org website download the tar file.
[Direct Link](https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz)
[Website Link](https://hadoop.apache.org/releases.html)

Once downloaded execute:
(To extract the tar file)

`tar -zxvf ~/Downloads/hadoop-3.3.6.tar.gz `

For all the configuration below ensure you are in `hadoop-3.3.6/etc/hadoop` directory.

`cd hadoop-3.3.6/etc/hadoop`

Many of the files might have `<configuration>` tag already so watch before you paste in new configurations.

Now open hadoop-env.h:

`sudo gedit hadoop-env.h`

Paste the following in hadoop-env.h:

`JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64`

(set the path for JAVA_HOME)
You might not need to use sudo in the following commands but to avoid permission issues I have added it to everything.
Let's configure other files similarly:

`sudo gedit core-site.xml`

```
<configuration> 
 <property> 
 <name>fs.defaultFS</name> 
 <value>hdfs://localhost:9000</value>  </property> 
 <property> 
<name>hadoop.proxyuser.dataflair.groups</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.dataflair.hosts</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.server.hosts</name> <value>*</value> 
 </property> 
 <property> 
<name>hadoop.proxyuser.server.groups</name> <value>*</value> 
 </property> 
</configuration>
```

`sudo gedit hdfs-site.xml`

Change \<User\> with your ubuntu username !

```
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href=''?>
<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
</configuration>
```

`sudo gedit mapred-site.xml`

```
<configuration> 
 <property> 
 <name>mapreduce.framework.name</name>  <value>yarn</value> 
 </property> 
 <property>
 <name>mapreduce.application.classpath</name> 
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> 
 </property> 
</configuration>
```

`sudo gedit yarn-site.xml`

```
<configuration> 
 <property> 
 <name>yarn.nodemanager.aux-services</name> 
 <value>mapreduce_shuffle</value> 
 </property> 
 <property> 
 <name>yarn.nodemanager.env-whitelist</name> 
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREP END_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> 
 </property> 
</configuration>
```
>


Hadoop is now configured.

Next execute following one by one:

```
ssh
ssh localhost 
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 
chmod 0600 ~/.ssh/authorized_keys 
hadoop-3.3.6/bin/hdfs namenode -format
export PDSH_RCMD_TYPE=ssh
```

To start hadoop
`start-all.sh`

To check if hadoop is running go to [http://localhost:9870/](http://localhost:9870/)

To stop hadoop
`stop-all.sh`


This is an update version to this [article](https://codewitharjun.medium.com/install-hadoop-on-ubuntu-operating-system-6e0ca4ef9689)