Build Hadoop 2.X Fully distributed Environment by Ubuntu 16.04
Tags: awscommunity, ubuntu, hadoop
Published: 2022-03-29 01:30:22

# [1.Experimental model and architecture description](#1)
# [2.Basic environment configuration](#2)
# [3.JDK Hadoop environment configuration](#3)
# [4.Hadoop configuration file modification](#4)
# [5.Start Hadoop cluster (all in AdServer operation)](#5)
<h2 id="1">1.Experimental model and architecture description</h2>

### Written in the front, these pictures are from another blog of mine. Don't worry
### My Chinese Blog:[CSDN Address](https://blog.csdn.net/fly1574)

##### This experiment uses three Ubuntu 16 04 instance
| HostName | IPAddress | OS|Running Services|Role|
|--|--|--|--|--|
|adserver| 192.168.200.10| ubuntu 16.04|NameNode、SecondaryNameNode、ResourceManager、JobHistoryServer|Master|
|monserver |192.168.200.20|ubuntu 16.04|DataNode、NodeManager|Slave|
|osdserver-1|192.168.200.30 |ubuntu 16.04|DataNode、NodeManager|Slave|


<h2 id="2">2.Basic environment configuration</h2>

#### Change Hostname and the network interface name
Ubuntu 16.04 static IP address only
1. Check the IP address first
![在这里插入图片描述](https://img-blog.csdnimg.cn/d37f23be25a8415498227acc397b0996.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

Modify the network card configuration file to make it a static IP address. After modification, restart the instance to take effect
`$ sudo vi /etc/network/interfaces`![在这里插入图片描述](https://img-blog.csdnimg.cn/aa2034f807374d438efb64b4007bc905.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
Modify host name, use here

```powershell
$ sudo hostnamectl set-hostname YOUR_HOSTNAME
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/87e23e4df7d846fcacd96cb8a67a8ec0.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/2de7104fce08451f826fd96edb60a871.png)
#### Modify hosts and configure FQDN domain name
![在这里插入图片描述](https://img-blog.csdnimg.cn/ccdd3b3ba0744d60958d02e24c97192f.png)

#### Create Hadoop users and configure password free login. Each node needs to be configured

```powershell
sudo useradd -d /home/hadoop -m hadoop 
sudo passwd hadoop 
echo "hadoop ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/hadoop 
sudo chmod 0440 /etc/sudoers.d/hadoop
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/7c45d0e0345041bb96a5a0a7dea5aa0c.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)




To configure SSH password free login, you need to install openssh-server(sudo apt get install openssh-server) first 

```powershell
ubuntu@adserver:~$ ssh-keygen 
ubuntu@adserver:~$ ssh-copy-id adserver
ubuntu@adserver:~$ ssh-copy-id monserver
ubuntu@adserver:~$ ssh-copy-id osdserver-1
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/3aa24b0ab5e344699422a73fcf55f3c5.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

![在这里插入图片描述](https://img-blog.csdnimg.cn/0cecff9bdee44214b397678e74d55b2d.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

```powershell
ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@adserver  
ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@monserver  
ubuntu@adserver:~$ ssh-copy-id -i .ssh/id_rsa.pub hadoop@osdserver-1 
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/7bf46ac8daf54943b38a759da9e9f79e.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/375987dd4766489d9f94db5012fc3637.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)






<h2 id="3">3.JDK Hadoop environment</h2>

### 1. Configure JDK environment. Jdk-8u77 is used here


**Download JDK**    [jdk-8u77-linux-x64.tar.gz](https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html)
https://www.oracle.com/java/technologies/javase/javase8-archive-downloads.html
![在这里插入图片描述](https://img-blog.csdnimg.cn/57b71e87f95a477db5a9ca99a0e284da.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

```powershell
ubuntu@adserver:~$ ls -lh
total 173M
-rw-rw-r-- 1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz
ubuntu@adserver:~$ tar -zxf jdk-8u77-linux-x64.tar.gz 
ubuntu@adserver:~$ ls -lh
total 173M
drwxr-xr-x 8 ubuntu ubuntu 4.0K Mar 21  2016 jdk1.8.0_77
-rw-rw-r-- 1 ubuntu ubuntu 173M Mar 28 09:11 jdk-8u77-linux-x64.tar.gz
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/83da128844b04615955d746f29972a05.png)

```powershell
ubuntu@adserver:~$ sudo mkdir /usr/lib/jdk      
ubuntu@adserver:~$ sudo mv jdk1.8.0_77/ /usr/lib/jdk/
ubuntu@adserver:~$ sudo ls /usr/lib/jdk/
jdk1.8.0_77
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/431cedb44ae94f7fb1e60a455d54a807.png)

```powershell
ubuntu@adserver:~$ sudo vi /etc/profile 
Add JDK environment
#JDK
export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77 
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/7d1970de82c44d9e9daed0f6f297619e.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/4310ef8b3a80435890ba29e7a1173088.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/e58eb0d53f9c417d821011e37ecf15ff.png)

### 2. Configure Hadoop environment
Download Hadoop[Hadoop-2.7.2](https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz)
Download Link：
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz

#### Unzip：

```powershell
ubuntu@adserver:~$ tar -zxf hadoop-2.7.2.tar.gz 
ubuntu@adserver:~$ ls -lh
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/5c5fe6a266ef40229f6fe74830936012.png)

#### Move Hadoop to /usr/local/ directory![在这里插入图片描述](https://img-blog.csdnimg.cn/5e8f57b974294ef0ab04f723ad527857.png)

#### Add Hadoop environment variable

```powershell
$ sudo vi /etc/profile

#HADOOP
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/797b9650ceb24d76b2f152b2507f4ccc.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

```powershell
$ vi ~/.bashrc
#HADOOP
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/7351717330864c88b83c47a9ac8b7da3.png)

```powershell
$ source /etc/profile
$ source ~/.bashrc
$ hadoop version
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/57d372f23d2b4109bfdc25758a2878cc.png)






<h2 id="4">4.Hadoop configuration file modification</h2>

### Modify Hadoop configuration file

**Modify the /hadoop-2.7.2/etc/hadoop directory hadoop-env.sh、yarn-env.sh、slaves、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml**

### 1. Create the tmp folder and its subdirectories under the Hadoop directory

```powershell
ubuntu@adserver:~$ sudo mkdir -p /usr/local/hadoop-2.7.2/tmp/dfs/data
ubuntu@adserver:~$ sudo mkdir -p /usr/local/hadoop-2.7.2/tmp/dfs/name
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/16fd695fec634ddd877847de9f9c2698.png)
## Modify profile:
#### First enter the corresponding folder

```powershell
ubuntu@adserver:~$ cd /usr/local/hadoop-2.7.2/etc/hadoop/
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/8c49417fe2fc42779397763b275f2b42.png)
### 2. Add Java_ Home to Hadoop environment profile
##### ① Add Java home to Hadoop env sh

```powershell
export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77

```
![在这里插入图片描述](https://img-blog.csdnimg.cn/74898c59ace947fa929e38d1705c7233.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

##### ② Add Java home to yarn env SH, just add it directly in the first line

```powershell
export JAVA_HOME=/usr/lib/jdk/jdk1.8.0_77
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/113bea062e014d59af4f33423b89c381.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
##### ③ Add slave hostname to slave

```powershell
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi slaves 
monserver
osdserver-1
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/2e6f40e9ca52438f9a92879cb3496aeb.png)

### 3. Modify the corresponding configuration file
##### ④Modification core-site.xml

```powershell
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi core-site.xml 
```

Add the following contents to `<configuration></configuration>`
```powershell
<property>
                <name>fs.defaultFS</name>
                <value>hdfs://adserver:8020</value>
       </property>
       <property>
               <name>hadoop.tmp.dir</name>
               <value>file:/usr/local/hadoop-2.7.2/tmp</value>
               <description>Abase for other temporary   directories.</description>
       </property>
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/8b42e28996f24e1ba9632724f694bdff.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

##### ⑤Modify hdfs-site.xml file,

```powershell
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi hdfs-site.xml 
```

Add the following contents to `<configuration></configuration>`

```powershell
       <property>
                <name>dfs.namenode.secondary.http-address</name>
               <value>adserver:50090</value>
       </property>
     <property>
             <name>dfs.namenode.name.dir</name>
             <value>file:/usr/local/hadoop-2.7.2/tmp/dfs/name</value>
       </property>
      <property>
              <name>dfs.datanode.data.dir</name>
              <value>file:/usr/local/hadoop-2.7.2/tmp/dfs/data</value>
       </property>
       <property>
               <name>dfs.replication</name>
               <value>3</value>
        </property>
        <property>
                 <name>dfs.webhdfs.enabled</name>
                  <value>true</value>
         </property>
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/84f72f20a0a64aff92e2f9a25141cfeb.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
##### ⑥Modify mapred-site,xml, you need to copy the file as mapred-site.xml, and then

```powershell
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ cp mapred-site.xml.template mapred-site.xml
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi mapred-site.xml
```

Add the following contents to `<configuration></configuration>`

```powershell
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>adserver:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>adserver:19888</value>
        </property>
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/0d544b1a0ce8435e9f2d342591fc1fc8.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
##### ⑦、Modify yarn-site.xml

```powershell
ubuntu@adserver:/usr/local/hadoop-2.7.2/etc/hadoop$ vi yarn-site.xml 
```

Add the following contents to `<configuration></configuration>`

```powershell
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>adserver:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>adserver:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>adserver:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>adserver:8033</value>
  </property>
  <property>
     <name>yarn.resourcemanager.webapp.address</name>
     <value>adserver:8088</value>
  </property>
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/ad4ab8a57c8e4e52b87d51ac25d515b4.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)


### 4.Use the SCP command to add /etc/hosts, /etc/profile, ~/.bashrc, JDK and Hadoop are distributed to two slave nodes respectively. Here is only a demonstration of copying to monserver

```powershell
ubuntu@adserver:~$ scp /etc/hosts ubuntu@monserver:~/
ubuntu@adserver:~$ scp -r /etc/profile ubuntu@monserver:~/           
ubuntu@adserver:~$ scp -r .bashrc  ubuntu@monserver:~/                 
ubuntu@adserver:~$ scp -r /usr/local/hadoop-2.7.2/  ubuntu@monserver:~/
ubuntu@adserver:~$ scp -r /usr/lib/jdk/  ubuntu@monserver:~/
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/0adfd1eb6c2049ba8cde10cdf0a6aa35.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/be3ad94f262c4507bc1c1b1c991b76fb.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/0e761e981a3947a8b8dd1af425d98e9d.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

```powershell
ubuntu@monserver:~$ sudo mv hosts /etc/hosts
ubuntu@monserver:~$ sudo mv hadoop-2.7.2/ /usr/local/
ubuntu@monserver:~$ sudo mv jdk/ /usr/lib/           
ubuntu@monserver:~$ sudo update-alternatives --install /usr/bin/java java /usr/lib/jdk/jdk1.8.0_77/bin/java 300 
ubuntu@monserver:~$ sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jdk/jdk1.8.0_77/bin/javac 300
ubuntu@monserver:~$ source /etc/profile
ubuntu@monserver:~$ source .bashrc     
ubuntu@monserver:~$ java -version
ubuntu@monserver:~$ hadoop version
```



![在这里插入图片描述](https://img-blog.csdnimg.cn/b057c5ed66ce467c91c2e1d57bc97a41.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)
![在这里插入图片描述](https://img-blog.csdnimg.cn/82e6cee3c57a4c37b1d5bb90be80b8e0.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

#### Set the folder permissions of all nodes hadoop-2.7.2 to 0777：
#### Set the folder permissions of all nodes hadoop-2.7.2 to 0777：
#### Set the folder permissions of all nodes hadoop-2.7.2 to 0777：

```powershell
ubuntu@adserver:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2
ubuntu@monserver:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2
ubuntu@osdserver-1:~$  sudo chmod -R 0777 /usr/local/hadoop-2.7.2
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/2d75d739d1fa4d4b9659990cc030f847.png)
<h2 id="5">5.Start Hadoop cluster (all in AdServer operation)</h2>

### ① initialization namenode
ubuntu@adserver:~$ hadoop namenode -format 
Note: initialization is required for the first run, but not after.
If it runs successfully, it should return exiting with status 0 and prompt shutting down namenode at AdServer/xxx.xxx.xxx.XX (IP address of AdServer). The specific results are shown in the figure below：
![在这里插入图片描述](https://img-blog.csdnimg.cn/9b24d6a5116749cdb148116e82a47501.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

### ②Start Hadoop daemons (namenode, datanode, ResourceManager, nodemanager, etc.)）

#### A、Start namenode, secondarynamenode and datanode first
##### Execute on the AdServer node

```powershell
ubuntu@adserver:~$start-dfs.sh
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/90904b2e309246e0a41e80983c14261c.png)
##### At this time, the processes running on the master node are：NameNode、SecondaryNameNode  
##### At this time, the processes running on the slave node are：DataNode  
![在这里插入图片描述](https://img-blog.csdnimg.cn/9e3f90f0b6c6401a92b3dc27cd6febc4.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/fd01774fcabb4d9bb815e2b693ede09c.png)

#### B、Start ResourceManager、NodeManager

```powershell
$ start-yarn.sh
```
![在这里插入图片描述](https://img-blog.csdnimg.cn/57ad6bc23bdd4d8ea16a8b0e7fb256af.png)
##### Yarn is separated from MapReduce and is responsible for resource management and task scheduling. Yarn runs on MapReduce and provides high availability and scalability  
##### At this time, the processes running on the master node are：NameNode、SecondaryNameNode、ResourceManager
##### The processes running on the slave node are：DataNode、NodeManager  
#### C、Start JobHistoryServer

```powershell
$ mr-jobhistory-daemon.sh start historyserver
```

##### Note: a jobhistoryserver process will be added to the master node
### After TMP and logP directories on each node are created several times, be sure to delete them again

View the operation status of three nodes

```powershell
ubuntu@adserver:~$ jps
ubuntu@adserver:~$ ssh monserver "/usr/lib/jdk/jdk1.8.0_77/bin/jps"
ubuntu@adserver:~$ ssh osdserver-1 "/usr/lib/jdk/jdk1.8.0_77/bin/jps"   
```

![在这里插入图片描述](https://img-blog.csdnimg.cn/8bf1b4b4943d4edf82e37852f5391fa0.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)

![在这里插入图片描述](https://img-blog.csdnimg.cn/ea1d8b53460a4ebf87be7c2cd158b6b9.png?x-oss-process=image/watermark,type_d3F5LXplbmhlaQ,shadow_50,text_Q1NETiBA5oyH5YmR,size_20,color_FFFFFF,t_70,g_se,x_16)


# If this article is helpful to you, please like it and Follow Me. Thank you,Have a Nice Day!!!!!