A practical tutorial on building a fully distributed Hadoop environment under Ubuntu 16.4

Preface

This article mainly introduces the relevant content about ubantu 16.4 Hadoop fully distributed construction, sharing it for your reference and learning. Let's take a look at the detailed introduction.

A virtual machine

1. Mount the virtual machine in NAT network card mode

2. It is best to change the host name of several virtual machines to use, static IP /etc/network/interface, here are three hosts s101 s102 s103 ubantu, change the /etc/hostname file

3. Install ssh

On the first host, s101 creates a public and private key

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

>cd .ssh

>cp id_rsa.pub >authorized_keys Create a key library

Transfer id_rsa.pub to other hosts and put it in the .ssh directory

Through the server nc -l 8888 >~/.ssh/authorized_keys

Client nc s102 8888 <id_rsa.pub

Start installing Hadoop/jdk

1. Install VM-tools to facilitate dragging files from win 10 to ubuntu

2. Create a directory /soft

3. Change the group chown ubantu:ubantu /soft to facilitate file transfer with permissions

4. Put the file into /soft (you can use cp/mv src dst from the desktop)

tar -zxvf jdk or hadoop automatically creates a decompression directory

Configure the installation environment (/etc/environment)

1. Add JAVA_HOME=/soft/jdk-...jdk directory

2. Add HADOOP_HOME=/soft/hadoop (Hadoop directory)

3. Add /soft/jdk-...jdk/bin:/soft/hadoop/bin/:/soft/hadoop/sbin to the path

4. Check the version number through java -version Success

5.hadoop version has version number Success

Start configuring the four major HDFS files core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml

core-site.xml

<configuration>
 <property>
 <name>fs.defaultFS</name>
 <value>hdfs://s101:9000</value>
 </property>

</configuration>

2.hdfs-site.xml

<configuration>
<!-- Configurations for NameNode: -->
<property>
 <name>dfs.replication</name>
 <value>2</value>
</property>

<property>
 <name>dfs.namenode.name.dir</name>
 <value>file:/data/hdfs/name</value>
</property>

<property>
 <name>dfs.datanode.data.dir</name>
 <value>file:/data/hdfs/data</value>
</property>

<property>
 <name>dfs.namenode.secondary.http-address</name>
 <value>s101:50090</value>
</property>

<property>
 <name>dfs.namenode.http-address</name>
 <value>s101:50070</value>
 <description>
 The address and the base port where the dfs namenode web ui will listen on.
 If the port is 0 then the server will start on a free port.
 </description>
</property> 

<property>
 <name>dfs.namenode.checkpoint.dir</name>
 <value>file:/data/hdfs/checkpoint</value>
</property>

<property>
 <name>dfs.namenode.checkpoint.edits.dir</name>
 <value>file:/data/hdfs/edits</value>
</property>
</configuration>

3. mapred-site.xml

<configuration>
 <property>
 <name>mapreduce.framework.name</name>
 <value>yarn</value>
 </property>
</configuration>

4.yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
 <property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
 </property>
 <property>
 <name>yarn.resourcemanager.hostname</name>
 <value>s101</value>
 </property>
</configuration>

This is half the battle. . . . . . . . . . . . . .

Create a folder

mkdir /data/hdfs/tmp
mkdir /data/hdfs/var
mkdir /data/hdfs/logs
mkdir /data/hdfs/dfs
mkdir /data/hdfs/data
mkdir /data/hdfs/name
mkdir /data/hdfs/checkpoint
mkdir /data/hdfs/edits

Remember to modify the directory permissions

• sudo chown ubantu:ubantu /data

Next, transfer the /soft folder to other hosts

Create the xsync executable

sudo touch xsync

sudo chmod 777 xsync permissions become executable files

sudo nano xsync

#!/bin/bash
pcount=$#
if((pcount<1));then
 echo no args;
 exit;
fi

p1=$1;
fname=`basename $p1`
pdir=`cd -P $(dirname $p1);pwd`

cuser=`whoami`
for((host=102 ; host<105 ; host=host+1));do
 echo --------s$host--------
 rsync -rvl $pdir/$fname $cuser@s$host:$pdir
done

xsync /soft --------> will transfer the folder to other hosts

xsync /data

Create xcall to pass commands to other hosts

#!/bin/bash
pcount=$#
if((pcount<1));then
 echo no args;
 exit;
fi
echo --------localhost--------

$@
for ((host=102;host<105;host=host+1));do
 echo --------$shost--------
 ssh s$host $@
done

Don't worry, it's almost over.

You also need to configure workers to ask for prices

• Put the host name that needs to be configured as a data node (DataNode) in it, one per line

Pay attention to the key point

Format hadoop -namenode -format first

Restart start-all.sh

View the process xcall jps

Go to website

Did you really want to cry? You succeeded! ! !

There were many problems in the middle

1. Insufficient rsync permissions: Delete the folder and change the folder permissions chown

2. Learn to read logs

Summarize

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. If you have any questions, you can leave a message to communicate. Thank you for your support for 123WORDPRESS.COM.