Common considerations for building a Hadoop 3.2.0 cluster

One port changes

In version 3.2.0, the namenode page port is 9870 and the datanode management port is 8088, so these two ports need to be opened for web page access.

But there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation

In hadoop-env.sh, you need to configure not only java-home but also the following user variables:

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_191
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

3.2.0 version mapred-site.xml file configuration needs to add mapreduce.application.classpath attribute

<property> <name>mapreduce.application.classpath</name> <value> /usr/local/hadoop3/etc/hadoop, /usr/local/hadoop3/share/hadoop/common/*, /usr/local/hadoop3/share/hadoop/common/lib/*, /usr/local/hadoop3/share/hadoop/hdfs/*, /usr/local/hadoop3/share/hadoop/hdfs/lib/*, /usr/local/hadoop3/share/hadoop/mapreduce/*, /usr/local/hadoop3/share/hadoop/mapreduce/lib/*, /usr/local/hadoop3/share/hadoop/yarn/*, /usr/local/hadoop3/share/hadoop/yarn/lib/* </value> </property>

Four salave files are programmed into workers files, and the content configuration is the same

5. The startup is successful without error, but the most annoying problem is that the number of Live Nodes is 0

Solution steps:

First check the datanode log:

If this occurs, it means that there is a problem with the connection configuration. Check the core-site.xml configuration, which is used to configure the communication between datanode and namnode:

At a glance

There is a problem with the connection address configured here

hdfs should be configured as the address of namnode, not the address of datanode. All nodes have the same address. After configuration, restart is successful.

There are also reports online that /etc/hots resolution failed. This is also a reason. Just configure the domain name mapping. The important thing is to first look at the log analysis to find out what the problem is.

Last successful page

The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM.