Hadoop 3.2.3 分布式集群部署Ubuntu 20.04 双节点 5 步配置与 4 大常见错误排查1. 环境准备与系统配置在开始部署Hadoop集群之前需要确保两台Ubuntu 20.04服务器已正确配置。以下是基础环境准备的关键步骤1.1 主机名与hosts配置首先需要为两台服务器分别设置主机名如master和slave并确保它们能够通过主机名互相解析# 在master节点执行 sudo hostnamectl set-hostname master # 在slave节点执行 sudo hostnamectl set-hostname slave接着修改所有节点的/etc/hosts文件添加IP与主机名的映射关系192.168.1.100 master 192.168.1.101 slave1.2 SSH免密登录配置Hadoop集群管理需要主节点能够无密码访问所有从节点。以下是配置步骤# 在master节点生成密钥对 ssh-keygen -t rsa -P -f ~/.ssh/id_rsa # 将公钥复制到本地和slave节点 cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys ssh-copy-id -i ~/.ssh/id_rsa.pub slave1.3 防火墙与SELinux确保防火墙允许Hadoop相关端口通信sudo ufw allow 22 sudo ufw allow 9000 sudo ufw allow 50070 sudo ufw enable2. JDK安装与配置Hadoop运行依赖Java环境推荐使用OpenJDK 82.1 安装OpenJDKsudo apt update sudo apt install -y openjdk-8-jdk2.2 环境变量配置编辑/etc/profile文件添加以下内容export JAVA_HOME/usr/lib/jvm/java-8-openjdk-amd64 export PATH$PATH:$JAVA_HOME/bin使配置立即生效source /etc/profile验证安装java -version3. Hadoop安装与基础配置3.1 下载与解压从Apache官网下载Hadoop 3.2.3并解压wget https://archive.apache.org/dist/hadoop/core/hadoop-3.2.3/hadoop-3.2.3.tar.gz tar -xzvf hadoop-3.2.3.tar.gz -C /usr/local/ mv /usr/local/hadoop-3.2.3 /usr/local/hadoop3.2 环境变量配置在/etc/profile中添加Hadoop环境变量export HADOOP_HOME/usr/local/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin3.3 核心配置文件修改Hadoop的核心配置文件位于$HADOOP_HOME/etc/hadoop目录下core-site.xml配置configuration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/usr/local/hadoop/tmp/value /property /configurationhdfs-site.xml配置configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name valuefile://${hadoop.tmp.dir}/dfs/name/value /property property namedfs.datanode.data.dir/name valuefile://${hadoop.tmp.dir}/dfs/data/value /property /configuration4. 集群部署与启动4.1 配置workers文件在master节点上配置从节点列表echo slave $HADOOP_HOME/etc/hadoop/workers4.2 格式化HDFS首次启动前需要格式化NameNodehdfs namenode -format4.3 启动集群使用以下命令启动HDFS和YARNstart-dfs.sh start-yarn.sh验证集群状态jps # 应在master看到NameNode、ResourceManager等进程5. 常见错误排查5.1 NativeCodeLoader警告错误现象WARN util.NativeCodeLoader: Unable to load native-hadoop library解决方案export HADOOP_OPTS-Djava.library.path$HADOOP_HOME/lib/native5.2 MRAppMaster类加载失败错误现象Class org.apache.hadoop.mapreduce.v2.app.MRAppMaster not found解决方案 在yarn-site.xml中添加property nameyarn.application.classpath/name value$HADOOP_HOME/etc/hadoop:$HADOOP_HOME/share/hadoop/common/lib/*:$HADOOP_HOME/share/hadoop/common/*:$HADOOP_HOME/share/hadoop/hdfs:$HADOOP_HOME/share/hadoop/hdfs/lib/*:$HADOOP_HOME/share/hadoop/hdfs/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/yarn:$HADOOP_HOME/share/hadoop/yarn/lib/*:$HADOOP_HOME/share/hadoop/yarn/*/value /property5.3 安全模式问题错误现象Name node is in safe mode解决方案hdfs dfsadmin -safemode leave5.4 连接超时问题错误现象Retrying connect to server: slave/192.168.1.101:41589解决方案 检查网络连接和防火墙设置确保所有节点间的通信端口开放。