HadoopApache开源的分布式大数据框架用普通廉价服务器组成集群实现海量数据存储 海量数据计算。核心解决单机硬盘存不下 TB/PB 级数据、单机计算太慢的问题。两大核心组件HDFSHadoop Distributed File System分布式文件存储 把大文件拆成小块分散存到多台机器多副本备份不怕某台机器宕机丢数据。MapReduce分布式计算框架 把计算任务拆分分发到各个数据所在机器并行运算移动计算不移动大量数据。现代Hadoop生态还包含YARN资源调度管理器管理集群CPU内存、Hive、HBase、SparkSpark可替代MapReduce做计算等。作用用处海量数据存储日志、爬虫数据、业务流水几十TB~PB级别普通硬盘存不下HDFS横向加机器就扩容。离线大数据分析网站访问日志分析、用户行为统计、电商订单统计、报表、数据仓库。MapReduce适合离线批量处理不适合毫秒级实时查询。支撑大数据生态底座Hive做数仓、HBase做分布式数据库、Spark/Flink跑计算很多都跑在Hadoop YARN集群上。hadoop高可用解决普通HDFS单点故障NameNode挂掉整个集群不可用。角色分配主机运行进程说明server1NameNode(nn1)、DFSZKFailoverController(ZKFC)、ResourceManager(rm1)、DataNodeHDFS主备其中一台RM主机server5NameNode(nn2)、DFSZKFailoverController(ZKFC)、ResourceManager(rm2)、DataNodeHDFS另一台备NNRM备机server2QuorumPeerMain(Zookeeper)、JournalNode、DataNode三台合一ZK JN DNserver3QuorumPeerMain(Zookeeper)、JournalNode、DataNode三台合一ZK JN DNserver4QuorumPeerMain(Zookeeper)、JournalNode、DataNode三台合一ZK JN DNzookeeper集群部署先停止原有Hadoop集群[hadoopserver1 hadoop]$ sbin/stop-all.sh把zookeeper压缩包移动到hadoop普通用户家目录登入普通用户后解压缩[rootserver1 ~]# mv apache-zookeeper-3.8.6-bin.tar.gz /home/hadoop/[rootserver1 ~]# su - hadoop[hadoopserver1 ~]$ tar zxf apache-zookeeper-3.8.6-bin.tar.gzserver2、server3、server4都一样先挂载server1的/home/hadoop共享目录每台机子都需要[rootserver2 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/将所有节点中的临时文件中的内容删掉防止后续部署有冲突报错[hadoopserver2 ~]$ rm -fr /tmp/*Zookeeper配置文件修改所有节点zoo.cfg文件内容完全一模一样不需要改只改myid文件[hadoopserver2 ~]$ cd apache-zookeeper-3.8.6-bin/conf/[hadoopserver2 conf]$ cp zoo_sample.cfg zoo.cfg[hadoopserver2 conf]$ vim zoo.cfg[hadoopserver2 conf]$ mkdir /tmp/zookeeper配置myid集群标识只有id输入不一致但每个节点的文件内容与上方文件中设置的id保持一致[hadoopserver2 conf]$ echo 1 /tmp/zookeeper/myid[hadoopserver3 ~]$ echo 2 /tmp/zookeeper/myid[hadoopserver4 ~]$ echo 3 /tmp/zookeeper/myid启动集群并查看状态要全部都启动后查看集群节点的角色[hadoopserver2 apache-zookeeper-3.8.6-bin]$ bin/zkServer.sh start[hadoopserver2 apache-zookeeper-3.8.6-bin]$ bin/zkServer.sh status查看状态选举server3作为leaderhadoop新加一台机子server5安装nfs-utils创建普通用户uidgid要与其余的节点都一致如果不一样可以手动修改成一样的然后挂载server1登入普通用户hadoop[rootserver5 ~]# groupmod -g 1000 hadoop[rootserver5 ~]# usermod -u 1000 -g 1000 hadoop在server1上编辑文件[hadoopserver1 hadoop]$ vim core-site.xmlfs.defaultFS原来写hdfs:// 主机名现在改为逻辑集群名mastersha.zookeeper.quorum填写zookeeper集群地址列表[hadoopserver1 hadoop]$ vim hdfs-site.xmlconfiguration property namedfs.replication/name value3/value /property property namedfs.nameservices/name valuemasters/value /property property namedfs.ha.namenodes.masters/name valueh1,h2/value /property property namedfs.namenode.rpc-address.masters.h1/name value192.168.159.133:9000/value /property property namedfs.namenode.http-address.masters.h1/name value192.168.159.133:9870/value /property property namedfs.namenode.rpc-address.masters.h2/name value192.168.159.137:9000/value /property property namedfs.namenode.http-address.masters.h2/name value192.168.159.137:9870/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://192.168.159.134:8485;192.168.159.135:8485;192.168.159.136:8485/masters/value /property property namedfs.journalnode.edits.dir/name value/tmp/journaldata/value /property property namedfs.ha.automatic-failover.enabled/name valuetrue/value /property property namedfs.client.failover.proxy.provider.masters/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name value sshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hadoop/.ssh/id_rsa/value /property property namedfs.ha.fencing.ssh.connect-timeout/name value30000/value /property /configuration格式化[hadoopserver1 hadoop]$ bin/hdfs namenode -format把格式化之后元数据目录拷贝给第二台namenodeserver5[hadoopserver1 hadoop]$ scp -r /tmp/hadoop-hadoop server5:/tmp所有journalnode节点(server2、3、4)启动journalnode进程[hadoopserver2 hadoop]$ bin/hdfs --daemon start journalnode[hadoopserver1 hadoop]$ bin/hdfs zkfc -formatZK格式化zookeeper在zk里面创建hdfs-ha持久化节点保存active/standby状态、锁信息。启动hdfs集群[hadoopserver1 hadoop]$ sbin/start-dfs.sh此时访问浏览器一切正常server1状态为activeserver5为standby故障切换将进程直接杀掉[hadoopserver1 hadoop]$ jps19184 DFSZKFailoverController18627 NameNode19467 Jps[hadoopserver1 hadoop]$ kill -9 18627[hadoopserver1 hadoop]$ jps19184 DFSZKFailoverController19760 Jps杀掉以后server5接管为masterserver1连接不上再次启动后server1状态为standby测试server5放入一些数据数据访问也正常[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml inputyarn高可用编辑文件[hadoopserver1 hadoop]$ vim mapred-site.xmlconfiguration property namemapreduce.framework.name/name valueyarn/value /property property namemapreduce.application.classpath/name value$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value /property /configuration[hadoopserver1 hadoop]$ vim yarn-site.xmlconfiguration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.env-whitelist/name valueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value /property property nameyarn.resourcemanager.ha.enabled/name valuetrue/value /property property nameyarn.resourcemanager.cluster-id/name valueRM_CLUSTER/value /property property nameyarn.resourcemanager.ha.rm-ids/name valuerm1,rm2/value /property property nameyarn.resourcemanager.hostname.rm1/name valueserver1/value /property property nameyarn.resourcemanager.hostname.rm2/name valueserver5/value /property property nameyarn.resourcemanager.recovery.enabled/name valuetrue/value /property property nameyarn.resourcemanager.store.class/name valueorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore/value /property property nameyarn.resourcemanager.zk-address/name value192.168.159.134:2181,192.168.159.135:2181,192.168.159.136:2181/value /property /configuration启动yarn集群[hadoopserver1 hadoop]$ sbin/start-yarn.shstart‑yarn.sh脚本会读取yarn‑site里面rm‑ids列表自动在rm1(server1)、rm2(server5)两台机器启动ResourceManager所有DataNode节点启动NodeManager。此时浏览器访问server1的8088端口杀掉RM的进程可以看到server5此时的状态为active[hadoopserver1 hadoop]$ jps19184 DFSZKFailoverController20370 NameNode39420 ResourceManager40029 Jps[hadoopserver1 hadoop]$ kill -9 39420此时浏览器访问server5的8088端口可以看到节点正常[hadoopserver1 hadoop]$ bin/yarn --daemon start resourcemanager再次启动server1的状态变为standby