说明在kubernetes集群部署Apache Hadoop定制版本实现单namenode单datanode实现了数据持久化数据可靠性依赖k8s的pvc和pv。组件高可用依赖k8s的statefulset。若想增强数据存储可靠性应当给k8s集群引入可靠的storageclass然后修改以下yaml文件中的storageclass使存储的数据不会因node故障不可用。配置configmap.yamlapiVersion: v1 kind: ConfigMap metadata: name: hadoop-config data: core-site.xml:|?xmlversion1.0?configurationpropertynamehadoop.tmp.dir/namevalue/opt/hadoop/tmpdata/value/propertypropertynamefs.defaultFS/namevaluehdfs://namenode-0.namenode.hadoop:9000/value/propertypropertynamehadoop.proxyuser.hadoop.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hadoop.groups/namevalue*/value/propertypropertynamehadoop.proxyuser.hue.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hue.groups/namevalue*/value/propertypropertynamehadoop.proxyuser.hive.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hive.groups/namevalue*/value/propertypropertynamehadoop.proxyuser.root.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.root.groups/namevalue*/value/property/configurationhdfs-site.xml:|?xmlversion1.0?configurationpropertynamedfs.namenode.name.dir/namevalue/opt/hadoop/data/nn/value/propertypropertynamedfs.datanode.data.dir/namevalue/opt/hadoop/data/dn/value/propertypropertynamedfs.namenode.rpc-address/namevaluenamenode-0.namenode.hadoop:9000/value/propertypropertynamedfs.namenode.rpc-bind-host/namevalue0.0.0.0/value/propertypropertynamedfs.namenode.http-bind-host/namevalue0.0.0.0/value/propertypropertynamedfs.replication/namevalue1/value/propertypropertynamedfs.permissions.enabled/namevaluefalse/value/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/propertypropertynamedfs.namenode.datanode.registration.ip-hostname-check/namevaluefalse/valuedescription当namenode反向解析datanode失败时,改为false。/description/property/configurationyarn-site.xml:|?xmlversion1.0?configurationpropertynameyarn.resourcemanager.hostname/namevalueresourcemanager-0.resourcemanager.hadoop/value/propertypropertynameyarn.resourcemanager.bind-host/namevalue0.0.0.0/value/propertypropertynameyarn.resourcemanager.webapp.address/namevalue0.0.0.0:8088/value/propertypropertynameyarn.nodemanager.bind-host/namevalue0.0.0.0/value/propertypropertynameyarn.nodemanager.pmem-check-enabled/namevaluefalse/value/propertypropertynameyarn.nodemanager.delete.debug-delay-sec/namevalue600/value/propertypropertynameyarn.nodemanager.vmem-check-enabled/namevaluefalse/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.aux-services.mapreduce_shuffle.class/namevalueorg.apache.hadoop.mapred.ShuffleHandler/value/propertypropertynameyarn.acl.enable/namevaluefalse/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value/property/configurationmapred-site.xml:|?xmlversion1.0?configurationpropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynameyarn.app.mapreduce.am.env/namevalueHADOOP_MAPRED_HOME/opt/hadoop/value/propertypropertynamemapreduce.map.env/namevalueHADOOP_MAPRED_HOME/opt/hadoop/value/propertypropertynamemapreduce.reduce.env/namevalueHADOOP_MAPRED_HOME/opt/hadoop/value/property/configurationcapacity-scheduler.xml:|?xmlversion1.0?configurationpropertynameyarn.scheduler.capacity.maximum-applications/namevalue10000/value/propertypropertynameyarn.scheduler.capacity.maximum-am-resource-percent/namevalue0.1/value/propertypropertynameyarn.scheduler.capacity.resource-calculator/namevalueorg.apache.hadoop.yarn.util.resource.DefaultResourceCalculator/value/propertypropertynameyarn.scheduler.capacity.root.queues/namevaluedefault/value/propertypropertynameyarn.scheduler.capacity.root.default.capacity/namevalue100/value/propertypropertynameyarn.scheduler.capacity.root.default.user-limit-factor/namevalue1/value/propertypropertynameyarn.scheduler.capacity.root.default.maximum-capacity/namevalue100/value/propertypropertynameyarn.scheduler.capacity.root.default.state/namevalueRUNNING/value/propertypropertynameyarn.scheduler.capacity.root.default.acl_submit_applications/namevalue*/value/propertypropertynameyarn.scheduler.capacity.root.default.acl_administer_queue/namevalue*/value/propertypropertynameyarn.scheduler.capacity.node-locality-delay/namevalue40/value/propertypropertynameyarn.scheduler.capacity.root.default.acl_application_max_priority/namevalue*/value/propertypropertynameyarn.scheduler.capacity.root.default.maximum-application-lifetime/namevalue-1/value/propertypropertynameyarn.scheduler.capacity.rack-locality-additional-delay/namevalue-1/value/property/configuration配置namenode.yaml容器镜像可以改为自己构建的。apiVersion: v1 kind: Service metadata: name: namenode labels: app: hadoop-namenode spec: ports: - port:9870name: web - port:9000name: rpc clusterIP: None selector: app: hadoop-namenode --- apiVersion: v1 kind: Service metadata: name: namenode-external labels: app: hadoop-namenode spec: type: NodePort ports: - port:9870name: web targetPort:9870nodePort:30070- port:9000name: rpc targetPort:9000nodePort:30090selector: app: hadoop-namenode --- apiVersion: apps/v1 kind: StatefulSet metadata: name: namenode spec: serviceName: namenode replicas:1selector: matchLabels: app: hadoop-namenode template: metadata: labels: app: hadoop-namenode spec:# --- 1. 初始化容器负责权限管理、目录准备和按需格式化 ---initContainers: - name: init-namenode image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent securityContext: runAsUser:0privileged:truecommand:[/bin/bash,-c]args: -|set-eechoStep 1: Preparing directories and permissions...mkdir-p/opt/hadoop/data/nn# 确保 hadoop 用户拥有数据目录权限chown-Rhadoop:hadoop /opt/hadoop/dataechoStep 2: Checking if NameNode needs formatting...if[!-f/opt/hadoop/data/nn/current/VERSION];thenechoNo VERSION file found. Formatting NameNode...# 使用非交互模式强制格式化suhadoop-chdfs namenode -format -nonInteractiveechoFormatting completed successfully.elseechoNameNode already formatted (VERSION file exists). Skipping.grep-iclusterID/opt/hadoop/data/nn/current/VERSIONfivolumeMounts: - name: hadoop-nn-data mountPath: /opt/hadoop/data/nn subPath: nn - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml# --- 2. 主容器仅负责运行 NameNode 服务 ---containers: - name: namenode image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent command:[/bin/bash,-c]# 使用 exec 确保 namenode 是 PID 1能够接收系统信号实现优雅停机args:[exec hdfs namenode]env: - name: HADOOP_CONF_DIR value:/opt/hadoop/etc/hadoop- name: HADOOP_LOG_DIR value:/opt/hadoop/logs- name: HADOOP_HEAPSIZE value:1024ports: - containerPort:9870name: web - containerPort:9000name: rpc# 资源限制保证 Pod 的 QOS 等级resources: requests: memory:2Gicpu:500mlimits: memory:4Gicpu:1000m# 健康检查通过 JMX 接口确保 NameNode 真正可用readinessProbe: httpGet: path: /jmx port:9870initialDelaySeconds:40periodSeconds:10livenessProbe: httpGet: path: /jmx port:9870initialDelaySeconds:100periodSeconds:30volumeMounts: - name: hadoop-nn-data mountPath: /opt/hadoop/data/nn subPath: nn - name: hadoop-logs mountPath: /opt/hadoop/logs - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml volumes: - name: hadoop-config-volume configMap: name: hadoop-config - name: hadoop-logs emptyDir:{}# --- 3. 存储定义 ---volumeClaimTemplates: - metadata: name: hadoop-nn-data spec: accessModes:[ReadWriteOnce]storageClassName:localresources: requests: storage: 20Gi配置datanode.yamlapiVersion: v1 kind: Service metadata: name: datanode labels: app: hadoop-datanode spec: ports: - port:9864name: web - port:9866name: data clusterIP: None selector: app: hadoop-datanode --- apiVersion: v1 kind: Service metadata: name: datanode-external labels: app: hadoop-datanode spec: type: NodePort ports: - port:9864name: web targetPort:9864# nodePort 留空系统将自动分配 30000-32767 之间的随机端口- port:9866name: data targetPort:9866selector: app: hadoop-datanode --- apiVersion: apps/v1 kind: StatefulSet metadata: name: datanode spec: serviceName: datanode replicas:1selector: matchLabels: app: hadoop-datanode template: metadata: labels: app: hadoop-datanode spec: securityContext: fsGroup:1000initContainers:# --- 1. 等待 NameNode RPC 端口就绪 ---- name: wait-for-namenode image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent command:[/bin/sh,-c]args: -|echo⏳ 正在检测 NameNode RPC 端口 (9000)...while!nc-znamenode-0.namenode.hadoop9000;doecho 等待 NameNode 响应中...sleep5doneecho✅ NameNode RPC 已就绪# --- 2. 动态权限校准 (自动识别 hadoop 用户 UID) ---- name: init-permissions image: zhuyifeiruichuang/hadoop:3.1.1 securityContext: runAsUser:0privileged:truecommand:[/bin/sh,-c]args: -|echo 正在准备数据目录并校准权限...mkdir-p/opt/hadoop/data/dn# 动态检测镜像内是否存在 hadoop 用户并授权ifidhadoop/dev/null21;thenecho检测到 hadoop 用户执行 chown hadoop:hadoopchown-Rhadoop:hadoop /opt/hadoop/dataelseecho未检测到 hadoop 用户回退使用 UID 1000chown-R1000:1000 /opt/hadoop/datafi# 授予 775 权限确保 DataNode 内部检查通过chmod-R775/opt/hadoop/dataecho✅ 权限校准完成containers: - name: datanode image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent# 以 root 运行进程是解决 K8s 挂载卷 chmod 权限报错 (EPERM) 的最稳妥方案securityContext: runAsUser:0command:[/bin/bash,-c]args: -|exportHADOOP_CONF_DIR/opt/hadoop/etc/hadoopecho 启动 DataNode...exechdfs datanode env: - name: HADOOP_CONF_DIR value:/opt/hadoop/etc/hadoopports: - containerPort:9864name: web - containerPort:9866name: data resources: requests: memory:2Gicpu:500mlimits: memory:4Gicpu:1000mlivenessProbe: httpGet: path: / port:9864initialDelaySeconds:60periodSeconds:20readinessProbe: httpGet: path: / port:9864initialDelaySeconds:20periodSeconds:10volumeMounts: - name: hadoop-dn-data mountPath: /opt/hadoop/data/dn subPath: dn - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/yarn-site.xml subPath: yarn-site.xml volumes: - name: hadoop-config-volume configMap: name: hadoop-config# --- 存储卷模板 ---volumeClaimTemplates: - metadata: name: hadoop-dn-data spec: accessModes:[ReadWriteOnce]storageClassName:localresources: requests: storage: 20Gi配置nodemanager.yamlapiVersion: v1 kind: Service metadata: name: nodemanager labels: app: hadoop-nodemanager spec: ports: - port:8042name: web clusterIP: None selector: app: hadoop-nodemanager --- apiVersion: v1 kind: Service metadata: name: nodemanager-external labels: app: hadoop-nodemanager spec: type: NodePort ports: - port:8042name: web targetPort:8042selector: app: hadoop-nodemanager --- apiVersion: apps/v1 kind: StatefulSet metadata: name: nodemanager spec: serviceName: nodemanager replicas:1selector: matchLabels: app: hadoop-nodemanager template: metadata: labels: app: hadoop-nodemanager spec: containers: - name: nodemanager image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent command:[yarn,nodemanager]env: - name: HADOOP_CONF_DIR value:/opt/hadoop/etc/hadoopports: - containerPort:8042volumeMounts: - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/yarn-site.xml subPath: yarn-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/mapred-site.xml subPath: mapred-site.xml volumes: - name: hadoop-config-volume configMap: name: hadoop-config配置resourcemanager.yamlapiVersion: v1 kind: Service metadata: name: resourcemanager labels: app: hadoop-resourcemanager spec: ports: - port:8088name: web - port:8032name: rpc clusterIP: None selector: app: hadoop-resourcemanager --- apiVersion: v1 kind: Service metadata: name: resourcemanager-external labels: app: hadoop-resourcemanager spec: type: NodePort ports: - port:8088name: web targetPort:8088- port:8032name: rpc targetPort:8032selector: app: hadoop-resourcemanager --- apiVersion: apps/v1 kind: StatefulSet metadata: name: resourcemanager spec: serviceName: resourcemanager replicas:1selector: matchLabels: app: hadoop-resourcemanager template: metadata: labels: app: hadoop-resourcemanager spec: containers: - name: resourcemanager image: zhuyifeiruichuang/hadoop:3.1.1 imagePullPolicy: IfNotPresent command:[yarn,resourcemanager]env: - name: HADOOP_CONF_DIR value:/opt/hadoop/etc/hadoopports: - containerPort:8088- containerPort:8032volumeMounts: - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/core-site.xml subPath: core-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/hdfs-site.xml subPath: hdfs-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/yarn-site.xml subPath: yarn-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/mapred-site.xml subPath: mapred-site.xml - name: hadoop-config-volume mountPath: /opt/hadoop/etc/hadoop/capacity-scheduler.xml subPath: capacity-scheduler.xml volumes: - name: hadoop-config-volume configMap: name: hadoop-config部署kubectl create namespace hadoop kubectl apply-fconfigmap.yaml-nhadoop kubectl apply-fnamenode.yaml-nhadoop kubectl apply-fdatanode.yaml-nhadoop kubectl apply-fresourcemanager.yaml-nhadoop kubectl apply-fnodemanager.yaml-nhadoop部署后查询rootmaster Mon Nov24[10:05:40]:/opt/bigdata2/hadoop/v3.1.1/v4# kubectl get all -n hadoopNAME READY STATUS RESTARTS AGE pod/datanode-01/1 Running09m20s pod/namenode-01/1 Running3(14m ago)15m pod/nodemanager-01/1 Running08s pod/resourcemanager-01/1 Running076s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE service/datanode ClusterIP Nonenone9864/TCP,9866/TCP 20m service/namenode ClusterIP Nonenone9870/TCP,9000/TCP,8020/TCP 27m service/nodemanager ClusterIP Nonenone8042/TCP 8s service/resourcemanager ClusterIP Nonenone8088/TCP,8032/TCP 7m9s NAME READY AGE statefulset.apps/datanode1/1 20m statefulset.apps/namenode1/1 27m statefulset.apps/nodemanager1/1 8s statefulset.apps/resourcemanager1/1 76s rootmaster Mon Nov24[10:05:42]:/opt/bigdata2/hadoop/v3.1.1/v4# kubectl get pvc -n hadoopNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE hadoop-dn-data-datanode-0 Bound pvc-1ce19f52-729d-4d2e-8b4a-6c064b5901fe 20Gi RWOlocal26m hadoop-nn-data-namenode-0 Bound pvc-8d162dd9-c732-4337-acd2-01945a9ff929 10Gi RWOlocal33m