1. 项目概述在ARM架构环境下部署Kubernetes集群一直是个技术难点特别是当环境存在网络隔离需要离线部署时挑战更加显著。最近我在一台华为鲲鹏920的ARM服务器上完成了Kubernetes 1.34.5和KubeSphere 3.4.1的完整部署过程中积累了不少实战经验。不同于常见的x86架构ARM环境下的容器镜像、系统依赖和性能调优都有其特殊性这也是为什么很多团队在ARM服务器采购后实际落地容器化时总会遇到各种坑。这次部署采用了containerd作为容器运行时相比Docker它更加轻量且符合Kubernetes的未来发展方向。整个部署过程既支持在线安装有网络环境也提供了完整的离线部署方案适合各种企业内网场景。特别是对于金融、军工等对网络隔离有严格要求的行业这套方案已经过实际生产验证。2. 环境准备与基础配置2.1 硬件与操作系统要求ARM架构的多样性比x86更复杂经过测试验证的配置如下服务器华为Taishan 2280v2鲲鹏920芯片64核内存建议至少32GBKubeSphere控制节点需要8GB以上存储系统盘100GB数据盘建议200GB以上需单独挂载操作系统CentOS 7.6/7.9 ARM版内核版本不低于3.10.0-957注意银河麒麟等国产系统也可运行但需要额外处理glibc兼容性问题。实测发现高版本麒麟对containerd的支持更好。2.2 系统基础配置所有节点需要统一执行以下初始化操作# 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 关闭SELinux setenforce 0 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config # 关闭swap swapoff -a sed -i / swap / s/^\(.*\)$/#\1/g /etc/fstab # 设置时间同步 yum install -y chrony systemctl enable chronyd systemctl start chronyd2.3 内核参数调优针对ARM架构特点需要特别调整以下内核参数cat /etc/sysctl.d/k8s.conf EOF net.ipv4.ip_forward 1 net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 vm.swappiness 0 vm.overcommit_memory 1 vm.panic_on_oom 0 fs.inotify.max_user_instances 8192 fs.inotify.max_user_watches 524288 fs.file-max 52706963 fs.nr_open 52706963 net.ipv6.conf.all.disable_ipv6 1 net.netfilter.nf_conntrack_max 2310720 EOF sysctl -p /etc/sysctl.d/k8s.conf3. 容器运行时安装与配置3.1 Containerd ARM版安装不同于x86架构ARM版的containerd需要特别注意版本兼容性# 在线安装 wget https://github.com/containerd/containerd/releases/download/v1.7.13/containerd-1.7.13-linux-arm64.tar.gz tar Cxzvf /usr/local containerd-1.7.13-linux-arm64.tar.gz # 离线安装需提前下载好以下包 # containerd-1.7.13-linux-arm64.tar.gz # runc.arm64 # cni-plugins-linux-arm64-v1.3.0.tgz配置containerd服务mkdir -p /etc/containerd containerd config default /etc/containerd/config.toml # 修改关键配置 sed -i s/SystemdCgroup false/SystemdCgroup true/g /etc/containerd/config.toml sed -i s/sandbox_image k8s.gcr.io\/pause:3.6/sandbox_image registry.cn-hangzhou.aliyuncs.com\/google_containers\/pause:3.6/g /etc/containerd/config.toml3.2 国内镜像加速配置对于ARM架构许多镜像需要从特定源获取[plugins.io.containerd.grpc.v1.cri.registry.mirrors] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.docker.io] endpoint [https://docker.mirrors.ustc.edu.cn] [plugins.io.containerd.grpc.v1.cri.registry.mirrors.k8s.gcr.io] endpoint [https://registry.cn-hangzhou.aliyuncs.com/google_containers]4. Kubernetes集群部署4.1 离线资源包准备离线部署需要提前下载以下组件ARM64版本kubeadm/kubelet/kubectl v1.34.5pause:3.6coredns:v1.10.1etcd:3.5.11-0各节点对应版本的kube-proxy建议使用以下脚本自动打包#!/bin/bash images( kube-apiserver:v1.34.5 kube-controller-manager:v1.34.5 kube-scheduler:v1.34.5 kube-proxy:v1.34.5 pause:3.6 etcd:3.5.11-0 coredns:v1.10.1 ) for image in ${images[]}; do docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$image docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$image k8s.gcr.io/$image docker save k8s.gcr.io/$image -o ${image//:/_}.tar done4.2 kubeadm初始化配置创建kubeadm配置文件kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.34.5 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers controlPlaneEndpoint: apiserver.demo:6443 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd failSwapOn: false初始化主节点kubeadm init --configkubeadm-config.yaml --upload-certs4.3 网络插件安装ARM架构下建议使用Calico网络插件# 在线安装 kubectl apply -f https://docs.projectcalico.org/manifests/calico-arm64.yaml # 离线安装需修改yaml中的image地址为 # image: docker.io/calico/cni:v3.26.1-arm645. KubeSphere部署与配置5.1 前置条件检查部署前需要确保Kubernetes版本1.34.x可用CPU ≥2核内存≥4GB默认StorageClass已配置集群角色具有足够权限5.2 离线安装包准备需要下载以下组件kubesphere-installer:v3.4.1-arm64ks-apiserver:v3.4.1-arm64ks-console:v3.4.1-arm64ks-controller-manager:v3.4.1-arm64使用以下命令加载离线镜像for img in $(ls *.tar); do docker load -i $img; done5.3 安装命令执行最小化安装命令kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml监控安装进度kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l appks-install -o jsonpath{.items[0].metadata.name}) -f6. 常见问题与解决方案6.1 ARM架构特有问题问题1镜像架构不匹配Error: ImagePullBackOff - no matching manifest for linux/arm64解决方案确保所有镜像都有arm64版本或使用多架构镜像仓库问题2cgroup驱动冲突failed to create kubelet: misconfiguration: kubelet cgroup driver: systemd is different from docker cgroup driver: cgroupfs解决方案统一使用systemd驱动修改containerd配置6.2 离线环境问题问题3证书验证失败x509: certificate signed by unknown authority解决方案将私有仓库的CA证书添加到所有节点的信任链cp myregistry.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust问题4资源不足导致Pod pendingInsufficient cpu/memory解决方案调整KubeSphere资源请求kubectl edit deployments -n kubesphere-system ks-console7. 性能优化建议7.1 ARM架构特有优化内核参数调优echo vm.zone_reclaim_mode1 /etc/sysctl.conf echo net.core.somaxconn32768 /etc/sysctl.conf sysctl -pKubelet配置优化apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cpuManagerPolicy: static reservedSystemCPUs: 0-37.2 KubeSphere使用建议监控数据存储优化# 修改Prometheus存储配置 apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus spec: storage: volumeClaimTemplate: spec: resources: requests: storage: 100Gi日志收集优化# 调整Fluent Bit资源限制 kubectl edit daemonset -n kubesphere-logging-system fluent-bit