k8s部署kubeadm方式部署环境及版本系统版本CentOS Linux release7.9.2009 k8s版本v1.28.15 docker版本26.1.4 containerd版本1.6.33 calico版本v3.25.0准备主机ip主机名角色配置192.168.1.131vm131master2核4G50G192.168.1.132vm132node12核4G50G192.168.1.133vm133node22核4G50G1、修改主机名#masterhostnamectl set-hostname vm131#node1hostnamectl set-hostname vm132#node2hostnamectl set-hostname vm133#修改hosts文件vim/etc/hosts192.168.1.131 vm131192.168.1.132 vm132192.168.1.133 vm1332、关闭自带防火墙# 关闭SELinuxsetenforce0sed-is/SELINUXenforcing/SELINUXdisabled/g/etc/selinux/config# 关闭Firewalld并禁止自启动systemctl stop firewalld systemctl disable firewalld# 设置iptables规则iptables-Fiptables-Xiptables-F-tnatiptables-X-tnatiptables-PFORWARD ACCEPT# 设置iptables解决iptables而导致流量无法正确路由的问题catEOF/etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_nonlocal_bind 1 net.ipv4.ip_forward 1 vm.swappiness 0 vm.overcommit_memory 1 EOFsysctl--system# 关闭swapswapoff-afree–hsed-is/dev/mapper/centos-swap#/dev/mapper/centos-swapg/etc/fstab3、时间同步yum-yinstallntp systemctlenablentpd systemctl start ntpd timedatectl set-timezone Asia/Shanghai ntpdate-utime.nist.govdate#如果遇到yum源的问题可以通过以下方式解决curl-o/etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all#清除缓存yum makecache#生成缓存一、安装docker#安装编译器环境yum-yinstallgcc gcc-c#卸载之前下载的docker防止和之前安装过的docker出现冲突yum removedocker\docker-client\docker-client-latest\docker-common\docker-latest\docker-latest-logrotate\docker-logrotate\docker-engine#安装yum-utilsyuminstall-yyum-utils#将docker-ce.repo添加到/etc/yum.repos.d/yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo#提升yum安装的速度yum makecache fast#安装dockeryuminstalldocker-ce docker-ce-cli containerd.io-y#启动dockersystemctl startdocker#开机自启动systemctlenabledocker二、安装containerdyuminstallcontainerd-y#配置/etc/containerd/config.toml#如果你不打算使用跟踪功能可以在 containerd 配置文件中禁用该插件。vim/etc/containerd/config.toml#disabled_plugins [cri][plugins.io.containerd.grpc.v1.cri]disablefalse#配置完成后重启 containerdsystemctl restart containerd#如果你需要启用 tracing可以配置跟踪端点。在 containerd 的配置文件中提供正确的 tracing 端点配置#[plugins.io.containerd.tracing.processor.v1.otlp]# endpoint your-tracing-endpoint:port#根据你所使用的 tracing 服务如 OpenTelemetry Collector来提供适当的端点信息。#配置完成后重启 containerd#systemctl restart containerd三、Master节点安装kubeadm1、安装kubelet 和kubeadm以及kubectlcatEOF/etc/yum.repos.d/kubernetes.repo[kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck0 repo_gpgcheck0 excludekubelet kubeadm kubectl cri-tools kubernetes-cni EOFyum clean all yum makecacheyuminstall-ykubelet kubeadm kubectl--disableexcludeskubernetes systemctlenable--nowkubelet2、下载所需要的镜像#查看需要下载的镜像kubeadm config images list#查看结果I030611:27:12.88672121939version.go:256]remote version is much newer: v1.32.2;falling back to: stable-1.28 registry.k8s.io/kube-apiserver:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15 registry.k8s.io/pause:3.9 registry.k8s.io/etcd:3.5.9-0 registry.k8s.io/coredns/coredns:v1.10.1#下载镜像注意namespace是k8s.ioctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6#重新打标签ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 registry.k8s.io/kube-apiserver:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 registry.k8s.io/pause:3.9 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 registry.k8s.io/etcd:3.5.9-0 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 registry.k8s.io/coredns/coredns:v1.10.1#删除aliyun镜像foriteminctr-nk8s.io images list|awk$1 ~ /registry.cn-hangzhou.aliyuncs.com/ {print $1};doctr-nk8s.io imagesrm${item};done3、更改kubelet参数vim/etc/sysconfig/kubelet#改为如下参数KUBELET_EXTRA_ARGS--cgroup-driversystemd4、kubeadm初始化kubeadm init#出现Your Kubernetes control-plane has initialized successfully!说明初始化成功#根据提示执行如下命令mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/config四、Node节点安装kubeadm1、安装kubeadm kubeletyum-yinstallkubeadm kubelet systemctlenablekubelet.service2、下载所需要的镜像#查看需要下载的镜像kubeadm config images list#查看结果I030611:27:12.88672121939version.go:256]remote version is much newer: v1.32.2;falling back to: stable-1.28 registry.k8s.io/kube-apiserver:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15 registry.k8s.io/pause:3.9 registry.k8s.io/etcd:3.5.9-0 registry.k8s.io/coredns/coredns:v1.10.1#下载镜像注意namespace是k8s.ioctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 ctr-nk8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6#重新打标签ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.15 registry.k8s.io/kube-apiserver:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.15 registry.k8s.io/kube-controller-manager:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.15 registry.k8s.io/kube-scheduler:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.15 registry.k8s.io/kube-proxy:v1.28.15 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 registry.k8s.io/pause:3.9 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0 registry.k8s.io/etcd:3.5.9-0 ctr-nk8s.io images tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 registry.k8s.io/coredns/coredns:v1.10.1#删除aliyun镜像foriteminctr-nk8s.io images list|awk$1 ~ /registry.cn-hangzhou.aliyuncs.com/ {print $1};doctr-nk8s.io imagesrm${item};done3、更改kubelet参数vim/etc/sysconfig/kubelet#改为如下参数KUBELET_EXTRA_ARGS--cgroup-driversystemd4、加入Master#token来自master节点执行kubeinit的结果kubeadmjoin192.168.1.131:6443--token51lpgy.tue7o5rnwi3h62ql\--discovery-token-ca-cert-hash sha256:200ad4e8f13649b3cd14db97cdaf842d97f4dbbca0cbec123c06b2a3c687ede1五、安装网络插件1、在Master节点拉取并修改calico.yml文件#下载calico的yaml文件Master节点操作yuminstallwget-ymkdir-pv/data/yamlcd/data/yamlwgethttps://docs.tigera.io/archive/v3.25/manifests/calico.yaml#修改calico的yaml文件- name: CALICO_IPV4POOL_CIDR value:10.244.0.0/16#在CLUSTER_TYPE那行下面添加interface根据自己的主机网卡名修改- name: IP_AUTODETECTION_METHOD value:interfaceens1922、下载所需要的镜像Master和Node都要操作mkdir-pv/data/tarcd/data/tar#需要先用docker下载用ctr测试下载不下来dockerpull calico/cni:v3.25.0dockerpull calico/node:v3.25.0dockerpull calico/kube-controllers:v3.25.0#导出docker镜像dockersave-ocni.tar calico/cni:v3.25.0dockersave-onode.tar calico/node:v3.25.0dockersave-okube-controllers.tar calico/kube-controllers:v3.25.0#用ctr导入镜像namespace是k8s.io前面两步可以在一台服务器上操作最后这步都要操作ctr-nk8s.io imagesimportcni.tar ctr-nk8s.io imagesimportnode.tar ctr-nk8s.io imagesimportkube-controllers.tar#验证是否导入成功ctr-nk8s.io images list|grepcalico3、安装calico插件#只在Master节点操作即可cd/data/yaml kubectl apply-fcalico.yaml[rootvm131 ~]# kubectl get pods -ANAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-658d97c59c-fc46q1/1 Running046m kube-system calico-node-4cc9q1/1 Running046m kube-system calico-node-l6ch21/1 Running046m kube-system calico-node-r27jj1/1 Running046m kube-system coredns-5dd5756b68-84cww0/1 Running04h kube-system coredns-5dd5756b68-nd9v80/1 Running04h kube-system etcd-vm1311/1 Running04h1m kube-system kube-apiserver-vm1311/1 Running04h1m kube-system kube-controller-manager-vm1311/1 Running1(3h9m ago)4h1m kube-system kube-proxy-26qwm1/1 Running03h44m kube-system kube-proxy-bj5xg1/1 Running04h kube-system kube-proxy-wgfnz1/1 Running03h43m kube-system kube-scheduler-vm1311/1 Running04h1m#出现上面情况可以尝试重启coreDNS#plugin/kubernetes: Kubernetes API connection failure: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeoutkubectl rollout restart deployment coredns-nkube-system六、验证[rootvm131 ~]# kubectl get pods -ANAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-658d97c59c-t678p1/1 Running05m26s kube-system calico-node-5zj2n1/1 Running05m26s kube-system calico-node-hjdcm1/1 Running05m26s kube-system calico-node-hrwh41/1 Running05m26s kube-system coredns-967d5bb69-sb9xx1/1 Running031m kube-system coredns-967d5bb69-sngxg1/1 Running031m kube-system etcd-vm1311/1 Running04h34m kube-system kube-apiserver-vm1311/1 Running04h34m kube-system kube-controller-manager-vm1311/1 Running1(3h43m ago)4h34m kube-system kube-proxy-26qwm1/1 Running04h17m kube-system kube-proxy-bj5xg1/1 Running04h34m kube-system kube-proxy-wgfnz1/1 Running04h16m kube-system kube-scheduler-vm1311/1 Running04h34m[rootvm131 ~]# kubectl get nodeNAME STATUS ROLES AGE VERSION vm131 Ready control-plane 4h34m v1.28.2 vm132 Readynone4h16m v1.28.2 vm133 Readynone4h17m v1.28.2七、问题记录crictl images 报错 WARN[0000]image connect using default endpoints:[unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you shouldsetthe endpoint instead. E030516:42:46.92225222812remote_image.go:119]ListImages with filter from image service failederrrpc error: code Unavailable desc connection error: desc \transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\filterImageFilter{Image:ImageSpec{Image:,Annotations:map[string]string{},},}FATA[0000]listing images: rpc error: codeUnavailable descconnection error: desctransport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory解决exportCONTAINER_RUNTIME_ENDPOINTunix:///run/containerd/containerd.sockexportIMAGE_SERVICE_ENDPOINTunix:///run/containerd/containerd.sock 配置 crictl 默认连接 containerd 编辑 crictl 配置文件mkdir-p/etc/crictlechoruntime-endpoint: unix:///run/containerd/containerd.sock/etc/crictl.yamlvim/etc/containerd/config.toml#disabled_plugins [cri][plugins.io.containerd.grpc.v1.cri]disablefalse禁用跟踪插件 如果你不打算使用跟踪功能可以在 containerd 配置文件中禁用该插件。在 /etc/containerd/config.toml 文件中查找并禁用跟踪插件[plugins.io.containerd.tracing.processor.v1.otlp]enabledfalse然后重启 containerdsudosystemctl restart containerd 配置 tracing endpoint 如果你需要启用 tracing可以配置跟踪端点。在 containerd 的配置文件中提供正确的 tracing 端点配置[plugins.io.containerd.tracing.processor.v1.otlp]endpointyour-tracing-endpoint:port根据你所使用的 tracing 服务如 OpenTelemetry Collector来提供适当的端点信息。 配置完成后重启 containerdsudosystemctl restart containerd 总结 如果不使用跟踪功能可以简单地忽略这个信息或禁用该插件。 如果需要启用跟踪功能确保配置了正确的 tracing endpoint