## 1 实验环境规划|主机名称|IP地址|角色|| ---- | ---- | ---- ||harbor|172.25.254.11|Ansible控制节点、Harbor私有镜像仓库||master|172.25.254.12|K8s控制平面(Master节点)||node1|172.25.254.13|K8s工作Node节点||node2|172.25.254.14|K8s工作Node节点|主机名解析/etc/hosts所有节点统一配置172.25.254.11 harbor reg.timinglee.org172.25.254.12 master172.25.254.13 node1172.25.254.14 node2## 2 部署原理分析 作业要求的个人理解说明K8s从1.24版本之后**正式移除docker作为内置容器运行时支持**K8s只遵循CRI容器运行时接口docker本身不实现CRI所以需要cri‑dockerd中间适配器让K8s可以调用docker。### 整套方案各组件分工1. **Harbor(172.25.254.11)**- 提供http本地yum源内网分发docker-ce安装包- HTTPS私有镜像仓库集群拉取镜像全部走内网harbor不需要外网。2. **Ansible自动化**代替人工逐台敲命令批量完成yum源、docker安装、内核参数、证书分发、cri‑dockerd部署、kubeadm/kubelet软件安装。对比二进制部署不需要手动每台机器写systemd文件对比原生kubeadm部署底层细节可控。3. **cri‑dockerd**桥梁组件接收kubelet的CRI请求转发给docker引擎。4. **kubeadm**官方集群初始化工具初始化控制平面生成join命令让node节点加入集群。 和其他部署方案对比- kubeadm默认containerd官方推荐- kubeadmcri‑dockerd保留docker适合老业务迁移- 二进制部署完全手动所有组件学习原理操作繁琐- 本实验方案Ansiblekubeadmcri‑dockerd兼顾自动化同时可以看清每一步配置。 K8s部署硬性前提必须全部满足否则集群启动失败1. 所有节点关闭swap交换分区2. 内核模块br_netfilter加载bridge‑nf iptables参数开启3. 主机名互相解析时间同步4. CRI socket路径配置正确kubeadm识别cri‑dockerd5. Harbor证书分发到所有节点系统信任私有仓库证书## 3 Harbor节点搭建本地Yum源与私有镜像仓库 全部操作在rootharbor执行### 3.1 制作docker本地yum源bashcat /etc/yum.repos.d/docker.repo EOF[docker]name dockerbaseurl https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/gpgcheck 0EOFdnf install httpd createrepo -ymkdir /var/www/html/docker/ -p#修改httpd监听端口4444sed -i s/^Listen 80/Listen 4444/ /etc/httpd/conf/httpd.confsystemctl enable --now httpd#下载docker rpm包到本地dnf install docker-ce --downloadonly --destdir /mnt/ -ymv /mnt/*.rpm /var/www/html/docker/createrepo -v /var/www/html/docker/#harbor本机使用本地http源cat /etc/yum.repos.d/docker.repo EOF[docker]name dockerbaseurl http://172.25.254.11:4444/dockergpgcheck 0EOF### 3.2 生成harbor HTTPS自签名证书bashmkdir /data/certs -popenssl req -newkey rsa:4096 \-nodes -sha256 -keyout /data/certs/timinglee.org.key \-addext subjectAltName DNS:reg.timinglee.org \-x509 -days 365 -out /data/certs/timinglee.org.crt交互填写Country Name (2 letter code) [XX]:CNState or Province Name (full name) []:ShannxiLocality Name (eg, city) [Default City]:XianOrganization Name (eg, company) [Default Company Ltd]:kubernetesOrganizational Unit Name (eg, section) []:harborCommon Name (eg, your name or your servers hostname) []:reg.timinglee.orgEmail Address []:admintiminglee.org把证书复制到ansible工作目录供ansible推送到所有节点bashcp /data/certs/timinglee.org.crt ~devops/ansible/### 3.3 部署Harbor离线仓库bashtar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/cd /opt/harbor/cp harbor.yml.tmpl harbor.ymlvim harbor.yml修改harbor.yml关键配置yamlhostname: reg.timinglee.orghttps:certificate: /data/certs/timinglee.org.crtprivate_key: /data/certs/timinglee.org.key执行安装bash./install.sh## 4 Ansible控制环境准备切换devops用户bashsu - devopsmkdir ansiblecd ansible### 4.1 inventory清单 inventoryini[all]harbor ansible_host172.25.254.11 ansible_userdevops ansible_ssh_private_key_file~/.ssh/id_ed25519master ansible_host172.25.254.12 ansible_userdevops ansible_ssh_private_key_file~/.ssh/id_ed25519node1 ansible_host172.25.254.13 ansible_userdevops ansible_ssh_private_key_file~/.ssh/id_ed25519node2 ansible_host172.25.254.14 ansible_userdevops ansible_ssh_private_key_file~/.ssh/id_ed25519[k8s_master]master[k8s_nodes]node1node2### 4.2 ansible.cfgini[defaults]inventory ./inventoryremote_user devopsask_pass falsebecome truebecome_method sudobecome_user rootinterpreter_python/usr/bin/python3[privilege_escalation]become_ask_pass false### 4.3 免密登录配置bashssh-keygen -t ed25519#全部回车不设置密码ssh-copy-id root172.25.254.12ssh-copy-id root172.25.254.13ssh-copy-id root172.25.254.14 再通过ansible在所有机器创建devops用户sudo免密bashansible all -m user -a namedevops statepresentansible all -m shell -a echo lee | passwd --stdin devopsansible all -m lineinfile -a path/etc/sudoers linedevops ALL(ALL) NOPASSWD: ALL insertafterEOF测试连通性bashansible all -m ping## 5 Ansible完整Playbook剧本 kubernetes-devops.ymlyaml---- name: 部署docker、cri-dockerd、k8s组件hosts: allgather_facts: truetasks:# K8s基础硬性环境- name: 关闭swap分区mount:name: swapstate: absentfstype: swap- name: fstab注释swaplineinfile:path: /etc/fstabregexp: ^.*swapstate: absent- name: 关闭firewalld测试环境service:name: firewalldstate: stoppedenabled: false- name: 配置/etc/hosts主机解析blockinfile:path: /etc/hostsblock: |172.25.254.11 harbor reg.timinglee.org172.25.254.12 master172.25.254.13 node1172.25.254.14 node2marker: # {mark} ANSIBLE MANAGED K8S HOSTS# docker本地yum源- name: setup docker repoyum_repository:name: dockerdescription: dockerbaseurl: http://172.25.254.11:4444/dockerfile: dockergpgcheck: no- name: install docker-cednf:name: docker-cestate: present- name: 修改docker.service增加--iptablestruereplace:path: /lib/systemd/system/docker.serviceregexp: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sockreplace: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue- name: docker daemon.json私有仓库配置copy:content: |{registry-mirrors:[https://reg.timinglee.org]}dest: /etc/docker/daemon.json# 内核模块与sysctl参数- name: 加载br_netfilter模块配置copy:dest: {{ item.dest }}content: {{ item.value }}loop:- dest: /etc/modules-load.d/docker_mod.confvalue: br_netfilter- dest: /etc/sysctl.d/docker.confvalue: |net.bridge.bridge-nf-call-iptables 1net.bridge.bridge-nf-call-ip6tables 1net.ipv4.ip_forward 1# 证书分发- name: 创建docker证书目录file:path: /etc/docker/certs.d/reg.timinglee.orgstate: directorymode: 0755- name: 推送harbor证书copy:src: {{ item.src }}dest: {{ item.dest }}loop:- src: ./timinglee.org.crtdest: /etc/docker/certs.d/reg.timinglee.org/ca.crt- src: ./timinglee.org.crtdest: /etc/pki/ca-trust/source/anchors/ca.crt- name: 更新系统CA信任 生效sysctlshell: update-ca-trust extract; sysctl --systemchanged_when: false# 部署 cri‑dockerd- name: 推送cri‑dockerd二进制与systemd文件copy:src: {{ item.src }}dest: {{ item.dest }}mode: {{ item.mode | default(0644) }}loop:- src: ./cri-dockerd/cri-dockerddest: /usr/bin/cri-dockerdmode: 0755- src: ./cri-dockerd/cri-docker.servicedest: /lib/systemd/system/cri-docker.service- src: ./cri-dockerd/cri-docker.socketdest: /lib/systemd/system/cri-docker.socket- name: systemd重载systemd:daemon_reload: yes- name: 启动docker cri‑dockerd相关服务service:name: {{ item }}state: restartedenabled: yesloop:- docker- systemd-modules-load.service- cri-docker.socket- cri-docker# 部署k8s yum源- name: setup k8s repoyum_repository:name: kubernetesdescription: kubernetesbaseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpmfile: kubernetesgpgcheck: no# master节点完整三件套- name: master节点安装k8s软件dnf:name:- kubeadm- kubelet- kubectlstate: presentwhen: inventory_hostname master# work节点安装kubeadm kubelet- name: work节点安装k8s软件dnf:name:- kubeadm- kubeletstate: presentwhen: inventory_hostname in [node1,node2]# bash命令补全- name: root bash补全配置lineinfile:path: /root/.bashrcline: |source (kubectl completion bash)source (kubeadm completion bash)when: inventory_hostname master- name: devops bash补全配置lineinfile:path: /home/devops/.bashrcline: |source (kubectl completion bash)source (kubeadm completion bash)create: yeswhen: inventory_hostname master- name: kubelet开机自启service:name: kubeletstate: startedenabled: yes## 6 执行Ansible自动化部署bash#测试连通ansible all -m ping#执行剧本ansible-playbook kubernetes-devops.yml## 7 Kubeadm初始化K8s集群登录master节点 172.25.254.12 ⚠️重点使用cri‑dockerd必须指定--cri-socket否则kubeadm会去找containerdbashkubeadm init \--apiserver-advertise-address172.25.254.12 \--image-repository reg.timinglee.org/k8s \--cri-socketunix:///var/run/cri-dockerd.sock \--pod-network-cidr10.244.0.0/16初始化完成后复制kubeconfigbashmkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config安装flannel CNI网络插件bashkubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml拿到kubeadm join命令在node1 node2执行加入集群。查看集群状态bashkubectl get nodeskubectl get pods -A8 实操踩坑总结1. swap没有关闭kubelet直接crash启动失败必须fstab也注释swap不能只临时swapoff。2. cri‑socket参数遗忘使用cri‑dockerd做CRIkubeadm init不加--cri-socketunix:///var/run/cri‑dockerd.sock会默认找containerd初始化直接报错。3. harbor证书问题仅仅docker目录放证书不够还要导入系统ca信任否则节点无法拉取harbor镜像。4. cri‑docker.socket没有启动**只启动cri‑docker.service会报错socket必须enable启动。5. 主机名解析问题集群节点之间必须互相解析主机名否则etcd、apiserver组件异常。6. ansible剧本执行报错优先检查文件路径、权限拷贝二进制文件mode设置755可执行权限。