部署环境前先做准备工作一.构建harbor镜像仓库1.部署docker本地仓库[rootharbor ~]# cat /etc/yum.repos.d/docker.repo EOF [docker] name docker baseurl https://mirrors.aliyun.com/docker-ce/linux/rhel/9.6/x86_64/stable/ gpgcheck 0 EOF [rootharbor ~]# dnf install httpd createrepo-y[rootharbor ~]# mkdir /var/www/html/docker/ -p [rootharbor ~]# vim /etc/httpd/conf/httpd.conf Listen 4444 [rootharbor ~]# systemctl enable --now httpd [rootharbor ~]# dnf install docker-ce --downloadonly --destdir /mnt/ -y[rootharbor ~]# mv /mnt/*.rpm /var/www/html/docker/ [rootharbor ~]# createrepo -v /var/www/html/docker/ [rootharbor ~]# cat /etc/yum.repos.d/docker.repo EOF [docker] name docker baseurl http://192.168.106.180:4444/docker gpgcheck 0 EOF [rootharbor ~]# dnf install docker-ce -y2.安装docker在harbor仓库节点[rootharbor ~]# dnf install docker-ce -y [rootharbor ~]# echo br_netfilter /etc/modules-load.d/docker_mod.conf [rootharbor ~]# modprobe -a br_netfilter [rootharbor ~]# vim /etc/sysctl.d/docker.conf net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 [rootharbor ~]# systemctl restart systemd-modules-load.service [rootharbor ~]# sysctl --system[rootharbor ~]# vim /lib/systemd/system/docker.service ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue [rootharbor ~]# systemctl daemon-reload [rootharbor ~]# systemctl enable --now docker3.生成key[rootharbor ~]# mkdir /data/certs -p [rootharbor ~]# openssl req -newkey rsa:4096 \ -nodes -sha256 -keyout /data/certs/timinglee.org.key \ -addext subjectAltName DNS:reg.timinglee.org \ -x509 -days 365 -out /data/certs/timinglee.org.crt You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ., the field will be left blank. ----- Country Name (2 letter code) [XX]:CN State or Province Name (full name) []:Shannxi Locality Name (eg, city) [Default City]:Xian Organization Name (eg, company) [Default Company Ltd]:kubernetes Organizational Unit Name (eg, section) []:harbor Common Name (eg, your name or your servers hostname) []:reg.timinglee.org Email Address []:admintiminglee.org4.编辑harbor配置文件[rootharbor ~]# tar zxf harbor-offline-installer-v2.5.4.tgz -C /opt/ [rootharbor ~]# cd /opt/harbor/ [rootharbor harbor]# ls common.sh harbor.v2.5.4.tar.gz harbor.yml.tmpl install.sh LICENSE prepare [rootharbor harbor]# cp harbor.yml.tmpl harbor.yml [rootharbor harbor]# vim harbor.yml hostname: reg.timinglee.org certificate: /data/certs/timinglee.org.crt private_key: /data/certs/timinglee.org.key harbor_admin_password: 123 [rootharbor harbor]# ./install.sh --with-chartmuseum#编写启动脚本 [rootharbor ~]# vim /lib/systemd/system/harbor.service [Unit] Descriptionharbor with Docker Compose Documentationhttps://reg.timinglee.com/compose/ Afterdocker.service network-online.target Requiresdocker.service [Service] Typeoneshot WorkingDirectory/opt/harbor ExecStart/usr/bin/docker compose up -d ExecStop/usr/bin/docker compose down RemainAfterExityes [Install] WantedBymulti-user.target [rootharbor harbor]# docker compose down WARN[0000] /opt/harbor/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion [] down 4/6 ⠇ Container registryctl Stopping 1.9s ✔ Container nginx Removed 0.2s ✔ Container chartmuseum Removed 0.2s ✔ Container harbor-jobservice Removed 0.1s ✔ Container harbor-portal Removed 0.1s ⠦ Container harbor-core Stopping [rootharbor ~]# systemctl enable --now harbor Created symlink /etc/systemd/system/multi-user.target.wants/harbor.service → /usr/lib/systemd/system/harbor.service. [rootharbor harbor]# docker compose ps5.启动并验证[rootharbor harbor]# mkdir /etc/docker/certs.d/reg.timinglee.org/ -p [rootharbor harbor]# cp /data/certs/timinglee.org.crt /etc/docker/certs.d/reg.timinglee.org/ca.crt [rootharbor harbor]# vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.106.180 harbor reg.timinglee.org [rootharbor harbor]# systemctl restart docker [rootharbor harbor]# docker compose up -d [rootharbor harbor]# docker login reg.timinglee.org -u admin Password: WARNING! Your credentials are stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded开始部署二.构建部署kubernetes所需主机主机名ip配置角色harbor192.168.106.180cpu1、mem 1Gharbor仓库master192.168.106.181cpu4、mem4Gmasterk8s集群控制节点node1192.168.106.182cpu2、mem 2Gworkerk8s集群工作节点node2192.168.106.183cpu2、mem 2Gworkerk8s集群工作节点1.所有主机配置1.1关闭swapsystemctl disable --now swap.target systemctl mask swap.target sed /swap/s/^/#/g -i /etc/fstab1.2所有主机彼此建立解析127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.106.181 master 192.168.106.182 node1 192.168.106.183 node2 192.168.106.180 reg.timinglee.org三.利用Ansible部署kubernetes运行环境1.安装ansible[rootharbor ~]# vim /etc/yum.repos.d/epel.repo [epel] name epel baseurl https://mirrors.aliyun.com/epel/9/Everything/x86_64/ gpgcheck 0 [rootharbor ~]# dnf install ansible-core.x86_64 ansible.noarch -y2.配置所需用户[rootharbor ~]# useradd devops [rootharbor ~]# echo 123 | passwd --stdin devops 更改用户 devops 的密码 。 passwd所有的身份验证令牌已经成功更新 [rootharbor ~]# su - devops [devopsharbor ~]$ mkdir ansible [devopsharbor ~]$ cd ansible/ [devopsharbor ansible]$3.生成ansible的临时配置文件并建立运行环境[devopsharbor ansible]$ vim inventory [servers] 192.168.106.181 192.168.106.182 192.168.106.183 [devopsharbor ansible]$ vim ansible.cfg [defaults] inventory /home/devops/ansible/inventory remote_user root ask_pass false host_key_checking false[devopsharbor ansible]$ ansible all -m user -a namedevops statepresent 192.168.106.182 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, changed: true, comment: , create_home: true, group: 1000, home: /home/devops, name: devops, shell: /bin/bash, state: present, system: false, uid: 1000 } 192.168.106.183 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, changed: true, comment: , create_home: true, group: 1000, home: /home/devops, name: devops, shell: /bin/bash, state: present, system: false, uid: 1000 } 192.168.106.181 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, changed: true, comment: , create_home: true, group: 1000, home: /home/devops, name: devops, shell: /bin/bash, state: present, system: false, uid: 1000 } [devopsharbor ansible]$ ansible all -m shell -a echo 123 | passwd --stdin devops 192.168.106.183 | CHANGED | rc0 更改用户 devops 的密码 。 passwd所有的身份验证令牌已经成功更新。 192.168.106.182 | CHANGED | rc0 更改用户 devops 的密码 。 passwd所有的身份验证令牌已经成功更新。 192.168.106.181 | CHANGED | rc0 更改用户 devops 的密码 。 passwd所有的身份验证令牌已经成功更新。 [devopsharbor ansible]$ ansible all -m lineinfile -a path/etc/sudoers linedevops ALL(ALL) NOPASSWD: ALL insertafterEOF 192.168.106.183 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, backup: , changed: true, msg: line added } 192.168.106.182 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, backup: , changed: true, msg: line added } 192.168.106.181 | CHANGED { ansible_facts: { discovered_interpreter_python: /usr/bin/python3 }, backup: , changed: true, msg: line added }4.生成ansible的主配置文件并测试[devopsharbor ansible]$ vim ansible.cfg [devopsharbor ansible]$ ansible all -m shell -a whoami 192.168.106.182 | CHANGED | rc0 root 192.168.106.183 | CHANGED | rc0 root 192.168.106.181 | CHANGED | rc0 root5.利用ansible剧本部署docker[devopsharbor ansible]$ vim kubernetes-devops.yml - name: devops k8s from ansible hosts: all tasks: - name: setup docker repo yum_repository: name: docker description: docker baseurl: http://192.168.106.180:4444/docker file: docker gpgcheck: no - name: install docker dnf: name: docker-ce state: present - name: setup docker.service replace: path: /lib/systemd/system/docker.service regexp: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock replace: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue - name: setup docker registry copy: content: | { registry-mirrors:[https://reg.timinglee.org] } dest: /etc/docker/daemon.json - name: setup docker certs file: path: /etc/docker/certs.d/reg.timinglee.org state: directory - name: cp certs file copy: src: {{item.src}} dest: {{item.dest}} loop: - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt dest: /etc/pki/ca-trust/source/anchors/ca.crt - name: load module copy: dest: {{item.dest}} content: {{item.value}} loop: - dest: /etc/modules-load.d/docker_mod.conf value: br_netfilter - dest: /etc/sysctl.d/docker.conf value: | net.bridge.bridge-nf-call-iptables 1 net.bridge.bridge-nf-call-ip6tables 1 net.ipv4.ip_forward 1 - name: update ca cert shell: update-ca-trust extract;sysctl --system - name: start services service: name: {{item}} state: restarted enabled: yes loop: - docker - systemd-modules-load.service [devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml6.利用ansible安装cri-dockerd[devopsharbor ansible]$ tar zxf cri-dockerd-0.4.4.amd64.tgz [devopsharbor ansible]$ ls ansible.cfg cri-dockerd cri-dockerd-0.4.4.amd64.tgz inventory kubernetes-devops.yml [devopsharbor ansible]$ cd cri-dockerd/ [devopsharbor cri-dockerd]$ ls cri-dockerd cri-docker.service cri-docker.socket [devopsharbor cri-dockerd]$ vim cri-docker.service [Service] Typenotify ExecStart/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugincni --pod-infra-container-imagereg.timinglee.org/k8s/pause:3.10.1 ExecReload/bin/kill -s HUP $MAINPID TimeoutSec0 RestartSec2 Restartalways[devopsharbor ansible]$ vim kubernetes-devops.yml - name: devops k8s from ansible hosts: all tasks: - name: setup docker repo yum_repository: name: docker description: docker baseurl: http://192.168.106.180:4444/docker file: docker gpgcheck: no - name: install docker dnf: name: docker-ce state: present - name: setup docker.service replace: path: /lib/systemd/system/docker.service regexp: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock replace: ExecStart/usr/bin/dockerd -H fd:// --containerd/run/containerd/containerd.sock --iptablestrue - name: setup docker registry copy: content: | { registry-mirrors:[https://reg.timinglee.org] } dest: /etc/docker/daemon.json - name: setup docker certs file: path: /etc/docker/certs.d/reg.timinglee.org state: directory - name: cp certs file copy: src: {{item.src}} dest: {{item.dest}} loop: - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt dest: /etc/docker/certs.d/reg.timinglee.org/ca.crt - src: /etc/docker/certs.d/reg.timinglee.org/ca.crt dest: /etc/pki/ca-trust/source/anchors/ca.crt - name: stup cri-dockerd copy: src: {{item.src}} dest: {{item.dest}} mode: 0755 loop: - src: ./cri-dockerd/cri-dockerd dest: /usr/bin/cri-dockerd - src: ./cri-dockerd/cri-docker.service dest: /lib/systemd/system/cri-docker.service - src: ./cri-dockerd/cri-docker.socket dest: /lib/systemd/system/cri-docker.socket - name: update ca cert shell: update-ca-trust extract;sysctl --system;systemctl daemon-reload - name: start services service: name: {{item}} state: restarted enabled: yes loop: - docker - systemd-modules-load.service - cri-docker [devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml7.利用ansible部署k8s软件在原来的基础上添加 [devopsharbor ansible]$ vim kubernetes-devops.yml - name: install kubernetes from ansible hosts: all tasks: - name: setup k8s repo yum_repository: name: kubernetes description: kubernetes baseurl: https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.35/rpm file: kubernetes gpgcheck: no - name: install software in master dnf: name: {{item}} state: present loop: - kubeadm-1.35.7-150500.1.1 - kubelet-1.35.7-150500.1.1 - kubectl-1.35.7-150500.1.1 when: inventory_hostname 192.168.106.181 - name: install software in work node dnf: name: {{item}} state: present loop: - kubeadm-1.35.7-150500.1.1 - kubelet-1.35.7-150500.1.1 when: inventory_hostname 192.168.106.182 or inventory_hostname 192.168.106.183 - name: setup kubectl kubeadm complication lineinfile: path: /root/.bashrc line: | source (kubectl completion bash) source (kubeadm completion bash) when: inventory_hostname 192.168.106.181 - name: start kubelet service: name: kubelet state: started enabled: yes [devopsharbor ansible]$ ansible-playbook kubernetes-devops.yml#让补齐功能生效 [rootk8s-master mnt]# source ~/.bashrc [rootk8s-master mnt]# kubectl 命令后按2此tab键 alpha (Commands for features in alpha) annotate (更新一个资源的注解) api-resources (Print the supported API resources on the server) api-versions (Print the supported API versions on the server, in the form of group/version) apply (Apply a configuration to a resource by file name or stdin) attach (挂接到一个运行中的容器) auth (Inspect authorization) autoscale (Auto-scale a deployment, replica set, stateful set, or replication controller) certificate (Modify certificate resources) [rootk8s-master mnt]# kubeadm certs config init kubeconfig token version completion help join reset upgrade [rootk8s-master mnt]# systemctl status kubelet.service ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: activating (auto-restart) (Result: exit-code) since Tue 2026-08-18 17:09:25 CST; 1s ago Docs: https://kubernetes.io/docs/ Process: 79433 ExecStart/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ Main PID: 79433 (codeexited, status1/FAILURE)四.安装构建kubernetes 集群所需软件1.master节点dnf install kubelet kubeadm kubectl -y systemctl enable --now kubelet.service2.node节点dnf install kubelet kubeadm -y systemctl enable --now kubelet.service3.master节点中 kubectl 和kubeadm 补齐[rootmaster ~]# echo source (kubectl completion bash) ~/.bashrc [rootmaster ~]# echo source (kubeadm completion bash) ~/.bashrc [rootmaster ~]# source ~/.bashrc4.下载kubernetes集群所需镜像1.1查看镜像[rootmaster ~]# kubeadm config images list I0823 00:52:36.984061 67769 version.go:260] remote version is much newer: v1.36.4; falling back to: stable-1.35 registry.k8s.io/kube-apiserver:v1.35.8 registry.k8s.io/kube-controller-manager:v1.35.8 registry.k8s.io/kube-scheduler:v1.35.8 registry.k8s.io/kube-proxy:v1.35.8 registry.k8s.io/coredns/coredns:v1.13.1 registry.k8s.io/pause:3.10.1 registry.k8s.io/etcd:3.6.6-01.2下载镜像[rootmaster ~]# kubeadm config images pull \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.35.8 \ --cri-socketunix:///var/run/cri-dockerd.sock [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.35.8 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.35.8 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.35.8 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.35.8 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.13.1 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.10.1 [config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.6.6-01.3上传镜像到本地harbor[rootmaster ~]# docker login reg.timinglee.org -u admin Password: WARNING! Your credentials are stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded [rootmaster ~]# docker images --format {{.Repository}}:{{.Tag}} | awk -F / /google/{system(docker tag $0 reg.timinglee.org/k8s/$3)} [rootmaster ~]# docker images --format {{.Repository}}:{{.Tag}} | awk -F / /timinglee/{system(docker push $0)}五.在master中初始化kubernetes集群1.在master中完成集群初始化kubeadm init --pod-network-cidr10.244.0.0/16 --image-repository reg.timinglee.org/k8s --kubernetes-version v1.35.8 --cri-socketunix:///var/run/cri-dockerd.sock#每个人不一样其他主机加入本集群的凭证kubeadm join 192.168.106.181:6443 --token en3dzr.20ytpl4eqkrquovx \--discovery-token-ca-cert-hash sha256:3603dd7bbfbe057278a66a24cdb0bc66ad74a950ea01c44d5fc2b209330f85f8#如果忘记kubeadm join 192.168.106.181:6443 --token en3dzr.20ytpl4eqkrquovx \--discovery-token-ca-cert-hash sha256:3603dd7bbfbe057278a66a24cdb0bc66ad74a950ea01c44d5fc2b209330f85f8#如果初始化出问题[rootmaster ~]# kubeadm reset --cri-socketunix:///var/run/cri-dockerd.sock #可以重置集群设定2.添加kubernets环境变量到本机[rootmaster ~]# echo export KUBECONFIG/etc/kubernetes/admin.conf ~/.bash_profile [rootmaster ~]# source ~/.bash_profile [rootmaster ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady control-plane 3m8s v1.35.83.添加node节点到本集群可以看到集群中主机但是因为网络插件问题状态是NotReady4.安装网络插件[rootk8s-master ~]# tar zxf kube-flannel-v0.28.9.tar.gz [rootk8s-master ~]# ls flannel kube-flannel-v0.28.9.tar.gz [rootk8s-master ~]# cd flannel/ [rootk8s-master flannel]# docker load -i kube-flannel-v0.28.9.tar Loaded image: ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 Loaded image: ghcr.io/flannel-io/flannel:v0.28.9 [rootmaster ~]# docker tag ghcr.io/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 [rootmaster ~]#docker push reg.timinglee.org/flannel-io/flannel-cni-plugin:v1.9.1-flannel3 The push refers to repository [reg.timinglee.org/flannel-io/flannel-cni-plugin] 2c8aa52d4746: Pushed 5aa68bbbc67e: Pushed v1.9.0-flannel1: digest: sha256:b3d30c221113b30fea3e8a7fccb145e929b097d0319b9eeb6b5a591b10b5c671 size: 739 [rootmaster ~]# docker tag ghcr.io/flannel-io/flannel:v0.28.9 reg.timinglee.org/flannel-io/flannel:v0.28.9 [rootmaster ~]#docker push reg.timinglee.org/flannel-io/flannel:v0.28.9 The push refers to repository [reg.timinglee.org/flannel-io/flannel] 5668da16a30b: Pushed 5f70bf18a086: Pushed 00012e17b6cc: Pushed 9738bb9596cf: Pushed b6233bc105d7: Pushed 03767449b95b: Pushed 9a7d8cf9ff51: Pushed 13a60c10faeb: Pushed 70dc5e033175: Pushed 256f393e029f: Pushed v0.28.1: digest: sha256:e671adbc267460164555159210066d3304a43e3b5dd85cc0b5b6ad62e83aab52 size: 2414 [rootk8s-master flannel]# grep -n image: kube-flannel.yml #要改内容所在行的位置 148: image: flannel-io/flannel:v0.28.9 175: image: flannel-io/flannel-cni-plugin:v1.9.1-flannel3 186: image: flannel-io/flannel:v0.28.9 [rootmaster ~]# kubectl apply -f kube-flannel.yml namespace/kube-flannel created serviceaccount/flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds createdk8s集群部署好了