环境说明k8s集群单主机版本1.35.5显卡nvidia系统ubuntu24.04容器运行时containerd 2.x容器运行时使用docker或k8s集群部署操作参考上一篇文章GPU虚拟化容器进行时部署配置安装containerdapt install -y containerd生成默认配置文件containerd 的核心配置文件位于 /etc/containerd/config.toml。如果该文件不存在可以使用以下命令生成默认配置mkdir -p /etc/containerd containerd config default /etc/containerd/config.toml修改配置文件vim /etc/containerd/config.toml找到[plugins.io.containerd.cri.v1.images.registry]将config_path /etc/containerd/certs.d:/etc/docker/certs.d注释新增一行修改后示例[plugins.io.containerd.cri.v1.images.registry]# config_path /etc/containerd/certs.d:/etc/docker/certs.dconfig_path /etc/containerd/certs.d重启containerdsystemctl restart containerd.service因为conertainerd实现物理隔离按需读取所以按域名分目录添加docker.io仓库mkdir -p /etc/containerd/certs.d/docker.io cat /etc/containerd/certs.d/docker.io/hosts.toml EOF server https://registry-1.docker.io [host.https://docker.m.daocloud.io] capabilities [pull, resolve] [host.https://docker.1panel.live] capabilities [pull, resolve] [host.https://docker.1ms.run] capabilities [pull, resolve] [host.https://docker.xuanyuan.me] capabilities [pull, resolve] EOF添加registry.k8s.io仓库mkdir -p /etc/containerd/certs.d/registry.k8s.io cat /etc/containerd/certs.d/registry.k8s.io/hosts.toml EOF server https://registry.k8s.io [host.https://registry.aliyuncs.com/google_containers] capabilities [pull, resolve] EOF不需要重启containerd支持热更新配置文件完成后树形总览rootk8s-master:~# tree /etc/containerd//etc/containerd/├── certs.d│ ├── docker.io│ │ └── hosts.toml│ └── registry.k8s.io│ └── hosts.toml└── config.toml安装crictlcurl -L https://gh-proxy.com/https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.35.0/crictl-v1.35.0-linux-amd64.tar.gz | sudo tar -C /usr/local/bin -xz crictl version cat /etc/crictl.yaml EOF runtime-endpoint: unix:///var/run/containerd/containerd.sock image-endpoint: unix:///var/run/containerd/containerd.sock timeout: 10 debug: false EOF crictl version测试及注意点注意1.原生调试工具ctr 默认不走仓库需要指定加上 --hosts-dir /etc/containerd/certs.d2.crictl是Kubernetes CRI 接口的调试工具拉取的镜像默认在containerd的k8s.io命名空间而ctr拉取的镜像默认在default命名空间# 测试 Docker Hub crictl pull docker.io/library/nginx:latest #测试ctr拉取 ctr images pull --hosts-dir /etc/containerd/certs.d/ docker.io/library/nginx:latest安装NV驱动这里自动检测并安装合适的驱动也可以自己安装需要的驱动ubuntu-drivers autoinstall驱动安装完成之后需要重启reboot验证nvidia-smi配置NVIDIA GPU运行时安装容器工具包 NVIDIA Container Toolkit官网链接Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkithttps://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html具体命令#安装前置依赖 sudo apt-get update sudo apt-get install -y --no-install-recommends \ ca-certificates \ curl \ gnupg2 #配置仓库 curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed s#deb https://#deb [signed-by/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list #更新软件包列表 sudo apt-get update #安装容器工具包 export NVIDIA_CONTAINER_TOOLKIT_VERSION1.19.1-1 sudo apt-get install -y \ nvidia-container-toolkit${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ nvidia-container-toolkit-base${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container-tools${NVIDIA_CONTAINER_TOOLKIT_VERSION} \ libnvidia-container1${NVIDIA_CONTAINER_TOOLKIT_VERSION}配置containerd需要重启containerdsudo nvidia-ctk runtime configure --runtimecontainerd sudo systemctl restart containerd部署GPU 设备插件NVIDIA Device Plugin为了让 kubelet 能够感知到节点上的 GPU 设备建议参考文档GitHub - NVIDIA/k8s-device-plugin: NVIDIA device plugin for Kubernetes · GitHubNVIDIA device plugin for Kubernetes. Contribute to NVIDIA/k8s-device-plugin development by creating an account on GitHub.https://github.com/NVIDIA/k8s-device-plugin需要执行的命令创建nvidia运行时类因为插件需要运行在NVIDIA Container Toolkit中才能让k8s发现GPU如果 Device Plugin 已自动创建则可跳过此步cat EOF | kubectl apply -f - apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: nvidia handler: nvidia EOF创建DaemonSetwget https://gh-proxy.com/https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml kubectl apply -f nvidia-device-plugin.yml为 DaemonSet 指定 nvidia 运行时kubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system \ --typejson \ -p[{op: add, path: /spec/template/spec/runtimeClassName, value: nvidia}]发现问题发现这个DaemonSet无法启动rootk8s-master:~# kubectl get daemonsets.apps -n kube-system NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE calico-node 1 1 1 1 1 kubernetes.io/oslinux 53m kube-proxy 1 1 1 1 1 kubernetes.io/oslinux 67m nvidia-device-plugin-daemonset 0 0 0 0 0 none 7m32s查看具体信息发现是因为我们单节点默认master节点有污点无法进行调度kubectl describe daemonsets.apps -n kube-system去除污点仅测试生产慎用查看污点命令中的 ”k8s-master“ 是节点名称根据实际替换kubectl describe node k8s-master | grep -A 5 Taints:rootk8s-master:~# kubectl describe node k8s-master | grep -A 5 Taints:Taints: node-role.kubernetes.io/control-plane:NoScheduleUnschedulable: falseLease:HolderIdentity: k8s-masterAcquireTime: unsetRenewTime: Fri, 03 Jul 2026 15:04:27 0800发现污点 ”node-role.kubernetes.io/control-plane:NoSchedule“添加容忍度建议添加容忍度kubectl patch daemonset nvidia-device-plugin-daemonset -n kube-system \ --typejson \ -p[{op: add, path: /spec/template/spec/tolerations/-, value: {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule}}]也可以选择去除污点因为本环境是单机后续的pod需要调度在本机master所以这里选择去掉污点命令中的 ”k8s-master“ 是节点名称根据实际替换kubectl taint nodes k8s-master node-role.kubernetes.io/control-plane:NoSchedule-结果此时查看发现插件成功运行了查看节点资源kubectl describe node k8s-master | grep -A10 Allocatable验证1.创建一个测试pod测试k8s是否能够识别、调用GPUcat gpu-test.yaml EOF apiVersion: v1 kind: Pod metadata: name: gpu-test spec: restartPolicy: Never runtimeClassName: nvidia # 指定容器运行时 tolerations: # 如果在 Master 上测试需要此容忍度 - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: nvidia-smi image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nvidia/cuda:12.4.0-base-ubuntu22.04 command: [nvidia-smi] # 执行的命令 resources: limits: nvidia.com/gpu: 1 # 请求 1 张 GPU EOF kubectl apply -f gpu-test.yaml kubectl logs gpu-test可以看到成功看到GPU信息证明k8s能够成功识别并调用GPU2.验证计算能力nvidia-smi 通过只代表设备可见不代表 CUDA 计算正常cat gpu-computer-test.yaml EOF apiVersion: v1 kind: Pod metadata: name: gpu-compute-test spec: restartPolicy: Never runtimeClassName: nvidia tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: cuda-test image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0 command: [/cuda-samples/vectorAdd] resources: limits: nvidia.com/gpu: 1 EOF kubectl apply -f gpu-computer-test.yaml kubectl logs pods/gpu-compute-test看到 “Test PASSED”证明测试通过3.验证资源分配查看已有几张卡被使用cat gpu-resource-test.yaml EOF apiVersion: v1 kind: Pod metadata: name: gpu-resource-test spec: runtimeClassName: nvidia tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: sleep image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nvidia/cuda:12.4.0-base-ubuntu22.04 command: [sleep, 3600] resources: limits: nvidia.com/gpu: 1 EOF kubectl apply -f gpu-resource-test.yaml kubectl describe node k8s-master | grep -A10 Allocated resources最后测试完毕记得释放资源kubectl delete -f gpu-resource-test.yaml