Kubernetes Redis 7.2 集群部署：NFS存储+StatefulSet 6节点配置详解

Kubernetes Redis 7.2 集群部署NFS存储StatefulSet 6节点配置详解Redis作为高性能的内存数据库在分布式系统中扮演着关键角色。本文将深入探讨如何在Kubernetes环境中利用StatefulSet和NFS存储部署一个高可用的Redis 7.2集群3主3从架构。不同于传统手动配置方式我们将采用Kubernetes原生资源实现自动化部署和管理显著提升运维效率。1. 架构设计与核心组件在Kubernetes中部署Redis集群需要考虑以下几个关键因素数据持久化Redis作为有状态服务需要确保数据在Pod重启或迁移时不丢失服务发现集群节点需要能够相互发现并建立连接高可用性主从节点应分布在不同的物理节点上自动化运维包括集群初始化、故障恢复等操作我们选择的解决方案组合如下组件用途优势StatefulSet管理Redis Pod提供稳定的网络标识和有序部署NFS StorageClass动态提供持久化存储支持多节点读写简化PV管理Headless Service内部DNS发现为每个Pod提供唯一域名ClusterIP Service外部访问入口统一访问入口负载均衡Init Job自动初始化集群替代手动执行redis-cli命令关键配置要点每个Redis Pod独占一个PV确保数据隔离通过Pod反亲和性(Anti-Affinity)分散主节点使用ConfigMap统一管理Redis配置文件通过Job自动完成集群节点握手和槽位分配2. 环境准备与NFS配置2.1 NFS服务器设置首先在任意节点可以是K8s集群外部署NFS服务# 安装NFS服务端 sudo apt-get install nfs-kernel-server -y # 创建共享目录 sudo mkdir -p /nfs/redis sudo chmod -R 777 /nfs/redis # 配置导出目录 echo /nfs/redis *(rw,sync,no_subtree_check,no_root_squash) | sudo tee -a /etc/exports # 应用配置 sudo exportfs -ra sudo systemctl restart nfs-kernel-server2.2 Kubernetes端NFS配置在Kubernetes集群中创建StorageClass实现动态PV供给# nfs-storageclass.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: redis-nfs-sc provisioner: example.com/nfs parameters: archiveOnDelete: false mountOptions: - hard - nfsvers4.1提示如果使用云服务商的托管Kubernetes建议使用对应的CSI驱动如AWS EFS CSI替代原生NFS3. Redis集群核心配置3.1 ConfigMap配置创建包含Redis配置和初始化脚本的ConfigMap# redis-configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: redis-cluster-config data: redis.conf: | cluster-enabled yes cluster-require-full-coverage no cluster-node-timeout 15000 cluster-config-file /data/nodes.conf cluster-migration-barrier 1 appendonly yes protected-mode no port 6379 dir /data update-node.sh: | #!/bin/sh REDIS_NODES/data/nodes.conf if [ -f ${REDIS_NODES} ]; then sed -i -e /myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/ ${REDIS_NODES} fi exec $3.2 StatefulSet配置这是最核心的部署文件实现了6节点Redis集群# redis-statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: redis-cluster spec: serviceName: redis-cluster replicas: 6 selector: matchLabels: app: redis-cluster template: metadata: labels: app: redis-cluster spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - redis-cluster topologyKey: kubernetes.io/hostname containers: - name: redis image: redis:7.2-alpine command: [/conf/update-node.sh, redis-server, /conf/redis.conf] ports: - containerPort: 6379 name: client - containerPort: 16379 name: gossip env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP volumeMounts: - name: conf mountPath: /conf readOnly: false - name: data mountPath: /data readOnly: false volumes: - name: conf configMap: name: redis-cluster-config defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ ReadWriteOnce ] storageClassName: redis-nfs-sc resources: requests: storage: 5Gi关键参数解析podAntiAffinity确保Redis Pod分散在不同节点update-node.sh动态更新节点配置文件中的IP地址volumeClaimTemplates为每个Pod自动创建PVC3.3 服务配置创建两种Service分别用于集群内部通信和外部访问# redis-services.yaml apiVersion: v1 kind: Service metadata: name: redis-cluster spec: clusterIP: None ports: - port: 6379 targetPort: 6379 name: client - port: 16379 targetPort: 16379 name: gossip selector: app: redis-cluster --- apiVersion: v1 kind: Service metadata: name: redis-access spec: type: ClusterIP ports: - port: 6379 targetPort: 6379 selector: app: redis-cluster4. 集群初始化与验证4.1 自动初始化Job传统方式需要手动执行redis-cli命令初始化集群我们通过Job实现自动化# redis-init-job.yaml apiVersion: batch/v1 kind: Job metadata: name: redis-cluster-init spec: template: spec: containers: - name: cluster-init image: redis:7.2-alpine command: - sh - -c - | sleep 10 redis-cli --cluster create \ redis-cluster-0.redis-cluster:6379 \ redis-cluster-1.redis-cluster:6379 \ redis-cluster-2.redis-cluster:6379 \ redis-cluster-3.redis-cluster:6379 \ redis-cluster-4.redis-cluster:6379 \ redis-cluster-5.redis-cluster:6379 \ --cluster-replicas 1 --cluster-yes restartPolicy: OnFailure4.2 部署流程按顺序应用所有配置文件kubectl apply -f nfs-storageclass.yaml kubectl apply -f redis-configmap.yaml kubectl apply -f redis-services.yaml kubectl apply -f redis-statefulset.yaml # 等待所有Pod就绪 kubectl wait --forconditionReady pod -l appredis-cluster --timeout300s # 初始化集群 kubectl apply -f redis-init-job.yaml4.3 集群验证检查集群状态# 查看Pod运行状态 kubectl get pods -l appredis-cluster -o wide # 检查集群节点关系 kubectl exec redis-cluster-0 -- redis-cli cluster nodes # 测试数据读写 kubectl exec -it redis-cluster-0 -- redis-cli set test-key hello kubectl exec -it redis-cluster-1 -- redis-cli get test-key5. 高级配置与优化5.1 资源限制与探针在生产环境中应为Redis容器配置资源限制和健康检查resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi livenessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 periodSeconds: 105.2 密码认证增强安全性可启用Redis密码认证# 在ConfigMap的redis.conf中添加 requirepass your-strong-password masterauth your-strong-password # 初始化Job需要相应修改 redis-cli --cluster create ... -a your-strong-password5.3 监控与告警建议配置Prometheus监控Redis集群# 添加Annotations暴露指标 annotations: prometheus.io/scrape: true prometheus.io/port: 91216. 故障排查与维护6.1 常见问题处理节点无法加入集群检查Pod间网络连通性验证DNS解析是否正常查看Redis日志kubectl logs pod-name数据持久化问题检查PVC/PV状态kubectl get pvc,pv验证NFS服务器连接和权限检查存储空间使用情况6.2 集群扩容横向扩展Redis集群从6节点扩展到9节点# 扩展StatefulSet kubectl scale statefulset redis-cluster --replicas9 # 添加新节点到集群 kubectl exec redis-cluster-0 -- redis-cli --cluster add-node \ redis-cluster-6.redis-cluster:6379 \ redis-cluster-0.redis-cluster:63796.3 备份与恢复建议的备份策略定期对NFS存储进行快照使用Redis的BGSAVE命令创建RDB备份通过redis-cli --cluster backup命令导出集群数据# 示例备份命令 kubectl exec redis-cluster-0 -- redis-cli bgsave