一、环境准备虚拟机个人最小可用192.168.234.129 1c2g192.168.234.130 1c2g192.168.234.131 1c2g系统盘20G独立空白裸盘每节点至少 1 块10G无分区、无格式化二、统一基础配置所有节点执行1. 设置主机名 hosts# 节点1hostnamectl set-hostname ceph-node1# 节点2hostnamectl set-hostname ceph-node2# 节点3hostnamectl set-hostname ceph-node3所有节点/etc/hosts加上192.168.234.129 ceph-node1192.168.234.130 ceph-node2192.168.234.131 ceph-node32. 关闭防火墙、SELinuxsystemctl stop firewalldsystemctl disable firewalldsetenforce 0sed -i s/^SELINUXenforcing/SELINUXdisabled/ /etc/selinux/config3. 时间同步yum install -y chronysystemctl enable --now chronyd4. 无密码 SSHadmin 节点到其他节点在 ceph-node1ssh-keygenssh-copy-id ceph-node1ssh-copy-id ceph-node2ssh-copy-id ceph-node3三、使用 cephadm 部署1. 安装 cephadm只在 ceph-node1admin 节点 执行cat /etc/yum.repos.d/ceph-quincy.repo EOF[ceph-quincy]nameCeph Quincy x86_64baseurlhttps://mirrors.aliyun.com/ceph/rpm-quincy/el9/x86_64/enabled1gpgcheck1gpgkeyhttps://mirrors.aliyun.com/ceph/keys/release.asc[ceph-quincy-noarch]nameCeph Quincy noarchbaseurlhttps://mirrors.aliyun.com/ceph/rpm-quincy/el9/noarch/enabled1gpgcheck1gpgkeyhttps://mirrors.aliyun.com/ceph/keys/release.ascEOFdnf clean alldnf makecachednf install -y cephadm2. 引导集群cephadm bootstrap --mon-ip 192.168.234.129dashboard 地址、用户名 admin、密码 保存好3. 拷贝ceph.pub 集群密钥ssh-copy-id -f -i /etc/ceph/ceph.pub rootceph-node2ssh-copy-id -f -i /etc/ceph/ceph.pub rootceph-node34. 加入其他节点cephadm shell -- ceph orch host add ceph-node2cephadm shell -- ceph orch host add ceph-node3查看节点[rootlocalhost ~]# cephadm shell -- ceph orch host lsInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326fHOST ADDR LABELS STATUSceph-node1 192.168.234.129 _adminceph-node2 192.168.234.130ceph-node3 192.168.234.1315. 给所有节点打标签ceph orch host label add ceph-node1 _adminceph orch host label add ceph-node1 monceph orch host label add ceph-node2 monceph orch host label add ceph-node3 mon查看节点[rootlocalhost ~]# cephadm shell -- ceph orch host lsInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326fHOST ADDR LABELS STATUSceph-node1 192.168.234.129 _admin,monceph-node2 192.168.234.130 monceph-node3 192.168.234.131 mon四、添加 OSD存储盘1. 查看可用裸设备[rootlocalhost ~]# cephadm shell -- ceph orch device lsInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326fHOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONSceph-node1 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 27m agoceph-node2 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 8m agoceph-node3 /dev/nvme0n2 ssd VMware_Virtual_NVMe_Disk_VMware_NVME_0000 10.0G Yes 3m ago2. 自动扫描并创建 OSDceph orch apply osd --all-available-devices3. 查看 OSD 状态[rootlocalhost ~]# cephadm shell -- ceph osd treeInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326fID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF-1 0.02939 root default-7 0.00980 host ceph-node10 ssd 0.00980 osd.0 up 1.00000 1.00000-5 0.00980 host ceph-node21 ssd 0.00980 osd.1 up 1.00000 1.00000-3 0.00980 host ceph-node32 ssd 0.00980 osd.2 up 1.00000 1.00000[rootlocalhost ~]# cephadm shell -- ceph -sInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326fcluster:id: 044dbd4a-7c37-11f1-8276-000c295d03cdhealth: HEALTH_OKservices:mon: 3 daemons, quorum ceph-node1,ceph-node2,ceph-node3 (age 46m)mgr: ceph-node1.kvwqoy(active, since 66m), standbys: ceph-node2.jujajlosd: 3 osds: 3 up (since 43m), 3 in (since 43m)data:pools: 1 pools, 1 pgsobjects: 2 objects, 449 KiBusage: 885 MiB used, 29 GiB / 30 GiB availpgs: 1 activeclean五、访问 DashboardWeb 界面查看访问地址[rootlocalhost ~]# cephadm shell -- ceph mgr servicesInferring fsid 044dbd4a-7c37-11f1-8276-000c295d03cdInferring config /var/lib/ceph/044dbd4a-7c37-11f1-8276-000c295d03cd/mon.ceph-node1/configUsing ceph image with id 259b35566514 and tag v17 created on 2024-11-26 00:45:38 0000 UTCquay.io/ceph/cephsha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f{dashboard: https://192.168.234.129:8443/,prometheus: http://192.168.234.129:9283/}用前面保存的账号密码登录访问http://192.168.234.129:8443/六、常用验证命令ceph -s # 集群状态ceph osd tree # OSD 拓扑ceph df # 存储用量ceph mon stat # monitor 状态ceph mgr stat # manager 状态