一前言高性能计算HPC是一个计算机集群系统它通过各种互联技术将多个计算机系统连接在一起利用所有被连接系统的综合计算能力来处理大型计算问题所以又通常被称为高性能计算集群。当服务器具备IPMI接口时BMC/IPMI web后台可以看到硬件的运行情况比如CPU温度、内存温度、风扇转速、主板电压等这些数据默认只能通过登录web后台或者ipmitool等接口工具来实时查看看不到历史曲线图也做不到监控告警当服务器出现问题时除了日志之外这些物理传感器指标绘制的趋势图更有利于我们判断服务器在历史异常时间点的硬件状态趋势变化情况或者实时监测指标到达设定的阈值时自动告警因此本文油然而生。二准备该项目采用docker部署1基础环境准备部署a离线安装docker环境下载适合该项目的docker版本离线安装包下载地址https://download.docker.com/linux/static/stable/b上传docker安装包解压docker安装包rooth100-NF5280-M7-A0-R0-00:/opt# tar -zxvf docker-29.1.0.tgzc复制docker到/usr/bin/rooth100-NF5280-M7-A0-R0-00:/opt# cp docker/* /usr/bin/d编辑docker启动服务rooth100-NF5280-M7-A0-R0-00:/opt# vim /etc/systemd/system/docker.service[Unit] DescriptionDocker Application Container Engine Documentationhttps://docs.docker.com Afternetwork-online.target firewalld.service Wantsnetwork-online.target [Service] Typenotify ExecStart/usr/bin/dockerd ExecReload/bin/kill -s HUP $MAINPID LimitNOFILE1048576 LimitNPROC1048576 LimitCOREinfinity TimeoutStartSec0 Delegateyes KillModeprocess Restarton-failure StartLimitBurst3 StartLimitInterval60s [Install] WantedBymulti-user.targete启动docker服务并验证rooth100-NF5280-M7-A0-R0-00:/opt# systemctl start dockerrooth100-NF5280-M7-A0-R0-00:/opt# systemctl status dockerf上传prometheusgrafana,antilax3_ipmi,自定义fc_ipmi镜像包加载镜像rooth100-NF5280-M7-A0-R0-00:/opt# docker load -i prometheus.tarrooth100-NF5280-M7-A0-R0-00:/opt# docker load -i grafana.tar#构建自定义ipmi镜像,主要修改项目服务器bmc信息dockerfile文件配置rooth100-NF5280-M7-A0-R0-00:/opt/ipmi# cat ipmi_remote3011.yml modules: default: user: xxxx pass: xxxxxxxxxx privilege: admin driver: LAN_2_0 xx.xx.xx.xx: user: xxxx pass: xxxxxxxxx privilege: admin driver: LAN_2_0 collectors: - bmc - ipmi - dcmi - chassis - sel custom_args: bmc: - bmc-info ipmi: - ipmimonitoring dcmi: - ipmi-dcmi chassis: - ipmi-chassis sel: - ipmi-sel rooth100-NF5280-M7-A0-R0-00:/opt/ipmi# cat Dockerfile FROM antilax3/ipmi-exporter # 本地安装软件 #RUN apt-get update apt-get install -y curl # 设置工作目录 WORKDIR /app # 拷贝文件 COPY myscript.sh . COPY ipmi_remote3011.yml . # 设置执行权限 RUN chmod x myscript.sh RUN touch /app/test.log # 入口点 ENTRYPOINT [/app/myscript.sh]#构建自定义ipmi采集镜像rooth100-NF5280-M7-A0-R0-00:/opt/ipmi# docker build -t fc_ipmi .rooth100-NF5280-M7-A0-R0-00:/opt# docker imagesg起动ipmi采集服务prometheus监控服务grafana可视化服务#创建每个服务持久化配置文件目录,这里主要是prometheusgrafanarooth100-NF5280-M7-A0-R0-00:/opt# mkdir -p -m 777 prometheus/config#prometheus下主要是配置文件和告警规则rooth100-NF5280-M7-A0-R0-00:/opt/prometheus/config# cat prometheus.yml # my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global evaluation_interval. rule_files: - /etc/prometheus/rules/*.yml # - second_rules.yml # A scrape configuration containing exactly one endpoint to scrape: # Here its Prometheus itself. scrape_configs: # The job name is added as a label jobjob_name to any timeseries scraped from this config. - job_name: prometheus # metrics_path defaults to /metrics # scheme defaults to http. static_configs: - targets: [xx.xx.xx.xx:9090] - job_name: ipmi_exporter static_configs: - targets: [xx.xx.xx.xx] - targets: [xx.xx.xx.xx] params: module: [default] scrape_interval: 1m scrape_timeout: 30s metrics_path: /ipmi scheme: http file_sd_configs: - files: - /etc/ipmi_targets.yml refresh_interval: 5m relabel_configs: - source_labels: [__address__] separator: ; regex: (.*) target_label: __param_target replacement: ${1} action: replace - source_labels: [__param_target] separator: ; regex: (.*) target_label: instance replacement: ${1} action: replace - separator: ; regex: .* target_label: __address__ #该地址为ipmi服务地址 replacement: xx.xx.xx.xx:3011 action: replace #监控ipmi的告警规则 rooth100-NF5280-M7-A0-R0-00:/opt/prometheus/config/rules# cat Memory_hardware.yml groups: - name: Memory_hardware rules: - alert: Memory_hardware expr: ipmi_sensor_state{typeMemory} 1 for: 3m labels: user: caizh annotations: summary: Instance {{ $labels.instance }} 内存硬件警告 description: {{ $labels.instance }} of job {{$labels.job}} 内存硬件警告,当前状态[{{ $value }}].rooth100-NF5280-M7-A0-R0-00:/opt# mkdir -p -m 777 prometheus/datarooth100-NF5280-M7-A0-R0-00:/opt# mkdir -p -m 777 grafana/data#ipmic采集 docker run -d --restartalways -v /etc/localtime:/etc/localtime:ro -p 3011:3011 --name fc_ipmi fc_ipmi #prometheus监控 docker run -d --name prometheus --restartalways -v /etc/localtime:/etc/localtime:ro -p 9090:9090 -v /opt/prometheus/data:/prometheus -v /opt/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml -v /opt/prometheus/config/rules:/etc/prometheus/rules prom/prometheus #grafana可视化 docker run -d --restartalways -v /etc/localtime:/etc/localtime:ro -p 3000:3000 --namegrafana -v /opt/grafana/data:/var/lib/grafana grafana/grafana-enterprise:11.3.0-ubuntuh服务验证2效果展示