环境被监控系统版本redhat7.9 splunk版本9.X.X splunk服务器地址10.10.10.10前提条件splunk服务器已经配置好了告警邮件服务器接收9997端口数据已新建索引testdata终端可以访问splunk服务器的9997/tcp端口一、被监控redhat系统安装客户端1、将安装包上传到redhat系统/tmp目录2、安装splunk客户端rpm -ivh splunkforwarder-9.3.2-d8bb32809498.x86_64.rpm安装过程中出现的下述错误对客户端基本功能使用不会造成影响[rootXXX tmp]# rpm -ivh splunkforwarder-9.3.2-d8bb32809498.x86_64.rpmwarning: splunkforwarder-9.3.2-d8bb32809498.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEYPreparing... ################################# [100%]Updating / installing...1:splunkforwarder-9.3.2-d8bb3280949################################# [100%]find: ‘/opt/splunkforwarder/lib/python3.7/site-packages’: No such file or directoryfind: ‘/opt/splunkforwarder/lib/python3.9/site-packages’: No such file or directorycomplete3、配置客户端日志外发新建并编辑outputs.conf文件vi /opt/splunkforwarder/etc/system/local/outputs.conf内容为[tcpout] defaultGroup primary_splunkservers [tcpout:primary_splunkservers] server 10.10.10.10:9997 compressed true4、配置客户端日志收集新建并编辑inputs.conf文件vi /opt/splunkforwarder/etc/system/local/inputs.conf内容为[monitor:///var/log/messages] sourcetype linux_messages_log index testdata disabled 05、运行splunk客户端启动客户端服务/opt/splunkforwarder/bin/splunk start[rootxxxx tmp]# /opt/splunkforwarder/bin/splunk startWarning: Attempting to revert the SPLUNK_HOME ownershipWarning: Executing chown -R splunkfwd:splunkfwd /opt/splunkforwarderSPLUNK GENERAL TERMSLast Updated: August 12, 2021These Splunk General Terms (General Terms) between Splunk Inc., a Delawarecorporation, with its principal place of business at 270 Brannan Street, SanFrancisco, California 94107, U.S.A (Splunk or we or us or our) and you(Customer or you or your) apply to the purchase of licenses andsubscriptions for Splunks Offerings. By clicking on the appropriate button,or by downloading, installing, accessing or using the Offerings, you agree tothese General Terms. If you are entering into these General Terms on behalf ofCustomer, you represent that you have the authority to bind Customer. If youdo not agree to these General Terms, or if you are not authorized to acceptthe General Terms on behalf of the Customer, do not download, install, access,or use any of the Offerings.See the General Terms Definitions Exhibit attached for definitions ofcapitalized terms not defined herein.XXXXXXXXXXXXXXXXXXXXX中间有很多内容直接拖到底Splunk Preexisting IP means, with respect to any CI Services Materials, allassociated Splunk technology and all Intellectual Property Rights created oracquired: (a) prior to the date of the Statement of Work that includes suchCI Services Materials, or (b) after the date of such Statement of Work butindependently of the CI Services provided under such Statement of Work.Statement of Work means the statements of work and/or any and all applicableOrders, that describe the specific services to be performed by Splunk,including any materials and deliverables to be delivered by Splunk.Do you agree with this license? [y/n]:yThis appears to be your first time running this version of Splunk.Splunk software must create an administrator account during startup. Otherwise, you cannot log in.Create credentials for the administrator account.Characters do not appear on the screen when you type in credentials.Please enter an administrator username: adminadminPassword must contain at least:* 8 total printable ASCII character(s).Please enter a new password:Please confirm new password:PasswordCreating unit file...Important: splunk will start under systemd as user: splunkfwdThe unit file has been created.Splunk CSI: Logfiles.Checking prerequisites...Checking mgmt port [8089]: openCreating: /opt/splunkforwarder/var/lib/splunkCreating: /opt/splunkforwarder/var/run/splunkCreating: /opt/splunkforwarder/var/run/splunk/appserver/i18nCreating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/cssCreating: /opt/splunkforwarder/var/run/splunk/uploadCreating: /opt/splunkforwarder/var/run/splunk/search_telemetryCreating: /opt/splunkforwarder/var/run/splunk/search_logCreating: /opt/splunkforwarder/var/spool/splunkCreating: /opt/splunkforwarder/var/spool/dirmoncacheCreating: /opt/splunkforwarder/var/lib/splunk/authDbCreating: /opt/splunkforwarder/var/lib/splunk/hashDbCreating: /opt/splunkforwarder/var/run/splunk/sessionsNew certs have been generated in /opt/splunkforwarder/etc/auth.Checking conf files for problems...DoneChecking default conf files for edits...Validating installed files against hashes from /opt/splunkforwarder/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-x86_64-manifestAll installed files intact.DoneAll preliminary checks passed.Starting splunk server daemon (splunkd)...Done[ OK ]客户端这边的账户密码正常情况下不会使用到但建议记录到文档中留存。二、将日志正文内容添加到message字段1、在splunk平台查看收集到的日志在新搜索中搜索indextestdata即可搜索到/var/log/messages中的日志此时正文内容不属于任何字段。2、在当前页面右侧点击“提取新字段”3、在新出现的窗口中随便选择“我更喜欢自己编写正则表达式”4、正则表达式填写 ^(?Pmessage.) 即获取日志的所有正文内容点击预览查看是否所有的日志正文都变成message字段了可以看到_raw字段中所有内容都变蓝了且和message字段中的内容一致这说明正则表达式的内容获取了日志的所有正文内容。5、点击保存6、在新出现的窗口查看信息是否正确字段的提取名称默认即可7、点击完成会出现提示成功的窗口8、再看之前的日志已经多了一行message信息三、根据message信息设置邮件告警新建邮件告警搜索内容如下根据上述内容设置系统日志中出现error字样发送告警效果如下