节点1数据库软件顺利打完补丁但是节点2在打软件补丁的过程中由于虚拟机异常被迫重启导致应用补丁中断开机后检查节点2数据库软件补丁已应用但无法正常开启节点2数据库实例注打补丁前一定记得要先把软件目录tar一份作为备份。以下的问题就是没备份不得已的办法。数据库启动报错[gridrac2 ~]$ srvctl start instance-dorcl-iorcl2 PRCR-1013:Failed to start resource ora.orcl.db PRCR-1064:Failed to start resource ora.orcl.db onnoderac2 CRS-5017: The resource actionora.orcl.db startencountered the following error: ORA-03113: end-of-file on communication channel Process ID:0Session ID:0Serial number:0.For details refer to(:CLSN00107:)in/u01/grid/11.2.0.4/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log.CRS-2674: Start ofora.orcl.dbonrac2failed -------------------------------------------------------------------[oraclerac2 ~]$ sqlplus / as sysdba SQL*Plus: Release11.2.0.4.0 Production on Tue Jul1410:18:362026Copyright(c)1982,2013, Oracle. All rights reserved. Connected to an idle instance. SQLstartup nomount ORA-03113: end-of-file on communication channel SQLexitDisconnected查看数据库日志数据库日志没有任何报错信息oraagent_oracle.log 也没有明确的信息Tue Jul1410:18:402026Starting ORACLE instance(normal)************************ Large Pages Information ******************* Per process system memlock(soft)limitUNLIMITED Total Shared Global RegioninLarge Pages0KB(0%)Large Pages used by this instance:0(0KB)Large Pages unused system wide0(0KB)Large Pages configured system wide0(0KB)Large Page size2048KB RECOMMENDATION: Total System Global Area size is1906MB. For optimal performance, prior to the next instance restart:1. Increase the number of unused large pages by at least953(page size2048KB, total size1906MB)system wide to get100% of the System Global Area allocated with large pages ******************************************************************** LICENSE_MAX_SESSION0LICENSE_SESSIONS_WARNING0Initial number of CPU is1Number of processor coresinthe system is1Number of processor socketsinthe system is1集群状态检查[gridrac2 ~]$ crsctlstatres-t-------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.FRA.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.LISTENER.lsnr ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.OCR.dg ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.asm ONLINE ONLINE rac1 Started ONLINE ONLINE rac2 Started ora.gsd OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 ora.net1.network ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.ons ONLINE ONLINE rac1 ONLINE ONLINE rac2 ora.registry.acfs ONLINE ONLINE rac1 ONLINE ONLINE rac2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr1ONLINE ONLINE rac1 ora.cvu1ONLINE ONLINE rac1 ora.oc4j1ONLINE ONLINE rac2 ora.orcl.db1ONLINE ONLINE rac1 Open2ONLINE OFFLINE Instance Shutdown ora.rac1.vip1ONLINE ONLINE rac1 ora.rac2.vip1ONLINE ONLINE rac2 ora.scan1.vip1ONLINE ONLINE rac1通过检查环境变量、集群日志、操作系统日志、数据库日志、系统参数等均未发现任何问题检查 bin/oracle 文件紧接着检查一个最容易出问题但被忽略的文件$ORACLE_HOME/bin/oracle节点1[oraclerac1 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239987343Jul1021:27 /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oraclerac1 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle节点2[oraclerac2 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239626641Dec12025/u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oraclerac2 ~]$ md5sum$ORACLE_HOME/bin/oracle 01e996bdb9127b1d65724331f7203dfe /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle发现 两边节点的 oracle 二进制文件不一致节点2的二进制文件还停留在 未打补丁的状态将节点2 oracle 文件替换为节点1相同 oracle 文件[oraclerac2 ~]$mv$ORACLE_HOME/bin/oracle /home/oracle/[oraclerac2 ~]$scprac1:$ORACLE_HOME/bin/oracle$ORACLE_HOME/bin/oracle oracle100% 229MB114.4MB/s 00:02[oraclerac2 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle## 使用 setasmgidwrap 工具修复权限[rootrac2 ~]# /u01/grid/11.2.0.4/bin/setasmgidwrap oracle_binary_path/u01/app/oracle/product/11.2.0.4/db_1/bin/oracle## 确认 oracle 二进制文件与 rac1 一致[oraclerac2 ~]$ ll$ORACLE_HOME/bin/oracle -rwsr-s--x1oracle asmadmin239987343Jul1410:27 /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle[oraclerac2 ~]$ md5sum$ORACLE_HOME/bin/oracle c4f967050767e21c783133afd56dd08a /u01/app/oracle/product/11.2.0.4/db_1/bin/oracle开启节点2实例可以正常启动[oraclerac2 ~]$ srvctl start instance-dorcl-iorcl2[oraclerac2 ~]$ps-ef|grepsmon root33461010:33 ? 00:00:09 /u01/grid/11.2.0.4/bin/osysmond.bin oracle157891010:53 ? 00:00:00 ora_smon_orcl2[gridrac2 ~]$ crsctlstatres-tora.orcl.db1ONLINE ONLINE rac1 Open2ONLINE ONLINE rac2 Open