故障排查11 分钟阅读
Oracle RAC 触发 BUG，crsd 无限重启！
大家好，这里是公众号 DBA学习之路，分享一些学习数据库路上的知识和经验。 @TOC目录 今天检查一套 Oracle RAC 12.2.0.1 数据库，检查集群状态时，发现集群命令一直夯着没反应： !image.pngh
2025年3月24日阅读—点赞—收藏—
oraclerac
内容阅读
在知识库中专注阅读，并随时返回相关工具与课程
大家好，这里是公众号 DBA学习之路，分享一些学习数据库路上的知识和经验。
前言
本章目标：掌握本章核心知识点 前置要求：完成前序章节学习 预计时长：60 分钟
今天检查一套 Oracle RAC 12.2.0.1 数据库，检查集群状态时，发现集群命令一直夯着没反应：
image.png
经过一顿分析，终于解决问题，比较简单，这里分享一下处理过程。
问题分析
首先怀疑是集群资源挂了，查看集群资源，发现 ora.crsd 挂了：
1## 节点一2[grid@lucifer1 ~]$ crsctl stat res -t -init3--------------------------------------------------------------------------------4Name Target State Server State details 5--------------------------------------------------------------------------------6Cluster Resources7--------------------------------------------------------------------------------8ora.asm9 1 ONLINE ONLINE mesdb0 Started,STABLE10ora.cluster_interconnect.haip11 1 ONLINE ONLINE mesdb0 STABLE12ora.crf13 1 ONLINE ONLINE mesdb0 STABLE14ora.crsd15 1 ONLINE OFFLINE STABLE16ora.cssd17 1 ONLINE ONLINE mesdb0 STABLE18ora.cssdmonitor19 1 ONLINE ONLINE mesdb0 STABLE20ora.ctssd21 1 ONLINE ONLINE mesdb0 OBSERVER,STABLE22ora.diskmon23 1 OFFLINE OFFLINE STABLE24ora.evmd25 1 ONLINE ONLINE mesdb0 STABLE26ora.gipcd27 1 ONLINE ONLINE mesdb0 STABLE28ora.gpnpd29 1 ONLINE ONLINE mesdb0 STABLE30ora.mdnsd31 1 ONLINE ONLINE mesdb0 STABLE32ora.storage33 1 ONLINE ONLINE mesdb0 STABLE34--------------------------------------------------------------------------------3536## 节点二37[grid@lucifer2 ~]$ crsctl stat res -t38CRS-4535: Cannot communicate with Cluster Ready Services39CRS-4000: Command Status failed, or completed with errors.检查 crs alert.log 日志:
12025-02-24 06:11:42.105 [ORAROOTAGENT(29459)]CRS-8500: Oracle Clusterware ORAROOTAGENT 进程以操作系统进程 ID 29459 开头22025-02-24 06:12:42.142 [ORAROOTAGENT(29459)]CRS-5818: 已中止命令 'check' (对于资源 'ora.crsd')。详细资料见 (:CRSAGF00113:) {0:15:2} (位于 /oracle/app/grid/diag/crs/mesdb0/crs/trace/ohasd_orarootagent_root.trc)。32025-02-24 06:13:20.260 [CRSD(30357)]CRS-8500: Oracle Clusterware CRSD 进程以操作系统进程 ID 30357 开头42025-02-24 06:13:22.541 [CRSD(30357)]CRS-1019: 主机 mesdb0 上的 OCR 服务已退出。详细资料见 /oracle/app/grid/diag/crs/mesdb0/crs/trace/crsd.trc52025-02-24T06:13:22.563713+08:006Errors in file /oracle/app/grid/diag/crs/mesdb0/crs/trace/crsd.trc (incident=41):7CRS-1019 [] [] [] [] [] [] [] [] [] [] [] []8Incident details in: /oracle/app/grid/diag/crs/mesdb0/crs/incident/incdir_41/crsd_i41.trc9102025-02-24 06:13:22.584 [CRSD(30357)]CRS-8505: Oracle Clusterware CRSD 进程 (具有操作系统进程 ID 30357) 遇到内部错误 CRS-01019检查 crsd 日志：
12025-02-24 06:13:22.514 : OCRMSG:3187623680: prom_listen: Port str [a0f4-81a3-c06c-03aa]22025-02-24 06:13:22.514 : OCRSRV:3187623680: proath_listen: listening to remote requests at portstr [a0f4-81a3-c06c-03aa]32025-02-24 06:13:22.518 : OCRMSG:3168728832: prom_listen: Port str [ab1d-0688-2d30-7387]42025-02-24 06:13:22.518 : OCRSRV:3168728832: th_invalidate_cache: listening to cache_invalidation requests at portstr [ab1d-0688-2d30-7387]52025-02-24 06:13:22.522 : OCRMSG:3166627584: prom_listen: Port str [c71c-c1a3-dc88-994f]62025-02-24 06:13:22.522 : OCRSRV:3166627584: proath_listen: listening to remote rim requests at portstr [c71c-c1a3-dc88-994f]72025-02-24 06:13:22.533 : OCRMAS:3164526336: th_calc_av: Configured Active Patch Level [0]82025-02-24 06:13:22.533 : OCRMAS:3164526336: th_calc_av:5'': Return persisted APL [0]9 OCRMAS:3164526336: th_calc_av:5': Return persisted AV [203424000] [12.2.0.1.0]102025-02-24 06:13:22.535 : OCRMAS:3164526336: th_master_prereg: Persistent upgrade state retrieved from OCR is [0].112025-02-24 06:13:22.537 : OCRMAS:3164526336: th_master_prereg: Persistent upgrade toversion buffer retrieved from OCR is [12.2.0.1.0]. Setting toversion to [203424000].122025-02-24 06:13:22.541 : CSSCLNT:3164526336: clssgsGroupJoin: member in use group(1/ocrlocal)132025-02-24 06:13:22.541 : default:3164526336: procr_reg_localgrp: Error [14] from clssgsreglocalgrp(). Return [23].142025-02-24 06:13:22.541 : default:3164526336: SLOS : [clsuSlosFormatDiag called with non-error slos.]15162025-02-24 06:13:22.541 : OCRMAS:3164526336: th_master_register: Failed to register in OCRLOCAL group. Retval:[23]172025-02-24 06:13:22.541 : OCRAPI:3164526336: procr_ctx_set_invalid: ctx is in state [6].182025-02-24 06:13:22.541 : OCRAPI:3164526336: procr_ctx_set_invalid: ctx set to invalid19Trace file /oracle/app/grid/diag/crs/mesdb0/crs/trace/crsd.trc20Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.21DDE: Flood control is not active222025-02-24T06:13:22.564565+08:0023Incident 41 created, dump file: /oracle/app/grid/diag/crs/mesdb0/crs/incident/incdir_41/crsd_i41.trc24CRS-1019 [] [] [] [] [] [] [] [] [] [] [] []252025-02-24 06:13:22.706 : OCRAPI:3164526336: procr_ctx_set_invalid: Aborting...26Trace file /oracle/app/grid/diag/crs/mesdb0/crs/trace/crsd.trc27Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.28 default:2552033344: 1: clskec:has:CLSU:910 4 args[CLSD00302][mod=clsdadr.c][loc=(:CLSD00302:)][msg=clsdAdrInit: Trace file size and number of segments fetched from environemnt variable: ORA_DAEMON_TRACE_FILE_OPTIONS filesize=26214400,numsegments=10]2930 CLSB:2552033344: Argument count (argc) for this daemon is 231 CLSB:2552033344: Argument 0 is: /oracle/app/12.2.0/grid/bin/crsd.bin32 CLSB:2552033344: Argument 1 is: reboot332025-02-24 06:13:22.829 : CSSCLNT:2552033344: clsssinit: initialized context: (0x4edf930) flags 0x207342025-02-24 06:13:22.829 : CRSMAIN:2552033344: First attempt: init CSS context succeeded.352025-02-24 06:13:22.829 : CRSMAIN:2552033344: Start mode: normal362025-02-24 06:13:22.831 : CLSDMT:2343307008: PID for the Process [30402], connkey CRSD372025-02-24 06:13:23.745 : CRSMAIN:2552033344: CRS Daemon Starting382025-02-24 06:13:23.745 : CRSMAIN:2343307008: Process environment is not initialized yet!392025-02-24 06:13:23.746 : CRSD:2552033344: Logging level for Module: clsdadr 0402025-02-24 06:13:23.746 : CRSD:2552033344: Logging level for Module: clsdnreg 0412025-02-24 06:13:23.746 : CRSD:2552033344: Logging level for Module: clsdynam 0查看 trace dump 日志：
1----- Invocation Context Dump -----2Address: 0x7f1a9c0243403Phase: 34flags: 0x10E00005Incident ID: 416Error Descriptor: CRS-1019 [] [] [] [] [] [] [] [] [] [] [] []7Error class: 08Problem Key # of args: 09Number of actions: 1010----- Incident Context Dump -----11Address: 0x7f1abc9d99d012Incident ID: 4113Problem Key: CRS 101914Error: CRS-1019 [] [] [] [] [] [] [] [] [] [] [] []15[00]: dbgePostErrorDirectVaList_int [diag_dde]16[01]: dbgePostErrorDirect [diag_dde]17[02]: clsdAdrPostError []18[03]: clsdadrpr_CreateIncidentCheck []19[04]: clsdadrprAlert []20[05]: clsd_alertprintft []21[06]: proath_master_exit_helper []<-- Signaling22[07]: proath_master_register []23[08]: proath_master []24[09]: start_thread []25MD [00]: 'Client ProcId'='crsd.bin@mesdb0.30357_139752810403584' (0x0)26Impact 0:27Impact 1:28Impact 2:29Impact 3:30Derived Impact:31----- END Incident Context Dump -----看着很像是 BUG，在 MOS 搜索后发现有一个文章很匹配：
MOS 截图如下：
image.png
image.png
image.png
image.png
MOS 内容与问题日志完全一致，确认是 BUG，需要进行补丁修复：
image.png
解决问题
下载 BUG 所需补丁 Patch 24396050: LNX64-12.2-CRS: CRSD.BIN FAILED SEVERAL TIMES WITH ERROR CRS-1019
image.png
更新 OPatch
查看补丁 README：You must use the OPatch utility version 12.2.0.1.5 or later to apply this patch.
检查当前 OPatch 补丁是否符合要求：
1[grid@mesdb0 ~]$ cd $ORACLE_HOME/OPatch/2[grid@mesdb0 OPatch]$ ./opatch version3OPatch Version: 12.2.0.1.645OPatch succeeded.符合要求，不需要更新 OPatch。
解压补丁
1## root 执行2unzip -q /soft/p24396050_122010_Linux-x86-64.zip -d /soft/3chown -R oracle:oinstall /soft/24396050安装补丁
1## root 执行2export GI_HOME=/oracle/app/12.2.0/grid34## 安装前检查5$GI_HOME/OPatch/opatchauto apply /soft/24396050 -analyze67## 安装补丁8$GI_HOME/OPatch/opatchauto apply /soft/24396050 -oh $GI_HOME安装补丁后重启系统验证集群已经恢复正常。