高可用 & 容灾17 分钟阅读
Oracle 19C RAC 集群起不来了，ora.storage 的锅？
大家好，这里是公众号 DBA学习之路，分享一些学习数据库路上的知识和经验。 @TOC目录 半夜 11 点多接到电话，说有一个客户的生产库宕了一个节点，需要紧急恢复一下。赶紧下床打开电脑，一顿操作连上了客户的数据库主机，是
2025年2月20日阅读—点赞—收藏—
oraclerac
内容阅读
在知识库中专注阅读，并随时返回相关工具与课程
大家好，这里是公众号 DBA学习之路，分享一些学习数据库路上的知识和经验。
前言
本章目标：掌握本章核心知识点 前置要求：完成前序章节学习 预计时长：60 分钟
半夜 11 点多接到电话，说有一个客户的生产库宕了一个节点，需要紧急恢复一下。赶紧下床打开电脑，一顿操作连上了客户的数据库主机，是一套 19C（19.22） RAC 2 节点数据库，看了下是节点 2 宕掉了，集群起不来了。
经过和客户的沟通之后，解决了这个问题，本文记录一下分析与解决过程，分享给大家一起进步。
问题分析
连上主机之后，看了下节点 2 的集群状态，发现是启动集群卡在了 ora.storage 起不来：
通过
crsctl stat res -t -init检查启动进度时，我们可以看到在启动ora.storage资源时会 HANG 住，等待一顿时间后就会超时。
1ora.storage2 1 ONLINE OFFLINE STABLE3--------------------------------------------------------------------------------查看 ohasd.trc 日志（已排除存储相关问题）：
12025-02-19 00:23:02.300 : CRSPE:2791253760: [ INFO] {0:1:29} Got agent-specific msg: ORA-12541: TNS:no listener2ORA-12541: TNS:no listener342025-02-19 00:23:02.300 : AGFW:2803861248: [ INFO] {0:1:29} Agfw Proxy Server sending the reply to PE for message:RESOURCE_START[ora.storage 1 1] ID 45098:147162025-02-19 00:23:02.300 : CRSPE:2791253760: [ INFO] {0:1:29} Received reply to action [Start] message ID: 147172025-02-19 00:23:02.300 : CRSPE:2791253760: [ INFO] {0:1:29} Got agent-specific msg: CRS-5055: unable to connect to an ASM instance because no ASM inst8ance is running in the cluster9102025-02-19 00:23:02.300 : CRSPE:2791253760: [ INFO] {0:1:29} Received reply to action [Start] message ID: 1471112025-02-19 00:23:02.301 : CRSMAIN:2791253760: [ NONE] {0:1:29} {0:1:29} Created alert : (:CRSPE00221:) : Start action timed out!122025-02-19 00:23:02.301 : CRSPE:2791253760: [ INFO] {0:1:29} Start action failed with error code: 3132025-02-19 00:23:02.301 : CRSRPT:2789152512: [ INFO] {0:1:29} Published to EVM CRS_ACTION_FAILURE for ora.storage可以看到报错是 ORA-12541: TNS:no listener，查看 ohasd_orarootagent_root.trc 日志：
1025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3182232025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] ORA-12541: TNS:no listener4ORA-12541: TNS:no listener5672025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] Category: 7892025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] DepInfo: 1254110112025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] ADR is not properly configured12132025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} [ora.storage] -- trace dump end --14152025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} Thread:kgfoCheckMountExt isRunning is reset to false here162025-02-19 00:14:23.824 : USRTHRD:3772593920: [ INFO] {0:1:29} Thread:kgfoCheckMountExt isFinished set to true172025-02-19 00:14:23.824 : USRTHRD:3764188928: [ INFO] {0:1:29} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned182025-02-19 00:14:23.824 : USRTHRD:3764188928: [ INFO] {0:1:29} Thread:kgfoCheckMountExt Tasklet::doTask(kgfoCheckMountExt) executed in 9 seconds192025-02-19 00:14:23.824 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] StorageAgent::parsekgforetcodes retcode = 7, kgfoCheckMount(CRS), flag 2202025-02-19 00:14:23.824 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 12541, other: OR21A-12541: TNS:no listener22ORA-12541: TNS:no listener23242025-02-19 00:14:23.824 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] StorageAgent::check 260 kgfo returncode 1252025-02-19 00:14:23.824 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] (:CLSN00140:)StorageAgent::check 300 parsekgforretcodes OCR dgName CRS state 1262025-02-19 00:14:23.824 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] Storage::start waiting for check to not return PARTIAL or UNPLANNED_OFFLINE 1272025-02-19 00:14:23.827 :CLSDYNAM:3764188928: [ora.storage]{0:1:29} [start] CssGroup::regis registration of groupName:ASM_NETWORK mbrid:-1 m_grpNum:2 regType28:1 succeeded可以发现再启动 ora.storage 时报错 ORA-12541: TNS:no listener，连接私网监听 ASM_NETWORK 报错，看到这里我也是有点懵逼的，在 MOS 上也搜不到相关问题记录。
在网上搜索到两篇问题现象比较相近的文章：
看了一下，大致原因都是人工重启主机导致，需要重新加载一下私网监听 ASMNET1LSNR_ASM。
随后咨询了一下客户，果然是节点 2 实例 crash 之后无法启动，执行 reboot 重启主机没有反应，所以使用管理端强制重启了主机，主机起来之后就一直 hang 在这里。
那看来现象基本吻合，看一下节点 1 的私网监听 ASMNET1LSNR_ASM 状态：
1[grid@mesdb1 ~]$ lsnrctl stat ASMNET1LSNR_ASM23LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:23:3145Copyright (c) 1991, 2023, Oracle. All rights reserved.67Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))8STATUS of the LISTENER9------------------------10Alias ASMNET1LSNR_ASM11Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production12Start Date 31-JAN-2025 18:18:3913Uptime 18 days 6 hr. 4 min. 51 sec14Trace Level off15Security ON: Local OS Authentication16SNMP OFF17Listener Parameter File /u01/app/19.3.0/grid/network/admin/listener.ora18Listener Log File /u01/app/grid/diag/tnslsnr/mesdb1/asmnet1lsnr_asm/alert/log.xml19Listening Endpoints Summary...20 (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=ASMNET1LSNR_ASM)))21 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=11.11.11.1)(PORT=1525)))22Services Summary...23Service "+ASM" has 2 instance(s).24 Instance "+ASM1", status READY, has 2 handler(s) for this service...25 Instance "+ASM2", status READY, has 1 handler(s) for this service...26Service "+ASM_CRS" has 2 instance(s).27 Instance "+ASM1", status READY, has 2 handler(s) for this service...28 Instance "+ASM2", status READY, has 1 handler(s) for this service...29Service "+ASM_DATA" has 2 instance(s).30 Instance "+ASM1", status READY, has 2 handler(s) for this service...31 Instance "+ASM2", status READY, has 1 handler(s) for this service...32Service "+ASM_FRA" has 2 instance(s).33 Instance "+ASM1", status READY, has 2 handler(s) for this service...34 Instance "+ASM2", status READY, has 1 handler(s) for this service...35The command completed successfully可以看到 2 个实例都存在，明明节点 2 的 ASM 实例已经宕掉了，按理说应该只有节点 1 的存在才对。再次查看节点 1 的私网监听的 service：
1[grid@mesdb1 ~]$ lsnrctl services ASMNET1LSNR_ASM23LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:26:4245Copyright (c) 1991, 2023, Oracle. All rights reserved.67Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))8Services Summary...9Service "+ASM" has 2 instance(s).10 Instance "+ASM1", status READY, has 2 handler(s) for this service...11 Handler(s):12 "DEDICATED" established:0 refused:0 state:ready13 REMOTE SERVER14 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))15 "DEDICATED" established:0 refused:0 state:ready16 LOCAL SERVER17 Instance "+ASM2", status READY, has 1 handler(s) for this service...18 Handler(s):19 "DEDICATED" established:4578 refused:0 state:ready20 REMOTE SERVER21 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.2)(PORT=1525)))22Service "+ASM_CRS" has 2 instance(s).23 Instance "+ASM1", status READY, has 2 handler(s) for this service...24 Handler(s):25 "DEDICATED" established:0 refused:0 state:ready26 REMOTE SERVER27 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))28 "DEDICATED" established:0 refused:0 state:ready29 LOCAL SERVER30 Instance "+ASM2", status READY, has 1 handler(s) for this service...31 Handler(s):32 "DEDICATED" established:4578 refused:0 state:ready33 REMOTE SERVER34 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.2)(PORT=1525)))35Service "+ASM_DATA" has 2 instance(s).36 Instance "+ASM1", status READY, has 2 handler(s) for this service...37 Handler(s):38 "DEDICATED" established:0 refused:0 state:ready39 REMOTE SERVER40 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))41 "DEDICATED" established:0 refused:0 state:ready42 LOCAL SERVER43 Instance "+ASM2", status READY, has 1 handler(s) for this service...44 Handler(s):45 "DEDICATED" established:4578 refused:0 state:ready46 REMOTE SERVER47 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.2)(PORT=1525)))48Service "+ASM_FRA" has 2 instance(s).49 Instance "+ASM1", status READY, has 2 handler(s) for this service...50 Handler(s):51 "DEDICATED" established:0 refused:0 state:ready52 REMOTE SERVER53 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))54 "DEDICATED" established:0 refused:0 state:ready55 LOCAL SERVER56 Instance "+ASM2", status READY, has 1 handler(s) for this service...57 Handler(s):58 "DEDICATED" established:4578 refused:0 state:ready59 REMOTE SERVER60 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.2)(PORT=1525)))61The command completed successfully确实可以看到有节点 2 的监听服务依然存在，这里参考上述文章的解决方案是重载私网监听。
解决方案
在节点 1 上手动执行私网监听重载：
1[grid@mesdb1 ~]$ lsnrctl reload ASMNET1LSNR_ASM23LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:27:4045Copyright (c) 1991, 2023, Oracle. All rights reserved.67Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))8The command completed successfully9[grid@mesdb1 ~]$ lsnrctl services ASMNET1LSNR_ASM1011LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:27:421213Copyright (c) 1991, 2023, Oracle. All rights reserved.1415Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))16The listener supports no services17The command completed successfully18[grid@mesdb1 ~]$ lsnrctl services ASMNET1LSNR_ASM1920LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:27:442122Copyright (c) 1991, 2023, Oracle. All rights reserved.2324Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))25The listener supports no services26The command completed successfully27[grid@mesdb1 ~]$ lsnrctl services ASMNET1LSNR_ASM2829LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:27:453031Copyright (c) 1991, 2023, Oracle. All rights reserved.3233Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))34The listener supports no services35The command completed successfully手动注册一下监听：
1[grid@mesdb1 ~]$ sqlplus / as sysasm23SQL*Plus: Release 19.0.0.0.0 - Production on Wed Feb 19 00:27:48 20254Version 19.22.0.0.056Copyright (c) 1982, 2023, Oracle. All rights reserved.789Connected to:10Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production11Version 19.22.0.0.01213SQL> alter system register;1415System altered.1617SQL> exit 18Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production19Version 19.22.0.0.0再次查看私网监听状态：
1[grid@mesdb1 ~]$ lsnrctl services ASMNET1LSNR_ASM23LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 19-FEB-2025 00:28:0345Copyright (c) 1991, 2023, Oracle. All rights reserved.67Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM)))8Services Summary...9Service "+ASM" has 1 instance(s).10 Instance "+ASM1", status READY, has 2 handler(s) for this service...11 Handler(s):12 "DEDICATED" established:0 refused:0 state:ready13 REMOTE SERVER14 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))15 "DEDICATED" established:0 refused:0 state:ready16 LOCAL SERVER17Service "+ASM_CRS" has 1 instance(s).18 Instance "+ASM1", status READY, has 2 handler(s) for this service...19 Handler(s):20 "DEDICATED" established:0 refused:0 state:ready21 REMOTE SERVER22 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))23 "DEDICATED" established:0 refused:0 state:ready24 LOCAL SERVER25Service "+ASM_DATA" has 1 instance(s).26 Instance "+ASM1", status READY, has 2 handler(s) for this service...27 Handler(s):28 "DEDICATED" established:0 refused:0 state:ready29 REMOTE SERVER30 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))31 "DEDICATED" established:0 refused:0 state:ready32 LOCAL SERVER33Service "+ASM_FRA" has 1 instance(s).34 Instance "+ASM1", status READY, has 2 handler(s) for this service...35 Handler(s):36 "DEDICATED" established:0 refused:0 state:ready37 REMOTE SERVER38 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=11.11.11.1)(PORT=1525)))39 "DEDICATED" established:0 refused:0 state:ready40 LOCAL SERVER41The command completed successfully已经没有节点 2 的监听服务存在了。再次尝试开启节点 2 的集群服务：
1[grid@mesdb2 ~]$ which crsctl2/u01/app/19.3.0/grid/bin/crsctl3[grid@mesdb2 ~]$ exit4logout5[root@mesdb2 ~]# /u01/app/19.3.0/grid/bin/crsctl start cluster6CRS-2672: Attempting to start 'ora.crsd' on 'mesdb2'7CRS-2676: Start of 'ora.crsd' on 'mesdb2' succeeded集群开启很顺利，问题解决。
写在最后
这次问题的解决过程告诉我们，表象并不总是反映真正的原因，我们需要透过现象深入分析，最终才能找到问题的根源和有效的解决方案！
ora.storage： 这真不是我的锅啊！
当然，这次的宕机问题到这其实仍未完全解决，尽管集群已恢复正常，但节点 2 的实例崩溃的原因以及如何恢复的问题仍需进一步探讨。我将在下一篇文章中继续分享。