在知识库中专注阅读，并随时返回相关工具与课程
最近部署了一套 Oracle 19C RAC 环境，安装完成后关机打包发往异地机房。上架后发现其中一个节点无法正常启动，集群卡在 cssd 资源启动阶段。心跳网络采用光纤连接，通过交换机的独立 VLAN，节点间可以正常 ping 通，但是 RAC 只能单节点运行。经过深入分析，最终定位到是 MTU 配置问题导致的故障。 本文记录详细的问题分析过程和解决方案。 Oracle RAC 中一个节点启动失
本章目标：掌握本章核心知识点 前置要求：完成前序章节学习 预计时长：60 分钟
最近部署了一套 Oracle 19C RAC 环境，安装完成后关机打包发往异地机房。上架后发现其中一个节点无法正常启动，集群卡在 cssd 资源启动阶段。心跳网络采用光纤连接，通过交换机的独立 VLAN，节点间可以正常 ping 通，但是 RAC 只能单节点运行。经过深入分析，最终定位到是 MTU 配置问题导致的故障。
本文记录详细的问题分析过程和解决方案。
Oracle RAC 中一个节点启动失败，集群状态显示：
1[root@orcl02:/root]# crsctl stat res -t -init2--------------------------------------------------------------------------------3Name Target State Server State details 4--------------------------------------------------------------------------------5Cluster Resources6--------------------------------------------------------------------------------7ora.asm8 1 ONLINE OFFLINE STABLE9ora.cluster_interconnect.haip10 1 ONLINE OFFLINE STABLE11ora.crf12 1 ONLINE ONLINE orcl02 STABLE13ora.crsd14 1 ONLINE OFFLINE STABLE15ora.cssd16 1 ONLINE OFFLINE orcl02 STARTING17ora.cssdmonitor18 1 ONLINE ONLINE orcl02 STABLE19ora.ctssd20 1 ONLINE OFFLINE STABLE21ora.diskmon22 1 OFFLINE OFFLINE STABLE23ora.drivers.acfs24 1 ONLINE ONLINE orcl02 STABLE25ora.evmd26 1 ONLINE INTERMEDIATE orcl02 STABLE27ora.gipcd28 1 ONLINE ONLINE orcl02 STABLE29ora.gpnpd30 1 ONLINE ONLINE orcl02 STABLE31ora.mdnsd32 1 ONLINE ONLINE orcl02 STABLE33ora.storage34 1 ONLINE OFFLINE STABLE集群在 cssd 资源启动阶段 HANG 住。
检查 CRS 日志 ($ORACLE_BASE/diag/crs/orcl02/crs/trace/alert.log)：
12025-11-11 12:21:57.020 [GIPCD(127382)]CRS-7517: The Oracle Grid Interprocess Communication (GIPC) failed to identify the Fast Node Death Detection (FNDD).232025-11-11 12:23:07.241 [OCSSD(130761)]CRS-1621: The IPMI configuration data for this node stored in the Oracle registry is incomplete; details at (:CSSNK00002:) in /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc42025-11-11 12:23:07.241 [OCSSD(130761)]CRS-1617: The information required to do node kill for node orcl02 is incomplete; details at (:CSSNM00004:) in /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc52025-11-11 12:23:37.324 [OCSSD(130761)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'orcl01'.672025-11-11 12:28:39.020 [OCSSD(130761)]CRS-7500: The Oracle Grid Infrastructure process 'ocssd' failed to establish Oracle Grid Interprocess Communication (GIPC) high availability connection with remote node 'orcl01'.892025-11-11 12:32:58.470 [OCSSD(130761)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc.102025-11-11 12:32:58.469 [CSSDAGENT(130651)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:5:4} in /u01/app/grid/diag/crs/orcl02/crs/trace/ohasd_cssdagent_root.trc.112025-11-11 12:32:58.506 [OHASD(126708)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00221:) {0:5:4} in /u01/app/grid/diag/crs/orcl02/crs/trace/ohasd.trc.122025-11-11 12:32:59.470 [OCSSD(130761)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc132025-11-11 12:32:59.470 [OCSSD(130761)]CRS-1603: CSSD on node orcl02 has been shut down.142025-11-11 12:33:00.151 [OCSSD(130761)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00086:) in /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc.152025-11-11T12:33:04.480316+08:0016Errors in file /u01/app/grid/diag/crs/orcl02/crs/trace/ocssd.trc (incident=17):17CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []18Incident details in: /u01/app/grid/diag/crs/orcl02/crs/incident/incdir_17/ocssd_i17.trc19202025-11-11 12:33:04.471 [OCSSD(130761)]CRS-8503: Oracle Clusterware process OCSSD with operating system process ID 130761 experienced fatal signal or exception code 6.日志明确显示节点 orcl02 无法与节点 orcl01 建立 GIPC 高可用连接。
进一步分析 cssd 日志 ($ORACLE_BASE/diag/crs/orcl02/crs/trace/ocssd.trc)：
12025-11-11 13:07:10.563 : CSSD:909854464: [ INFO] clssnmvDHBValidateNCopy: node 1, orcl01, has a disk HB, but no network HB, DHB has rcfg 658329638, wrtcnt, 36406, LATS 5334194, lastSeqNo 36403, uniqueness 1762834827, timestamp 1762837626/532312422025-11-11 13:07:10.564 : CSSD:897136384: [ INFO] clssscSelect: gipcwait returned with status gipcretTimeout (16)日志表明心跳网络存在通信问题。
首先检查基础网络连通性：
1[root@orcl02:/tmp/mcasttest]# ping orcl01-priv2PING orcl01-priv (1.1.1.1) 56(84) bytes of data.364 bytes from orcl01-priv (1.1.1.1): icmp_seq=1 ttl=64 time=0.053 ms464 bytes from orcl01-priv (1.1.1.1): icmp_seq=2 ttl=64 time=0.044 ms564 bytes from orcl01-priv (1.1.1.1): icmp_seq=3 ttl=64 time=0.116 ms心跳 IP 可以正常 ping 通，防火墙已关闭。
查询 MOS 文档，发现类似案例：OCI DBCS : Failed to start CRS on first RAC node - (GIPC) failed to identify the Fast Node Death Detection (FNDD). (Doc ID 2969313.1)
该案例指出问题根源在于节点间 MTU 配置不一致：
检查两个节点的 MTU 配置：
1## 节点12[root@orcl01:/home/grid]# ifconfig bond1|grep mtu3bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 900045## 节点26[root@orcl02:/home/grid]$ ifconfig bond1|grep mtu7bond1: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 9000节点间的 MTU 配置一致，均为 9000。
使用 Oracle Cluster Verification Utility (CVU) 验证一下 RAC 集群节点间的网络连通性：
1[grid@orcl01:/home/grid]$ cluvfy comp nodecon -n all -verbose23Performing following verification checks ...45 Node Connectivity ...6 Hosts File ...7 Node Name Status 8 ------------------------------------ ------------------------9 orcl01 passed 10 orcl02 passed 11 Hosts File ...PASSED1213Interface information for node "orcl02"1415 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 16 ------ --------------- --------------- --------------- --------------- ----------------- ------17 bond0 192.168.6.206 192.168.6.0 0.0.0.0 192.168.6.1 8C:84:74:75:EF:00 1500 18 bond1 1.1.1.2 1.1.1.0 0.0.0.0 192.168.6.1 8C:84:74:DA:B8:E0 9000 1920Interface information for node "orcl01"2122 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 23 ------ --------------- --------------- --------------- --------------- ----------------- ------24 bond0 192.168.6.205 192.168.6.0 0.0.0.0 192.168.6.1 8C:84:74:75:F2:00 1500 25 bond0 192.168.6.207 192.168.6.0 0.0.0.0 192.168.6.1 8C:84:74:75:F2:00 1500 26 bond0 192.168.6.209 192.168.6.0 0.0.0.0 192.168.6.1 8C:84:74:75:F2:00 1500 27 bond0 192.168.6.208 192.168.6.0 0.0.0.0 192.168.6.1 8C:84:74:75:F2:00 1500 28 bond1 1.1.1.1 1.1.1.0 0.0.0.0 192.168.6.1 8C:84:74:DA:D3:10 9000 2930Check: MTU consistency on the private interfaces of subnet "1.1.1.0"3132 Node Name IP Address Subnet MTU 33 ---------------- ------------ ------------ ------------ ----------------34 orcl02 bond1 1.1.1.2 1.1.1.0 9000 35 orcl01 bond1 1.1.1.1 1.1.1.0 9000 3637Check: MTU consistency of the subnet "192.168.6.0".3839 Node Name IP Address Subnet MTU 40 ---------------- ------------ ------------ ------------ ----------------41 orcl02 bond0 192.168.6.206 192.168.6.0 1500 42 orcl01 bond0 192.168.6.205 192.168.6.0 1500 43 orcl01 bond0 192.168.6.207 192.168.6.0 1500 44 orcl01 bond0 192.168.6.209 192.168.6.0 1500 45 orcl01 bond0 192.168.6.208 192.168.6.0 1500 4647 Source Destination Connected? 48 ------------------------------ ------------------------------ ----------------49 orcl01[bond0:192.168.6.205] orcl02[bond0:192.168.6.206] yes 50 orcl01[bond0:192.168.6.205] orcl01[bond0:192.168.6.207] yes 51 orcl01[bond0:192.168.6.205] orcl01[bond0:192.168.6.209] yes 52 orcl01[bond0:192.168.6.205] orcl01[bond0:192.168.6.208] yes 53 orcl02[bond0:192.168.6.206] orcl01[bond0:192.168.6.207] yes 54 orcl02[bond0:192.168.6.206] orcl01[bond0:192.168.6.209] yes 55 orcl02[bond0:192.168.6.206] orcl01[bond0:192.168.6.208] yes 56 orcl01[bond0:192.168.6.207] orcl01[bond0:192.168.6.209] yes 57 orcl01[bond0:192.168.6.207] orcl01[bond0:192.168.6.208] yes 58 orcl01[bond0:192.168.6.209] orcl01[bond0:192.168.6.208] yes 5960 Source Destination Connected? 61 ------------------------------ ------------------------------ ----------------62 orcl01[bond1:1.1.1.1] orcl02[bond1:1.1.1.2] yes 63 Check that maximum (MTU) size packet goes through subnet ...FAILED (PRVG-12885, PRVG-12884, PRVG-2043)64 subnet mask consistency for subnet "192.168.6.0" ...PASSED65 subnet mask consistency for subnet "1.1.1.0" ...PASSED66 Node Connectivity ...FAILED (PRVG-12885, PRVG-12884, PRVG-2043)67 Multicast or broadcast check ...68Checking subnet "1.1.1.0" for multicast communication with multicast group "224.0.0.251"69 Multicast or broadcast check ...PASSED7071Verification of node connectivity was unsuccessful on all the specified nodes. 727374Failures were encountered during execution of CVU verification request "node connectivity".7576Node Connectivity ...FAILED77 Check that maximum (MTU) size packet goes through subnet ...FAILED78 PRVG-12885 : ICMP packet of MTU size "9000" does not go through subnet79 "1.1.1.0".80 PRVG-12884 : Maximum (MTU) size packet check failed on subnets "1.1.1.0"8182 orcl01: PRVG-2043 : Command "/bin/ping 1.1.1.2 -c 1 -w 3 -M do -s83 8972 " failed on node "orcl01" and produced the following84 output:85 PING 1.1.1.2 (1.1.1.2) 8972(9000) bytes of data.8687 --- 1.1.1.2 ping statistics ---88 3 packets transmitted, 0 received, 100% packet loss, time89 2074ms909192939495CVU operation performed: node connectivity96Date: Nov 11, 2025 1:10:21 PM97CVU version: 19.28.0.0.0 (070125x8664)98Clusterware version: 19.0.0.0.099CVU home: /u01/app/19.3.0/grid100Grid home: /u01/app/19.3.0/grid101User: grid102Operating system: Linux4.18.0-553.el8_10.x86_64CVU 检测发现 MTU 为 9000 的数据包无法通过子网：
1Node Connectivity ...FAILED2 Check that maximum (MTU) size packet goes through subnet ...FAILED3 PRVG-12885 : ICMP packet of MTU size "9000" does not go through subnet4 "1.1.1.0".5 PRVG-12884 : Maximum (MTU) size packet check failed on subnets "1.1.1.0"67 orcl01: PRVG-2043 : Command "/bin/ping 1.1.1.2 -c 1 -w 3 -M do -s8 8972 " failed on node "orcl01" and produced the following9 output:10 PING 1.1.1.2 (1.1.1.2) 8972(9000) bytes of data.1112 --- 1.1.1.2 ping statistics ---13 3 packets transmitted, 0 received, 100% packet loss, time14 2074ms使用 /bin/ping 1.1.1.2 -c 1 -w 3 -M do -s 8972 进行 MTU 大小测试：
1# 测试 9000 MTU - 失败2[grid@orcl01:/home/grid]$ ping 1.1.1.2 -c 3 -M do -s 89723PING 1.1.1.2 (1.1.1.2) 8972(9000) bytes of data.4^C5--- 1.1.1.2 ping statistics ---62 packets transmitted, 0 received, 100% packet loss, time 1021ms78# 测试 1500 MTU - 成功9[grid@orcl01:/home/grid]$ ping 1.1.1.2 -c 3 -M do -s 147210PING 1.1.1.2 (1.1.1.2) 1472(1500) bytes of data.111480 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=0.177 ms121480 bytes from 1.1.1.2: icmp_seq=2 ttl=64 time=0.127 ms13^C14--- 1.1.1.2 ping statistics ---152 packets transmitted, 2 received, 0% packet loss, time 1032ms16rtt min/avg/max/mdev = 0.127/0.152/0.177/0.025 ms测试结果表明：MTU 1500 通信正常，但 MTU 9000 出现 100% 丢包。经与网络工程师确认，交换机未配置 Jumbo Frame 支持，破案了！
要求网络工程师在交换机上启用 Jumbo Frame 支持，配置 MTU 大于 9000。交换机配置完成后，重新测试：
1[grid@orcl01:/home/grid]$ /bin/ping 1.1.1.2 -c 1 -w 3 -M do -s 89722PING 1.1.1.2 (1.1.1.2) 8972(9000) bytes of data.38980 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=0.232 ms45--- 1.1.1.2 ping statistics ---61 packets transmitted, 1 received, 0% packet loss, time 0ms7rtt min/avg/max/mdev = 0.232/0.232/0.232/0.000 msMTU 9000 通信测试成功。
重启两个节点的集群服务：
1crsctl stop crs -f2crsctl start crs检查集群状态：
1[root@orcl01:/soft]# crsctl stat res -t 2--------------------------------------------------------------------------------3Name Target State Server State details 4--------------------------------------------------------------------------------5Local Resources6--------------------------------------------------------------------------------7ora.LISTENER.lsnr8 ONLINE ONLINE orcl01 STABLE9 ONLINE ONLINE orcl02 STABLE10ora.chad11 ONLINE ONLINE orcl01 STABLE12 ONLINE ONLINE orcl02 STABLE13ora.net1.network14 ONLINE ONLINE orcl01 STABLE15 ONLINE ONLINE orcl02 STABLE16ora.ons17 ONLINE ONLINE orcl01 STABLE18 ONLINE ONLINE orcl02 STABLE19ora.proxy_advm20 OFFLINE OFFLINE orcl01 STABLE21 OFFLINE OFFLINE orcl02 STABLE22--------------------------------------------------------------------------------23Cluster Resources24--------------------------------------------------------------------------------25ora.ARCH.dg(ora.asmgroup)26 1 ONLINE ONLINE orcl01 STABLE27 2 ONLINE ONLINE orcl02 STABLE28ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)29 1 ONLINE ONLINE orcl01 STABLE30 2 ONLINE ONLINE orcl02 STABLE31ora.DATA.dg(ora.asmgroup)32 1 ONLINE ONLINE orcl01 STABLE33 2 ONLINE ONLINE orcl02 STABLE34ora.LISTENER_SCAN1.lsnr35 1 ONLINE ONLINE orcl02 STABLE36ora.OCR.dg(ora.asmgroup)37 1 ONLINE ONLINE orcl01 STABLE38 2 ONLINE ONLINE orcl02 STABLE39ora.asm(ora.asmgroup)40 1 ONLINE ONLINE orcl01 Started,STABLE41 2 ONLINE ONLINE orcl02 Started,STABLE42ora.asmnet1.asmnetwork(ora.asmgroup)43 1 ONLINE ONLINE orcl01 STABLE44 2 ONLINE ONLINE orcl02 STABLE45ora.cvu46 1 ONLINE ONLINE orcl02 STABLE47ora.orcl.db48 1 ONLINE ONLINE orcl01 Open,HOME=/u01/app/o49 racle/product/19.3.050 /db,STABLE51 2 ONLINE ONLINE orcl02 Open,HOME=/u01/app/o52 racle/product/19.3.053 /db,STABLE54ora.qosmserver55 1 ONLINE ONLINE orcl02 STABLE56ora.scan1.vip57 1 ONLINE ONLINE orcl02 STABLE58ora.orcl01.vip59 1 ONLINE ONLINE orcl01 STABLE60ora.orcl02.vip61 1 ONLINE ONLINE orcl02 STABLE62--------------------------------------------------------------------------------集群所有资源正常启动，问题解决。
心跳网卡的 MTU 默认是 1500，交换机的默认 MTU 是 1500，当在系统层面修改网卡配置 MTU 为 9000 之后，交换机没有配置，这时候就会无法进行通信，100% 丢包。
为什么 Oracle 建议配置心跳网卡 MTU 为 9000？
可以参考 MOS 文档：Recommendation for the Real Application Cluster Interconnect and Jumbo Frames (Doc ID 341788.1)
配置巨型帧的优势：
修改私网 MTU 配置可参考 MOS 文档：如何在 oracle 集群环境下修改私网信息 (Doc ID 2103317.1)。
本次故障的根本原因是端到端的 MTU 配置不一致。虽然节点操作系统层面配置了 9000 字节的 MTU，但中间网络设备的 MTU 仍保持默认的 1500 字节，导致大数据包传输失败。