author：skate
time：2011/12/01
环境：
os：CentOS release 5.5
db: 11.2.0.2.0 - 64bit Production
physical dataguard
现象：只是报ora-00600: 内部错误代码, 参数: [4415]；数据库还是可以正常提供服务
诊断：
这套oracle11g的DG已经运行一段时间了，在调整了些小问题后，目前运行还是比较稳定，今天
在primary库收到报警，在alertlog里的错误信息如下：
Wed Nov 30 17:58:28 2011
Archived Log entry 1547 added for thread 1 sequence 7842 ID 0xa47b04d0 dest 1:
Wed Nov 30 18:21:33 2011
Media Recovery Waiting for thread 1 sequence 7844
Wed Nov 30 18:21:34 2011
RFS[5]: Selected log 7 for thread 1 sequence 7844 dbid -1535431216 branch 760042576
Recovery of Online Redo Log: Thread 1 Group 7 Seq 7844 Reading mem 0
Mem# 0: /oracle/oradata/skatedb/sdbyredo07.log
Wed Nov 30 18:21:41 2011
Archived Log entry 1548 added for thread 1 sequence 7843 ID 0xa47b04d0 dest 1:
Wed Nov 30 18:28:20 2011
Errors in file /oracle/app/diag/rdbms/skate03/skatedb/trace/skatedb_ora_31048.trc (incident=56617):
ora-00600: 内部错误代码, 参数: [4415], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/diag/rdbms/skate03/skatedb/incident/incdir_56617/skatedb_ora_31048_i56617.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
System State dumped to trace file /oracle/app/diag/rdbms/skate03/skatedb/incident/incdir_56617/skatedb_ora_31048_i56617.trc
Wed Nov 30 18:29:19 2011
Dumping diagnostic data in directory=[cdmp_20111130182919], requested by (instance=1, osid=31048), summary=[incident=56617].
Wed Nov 30 18:29:25 2011
Errors in file /oracle/app/diag/rdbms/skate03/skatedb/trace/skatedb_ora_31048.trc (incident=56618):
ora-00603: ORACLE 服务器会话因致命错误而终止
ora-00600: 内部错误代码, 参数: [4415], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/app/diag/rdbms/skate03/skatedb/incident/incdir_56618/skatedb_ora_31048_i56618.trc
Dumping diagnostic data in directory=[cdmp_20111130182926], requested by (instance=1, osid=31048), summary=[incident=56618].
opiodr aborting process unknown ospid (31048) as a result of ora-603
Wed Nov 30 18:29:29 2011
Sweep [inc][56618]: completed
Sweep [inc][56617]: completed
Sweep [inc2][56618]: completed
Sweep [inc2][56617]: completed
Wed Nov 30 18:44:01 2011
RFS[5]: Selected log 8 for thread 1 sequence 7845 dbid -1535431216 branch 760042576
Wed Nov 30 18:44:20 2011
Archived Log entry 1549 added for thread 1 sequence 7844 ID 0xa47b04d0 dest 1:
Wed Nov 30 18:44:20 2011
Media Recovery Waiting for thread 1 sequence 7845 (in transit)
Recovery of Online Redo Log: Thread 1 Group 8 Seq 7845 Reading mem 0
Mem# 0: /oracle/oradata/skatedb/sdbyredo08.log
Wed Nov 30 19:08:17 2011
查看trace文件
[root@skatedb03 ~]# more /oracle/app/diag/rdbms/skate03/skatedb/incident/incdir_56618/skatedb_ora_31048_i56618.trc
Dump file /oracle/app/diag/rdbms/skate03/skatedb/incident/incdir_56618/skatedb_ora_31048_i56618.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/app/oracle/product/11.2.0/db_1
System name: Linux
Node name: skatedb03
Release: 2.6.18-194.el5
Version: #1 SMP Fri Apr 2 14:58:14 EDT 2010
Machine: x86_64
Instance name: skatedb
Redo thread mounted by this instance: 1
Oracle process number: 77
Unix process pid: 31048, image: oracle@skatedb03
*** 2011-11-30 18:29:25.696
*** SESSION ID:(1261.4657) 2011-11-30 18:29:25.696
*** CLIENT ID:() 2011-11-30 18:29:25.696
*** SERVICE NAME:(SYS$USERS) 2011-11-30 18:29:25.696
*** MODULE NAME:(PL/SQL Developer) 2011-11-30 18:29:25.696
*** ACTION NAME:(SQL Window - select * from STATS$DATABASE_INSTANCE select * from) 2011-11-30 18:29:25.696
Dump continued from file: /oracle/app/diag/rdbms/skate03/skatedb/trace/skatedb_ora_31048.trc
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-00600: 内部错误代码, 参数: [4415], [], [], [], [], [], [], [], [], [], [], []
========= Dump for incident 56618 (ORA 603) ========
*** 2011-11-30 18:29:25.697
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+36 call kgdsdst() 000000000 ? 000000000 ?
7FFF58395F98 ? 000000001 ?
000000001 ? 000000002 ?
ksedst1()+98 call skdstdst() 000000000 ? 000000000 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
ksedst()+34 call ksedst1() 000000000 ? 000000001 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
dbkedDefDump()+2741 call ksedst() 000000000 ? 000000001 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
ksedmp()+36 call dbkedDefDump() 000000003 ? 000000002 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
ksfdmp()+64 call ksedmp() 000000003 ? 000000002 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
dbgexPhaseII()+1764 call ksfdmp() 000000003 ? 000000002 ?
7FFF58395F98 ? 000000001 ?
000000000 ? 000000002 ?
dbgexProcessError() call dbgexPhaseII() 2B8F76409718 ? 2B8F767AA6A8 ?
+2651 7FFF583A2310 ? 000000001 ?
000000000 ? 000000002 ?
dbgeExecuteForError call dbgexProcessError() 2B8F76409718 ? 2B8F767AA6A8 ?
()+83 000000001 ? 000000000 ?
100000000 ? 000000002 ?
dbgePostErrorKGE()+ call dbgeExecuteForError 2B8F76409718 ? 2B8F767AA6A8 ?
2131 () 000000001 ? 000000001 ?
000000000 ? 000000002 ?
dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 000000000 ? 2B8F768412A0 ?
63 00000025B ? 2B8F767AA6A8 ?
100000000 ? 000000002 ?
kgeade()+351 call dbkePostKGE_kgsf() 00B7D7EA0 ? 2B8F768412A0 ?
00000025B ? 2B8F767AA6A8 ?
100000000 ? 000000002 ?
kgefec()+171 call kgeade() 00B7D7EA0 ? 00B7D8050 ?
2B8F768412A0 ? 00000025B ?
100000000 ? 000000002 ?
ktc_die()+124 call kgefec() 00B7D7EA0 ? 2B8F768412A0 ?
000000000 ? 000000001 ?
100000000 ? 000000002 ?
k2send()+7723 call ktc_die() 00B7D7EA0 ? 2B8F768412A0 ?
000000000 ? 000000001 ?
100000000 ? 000000002 ?
xctRollbackTxn()+52 call k2send() 000000003 ? 2B8F768412A0 ?
6 000000000 ? 000000000 ?
7FFF583A4050 ? 7FFF583A40B4 ?
kpoltxen()+158 call xctRollbackTxn() 000000000 ? 009949E30 ?
00000062E ? 000000000 ?
7FFF583A4050 ? 7FFF583A40B4 ?
kpotxen()+2880 call kpoltxen() 000000000 ? 009949E30 ?
00000062E ? 000000000 ?
7FFF583A4050 ? 7FFF583A40B4 ?
opiodr()+910 call kpotxen() 000000000 ? 009949E30 ?
7FFF583A70E0 ? 000000000 ?
7FFF583A4050 ? 7FFF583A40B4 ?
ttcpip()+2289 call opiodr() 000000068 ? 00000000C ?
7FFF583A70E0 ? 000000000 ?
0098AE420 ? 7FFF583A40B4 ?
opitsk()+1665 call ttcpip() 00B7EDB10 ? 009247700 ?
7FFF583A70E0 ? 000000000 ?
7FFF583A6B40 ? 7FFF583A72D8 ?
opiino()+961 call opitsk() 00B7EDB10 ? 000000001 ?
7FFF583A70E0 ? 000000000 ?
7FFF583A6B40 ? 7FFF583A72D8 ?
opiodr()+910 call opiino() 00000003C ? 000000004 ?
7FFF583A8868 ? 000000000 ?
7FFF583A6B40 ? 7FFF583A72D8 ?
opidrv()+565 call opiodr() 00000003C ? 000000004 ?
7FFF583A8868 ? 000000000 ?
0098ADD40 ? 7FFF583A72D8 ?
sou2o()+98 call opidrv() 00000003C ? 000000004 ?
7FFF583A8868 ? 000000000 ?
0098ADD40 ? 7FFF583A72D8 ?
opimai_real()+128 call sou2o() 7FFF583A8840 ? 00000003C ?
000000004 ? 7FFF583A8868 ?
0098ADD40 ? 7FFF583A72D8 ?
ssthrdmain()+252 call opimai_real() 000000002 ? 7FFF583A8A30 ?
000000004 ? 7FFF583A8868 ?
0098ADD40 ? 7FFF583A72D8 ?
main()+196 call ssthrdmain() 000000002 ? 7FFF583A8A30 ?
000000001 ? 000000000 ?
0098ADD40 ? 7FFF583A72D8 ?
__libc_start_main() call main() 000000002 ? 7FFF583A8BD8 ?
+244 000000001 ? 000000000 ?
0098ADD40 ? 7FFF583A72D8 ?
_start()+36 call __libc_start_main() 000A078B8 ? 000000002 ?
7FFF583A8BC8 ? 000000000 ?
0098ADD40 ? 000000002 ?
........
.......
从trace文件可以看到：
*** MODULE NAME:(PL/SQL Developer) 2011-11-30 18:29:25.696
*** ACTION NAME:(SQL Window - select * from STATS$DATABASE_INSTANCE select * from) 2011-11-30 18:29:25.696
这说明是人为操作的，执行了一个select查询触发了这个故障。
到metlink查找，发现这个是bug 9531380，触发这个bug，可能是如下原因：
A db-link created on the primary and is propageted to the standby.A synonym on the primay using the dblink is progated to the standby.Opening the standby in READ ONLY and selecting the synonym is fine.Using a commit afterwards fails with ORA-600 [4415]
翻译：
在主库通过dblink访问standby库的同义词，并采用了commit的操作就会触发这个bug。
而在trace的stack里没有看到commit相关操作，而是看到了函数xctRollbackTxn()，说明
有rollback操作，因为在read only的standby库是不存在事务的，执行事务操作是有问题的。
从时间上看，那个时间是我在调试程序，我查了远程standby库的同义词，并rollback了。
从tracelog来看，rollback也会触发这个600的错误。这个错误据我的故障来说，对库
没有什么影响，如果是人为触发的，可以不用管它。oracle在oracle11.2.0.3之后修复了这个bug。
-----end----