I have 10.2.0.4 2 node RAC system on AIX 6 Operating system.

Node1 has been crashed wiht those error messages:

Thu Jun 14 12:20:11 2012
Errors in file /oracle10g/app/oracle/admin/PROD/bdump/PROD1_s001_889010.trc:
ORA-07445: exception encountered: core dump [kghsrch+0034] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []
Thu Jun 14 12:20:12 2012


Trace dumping is performing id=[cdmp_20120614122012]
Thu Jun 14 12:20:15 2012
Errors in file /oracle10g/app/oracle/admin/PROD/bdump/PROD1_j000_1859640.trc:
ORA-07445: exception encountered: core dump [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0x2C21504C05B0A36] [] []
Thu Jun 14 12:20:18 2012
Trace dumping is performing id=[cdmp_20120614122016]
Thu Jun 14 12:20:25 2012
Errors in file /oracle10g/app/oracle/admin/PROD/bdump/PROD1_mmon_823446.trc:
ORA-07445: Message 7445 not found; No message file for product=RDBMS, facility=ORA; arguments: [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0x2C21504C05B0A36]
Thu Jun 14 12:20:28 2012
Trace dumping is performing id=[cdmp_20120614122026]
Thu Jun 14 12:20:36 2012
Errors in file /oracle10g/app/oracle/admin/PROD/udump/PROD1_ora_471104.trc:
ORA-07445: exception encountered: core dump [kghlkremf+0024] [SIGSEGV] [Address not mapped to object] [0x2C21504C05B0A36] [] []
ORA-01403: no data found
Thu Jun 14 12:20:36 2012
Errors in file /oracle10g/app/oracle/admin/PROD/bdump/PROD1_pmon_712900.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [kghfrh:ds] [0x70000011AFFF018]
Thu Jun 14 12:20:37 2012
Errors in file /oracle10g/app/oracle/admin/PROD/bdump/PROD1_pmon_712900.trc:

We have been started to investigate error message by checking below note and also trc files:

Troubleshoot an ORA-600 or ORA-7445 Error Using the Error Lookup Tool [ID 153788.1]

After some searching We noticed that the bug 5868257 is the cause for this issue beacuse of Heap memory corruption and ORA-7445 error has been reported.

This error cause&solution mention in below doc:
Smon Process Terminated Because Of ORA-07445 [Kghsrch()+128] [ID 1189894.1]

For can fix that error We need to apply one-off Patch 5868257.

PS:This fix has been included in 10.2.0.4.1 PSU.In this case applying PSU4 (Patch 9352164) has resolved the issue (PSU4 includes PSU1).The bug are solved in 10.2.0.5 and 11.1.0.6.apply patch of upgrade your db.

Advertisements