I have 11.2.0.1.6 2 node RAC system with ASM on AIX 6.1 64 bit platform. One of the node has been rebooted. While We were investigating issue We noticed LMDO process terminating the instance.

We have below in alert.log on victim node:

LMS2 (ospid: 28770960) received an instance eviction notification from instance 2 [2]
LMON received an instance eviction notification from instance 2
The instance eviction reason is 0x2
The instance eviction map is 1
PMON (ospid: 13435162): terminating the instance due to error 481
/oracle11g/VIS/db/diag/rdbms/VIS/rac_node1/trace/rac_node1_diag_19530038.trc

While We were investigating trc file rac_node1_diag_19530038.trc
=========================
Enqueue blocker waiting on ‘KSV master wait’ <<<<<<<<<<< This is the error message
Dumping global systemstates of the ASM instances, check DIAG traces under the ASM instances. <<<<<< We need to check ASM log

Performing diagnostic data dump for this instance
===================================================
SYSTEM STATE (level=10)
————
System global information:
processes: base 0x700000df8eb8550, size 5000, cleanup 0x700000d90f2e278
allocation: free sessions 0x700000d992170c0, free calls 0x0
control alloc errors: 1092 (process), 1092 (session), 1092 (call)
PMON latch cleanup depth: 0
seconds since PMON’s last scan for dead processes: 44
system statistics:
alert_+ASM1.log
====================
NOTE: ASM client rac_node1:VIS disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Process state recorded in trace file /oracle11g/VIS/base/11.2.0/diag/asm/+asm/+ASM1/trace/+ASM1_ora_57016454.trc
Tue Oct 23 09:37:21 2012
NOTE: killing foreground process 45 (10093158) for state cleanup
NOTE: killing foreground process 35 (15335956) for state cleanup
NOTE: killing foreground process 44 (26477216) for state cleanup

We checked /oracle11g/VIS/base/11.2.0/diag/asm/+asm/+ASM1/trace/+ASM1_diag_13500576.trc

We see the following in the trace file:
“Enqueue blocker waiting on ‘KSV master wait'”

After make some search We noticed that We are hitting a known problem.
This problem is due to the next bug:
Bug 11800170 – Many ASM file metadata / KSV master waits [ID 11800170.8]

Details of bug:
====================
Abstract: Many ASM file metadata / KSV master waits
Prod/Comp: DB 5/RDBMS
Affects: Vers >=11 and BELOW 12.1 Specifically: 11.2.0.1 11.2.0.2
Fixed Releases: 11.2.0.2.3 11.2.0.2.BP08 11.2.0.3 12.1.0.0 WIN:B202P07
Tags: ASM PERF
Hooks:
NEED:ASM “WAITEVENT:ASM file metadata operation”
“WAITEVENT:KSV master wait” STACKHAS:kcfis_tablespace_is_on_sage PARAMETER:_allow_cell_smart_scan_attr

DB sessions may show excessive waits for “ASM file metadata operation” / “KSV master wait” waits for systems NOT running on Exadata dueto calls into ASM to query “cell.smart” data on non-Exadata systems.Such calls are not needed.
Rediscovery Notes: ASM is used. Exadata is NOT used. The waiting DB sessions shows stacks including kcfis_tablespace_is_on_sage().

If you don’t use Exadata than Workaround for this error message is set “_allow_cell_smart_scan_attr” = false which comes True by default.

Advertisements