On my 11.2&2 node RAC system, one of the node has been rebooted. While I was investigating issue I noticed LMDO process terminating the instance.

We have below in alert.log on victim node:

LMS2 (ospid: 28770960) received an instance eviction notification from instance 2 [2]
LMON received an instance eviction notification from instance 2
The instance eviction reason is 0x2
The instance eviction map is 1
PMON (ospid: 13435162): terminating the instance due to error 481
/oracle11g/db/diag/rdbms/VIS/rac_node1/trace/rac_node1_diag_19530038.trc

While I was investigating trc file rac_node1_diag_19530038.trc, I noticed below error messages

Enqueue blocker waiting on ‘KSV master wait’ <<<<<<<<<<< This is the error message
Dumping global systemstates of the ASM instances, check DIAG traces under the ASM instances. <<<<<< We need to check ASM log

From trc:

SYSTEM STATE (level=10)
————
System global information:
processes: base 0x700000df8eb8550, size 5000, cleanup 0x700000d90f2e278
allocation: free sessions 0x700000d992170c0, free calls 0x0
control alloc errors: 1092 (process), 1092 (session), 1092 (call)
PMON latch cleanup depth: 0
seconds since PMON’s last scan for dead processes: 44
system statistics:
alert_+ASM1.log
====================
NOTE: ASM client rac_node1:VIS disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Process state recorded in trace file /oracle11g/base/11.2.0/diag/asm/+asm/+ASM1/trace/+ASM1_ora_57016454.trc
Tue Mar 23 09:37:21 2020
NOTE: killing foreground process 45 (10093158) for state cleanup
NOTE: killing foreground process 35 (15335956) for state cleanup
NOTE: killing foreground process 44 (26477216) for state cleanup

I started to check +ASM1_diag_13500576.trc file.

From  trace file:
“Enqueue blocker waiting on ‘KSV master wait’”

After make some search I noticed that I am hitting a known problem.
This problem is due to the next bug:
Bug 11800170 – Many ASM file metadata / KSV master waits [ID 11800170.8]

Details of bug:

Abstract: Many ASM file metadata / KSV master waits

DB sessions may show excessive waits for “ASM file metadata operation” / “KSV master wait” waits for systems NOT running on Exadata dueto calls into ASM to query “cell.smart” data on non-Exadata systems.Such calls are not needed.
Rediscovery Notes: ASM is used. Exadata is NOT used. The waiting DB sessions shows stacks including kcfis_tablespace_is_on_sage().

If you don’t use Exadata than Workaround for this error message is set “_allow_cell_smart_scan_attr” = false which comes True by default.

 

Reklam