Issue: ASM diskgroup was forcibly dismounted during an online EMC SAN storage change

ORA-00204: error in reading (block 32, # blocks 1) of control file

ORA-00202: control file: ‘+DATA/ostdp/controlfile/current.258.935405159’

ORA-15081: failed to submit an I/O operation to a disk

WARNING: Read Failed. group:1 disk:10 AU:3 offset:540672 size:16384

WARNING: failed to read mirror side 1 of virtual extent 3 logical extent 0 of file 258 in group [1.3486014431]

from disk DATA19  allocation unit 3 reason error; if possible, will try another mirror side

Errors in file /u00/app/oracle/diag/rdbms/ostdp/ostdp/trace/ostdp_arc1_32990.trc:

ORA-00202: control file: ‘+DATA/ostdp/controlfile/current.258.935405159’

ORA-15081: failed to submit an I/O operation to a disk

SUCCESS: diskgroup DATA was dismounted

SUCCESS: diskgroup DATA was dismounted

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: ‘+DATA/ostdp/controlfile/current.258.935405159’

ORA-15078: ASM diskgroup was forcibly dismounted

ORA-15078: ASM diskgroup was forcibly dismounted Errors in file /u00/app/oracle/diag/rdbms/ostdp/ostdp/trace/ostdp_ckpt_32852.trc:

ORA-00221: error on write to control file

ORA-00206: error in writing (block 3, # blocks 1) of control file

ORA-00202: control file: ‘+DATA/ostdp/controlfile/current.258.935405159’

ORA-15078: ASM diskgroup was forcibly dismounted

ORA-15078: ASM diskgroup was forcibly dismounted CKPT (ospid: 32852): terminating the instance due to error 221

high water mark = 569 Instance terminated by CKPT, pid = 32852 USER (ospid: 94619): terminating the instance Instance terminated by USER, pid = 94619

/etc/multipath.conf

/etc/sysconfig/oracleasm

/var/log/messages

1) There was an activity at Storage side , This Change is related to Hardware DIMM Replacement

2) Description  DIMM Replacement – DIMM 15 of Director 3 of ENG-2 in System Bay 2 CE to replace the director 3D.

It is online activity and will be done by EMC engineering team.

3) This is online change as per storage team, None of the servers effected except DB prod server

4) One of the SAN was brought down as per the storage change. The time it was down, our Production DB went down

due to DATA disk groups were dismounted.

5) The Reason for DATA Disk group went down:

Multipath Issue at ASM side where ASM disks are not configured to use Multipath to failover whenever there is

any SAN link down during this kind of change.

6) The file:/etc/sysconfig/oracleasm shows following details –

ORACLEASM_SCANORDER=””

ORACLEASM_SCANEXCLUDE=””

We see that Multipathing device is not set in parameter “ORACLEASM_SCANORDER”.

Due to this ASMLIB was still using single path instead of Multipath device.

and hence when one link went down, it did not use the multipath and there were I/O errors and the DATA diskgroup got dismounted.

1)Edit the file /etc/sysconfig/oracleasm file and change the following parameters to below values –

               ORACLEASM_SCANORDER=”dm”

               ORACLEASM_SCANEXCLUDE=”sd”

2)Shutdown all database(s) that use the ASM Instance and then shutdown the ASM Instance.

3)Once this is done issue the following command as the root user =

#/etc/init.d/oracleasm stop             #/etc/init.d/oracleasm start

4)Start the ASM Instance and the database(s).