Oracle Corner

After a RAC node reboot, one of the ASM disks didn't get mounted.

We could see the disk (partition) /dev/> with "fdisk -l", but for some reason ASM either couldn't see it, or couldn't recognise it as a ASM disk.

We looked at the ASM header (with kfed command). The header was correct. We checked the other blocks of the disk, they were all OK. Then we noticed that ASM label was empty, and it's not supposed to be.

NOTE: Disk ORADATA2_0000 in mode 0x7f marked for de-assignment

NOTE: Disk  in mode 0x7f marked for de-assignment

ERROR: diskgroup ORADATA2 was not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk "1" is missing from group number "3"

2015-04-09 14:13:21.421: [ default]failed to initialize skgp context

2015-04-09 14:13:21.422: [ default]slos op  :  sslssreghdlr

2015-04-09 14:13:21.422: [ default]slos dep :  Error 0 (0)

2015-04-09 14:13:21.422: [ default]slos loc :  sskgpinit1

2015-04-09 14:13:21.422: [ default]slos info:[   CLWAL]clsw_Initialize: OLR initlevel [30000]

2015-04-09 14:13:21.423: [ default]a_init: Unable to get log name. Retval:[-4]

ASM disk lost the ASM label.

List the label:

$ asmtool -I -l "/dev/oracleasm" -n /dev/> -a label  

     asmtool: Disk "/dev/>" is marked an ASM disk with the label ""  

As you can see, the label is "" which is wrong. It should be "ORADATA22"

Stop the RAC on that node, change the label.

su - root

[root@bin]# crsctl stop crs

[root@bin]# /etc/init.d/oracleasm force-renamedisk /dev/> ORADATA22 

[root@bin]# asmtool -I -l "/dev/oracleasm" -n /dev/> -a label  

asmtool: Disk "/dev/>" is marked an ASM disk with the label "ORADATA22"

[root@bin]# oracleasm scandisks

We needed to change the label on the second node, too.