In this blog post, we would see how to remove quorum disks in Exadata machine. We have one 1/8th Rack Exadata machine for Lab activity where ran into missing quorum disks for Voting Disk and I had to fix that in order to get the system patched in Rolling fashion to maintain high availability.
To patch the Exadata system in rolling fashion, it requires voting disks to reside on HIGH redundancy diskgroup which is not possible in 1/8th Rack Exadata system since there is only 3 cell nodes in there, and for High redundancy it requires five copies of voting disk.
So, in this scenario - we add a separate HDD drive on both compute nodes and share that between the nodes to be used for High redundancy diskgroup creation - there it comes the term "Quorum Disk"
We have +DATAC1 diskgroup configured with HIGH redundancy where our voting disk resides but two of the quorum disks being used in there went missing so our voting disks had only three copies available and CRS was not coming up.
I had to remove the quorum disks from both compute nodes and then configure it back to fix the issue.
SQL> l 1* select disk_number, GROUP_NUMBER, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, STATE, path from v$asm_disk where GROUP_NUMBER=1 SQL> / DISK_NUMBER GROUP_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE PATH ----------- ------------ ------- ------------ ------- -------- ---------------------------------------------------------------------- 2 1 MISSING UNKNOWN OFFLINE NORMAL 3 1 MISSING UNKNOWN OFFLINE NORMAL 14 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_05_exad02cel03 13 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_00_exad02cel03 15 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_02_exad02cel03 7 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03 8 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_04_exad02cel03 0 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.1;10.0.0.2/DATAC1_CD_01_exad02cel03 4 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02 16 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_02_exad02cel02 17 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_01_exad02cel02 5 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_00_exad02cel02 6 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_03_exad02cel02 11 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.3;10.0.0.4/DATAC1_CD_04_exad02cel02 19 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.5;10.0.0.6/DATAC1_CD_04_exad02cel01 1 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01 18 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.5;10.0.0.6/DATAC1_CD_00_exad02cel01 9 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.5;10.0.0.6/DATAC1_CD_03_exad02cel01 12 1 CACHED MEMBER ONLINE NORMAL o/10.0.0.5;10.0.0.6/DATAC1_CD_01_exad02cel01
Due to missing quorum disk in the DATAC1 diskgroup, we have only three copies of voting disks available as of now.
[root@exa01dbadm02 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 94acaa401bd34fe0bf261af4437fc75e (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
2. ONLINE 7680d9f206544ff1bf27661bf8cc3250 (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
3. ONLINE b632f75fe4884f5cbf3627faa6bbbe07 (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
Located 3 voting disk(s).
[root@exa01dbadm02 ~]#I moved the voting disks to +RECOC1 diskgroup which was configured with NORMAL REDUNDANCY.
[oragrid@exa01dbadm01 ~]$ crsctl replace votedisk +RECOC1
Successful addition of voting disk 783e7633ea934fd0bff78c8b8dbe5a66.
Successful addition of voting disk a5cf9e2d99134f69bf6bb15c98044623.
Successful addition of voting disk d5d20cd6ff3e4fcdbf68df28cb18d030.
Successful deletion of voting disk 94acaa401bd34fe0bf261af4437fc75e.
Successful deletion of voting disk 7680d9f206544ff1bf27661bf8cc3250.
Successful deletion of voting disk b632f75fe4884f5cbf3627faa6bbbe07.
Successful deletion of voting disk e47d60407b054f3bbf5dc07df8080ee6.
Successfully replaced voting disk group with +RECOC1.
CRS-4266: Voting file(s) successfully replaced
[oragrid@exa01dbadm01 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 783e7633ea934fd0bff78c8b8dbe5a66 (o/10.0.0.5;10.0.0.6/RECOC1_CD_02_exad02cel01) [RECOC1]
2. ONLINE a5cf9e2d99134f69bf6bb15c98044623 (o/10.0.0.3;10.0.0.4/RECOC1_CD_02_exad02cel02) [RECOC1]
3. ONLINE d5d20cd6ff3e4fcdbf68df28cb18d030 (o/10.0.0.1;10.0.0.2/RECOC1_CD_02_exad02cel03) [RECOC1]
Located 3 voting disk(s).
[oragrid@exa01dbadm01 ~]$Now we can go ahead and drop the missing quorum disks from the DATAC1 diskgroup forcefully.
[oragrid@exa01dbadm01 ~]$ sqlplus "/as sysasm"
SQL*Plus: Release 12.2.0.1.0 Production on Thu Jul 30 12:06:27 2020
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL>
SQL> set lines 300
SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm01 FORCE;
Diskgroup altered.
SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm02 FORCE;
Diskgroup altered.
SQL>Delete the quorum disk device from the node1 using quorumdiskmgr utility.
[root@exa01dbadm01 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.
Delete the quorum disk target from the node1 using quorumdiskmgr utility.
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm01
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1
Now delete quorum disk configuration from the node1.
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --config
[Success] Successfully deleted quorum disk configurations
[root@exa01dbadm01 oracle.SupportTools]#Now, delete the quorum disk device from node2.
[root@exa01dbadm02 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.
Delete quorum disk target from node2.
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm02
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1
Delete quorum disk configuration from node 2.
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --config
[Success] Successfully deleted quorum disk configurations
[root@exa01dbadm02 oracle.SupportTools]#At this point, we are done with removing the quorum disk configuration from our Exadata System, in next post, I would detail how we configure it in the system.
Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.
Twitter : https://twitter.com/rajsoft8899
Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/
Facebook : https://www.facebook.com/rkushawaha
