Moneycontrol Brokerage Recos

Monday, August 31, 2020

Removing the Quorum Disk Configuration in Exadata

 

In this blog post, we would see how to remove quorum disks in Exadata machine. We have one 1/8th Rack Exadata machine for Lab activity where ran into missing quorum disks for Voting Disk and I had to fix that in order to get the system patched in Rolling fashion to maintain high availability.

To patch the Exadata system in rolling fashion, it requires voting disks to reside on HIGH redundancy diskgroup which is not possible in 1/8th Rack Exadata system since there is only 3 cell nodes in there, and for High redundancy it requires five copies of voting disk.


So, in this scenario - we add a separate HDD drive on both compute nodes and share that between the nodes to be used for High redundancy diskgroup creation - there it comes the term "Quorum Disk"


We have +DATAC1 diskgroup configured with HIGH redundancy where our voting disk resides but two of the quorum disks being used in there went missing so our voting disks had only three copies available and CRS was not coming up.


I had to remove the quorum disks from both compute nodes and then configure it back to fix the issue.


SQL> l 
  1* select disk_number, GROUP_NUMBER, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, STATE, path from v$asm_disk where GROUP_NUMBER=1
SQL> / 

DISK_NUMBER GROUP_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    PATH 
----------- ------------ ------- ------------ ------- -------- ---------------------------------------------------------------------- 
          2            1 MISSING UNKNOWN      OFFLINE NORMAL 
          3            1 MISSING UNKNOWN      OFFLINE NORMAL 
         14            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_05_exad02cel03
         13            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_00_exad02cel03
         15            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_02_exad02cel03
          7            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03
          8            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_04_exad02cel03
          0            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_01_exad02cel03
          4            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02
         16            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_02_exad02cel02
         17            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_01_exad02cel02
          5            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_00_exad02cel02
          6            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_03_exad02cel02
         11            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_04_exad02cel02
         19            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_04_exad02cel01
          1            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01
         18            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_00_exad02cel01
          9            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_03_exad02cel01
         12            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_01_exad02cel01


Due to missing quorum disk in the DATAC1 diskgroup, we have only three copies of voting disks available as of now.

[root@exa01dbadm02 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   94acaa401bd34fe0bf261af4437fc75e (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   7680d9f206544ff1bf27661bf8cc3250 (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   b632f75fe4884f5cbf3627faa6bbbe07 (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
Located 3 voting disk(s).
[root@exa01dbadm02 ~]#


I moved the voting disks to +RECOC1 diskgroup which was configured with NORMAL REDUNDANCY.


[oragrid@exa01dbadm01 ~]$ crsctl replace votedisk +RECOC1
Successful addition of voting disk 783e7633ea934fd0bff78c8b8dbe5a66.
Successful addition of voting disk a5cf9e2d99134f69bf6bb15c98044623.
Successful addition of voting disk d5d20cd6ff3e4fcdbf68df28cb18d030.
Successful deletion of voting disk 94acaa401bd34fe0bf261af4437fc75e.
Successful deletion of voting disk 7680d9f206544ff1bf27661bf8cc3250.
Successful deletion of voting disk b632f75fe4884f5cbf3627faa6bbbe07.
Successful deletion of voting disk e47d60407b054f3bbf5dc07df8080ee6.
Successfully replaced voting disk group with +RECOC1.
CRS-4266: Voting file(s) successfully replaced


[oragrid@exa01dbadm01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   783e7633ea934fd0bff78c8b8dbe5a66 (o/10.0.0.5;10.0.0.6/RECOC1_CD_02_exad02cel01) [RECOC1]
 2. ONLINE   a5cf9e2d99134f69bf6bb15c98044623 (o/10.0.0.3;10.0.0.4/RECOC1_CD_02_exad02cel02) [RECOC1]
 3. ONLINE   d5d20cd6ff3e4fcdbf68df28cb18d030 (o/10.0.0.1;10.0.0.2/RECOC1_CD_02_exad02cel03) [RECOC1]
Located 3 voting disk(s).
[oragrid@exa01dbadm01 ~]$


Now we can go ahead and drop the missing quorum disks from the DATAC1 diskgroup forcefully.


[oragrid@exa01dbadm01 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.2.0.1.0 Production on Thu Jul 30 12:06:27 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>
SQL> set lines 300
SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm01 FORCE;

Diskgroup altered.

SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm02 FORCE;

Diskgroup altered.

SQL>


Delete the quorum disk device from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.

Delete the quorum disk target from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm01
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1


Now delete quorum disk configuration from the node1.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --config

[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#




Now, delete the quorum disk device from node2.

[root@exa01dbadm02 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.

Delete quorum disk target from node2.

[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm02
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1

Delete quorum disk configuration from node 2.

[root@exa01dbadm02 oracle.SupportTools]#  ./quorumdiskmgr --delete --config
[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#


At this point, we are done with removing the quorum disk configuration from our Exadata System, in next post, I would detail how we configure it in the system.





Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.


Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha