Configure Quorum Disk in Exadata

In last article[click here] - we learnt how to remove quorum disk from the Exadata system, now here we would see it we add and configure that.

Before we are adding quorum disk configuration in the system, you need to have IB switches interface names, ASM binary owner and group handy as that is required to create the config.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#

Do the same as above on node2 as well.

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#

Check if the quoum disk configuration is created as below from node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.1

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.3

[root@exa01dbadm02 oracle.SupportTools]#

Now, create quorum disk target for DATAC1 diskgroup which is visible to both compute nodes, before it you need to identify IB interface IPs of both nodes as that is required here.

Run the command on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm01.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01.

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm02.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02.

[root@exa01dbadm02 oracle.SupportTools]#

Now you can list the quorum disk targets on node1 and node2 as below to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm02 oracle.SupportTools]#

Create quorum disk device as below on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm02 oracle.SupportTools]#

Now you can list the quorum disk device on node1 and node2 to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm02 oracle.SupportTools]#

Now, check in the ASM instance if quorum disk devices are visible to ASM to be used for the DATAC1 diskgroup as below, we can see that quorum disk devices are available as CANDIDATE disks that we can add in to the diskgroup.

SQL> l
  1* SELECT inst_id, label, path, mode_status, header_status FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%'
SQL> /

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE

SQL>

Add the quorum devices in to the diskgroup as below.

SQL> ALTER DISKGROUP datac1 ADD QUORUM FAILGROUP exa01dbadm01 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm01'
QUORUM FAILGROUP exa01dbadm02 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm02';  2

Diskgroup altered.

SQL>

After disk addition, we can see that now these are parts of diskgroup and status is MEMBER.

SQL> SELECT inst_id, label, path, mode_status, header_status
FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%';  2

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER

SQL>

Finally, replaced/moved the voting disks back to DATAC1 diskgroup and five copies of files are now available.

[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#

After CRS restart : I bounced the CRS on both nodes to check if things are coming up normally.

[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#

Our missing quorum disk issue stands fixed here......!!

Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.

Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha

How to find number of instances configured in your RAC Environment?

Below are the ways you can find out how many number of instances configured in your Oracle RAC cluster Environment.


1 - Query the V$ACTIVE_INSTANCES view to determine the number of instances involved in your RAC configuration.


SQL> desc v$active_instances;
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 INST_NUMBER                                        NUMBER
 INST_NAME                                          VARCHAR2(60)



SQL> select * from v$active_instances;

INST_NUMBER INST_NAME
----------- ------------------------------------------------------------
          1 rac1.rajdbsolutions.com:ractst1
          2 rac2.rajdbsolutions.com:ractst2







2 - You can find the same answer at OS level SRVCTL command line utility as below.


-bash-3.2$ srvctl config database -d ractst
Database unique name: ractst
Database name: ractst
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ractst/spfileractst.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ractst
Database instances: ractst1,ractst2
Disk Groups: DATA
Services:
Database is administrator managed


Note : - In the above srvctl config command output you can see the number of instances (ractst1,ractst2) are listed by comma at Database Instances clause that are configured in current RAC configuration environment.

What is Cache Fusion?

Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block,it is easy to get the block image from the instance which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects.

The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cachce fusion.

What is GRD?

GRD stands for Global Resource Directory.

The GES and GCS maintains records of the statuses of each datafile and each cached block using global resource directory.This process is referred to as cache fusion and helps in data integrity.

Mention the Oracle RAC software components

Oracle RAC is composed of two or more database instances.
They are composed of Memory structures and background processes same as the single instance database.

Oracle RAC instances use two processes:
1- GES(Global Enqueue Service)
2- GCS(Global Cache Service) that enable cache fusion.

Oracle RAC instances are composed of following background processes:
ACMS—Atomic Controlfile to Memory Service (ACMS)
GTX0-j—Global Transaction Process
LMON—Global Enqueue Service Monitor
LMD—Global Enqueue Service Daemon
LMS—Global Cache Service Process
LCK0—Instance Enqueue Process
RMSn—Oracle RAC Management Processes (RMSn)
RSMN—Remote Slave Monitor

Oracle Clusterware Processes on Linux

What are Oracle Clusterware processes for 10g on Unix and Linux:

Cluster Synchronization Services (ocssd) — Manages cluster node membership and runs as the oracle user; failure of this process results in cluster restart.

 Cluster Ready Services (crsd) — The crs process manages cluster resources (which could be a database, an instance, a service, a Listener, a virtual IP (VIP) address, an application process, and so on) based on the resource’s configuration information that is stored in the OCR. This includes start, stop, monitor and failover operations. This process runs as the root user

 Event manager daemon (evmd) —A background process that publishes events that crs creates. 

Process Monitor Daemon (OPROCD) —This process monitor the cluster and provide I/O fencing.

OPROCD performs its check, stops running, and if the wake up is beyond the expected time, then
OPROCD resets the processor and reboots the node. An OPROCD failure results in Oracle Clusterware restarting the node. OPROCD uses the hangcheck timer on Linux platforms.

 RACG (racgmain, racgimon) —Extends clusterware to support Oracle-specific requirements and complex resources. Runs server callout scripts when FAN events occur.