Configure Quorum Disk in Exadata

In last article[click here] - we learnt how to remove quorum disk from the Exadata system, now here we would see it we add and configure that.

Before we are adding quorum disk configuration in the system, you need to have IB switches interface names, ASM binary owner and group handy as that is required to create the config.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#

Do the same as above on node2 as well.

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#

Check if the quoum disk configuration is created as below from node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.1

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.3

[root@exa01dbadm02 oracle.SupportTools]#

Now, create quorum disk target for DATAC1 diskgroup which is visible to both compute nodes, before it you need to identify IB interface IPs of both nodes as that is required here.

Run the command on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm01.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01.

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm02.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02.

[root@exa01dbadm02 oracle.SupportTools]#

Now you can list the quorum disk targets on node1 and node2 as below to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm02 oracle.SupportTools]#

Create quorum disk device as below on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm02 oracle.SupportTools]#

Now you can list the quorum disk device on node1 and node2 to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm01 oracle.SupportTools]#

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm02 oracle.SupportTools]#

Now, check in the ASM instance if quorum disk devices are visible to ASM to be used for the DATAC1 diskgroup as below, we can see that quorum disk devices are available as CANDIDATE disks that we can add in to the diskgroup.

SQL> l
  1* SELECT inst_id, label, path, mode_status, header_status FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%'
SQL> /

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE

SQL>

Add the quorum devices in to the diskgroup as below.

SQL> ALTER DISKGROUP datac1 ADD QUORUM FAILGROUP exa01dbadm01 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm01'
QUORUM FAILGROUP exa01dbadm02 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm02';  2

Diskgroup altered.

SQL>

After disk addition, we can see that now these are parts of diskgroup and status is MEMBER.

SQL> SELECT inst_id, label, path, mode_status, header_status
FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%';  2

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER

SQL>

Finally, replaced/moved the voting disks back to DATAC1 diskgroup and five copies of files are now available.

[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#

After CRS restart : I bounced the CRS on both nodes to check if things are coming up normally.

[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#

Our missing quorum disk issue stands fixed here......!!

Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.

Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha

Removing the Quorum Disk Configuration in Exadata

 

In this blog post, we would see how to remove quorum disks in Exadata machine. We have one 1/8th Rack Exadata machine for Lab activity where ran into missing quorum disks for Voting Disk and I had to fix that in order to get the system patched in Rolling fashion to maintain high availability.

To patch the Exadata system in rolling fashion, it requires voting disks to reside on HIGH redundancy diskgroup which is not possible in 1/8th Rack Exadata system since there is only 3 cell nodes in there, and for High redundancy it requires five copies of voting disk.

So, in this scenario - we add a separate HDD drive on both compute nodes and share that between the nodes to be used for High redundancy diskgroup creation - there it comes the term "Quorum Disk"

We have +DATAC1 diskgroup configured with HIGH redundancy where our voting disk resides but two of the quorum disks being used in there went missing so our voting disks had only three copies available and CRS was not coming up.

I had to remove the quorum disks from both compute nodes and then configure it back to fix the issue.

SQL> l 
  1* select disk_number, GROUP_NUMBER, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, STATE, path from v$asm_disk where GROUP_NUMBER=1
SQL> / 

DISK_NUMBER GROUP_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    PATH 
----------- ------------ ------- ------------ ------- -------- ---------------------------------------------------------------------- 
          2            1 MISSING UNKNOWN      OFFLINE NORMAL 
          3            1 MISSING UNKNOWN      OFFLINE NORMAL 
         14            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_05_exad02cel03
         13            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_00_exad02cel03
         15            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_02_exad02cel03
          7            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03
          8            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_04_exad02cel03
          0            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_01_exad02cel03
          4            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02
         16            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_02_exad02cel02
         17            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_01_exad02cel02
          5            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_00_exad02cel02
          6            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_03_exad02cel02
         11            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_04_exad02cel02
         19            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_04_exad02cel01
          1            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01
         18            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_00_exad02cel01
          9            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_03_exad02cel01
         12            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_01_exad02cel01

Due to missing quorum disk in the DATAC1 diskgroup, we have only three copies of voting disks available as of now.

[root@exa01dbadm02 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   94acaa401bd34fe0bf261af4437fc75e (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   7680d9f206544ff1bf27661bf8cc3250 (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   b632f75fe4884f5cbf3627faa6bbbe07 (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
Located 3 voting disk(s).
[root@exa01dbadm02 ~]#

I moved the voting disks to +RECOC1 diskgroup which was configured with NORMAL REDUNDANCY.

[oragrid@exa01dbadm01 ~]$ crsctl replace votedisk +RECOC1
Successful addition of voting disk 783e7633ea934fd0bff78c8b8dbe5a66.
Successful addition of voting disk a5cf9e2d99134f69bf6bb15c98044623.
Successful addition of voting disk d5d20cd6ff3e4fcdbf68df28cb18d030.
Successful deletion of voting disk 94acaa401bd34fe0bf261af4437fc75e.
Successful deletion of voting disk 7680d9f206544ff1bf27661bf8cc3250.
Successful deletion of voting disk b632f75fe4884f5cbf3627faa6bbbe07.
Successful deletion of voting disk e47d60407b054f3bbf5dc07df8080ee6.
Successfully replaced voting disk group with +RECOC1.
CRS-4266: Voting file(s) successfully replaced


[oragrid@exa01dbadm01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   783e7633ea934fd0bff78c8b8dbe5a66 (o/10.0.0.5;10.0.0.6/RECOC1_CD_02_exad02cel01) [RECOC1]
 2. ONLINE   a5cf9e2d99134f69bf6bb15c98044623 (o/10.0.0.3;10.0.0.4/RECOC1_CD_02_exad02cel02) [RECOC1]
 3. ONLINE   d5d20cd6ff3e4fcdbf68df28cb18d030 (o/10.0.0.1;10.0.0.2/RECOC1_CD_02_exad02cel03) [RECOC1]
Located 3 voting disk(s).
[oragrid@exa01dbadm01 ~]$

Now we can go ahead and drop the missing quorum disks from the DATAC1 diskgroup forcefully.

[oragrid@exa01dbadm01 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.2.0.1.0 Production on Thu Jul 30 12:06:27 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>
SQL> set lines 300
SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm01 FORCE;

Diskgroup altered.

SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm02 FORCE;

Diskgroup altered.

SQL>

Delete the quorum disk device from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.




Delete the quorum disk target from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm01
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1




Now delete quorum disk configuration from the node1.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --config

[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#

Now, delete the quorum disk device from node2.

[root@exa01dbadm02 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.




Delete quorum disk target from node2.

[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm02
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1




Delete quorum disk configuration from node 2.

[root@exa01dbadm02 oracle.SupportTools]#  ./quorumdiskmgr --delete --config
[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#

At this point, we are done with removing the quorum disk configuration from our Exadata System, in next post, I would detail how we configure it in the system.

Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.

Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha

ORA-15039: diskgroup not dropped

You would get this error when you try to drop an ASM diskgroup that is mounted by an another ASM instance as well e.g. in case of Oracle RAC database clusters, ASM diskgroup shared/mounted by all cluster nodes in oracle RAC configuration.

ASMCMD> dropdg EBS_DATA

ORA-15039: diskgroup not dropped

ORA-15073: diskgroup EBS_DATA is mounted by another ASM instance (DBD ERROR: OCIStmtExecute)

ASMCMD>

Let's verify what all instances have mounted this EBS_DATA diskgroup with following command.

SQL> select inst_id, name, state from gv$asm_diskgroup;

   INST_ID NAME                           STATE
---------- ------------------------------ -----------
         2 VOTEDATA                       MOUNTED
         2 ORCL_DATA                      MOUNTED
         2 ORCL_FRA                       MOUNTED
         2 EBS_DATA                       MOUNTED
         1 VOTEDATA                       MOUNTED
         1 ORCL_DATA                      MOUNTED
         1 ORCL_FRA                       MOUNTED
         1 EBS_DATA                       MOUNTED

8 rows selected.

In above output, you can see that EBS_DATA diskgroup has been mounted by INST_ID=1 and INST_ID=2. 

Now let's go to node 2 and dismount the EBS_DATA diskgroup by logging to ASM instance as "SYSASM"

SQL> alter diskgroup ebs_data dismount;

Diskgroup altered.


SQL>

ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   1024000    92279                0           92279              0             N  ORCL_DATA/
MOUNTED  EXTERN  N         512   4096  1048576    307200   307101                0          307101              0             N  ORCL_FRA/
MOUNTED  NORMAL  N         512   4096  1048576   1024000  1023705           256000          383852              0             N  EBS_DATA/
MOUNTED  HIGH    N         512   4096  1048576     51200    49944            20480            9821              0             Y  VOTEDATA/

Now try to drop the diskgroup, and it works.

ASMCMD> dropdg ebs_data

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   1024000    92279                0           92279              0             N  ORCL_DATA/
MOUNTED  EXTERN  N         512   4096  1048576    307200   307101                0          307101              0             N  ORCL_FRA/
MOUNTED  HIGH    N         512   4096  1048576     51200    49944            20480            9821              0             Y  VOTEDATA/
ASMCMD>

Hope it helps, subscribe to this blog to stay updated on latest Oracle Technologies and new articles.

Twitter : @rajsoft8899
Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

CRS-4046: Invalid Oracle Clusterware configuration

Received below error while running root.sh script in Oracle 11g R2 (11.2.0.4) on node one in a two node RAC configuration. It was due to leftover processes from previous cluster install.


[root@RAC1 Clusterware]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Create failed, or completed with errors.
Failure initializing entries in /etc/oracle/scls_scr/rac1
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed




Below illustrated steps resolved the issue.

Clean up the leftover cluster processes by executing below command on the node where root.sh script fails.


[root@RAC1 install]# ./rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: ./crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
# You must kill processes or reboot the system to properly #
# cleanup the processes started by Oracle clusterware          #
################################################################
Removing Trace File Analyzer
Failure in execution (rc=-1, 256, No such file or directory) for command /etc/init.d/ohasd deinstall
error: package cvuqdisk is not installed
Successfully deconfigured Oracle clusterware stack on this node




=>> Once de-configuration for Oracle clusterware completes successful reboot the server once de-config of clusterware is done as above and re-try executing root.sh again.



[root@RAC1 grid]# ./root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert

.

.

.

.

Adding Clusterware entries to upstart

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group VOTE_DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk ae113dc9ff954f7bbf65bccce936df15.

Successful addition of voting disk 7d464ed047e64fb8bf349015e04dc69f.

Successful addition of voting disk 8724064160f74fe7bf8f85bf258d13e9.

Successfully replaced voting disk group with +VOTE_DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   ae113dc9ff954f7bbf65bccce936df15 (/dev/oracleasm/disks/DISK1) [VOTE_DATA]

 2. ONLINE   7d464ed047e64fb8bf349015e04dc69f (/dev/oracleasm/disks/DISK2) [VOTE_DATA]

 3. ONLINE   8724064160f74fe7bf8f85bf258d13e9 (/dev/oracleasm/disks/DISK3) [VOTE_DATA]

Located 3 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.VOTE_DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.VOTE_DATA.dg' on 'rac1' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Hope it would help someone....!!

[INS-20802] Grid Infrastructure failed During Grid Installation On Windows OS

It was a challenging day to to troubleshoot the issue of cluster configuration failure while setting up 2 Node Oracle 11gR2(11.2.0.4) RAC cluster on Windows Server 2012 R2 operating system for one of our customer. At every attempt Grid Confuguration for cluster was getting failed while Installing Oracle Grid Infrastructure software for cluster.

InstallAction log did't have much clue about the GI configuration failure but rootcrs_racnode1.log located at $GRID_HOME\cfgtoollogs\rootcrs_racnode1.log file provided the root cause of cluster configuration failure which reveals that "The driver is not currently installed on this node."

Below is the entries from rootcrs_racnode1.log log file from Node 1.


From Log file : C:\app\11.2.0\grid\cfgtoollogs\rootcrs_racnode1.log
------------------------------------------------------

2016-06-09 08:22:36: The 'ROOTCRS_ACFSINST' is either in START/FAILED state
2016-06-09 08:22:36: Executing 'E:\app\11.2.0\grid\bin\acfsroot.bat install'
2016-06-09 08:22:36: Executing cmd: E:\app\11.2.0\grid\bin\acfsroot.bat install
2016-06-09 08:22:37: Command output:
>  ACFS-9300: ADVM/ACFS distribution files found.
>  ACFS-9307: Installing requested ADVM/ACFS software.
>  acfsinstall: ACFS-09420: The driver is not currently installed on this node.
>  acfsinstall: ACFS-09411: CreateService succeeded.
>  acfsinstall: CLSU-00100: Operating System function: StartDriver failed with error data: 31
>  acfsinstall: CLSU-00101: Operating System error message: A device attached to the system is not functioning.
>  acfsinstall: CLSU-00103: error location: StartDriver_
>  acfsinstall: CLSU-00104: additional error information: W
>  acfsinstall: ACFS-09419: StartService failed.
>  acfsinstall: ACFS-09401: Failed to install the driver.
>
>  ACFS-9340: failed to install OKS driver.
>  acfsinstall: ACFS-09420: The driver is not currently installed on this node.
>  acfsinstall: ACFS-09411: CreateService succeeded.
>  acfsinstall: CLSU-00100: Operating System function: StartDriver failed with error data: 1068
>  acfsinstall: CLSU-00101: Operating System error message: The dependency service or group failed to start.
>  acfsinstall: CLSU-00103: error location: StartDriver_
>  acfsinstall: CLSU-00104: additional error information: J
>  acfsinstall: ACFS-09419: StartService failed.
>  acfsinstall: ACFS-09401: Failed to install the driver.
>
>  ACFS-9340: failed to install ADVM driver.
>  acfsinstall: ACFS-09420: The driver is not currently installed on this node.
>  acfsinstall: ACFS-09411: CreateService succeeded.
>  acfsinstall: CLSU-00100: Operating System function: StartDriver failed with error data: 1068
>  acfsinstall: CLSU-00101: Operating System error message: The dependency service or group failed to start.
>  acfsinstall: CLSU-00103: error location: StartDriver_
>  acfsinstall: CLSU-00104: additional error information: ]
>  acfsinstall: ACFS-09419: StartService failed.
>  acfsinstall: ACFS-09401: Failed to install the driver.
>
>  ACFS-9340: failed to install ACFS driver.
>  ACFS-9310: ADVM/ACFS installation failed.
>End Command output
2016-06-09 08:22:37: E:\app\11.2.0\grid\bin\acfsroot.bat install ... failed
2016-06-09 08:22:37: USM driver install status is 0
2016-06-09 08:22:37: USM driver install actions failed
2016-06-09 08:22:37: Running as user Administrator: E:\app\11.2.0\grid\bin\cluutil -ckpt -oraclebase E:\app\Administrator -writeckpt -name ROOTCRS_ACFSINST -state FAIL
2016-06-09 08:22:37: s_run_as_user2: Running E:\app\11.2.0\grid\bin\cluutil -ckpt -oraclebase E:\app\Administrator -writeckpt -name ROOTCRS_ACFSINST -state FAIL
2016-06-09 08:22:37: E:\app\11.2.0\grid\bin\cluutil successfully executed

2016-06-09 08:22:37: Succeeded in writing the checkpoint:'ROOTCRS_ACFSINST' with status:FAIL
2016-06-09 08:22:37: CkptFile: E:\app\Administrator\Clusterware\ckptGridHA_win1.xml
2016-06-09 08:22:37: Sync the checkpoint file 'E:\app\Administrator\Clusterware\ckptGridHA_win1.xml'




Solutions :
=========

As per the logs pasted above, I came about an unpublished BUG 17927204 - ACFS SUPPORT FOR WINDOWS 2012R2  in Oracle Grid Infrastructure version 11.2.0.4 itself so in order to resolve the cluster configuration issue I downloaded the one-off patch (p22839608_112040_MSWIN-x86-64) from MOS to be applied on #GRID_HOME binaries on both nodes in the cluster. And in order to apply the Patch mentioned above, you will have to download relevant Opatch utility (p6880880_112000_MSWIN-x86-64) from MOS.

Please refer MOS Doc ID 1987371.1 for its details.


Once you have downloaded both patches mentioned above please follow the below steps for successful Grid Infrastructure Installation.


1 - Clean the currently failed run of GI from both nodes (it includes deinstall of GI and removal of all related entries from Windows Registry.)

Click Here to see how to clean failed Grid Installation.

2 - Bounce the nodes once you are done with step 1.

3 - Run the GI Installer (setup.exe) for GI installation and choose to install Grid Infrastructure software only.

Note :- In step 3 you will have to install GI software on each individual nodes in the cluster as it won't do GI installation on remote nodes at once.

4 - Once the GI software(Software only) is installed on both the nodes then replace(or rename the existing Opatch directory in Grid Home. e.g. Opatch_old) Opatch folder in Grid Home with the one you have downloaded above (p6880880_112000_MSWIN-x86-64) on both nodes.

5 - Verify the Opatch utility is working fine with : opatch.exe lsinv command and apply the one-off patch(p22839608_112040_MSWIN-x86-64) on Grid Home at both nodes.

6 - After successful application of Opatch on both nodes, Start the Grid Infrastructure Configuration for the cluster as mentioned below.

Navigate to and run $GRID_HOME/crs/config/config.bat

Note : - It will launch GUI of Grid Infrastructure Configuration and proceed as per its steps gets prompted on interface. Please note that if your GI configuration still fails at 33% then don't just cancel the installation, just click OK on the error prompt and click on "Retry" to continue with the installation.


Hope it wold help to resolve the issue. 

Adding a new OCR device/file

To avoid Single Point Of Failure of OCR we should add/have multiple OCR device/file on separate storage. We can have upto 5 OCR device/file on our cluster configuration. Below are the steps outlined how we can add a new OCR to our Cluster configuration.


Step 1 - Let's first find how many OCR device/file already exists.


[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2616
         Available space (kbytes) :     259504
         ID                       :  170058601
         Device/File Name         : +VOTE_DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded




Note :- In the preceding output we can see only one OCR file exists at location +VOTE_DATA diskgroup.





Step 2 - Add a new OCR device/file.


[root@rac1 ~]# ocrconfig -add '+VOTE_DATA'
PROT-29: The Oracle Cluster Registry location is already configured

Note :- We can't add another OCR device/file on the same file-system or diskgroup hence we need to add the new OCR file to a new/separate device/file as adding the other OCR files on the same device or diskgroup doesn't avoid SPOF(Single Poing Of Failure)



[root@rac1 ~]# ocrconfig -add '+FLASH'


Again verify if new OCR device/file is added to +FLASH diskgroup.

[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2616
         Available space (kbytes) :     259504
         ID                       :  170058601
         Device/File Name         : +VOTE_DATA
                                    Device/File integrity check succeeded
         Device/File Name         :     +FLASH
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded



[root@rac1 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
         Device/File Name         : +VOTE_DATA
         Device/File Name         :     +FLASH



Note : As we can see above, a new OCR device/file file has been added to +FLASH diskgroup. After adding the new OCR file once check the integrity of OCR using ocrcheck command.

Significance of OCRCONFIG

OCRCONFIG:

Use the ocrconfig command to manage OCR. Using this utility you can import, export, add, delete, restore, overwrite, backup, repair, replace, move, upgrade, or downgrade OCR.


Below are the options can be used with ocrconfig command.
---------------------------------------------

[root@rac1 ~]# ocrconfig
Name:
        ocrconfig - Configuration tool for Oracle Cluster/Local Registry.

Synopsis:
        ocrconfig [option]
        option:
                [-local] -export <filename>
                                                    - Export OCR/OLR contents to a file
                [-local] -import <filename>         - Import OCR/OLR contents from a file
                [-local] -upgrade [<user> [<group>]]
                                                    - Upgrade OCR from previous version
                -downgrade [-version <version string>]
                                                    - Downgrade OCR to the specified version
                [-local] -backuploc <dirname>       - Configure OCR/OLR backup location
                [-local] -showbackup [auto|manual]  - Show OCR/OLR backup information
                [-local] -manualbackup              - Perform OCR/OLR backup
                [-local] -restore <filename>        - Restore OCR/OLR from physical backup
                -replace <current filename> -replacement <new filename>
                                                    - Replace a OCR device/file <filename1> with <filename2>
                -add <filename>                     - Add a new OCR device/file
                -delete <filename>                  - Remove a OCR device/file
                -overwrite                          - Overwrite OCR configuration on disk
                -repair -add <filename> | -delete <filename> | -replace <current filename> -replacement <new filename>
                                                    - Repair OCR configuration on the local node
                -help                               - Print out this help information

Note:
        * A log file will be created in
        $ORACLE_HOME/log/<hostname>/client/ocrconfig_<pid>.log. Please ensure
        you have file creation privileges in the above directory before
        running this tool.
        * Only -local -showbackup [manual] is supported.
        * Use option '-local' to indicate that the operation is to be performed on the Oracle Local Registry.




1 - Let's use the first option (-local -export) with ocrconfig command to export the contents of OCR/OLR to a text file as a backup.


[root@rac1 ~]# ocrconfig -local -export /u01/app/OCR_Local_export.txt

OCR local file has been export and an export listed below has been created.

[root@rac1 ~]# ls -ltr /u01/app/OCR_Local_export.txt
-rw-r--r-- 1 root root 73429 Jun  7 09:00 /u01/app/OCR_Local_export.txt



Note: -

This section lists the following OCRCONFIG commands: Click on any commands listed below to get its detailed usage info.

• ocrconfig -add
• ocrconfig -backuploc
• ocrconfig -delete
• ocrconfig -downgrade
• ocrconfig -export
• ocrconfig -import
• ocrconfig -manualbackup
• ocrconfig -overwrite
• ocrconfig -repair
• ocrconfig -replace
• ocrconfig -restore
• ocrconfig -showbackup
• ocrconfig -upgrade
• 
  

OCRCHECK : Oracle Cluster Registry Check utility

OCRCHECK:

The OCRCHECK utility displays the version of the OCR's block format, total space available and used space, OCRID, and the OCR locations that you have configured. OCRCHECK performs a block-by-block checksum operation for all of the blocks in all of the OCRs that you have configured. It also returns an individual status for each file and a result for the overall OCR integrity check.


Note:
Oracle supports using the ocrcheck command when, at a minimum, the Oracle Cluster Ready Services stack is OFFLINE on all nodes in the cluster because the command will run even if the stack is ONLINE but it can falsely indicate that the OCR is corrupt if the check happens while an update to OCR is underway.



Syntax
ocrcheck [-local] [-config] [-details] [-help]



[root@rac2 ~]# ocrcheck -help
Name:
        ocrcheck - Displays health of Oracle Cluster/Local Registry.

Synopsis:
        ocrcheck [-config] [-local]

  -config       Displays the configured locations of the Oracle Cluster Registry.
                This can be used with the -local option to display the configured
                location of the Oracle Local Registry
  -local        The operation will be performed on the Oracle Local Registry.



Notes:
        A log file will be created in
        $ORACLE_HOME/log/<hostname>/client/ocrcheck_<pid>.log.
        File creation privileges in the above directory are needed
        when running this tool.




[root@rac2 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2616
         Available space (kbytes) :     259504
         ID                       :  170058601
         Device/File Name         : +VOTE_DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded



Note :- When we simply run only ocrcheck command as root user without any option then it checks the integrity of Oracle Clusterware Registry and reveals the OCR version, Total Space, Used Space and Available space. It also displays the File ID and the location where OCR is located for global access as shown in preceding output.




[root@rac2 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
         Device/File Name         : +VOTE_DATA


Note :- When we run the ocrcheck command with root user with -config option then it displays the device/diskgroup where cluster OCR is located.





 [root@rac2 ~]# ocrcheck -config -local
Oracle Local Registry configuration is :
         Device/File Name         : /u01/app/11.2.0/grid/cdata/rac2.olr
[root@rac2 ~]#


Note :- When we run the ocrcheck -config along with -local option then it displays the Oracle Local Repository(OLR), the local version of OCR  located on local node as shown in the preceding example.





Note :-

A log file will be created in
        $GRID_HOME/log/<hostname>/client/ocrcheck_<pid>.log.
        File creation privileges in the above directory are needed
        when running this tool.


A log file for each ocrcheck command run will be created under $GRID_HOME/log/<hostname>/client/ocrcheck_<pid>.log. Let's take a look at this with an example as explained below.



1 - First let's check the timestamp at the RAC node were we will be running the ocrcheck command.

[root@rac2 client]# date

Mon Jun  6 10:28:03 IST 2016

2 - Now, let's run the ocrcheck command.

[root@rac2 client]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       2616

         Available space (kbytes) :     259504

         ID                       :  170058601

         Device/File Name         : +VOTE_DATA

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

3 - Now let's go to log location ($GRID_HOME/log/<hostname>/client/ocrcheck_<pid>.log) and check for the new log file is created.

[root@rac2 client]# pwd

/u01/app/11.2.0/grid/log/rac2/client

[root@rac2 client]# ls -ltr | tail

-rw-r----- 1 root    root       256 Jun  3 12:14 ocrconfig_7211.log

-rw-r----- 1 root    root       256 Jun  3 12:21 ocrconfig_7460.log

-rw-r----- 1 root    root       342 Jun  3 12:21 ocrconfig_7469.log

-rw-r--r-- 1 oragrid oinstall  1612 Jun  6 09:56 oclskd.log

-rw-r--r-- 1 oragrid oinstall 21138 Jun  6 09:56 olsnodes.log

-rw-r--r-- 1 root    root       379 Jun  6 09:58 ocrcheck_6646.log

-rw-r--r-- 1 root    root       379 Jun  6 09:58 ocrcheck_6669.log

-rw-r----- 1 root    root       255 Jun  6 09:59 ocrcheck_6684.log

-rw-r----- 1 root    root       255 Jun  6 09:59 ocrcheck_6689.log

-rw-r--r-- 1 root    root       379 Jun  6 10:28 ocrcheck_7621.log

[root@rac2 client]# date

Mon Jun  6 10:28:52 IST 2016

Note :- We can see a new log file has been created marked in red-color above.

Contents of ocrcheck_7621.log

[root@rac2 client]# cat ocrcheck_7621.log

Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.

2016-06-06 10:28:36.677: [OCRCHECK][3038611136]ocrcheck starts...

2016-06-06 10:28:37.212: [OCRCHECK][3038611136]protchcheck: OCR status : total = [262120], used = [2616], avail = [259504]

2016-06-06 10:28:40.939: [OCRCHECK][3038611136]Exiting [status=success]...

How to Enable Archiving in Oracle RAC environment?

Enabling ARCHIVELOG in Oracle RAC environment.


Below verification reveals that our current RAC cluster database is in NOARCHIVE LOG mode.

SQL> archive log list
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     4
Current log sequence           5



Step 1 - Shut-down the database across the nodes from any node in the cluster using below command.

-bash-3.2$ srvctl stop database -d ractst -o immediate


Let's verify if database instances are down across the cluster nodes.

-bash-3.2$ srvctl status database -d ractst
Instance ractst1 is not running on node rac1
Instance ractst2 is not running on node rac2


Step 2 - Mount the database instances using below command.

-bash-3.2$ srvctl start database -d ractst -o mount


Instances are started now - in Mount state.

-bash-3.2$ srvctl status database -d ractst
Instance ractst1 is running on node rac1
Instance ractst2 is running on node rac2



Note : - Before Oracle 11g R2, we used to disable the INIT parameter cluster_database in order to enable and disable archiving in RAC environment.


Step 3 - Enable the archiving now.


-bash-3.2$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Fri Jun 3 13:00:52 2016

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     4
Next log sequence to archive   5
Current log sequence           5


Also open the database at 2nd node and check archiving status as below.

SQL> alter database open;

Database altered.

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     2
Next log sequence to archive   3
Current log sequence           3



That's it...Hope it would help someone....

CSS is not ready. Received status 3 from CSS. Waiting for good status ..

Today I was just playing with 2 node Oracle RAC cluster and 1st node went down suddenly. I could see the following entries in the CRS log file.


CRS log file:
========

2016-05-17 09:58:18.944: [ CSSCLNT][3038205648]clssscConnect: gipc request failed with 29 (0x16)
2016-05-17 09:58:18.944: [ CSSCLNT][3038205648]clsssInitNative: connect failed, rc 29
2016-05-17 09:58:18.944: [  CRSRTI][3038205648] CSS is not ready. Received status 3 from CSS. Waiting for good status ..

2016-05-17 09:58:19.945: [ CSSCLNT][3038205648]clssscConnect: gipc request failed with 29 (0x16)
2016-05-17 09:58:19.945: [ CSSCLNT][3038205648]clsssInitNative: connect failed, rc 29
2016-05-17 09:58:19.945: [  CRSRTI][3038205648] CSS is not ready. Received status 3 from CSS. Waiting for good status .. 




From OCSSD log file:
==============

==> ocssd.log <==
2016-05-17 10:27:16.996: [    CSSD][2998332304]clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID
=ef2b0e79-00000000-5893))
2016-05-17 10:27:16.996: [    GPnP][3038959296]clsgpnp_Init: [at clsgpnp0.c:404] gpnp tracelevel 3, component tracelevel 0
2016-05-17 10:27:16.996: [    GPnP][3038959296]clsgpnp_Init: [at clsgpnp0.c:534] '/u01/app/11.2.0/grid' in effect as GPnP home base.
2016-05-17 10:27:17.079: [GIPCCLSA][2998332304]gipcmodClsaCompleteAccept: failed on clsaauthstart ret clsaretOSD (8), endp 0x8b26a10 [0
00000000000002f] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=fa51a6a2-77bcb030-5893))', remot
eAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=77bcb030-fa51a6a2-5834))', numPend 5, numReady 0, numDone 0, numDead 0
, numTransfer 0, objFlags 0x16ca, pidPeer 0, flags 0x603710, usrFlags 0x14000 }
2016-05-17 10:27:17.079: [GIPCCLSA][2998332304]gipcmodClsaCompleteAccept: slos op  :  mkdir
2016-05-17 10:27:17.079: [GIPCCLSA][2998332304]gipcmodClsaCompleteAccept: slos dep :  No space left on device (28)
2016-05-17 10:27:17.079: [GIPCCLSA][2998332304]gipcmodClsaCompleteAccept: slos loc :  authprep6
2016-05-17 10:27:17.079: [GIPCCLSA][2998332304]gipcmodClsaCompleteAccept: slos info:  failed to make dir /u01/app/11.2.0/grid/auth/css/
rac1/A2526853
2016-05-17 10:27:17.079: [GIPCXCPT][2998332304]gipcmodMuxTransferAccept: internal accept request failed endp 0x8b25c30 [000000000000001
b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=ef2b0e79-00000000-5893))', remoteAddr '', numP
end 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, flags 0x30613, usrFlags 0x10010 }, ret gipcretAuthFail
 (22)
2016-05-17 10:27:17.079: [ GIPCMUX][2998332304]gipcmodMuxTransferAccept: EXCEPTION[ ret gipcretAuthFail (22) ]  error during accept on
endp 0x8b25c30 [000000000000001b] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_)(GIPCID=ef2b0e79-000000
00-5893
[root@rac1 cssd]#



Cause : In the ocssd log file we can see that "No space left on device (28)" and when I checked at mount point /u01 found it was 100% full.


Resolution : I Just released space inside the mount /u01 and did reboot node1 and all cluster services came up automatically as expected at subsequent reboot.





Another cause and resolution:
====================

Sometimes, after maintenance tasks when we try to brinng up all CRS services then we get similary kind of issues due to lock files does exist inside /tmp or /var/tmp/.oracle directory


[root@rac1 tmp]# pwd
/tmp

We can see following files are there inside /tmp directory, just login as root user and navigate to /tmp or /var/tmp/ and delete(rm -rf *) these all and reboot the server.

[root@rac1 tmp]# ls -ltr /var/tmp/.oracle/
total 0
prw-r--r-- 1 root    root     0 Mar 13 15:37 npohasd
srwxrwxrwx 1 oragrid oinstall 0 Mar 16 01:28 s#6509.2
srwxrwxrwx 1 oragrid oinstall 0 Mar 16 01:28 s#6509.1
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:33 s#6543.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:33 s#6543.1
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:33 s#6571.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:33 s#6571.1
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:41 s#6481.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:41 s#6481.1
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:48 s#7342.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 08:48 s#7342.1
-rw-r--r-- 1 oragrid oinstall 0 May 12 10:08 ora_gipc_GPNPD_rac1_lock
srwxrwxrwx 1 oragrid oinstall 0 May 12 10:09 s#6486.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 10:09 s#6486.1
srwxrwxrwx 1 oragrid oinstall 0 May 12 10:11 s#7133.2
srwxrwxrwx 1 oragrid oinstall 0 May 12 10:11 s#7133.1
srwxrwxrwx 1 oragrid oinstall 0 May 15 14:29 ora_gipc_GPNPD_rac1
srwxrwxrwx 1 root    root     0 May 15 14:30 srac1DBG_CTSSD
srwxrwxrwx 1 oragrid oinstall 0 May 15 14:30 srac1DBG_EVMD
srwxrwxrwx 1 root    root     0 May 15 14:30 sprocr_local_conn_0_PROC
srwxrwxrwx 1 oragrid oinstall 0 May 15 14:30 sSYSTEM.evm.acceptor.auth
srwxrwxrwx 1 oragrid oinstall 0 May 15 14:30 sCevm
srwxrwxrwx 1 oragrid oinstall 0 May 15 14:30 sAevm
srwxrwxrwx 1 root    root     0 May 15 14:30 sCRSD_IPC_SOCKET_11
srwxrwxrwx 1 root    root     0 May 15 14:30 sora_crsqs
srwxrwxrwx 1 root    root     0 May 15 14:30 sCRSD_UI_SOCKET
srwxrwxrwx 1 oragrid oinstall 0 May 15 19:07 sLISTENER
srwxrwxrwx 1 oragrid oinstall 0 May 15 19:07 s#10780.2
srwxrwxrwx 1 oragrid oinstall 0 May 15 19:07 s#10780.1
-rw-r--r-- 1 oragrid oinstall 0 May 17 09:52 sOCSSD_LL_rac1__lock
srwxrwxrwx 1 root    root     0 May 17 09:57 srac1DBG_CRSD
srwxrwxrwx 1 root    root     0 May 17 10:10 srac1DBG_OHASD
srwxrwxrwx 1 root    root     0 May 17 10:10 sprocr_local_conn_0_PROL
srwxrwxrwx 1 root    root     0 May 17 10:10 sOHASD_UI_SOCKET
srwxrwxrwx 1 root    root     0 May 17 10:10 sOHASD_IPC_SOCKET_11
srwxrwxrwx 1 oragrid oinstall 0 May 17 10:10 srac1DBG_MDNSD
srwxrwxrwx 1 oragrid oinstall 0 May 17 10:10 srac1DBG_GIPCD
srwxrwxrwx 1 oragrid oinstall 0 May 17 10:12 srac1DBG_GPNPD
srwxrwx--- 1 oragrid oinstall 0 May 17 10:14 master_diskmon
srwxrwxrwx 1 oragrid oinstall 0 May 17 10:14 srac1DBG_CSSD
srwxrwxrwx 1 oragrid oinstall 0 May 17 10:14 sOCSSD_LL_rac1_


[root@rac1 tmp]# cd /var/tmp/.oracle/
[root@rac1 .oracle]# pwd
/var/tmp/.oracle

[root@rac1 .oracle]# rm -rf *

[root@rac1 .oracle]# ls -ltr
total 0



Note : In my case underline mount /u01 of GRID_HOME was exhausted and relasing the space resolved my issue.

How to compress SCAN_LISTENER log file in Oracle RAC environment?

Sometimes we are in need to compress SCAN listener log files to release space on underline mount point due huge growth of its size.

We follow the below steps to compress a SCAN listener log file.


Step 1 : Login as Grid Infrastructure user (it may be oracle as RDBMS user or oragrid/grid user in your environment, you can easily find it using - ps -ef | grep tns)

In my case ASM/GI is running with oragrid user.


[root@rac1 ~]# ps -ef | grep pmon
oragrid   5734     1  0 14:30 ?        00:00:00 asm_pmon_+ASM1
oracle    6440     1  0 14:31 ?        00:00:00 ora_pmon_ractst1
root      7145  7110  0 14:33 pts/1    00:00:00 grep pmon

[root@rac1 ~]# ps -ef | grep tns

root         9     2  0 14:27 ?        00:00:00 [netns]

oragrid   6279     1  0 14:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit

oragrid   6281     1  0 14:30 ?        00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit

root      7196  7110  0 14:34 pts/1    00:00:00 grep tns

Step 2 : Invoke LSNRCTL command line utility from GRID_HOME/bin directory.


Step 3 : set the current listener to the appropriate SCAN_LISTENER name you want to compress the log file as depicted below.


LSNRCTL> set current_listener LISTENER_SCAN1
Current Listener is LISTENER_SCAN1
LSNRCTL>

Now, if we check the status of listener then by default it is set to LISTENER_SCAN1 for now.

STATUS of the LISTENER

------------------------

Alias                     LISTENER_SCAN1

Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production

Start Date                15-MAY-2016 14:31:00

Uptime                    0 days 0 hr. 5 min. 30 sec

Trace Level               off

Security                  ON: Local OS Authentication

SNMP                      OFF

Listener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.ora

Listener Log File         /u01/app/11.2.0/grid/log/diag/tnslsnr/rac1/listener_scan1/alert/log.xml

Listening Endpoints Summary...

  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))

  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.100.105)(PORT=1521)))

Services Summary...

Service "ractst" has 2 instance(s).

  Instance "ractst1", status READY, has 1 handler(s) for this service...

  Instance "ractst2", status READY, has 1 handler(s) for this service...

Service "ractstXDB" has 2 instance(s).

  Instance "ractst1", status READY, has 1 handler(s) for this service...

  Instance "ractst2", status READY, has 1 handler(s) for this service...

The command completed successfully

Step 4 : Set the log_status to OFF for the current listener.

LSNRCTL> set log_status off

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))

LISTENER_SCAN1 parameter "log_status" set to OFF

The command completed successfully

Step 5 : Now, go to listener log location ( in my case, current listener is LISTENER_SCAN1 and it;s log file location can be obtained from status of the listener i.e. /u01/app/11.2.0/grid/log/diag/tnslsnr/rac1/listener_scan1).

[oragrid@rac1 trace]$ pwd

/u01/app/11.2.0/grid/log/diag/tnslsnr/rac1/listener_scan1/trace

[oragrid@rac1 trace]$ ls -ltr

total 220

-rw-r----- 1 oragrid oinstall 221096 May 15 14:37 listener_scan1.log

Step 6 : Compress or rename the SCAN LISTENER log file now.

 [oragrid@rac1 trace]$ gzip listener_scan1.log 

[oragrid@rac1 trace]$ ls -ltr

total 12

-rw-r----- 1 oragrid oinstall 11457 May 15 14:37 listener_scan1.log.gz  -- Compressed now.

Step 7 : Go back to the LSNRCTL prompt and turn ON the log_status parameter.

Note : If you are working on the same prompt then again you will have to set current listener to the same SCAN listener name to trun ON logging that you did OFF in step 4

LSNRCTL> set current_listener LISTENER_SCAN1
Current Listener is LISTENER_SCAN1

LSNRCTL> set log_status on
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
LISTENER_SCAN1 parameter "log_status" set to ON
The command completed successfully

Now you would see a new SCAN listener log file is created at that location.

[oragrid@rac1 trace]$ pwd
/u01/app/11.2.0/grid/log/diag/tnslsnr/rac1/listener_scan1/trace

[oragrid@rac1 trace]$ ls -ltr
total 16
-rw-r----- 1 oragrid oinstall 11457 May 15 14:37 listener_scan1.log.gz
-rw-r----- 1 oragrid oinstall    73 May 15 14:53 listener_scan1.log

Hope it would be a help !!

How to compress/purge a listener log file?

Sometimes listener log file grows very large in size and we have to shrink that in order to reclaim space on the mount. I see some people just wonder how they can compress the listener log file while it is currently in use in production system as they can't just stop and start the listener to compress/rename the file as unavailability of listener will cause user connections to fail and that will be encountered as production outage by end user.

Below I have outlined two steps where we tell listener to stop logging in the listener log file and then rename/compress/purge the listener log file and finally we can tell listener back that it can start logging the info in the listener.log file.


Step : 1 - First of all turn off the logging status of listner using below command at lsnrctl prompt.

LSNRCTL> SET LOG_STATUS OFF
Connecting to (ADDRESS=(PROTOCOL=TCP)(HOST=rac1.rajdbsolutions.com)(PORT=1522))
LISTENER parameter "log_status" set to OFF
The command completed successfully

After the above step, go to listener log location and rename the listener.log file to something else e.g. Listener_old.log

Again turn back the listener logging ON using below command.

LSNRCTL> SET LOG_STATUS ON
Connecting to (ADDRESS=(PROTOCOL=TCP)(HOST=rac1.rajdbsolutions.com)(PORT=1522))
LISTENER parameter "log_status" set to ON
The command completed successfully

When you turn ON the listener log using above command, a new listener.log file will be created there and it will be used hereafter for database connectivity.


Hope it would help someone...!!