Moneycontrol Brokerage Recos

Showing posts with label ASM. Show all posts
Showing posts with label ASM. Show all posts

Friday, September 4, 2020

Configure Quorum Disk in Exadata


In last article[click here] - we learnt how to remove quorum disk from the Exadata system, now here we would see it we add and configure that.


Before we are adding quorum disk configuration in the system, you need to have IB switches interface names, ASM binary owner and group handy as that is required to create the config.


[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#


Do the same as above on node2 as well.

[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oragrid --group=asmadmin --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#


Check if the quoum disk configuration is created as below from node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.1

[root@exa01dbadm01 oracle.SupportTools]#
[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oragrid
Group: asmadmin
ifaces: exadata_ib1 exadata_ib0
Initiatior name: iqn.1988-12.com.oracle:192.168.10.3

[root@exa01dbadm02 oracle.SupportTools]#


Now, create quorum disk target for DATAC1 diskgroup which is visible to both compute nodes, before it you need to identify IB interface IPs of both nodes as that is required here.


Run the command on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm01.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01.

[root@exa01dbadm01 oracle.SupportTools]#
[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Created logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1.
[Success] Created backstore QD_DATAC1_exa01dbadm02.
[Success] Created target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02.

[root@exa01dbadm02 oracle.SupportTools]#


Now you can list the quorum disk targets on node1 and node2 as below to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm01 oracle.SupportTools]#
[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Visible to: iqn.1988-12.com.oracle:192.168.10.1, iqn.1988-12.com.oracle:192.168.10.2, iqn.1988-12.com.oracle:192.168.10.3, iqn.1988-12.com.oracle:192.168.10.4
Discovered by:


[root@exa01dbadm02 oracle.SupportTools]#


Create quorum disk device as below on node1 and node2.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm01 oracle.SupportTools]#
[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.1, 192.168.10.2, 192.168.10.3, 192.168.10.4"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.1

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.2

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.3

[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.4

[root@exa01dbadm02 oracle.SupportTools]#



Now you can list the quorum disk device on node1 and node2 to validate.

[root@exa01dbadm01 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm01 oracle.SupportTools]#


[root@exa01dbadm02 oracle.SupportTools]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm02
Host name: exa01dbadm02
ASM disk group name: DATAC1
Size: 128 MB

Device path: /dev/exadata_quorum/QD_DATAC1_exa01dbadm01
Host name: exa01dbadm01
ASM disk group name: DATAC1
Size: 128 MB


[root@exa01dbadm02 oracle.SupportTools]#



Now, check in the ASM instance if quorum disk devices are visible to ASM to be used for the DATAC1 diskgroup as below, we can see that quorum disk devices are available as CANDIDATE disks that we can add in to the diskgroup.

SQL> l
  1* SELECT inst_id, label, path, mode_status, header_status FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%'
SQL> /

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  CANDIDATE
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  CANDIDATE

SQL>




Add the quorum devices in to the diskgroup as below.

SQL> ALTER DISKGROUP datac1 ADD QUORUM FAILGROUP exa01dbadm01 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm01'
QUORUM FAILGROUP exa01dbadm02 DISK '/dev/exadata_quorum/QD_DATAC1_exa01dbadm02';  2

Diskgroup altered.

SQL>


After disk addition, we can see that now these are parts of diskgroup and status is MEMBER.


SQL> SELECT inst_id, label, path, mode_status, header_status
FROM gv$asm_disk WHERE path LIKE '/dev/exadata_quorum/%';  2

   INST_ID LABEL                           PATH                                                                   MODE_ST HEADER_STATU
---------- ------------------------------- ---------------------------------------------------------------------- ------- ------------
         1 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         1 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm02      /dev/exadata_quorum/QD_DATAC1_exa01dbadm02                         ONLINE  MEMBER
         2 QD_DATAC1_exa01dbadm01      /dev/exadata_quorum/QD_DATAC1_exa01dbadm01                         ONLINE  MEMBER

SQL>



Finally, replaced/moved the voting disks back to DATAC1 diskgroup and five copies of files are now available.

[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#


After CRS restart : I bounced the CRS on both nodes to check if things are coming up normally.


[root@exa01dbadm02 oracle.SupportTools]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   cf9c32d1c0c34fafbf47caa1b77821b0 (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   e2e61cd646224f08bf2c4c47e719e8fd (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   7691da5101a34f95bfc56c3bec4e681b (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
 4. ONLINE   3dea97bdc5aa4f61bf0355abe31c1361 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm02) [DATAC1]
 5. ONLINE   cc5671a086eb4f38bf5c690666099d66 (/dev/exadata_quorum/QD_DATAC1_exa01dbadm01) [DATAC1]
Located 5 voting disk(s).
[root@exa01dbadm02 oracle.SupportTools]#


Our missing quorum disk issue stands fixed here......!!




Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.


Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha


Monday, August 31, 2020

Removing the Quorum Disk Configuration in Exadata

 

In this blog post, we would see how to remove quorum disks in Exadata machine. We have one 1/8th Rack Exadata machine for Lab activity where ran into missing quorum disks for Voting Disk and I had to fix that in order to get the system patched in Rolling fashion to maintain high availability.

To patch the Exadata system in rolling fashion, it requires voting disks to reside on HIGH redundancy diskgroup which is not possible in 1/8th Rack Exadata system since there is only 3 cell nodes in there, and for High redundancy it requires five copies of voting disk.


So, in this scenario - we add a separate HDD drive on both compute nodes and share that between the nodes to be used for High redundancy diskgroup creation - there it comes the term "Quorum Disk"


We have +DATAC1 diskgroup configured with HIGH redundancy where our voting disk resides but two of the quorum disks being used in there went missing so our voting disks had only three copies available and CRS was not coming up.


I had to remove the quorum disks from both compute nodes and then configure it back to fix the issue.


SQL> l 
  1* select disk_number, GROUP_NUMBER, MOUNT_STATUS, HEADER_STATUS, MODE_STATUS, STATE, path from v$asm_disk where GROUP_NUMBER=1
SQL> / 

DISK_NUMBER GROUP_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    PATH 
----------- ------------ ------- ------------ ------- -------- ---------------------------------------------------------------------- 
          2            1 MISSING UNKNOWN      OFFLINE NORMAL 
          3            1 MISSING UNKNOWN      OFFLINE NORMAL 
         14            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_05_exad02cel03
         13            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_00_exad02cel03
         15            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_02_exad02cel03
          7            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03
          8            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_04_exad02cel03
          0            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.1;10.0.0.2/DATAC1_CD_01_exad02cel03
          4            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02
         16            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_02_exad02cel02
         17            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_01_exad02cel02
          5            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_00_exad02cel02
          6            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_03_exad02cel02
         11            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.3;10.0.0.4/DATAC1_CD_04_exad02cel02
         19            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_04_exad02cel01
          1            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01
         18            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_00_exad02cel01
          9            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_03_exad02cel01
         12            1 CACHED  MEMBER       ONLINE  NORMAL   o/10.0.0.5;10.0.0.6/DATAC1_CD_01_exad02cel01


Due to missing quorum disk in the DATAC1 diskgroup, we have only three copies of voting disks available as of now.

[root@exa01dbadm02 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   94acaa401bd34fe0bf261af4437fc75e (o/10.0.0.5;10.0.0.6/DATAC1_CD_02_exad02cel01) [DATAC1]
 2. ONLINE   7680d9f206544ff1bf27661bf8cc3250 (o/10.0.0.3;10.0.0.4/DATAC1_CD_05_exad02cel02) [DATAC1]
 3. ONLINE   b632f75fe4884f5cbf3627faa6bbbe07 (o/10.0.0.1;10.0.0.2/DATAC1_CD_03_exad02cel03) [DATAC1]
Located 3 voting disk(s).
[root@exa01dbadm02 ~]#


I moved the voting disks to +RECOC1 diskgroup which was configured with NORMAL REDUNDANCY.


[oragrid@exa01dbadm01 ~]$ crsctl replace votedisk +RECOC1
Successful addition of voting disk 783e7633ea934fd0bff78c8b8dbe5a66.
Successful addition of voting disk a5cf9e2d99134f69bf6bb15c98044623.
Successful addition of voting disk d5d20cd6ff3e4fcdbf68df28cb18d030.
Successful deletion of voting disk 94acaa401bd34fe0bf261af4437fc75e.
Successful deletion of voting disk 7680d9f206544ff1bf27661bf8cc3250.
Successful deletion of voting disk b632f75fe4884f5cbf3627faa6bbbe07.
Successful deletion of voting disk e47d60407b054f3bbf5dc07df8080ee6.
Successfully replaced voting disk group with +RECOC1.
CRS-4266: Voting file(s) successfully replaced


[oragrid@exa01dbadm01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   783e7633ea934fd0bff78c8b8dbe5a66 (o/10.0.0.5;10.0.0.6/RECOC1_CD_02_exad02cel01) [RECOC1]
 2. ONLINE   a5cf9e2d99134f69bf6bb15c98044623 (o/10.0.0.3;10.0.0.4/RECOC1_CD_02_exad02cel02) [RECOC1]
 3. ONLINE   d5d20cd6ff3e4fcdbf68df28cb18d030 (o/10.0.0.1;10.0.0.2/RECOC1_CD_02_exad02cel03) [RECOC1]
Located 3 voting disk(s).
[oragrid@exa01dbadm01 ~]$


Now we can go ahead and drop the missing quorum disks from the DATAC1 diskgroup forcefully.


[oragrid@exa01dbadm01 ~]$ sqlplus "/as sysasm"

SQL*Plus: Release 12.2.0.1.0 Production on Thu Jul 30 12:06:27 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>
SQL> set lines 300
SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm01 FORCE;

Diskgroup altered.

SQL> ALTER DISKGROUP DATAC1 DROP QUORUM DISK QD_DATAC1_exa01dbadm02 FORCE;

Diskgroup altered.

SQL>


Delete the quorum disk device from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.

Delete the quorum disk target from the node1 using quorumdiskmgr utility.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm01
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm01
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm01DATAC1


Now delete quorum disk configuration from the node1.

[root@exa01dbadm01 oracle.SupportTools]# ./quorumdiskmgr --delete --config

[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm01 oracle.SupportTools]#




Now, delete the quorum disk device from node2.

[root@exa01dbadm02 ~]# cd /opt/oracle.SupportTools/
[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --device
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm01.
[Success] Successfully deleted device /dev/exadata_quorum/QD_DATAC1_exa01dbadm02.

Delete quorum disk target from node2.

[root@exa01dbadm02 oracle.SupportTools]# ./quorumdiskmgr --delete --target
[Success] Successfully removed target iqn.2015-05.com.oracle:qd--datac1--exa01dbadm02
[Success] Successfully removed backstore QD_DATAC1_exa01dbadm02
[Success] Successfully removed logical volume /dev/VGExaDb/LVDbVdexa01dbadm02DATAC1

Delete quorum disk configuration from node 2.

[root@exa01dbadm02 oracle.SupportTools]#  ./quorumdiskmgr --delete --config
[Success] Successfully deleted quorum disk configurations

[root@exa01dbadm02 oracle.SupportTools]#


At this point, we are done with removing the quorum disk configuration from our Exadata System, in next post, I would detail how we configure it in the system.





Hope it helps, thanks for reading, please subscribe to this blog to stay updated with latest news on Oracle Cloud Infrastructure and Oracle Autonomous Database Cloud Services and new articles.


Twitter : https://twitter.com/rajsoft8899

Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Facebook : https://www.facebook.com/rkushawaha


Tuesday, August 13, 2019

ORA-15039: diskgroup not dropped


You would get this error when you try to drop an ASM diskgroup that is mounted by an another ASM instance as well e.g. in case of Oracle RAC database clusters, ASM diskgroup shared/mounted by all cluster nodes in oracle RAC configuration.


ASMCMD> dropdg EBS_DATA

ORA-15039: diskgroup not dropped

ORA-15073: diskgroup EBS_DATA is mounted by another ASM instance (DBD ERROR: OCIStmtExecute)

ASMCMD>



Let's verify what all instances have mounted this EBS_DATA diskgroup with following command.


SQL> select inst_id, name, state from gv$asm_diskgroup;

   INST_ID NAME                           STATE
---------- ------------------------------ -----------
         2 VOTEDATA                       MOUNTED
         2 ORCL_DATA                      MOUNTED
         2 ORCL_FRA                       MOUNTED
         2 EBS_DATA                       MOUNTED
         1 VOTEDATA                       MOUNTED
         1 ORCL_DATA                      MOUNTED
         1 ORCL_FRA                       MOUNTED
         1 EBS_DATA                       MOUNTED

8 rows selected.


In above output, you can see that EBS_DATA diskgroup has been mounted by INST_ID=1 and INST_ID=2. 


Now let's go to node 2 and dismount the EBS_DATA diskgroup by logging to ASM instance as "SYSASM"


SQL> alter diskgroup ebs_data dismount;

Diskgroup altered.


SQL>



ASMCMD> lsdg

State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   1024000    92279                0           92279              0             N  ORCL_DATA/
MOUNTED  EXTERN  N         512   4096  1048576    307200   307101                0          307101              0             N  ORCL_FRA/
MOUNTED  NORMAL  N         512   4096  1048576   1024000  1023705           256000          383852              0             N  EBS_DATA/
MOUNTED  HIGH    N         512   4096  1048576     51200    49944            20480            9821              0             Y  VOTEDATA/



Now try to drop the diskgroup, and it works.

ASMCMD> dropdg ebs_data

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   1024000    92279                0           92279              0             N  ORCL_DATA/
MOUNTED  EXTERN  N         512   4096  1048576    307200   307101                0          307101              0             N  ORCL_FRA/
MOUNTED  HIGH    N         512   4096  1048576     51200    49944            20480            9821              0             Y  VOTEDATA/
ASMCMD>



Hope it helps, subscribe to this blog to stay updated on latest Oracle Technologies and new articles.

Twitter : @rajsoft8899
Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/

Sunday, April 24, 2016

Add a new ASM disk to an existing ASM Diskgroup.



Suppose your DATAGROUP1 diskgroup got filled up over defined threshold and there is nothing inside there to be purged in order to reclaim the space then you will need to add some space to that exising Diskgroup.

In that case, Systems Team will add a new HDD device to the host and represent that as a LUN in there and root user will have to create a new ASM disk for that newly added HDD device to be unsed in existing ASM diskgroup.

In our case, new HDD device /dev/sdm1 added on the host as below.


root@rac1 ~]# fdisk -l /dev/sdm

Disk /dev/sdm: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

  Device Boot   Start     End   Blocks  Id System
/dev/sdm1        1    39162  314568733+ 83 Linux


Here lets verify the ASM disks already created on the host using below command.

[root@rac1 ~]# /etc/init.d/oracleasm listdisks
DATA22
DATA23
DATA24
DATA25
DATA26
DATA27
DATA28
DATA29
RECO4


Now let's create a new ASM disk on the new device added as below.


[root@rac1 ~]# /etc/init.d/oracleasm createdisk DATA30 /dev/sdm1
Marking disk "DATA30" as an ASM disk:           [ OK ]


Again verify if new ASM disk created above is listed now.

[root@rac1 ~]# /etc/init.d/oracleasm listdisks
DATA22
DATA23
DATA24
DATA25
DATA26
DATA27
DATA28
DATA29
DATA30     --- Here new ASM disk is created now.
RECO4


Now, it is time to login in ASM instance at SQL prompt and make sure if the new disk is PROVISIONED.

[+ASM]-->sqlplus "/as sysasm"
SQL*Plus: Release 11.2.0.3.0 Production on Sat Mar 12 16:46:09 2016

Copyright (c) 1982, 2011, Oracle. All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option

SQL> select path, header_status from v$asm_disk order by 1;

PATH           HEADER_STATUS
------------------------- ------------------------------------
ORCL:DATA22        MEMBER
ORCL:DATA23        MEMBER
ORCL:DATA24        MEMBER
ORCL:DATA25        MEMBER
ORCL:DATA26        MEMBER
ORCL:DATA27        MEMBER
ORCL:DATA28        MEMBER
ORCL:DATA29        MEMBER
ORCL:DATA30        PROVISIONED
ORCL:RECO4        MEMBER

10 rows selected.


Note : Above we can see new ASM disk created above is PROVISIONED.


Below is the command just to note down the current status/usage of existing ASM diskgroup DATAGROUP1

SQL> select NAME,TOTAL_MB,FREE_MB from v$asm_diskgroup;

NAME
------------------------------------------------------------------------------------------
 TOTAL_MB  FREE_MB
---------- ----------
DATAGROUP1
  6498256   292494

RECOGROUP1
  307196   177404




Let's add the new ASM disk to the diskgroup now.


SQL> alter DISKGROUP DATAGROUP1 ADD DISK 'ORCL:DATA30';

Diskgroup altered.


Again lets verify the new size of ASM diskgroup.


SQL> select NAME,TOTAL_MB,FREE_MB from v$asm_diskgroup;

NAME        TOTAL_MB  FREE_MB
----------------- -------------- ---------------
DATAGROUP1  6805452   599686
RECOGROUP1  307196    177404



Finally we can see that newly added ASM disk is the member of the diskgroup using below command.

SQL> select path, header_status from v$asm_disk order by 1;

PATH           HEADER_STATUS
------------------------- ------------------------------------
ORCL:DATA22        MEMBER
ORCL:DATA23        MEMBER
ORCL:DATA24        MEMBER
ORCL:DATA25        MEMBER
ORCL:DATA26        MEMBER
ORCL:DATA27        MEMBER
ORCL:DATA28        MEMBER
ORCL:DATA29        MEMBER
ORCL:DATA30        MEMBER
ORCL:RECO4        MEMBER

10 rows selected.


Hope it would help someone.....!!

Tuesday, February 2, 2016

ORA-15260: permission denied on ASM disk group


ORA-15260 error comes when you try to drop an ASM diskgroup while connecting as SYSDBA.

SQL> drop diskgroup DATA;
drop diskgroup DATA
*
ERROR at line 1:
ORA-15260: permission denied on ASM disk group


Note : - As we all know that only one ASM instance is required to run multiple databases on a single server so if we have logged in one of the database with sysdba that is associated with that ASM instance and try to drop a disk group which is being used by other databases as well then oracle will not let that happen and throws above error (Suppose if diskgroup gets droped by SYSDBA having connected with one of the database then what would happen to other databases when underlying storage media is not available?) .

So, Oracle 11g onward, oracle requires SYSASM privilege do CREATE or DROP an ASM diskgroup.

let's connect SYS as SYSASM and try to drop the ASM diskgroup now.


SQ> conn /as sysasm

Connected.


SQL> drop diskgroup DATA including contents;

Diskgroup dropped.


SQL> drop diskgroup FRA including contents;

Diskgroup dropped.


Note:- Now you can see diskgroups are dropped being connected with sysasm.


Feel free to contact if any confusion.

Wednesday, July 15, 2015

Oracle 11g R2 Grid Infrastructure/ASM Setup for Standalone Server Step By Step



Oracle Database Automatic Storage Management(ASM):
-----------------------------------------------------------------------------------



DBA's sometimes maintain thousands of datafiles for each database they manage, so an Oracle storage solution should provide both high-performance I/O and failure-proof storage hardware. In fact, file and I/O management what what usually takes up a large part of an Oracle DBA's time. With Oracle's new ASM feature, you can automate traditional file management task. Under an ASM system, the Oracle DBA is in change of the management of physical storage from within Oracle's framework, instead of relying on the system administrator. Using the ASM diskgroups, you can address sets of disk simultaneously, instead of individual disks, and the database can dynamically configure storage based on changing workloads. By allowing the Oracle DBA the flexibility to manage complex storage-management devices across various server and storage platforms, ASM becomes a crucial part of Oracle's grid computing initiative.


ASM is built on OMF, which means you don't have to worry about specifying filenames and location when creating new databases - all you have to do is identifying an ASM disk group, which consists of a set of disks. When you create a database or add a file, you can use familiar CREATE, ALTER and DROP SQL statements to allocate disk space. ASM acts as Oracle's built-in Logical Volume Manager by handling stripping and mirroring functions previously managed by third party tools. Under ASM, disk are grouped and managed by the database itself and made available for creating tablespaces. You don't have to mount the files as with the normal Linux or UNIX file systems. You also can't use the traditional tools, such as cp and tar, to copy the ASM files, nor you can describe them using the ls command. The database holds all information regarding ASM files. If you use ASM for an oracle file, the operating system can't see it. but RMAN and Oracle's other tool can.

For example, issue the following command:

SQL> ALTER DATABASE BACKUP CONTROFILE TO TRACE AS <filename>;

The file generated by the previous statement will display the names of any ASM files. If ASM uses fully qualified names, you can see datafiles in the views such as V$DATAFILE and V$LOGFILE.
When assigning a file to a tablespace or other object in an ASM file system, you don't need to know its name, You can simply refer to a disk group, and ASM automatically generates the filename.
Instead of learning to utilize a whole set of commands to manage ASM database, you can just use the OEM Database Control to manage virtually all ASM operations. You can create a new ASM instance with the DBCA or with the Oracle Universal Installer(which uses DBCA behind the scenes) and you can migrate an exising database to an ASM system with the Database Control.



Benefits of ASM
============

By using ASM, you can manage data by selecting reliability and performance characteristics for data classes, rather than working with the large storage systems on a per-file basis. An ASM file system offers the following benefits:

1 - ASM provides automatic load balancing over all the available disks, thus reducing hot spots in the file system.

2 - ASM prevents fragmentation of disks, so you don't need to manually relocate data to tune I/O performance.

3 - Adding disks is straightforward - ASM automatically performs online disk reorganization when you add or remove storage.

4 - ASM uses redundancy features available in intelligent storage arrays.

5 - The ASM storage system stores all types of database files.

6 - ASM makes your file management tasks easier, because you will be dealing with just a few groups of disks, rather than a multitude of database files. ASM automatically creates the database files and places then in appropriate disk groups.

7 - ASM does mirroring and stripping, which in turn increases reliability and performance. You can select different reliability and performance characteristics for various types of data. For example, you can use fine-grained stripping for redo log files and coarse-grained stripping for regular datafiles.

8 - ASM is free!


Please refer the book  - Expert Oracle Database 11g Administration, written by Sam R. Alapati for detailed information.
===================================================


I have used Oracle Enterprise Linux 5.8 platform for this paper and all required pre-requisites are already done on the host. I would show step by step installation of Grid Infrastructure and it doesn't include Oracle 11g R2 RDBMS installation as its database binary is already installed over there.


This Paper has two parts:

1 - Configuring and Installing Oracle 11g R2 Grid Infrastructure.

2 - Create Oracle 11g R2 Database based on ASM storage.


Let's Proceed with the 1st part : Configure and Install Grid Infrastructure


Step : 1 - Download Oracle 11g R2 Grid Infrastructure and Oracle 11g R2 database software from Oracle portal [http://www.oracle.com/technetwork/database/clusterware/overview/index-096607.html]


linux_11gR2_grid.zip  --- Grid Infrastructure Software

linux_11gR2_database_1of2.zip   --- Database Software Part - I
linux_11gR2_database_2of2.zip   --- Database Software Part - II



[root@localhost disks]# rpm -qa | grep oracleasm
oracleasm-support-2.1.7-1.el5

Note : - As we are using OEL 5.8 so oracle has already included ASM library packages in there by default so no need to install them separately. All you need is just a oracleasm-support package.


Step : 2 - In order to install Oracle 11g R2 Grid Infrastructure with success, list of below OS rpm packages need to installed on the server.


32-bit (x86) Installations
------------------------

binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.52
glibc-headers-2.5
kernel-headers-2.6.18
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libgomp-4.1.2
libstdc++-4.1.2
libstdc++-devel-4.1.2
make-3.81
pdksh-5.2.14
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11


Please Refer Oracle Doc - [http://docs.oracle.com/cd/E11882_01/install.112/e41961.pdf] for completed OS level configuration for Grid Infrastructure Installation.


Step : 3 - Create appropriate OS groups and oragrid user for Grid installation.

[root@localhost ~]# groupadd oinstall
[root@localhost ~]# groupadd osdba
[root@localhost ~]# groupadd asmadmin
[root@localhost ~]# groupadd asmdba
[root@localhost ~]# groupadd asmoper
[root@localhost ~]# groupadd osoper


[root@localhost ~]# useradd -g oinstall -G asmdba,asmadmin,asmoper oragrid


[oragrid@localhost ~]$ id
uid=1101(oragrid) gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1202(asmadmin),1203(asmoper)

[root@localhost ~]# useradd -u 1101 -g oinstall -G dba oracle - This account is used for Oracle Database installation.


Step : 4 -  We will be installing Oracle 11g R2 GI under "oragrid" user which would be separate from oracle database user (oracle). As we know, Oracle Automatic Storage Management needs devices not formatted with any file system so we will be adding 4 new Hard Disks in the VM, first two of them will be used for  OCR_DATA disk group which has Normal redundancy, in our case. Third disk will be used for FRA diskgroup and 4th device will be used for DATA diskgroup to store all database files in there.

a) Click on VM at menu bar and click on Settings.



b) Now select Hard Disk at left panel and click on Add at bottom.



c) Again select Hard Disk and Click on Next.



d) Select SCSI(Recommended) Option and Click on Next.

e) Select "Create a new virtual disk"and click on Next.


f) Specify the disk size to be added - 2.00GB and click on Next.



g) Click on Finish then you would see a new HDD of 2GB added.



Following the above steps(a-g) add two other 2GB device and add another 20GB device as depicted below.



Step : 5 - In order to list the devices using fdisk -l command, restart the system.



Step : 6 - Create partitions of the devices you added above. 

[root@localhost ~]# fdisk -l

Disk /dev/sda: 85.8 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        5235    41945715   83  Linux
/dev/sda3            5236        7846    20972857+  83  Linux
/dev/sda4            7847       10443    20860402+   5  Extended
/dev/sda5            7847        8368     4192933+  82  Linux swap / Solaris
/dev/sda6            8369        8559     1534176   83  Linux

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn't contain a valid partition table

Disk /dev/sdd: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/sde: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sde doesn't contain a valid partition table


Above highlighted Disks are the Hard Disk devices you added in previous step 4. Now let's create partition of these one by one.


a) [root@localhost ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n   - here n stand for Create New partition

Command action
   e   extended
   p   primary partition (1-4)
p  --  Here p for primary partition
Partition number (1-4): 1  -- Given 1 here
First cylinder (1-261, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):   -- Press Enter to accept its default value.
Using default value 261

Command (m for help): w  -- press w here to save the info in partition table and quit.
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.


======= Same way partition all other 3 devices added and listed above ======



Now you would partitions are created for all the devices as follows:

[root@localhost ~]# fdisk -l

Disk /dev/sda: 85.8 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        5235    41945715   83  Linux
/dev/sda3            5236        7846    20972857+  83  Linux
/dev/sda4            7847       10443    20860402+   5  Extended
/dev/sda5            7847        8368     4192933+  82  Linux swap / Solaris
/dev/sda6            8369        8559     1534176   83  Linux

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         261     2096451   83  Linux

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1         261     2096451   83  Linux

Disk /dev/sdd: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1         261     2096451   83  Linux

Disk /dev/sde: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1        2610    20964793+  83  Linux


Step : 7 - Create Oracle ASM disks to make it available to Oracle ASM to created disk groups for the grid/database data storage.


[root@localhost ~]# oracleasm createdisk DISK1 '/dev/sdb1'
Writing disk header: done
Instantiating disk: done

[root@localhost ~]# oracleasm createdisk DISK2 '/dev/sdc1'
Writing disk header: done
Instantiating disk: done

[root@localhost ~]# oracleasm createdisk DISK3 '/dev/sdd1'
Writing disk header: done
Instantiating disk: done

[root@localhost ~]# oracleasm createdisk DISK4 '/dev/sde1'
Writing disk header: done
Instantiating disk: done



Step : 8 - Configure the Oracle ASM library owner and start mode of ASM library driver on start-up and scanning of Oracle ASM disk on system reboot automatically for you so that you don't have to start ASM services manually.


[root@localhost ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [oragrid]: 
Default group to own the driver interface [oinstall]: 
Start Oracle ASM library driver on boot (y/n) [y]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done



Note : In this case oracleasm was already configured for the user oragrid so I didn't provide any input for default user and default group but if in your case the user is root then provide appropriate user and default group accordingly. Now if you go the physical location of disks created, you would see its owner has changed to oragrid as follows.

[root@localhost disks]# pwd
/dev/oracleasm/disks

[root@localhost disks]# ls -ltr
total 0
brw-rw---- 1 oragrid oinstall 8, 65 Jul 14 15:09 DISK4
brw-rw---- 1 oragrid oinstall 8, 49 Jul 14 15:09 DISK3
brw-rw---- 1 oragrid oinstall 8, 33 Jul 14 15:09 DISK2
brw-rw---- 1 oragrid oinstall 8, 17 Jul 14 15:09 DISK1


Note : If above user and group don't belong to the user/group under which you are running Oracle Universal Installer to install Grid Infrastructure then you won't to be able to see the list of disks on disk group creation page. This is where if you are not properly configured/set then you won't be able to see any disks at GUI even try with changing the disk discovery path so be ensured you have configured oracle asm($oracleasm configure -i) with correct user/group.



Step : 9 - Check to see the list of ASM disks created.


[root@localhost ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@localhost ~]# 


[root@localhost ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4

Fine ASM disks are created now.


Step : 10 - Now login with oragrid user(I have created this separate user to run ASM instance. we will be using oracle user for oracle database instance).

Navigate to the appropriate installer location and run the ./runInstaller

[oragrid@localhost grid]$ ./runInstaller 
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 80 MB.   Actual 998 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 4094 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-07-14_12-34-17PM. Please wait ...




Select Install and Configure Grid Infrastructure for  Standalone Server and Click on Next >> again click on Next.



Well, no disks are listed here to be used to create Disk Group so we will change the disk discovery path where oracle ASM disks physically located. Click on "Change Discovery Path" and give this path in the new pop-up box - /dev/oracleasm/disks(this is where your oracle ASM disks physically exists) and click on OK


Now we can see all disks are listed as in the above Fig.


TO store OCR/voting disk and all we will create OCR_DATA disk group with "Normal Redundancy" with two disks as Normal Redundancy requires at least two disks for mirroring its data across disks to prevent disk failure and survive its ASM operation. Click on Next.



Here I have chosen same password for SYS ans ASMSNMP accounts but if you want to separate then choose accordingly.


As Oracle recommends to use complex password so it flashes a warning message if we choose normal password so you can ignore and click on Next.


Click on Next.


Click on Next.



It's all ignorable so just click on Next but don't compromise for production server :-)




Step : 11 - Finally it asks to run root.sh script as root user in order to complete the installation process and Click on OK.


[root@localhost ~]# /u01/app/oragrid/product/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oragrid
    ORACLE_HOME=  /u01/app/oragrid/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-07-14 13:07:21: Checking for super user privileges
2015-07-14 13:07:21: User has super user privileges
2015-07-14 13:07:21: Parsing the host name
Using configuration parameter file: /u01/app/oragrid/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE 
Creating OCR keys for user 'oragrid', privgrp 'oinstall'..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-5-8.0.2




localhost     2015/07/14 13:07:47     /u01/app/oragrid/product/11.2.0/grid/cdata/localhost/backup_20150714_130747.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4094 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.


Finally Click on Close button page.


Step : 12 - Now Let's verify our Grid Infrastructure Installation.

[oragrid@localhost ~]$ ps -ef | grep pmon
oragrid  10309 10151  0 17:03 pts/1    00:00:00 grep pmon
oragrid  10618     1  0 13:15 ?        00:00:03 asm_pmon_+ASM


[oragrid@localhost ~]$ ps -ef | grep css
oragrid  10311     1  0 13:14 ?        00:00:08 /u01/app/oragrid/product/11.2.0/grid/bin/cssdagent
oragrid  10348     1  1 13:14 ?        00:02:31 /u01/app/oragrid/product/11.2.0/grid/bin/ocssd.bin 
oragrid  10367 10151  0 17:03 pts/1    00:00:00 grep css

[oragrid@localhost ~]$ ps -ef | grep asm
oragrid  10618     1  0 13:15 ?        00:00:04 asm_pmon_+ASM
oragrid  10622     1  0 13:15 ?        00:00:46 asm_vktm_+ASM
oragrid  10628     1  0 13:15 ?        00:00:00 asm_gen0_+ASM
oragrid  10632     1  0 13:15 ?        00:00:01 asm_diag_+ASM
oragrid  10636     1  0 13:15 ?        00:00:00 asm_psp0_+ASM
oragrid  10640     1  0 13:15 ?        00:00:09 asm_dia0_+ASM
oragrid  10644     1  0 13:15 ?        00:00:00 asm_mman_+ASM
oragrid  10648     1  0 13:15 ?        00:00:03 asm_dbw0_+ASM
oragrid  10652     1  0 13:15 ?        00:00:01 asm_lgwr_+ASM
oragrid  10656     1  0 13:15 ?        00:00:01 asm_ckpt_+ASM
oragrid  10660     1  0 13:15 ?        00:00:00 asm_smon_+ASM
oragrid  10664     1  0 13:15 ?        00:00:03 asm_rbal_+ASM
oragrid  10668     1  0 13:15 ?        00:00:10 asm_gmon_+ASM
oragrid  10672     1  0 13:15 ?        00:00:01 asm_mmon_+ASM
oragrid  10676     1  0 13:15 ?        00:00:03 asm_mmnl_+ASM


We can see ASM instance is running. Let's login and validate its instance.

[oragrid@localhost ~]$ . oraenv
ORACLE_SID = [] ? +ASM
The Oracle base for ORACLE_HOME=/u01/app/oragrid/product/11.2.0/grid is /u01/app/oragrid

[oragrid@localhost ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 14 17:05:59 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Automatic Storage Management option

SQL> set linesize 220
SQL> select * from v$instance;

INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME                                                        VERSION           STARTUP_T STATUS       PAR    THREAD# ARCHIVE LOG_SWITCH_WAIT LOGINS     SHU DATABASE_STATUS
--------------- ---------------- ---------------------------------------------------------------- ----------------- --------- ------------ --- ---------- ------- --------------- ---------- --- -----------------
INSTANCE_ROLE      ACTIVE_ST BLO
------------------ --------- ---
              1 +ASM             localhost.localdomain                                            11.2.0.1.0        14-JUL-15 STARTED      NO           0 STOPPED               ALLOWED    NO  ACTIVE
UNKNOWN            NORMAL    NO


SQL> select name from v$asm_diskgroup;

NAME
------------------------------
OCR_DATA
DATA
FRA


[oragrid@localhost ~]$ asmcmd

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576     20473    19017                0           19017              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576      2047     1735                0            1735              0             N  FRA/
MOUNTED  NORMAL  N         512   4096  1048576      4094     3974                0            1987              0             N  OCR_DATA/


Note : - I have created two other diskgroups named DATA and FRA to be used for database creation in part 2. I will explain how to create and manage an Oracle ASM diskgroup in a separate post.

Oracle 11g R2 Grid Infrastructure Installation completed successfully now.



Let's proceed to create a database based on the above Automatic Storage Management.

 As earlier I discussed that we have separate user for oracle database instance  and oracle binary already installed over there so in order to create a database based on ASM storage let's just switch to oracle user and run DBCA from its home 


Step : 1 - Launch DBA and click on Next. On Create Database page click on Next.

Step : 2 - Select General Purpose or Transaction Procession >> Click on Next

Step : 3 - Provided database name : testdb >> Click on Next.

Step : 4 -  Click on Next.

Step : 5 - Provide passwords and click on Next.

Step : 6 - Select Automatic Storage Management for Storage Type option. For Storage Locations - Use Oracle-Managed Files >> Click on browse and select appropriate diskgroup(DATA in our case) to be used for oracle database files and click on OK. >> Finally click on Next and provide the password you given earlier in step 5.

Step : 7 - Select option Flash Recovery Area and click on browse to choose appropriate diskgroup to be used for FRA and click enable archiving option and click on Next.

Step : 8 - Click on Next.

Step : 9 - Click on Next.

Step : 10 - Click on Next.

Step : 11 - Click on Finish and OK in order to starting database creation.



Now Database creation completed so lets verify its pmon process.

[oracle@localhost ~]$ ps -ef | grep pmon
oragrid  10618     1  0 13:15 ?        00:00:04 asm_pmon_+ASM
oracle   16770     1  0 17:43 ?        00:00:00 ora_pmon_testdb
oracle   26885 10204  0 18:02 pts/2    00:00:00 grep pmon


Note :- Here you can see that ASM instance is running under oragrid user and oracle database instance is running under oracle user.


[oracle@localhost ~]$ . oraenv   -- To set correct environment 
ORACLE_SID = [asmdb] ? testdb  -- given testdb in order to set its env
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@localhost ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jul 14 18:04:06 2015

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> set linesize 220
SQL> select name from v$datafile;

NAME
----------------------------------------------------------------
+DATA/testdb/datafile/system.260.885058597
+DATA/testdb/datafile/sysaux.265.885058597
+DATA/testdb/datafile/undotbs1.264.885058597
+DATA/testdb/datafile/users.263.885058599

SQL> show parameter control

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time        integer     7
control_files                        string      +DATA/testdb/controlfile/curre
                                                 nt.262.885058851, +FRA/testdb/
                                                 controlfile/current.256.885058
                                                 853
control_management_pack_access       string      DIAGNOSTIC+TUNING
SQL> 


SQL> select group#, status, member from v$logfile;

    GROUP# STATUS  MEMBER
---------- ------- ------------------------------------------------------------
         3         +DATA/testdb/onlinelog/group_3.258.885058861
         3         +FRA/testdb/onlinelog/group_3.257.885058863
         2         +DATA/testdb/onlinelog/group_2.259.885058859
         2         +FRA/testdb/onlinelog/group_2.258.885058859
         1         +DATA/testdb/onlinelog/group_1.261.885058855
         1         +FRA/testdb/onlinelog/group_1.259.885058857



So our database is finally created over Automatic Storage Management(ASM) storage.


Hope it would help someone....!!


Wednesday, June 17, 2015

Migrate Non-ASM Oracle Database to ASM Storage:

A friend, Vishnu Sai asked me if I have any document to migrate a Non-ASM Database to ASM storage so I thought to create a quick note for him today. I have created a ASM diskgroup named '+DATA' and will be using the same to store all database files in there.


Step 1 - List the database files need to be migrated to ASM storage:


SET TERMOUT OFF;
COLUMN current_instance NEW_VALUE current_instance NOPRINT;
SELECT rpad(instance_name, 17) current_instance FROM v$instance;
SET TERMOUT ON;

PROMPT
PROMPT +------------------------------------------------------------------------+
PROMPT | Report   : Data File Report (all physical files)                       |
PROMPT | Instance : &current_instance                                           |
PROMPT +------------------------------------------------------------------------+

SET ECHO        OFF
SET FEEDBACK    6
SET HEADING     ON
SET LINESIZE    180
SET PAGESIZE    50000
SET TERMOUT     ON
SET TIMING      OFF
SET TRIMOUT     ON
SET TRIMSPOOL   ON
SET VERIFY      OFF

CLEAR COLUMNS
CLEAR BREAKS
CLEAR COMPUTES

COLUMN tablespace      FORMAT a30                 HEADING 'Tablespace Name / File Class'
COLUMN filename        FORMAT a75                 HEADING 'Filename'
COLUMN filesize        FORMAT 9,999,999,999,999   HEADING 'File Size'
COLUMN autoextensible  FORMAT a4                  HEADING 'Auto'
COLUMN increment_by    FORMAT 999,999,999,999     HEADING 'Next'
COLUMN maxbytes        FORMAT 999,999,999,999     HEADING 'Max'

BREAK ON report

COMPUTE sum OF filesize  ON report

SELECT /*+ ordered */
    d.tablespace_name                     tablespace
  , d.file_name                           filename
  , d.bytes                               filesize
  , d.autoextensible                      autoextensible
  , d.increment_by * e.value              increment_by
  , d.maxbytes                            maxbytes
FROM
    sys.dba_data_files d
  , v$datafile v
  , (SELECT value
     FROM v$parameter
     WHERE name = 'db_block_size') e
WHERE
  (d.file_name = v.name)
UNION
SELECT
    d.tablespace_name                     tablespace
  , d.file_name                           filename
  , d.bytes                               filesize
  , d.autoextensible                      autoextensible
  , d.increment_by * e.value              increment_by
  , d.maxbytes                            maxbytes
FROM
    sys.dba_temp_files d
  , (SELECT value
     FROM v$parameter
     WHERE name = 'db_block_size') e
UNION
SELECT
    '[ ONLINE REDO LOG ]'
  , a.member
  , b.bytes
  , null
  , TO_NUMBER(null)
  , TO_NUMBER(null)
FROM
    v$logfile a
  , v$log b
WHERE
    a.group# = b.group#
UNION
SELECT
    '[ CONTROL FILE    ]'
  , a.name
  , TO_NUMBER(null)
  , null
  , TO_NUMBER(null)
  , TO_NUMBER(null)
FROM
    v$controlfile a
ORDER BY 1,2
/


=====================
Tablespace Name / File Class   Filename                                                    File Size Auto             Next              Max
------------------------------ -------------------------------------------------- ------------------ ---- ---------------- ----------------
SYSAUX                         /u01/app/oracle/oradata/testdb/sysaux01.dbf               450,887,680 YES        10,485,760   34,359,721,984
SYSTEM                         /u01/app/oracle/oradata/testdb/system01.dbf               702,545,920 YES        10,485,760   34,359,721,984
TEMP                           /u01/app/oracle/oradata/testdb/temp01.dbf                  20,971,520 YES           655,360   34,359,721,984
UNDOTBS1                       /u01/app/oracle/oradata/testdb/undotbs01.dbf               31,457,280 YES         5,242,880   34,359,721,984
USERS                          /u01/app/oracle/oradata/testdb/users01.dbf                  5,242,880 YES         1,310,720   34,359,721,984
[ CONTROL FILE    ]            /u01/app/oracle/flash_recovery_area/testdb/control
                               02.ctl

[ CONTROL FILE    ]            /u01/app/oracle/oradata/testdb/control01.ctl
[ ONLINE REDO LOG ]            /u01/app/oracle/oradata/testdb/redo01.log                  52,428,800
[ ONLINE REDO LOG ]            /u01/app/oracle/oradata/testdb/redo02.log                  52,428,800
[ ONLINE REDO LOG ]            /u01/app/oracle/oradata/testdb/redo03.log                  52,428,800
                                                                                  ------------------
sum                                                                                    1,368,391,680

10 rows selected.


Above is the list of all database files these we will be migrating to ASM storage from filesystem.


Step 2 - If your database is running using pfile then just edit the following parameters pointing to appropriate ASM diskgroups in the pfile and shutdown the database normal and start in nomount state.

control_files='=+DATA'
db_create_file_dest='+DATA'
db_recovery_file_dest='+DATA'

and if your database is running using spfile then simply alter the parameters accordingly.


SQL> ALTER SYSTEM SET CONTROL_FILES='+DATA' SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET DB_CREATE_FILE_DEST='+DATA' SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='+DATA' SCOPE=SPFILE;

System altered.


Step 3 - Shutdown the DB normally and start it up in nomount state.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.


SQL> startup nomount;
ORACLE instance started.

Total System Global Area  841162752 bytes
Fixed Size                  1339768 bytes
Variable Size             494931592 bytes
Database Buffers          339738624 bytes
Redo Buffers                5152768 bytes



Step 4 - Restore the control file  to ASM diskgroup '+DATA' from its original filesystem location.


[oracle@localhost ~]$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Wed Jun 17 15:57:43 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: TESTDB (not mounted)



RMAN> restore controlfile from '/u01/app/oracle/flash_recovery_area/testdb/control02.ctl';

Starting restore at 17-JUN-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=24 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+DATA/testdb/controlfile/current.264.882633485
Finished restore at 17-JUN-15

RMAN>


Step 5 - As of now, we have migrated the database controlfile to ASM storage now so lets start the database in MOUNT state.

SQL> alter database mount;

Database altered.



Step 6 - Backup the database as copy to the ASM storage.

RMAN> backup as copy database format '+DATA';

Starting backup at 17-JUN-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=22 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=/u01/app/oracle/oradata/testdb/system01.dbf
output file name=+DATA/testdb/datafile/system.263.882633749 tag=TAG20150617T160228 RECID=1 STAMP=882633766
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=/u01/app/oracle/oradata/testdb/sysaux01.dbf
output file name=+DATA/testdb/datafile/sysaux.262.882633775 tag=TAG20150617T160228 RECID=2 STAMP=882633789
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=/u01/app/oracle/oradata/testdb/undotbs01.dbf
output file name=+DATA/testdb/datafile/undotbs1.269.882633801 tag=TAG20150617T160228 RECID=3 STAMP=882633800
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:02
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=+DATA/testdb/controlfile/backup.259.882633801 tag=TAG20150617T160228 RECID=4 STAMP=882633802
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=/u01/app/oracle/oradata/testdb/users01.dbf
output file name=+DATA/testdb/datafile/users.258.882633803 tag=TAG20150617T160228 RECID=5 STAMP=882633803
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 17-JUN-15
channel ORA_DISK_1: finished piece 1 at 17-JUN-15
piece handle=+DATA/testdb/backupset/2015_06_17/nnsnf0_tag20150617t160228_0.257.882633807 tag=TAG20150617T160228 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:02
Finished backup at 17-JUN-15



Step 7 - So now we have a copy of our current database at ASM storage now we can switch the database to copy.


RMAN> SWITCH DATABASE TO COPY;

datafile 1 switched to datafile copy "+DATA/testdb/datafile/system.263.882633749"
datafile 2 switched to datafile copy "+DATA/testdb/datafile/sysaux.262.882633775"
datafile 3 switched to datafile copy "+DATA/testdb/datafile/undotbs1.269.882633801"
datafile 4 switched to datafile copy "+DATA/testdb/datafile/users.258.882633803"

RMAN>



Step 8 - Now open the database.


SQL> alter database open;

Database altered.

SQL>


Step 9 - Now create the new temp file and drop the old one so that it is also stored on ASM.


SQL> select tablespace_name, file_name, bytes from dba_temp_files;

TABLESPACE_NAME                FILE_NAME                                               BYTES
------------------------------ -------------------------------------------------- ----------
TEMP                           /u01/app/oracle/oradata/testdb/temp01.dbf            20971520


SQL> alter tablespace temp add tempfile size 50m autoextend on maxsize 2048m;

Tablespace altered.

SQL> select tablespace_name, file_name, bytes from dba_temp_files;

TABLESPACE_NAME                FILE_NAME                                               BYTES
------------------------------ -------------------------------------------------- ----------
TEMP                           /u01/app/oracle/oradata/testdb/temp01.dbf            20971520
TEMP                           +DATA/testdb/tempfile/temp.256.882634443             52428800


Now drop the old temp file which is stored on Filesystem.


SQL> alter database tempfile '/u01/app/oracle/oradata/testdb/temp01.dbf' drop including datafiles;

Database altered.

SQL> select tablespace_name, file_name, bytes from dba_temp_files;

TABLESPACE_NAME                FILE_NAME                                               BYTES
------------------------------ -------------------------------------------------- ----------
TEMP                           +DATA/testdb/tempfile/temp.256.882634443             52428800



Step 10 - Let's migrate the database redo log files to ASM storage now.

SQL> select a.group#, a.member, b.bytes, b.status from v$logfile a, v$log b where a.group# = b.group#;

    GROUP# MEMBER                                                  BYTES STATUS
---------- -------------------------------------------------- ---------- ----------------
         3 /u01/app/oracle/oradata/testdb/redo03.log            52428800 CURRENT
         2 /u01/app/oracle/oradata/testdb/redo02.log            52428800 ACTIVE
         1 /u01/app/oracle/oradata/testdb/redo01.log            52428800 INACTIVE



SQL> alter database drop logfile group 1;

Database altered.

SQL> alter database add logfile group 1 size 50m;

Database altered.



SQL> select a.group#, a.member, b.bytes, b.status from v$logfile a, v$log b where a.group# = b.group#;

    GROUP# MEMBER                                                  BYTES STATUS
---------- -------------------------------------------------- ---------- ----------------
         3 /u01/app/oracle/oradata/testdb/redo03.log            52428800 INACTIVE
         2 /u01/app/oracle/oradata/testdb/redo02.log            52428800 CURRENT
         1 +DATA/testdb/onlinelog/group_1.287.882635039         52428800 INACTIVE
         1 +DATA/testdb/onlinelog/group_1.286.882635039         52428800 INACTIVE

SQL> alter database drop logfile group 3;

Database altered.

SQL> alter database add logfile group 3 size 50m;

Database altered.



SQL> select a.group#, a.member, b.bytes, b.status from v$logfile a, v$log b where a.group# = b.group#;

    GROUP# MEMBER                                                  BYTES STATUS
---------- -------------------------------------------------- ---------- ----------------
         3 +DATA/testdb/onlinelog/group_3.282.882635099         52428800 INACTIVE
         2 /u01/app/oracle/oradata/testdb/redo02.log            52428800 INACTIVE
         1 +DATA/testdb/onlinelog/group_1.287.882635039         52428800 CURRENT
         1 +DATA/testdb/onlinelog/group_1.286.882635039         52428800 CURRENT
         3 +DATA/testdb/onlinelog/group_3.280.882635099         52428800 INACTIVE

SQL> alter database drop logfile group 2;

Database altered.

SQL> alter database add logfile group 2 size 50m;

Database altered.



Now we are all migrated to ASM storage.


SQL> select a.group#, a.member, b.bytes, b.status from v$logfile a, v$log b where a.group# = b.group#;

    GROUP# MEMBER                                                  BYTES STATUS
---------- -------------------------------------------------- ---------- ----------------
         3 +DATA/testdb/onlinelog/group_3.282.882635099         52428800 INACTIVE
         2 +DATA/testdb/onlinelog/group_2.289.882635185         52428800 UNUSED
         1 +DATA/testdb/onlinelog/group_1.287.882635039         52428800 CURRENT
         1 +DATA/testdb/onlinelog/group_1.286.882635039         52428800 CURRENT
         3 +DATA/testdb/onlinelog/group_3.280.882635099         52428800 INACTIVE
         2 +DATA/testdb/onlinelog/group_2.290.882635187         52428800 UNUSED




Finally let's verify if all database files are migrated to ASM storage now:



Tablespace Name / File Class   Filename                                                                             File Size Auto             Next           Max
------------------------------ --------------------------------------------------------------------------- ------------------ ---- ---------------- ----------------
SYSAUX                         +DATA/testdb/datafile/sysaux.262.882633775                                         450,887,680 YES        10,485,760   34,359,721,984
SYSTEM                         +DATA/testdb/datafile/system.263.882633749                                         702,545,920 YES        10,485,760   34,359,721,984
TEMP                           +DATA/testdb/tempfile/temp.256.882634443                                            52,428,800 YES        52,428,800    2,147,483,648
UNDOTBS1                       +DATA/testdb/datafile/undotbs1.269.882633801                                        31,457,280 YES         5,242,880   34,359,721,984
USERS                          +DATA/testdb/datafile/users.258.882633803                                            5,242,880 YES         1,310,720   34,359,721,984
[ CONTROL FILE    ]            +DATA/testdb/controlfile/current.264.882633485
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_1.286.882635039                                        52,428,800
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_1.287.882635039                                        52,428,800
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_2.289.882635185                                        52,428,800
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_2.290.882635187                                        52,428,800
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_3.280.882635099                                        52,428,800
[ ONLINE REDO LOG ]            +DATA/testdb/onlinelog/group_3.282.882635099                                        52,428,800
                                                                                                           ------------------
sum                                                                                                             1,557,135,360

12 rows selected.


Conclusion : Finally we are migrated from Non-ASM storage to ASM now.