Moneycontrol Brokerage Recos

Showing posts with label Performance Tuning. Show all posts
Showing posts with label Performance Tuning. Show all posts

Tuesday, August 13, 2019

KSFV I/O slave Ic01 dp=0x132e70f9f3 exiting



This is just an informational message written in the alert log when you have set either of the following parameters in the database parameter file other than its default value.


Alert message:

KSFV I/O slave Ic01 dp=0x132e70f9f3 exiting

The alert message just indicates the startup and shutdown of backup i/o slave processes.

Default values :

SQL> show parameter slave

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
backup_tape_io_slaves                boolean     FALSE
dbwr_io_slaves                       integer     0
SQL>


Non-default values:

BACKUP_TAPE_IO_SLAVES = TRUE

or

DBWR_IO_SLAVES > 0



Hope it helps, subscribe to this blog to stay updated on latest Oracle Technologies and new articles.

Twitter : @rajsoft8899


Tuesday, October 17, 2017

ORION - ORacle I/O Numbers - I/O Bench-marking Tool in Oracle



ORION - ORacle I/O Numbers :  An I/O Bench-marking Tool in Oracle


Generally, you do the workload testing of your production database system or the new hardware arrived in your data center when your Oracle RDBMS server already running with actual application data but you can simulate the similar I/O testing with Oracle provided Tool called "Orion", an I/O bench-marking Tool. It helps you predict the performance of I/O Bench-marking for new hardware system for Oracle database.

The ORION tool has been available for some time now and honestly I was not aware of it so far. While exploring the advanced performance tuning of Oracle RDBMS, I came to know about this nice pre-bundled script to evaluate the I/O performance of your Oracle database on new hardware where you are going to deploy or migrate your database.

Initially this tool was available to download from Oracle OTN but since Oracle 11.2 onward, it is bundled inside Orale RDBMS database software package itself. So you can invoke from $ORACLE_HOME/bin directory to work with it.

ORION works best during the evaluation phase of new platform without application-data on your logical unit numbers(LUNs). It is specifically designed to simulate the I/O workload on your new hardware system for Oracle database. This package uses asynchronous I/O and large pages for read and writes to concurrently submit I/O requests to operating system. Asynchronous I/O uses "libaio" on Linux platform. You can quite easily see that the ORION slave processes use "io_submit" and "io_getevents" by tracing strace on this.


Here, let's go ahead and fake some LUNs to perform evaluation of I/O on it.

-bash-4.1$ dd if=/dev/zero of=lun01.file bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 24.0865 s, 44.6 MB/s


-bash-4.1$ dd if=/dev/zero of=lun02.file bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 12.9167 s, 83.1 MB/s


-bash-4.1$ dd if=/dev/zero of=lun03.file bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.00802 s, 1.1 GB/s


-bash-4.1$ dd if=/dev/zero of=lun04.file bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.01502 s, 1.1 GB/s



You can see LUNs are faked inside /u01 mount by listing the directory.

bash-4.1$ pwd
/u01
-bash-4.1$ ls -ltr *.file
-rw-r--r--. 1 oracle oinstall 1073741824 Oct 17 06:53 lun01.file
-rw-r--r--. 1 oracle oinstall 1073741824 Oct 17 06:53 lun02.file
-rw-r--r--. 1 oracle oinstall 1073741824 Oct 17 06:54 lun03.file
-rw-r--r--. 1 oracle oinstall 1073741824 Oct 17 06:54 lun04.file


Once the LUNs are created, make a group file with list of LUNs we created earlier with its obsolute file path. I created a group file /u01/orion.lun as below.

-bash-4.1$ cat /u01/orion.lun 
/u01/lun01.file
/u01/lun02.file
/u01/lun03.file
/u01/lun04.file


Finally, invoke orion executable from $ORACLE_HOME/bin location as below. As soon as you start the command to evaluate the I/O testing, this tool will tell you the approximate time it will take to complete.

-bash-4.1$ $ORACLE_HOME/bin/orion -run oltp -testname /u01/orion -hugenotneeded


Output should be like below.





Below I have generated a graph based on above out to relate Latency and and corresponding # No. of I/Os(Read) per second. You can realise that lower the latency the higher #No. of I/O(Reads)




As part of the I/O performance evaluation, orion tool creates multiple CSV files, text files with detailed metrics you should look at for evaluating IOPS, MBPS operations.


-rw-r--r--. 1 oracle oinstall   355 Oct 17 07:00 orion_20171017_0700_summary.txt
-rw-r--r--. 1 oracle oinstall     0 Oct 17 07:00 orion_20171017_0700_mbps.csv
-rw-r--r--. 1 oracle oinstall     0 Oct 17 07:00 orion_20171017_0700_lat.csv
-rw-r--r--. 1 oracle oinstall     0 Oct 17 07:00 orion_20171017_0700_iops.csv
-rw-r--r--. 1 oracle oinstall     0 Oct 17 07:00 orion_20171017_0700_hist.txt
-rw-r--r--. 1 oracle oinstall   515 Oct 17 07:00 orion_20171017_0700_trace.txt
-rw-r--r--. 1 oracle oinstall 56161 Oct 17 07:22 orion_20171017_0702_trace.txt
-rw-r--r--. 1 oracle oinstall   570 Oct 17 07:22 orion_20171017_0702_mbps.csv
-rw-r--r--. 1 oracle oinstall   811 Oct 17 07:22 orion_20171017_0702_lat.csv
-rw-r--r--. 1 oracle oinstall   742 Oct 17 07:22 orion_20171017_0702_iops.csv
-rw-r--r--. 1 oracle oinstall 19494 Oct 17 07:22 orion_20171017_0702_hist.txt
-rw-r--r--. 1 oracle oinstall  2008 Oct 17 07:22 orion_20171017_0702_summary.txt



Below is the contents I extracted from orion_20171017_0700_iops.csv
file and it has more details for Small I/O and Large I/O statistics in there.
 
This comma-separated-value file contains the rates sustained by small I/Os in IOPS. 
Each value corresponds to a data point test that used a fixed number of outstanding small and large I/Os. 
The number of outstanding small I/Os for a value is specified by its column header in the first row. 
The number of outstanding large I/Os for a value is specified by its row header in the first column. 


Small I/O 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80
Large I/O 587 667 595 428 361 338 340 340 336 341 342 351 339 335 346 350 351 353 352 351




Similarly, you can review all required .csv and other files to study the evaluation in details to get an idea how your oracle database will perform on new hardware.


Tuesday, August 30, 2016

SQL Tuning - 100% CPU Utilization






SQL Using 100% CPU
-----------------------------

After a weekend maintenance production activity(DB was bounced), EBS application started behaving abnormally, CPU utilisation was continuously being utilised 100%, whole production system was likely to be down for end users as they were not able to submit and process their requests.

I wondered around and get the AWR report for the problematic duration and found that TX – contention lock was there in the TOP 5 wait events and CBC latches as well. We cleared CBC latches but no help with that.

Investigated further and found a single SQL SELECT Statement (based on a view) was consuming TOP CPU order by CPU elapsed time and ordered by executions.


Same SQL was working fine usually but suddenly it started hogging whole Database server CPU. I found that Execution plan for that SQL statement was changed since DB server was bounced after maintenance activity.  



Please use below SQL to find the SQL Plan History of a particular SQL ID.
------------------------------------------------------------------------------------


set pagesize 1000
set linesize 200
column begin_interval_time format a20
column milliseconds_per_execution format 999999990.999
column rows_per_execution format 999999990.9
column buffer_gets_per_execution format 999999990.9
column disk_reads_per_execution format 999999990.9
break on begin_interval_time skip 1

SELECT
  to_char(s.begin_interval_time,'mm/dd hh24:mi')
    AS begin_interval_time,
  ss.plan_hash_value,
  ss.executions_delta,
  CASE
    WHEN ss.executions_delta > 0
    THEN ss.elapsed_time_delta/ss.executions_delta/1000
    ELSE ss.elapsed_time_delta
  END AS milliseconds_per_execution,
  CASE
    WHEN ss.executions_delta > 0
    THEN ss.rows_processed_delta/ss.executions_delta
    ELSE ss.rows_processed_delta
  END AS rows_per_execution,
  CASE
    WHEN ss.executions_delta > 0
    THEN ss.buffer_gets_delta/ss.executions_delta
    ELSE ss.buffer_gets_delta
  END AS buffer_gets_per_execution,
  CASE
    WHEN ss.executions_delta > 0
    THEN ss.disk_reads_delta/ss.executions_delta
    ELSE ss.disk_reads_delta
  END AS disk_reads_per_execution
FROM wrh$_sqlstat ss
INNER JOIN wrm$_snapshot s ON s.snap_id = ss.snap_id
WHERE ss.sql_id = '&sql_id'
AND ss.buffer_gets_delta > 0

ORDER BY s.snap_id, ss.plan_hash_value;





Below is the OLD SQL Plan which was working fine.
------------------------------------------------

BEGIN_INTERVAL_TIME  PLAN_HASH_VALUE EXECUTIONS_DELTA MILLISECONDS_PER_EXECUTION ROWS_PER_EXECUTION BUFFER_GETS_PER_EXECUTION DISK_READS_PER_EXECUTION
-------------------- --------------- ---------------- -------------------------- ------------------ ------------------------- ------------------------
08/14 05:15               3139549555                6                    758.744              345.0                   47354.0                      0.0
08/14 05:30               3139549555                6                    762.433              345.0                   47350.0                      0.0
08/14 05:45               3139549555               10                    763.121              345.0                   47350.0                      0.0
08/14 09:00               3139549555                4                   3081.986              348.0                  103319.3                   2141.3
08/14 10:30               3139549555                2                   1821.184              370.0                  126928.5                      2.5
08/14 11:15               3139549555                2                   1926.320              372.0                  135198.0                      0.0




Below is the BAD SQL Plan which optimiser chooses after maintenance activity. You can see dramatic increase in time per execution for the SQL Statement.
----------------------------------------------------------------------------------------

BEGIN_INTERVAL_TIME  PLAN_HASH_VALUE EXECUTIONS_DELTA MILLISECONDS_PER_EXECUTION ROWS_PER_EXECUTION BUFFER_GETS_PER_EXECUTION DISK_READS_PER_EXECUTION
-------------------- --------------- ---------------- -------------------------- ------------------ ------------------------- ------------------------
08/15 05:15               4140799271                1                 215288.221                1.0                13049295.0                   8088.0
08/15 21:00               4140799271                1                 292584.136                0.0                16131485.0                      8.0
08/15 21:15               4140799271                0              883156121.000                0.0                48242380.0                      0.0
08/15 21:30               4140799271                0              892885844.000                0.0                48965395.0                      0.0
08/15 21:45               4140799271                0              887075745.000                0.0                48552866.0                      0.0
08/15 22:01               4140799271                0              830391270.000                1.0                45468887.0                      0.0
08/15 22:15               4140799271                0              890962782.000                0.0                49497369.0                      0.0
08/15 22:30               4140799271                0              890992388.000                0.0                48430711.0                      0.0





Investigated further and found that base tables of underline view had STALE statistics due to which optimizer was not able to choose better execution plan for that SQL statement.

We gathered table statistics for those underline tables and immediately after that Oracle Optimizer chosen better execution plan and whole CPU utilization on database sever dragged down to normal.


BEGIN_INTERVAL_TIME  PLAN_HASH_VALUE EXECUTIONS_DELTA MILLISECONDS_PER_EXECUTION ROWS_PER_EXECUTION BUFFER_GETS_PER_EXECUTION DISK_READS_PER_EXECUTION
-------------------- --------------- ---------------- -------------------------- ------------------ ------------------------- ------------------------
08/17 10:31                576393603                8                   1476.348              366.0                   82579.3                      0.0
08/17 10:45                576393603               16                   1530.348              366.0                   86192.0                      0.0