ORA-27154: post/wait create failed

Today, we were doing Exadata X7-2 QFSDP APR 2019 patching, but during post patch steps, some of the cluster database instances failed to start with following error  - Over-all 62 databases running on this Exadata Machine.

Note : Recently we added 2 more databases to this Exadata box.

SQL> startup
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates
SQL> 

From one of the database alert log file:

Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 13833 (27 GB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total System Global Area size is 2050 MB. For optimal performance,
  prior to the next instance restart:
  1. Increase the number of unused large pages by
 at least 1025 (page size 2048 KB, total size 2050 MB) system wide to
  get 100% of the System Global Area allocated with large pages
********************************************************************
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_184111.trc:
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates

From the trace file:

*** 2019-08-17 10:10:25.566
Switching to regular size pages for segment size 100663296
Switching to regular size pages for segment size 2046820352
Switching to regular size pages for segment size 2097152

*** 2019-08-17 10:10:25.595
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=0, mask=0x0)
----- Error Stack Dump -----
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates

Semaphore current configuration in the system.

[root@exa01dbadm01 ~]# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 256
max semaphores per array = 1024
max semaphores system wide = 60000
max ops per semop call = 1024
semaphore max value = 32767







Means 60000 semaphores are available in the system and one semaphore identifier can accommodate a maximum of 1024 semaphores with the configuration above.

But ipcs command shows each semaphore identifier accommodates maximum of 514 (some 752) semaphores by Oracle. 

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x27219008 5242886    oracle     640        704
0x1d63fabc 5373959    oracle     640        514
0x1d63fabd 5406728    oracle     640        514
0x26f035f8 5603337    oracle     640        514
0x1d63fabe 5472266    oracle     640        514
0x26f035f9 5701643    oracle     640        514
0xeb773830 5898252    oracle     640        514
0x26f035fa 5832717    oracle     640        514
0x14422b02 11862119   oracle     640        514
0xa83d39fc 11993192   oracle     640        514
0xa83d39fd 12025961   oracle     640        514
0xa83d39fe 12058730   oracle     640        514
0xea1a4768 12189803   oracle     640        514
0xea1a4769 12222572   oracle     640        514
0xea1a476a 12255341   oracle     640        514
0xdcc4a7a4 12386414   oracle     640        514
0xdcc4a7a5 12419183   oracle     640        514
0xdcc4a7a6 12451952   oracle     640        514
0x75550980 12583025   oracle     600        514
0x75550981 12615794   oracle     600        514
0x75550982 12648563   oracle     600        514

So, maximum semaphores available on this system would be 514x256 = 131584

[root@exa01dbadm01 ~]# /sbin/sysctl -a | grep sem
kernel.sem = 1024       60000   1024    256

Per above, we see that we have less, max semaphores configured in the system i.e. 60000.
==================

SEMMNI should be increased to accomodate more semaphores.

[root@exa01dbadm01 ~]# /sbin/sysctl -a | grep sem
kernel.sem = 1024       60000   1024    256

I increased the max number of semaphores to be available on the system to 70000 from 60000

From :

[root@exa01dbadm01 ~]# /sbin/sysctl -a | grep sem
kernel.sem = 1024       60000   1024    256

To this:

[root@exa01dbadm01 ~]# /sbin/sysctl -a | grep sem
kernel.sem = 1024       70000   1024    256

And finally run the below to make new kernel values effective without server reboot.

/sbin/sysctl -p

Finally, tried to start the remaining DB instances and all came up normally.

[oracle@exa01dbadm01 ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 12.1.0.2.0 Production on Sat Aug 17 10:32:13 2019

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  2926472 bytes
Variable Size            1318045816 bytes
Database Buffers          822083584 bytes
Redo Buffers                4427776 bytes
Database mounted.
Database opened.
SQL>

Hope it helps, subscribe to this blog to stay updated on latest Oracle Technologies and new articles.

Twitter : @rajsoft8899
Linkedin : https://www.linkedin.com/in/raj-kumar-kushwaha-5a289219/