ORA-27123: Unable to attach a shared memory segment

This morning we had another brainteaser starting one of our databases.

The situiation was as following:

We have TEST/DEV server with a single Oracle 9.2.0.8 installation on it, installed as user "oracle"

Next to that there are several application users, called appt01, appt02, appt03 and appt04, all using this same ORACLE_HOME for their database.

All databases were running fine this morning. 

We shutdown the appt01 to install a patch. 

When trying to restart the database, suddenly this error appeared:

SQL> startup 
ORA-27123: Unable to attach a shared memory segment 
IBM AIX RISC System/6000 Error: 13: Permission denied


Using Google and MOS docs (115753.1 &167250.1), all information pointed to the incorrect file permissions of the "oracle" binary where the sticky bit has to be set. However this could be possible, because the patch installation rebuild this binary, it felt as not the correct reason. The file permssions were correct when looking at the file:

oracle@devrsv1% cd$ORACLE_HOME
oracle@devrsv1% ls -l ./bin/oracle
-rwsr-s--x    1 oracle   dba     70518002 Sep 29 12:09 ./bin/oracle

Investigating even more it was soon discovered that all other three database were able to stop and restart as expected. They showed no problems at all, even though they use the same ORACLE_HOME and settings.

After hours of searching and looking at the system, I finally turned to the OS.
Because the error message showed something about shared memory, I started investigating into that direction.
As user root, we executed:

# ipcs -mob

This statement usually shows all active shared memory segments, their sizes and the owners of the segments. In our case, only the output header appeared:


IPC status from /dev/mem as of Thu Sep 29 14:17:01 DFT 2011
T        ID     KEY        MODE       OWNER    GROUP NATTCH     SEGSZ

Now this showed that something was really wrong with the OS shared memory administration.
The other three databases were running, but also from these three no shared memory was shown, not even mentioning the standard OS shared memory pools.

We then decided to restart the complete server. And that solved the problem.
After the restart all databases started again as expected. Also the shared memory segments were correctly shown.
So even the error message displayed, usually points into the direction of the 'sticky bit' of the oracle binary, the problem can also be caused by a total shared memory problem of the system, caused by whatever reason.