The situiation was as following:
We have TEST/DEV server with a single Oracle 220.127.116.11 installation on it, installed as user "oracle"
Next to that there are several application users, called appt01, appt02, appt03 and appt04, all using this same ORACLE_HOME for their database.
All databases were running fine this morning.
We shutdown the appt01 to install a patch.
When trying to restart the database, suddenly this error appeared:
ORA-27123: Unable to attach a shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied
Using Google and MOS docs (115753.1 &167250.1), all information pointed to the incorrect file permissions of the "oracle" binary where the sticky bit has to be set. However this could be possible, because the patch installation rebuild this binary, it felt as not the correct reason. The file permssions were correct when looking at the file:
oracle@devrsv1% ls -l ./bin/oracle
-rwsr-s--x 1 oracle dba 70518002 Sep 29 12:09 ./bin/oracle
Investigating even more it was soon discovered that all other three database were able to stop and restart as expected. They showed no problems at all, even though they use the same ORACLE_HOME and settings.
After hours of searching and looking at the system, I finally turned to the OS.
Because the error message showed something about shared memory, I started investigating into that direction.
As user root, we executed:
# ipcs -mob
This statement usually shows all active shared memory segments, their sizes and the owners of the segments. In our case, only the output header appeared:
IPC status from /dev/mem as of Thu Sep 29 14:17:01 DFT 2011
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Now this showed that something was really wrong with the OS shared memory administration.
The other three databases were running, but also from these three no shared memory was shown, not even mentioning the standard OS shared memory pools.
We then decided to restart the complete server. And that solved the problem.
After the restart all databases started again as expected. Also the shared memory segments were correctly shown.
So even the error message displayed, usually points into the direction of the 'sticky bit' of the oracle binary, the problem can also be caused by a total shared memory problem of the system, caused by whatever reason.