SSH equivalence on RAC

A couple of months ago, we setup our first RAC cluster consisting of two LPAR on an IBM AIX Power6 server. Although it has been months ago, and the cluster has been running a production database now successfully for over a month, I do want to share the 'frustration's we've had with setting up the user-equivalence, needed for the installation.

For the newbie's among us, first of all a short explanation of the phrase "SSH equivalence":

SSH equivalence means that acrosse different servers a user, and especially its credentials and rights, are considered equal to the local server.

With SSH equivalence configured a logged on user can access a remote server without password if and only if his account with the same settings also exist on that remote server.

SSH equivalence is needed during the installation of the RAC software. All parts of the Oracle installation ( CRS, ASM and the Rdbms ) are each installed on the local host and afterwards copied to all remote nodes using this SSH equivalence.

First of all a short description of how to setup SSH equivalence. Of course the Oracle installation manual has a more detailed explanation to follow:

Generate a keypair on each node:

[crsprd@node1 ~]$ cd

[crsprd@node1 ~]$ mkdir .ssh

[crsprd@node1 ~]$ chmod 700 ~/.ssh

[crsprd@node1 ~]$ /usr/bin/ssh-keygen -t rsa

Create a authorized_keys file from the public keyfile:

[crsprd@node1 ~]$ cd .ssh

[crsprd@node1 ~]$ cat id_rsa.pub >> authorized_keys

[crsprd@node1 ~]$ ls

The result will be 3 files.

One of them is the file "authorized_keys"

Copy the authorized_keys file to the other node(s):

First make sure the .ssh directory exists in the home directory of the user on each node.

[crsprd@node1 ~]$ scp authorized_keys node2:/opt/crsprd/ora/home/.ssh/

On the question to accept the RSA keys, enter Yes:

It will add the remote-node to the local file called 'known_hosts'.

If this file does not exist it is created automatically.

[crsprd@node1 ~]$ scp authorized_keys node2:/opt/crsprd/ora/home/.ssh/

The authenticity of host 'node2 (xxx.xxx.0.181) can't be established.

RSA key fingerprint is 7e:60:60:ae:40:40:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'node1,xxx.xxx.0.181' (RSA) to the list of known hosts

authorized_keys 100% 828 7.5MB/s 00:00

Log on to the other node(s) as the same user en enter the .ssh directory.

[crsprd@node1 ~]$ ssh node2

The authenticity of host node2 (xxx.xxx.0.181) can’t be established.

RSA key fingerprint is z3:z3:33:z3:z3:33:zz:76:z3:z3:z3.

Are you sure you want to continue connecting? (yes/no)? yes

Add the local public key to the authorized keys file:

[crsprd@node2 ~]$ cd .ssh

[crsprd@node2 ~]$ cat id_rsa.pub >> authorized_keys

Repeat this action for every node in the the cluster, until you've added the last node.

Then start copying back this file to every node ( In this example there are only two nodes)

The result must/will be that on every host the authorized_keys file is identical

[crsprd@node2 ~]$ scp authorized_keys node1:/opt/crsprd/ora/home/.ssh/

Now you should have the SSH equivalence setup.

We followed these steps from the Oracle installation manual and then the problems started to begin:

The first problem we encountered, was that we were forced to enter the passphrase for the SSH connection every time, with every command.

This resulted in not being able to install any of the software applications.

Only with assistance of an onsite Oracle consultant, and only after he saw me configuring the SSH equivalence the problem was solved:

Don't enter a passphrase!!

Just press <Enter> when prompted.

Now that was a great thing to discover. We were trying to secure the SSH connection by means of this passphrase where Oracle always uses an empty passphrase ( i.e. according to this consultant).

As soon we had discovered this issue, we were able to start the installation of the CRS and it succeeded completely.

Then however problem two appeared. We were using seperate accounts for CRS, ASM and the Rdbms. In the oracle documentation we used, I never found a statement that when using multiple users all these users should have SSH equivalence seperately.

Okay, okay... afterwards I must say it is quite logical that all users should have this ability. But until we found this out, I was always assuming the CRS ( and therefore the crs owner ) would be responsible for the distibution of the installation files across the nodes.