Search this Blog

Wednesday, June 14, 2017

ORA-15036: disk '/dev/asmdata01' is truncated

Today I encountered one of these that you don't want to encounter.
We have this database server running Oracle 12c ( 12.1.0.2 ) on AIX 7.1

This server hosts 4 databases all around 400GB in size.
The databases are located on one ASM LUN connected through SAN storage of 1.5Tb

The need for yet another database on this server made us decide to increase the ASM disk to 2.1 TB and there we ran into problems. 
The increased diskspace on Storage level and OS level was not recognized by ASM. It kept showing a size of 1.5 TB.
We had to reboot the server anyway, becuase we also had to increase the memory size and from there on we were in trouble.
Starting ASM went fine, but the disk was not mountable, diplaying the error:

ORA-15036: disk '/dev/asmdata01' is truncated

The disksize on OS level showed it as the expected 2.1Tb, within ASM it showed as 0 Mb !!!



The 2Tb disk-limit was a known limit for databases up to Oracle 11.2. 
Since we are on Oracle 12.1 I did not expect to run into this problem.

The diskgroup however was created in the past with an 11.2 ASM and afterwards upgraded to 12.1
The diskgroup compatibility settings were also set to 12.1, but still the 2Tb limit seemed to be in effect.


The only solution we had was:
* Recreate the diskgroup and restore all database from backup, or
* Reduce the disksize (LUNsize) on the SAN and hope it gets reduced from the end of the disk where it just had been extended. 

We chose option 2 above option 1.
If option 2 failed, we were any convicted to option 1.

Luckily for us it worked!!
Reducing the LUN to 1.9 Tb, enabled ASM to recognize the disk and mount it.
All databases were OK afterwards, so no data was lost.

There was one step left in the end, whereas the disksize to ASM still was 1.5GTb

SQL> select total_mb from v$asm_diskgroup where name='DB01';

  TOTAL_MB
----------

   1572864

so we had to resize the disk and in the end all is OK now

SQL> alter diskgroup db01 resize all;

Diskgroup altered.

SQL> select total_mb from v$asm_diskgroup where name='DB01';

  TOTAL_MB
----------

   2046976


Wednesday, March 22, 2017

ORA-15067: command or option incompatible with diskgroup redundancy

At our company we had to replace the storage cabinets of a two node RAC cluster. The RAC cluster uses ASM diskgroups with NORMAL redundancy.
All diskgroups exist of two failgroups, with each failgroup located on one of the storage cabinets, for redundancy purposes, as seen here:

SQL> select   distinct a.name , b.failgroup, count(*) numdisks
  2  from     v$asm_diskgroup a
  3         , v$asm_disk b
  4  where  a.group_number = b.group_number
  5  order by name;

NAME         FAILGROUP             NUMDISKS
------------ --------------------- ----------
DB01         DB01_0000                     14
DB01         DB01_1000                     14
FB01         FB01_0000                      5
FB01         FB01_1000                      5

While all disks in FAILGROUPS DB01_0000 and FB01_0000 are on one storage cabinet the command to drop all disks on one 'side' of the ASM diskgroup at once would be the first option to choose:

However:

SQL> alter diskgroup fb01 drop disks in failgroup FB01_0000;
alter diskgroup fb01 drop disks in failgroup FB01_0000
*
ERROR at line 1:
ORA-15067: command or option incompatible with diskgroup redundancy


Since all data is mirrored across the two failure groups, one would expect to be able to drop one complete set of the mirror, so one complete failgroup.

All can be explained by the redundancy of the diskgroup. Using NORMAL redundancy, the diskgroup expects to have at least one disk available in the first failgroup that can be used as a destination for a failing disk in the other failgroup. This is also seen when querying the REQUIRED_MIRROR_FREE value of the diskgroup:

SQL> SELECT name, type, total_mb, free_mb, required_mirror_free_mb,usable_file_mb FROM V$ASM_DISKGROUP;

NAME                           TYPE     TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
------------------------------ ------ ---------- ---------- ----------------------- --------------
FB01                           NORMAL     512000     503598                   51200         226199
DB01                           NORMAL     819200     801612                  102400         349606

The REQUIRED_MIRROR_FREE values are equal to the size of a single disk in these diskgroups as seen here:

SQL> select distinct * from (
  2    select b.name, a.total_mb
  3    from v$asm_disk a
  4  , v$asm_diskgroup b
  5  where
  6  a.group_number=b.group_number);

NAME                             TOTAL_MB
------------------------------ ----------
DB01                               102400
FB01                                51200

So the NORMAL REDUNDANCY prevents us from dropping all disks at once in a failgroup.



Solution:

Drop all disks but one in the failgroup and then first add at least one new disk from the new storage cabinet
After that dropping the last disk from the old storage cabinet should be possible.




Friday, March 17, 2017

Thursday, February 9, 2017

ORA-12154 running dbca on a 12c RAC Cluster

When setting up a Oracle RAC cluster on 12.1.0.2 ( in our case patchlevel 170117 ), creating a database using dbca can run into an ORA-12154 unable to complete the database creation.


The setup of an Oracle RAC cluster is that you have first of all the Oracle Grid Infrastructure, the so-called clusterware. This software piece is the cluster, and it is responsible for everything else but the actual RAC database.
Everything else also includes the Listener (SCAN-listener and normal listener). For this matter the default location of an Oracle Installation is used for the listener config file, being:

$GRID_HOME/network/admin

 The RDBMS part of the installation is another software installation, with its own default location for SQL-net files, being:

$ORACLE_HOME/network/admin


Now when running 'dbca' and while creating a database, a tnsnames.ora file is being created in the $ORACLE_HOME/network/admin directory for self-reference of this newly created database.
As soon as the database is ready, it tries to register with the clusterware. At that moment the clusterware processes try to connect to the database and fail with an

ORA-12154 TNS Could not resolve service name


The reason for this is, that the clusterware home did not receive the tnsnames.ora information about the created database, resulting in an unresolvable service name.


Solution:

Before starting dbca, add support for the self-reference of the to-be created database into the $GRID_HOME/network/admin location, by creating an empty tnsnames.ora for the rdbms-user and a symbolic link to this file for the grid-user.


dbuser $ cd $ORACLE_HOME/network/admin
dbuser $ echo “NAMES.DIRECTORY_PATH=(TNSNAMES)” > sqlnet.ora
dbuser $ touch tnsnames.ora
dbuser $ chmod g+w . tnsnames.ora

Next:

griduser $ cd $GRID_HOME/network/admin
griduser $ ln –s /opt/oraadmin/network/admin/tnsnames.ora tnsnames.ora
griduser $ ln –s /opt/oraadmin/network/admin/ sqlnet.ora sqlnet.ora

In the above statements also a sqlnet.ora file is being created. This is (in our case) to prevent the grid-user from connecting to the still existing older version of this database by means of an EZCONNECT connection.

Now when dbca creates an entry in the tnsnames.ora file the grid-user is also aware of its existence and the ORA-12154 does not appear