Search this Blog

Wednesday, June 14, 2017

ORA-15036: disk '/dev/asmdata01' is truncated

Today I encountered one of these that you don't want to encounter.
We have this database server running Oracle 12c ( 12.1.0.2 ) on AIX 7.1

This server hosts 4 databases all around 400GB in size.
The databases are located on one ASM LUN connected through SAN storage of 1.5Tb

The need for yet another database on this server made us decide to increase the ASM disk to 2.1 TB and there we ran into problems. 
The increased diskspace on Storage level and OS level was not recognized by ASM. It kept showing a size of 1.5 TB.
We had to reboot the server anyway, becuase we also had to increase the memory size and from there on we were in trouble.
Starting ASM went fine, but the disk was not mountable, diplaying the error:

ORA-15036: disk '/dev/asmdata01' is truncated

The disksize on OS level showed it as the expected 2.1Tb, within ASM it showed as 0 Mb !!!



The 2Tb disk-limit was a known limit for databases up to Oracle 11.2. 
Since we are on Oracle 12.1 I did not expect to run into this problem.

The diskgroup however was created in the past with an 11.2 ASM and afterwards upgraded to 12.1
The diskgroup compatibility settings were also set to 12.1, but still the 2Tb limit seemed to be in effect.


The only solution we had was:
* Recreate the diskgroup and restore all database from backup, or
* Reduce the disksize (LUNsize) on the SAN and hope it gets reduced from the end of the disk where it just had been extended. 

We chose option 2 above option 1.
If option 2 failed, we were any convicted to option 1.

Luckily for us it worked!!
Reducing the LUN to 1.9 Tb, enabled ASM to recognize the disk and mount it.
All databases were OK afterwards, so no data was lost.

There was one step left in the end, whereas the disksize to ASM still was 1.5GTb

SQL> select total_mb from v$asm_diskgroup where name='DB01';

  TOTAL_MB
----------

   1572864

so we had to resize the disk and in the end all is OK now

SQL> alter diskgroup db01 resize all;

Diskgroup altered.

SQL> select total_mb from v$asm_diskgroup where name='DB01';

  TOTAL_MB
----------

   2046976