Data area doesn't grow bigger than ~160 GB

Posted by Kai Lahtinen on 18-Aug-2017 00:36

Hi!

One of our customers have a a following problem. One big table "printouts" is placed alone on a type II area. The area is in one variable size file, that currently  is almost 160 GB. The block size is 8k and the server is Windows server. If they try add records to the area, the database crashes with errors:

[2017/07/04@00:47:10.667+0300] P-162768     T-162772 I ABL   135: (9450)  bkioWrite:Insufficient disk space during write, fd 2864, len 16384, offset 19902184, file E:\db\datas\database_11.d1

[2017/07/04@00:47:10.698+0300] P-162768     T-162772 F ABL   135: (8897)  SYSTEM ERROR: Unable to extend database within area printouts.

[17/07/04@00:47:10.698+0300] P-162768 T-162772 1 4GL -- (Procedure: 'programX.p' Line:84) SYSTEM ERROR: Unable to extend database within area printouts. (8897)

Area settings in st file: "printouts":11,64;512 E:\db\datas\database_11.d1

   Statistics for Area: printouts

   Files in Area: printouts

  E:\db\datas\database_11.d1  163038707712
  Database Block Usage for Area: printouts
  Active blocks: 19901951
  Data blocks: 19901446
  Free blocks: 505
  Empty blocks: 235
  Total blocks: 19902186
  Extent blocks: 1
  Records/Block: 64

A couple of months old tabanalys for the area (when it worked):
AREA "printouts" : 11  BLOCK ANALYSIS
-------------------------------------------------

19470847 block(s) found in the area.

Current high water mark: 19470847

    505 free block(s) found in the area
    19470332 record block(s) found in the area
    0 index block(s) found in the area
    0 empty block(s) found in the area
    2 object block(s) found in the area
    2 cluster list block(s) found in the area
    1 object list block(s) found in the area
    2 cluster map block(s) found in the area

RECORD BLOCK SUMMARY FOR AREA "printouts" : 11
-------------------------------------------------------
                                       -Record Size (B)- ---Fragments--- Scatter
Table                Records    Size   Min   Max  Mean      Count Factor  Factor
PUB.printouts      912754005  123.4G    68 12382   145  917107791    1.0     1.0


We have studied this with co-workers and couldn't find solution. The customer has moved some old records from the area to an other place so we have managed so far, but we should solve this soon.

Any ideas what might be the problem?

Thanks

Kai

All Replies

Posted by George Potemkin on 18-Aug-2017 01:19

Did you post the area setting reported by a recent prostrct list? Maybe someone tried to add a new extent and then immediately removed it but left the first extent with the fixed size?

Does E disk have the free space?

Did you check the possible corruptions on file system level? Can you copy/delete/restore E:\db\datas\database_11.d1 ?

Posted by Kai Lahtinen on 18-Aug-2017 01:36

Hi!

I ran a new prostrct list and the result was the same: just one variable size data file for the area. E-disk has free space about 110 GB and no other database areas have faced this kind of problem.

The company resposible for the server has checked disks and found no problems. Disk system is a mirrored Hitachi Data Systems SAN.

As the database is in very important use 24/7 there's very little I can do online, but I try to find a place to restore last backup and do some testing with that.

Regards

Kai

Posted by George Potemkin on 18-Aug-2017 01:47

Can you add (online) new extent to the area? It will work either as a workaround or as a debugging of the issue.

Posted by cjbrandt on 18-Aug-2017 08:12

What is the version of Windows ?  Try to add another variable length extent to the area and allow OE and Windows to start using the new extent.

I couldn't tell from your previous post if other databases already had single extents larger than 160gb - " no other database areas have faced this kind of problem."

Limit might be with something besides OE.

If there is space available, try to create a test database with a single fixed extent of 175gb and see if it can be created.

If that works, take a binary dump of a table and load it back into the 175gb extent, repeat the load until you fill the extent and see if there are similar errors around 160gb.

Posted by Kai Lahtinen on 21-Aug-2017 06:33

Hi! Looks like addonline command solved the issue. The new variable size file has started growing and no error messages or crashes have occurred after the file addition.

Thanks

Kai

Posted by George Potemkin on 21-Aug-2017 06:48

So it's more likely that the root case was a problem with disks and this issue is not solved.

Posted by George Potemkin on 21-Aug-2017 13:30

What is the size of database_11.d1 after it was fixed by prostrct?

Is it larger than the offset reported by the error 9450?

If you have a short timeframe to test db then I'd recommend to update a few records with RECIDs at the end of database_11.d1.

For example:

for table where recid(table) eq myRecid:
  assign table.name = "x" + table.name.
  assign table.name = substring(table.name, 2).
end.

Db might crash with the same error 9450 if there is a problem with disk writes.

This thread is closed