Question Regarding New Index Rebuild Parameters

Hi Paul,

What was the ulimit -m for the process running the idxbuild?

Thanks

Marek

Hi Marek,

I was root.  All my ulimits were unlimited.

in this case one might expect that the amount of memory you have would be sufficient. but there is more than just the 2.6 gb of compressed index entries that has to be managed by the index rebuild code. Some of the data structures that are used are fairly large. though many improvements have been made, more are possible. focus has mainly been on speed so far.

Paul,

If I'm reading it right, the process can allocate max of 4GB of memory and 47Mb is already in use:

Process memory 49283072 of 4294967296 in use

TF memory is determined roughly based on the formula:

MIN((physicalMemory - InuseMemory), ((physicalMemory * -TF) / 100))

I suppose that physicalMemory is more about max memory available to the currently running process and not physical memory in the system (10.2B05+).

It looks like 90% of (4294967296 - 49283072) is about 3821115802 (3.6GB).

It appears to be around the same amount as taken for the merge pool:

Max merge pool entries estimated at: 3644 w/memory usage: 3821010944

Any overflow space needed (for 15GB table) will go to disk...

Is your OE11.2 a 32bit release?

Note that I'm a bit speculating here based on your log entries and what I remember. I'd have to run some refreshing tests to verify what I've said above.

Marek

Gus: I admit to not having attempted to calculate memory in use by _proutil.  I suppose it is possible that it was using considerably more memory though I don't see how it could have consumed 30-40 Gb.

Marek: It is OE 11.2.1 64 bit.  I was able to start the DB with a -B of 20 Gb RAM (not blocks) so I don't see an operating system limitation.  I don't understand where/how it calculated process max memory as 4 Gb.

Right, this is interesting. We determine that max is exactly 4GB (4294967296), which is the 32bit max.

Anyway, maybe it's limited by maxdsiz_64bit:

maxdsiz* = data segment size limit

... if I remember correctly 4GB is the default value on 64bit HP-UX

Marek has the key here:

When IDXBUILD ran, there was only 4GB available to the process and 47MB of that was already in use, not the 48GB you were expecting.

Process memory 49,283,072 of 4,294,967,296 in use

So the -TF memory part was : 3,865,470,566

being the lesser of: 'MIN((physicalMemory - InuseMemory), ((physicalMemory * -TF) / 100))

4,245,684,224 3,865,470,566

Added to the -TF (3,865,470,566) was the datascan overhead ~ 4MB and memory for all sort groups ~ 528MB,

which is roughly 122MB than the Process memory available.

Isn't Progress Maths fun ;)

Hmm...so the question becomes: why did _proutil think there was only 4 Gb available to the process?  It's kinda starting to sound like a bug.

What are your maxdsiz* values?

Hi Paul,

I still believe a process is restricted to maxdsiz_64bit in your case. The default value is 4GB.

Could you increase it (let's say to 8GB) and let us know if it changes anything?

If this (value of maxdsiz) is a legitimate design restriction and not a bug then it should be documented so customers can be aware of it and plan accordingly.

Based on hits in the KB this appears to be specific to HP-UX; is that the case?

This is a production box so it will be difficult to try. I will check the value and get back to you asap.

Hi Rob,

Yes, this one is specific to HP-UX. There are different limits for different memory types on different operating systems and their different releases. These are rather OS "design restrictions" and not directly related to Progress. Every system administrator of a specific OS is/should be well aware of such limitations. So, I'm not sure where else it should be documented except KB.

In general, maxdisz* is about data segment size, like LDR_CNTRL on AIX that regulates how much space is designated for data user stack, kernel stack, u-block, etc...anything allocated by malloc() (at the expense of shared memory).

Thanks

Marek

Hi Marek,

Fair enough, sysadmins and DBAs should have some level of OS knowledge.  But there is a difference between knowing one the one hand that a kernel parameter exists and what it is used for, and on the other hand how OE uses or is constrained by its value.

Given the relatively small number of fundamentally different architectures that the RDBMS runs in it would be helpful to have a "cheat sheet" of practical limits that we could refer to.  Even a brief KB article would be better than nothing.  

These limits may be self-evident to systems programmers, but most DBAs don't have that background.  I can see Paul's point that it is reasonable to assume that 64-bit idxbuild on a 64-bit box with lots of RAM should be able to allocate a lot more than 4 GB of memory for sort buffers.  

Marek: you are correct.  maxdsiz_64bit is 4Gb.

I don't expect I will have the opportunity to raise it and run another index rebuild since this is the prod box.  Maybe I can try on a test box if it has enough memory.

I guess this falls under an enhancement request: if _mprosrv can get multiple 4 Gb segments, then _proutil should be able to do the same.