Today there was a higher volume of activity in PASOE than normal and we started seeing some really strange errors in the agent log .
Here is an example..
[19/01/29@16:58:01.613-0500] P-007020 T-008596 1 AS-404 -- (Procedure: 'EntryProcedures\Server\StartupStateReset.p' Line:0) ** stget: out of storage. (6498) [19/01/29@16:58:01.614-0500] P-007020 T-008596 1 AS-404 MSAS Destroying Session AS-404 (crash). [19/01/29@16:58:01.727-0500] P-007020 T-008596 1 AS-Aux-0 MSAS Worker Thread exiting. Number: 6, Status: 10002 [19/01/29@16:58:01.729-0500] P-007020 T-005988 1 AS-ResourceMgr MSAS Spawning New Worker Thread. Number: 8 [19/01/29@16:58:06.639-0500] P-008356 T-009680 1 AS-Aux-0 MSAS Agent Starting Up -- Progress OpenEdge Release 11.7 build 1685. [19/01/29@16:58:06.640-0500] P-008356 T-009680 1 AS-Aux-0 -- Logging level set to = 1 [19/01/29@16:58:06.652-0500] P-008356 T-009680 1 AS-Listener MSAS Spawning New Worker Thread. Number: 4 [19/01/29@16:58:07.058-0500] P-008356 T-005232 1 AS-ResourceMgr MSAS Spawning New Worker Thread. Number: 5
Notice the message, "stget: out of storage" and the crash. Based on my reading, this appears to indicate that the PASOE agent, or perhaps the server as a whole, had run out of memory.
There is other evidence that there were memory problems. The same MS-agent was generating these messages too at the same time (and we NEVER see errors about SAX parsing):
And other processes on the machine were also unhappy. Another small, unrelated process logged this to the windows event log around the same moment of time:
But moments later, when I logged into the desktop to investigate, there were no problems at all and the server seemed totally peaceful.
I'm guessing that PASOE had some type of spasm and tried to momentarily consume all available memory on the entire system. Is there any other logging I should be looking at other than the MS-agent log and the Windows event log? Is there a way for me to gather better information if/when this reoccurs?
We are running OE 11.7.4 on Windows Server.
the message "stget out of storage" comes when the 4gl runtime is allocating
dynamic memory and the allocation fails. there are several reasons why this
might happen:
0) there is no more available address space - you are attempting to exceed
the per process limit.
1) the 4gl are attempting to allocate a large chunk of memory for something
and the address space is so fragmented that the largest contiguous chunk is
smaller than the requested amount.
2) the application is allocating memory for various things and does not delete
the items that are no longer needed.
3) you have too many 4gl sessions in one app server agent.
4) other nstuff that i cant think of at the moment.
the message "stget out of storage" comes when the 4gl runtime is allocating
dynamic memory and the allocation fails. there are several reasons why this
might happen:
0) there is no more available address space - you are attempting to exceed
the per process limit.
1) the 4gl are attempting to allocate a large chunk of memory for something
and the address space is so fragmented that the largest contiguous chunk is
smaller than the requested amount.
2) the application is allocating memory for various things and does not delete
the items that are no longer needed.
3) you have too many 4gl sessions in one app server agent.
4) other nstuff that i cant think of at the moment.
Yes, it sounds like the Sax Parser must had tried to allocate a massive amount of memory for an XML object, causing the session to crash. The good news is that it appears the server handled the agent crash and recovered. Can you see if there are any protrace files associated with the crash? If you didn't install the PDB files,they may not be that useful, but I wouldn't mind taking a look.
I'm pretty convinced that I ran out of memory because of a flurry of PASOE activity. It was probably a self-inflicted problem that occurred because of a client program that was making an unreasonable number of concurrent connections (possibly over 100).
But I hadn't ever seen the error message before, and I didn't know if the scope of the memory leak was local to a session, an ms-agent process, or the entire server. I suspect it was the entire server. But as soon as I was able to start doing some investigation, the problem had already cleared itself out.
I have a pretty good idea about which client program was responsible, and if changing the behavior of that program doesn't work then I will enable a higher level of logging in ASPlumbing, and start monitoring the Windows memory perf counters. Thanks, David