Memory problems in PASOE

Posted by tbergman on 02-Apr-2019 13:23

We have been experiencing memory problems in PASOE. Windows, 11.7.2. 

The usual symptom is failure to handle LONGCHARS properly and other memory related errors in the logs. A restart of the broker solves all issues, for a while.

I have in place a way of looking at all the Progress dynamic objects. This uses code similar to that described in the kbase here https://knowledgebase.progress.com/articles/Article/P124514.

I can run this at any time against any AppServer and see what's left behind. This has not yielded any clues or culprits.

We do use some .Net code on the AppServers and I'm aware that the technique above will not help in identifying problems possibly related to .Net GC or other possible .Net issues.

Does anyone have or know of a tool and/or technique to help with this in Progress? 

Thanks,

Tom

All Replies

Posted by frank.meulblok on 02-Apr-2019 14:15

Checking dynamic objects from the session handle is flawed, simply because it doesn't provide access to all types of object. (It *really* should have a FIRST-HANDLE from which you can get all handle-based objects, but even with that I don't think it catches memptrs/longchars.)

Also, I've not tried that approach on a PASOE, but I suspect it'll only report things for the one specific session it's called from - which may not be the session that's causing issues.

I've had more success with the logging-based variant myself. That one's described in knowledgebase.progress.com/.../P133306

Posted by dbeavon on 02-Apr-2019 16:13

You are talking about a leak in _mproapsv?  Can you quantify the rate at which you see a leak?  (1 GB a day or more?)  Almost all apps leak memory, its just a question of how fast.  We restart our PASOE instances once a week.  We leak about 5 GB a week, which is the total out of several ms-agent processes.

I have an open case about a memory leak in PASOE on Windows.  It didn't apply to Linux (apparently).  It may be fixed soon but will be on 11.7.5.  Hopefully there will be a KB once it is fixed.

Depending on the speed that you leak, you may consider different approaches.  If it is a very fast leak then I would definitely continue to read all those KB articles where Progress suggests some ways to track down dynamic objects and handles.  There were a few different KB's for that.  

But if it is a slow leak and a lot of legacy code is involved, then you may have diminishing returns on your troubleshooting efforts.  In my opinion life is too short to be helping ABL to manage its internal memory usage.  Another possibility is to just trim the sessions out of the ms-agent on a periodic basis (every hour or half hour).  This is very safe, and will only trim sessions that aren't actively being used.  The upshot is that any memory that was allocated within those sessions is NOT your problem anymore.  Once the ABL session doesn't exist anymore, it is the responsibility of Progress to release the related memory.  Some of it may not be returned to the OS, but at least it can be reused by future sessions.

Trimming sessions can be accomplished via the OE manager rest interface.  (I think you can also use JMX).  Here is a powershell script that shows how to trim inactive sessions from an ms-agent process:

community.progress.com/.../36461

(Posted by me on 26 Dec 2017 14:04, with lots of help from Irfan)

As an added benefit, when new PASOE sessions replace the old ones, they will also pick up any r-code changes as well. (This happens even if you are using the -q parameter on the agent).

Note that the strategy of trimming sessions has its own slow leak (which I mentioned before).  But that is a relatively small problem, and it is a problem which the smart people at Progress can fix on their side of things.  That is a leak which doesn't require us to review every line of legacy code that makes use of HANDLE, MEMPTR, and so on.

As far as the .Net side of things goes, you should be able to see a leak there fairly easy by taking memory dumps and using VS to debug managed memory.  Another tool I highly recommend is sysinternals "process explorer".  If you run it as admin, you will have a ".Net assemblies" tab and a ".Net performance" tab and they will tell you most of the stuff you need to know.  Eg. you will see appdomains, custom assemblies, the managed heap size, etc.

This thread is closed