OpenEdge 11.6 | How to find a memory leak on AppServer

Posted by atuldalvi123 on 12-Apr-2019 13:57

Hello All,

We're investigating a memory leak in our application.

Is there any utility or any setting which will report the memory leaks in the AppServer source code.

Any tips on how to tackle this kind of issues is appreciated!


Atul Dalvi

All Replies

Posted by Ivan Atanasov on 12-Apr-2019 15:37

You've probably done a quick search before but that's how Progress suggest finding memory leaks

Posted by Peter Judge on 12-Apr-2019 15:44

There are a few approaches you can take. These apply to all AVM/clients
  1. Use the DynObjects.* logging type to identify when OO and other objects are created and destroyed. Parse the logs generated to see what isn’t appropriately destroyed. There are knowledgebases and Communities posts on how to do this
  2. Do the same for Temp-Tables ; these entries also help you identify when you are unnecessarily passing temp-table data by-value (instead of by-reference)
  3. For AppServers, identify the start and end of a request (in activate and deactivate event procedures) the objects, persistent procedure and other stuff that’s in memory. Look at what’s increased in the deactivate, and decide whether that’s a leak. Use the SESSION handle’s FIRST-OBJECT, FIRST-PROCEDURE, FIRST-BUFFER, FIRST-TABLE and any other FIRST-* reference, and “walk the tree” by checking that reference’s NEXT-SIBLING until there are no more valid-handle/valid-objects.
Notice that you will need to decide whether additional/remaining references at the end of a request are leaks or not – in some cases there will be objects cached deliberately, which should not be considered leaks. There are some conversations on Communities in this vein.
For (3) above, you will have to add ABL code to gather the information; in newer versions of PASOE there are mechanisms to get some of this data without having to write ABL, though the analysis about leak-or-not will be the same.
- peter

Posted by dbeavon on 12-Apr-2019 16:49

Classic appserver or PASOE?  How fast is the leak?  Can you quantify it?  Almost all apps leak to some extent, and what is more interesting is how fast (1 GB an hour/day/week).  A "leak" can also be defined in different ways.  (Eg. Peter gave the example of objects that are cached deliberately, which may appear like a leak until closer inspection.  Another example is the use of classes that have static member data.  Or even the runtime itself may "leak" if it accumulates a lot of r-code in memory, or chooses not to instantly release memory allocations back to the OS in favor of keeping them for some future reuse.)

If there is in fact a measurable leak, then PASOE (if that is what we are talking about) will amplify if.  This is  because there are lots of ABL sessions that are hosted in a single process ( _mproapsv).

This question just came up a week ago, you may want to look at this thread too:

Posted by Patrick Tingen on 12-Apr-2019 18:05

What [mention:9e4ee96fac634b8f91b580e1fb4f7e71:e9ed411860ed4f2ba0265705b8793d05] suggested was what we did at a previous customer: we had a special program to testrun a procedure. Before the start of the program we created a fingerprint of all persistent procedures and tables currently active in your session. Then we ran the program, did an UNDO and made a second list of persistent stuff. We removed everything from the list that was already there (tip: store handle in a tt) and what remains is the stuff your program created but did not clean up.

Posted by jbijker on 15-Apr-2019 09:01

Just a caveat when you do use the NEXT-SIBLING method: In some cases you might get a handle / object back, but when you do a VALID-HANDLE or VALID-OBJECT on it it will return FALSE. It typically happens when you've created something inside a widget-pool and destroy the widget-pool later, but nowhere there is an explicit DELETE on that handle / object. Personally I think this is a bug because that should've be cleaned up as part of the widget pool, but for now you need to ignore these invalid handles / object and do another NEXT-SIBLING (yes you can do that, even though they don't point to a valid handle / object). If you don't you'll simply stop scanning the rest of the chain.

Posted by slegian-bva on 18-Apr-2019 00:56

That article and its associated script came in very handy for us just this Monday as we were chasing a ** stget out of storage error. Added the DynObjects.* loging and run the output through the script only to discouver some dynamic queries being created and never deleted... CREATE WIDGET-POOL is my new friend.

Posted by atuldalvi123 on 02-May-2019 12:39

Hello Peter,

Thanks for the reply. Actually, we have a PAS server specially running for the OpenEdge soap web services only. We have couple of OpenEdge soap services deployed on this PAS server and hundreds of programs running and linked under it. There are no OpenEdge classes involved in it only .p programs with Dataset handles, input-output Datasets, temp-tables, Dynamic Queries, buffers etc. I have below doubts on the options you suggested -

1. Will the Dynamic Objects Logging work in this case ?

2. Can you give some more details on the option (2) to identify temp-table entries. Any other logging type for temp tables ?

3. Also how can we implement option(3). I have code for this but Its a session or request specific right ? Can't we implement this at very higher level as a common code for all the web services ? otherwise I have to modify main program of each web service to write this code.

Thanks in advance.

Posted by Peter Judge on 02-May-2019 13:35

DynObjects tracks the following

The temp-table logging is described .

These 2 should give you good insight .

In PASOE you can always add activate and deactivate procedures, and those will run on every request (for SOAP, WEB, etc). That would be the place to add the checks IMO.

Posted by atuldalvi123 on 02-May-2019 13:42

Thanks Peter.

Posted by Ken McIntosh on 02-May-2019 17:05

Please note that if you can run the same application in 11.7.4 or later you can use the PASOE REST API to monitor for leaks.  The below article talks about using the ABL http client to call out to the REST API and includes references to code that will allow you to turn the getABLObjects functionality on/off, and another piece of code that can be used to fetch a list of leaked objects.

000095195 - How to call into the OEManager's REST API for insight into PASOE?

The benefit of using this is that you don't need to change your application code or use parameters to generate a multi-million line log file that you have to then parse in order to compare the creates with the deletes to find the list of leaked objects.  This utility is simply turned on/off as desired and then it does the analysis and simply returns where the leaks are in your code.

The only pain point I can think of is that you DO have to have the oemanager deployed to your pasoe instance.

Posted by atuldalvi123 on 15-May-2019 09:16

Hello All,

I tried DynObjects but in log report I did not find anything related to the dynamic objects as I don't have object programming in my code only input output temp tables and datasets and temp tables & datasets are simple not with handle object.

What else I can check to find out this memory leak on AppSeerver ?

Thanks in advance !!!

Posted by gus bjorklund on 15-May-2019 15:24

> On May 15, 2019, at 5:18 AM, atuldalvi123 wrote:


> only input output temp tables and datasets and temp tables & datasets are simple not with handle object

based on what you have said, it is likely that you are creating tem-tables and dataasets that are not being deleted when you don't need them anymore.

if you start client with

-clientlog foo -logentrytype "temp-tables:4"

then this code (admittedly crude) might be helpful and that you can use along with client-logging:

/* this program matches up temp-table create and delete log messages.

generated by the 4gl runtime clientlog facility.

-clientlog foo.log -logentrytype "temp-tables:4"

just cat your foo.log log files as generated by the 4GL into tt_all.log.

version 6, gus bjorklund, december 15, 2014.


using Progress.Database.TempTableInfo.

def var deb as logical no-undo initial false.

def var input1 as char no-undo.

def var input2 as char no-undo.

def var data_op as char no-undo.

def var data_name as char no-undo.

def var data_path as char no-undo.

def var data_line as char no-undo.

def var n as int no-undo.

def var m as int no-undo.

def var nmatches as int no-undo.

def var entries as int no-undo.

/* here we collect info about the orphans

a row is added for each create.

the row is eliminated when we see a delete */

def temp-table ttorphans no-undo

field tt_op as character

field tt_name as char

field tt_path as char

field tt_line as char

index ix1 tt_name.

/* here we collect tt name and number of creates

and deletes for each one */

def temp-table ttcounts no-undo

field theName as char

field path as char

field creates as int

field deletes as int

index ix1 theName.

/* read log entries for tt creates and deletes */

input from tt_all.log.



import unformatted input1.

trim (input1).

/* skip the stuff we don't want */

if (index (input1, "TEMP-TABLE")

if (index (input1, "Created")

data_op = entry (11, input1, " ").

data_name = entry (13, input1, " ").

if (index (input1, ")") > 0) then do:

input2 = substring (input1 , index (input1, ")") + 1).

data_path = entry (2, input2, " ") no-error.

data_line = entry (4, input2, " ") no-error.

if deb then

display data_op format "x(1)"

data_name format "x(25)"

data_path format "x(30)"

data_line format "x(6)"



if (data_op <> "Created") and (data_op <> "Deleted") then next inloop.

entries = entries + 1.

if (entries mod 25000) = 0 then display entries.

if (entries > 1000000) then leave.

find ttcounts where theName = data_name no-error.

if not available ttcounts then do:

create ttcounts.

assign theName = data_name

path = data_path

creates = 0

deletes = 0



if (data_op = "Created") then creates = creates + 1.

else deletes = deletes + 1.

find first ttorphans where (data_name = tt_name) and (tt_op <> data_op) no-error.

if available ttorphans then do:

/* have matching tt , get rid of old, drop new */

nmatches = nmatches + 2.

delete ttorphans.

next inloop.


/* no match, so add a new record */

create ttorphans.

assign tt_op = data_op

tt_name = data_name

tt_path = data_path

tt_line = data_line



message entries " records read," nmatches " matched".

output to nodelete.txt.

for each ttcounts where (creates > 0) and (deletes <> creates):

put theName format "x(20)"

(creates - deletes)



output close.

n = 0.

m = 0.

output to nodelete2.txt.

for each ttorphans:

put tt_path format "x(30)"

tt_name format "x(30)"


if tt_op = "Created" then n = n + 1.

if tt_op = "Deleted" then m = m + 1.


output close.

message n " creates and " m " deletes don't match".

def var tableCount as int no-undo.

def var procName as char no-undo.

def var infoHandle as handle no-undo.

repeat n = 1 to TempTableInfo:TempTableCount:

TempTableInfo:GetTableInfoByPosition (n, infoHandle, procName).

display infoHandle:name procName.


Posted by Laura Stern on 15-May-2019 15:36

You said you don’t have “object programming”, and only TT, DataSet parameters, no handles.  But previously you said you have dynamic queries and buffers. Those are tracked with the DynObjects logging.

Posted by dbeavon on 15-May-2019 16:19

I have a support case open with Progress at the moment regarding a memory leak in the _mproapsv.  I can give you more details if you need them; I think the SP will be available in 11.7.5.  Your issue may not be the same as ours.  It would be helpful if you gave us a lot more explanation about your problem.  How fast is the leak?  Can you quantify it?   What process name is actually leaking (_mproapsv.exe? java.exe? tomcat.exe?)

One thing I would suggest doing is create a reproducible that you can play with *outside* of production.  You can run your repro at high speed to amplify the problem.  That may also allow you to isolate the portion of your code that leaks.  In the end you might be left with something that you can send over to Progress tech support.

Are you running on windows or linux?  If windows then I would recommend a sysinternals tool:

This tool is easy to use.  And should help isolate the memory leak to a certain degree, and also quantify it.  I suspect your leak will be seen as allocations in the native heap.

Another thing I would do ... especially since your question hasn't been resolved after a month ... is to approach the PASOE problem from another angle.  You can install the "oemanager" API and use a REST interface to trim your inactive ABL sessions on a recurring schedule (eg. on the hour or half hour).  This is a very safe operation and could be enough to resolve a wide range of so-called "leaks".  (Depending on the definition of "leak".)  In the very least it will give you additional information to add to everything else that you know about your "leak".

Trimming inactive ABL sessions can be accomplished via the OE manager rest interface. Here is a powershell script that shows how to trim inactive sessions from an ms-agent process:

(Posted by me on 26 Dec 2017 14:04, with lots of help from Irfan)

When an inactive session is trimmed, PASOE will clear out all the memory that is directly associated with the session.  That may be half the battle.  When new PASOE client requests arrive, PASOE will create fresh new sessions to replace the ones that were trimmed.  

Yet another thing you may want to look at is the tomcat manager console:


The console may help in your troubleshooting, and may show when HTTP sessions are open for long periods of time.  I don't know about SOAP, but where APSV is concerned sometimes the HTTP sessions can become orphaned for long periods of time and hold onto resources in the MSagent, thereby causing "leaks".  This can be resolved administratively in the tomcat manager console by explicitly forcing the expiration of the HTTP sessions.

Good luck isolating the leaks.  As a side, I will point out that even Java and .Net code can have "leaks" when items on the heap are rooted in memory and the garbage collector is unable to clear everything out.  It would be nice if PASOE had an operation that was analogous to forcing a GC.Collect() operation, which is something to help determine when there is truly a leak.  The closest thing I've found to this (in the world of PASOE) is to trim inactive ABL sessions via the OE manager REST api.

Posted by Daniel Spinelli on 15-May-2019 18:09



Have you heard or worked with swagger?

With swagger you can identify the program that is causing the memory leak as well as the line that is not deleting DATASET.

Tell me if you want to know more than I can tell you.


Posted by atuldalvi123 on 16-May-2019 06:54

Hello Daniel,

No, I have not worked with swagger.

Can you give some more details on it?

And as I said I have few datasets defined in my programs but those are normal ones, no dynamic handles for datasets or output parameter handles.

How swagger will help into this?

Posted by Daniel Spinelli on 16-May-2019 12:58


First sorry for my English. I'm using google translator and I hope you can understand.

I believe that with swagger you will be able to identify the program that is causing the memory leak and know which DATASET is not being removed.

I solved my memory leak problem with swagger help where the programs were adjusted in front of your report.

The following is the step-by-step guide for you to enable and use swagger

- Step 1 - Enable OEmanager

cd /usr/wrk/instance

bin/ undeploy -u tomcat:yourpasswod oemanager     - Password default - tomcat

bin/ deploy -u tomcat:yourpassword -a oemanager /$DLC/servers/pasoe/extras/oemanager.war

vi webapps/oemanager/WEB-INF/oemgrSecurity-container.xml

//remove comments in line 35 <!-- Access to SwaggerUI. Disabled by default, user has to uncomment the below line to enable it -->


Sign in - hostname:port-instcance/manager/     -  do not forget / at the end

/oemanager  - Stop and Start

- Step 2 - Collect Agent ID

Sign in - hostname:port-instcance/oemanager/     - do not forget / at the end   - Usr: tomcat / Pwd: your password

Click "Get / application/{appName}/agents Get Agents"

Click "Try it out"

Inform Instance name and execute

copy "agentID"

- Step 3 - Enable agent for collection

Click " PUT /aaplication/{appName}/agents/{agentID}/ABLObjects/status Enable/Disable ABLObjects and click “Try it out”

Inform instance name and agentID

Change for


  "enable" : "true"


And Execute

Response Body will be changed for: "result": true

- Step 4 - Run your program or your system normally so that the collection is done

- Step 5 - Analyze collection

Click: GET / applications/{appName}/agents/{agentID}/ABLObjects Get ABLObjects Report and click in “Try it out

The screen below will show the handle used;

Do it Download

- Step 6 - Run again your program or your system normally so that the collection is done

- Step 7 - Analyze collection again

Click: GET / applications/{appName}/agents/{agentID}/ABLObjects Get ABLObjects Report and click in “Try it out

The screen below will show the handle used;

Do it Download

- Step 8 - Check the reports

If the handle of the first scan remains, it means that the program is not deleting the memory leak. It should be reviewed and corrected.

If it disappears and generates another, the program is correct without the addition of memory leak.

If the number of rows in the report increases, it means that there is memory leak and must be corrected. The handleID should always be refreshed.

With this tool it is possible to evaluate if the program is having memory leak and correct it. So the memory consumption will drop successfully.

- Step 9 - Disable agent for collection

Click " PUT /aaplication/{appName}/agents/{agentID}/ABLObjects/status Enable/Disable ABLObjects and click “Try it out”

Inform instance name and agentID

Change for


  "enable" : "false"


And Execute

END Step

I hope you understand and if you have any questions, I am at your disposal.


Posted by atuldalvi123 on 16-May-2019 14:12


Where I can set this -clientlog ? My client is not an OpenEdge client, its a .net client and they call the soap services developed in OpenEdge.

We just accept inputs from them, process those inputs in OpenEdge and returns back the output in datasets or temp table.

Posted by Daniel Spinelli on 16-May-2019 14:25


My client is also dotnet.

When you call the dotnet client, it calls a .p program in OpenEdge?

Posted by atuldalvi123 on 16-May-2019 14:29

Does 11.6 support swagger?

Posted by Daniel Spinelli on 16-May-2019 14:36

I'm in version 11.7

See if you can enable it in version 11.6

Posted by Irfan on 16-May-2019 15:05

Swagger is only available in 11.7.3+. If you are on 11.6, you will not find the API's to find the leaks. They are only available from 11.7.2+ onward. I think the choice for you would be enable DynamicObjects logging in the PASOE logEntryTypes and debug which objects are leaking from the agent log.

Posted by Daniel Spinelli on 16-May-2019 15:13


Posted by atuldalvi123 on 20-May-2019 12:59

Hello All,

On the AppServer source code, we have a code structure like 1 include that contains all its temporary table definition and related dataset definitions. We use it wherever we need it and declare the input or output parameters for a dataset without a by-reference or BIND.

Will this also be the reason for the memory leak ?

e.g. Sample.p


define output parameter dataset for dscust.




run CreateData.p(outputDATASET dsCust).




run UpdateData.p(input-output DATASET dsCust).




define output parameter dataset for dscust.




Thanks in advance.

Posted by Laura Stern on 21-May-2019 13:10

Not sure what you're asking now.  Statically defined temp-tables and datasets should not be leaking. If there is a leak there, it would be a bug in the AVM.  But right now, we are not aware of such a bug.  It doesn't matter how you pass them around as parameters.  It is only if you create the TTs or Datasets dynamically that you would be responsible for deleting them and there could be a leak there.

There's been a lot of information provided in this thread.  I think you need to follow up on one or more of the methods provided and then provide more concrete data as to what you tried and what the results were.  

Posted by Thomas Wurl on 21-May-2019 14:50

Yes! Just passing static dataset around should not be a / the problem ...

How do you know that you have a leak on AS? Just a feeling?

From what you are writing there may be some potential problems:

- Soap and XML. You parser may leave X-DOC, X-NODEREF ... in memory.

- Persistent procedures stay in memory...

How do you pass the datasets to your .Net frontend? C# OpenClient Proxy? Do you call "sample.p" directly from .Net or do you have some kind of "service adapter"?

To see what's going on the appserver (or even on a client) I use a simple tool.

- 1) Scan the SESSION on the Appserver and collect objects that stay in memory. Pass the data as a temp-table to the client.

- 2) Do something like calling your sample.p on the appserver

- 3) again 1)

Normally the result of 1) and 3) should show the same objects. If the object list grows then you may have a problem.

/* MemoryChecker.i/


   FIELD ObjectNumber AS INTEGER   LABEL "Number"

   FIELD ObjectHandle AS HANDLE    LABEL "Handle"

   FIELD ObjectType   AS CHARACTER FORMAT "x(32)"  LABEL "Type"

   FIELD ObjectName   AS CHARACTER FORMAT "x(80)"  LABEL "Name"

   INDEX pix IS PRIMARY UNIQUE ObjectNumber.


/* MemoryChecker.p */



DEFINE VARIABLE oObject AS Progress.Lang.Object NO-UNDO.


RUN addWidgets(hObject).
RUN addWidgets(hObject).
RUN addWidgets(hObject).
RUN addWidgets(hObject).
RUN addWidgets(hObject).
RUN addWidgets(hObject).
RUN addWidgets(hObject).

CREATE ttDynObject.
ASSIGN iSeq = iSeq + 1
ttDynObject.ObjectNumber = iSeq
ttDynObject.ObjectType = oObject:GetClass():TypeName
ttDynObject.ObjectName = oObject:ToString()
oObject = oObject:NEXT-SIBLING.

PROCEDURE addWidgets:

CREATE ttDynObject.
ASSIGN iSeq = iSeq + 1
ttDynObject.ObjectNumber = iSeq
ttDynObject.ObjectHandle = hObject
ttDynObject.ObjectType = hObject:TYPE
ttDynObject.ObjectName = hObject:NAME.
hObject = hObject:NEXT-SIBLING.

/* Client prog */


RUN MemoryChecker.p ON SERVER <Appserver> (OUTPUT TABLE ttDynObject).

Then you can load the temp-table into a browse or better a .Net grid with grouping. If you display the number of records of ttDynObject it's easy to see if something goes wrong.

All of this only makes sense if the client talks to the same appserver agent for every request. 
- Start the appserver with a single agent (in development).
- Make the appserver agent stateless bound for the time of checking. (Start a .p o the appserver persistent and delete it when you are done).

Good luck...








This thread is closed