Memory leak in OpenEdge.Net.HTTP classes?

Posted by pedromarcerodriguez on 01-Jun-2015 10:46


Hi,

 I am using OpenEdge.Net.HTTP classes to populate an elasticSearch instance.

To do it I am using the new available Openedge.Net.HTTP classes: 

USING OpenEdge.Net.HTTP.ClientBuilder FROM PROPATH.
USING OpenEdge.Net.HTTP.IHttpRequest FROM PROPATH.
USING OpenEdge.Net.HTTP.IHttpResponse FROM PROPATH.
USING OpenEdge.Net.HTTP.RequestBuilder FROM PROPATH.
USING Progress.Json.ObjectModel.JsonObject FROM PROPATH.
 
DEFINE VARIABLE oRequest    AS IhttpRequest     NO-UNDO.
DEFINE VARIABLE oResponse   AS IhttpResponse    NO-UNDO.
DEFINE VARIABLE httpUrl     AS CHARACTER        NO-UNDO.
DEFINE VARIABLE iCount      AS INTEGER          NO-UNDO.

MESSAGE "Start" VIEW-AS ALERT-BOX.

httpUrl = "http://localhost:9200/sports/customer".
FOR EACH customer no-lock:
    oRequest = RequestBuilder:Put(httpUrl + "/" + string(Customer.CustNum),new JsonObject()):Request.
    oResponse = ClientBuilder:Build():Client:Execute(oRequest).
    display customer.custNum.
    pause(0).
END.

This code executed with a running instance of ElasticSearch locally should work and the sports database is working fine, but I am finding that there are massive number of objects left behind, I am not sure if I am suppose to do some manual cleaning when using these classes or this code should be Ok like that and a lot of cleaning is missing in the OpenEdge.Net.HTTP classes. 

To give an idea of the problem, after executing this code, I am finding that the VM has grown to 247M.

 So, is it a problem of my code that should be use differently or is it the library not doing some cleansing?

 Cheers,

All Replies

Posted by TheMadDBA on 01-Jun-2015 10:53

I don't see any DELETE OBJECT references in your code. You should really be cleaning up after yourself, especially when using .NET.

Eventually when your OE objects get cleaned up the .NET GC will kick in. But try not to depend on GC if you want the most optimal memory usage.

Posted by pedromarcerodriguez on 01-Jun-2015 10:56

But I am not using .NET, these are Openedge classes available in Openedge.net.pl.

Posted by Peter Judge on 01-Jun-2015 10:57

Despite the "Net" part of the namespace, these classes are pure ABL. "net" means "network", in the same way that java does.
 
GC should kick in on each iteration here.
 
-- peter
 
[collapse]
From: TheMadDBA [mailto:bounce-TheMadDBA@community.progress.com]
Sent: Monday, 01 June, 2015 11:54
To: TU.OE.Development@community.progress.com
Subject: RE: [Technical Users - OE Development] Memory leak in OpenEdge.Net.HTTP classes?
 
Reply by TheMadDBA

I don't see any DELETE OBJECT references in your code. You should really be cleaning up after yourself, especially when using .NET.

Eventually when your OE objects get cleaned up the .NET GC will kick in. But try not to depend on GC if you want the most optimal memory usage.

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by TheMadDBA on 01-Jun-2015 10:58

Well that is unfortunate naming then :)

But good to know Peter. Thanks.

Posted by Peter Judge on 01-Jun-2015 10:58

One of the constructs the http client uses is a class called a ByteBucket. This object contains byte/memptr data and allows you to dynamically add data. There is an array of memptrs inside it, and these memptrs are created once and *reused*. The default is 3 16k memptrs. These memptrs should be destroyed/cleaned up when the ByteBucket is destroyed.
 
You can check whether there's a leak on the ABL side by running the LOG-MANAGER for DynObjects.* which will log OOABL objects and memptr creation/destruction if memory serves.
 
If there is a bug, please log it.
 
-- peter
 
[collapse]
From: pedromarcerodriguez [mailto:bounce-pedromarcerodriguez@community.progress.com]
Sent: Monday, 01 June, 2015 11:48
To: TU.OE.Development@community.progress.com
Subject: [Technical Users - OE Development] Memory leak in OpenEdge.Net.HTTP classes?
 
Thread created by pedromarcerodriguez

Hi,

 I am using OpenEdge.Net.HTTP classes to populate an elasticSearch instance.

To do it I am using the new available Openedge.Net.HTTP classes: 

USING OpenEdge.Net.HTTP.ClientBuilder FROM PROPATH.
USING OpenEdge.Net.HTTP.IHttpRequest FROM PROPATH.
USING OpenEdge.Net.HTTP.IHttpResponse FROM PROPATH.
USING OpenEdge.Net.HTTP.RequestBuilder FROM PROPATH.
USING Progress.Json.ObjectModel.JsonObject FROM PROPATH.
 
DEFINE VARIABLE oRequest    AS IhttpRequest     NO-UNDO.
DEFINE VARIABLE oResponse   AS IhttpResponse    NO-UNDO.
DEFINE VARIABLE httpUrl     AS CHARACTER        NO-UNDO.
DEFINE VARIABLE iCount      AS INTEGER          NO-UNDO.
 
MESSAGE "Start" VIEW-AS ALERT-BOX.
 
httpUrl = "http://localhost:9200/sports/customer".
FOR EACH customer no-lock:
    oRequest = RequestBuilder:Put(httpUrl + "/" + string(Customer.CustNum),new JsonObject()):Request.
    oResponse = ClientBuilder:Build():Client:Execute(oRequest).
    display customer.custNum.
    pause(0).
END.

This code executed with a running instance of ElasticSearch locally should work and the sports database is working fine, but I am finding that there are massive number of objects left behind, I am not sure if I am suppose to do some manual cleaning when using these classes or this code should be Ok like that and a lot of cleaning is missing in the OpenEdge.Net.HTTP classes. 

To give an idea of the problem, after executing this code, I am finding that the VM has grown to 247M.

 So, is it a problem of my code that should be use differently or is it the library not doing some cleansing?

 Cheers,

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by pedromarcerodriguez on 01-Jun-2015 11:03

Anyway, just tried adding a delete object for oRequest and oResponse inside the FOR EACH block with same result, 248M at the end of the execution.

Posted by Peter Judge on 01-Jun-2015 11:05

You're not the first, nor the last no doubt, to say that. We struggled to find a name that was accurate and short.
 
-- peter
 
[collapse]
From: TheMadDBA [mailto:bounce-TheMadDBA@community.progress.com]
Sent: Monday, 01 June, 2015 12:00
To: TU.OE.Development@community.progress.com
Subject: RE: [Technical Users - OE Development] Memory leak in OpenEdge.Net.HTTP classes?
 
Reply by TheMadDBA

Well that is unfortunate naming then :)

But good to know Peter. Thanks.

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by pedromarcerodriguez on 01-Jun-2015 11:09

Thanks Peter,

I have executed the code with logging for dynobjects. But once I got that log file, do you guys have any tool to validate what has been created/allocated against what has been destroyed/unallocated?

Posted by TheMadDBA on 01-Jun-2015 11:17

Just my fault for assuming that NET meant one of the classes that actually called .NET.

I suggest we just blame Microsoft :)

Posted by TheMadDBA on 01-Jun-2015 11:19

See this KB for some sample code to parse the log

knowledgebase.progress.com/.../P133306

Posted by Peter Judge on 01-Jun-2015 11:44

Works for me :)
 
[collapse]
From: TheMadDBA [mailto:bounce-TheMadDBA@community.progress.com]
Sent: Monday, 01 June, 2015 12:18
To: TU.OE.Development@community.progress.com
Subject: RE: [Technical Users - OE Development] Memory leak in OpenEdge.Net.HTTP classes?
 
Reply by TheMadDBA

Just my fault for assuming that NET meant one of the classes that actually called .NET.

I suggest we just blame Microsoft :)

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by Marian Edu on 01-Jun-2015 11:45

I would say fluent syntax might be the problem here, there is probably a client instance created and you have no reference to it

ClientBuilder:Build()

ClientBuilder:Build():Client:Execute(oRequest).#sthash.WVaBkgHP.dpuf

Posted by pedromarcerodriguez on 01-Jun-2015 12:44

Well, I had to change quite a few things to that program to get it to work with the format I was getting the log into it, plus I have addded  processing to compare allocate vs deallocate memory in longchars/memptrs (not sure how correct is my processing there). But finally there is a result:

My findings are:

 There are 58290k that remains allocated that in the log file are logged as handle 0 without an address (not sure what that means or if my processing is correct).

  Procedure sockethelper remains in memory.

  ParameterList object from DefaultRequestBuilder remains in memory.

 JsonObject from JsonBodyResponseFIlter remains in memory.

 There are several instances of allocated memory in ByteBucket (quite a lot) that never gets deallocated.

 There is one instance of allocated memory in ClientSocket that never gets deallocated.

It looks to me that there is somethin wrong there, but let me know, also if possible could you email me how to submit a bug?

Cheers,

PS. I have attached here my modified version of leakcheck.p in case is of anyone's interest.

Posted by pedromarcerodriguez on 01-Jun-2015 12:47

Not sure to understand what the problem is, whatever using fluent syntax or named instances shouldn't change the behaviour of the GC, should it?

And it should be the reponsability of the classes to delete whatever it is that needs to be deleted once the GC collects my instances, not mine as I don't need to know the internals of the library to use it.

Posted by Marian Edu on 01-Jun-2015 12:55

Agreed, I just don't trust GC that much although it's getting better from what I've heard :)

I expect if the GC kicks in and delete the objects fluid should work just fine and any allocated memory to be freed by the object that allocated it.

Posted by TheMadDBA on 01-Jun-2015 14:13

GC is always a little flaky in every language I have used. Even though this should work as is... it would be interesting to see what happens when you move all of the HTTP logic into a different procedure (new .p file) and run that code from inside the customer loop.

Either it behaves the same and shows a true memory leak or it doesn't and OE has a GC/bug issue.

Posted by pedromarcerodriguez on 02-Jun-2015 03:12

Tried that, moving the request calls to an independent procedure, same result, tried with a get operation instead of a put (just in case) same result but less memory used, I am pretty sure there is a memory leak there.

Could anyone point me to where can I submit the bug?

Also, is there someone else using these classes in a different way where that works around this memory leak? It seems surprising nobody has noticed, it leaks almost 0.25Mb for every call for me...

Posted by Peter Judge on 02-Jun-2015 07:27

Log bugs via tech support, please (ie the usual way).
 
-- peter
 
[collapse]
From: pedromarcerodriguez [mailto:bounce-pedromarcerodriguez@community.progress.com]
Sent: Tuesday, 02 June, 2015 04:13
To: TU.OE.Development@community.progress.com
Subject: RE: [Technical Users - OE Development] Memory leak in OpenEdge.Net.HTTP classes?
 
Reply by pedromarcerodriguez

Tried that, moving the request calls to an independent procedure, same result, tried with a get operation instead of a put (just in case) same result but less memory used, I am pretty sure there is a memory leak there.

Could anyone point me to where can I submit the bug?

Also, is there someone else using these classes in a different way where that works around this memory leak? It seems surprising nobody has noticed, it leaks almost 0.25Mb for every call for me...

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by Peter Judge on 02-Jun-2015 12:50

There's at least one leak caused by circular references held by the ClientSocket class, incidentally.

Posted by Peter Judge on 02-Jun-2015 14:59

I've logged issue PSC00333868 for this problem. Contact TS if you want your name associated with it (for notifications etc)..

Posted by Peter Judge on 03-Jun-2015 08:17

[apologies for the long post. the TL;DR is there were a couple of leaks, which are fixed in 11.6 and also be careful of reading too much into the log-manager's outout]

After some investigation I found 2 styles of leak

- the circular one I mentioned earlier, where  a .P is run persistently from an object and which holds a reference to the object, thus rendering garbage collection impotent.\

- i like to use a Initialize/Destroy pattern instead of trusting a constructor to do everything; this requires someone to call the Initialize method on object construction (done) and the Destroy method when the object is being destroyed. This wasn't always done.

The log-manager's output from DynObjects.* - especialy with a high enough log level - needs to be carefully read

- Entries that refer to LONGCHAR only, like the line below are intended for internal-to-ABL-core developers for their edification and delight (and not for ours). They can be safely ignored.

PutBytes OpenEdge.Core.ByteBucket @ 193    x166750    LONGCHAR    19


- Sometimes, the AVM creates objects internally from non-object arguments you pass it. The line below

NewRequest OpenEdge.Net.HTTP.DefaultRequestBuilder @ 54    0    Progress.Lang.Object    Progress.Lang.ParameterList


is the result of the following ABL

oRequest = dynamic-new string(oRequestType:TypeName)( GetOptionStringValue('method':u),
                                                          cast(GetOptionObjectValue('uri':u), URI)).
                

The AVM changes the 2 arguments into a Progress.Lang.ParameterList object over which you have no control. You have to take it on faith that that object will be "GC'ed"

- Objects that are 'held' by static members will only show up as created and not deleted. For instance, the following code

    /** Registry for mapping build types to their implementations */
    define static public property Registry as BuilderRegistry no-undo
        get():
            define variable oRegistry as BuilderRegistry no-undo.
            if not valid-object(RequestBuilder:Registry) then
            do:
                assign oRegistry = new BuilderRegistry().
                RequestBuilder:InitializeRegistry(oRegistry).                
                assign RequestBuilder:Registry = oRegistry.
            end.
            return RequestBuilder:Registry.
        end get.
        private set.

shows up in the logs as

propGet_Registry OpenEdge.Net.HTTP.RequestBuilder @ 74    1049    Progress.Lang.Object    OpenEdge.Net.HTTP.BuilderRegistry


However, there are no corresponding Deleted-by-GC entries. This is because of the fact that the references are held by static members which are destroyed by the AVM when the session shuts down; this destruction happens in an purely internal way, and so there are no logs of the event.


hth

Posted by pedromarcerodriguez on 03-Jun-2015 08:39

Hi Peter,

Thanks for your answer, it will prove very helpful to understand the log-manager's output.

One thing I noticed was that in the destroy method, the persistent procedure is deleted with a no-error clause, and every time that destructor was called (I debug that bit) the handle of the procedure was not valid, not sure if that adds any extra useful information, if not, just ignore it.

Cheers,

Posted by dbeavon on 19-Nov-2018 16:39

Peter was there ever any resolution to the memory leak in OpenEdge/Net/HTTP?

We just deployed PASOE to production for the first time and our MS agents are growing to use a lot of resident memory (well over 1 GB after just a couple hours).  These MS agent processes only have about 5 ABL sessions in each of them.  When I click the sessions and examine the "stack" data that is associated with each, it lists objects and persistent procedures that are occupying memory.  I see a lot of (non-STATIC) references in each session to "BuilderRegistry.r".  

There is usually 1 each of a variety of "STATIC" classes, and those don't bother me.  But the "BuilderRegistry" entries have arbitrary numbers for the "ObjectId" and there are lots of them! Image above.

I will open another ticket with tech support but I was hoping maybe I could shortcut that process if somebody else had encountered this already.  We are running PASOE on windows with OE version 11.7.4.

Thanks, David

Posted by Peter Judge on 19-Nov-2018 20:28

David,
 
This is expected behaviour – the HTTP client (and other components) use a registry of character/names + types (instance of Progress.Lang.Class) for a bunch of stuff . There are around 110 registry entries in the OpenEdge.Net.pl library and about 15 in the OpenEdge.Core.pl.  Each time you do a GET-CLASS() or equivalent, a  new instance of that class is created. These are, to the best of my understanding, static in the AVM (per session).
 
I would expect a certain number of objects to be created in the various builder registries and then for the agents’ memory consumption to plateau.
 
You’re looking at the stacks in OEE by the looks of things. If you use the ABL Objects report you get the same information but more of it. You can also make these calls from an OpenAPI doc that ‘s in 11.7.4. To set it up check out documentation.progress.com/.../index.html  .
 
You’ll need to get the agents, and then enable ABLObjects. Then run your stuff and then GetABLObjects.
 
This has the benefit of containing the OOABL type name. You can remove your types and see what’s left (keep Progress.* and OpenEdge.* references), and if that grows then there’s definitely a leak on our side.
 
 

Posted by dbeavon on 19-Nov-2018 21:24

OK, thanks for the explanation.  I was originally thinking that any objects showing up that didn't have the "STATIC" notation was an indication of a leak - especially if there were a large number of them.  But I didn't do much digging yet.  Now that you say this, I've noticed that my own singleton classes (listed with a "STATIC " notation) are always paired with one that does *not* have that same notation.  I suspect the "STATIC" one represents the type, and the one without that notation is the singleton instance of the type.

Yes, my screenshot was from the "stacks" link in the OEE console for PASOE. (The stacks link can be found on the individual sessions within the "agents" of an abl application).

This thread is closed