Memory-leak?

Posted by Bartel Dodeman on 13-Nov-2015 05:59

I'm using the Corticon 5.5 for java on Windows-2012. It seems that during each DS-call, the memory-usage for java increases, but afterwards does not decrease. During each call about 100,000 entity-instances are created. So after about 30 DS-calls the max (16Gb) is reached. Last step of each flow is to persist the created entities via a ServiceCallOut.

Do i need tot explicitly remove the created entities in the SCO?

In BRMS.properties i use the setting for not returning the created entities:

com.corticon.server.restrict.response.rulemessages.only=true

Greetings,

Bartel

All Replies

Posted by Gertjan Hendriks on 13-Nov-2015 08:01

Hi Bartel,

Have you tried to install the 5.5.1 update yet? It fixes a lot of (smaller) issues with 5.5.

-Gertjan

Posted by James Arsenault on 13-Nov-2015 16:26

Are you saying the process grows to 16gb and then you get an out-of-memory exception?
 
Jim
 

Posted by Gertjan Hendriks on 16-Nov-2015 10:12

Hi Bartel,

What version of the JRE are you using? Is it the standard 1.7 that came with 5.5? Did you make any changes to startServer.bat or jvm.properties?

Gertjan

Posted by Bartel Dodeman on 29-Nov-2015 04:32

Process grows to almost 16gb and then my DS-call times out (com.corticon.server.execution.queue.timeout=1800000). There's no memory-exception. Using Corticon 5.4.0.3 the proces grew up to 11gb after many calls and then stabilized.

With the help of Gertjan i installed 5.5.1-bugfix, but that did not make a difference.

Bartel

Posted by Bartel Dodeman on 29-Nov-2015 04:34

I'm using java1.8. I will try using the standard 1.7 again and see what happens.

Bartel

Posted by James Arsenault on 29-Nov-2015 09:09

Have you tested with your SCOs disabled? If there is a memory leak in them you'll need to profile your code to identify the cause.

Are you using EDC? Is it possible you're inadvertently running some query which is retrieving massive amounts of data?

Posted by Bartel Dodeman on 01-Dec-2015 03:28

My SCO runs several queries in order to create the required entities. One of those queries has a massive resultset (about 100000 rows). After the resultsets are processed (creating entities) the queries and database-connection are closed.  With Corticon 5.4.0.3 i did use a combination of EDC ans SCO for database-actions. Since Corticon 5.5.1. i perform the database-actions via my SCO exclusively and do not use EDC any more. As a kind of test i run many DS and after half a day CorticonServer-console showed the following log-enties :

Exception in thread "catalina-exec-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "catalina-exec-2"

Exception in thread "Timer-9" Exception in thread "Timer-10"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-9"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-10"

Exception in thread "Timer-12"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-12"

Exception in thread "Timer-13" Exception in thread "Timer-14"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-13"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-14"

Exception in thread "Timer-11"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-11"

Exception in thread "Timer-15"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-15"

Exception in thread "Timer-16"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-16"

Exception in thread "Timer-17"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-17"

Exception in thread "Timer-0" Exception in thread "Timer-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"

Exception in thread "Timer-3"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-3"

Exception in thread "Timer-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-2"

Exception in thread "Timer-8"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-8"

Exception in thread "Timer-6" Exception in thread "Timer-7"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-6"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-7"

Exception in thread "Timer-4" Exception in thread "Timer-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-4"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-5"

Could it be that the thread for each DS-call does not end?

Posted by James Arsenault on 01-Dec-2015 07:17

Any thing is possible but I don't think that's the cause. I would first look to see if your SCO is keeping a reference to the data it queries or the entities it creates in some way which persists between executions. The cumulative effect of this could be leading to the OOM error. Do you have access to, or experience with, a profiler? That would most likely point to the cause.

Posted by Gertjan Hendriks on 01-Dec-2015 07:33

Hi Bartel,

Is the use of Java 1.7 (instead of Java 1.8) causing a slower increase of used memory?

Did you add a RuleSheet to the RuleFlow yet, that clears the created instances after the SCO has performed the SQL INSERT?

- Gertjan

Posted by Bartel Dodeman on 01-Dec-2015 07:50

I already checked my SCO-class. It has no members that contains the queried data.

Altough i have no experience i will try a profiler. I'm leaning towards VisualVM ,but perhaps you do have a special recommendation?

Posted by Bartel Dodeman on 01-Dec-2015 08:33

Hi Gertjan,

Yes i tried both, but with no effect. I removed the created entities explicitly as a final step in my SCO. Total time for one DS-call increased with 2 minutes, but it did not have benefits in memory-usage.

Bartel

Posted by James Arsenault on 01-Dec-2015 08:48

Personally I like YourKit for java profiling.

Posted by James Arsenault on 01-Dec-2015 08:52

When looking at Java memory usage you need to be careful how to interpret the process size. Excluding actually hitting an OOM condition, the process size does not definitively identify how much memory is used. It's all subject to Java garbage collection. You could have a large process size but the heap usage is low; that just means Java hasn't freed the unused heap space.

Posted by Bartel Dodeman on 22-Dec-2015 07:09

Gertjan visited me in order to fix this problem. He re-compiled all of the ruleflows leaving the EDC-settings empty. The result of this is that i'm happy to have one CorticonServer-machine that keeps on running all my decisionservices without any problem. This happens to be the only server that runs on dedicated hardware and OS is Windows7-professional. All my other servers still show the memory-problem when running the most demanding decisionservice repeatedly. Those servers are virtual (ESX-server-pool) and OS is Windows-server2012.

So our next plan is to replace all virtual servers by dedicated hardware.

Bartel

Posted by Bartel Dodeman on 04-Jan-2016 04:40

We also fixed the memory-problem on the virtual servers. The problem was that the Xmx-setting in jvm.properties was equal to the amount of fysical memory. Decreasing the Xmx-setting with 1Gb fixed the problem.

Bartel

This thread is closed