Memory-leak?

Hi Bartel,

Have you tried to install the 5.5.1 update yet? It fixes a lot of (smaller) issues with 5.5.

-Gertjan

Hi Bartel,

What version of the JRE are you using? Is it the standard 1.7 that came with 5.5? Did you make any changes to startServer.bat or jvm.properties?

Gertjan

Process grows to almost 16gb and then my DS-call times out (com.corticon.server.execution.queue.timeout=1800000). There's no memory-exception. Using Corticon 5.4.0.3 the proces grew up to 11gb after many calls and then stabilized.

With the help of Gertjan i installed 5.5.1-bugfix, but that did not make a difference.

Bartel

I'm using java1.8. I will try using the standard 1.7 again and see what happens.

Bartel

Have you tested with your SCOs disabled? If there is a memory leak in them you'll need to profile your code to identify the cause.

Are you using EDC? Is it possible you're inadvertently running some query which is retrieving massive amounts of data?

My SCO runs several queries in order to create the required entities. One of those queries has a massive resultset (about 100000 rows). After the resultsets are processed (creating entities) the queries and database-connection are closed.  With Corticon 5.4.0.3 i did use a combination of EDC ans SCO for database-actions. Since Corticon 5.5.1. i perform the database-actions via my SCO exclusively and do not use EDC any more. As a kind of test i run many DS and after half a day CorticonServer-console showed the following log-enties :

Exception in thread "catalina-exec-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "catalina-exec-2"

Exception in thread "Timer-9" Exception in thread "Timer-10"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-9"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-10"

Exception in thread "Timer-12"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-12"

Exception in thread "Timer-13" Exception in thread "Timer-14"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-13"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-14"

Exception in thread "Timer-11"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-11"

Exception in thread "Timer-15"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-15"

Exception in thread "Timer-16"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-16"

Exception in thread "Timer-17"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-17"

Exception in thread "Timer-0" Exception in thread "Timer-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"

Exception in thread "Timer-3"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-3"

Exception in thread "Timer-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-2"

Exception in thread "Timer-8"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-8"

Exception in thread "Timer-6" Exception in thread "Timer-7"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-6"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-7"

Exception in thread "Timer-4" Exception in thread "Timer-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-4"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-5"

Could it be that the thread for each DS-call does not end?

Any thing is possible but I don't think that's the cause. I would first look to see if your SCO is keeping a reference to the data it queries or the entities it creates in some way which persists between executions. The cumulative effect of this could be leading to the OOM error. Do you have access to, or experience with, a profiler? That would most likely point to the cause.

Hi Bartel,

Is the use of Java 1.7 (instead of Java 1.8) causing a slower increase of used memory?

Did you add a RuleSheet to the RuleFlow yet, that clears the created instances after the SCO has performed the SQL INSERT?

- Gertjan

I already checked my SCO-class. It has no members that contains the queried data.

Altough i have no experience i will try a profiler. I'm leaning towards VisualVM ,but perhaps you do have a special recommendation?

Hi Gertjan,

Yes i tried both, but with no effect. I removed the created entities explicitly as a final step in my SCO. Total time for one DS-call increased with 2 minutes, but it did not have benefits in memory-usage.

Bartel

Personally I like YourKit for java profiling.

When looking at Java memory usage you need to be careful how to interpret the process size. Excluding actually hitting an OOM condition, the process size does not definitively identify how much memory is used. It's all subject to Java garbage collection. You could have a large process size but the heap usage is low; that just means Java hasn't freed the unused heap space.

Gertjan visited me in order to fix this problem. He re-compiled all of the ruleflows leaving the EDC-settings empty. The result of this is that i'm happy to have one CorticonServer-machine that keeps on running all my decisionservices without any problem. This happens to be the only server that runs on dedicated hardware and OS is Windows7-professional. All my other servers still show the memory-problem when running the most demanding decisionservice repeatedly. Those servers are virtual (ESX-server-pool) and OS is Windows-server2012.

So our next plan is to replace all virtual servers by dedicated hardware.

Bartel

We also fixed the memory-problem on the virtual servers. The problem was that the Xmx-setting in jvm.properties was equal to the amount of fysical memory. Decreasing the Xmx-setting with 1Gb fixed the problem.

Bartel