Classic AppServer GC overhead limit exceeded

Posted by James Palmer on 15-Dec-2015 09:23

Chaps, 

11.5.1, classic stateless appserver on Win 2012 R2

From time to time the AppServer crashes with "java.lang.OutOfMemoryError: GC overhead limit exceeded" in the log file. We than have to kill the AppServer processes and the Java processes in order to get it running again. 

Any ideas what we can do to fix this? Is this something that might be easy enough to track the cause of?

All Replies

Posted by Paul Koufalis on 15-Dec-2015 09:31

This is a well known issue.

The solution depends on your version of Java, but basically you write out the java heap data to a file and monitor the heap size using a script. Google for -verbose:gc. When the heap size hits a certain percentage of the max you schedule a restart of the broker JVM.

Posted by n.lewers on 15-Dec-2015 09:34

can you please post the java startup options that are currently being used and what version of java?  the error is related to java running out of memory for garbage collection, i believe the Xmx param would influence that.

Posted by James Palmer on 15-Dec-2015 09:38

This is from the admserv.log.

C:\Progress\OpenEdge\bin\jvmStart

-w @{WorkPath}

-o stdout

-e 1 C:\Progress\OpenEdge\jre\bin\java

-Xmx256m

-classpath C:\Progress\OpenEdge\jdk\lib\tools.jar;C:\Progress\OpenEdge\java\progress.jar -DInstall.Dir=C:\Progress\OpenEdge

-Djava.security.policy=C:\Progress\OpenEdge\java\java.policy

-DCanonicalName=fathom1.hob:ID=AppServer com.progress.ubroker.broker.ubroker

-t AS

-i Live

-r rmi://HOB:20931/Live

-f C:\Progress\OpenEdge\properties\ubroker.properties

Not sure of Java version. It's whatever is shipped with 11.5.1 - we haven't upgraded it.

Posted by Paul Koufalis on 15-Dec-2015 09:46

Run "java -fullversion" from a proenv then google for the appropriate -Xx parameters to enable logging to a file. For example, it might be -verbose:gc   -Xloggc:c:\temp\.javaheap.log.

You'll see what the output looks like then just write a simple script to parse it and generate an alert.

Posted by gus on 15-Dec-2015 09:51

try allocating more memory. i see you have -Xmx256m

try 512.

> On Dec 15, 2015, at 10:39 AM, James Palmer wrote:

>

> Update from Progress Community [https://community.progress.com/]

>

> James Palmer [https://community.progress.com/members/jdpjamesp]

>

> This is from the admserv.log.

>

> C:\Progress\OpenEdge\bin\jvmStart

>

> -w @{WorkPath}

>

> -o stdout

>

> -e 1 C:\Progress\OpenEdge\jre\bin\java

>

> -Xmx256m

>

> -classpath C:\Progress\OpenEdge\jdk\lib\tools.jar;C:\Progress\OpenEdge\java\progress.jar -DInstall.Dir=C:\Progress\OpenEdge

>

> -Djava.security.policy=C:\Progress\OpenEdge\java\java.policy

>

> -DCanonicalName=fathom1.hob:ID=AppServer com.progress.ubroker.broker.ubroker

>

> -t AS

>

> -i Live

>

> -r rmi://HOB:20931/Live

>

> -f C:\Progress\OpenEdge\properties\ubroker.properties

>

> Not sure of Java version. It's whatever is shipped with 11.5.1 - we haven't upgraded it.

>

> View online [https://community.progress.com/community_groups/openedge_rdbms/f/18/p/22003/77207#77207]

>

> You received this notification because you subscribed to the forum. To unsubscribe from only this thread, go here [https://community.progress.com/community_groups/openedge_rdbms/f/18/t/22003/mute].

>

> Flag [https://community.progress.com/community_groups/openedge_rdbms/f/18/p/22003/77207?AbuseContentId=9a02c9c5-559e-4760-8cde-13ef471ed6c8&AbuseContentTypeId=f586769b-0822-468a-b7f3-a94d480ed9b0&AbuseFlag=true] this post as spam/abuse.

Posted by James Palmer on 15-Dec-2015 09:52

Sorry Paul, getting

'java' is not recognized as an internal or external command,

operable program or batch file.

Posted by James Palmer on 15-Dec-2015 09:52

Thanks gus - will do.

Posted by Paul Koufalis on 15-Dec-2015 09:56

I thought it was in the PATH but I guess not. Run %DLC%\jre\bin\java -fullversion.  For 11.5.1 it is 1.7.0_45.

Increasing the heap size is definitely smart, especially if you are running OE Mgmt. If OEM is active I set the max heap size to 1 Gb or more. But you still need to monitor for leaks.

Posted by James Palmer on 15-Dec-2015 09:57

Ok worked out how to get the Java version:

java full version "1.7.0_45-b18"

Posted by n.lewers on 15-Dec-2015 10:26

fyi, the java version s/b 1.7 (knowledgebase.progress.com/.../what-version-of-jdk-is-shipped-with-oe-11-5), you can also type C:\Progress\OpenEdge\jre\bin\java -version at the command prompt.

You can try increasing the Xms to 512 or 1024 to see if that helps.  As far as monitoring and tuning related params to garbage collection from Paul's reply, you can look at the following link, www.cubrid.org/.../ .

I am more familiar with HPUX/Linux and I believe we normally used 512, however somebody with more windows server experience can share what they use for that param (as well as the other java:gc params).

Posted by James Palmer on 15-Dec-2015 10:30

Thanks everyone. I've increased the Xms to 512. See how that goes. Will have a bit of a look at that doc too.

Posted by gus on 15-Dec-2015 10:31

i wouldn't bother monitoring. too much work. increase heap size and if oom messages don't stop, increase it more.

> On Dec 15, 2015, at 11:27 AM, n.lewers wrote:

>

> Update from Progress Community [https://community.progress.com/]

>

> n.lewers [https://community.progress.com/members/n.lewers]

>

> fyi, the java version s/b 1.7 (knowledgebase.progress.com/.../what-version-of-jdk-is-shipped-with-oe-11-5) [http://knowledgebase.progress.com/articles/Article/what-version-of-jdk-is-shipped-with-oe-11-5)], you can also type C:\Progress\OpenEdge\jre\bin\java -version at the command prompt.

>

> You can try increasing the Xms to 512 or 1024 to see if that helps. As far as monitoring and tuning related params to garbage collection from Paul's reply, you can look at the following link, www.cubrid.org/.../ [http://www.cubrid.org/blog/dev-platform/how-to-tune-java-garbage-collection/] .

>

> I am more familiar with HPUX/Linux and I believe we normally used 512, however somebody with more windows server experience can share what they use for that param (as well as the other java:gc params).

>

> View online [https://community.progress.com/community_groups/openedge_rdbms/f/18/p/22003/77219#77219]

>

> You received this notification because you subscribed to the forum. To unsubscribe from only this thread, go here [https://community.progress.com/community_groups/openedge_rdbms/f/18/t/22003/mute].

>

> Flag [https://community.progress.com/community_groups/openedge_rdbms/f/18/p/22003/77219?AbuseContentId=74df39da-31fe-4cf6-9c7d-4d4c506922ed&AbuseContentTypeId=f586769b-0822-468a-b7f3-a94d480ed9b0&AbuseFlag=true] this post as spam/abuse.

Posted by Paul Koufalis on 15-Dec-2015 10:43

I am going to respectfully disagree with Gus on this one, especially in cases where OE Management is running and there are a lot of resources being monitored by the AdminServer JVM. And if the AppServer Broker JVM goes JOOM then you need to bounce it, which often times means kicking out all your users.

Monitor.

This thread is closed