Administering admin server

Posted by dbeavon on 08-Dec-2017 10:21

On a somewhat frequent basis, admin server's "java" process CPU will spike up to a high number, like 500%, and stay there for very long periods of time (sometimes indefinitely).  

Often times there is a correlation with some "bad" thing that a developer did in OEE/OEM,  - like click a link which is known to be "off-limits" (ie. viewing the database connection details for a single database user is "off-limits" because it gathers table/index statistics and that causes enormous CPU problems in admin server itself).

At other times, however there is not any known trigger for the high admin server CPU.  We'll see it running on multiple cores for no reason that anyone can think of.  In these cases it either eventually stops on its own, or someone has to kill it.

All that said, I have a two-part question.  First does anyone know if there is a way to set up detailed logging of admin server itself? (IE That we might read the admserv.log file and find out what it is spending so much CPU on).

Secondly, today we had another instance of high CPU and an hour later we got the following in the admserv.log (right around the time that the CPU activity finally went back to normal):

[2017/12/08@11:01:32.516-0500] [3] [STDERR]                Exception in thread "AdminServer Discovery - multicast://239.2.2.4:6840" 
[2017/12/08@11:01:32.517-0500] [3] [STDERR]                java.lang.OutOfMemoryError: GC overhead limit exceeded
[2017/12/08@11:01:32.517-0500] [3] [STDERR]                	at java.util.Arrays.copyOf(Arrays.java:2367)
[2017/12/08@11:01:32.517-0500] [3] [STDERR]                	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:134)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:118)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:419)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at java.lang.StringBuilder.append(StringBuilder.java:145)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at sun.rmi.server.Util.createStub(Util.java:278)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at sun.rmi.server.Util.createProxy(Util.java:140)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:196)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:310)
[2017/12/08@11:01:32.518-0500] [3] [STDERR]                	at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:252)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at java.rmi.server.UnicastRemoteObject.<init>(UnicastRemoteObject.java:150)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at java.rmi.server.UnicastRemoteObject.<init>(UnicastRemoteObject.java:136)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.common.networkevents.EventObject.<init>(EventObject.java:19)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.mf.RemoteAdminServerDiscovered.<init>(RemoteAdminServerDiscovered.java:33)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.mf.runtime.activemq.AMQDiscoveryListener.processEvent(AMQDiscoveryListener.java:39)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.mf.remote.discovery.AMQDiscoveryAgent.processAlive(AMQDiscoveryAgent.java:230)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.mf.remote.discovery.AMQDiscoveryAgent.processPacket(AMQDiscoveryAgent.java:202)
[2017/12/08@11:01:32.519-0500] [3] [STDERR]                	at com.progress.mf.remote.discovery.AMQDiscoveryAgent.run(AMQDiscoveryAgent.java:128)
[2017/12/08@11:01:32.519-0500] [3] [Fathom]                Cleaning set for OpenEdge Management files.dir directory for files older than 1440 minutes old.
[2017/12/08@11:01:32.520-0500] [3] [STDERR]                	at java.lang.Thread.run(Thread.java:744)
[2017/12/08@11:04:00.128-0500] [3] [STDERR]                SLF4J: Failed toString() invocation on an object of type [org.eclipse.jetty.io.SelectChannelEndPoint]
[2017/12/08@11:05:19.308-0500] [0] [*UnexpectedError*]   * recorded as exception #56 in file ads0.exp.
[2017/12/08@11:05:19.364-0500] [3] [STDERR]                SLF4J: Failed toString() invocation on an object of type [org.eclipse.jetty.io.SelectChannelEndPoint]
[2017/12/08@11:05:19.364-0500] [3] [STDERR]                java.lang.OutOfMemoryError: GC overhead limit exceeded
[2017/12/08@11:05:19.364-0500] [3] [STDERR]                java.lang.OutOfMemoryError: GC overhead limit exceeded

[2017/12/08@11:05:19.409-0500] [3] [STDERR]                Dec 08, 2017 11:05:19 AM com.orientechnologies.common.log.OLogManager log
[2017/12/08@11:05:19.409-0500] [3] [STDERR]                SEVERE: Error during WAL background flush
[2017/12/08@11:05:19.409-0500] [3] [STDERR]                java.lang.OutOfMemoryError: GC overhead limit exceeded


Can anyone help interpret this stuff above, and so we can avoid a recurrence of whatever it was that caused adminserver to flip out?

This is on HP-UX OE 11.6.3.  Also I should note that we are more than generous with memory, and adminserver's java process is started with these memory options: -Xmx2000m -Xms500m -XX:MaxPermSize=500m

Thanks in advance,

David

All Replies

Posted by jbijker on 11-Dec-2017 01:07

We had the same issue when starting up the admin server. We roped in Tech Support and In the end it was a file permission problem, see this knowledge base article:

knowledgebase.progress.com/.../Adminserver-using-100-CPU-and-falling-over-with-heap-space-errors-when-not-started-as-root-user

The 100% CPU in the title is misleading - it will chow up all CPU power it can, i.e. on a 4 CPU box it will get close to 400%.

Sorry but I have no idea how to enable verbose logging on the admin server itself.

Posted by jankeir on 11-Dec-2017 02:35

If desperate and tech support can't help, you could enable flight recording and use the flight recorder from Java Mission Control. Some docs are here: docs.oracle.com/.../run.htm www.oracle.com/.../java-mission-control-1998576.html

You don't need to download it, it comes bundled with the jdk, I have it in C:\Progress\OpenEdge116\jdk\bin\jmc.exe

I've done this occasionally (with great success) for pdsoe to find which plugin was taking all the cpu.

Mind that while as far as I understand you can use this on a dev machine for free, I believe this requires a license from oracle to do it on a production system.

This thread is closed