AdminServer using high amounts of CPU (OE 11.5.1.025)

Posted by Valeriy Bashkatov on 11-Jan-2017 03:59

Hello,

OS:SUSE Linux Enterprise Server 11 SP1

OpenEdge: 11.5.01.025

The Customer has just been updated to OpenEdge 11.5.1.025 from 10.2B.

After starting AdminServer using high amounts of CPU until it is stopped.  AdminServer not have any resources for autostart (no AppServer & WebSpeed brokers & databases)

In the SAR, we see that immediately after the start of the system  CPU value dramatically increases  and it remains high even after the start.

08:34:27 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:34:32 AM     all      6.91      0.00      2.86      4.43      0.00     85.80
08:34:37 AM     all      5.99      0.00      2.54      6.14      0.00     85.32
08:34:42 AM     all      6.03      0.00      2.47      7.70      0.00     83.80
08:34:47 AM     all      5.80      0.00      2.09      7.89      0.00     84.22
08:34:52 AM     all     10.27      0.00      2.39      8.27      0.00     79.06
08:34:57 AM     all      6.19      0.00      2.20      8.76      0.00     82.86
08:35:02 AM     all      7.20      0.00      3.30     14.57      0.00     74.93
...
...
...
08:41:27 AM     all     17.87      0.00      7.05     13.24      0.00     61.85 ------------ AdminServer start point
08:41:32 AM     all     16.64      0.00      9.02     12.19      0.00     62.15
08:41:37 AM     all     11.43      0.00      8.75     14.59      0.00     65.23
08:41:42 AM     all     13.44      0.00      8.73     14.57      0.00     63.25
08:41:47 AM     all     12.37      0.00      7.92     13.60      0.00     66.12
08:41:52 AM     all     11.05      0.00      8.05     14.87      0.00     66.03
08:41:57 AM     all     13.34      0.00      8.32     15.13      0.00     63.21
08:42:02 AM     all     12.77      0.00     12.28      9.16      0.00     65.79  
08:42:07 AM     all     13.43      0.00     14.20      8.37      0.00     64.00
08:42:12 AM     all      8.95      0.00     12.18      4.97      0.00     73.91
08:42:17 AM     all      8.50      0.00     14.91      4.61      0.00     71.99
08:42:22 AM     all      9.36      0.00     15.55      6.58      0.00     68.52
08:42:27 AM     all     11.40      0.00     15.65      6.69      0.00     66.26
08:42:32 AM     all     10.38      0.00     15.97      7.84      0.00     65.81
08:42:37 AM     all     14.84      0.00     13.73      8.96      0.00     62.46
08:42:42 AM     all     20.81      0.00     23.79      9.53      0.00     45.87
08:42:47 AM     all     25.90      0.00     52.66      7.96      0.00     13.48
08:42:52 AM     all     28.95      0.00     58.61      3.43      0.00      9.01
08:42:57 AM     all     29.98      0.00     53.98      5.53      0.00     10.50
08:43:02 AM     all     28.94      0.00     68.42      0.45      0.00      2.19
08:43:07 AM     all     29.31      0.00     69.65      0.06      0.00      0.99 --- AdminServer start end point (Fathom probe started.)
08:43:12 AM     all     30.39      0.00     69.20      0.05      0.00      0.36
08:43:17 AM     all     30.20      0.00     68.91      0.40      0.00      0.49
08:43:22 AM     all     30.80      0.00     65.05      1.22      0.00      2.93
08:43:27 AM     all     28.88      0.00     63.89      2.51      0.00      4.72
08:43:32 AM     all     28.52      0.00     67.04      1.23      0.00      3.21
08:43:37 AM     all     30.83      0.00     64.70      1.23      0.00      3.24
08:43:42 AM     all     28.32      0.00     69.24      1.11      0.00      1.33
08:43:47 AM     all     25.85      0.00     68.26      1.43      0.00      4.46
08:43:52 AM     all     26.10      0.00     67.16      0.72      0.00      6.02
08:43:57 AM     all     26.10      0.00     66.30      0.91      0.00      6.69
08:44:02 AM     all     26.86      0.00     68.25      1.22      0.00      3.67
08:44:07 AM     all     28.23      0.00     65.26      1.05      0.00      5.46
08:44:12 AM     all     24.97      0.00     71.88      0.68      0.00      2.47
08:44:17 AM     all     25.29      0.00     66.57      1.16      0.00      6.98
08:44:22 AM     all     23.32      0.00     68.89      2.18      0.00      5.61
08:44:27 AM     all     21.37      0.00     67.84      2.56      0.00      8.22
08:44:32 AM     all     23.29      0.00     70.27      1.44      0.00      5.01
08:44:37 AM     all     24.21      0.00     68.44      1.03      0.00      6.32
08:44:42 AM     all     18.67      0.00     70.15      2.44      0.00      8.73
08:44:47 AM     all     18.87      0.00     68.59      1.76      0.00     10.77
08:44:52 AM     all     22.58      0.00     69.50      1.40      0.00      6.52
08:44:57 AM     all     20.44      0.00     68.39      1.90      0.00      9.27
08:45:02 AM     all     22.35      0.00     66.35      1.17      0.00     10.13
08:45:07 AM     all     22.06      0.95     62.82      2.05      0.00     12.12
08:45:12 AM     all     17.86      0.00     65.69      0.61      0.00     15.84
08:45:17 AM     all     20.41      0.00     63.54      0.11      0.00     15.94
08:45:22 AM     all     19.41      0.00     58.43      0.02      0.00     22.14
08:45:27 AM     all     19.28      0.00     61.03      0.17      0.00     19.52
08:45:32 AM     all     17.03      0.00     64.64      0.67      0.00     17.66
08:45:37 AM     all     14.26      0.00     64.66      0.10      0.00     20.97
08:45:42 AM     all     18.03      0.00     66.56      0.10      0.00     15.31
08:45:47 AM     all     18.23      0.00     65.75      0.24      0.00     15.78
08:45:52 AM     all     15.08      0.00     66.38      0.04      0.00     18.51
08:45:57 AM     all     16.22      0.00     65.87      0.50      0.00     17.41
08:46:02 AM     all     20.03      0.00     69.77      0.25      0.00      9.95
08:46:07 AM     all     25.62      0.00     68.17      0.25      0.00      5.96
08:46:12 AM     all     22.91      0.00     72.83      0.51      0.00      3.74
08:46:17 AM     all     23.35      0.00     70.30      1.59      0.00      4.76
08:46:22 AM     all     24.97      0.00     67.41      0.80      0.00      6.82
08:46:27 AM     all     24.46      0.00     61.03      0.81      0.00     13.70 -- AdminServer start shutdown  (User name matches. Shutdown permitted.)
08:46:32 AM     all     20.55      0.00     56.72      2.50      0.00     20.22
08:46:37 AM     all     22.45      0.00     53.39      1.66      0.00     22.50
08:46:42 AM     all     21.82      0.00     52.73      0.95      0.00     24.50
08:46:47 AM     all     25.16      0.00     52.20      2.09      0.00     20.56
08:46:52 AM     all     21.88      0.00     54.79      1.44      0.00     21.88
08:46:57 AM     all     21.77      0.00     57.61      1.69      0.00     18.93
08:47:02 AM     all     23.82      0.00     53.88      2.48      0.00     19.82
08:47:07 AM     all     19.42      0.00     52.92      3.46      0.00     24.20
08:47:12 AM     all     15.27      0.00     52.88      3.47      0.00     28.39
08:47:17 AM     all     14.95      0.00     52.15      3.67      0.00     29.22
08:47:22 AM     all     18.41      0.00     52.66      0.31      0.00     28.63
08:47:27 AM     all     18.54      0.00     52.10      0.98      0.00     28.38
08:47:32 AM     all     15.33      0.00     51.86      0.65      0.00     32.17
08:47:37 AM     all     14.92      0.00     51.35      0.60      0.00     33.13
08:47:42 AM     all     18.06      0.00     53.32      1.43      0.00     27.19
08:47:47 AM     all     19.21      0.00     45.29      1.30      0.00     34.20
08:47:52 AM     all     15.68      0.00     45.21      1.74      0.00     37.36
08:47:57 AM     all     16.00      0.00     47.88      1.55      0.00     34.57
08:48:02 AM     all     14.24      0.00     41.73      0.41      0.00     43.63
08:48:07 AM     all     16.19      0.00     46.97      0.42      0.00     36.42 -- AdminServer end shutdown (OpenEdge Management cache database closed.)
08:48:12 AM     all     11.28      0.00     34.34      4.27      0.00     50.11
08:48:17 AM     all      4.85      0.00      8.11      0.84      0.00     86.20
08:48:22 AM     all      2.86      0.00      7.02      0.13      0.00     89.99
08:48:27 AM     all      4.74      0.00      6.88      1.69      0.00     86.68
08:48:32 AM     all      4.04      0.00      6.86      0.84      0.00     88.26
08:48:37 AM     all      4.72      0.00      7.21      0.59      0.00     87.48
08:48:42 AM     all      7.05      0.00      6.90      2.56      0.00     83.49
08:48:47 AM     all      4.22      0.00      6.89      0.61      0.00     88.28
08:48:52 AM     all      5.65      0.00      7.07      0.37      0.00     86.91
08:48:57 AM     all      4.76      0.00      7.03      0.20      0.00     88.00
08:49:02 AM     all      8.77      0.00      6.99      1.06      0.00     83.18
08:49:07 AM     all     13.72      0.00      7.99      1.41      0.00     76.88
08:49:12 AM     all     10.30      0.00      7.33      0.75      0.00     81.62
08:49:17 AM     all      6.85      0.00      2.43      2.60      0.00     88.12

We tried this recommendation (allowDiscovery=false) http://knowledgebase.progress.com/articles/Article/adminserver-using-high-amounts-of-cpu-and-network-resources-at-startup

It did not help.

Are there any ways to find out why AdminServer uses a lot of CPU?

Regards,
Valeriy

Posted by Valeriy Bashkatov on 24-Jan-2017 01:51

Hello,

We solved the problem by updating the OS from SLES 11.1 to SLES 11.3. After that the load on the server has stabilized.

Thank you all for your answers! I really appreciate it.

Regards,

Valeriy

All Replies

Posted by Matt Baker on 11-Jan-2017 07:13

Use $DLC/jdk/bin/jstack <pid> | somefilea few times a few seconds apart and compare the output to see what is running

Posted by Dmitri Levin on 13-Jan-2017 16:55

There is no jstack in my $DLC/jdk/bin. Does it come standard with OE or it need to be installed there? I have OE Advanced Enterprise Edition.

Posted by Matt Baker on 13-Jan-2017 21:02

jstack is a java utility that is part of the standard JDK distribution.  If you're missing it...

Well, from within OEE you can run /diagnostics/threaddump.jsp which is pretty much the same output.

Posted by Ruanne Cluer on 17-Jan-2017 04:38

If diagnostics/threaddump.jsp doesn’t work it will be because the webserver isn’t responding, in which case you need to use jstack instead.

As a matter of interest, were the cachedb properly closed before restarting? Do you see having to run WAL (a bit like bi recovery on OE) and rebuilding indexes on these databases when you restart the adminserver?

Posted by karasanr on 17-Jan-2017 08:54

In 11.5 Fathom uses more CPU. If you don't really use it. Please bring down fathom.

Posted by Valeriy Bashkatov on 24-Jan-2017 01:51

Hello,

We solved the problem by updating the OS from SLES 11.1 to SLES 11.3. After that the load on the server has stabilized.

Thank you all for your answers! I really appreciate it.

Regards,

Valeriy

This thread is closed