Seriously Slow PAS Start/Stop Operations (Windows 2012 R2 OE

Posted by dbeavon on 22-Nov-2017 10:11

When I need to make changes to the oepas1 configuration (eg. AVM parameters, propath, etc), I need to restart the entire pacific appserver (it isn't enough for the multi-session agent to restart).

I can live with that, but what bothers me is that stopping and restarting the pacific appserver takes a ton of time - like a whole minute or two.  I did a minimal amount of investigation; enough to see that adminserver (java) is launching  numerous, consecutive powershell scripts, each of which is quite CPU-intensive.  This unusual adminserver activity consists of the vast majority of the bottleneck in the stop/restart operations.  I'd guess that the time that the TomCat java process itself takes to stop or start is only a minimal (5-10%) of the overall time, and the majority of the time is related to these wierd adminserver-spawned powershell scripts.

I was wondering if this performance issue is typical in Windows or if there is something misconfigured on our Windows server (a 3.2 Ghz box, SSD, lots of ram).  I though I'd ask the question since I'm sitting here waiting for PAS to restart and I have a few minutes to spare (and don't need a coffee refill yet).

(Powershell scripts aren't good at doing tiny operations because the startup overhead is substantial.  Is there any reason why OE PAS in Windows relies so heavily on powershell?  Could this work - whatever it is - be rolled back into the adminserver java process itself?  Or assuming that .Net is required, couldn't there be a separate "adminserverhelper" service that would perform the various small operations that are otherwise being done in powershell?)

OpenEdge Development - Forum
Posted by Matt Baker on 22-Nov-2017 10:24

 
 
The powershell scripts are mostly calls to tcman env to fetch various values.  Mostly temp directory, pid file location, status and a few other things.  The adminserver has no way of resolving any of these values on its own since many of them are done via substitutions in catalina.properties or appserver.properties or set via shell scripts, and order is very important. 
 
 
You are right, powershell is expensive.  Most of it is compile time for the tcmanager.ps1 script.  Some of this has been addressed newer versions that make use of a status file shared between PAS and the AdminServer.  Other performance improvements are there in 11.7.2 which reduces the number of calls the adminserver makes to tcman.  Still not perfect yet.
 
 
 
 

All Replies

Posted by Brian K. Maher on 22-Nov-2017 10:18

Details on how you are stopping & starting the instance along with the OE version

Posted by Matt Baker on 22-Nov-2017 10:24

 
 
The powershell scripts are mostly calls to tcman env to fetch various values.  Mostly temp directory, pid file location, status and a few other things.  The adminserver has no way of resolving any of these values on its own since many of them are done via substitutions in catalina.properties or appserver.properties or set via shell scripts, and order is very important. 
 
 
You are right, powershell is expensive.  Most of it is compile time for the tcmanager.ps1 script.  Some of this has been addressed newer versions that make use of a status file shared between PAS and the AdminServer.  Other performance improvements are there in 11.7.2 which reduces the number of calls the adminserver makes to tcman.  Still not perfect yet.
 
 
 
 

Posted by dbeavon on 22-Nov-2017 12:25

Thanks.  Its good to know that this is a work-in-progress.  (I never expect to hear such a thing when I run into issues with classic appserver).

If you have any shortcuts I would appreciate knowing them.  (eg I was considering using pskill on all java/_mproapsv at the OS level and then just restarting admin server)?  Of course I would only use shortcuts in my dev environment.

Posted by Matt Baker on 22-Nov-2017 12:34

 
Use 11.7.2 if possible.  There were changes made to reduce a lot of the calls to tcman from adminserver.  Still not perfect but better.
 
Most of the problem is compiling the tcmanager.ps1 script.  It has to be compiled each time it is run, and it gets run a lot.
 
If you are desperate you can compile tcmanager.ps1 and tweak the tcman script to call the .exe instead of the shell script.  We showed performance of about >90% and very little CPU usage.
 
Other thing we ran into recently is that if nslookup <hostname> does not resolve quickly it can become a bottleneck as well.  It is used a lot in a few places (especially tcman create).
 
Or switch to linux.  The tcman shell scripts is much faster.
 

Posted by dbeavon on 22-Nov-2017 14:14

OK.  Thanks for the heads-up about the new version.  I will recommend to the team that we not go to production until we upgrade to 11.7.2.

It sounds like you aren't a fan of powershell.   In that case I'm not sure why you used it at all.  (It really isn't a direct equivalent to a Linux bash script.)

As a side-note, if you like your Linux scripts so much, there may now be the option of running them directly on Windows (see msdn.microsoft.com/.../about).  This feature is probably not compatible with older windows servers, and I'm not sure if performance would be any better, but there would be less mismatch between how you develop your scripts, and your adminserver would operate more similarly between Linux and Windows....  

Posted by Matt Baker on 22-Nov-2017 14:28

 
I am language agnostic.  The tcmanager.ps1 shell is quite large and takes a lot of CPU to compile it.
 
tcman.bat is a shell wrapper around tcmanger.ps1.  DOS batch scripts don’t have the needed capabilities to perform all the same tasks as the unix shell script equivalents, so powershell was chosen since it does and it is available on all the needed windows versions. 
 
WSL only runs on windows 10.
 

Posted by dbeavon on 04-Dec-2017 09:03

I was able to test out PAS in PDSOE 11.7.2.  It is now significantly faster when interacting with pas from the OEE console.  

Thanks for working on improving the performance of the OEE console on Windows.  The prior implementation (using a bunch more instances of powershell) was causing a lot of needless delay. I'd guess that an equivalent Linux installation of PAS was performing better in 11.6. Hopefully the performance of the Windows side is now on par with Linux.  I can't see why they should be that much different from each other, but I haven't had a chance to compare.

This thread is closed