How to Restart a PASOE ABL Application

Looks like /oemanager webapp returned a 500 which means an internal error.  Check the PASOE logs and see if there is more details as to why the agent failed to stop properly.   This isn't an OEM issue, but rather a /oemanager issue.

The shutdown through OEM is a stop of all each agent for the abl application, and then a restart of each webapp that contributes to the ABL application (as listed in openedge.properties).  This is all done through the /oemanager interface REST API.

Hi Matt.

There are errors in the catalina*.log but not sure what they mean exactly. We do not have any webapps deployed (not intentionally). We chose 'deploy', selected 'oeabl.war' in the 'Deploy ABL Application' dialog, and gave it a name. Afterwards, we set the propath and agent parameters.

2019-05-17T12:55:15.241-05:00 WARN  [thd-10] o.a.c.loader.WebappClassLoaderBase.clearReferencesThreads - The web application [oadevel] appears to have started a thread named [Camel (camel) thread #3 - ShutdownTask] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)

What would be the goal for restarting an ABL application?  

We use lots of distinct ABL applications within a single instance.  There is rarely a scenario where one of them needs to be managed independently of the others.  Most of our management is done by trimming ABL sessions (ie. within an application).  Or forcing the expiration of HTTP sessions.  Or restarting individual msagent processes (mproapsv).  Or restarting the entire instance.

There is a lot of configuration related to an ABL application, but changes to that will take effect dynamically.  Are you trying to restart to make a config change?

Another thing I've noticed is that PASOE is fairly resilient.  If you want to just terminate one of the mproapsv processes, you can do that at the OS level.  I've found that you can basically just kill it out of task manager and PASOE will start up another one to take its place as needed.  Of course this is a "quick-and-dirty" approach and Progress certainly wouldn't recommend it.

In production our preferred approach to do maintenance is by using load-balancing and the "gracefully" removing a server from the load.  Then after the server is inactive for a while, you can do any maintenance that is needed.

"TO the best of my knowledge, there's no "restart ABL app" option."

Thanks Peter.

I wonder what the 'Reload' button on the ABL Application tab is supposed to do? I couldn't find much information about it on the Hub (just the WebApp reload button). It would be nice if it did work. :)

"What would be the goal for restarting an ABL application?"

Testing enviroment. We need to empty persistent procedures that are running when they have been changed.

FWIW, I get this lovely error when trying to get the list of agents (via OEM). This error does not occur on a PASOE instance that only has a single ABL Application. :-\

"Get agent sessions failed for application oadevel: Cannot construct instance of `com.progress.isq.ipqos.resources.plugins.pas.entities.PASAgentSessionWrapper$Result` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('''') at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.progress.isq.ipqos.resources.plugins.pas.entities.PASAgentSessionWrapper["result"])"

Looks like /oemanager webapp fell over completely.  It isn't returning any valid JSON and OEM choked on the bad data.   Looks like /oemanager is returning an empty string.

Jeff,

Those errors are from Tomcat leak detector valve.  They indicate some thread failed to be shutdown (as the message says).  They are harmless by themselves as they are warnings, but webapps should be playing nice and cleanup after themselves on shutdown.  

Okay. If this is something that we are causing, I don't know how we are doing it.

The left over thread warning from Tomcat isn't something you should be able to cause.  The recommended path is to contact tech support if you care about the noise in the log file and they'll ask you a bunch of questions, and see if they can reproduce it to log a bug.

As for the agent not shutting down...that is a PASOE question and the process is the same, call tech support so they can investigate.  

As for the OEM choking on the bad output. Some improvements are warranted there to handle the bad output better.

I don't care about the noise. I wouldn't have noticed if you had told me to look at the log. :-)

I just want to be able to restart my ABL Application (or remove the agents as suggested) and I cannot do either one. Maybe we deployed it all wrong.

PDSOE is supposed to be doing this behind the scenes for you via OEM.  You can replicate this using the REST API's for PASOE as long as the /oemanager and /manager webapps are installed.

1. fetch list of webapps (and correlate what is listed in <instance>/conf/openedge.properties if needed)

documentation.progress.com/.../index.html

2. fetch each agent for each webapp

documentation.progress.com/.../index.html

3. request stop for the each agent

documentation.progress.com/.../index.html

4.

The restart of the webapp can be done through Tomcat's /manager application, or you can use tcman from command line to stop it and then restart it (this just calls the /oemanager text API).

docs.progress.com/.../Stop-a-Web-application-disable.html

OEM is bundling this all together through the restart button for you.  PDSOE is using the REST API that OEM exposes to do this.

It looks like step 3 is failing based on the error OEM is spitting out.

Hi Jeff,

Hi Shelly. I can duplicate consistently. Not sure if it's a bug or we are doing something wrong but I will log it.

This isn't using PDSOE. This is using OEM. We aren't deploying a .WAR file of our application but just modifying the PROPATH to point to our various testing deployments.

Our goal was to have multiple ABL Apps on a single PASOE instance on our testing server (instead of the 3-4 PASOE instances that we have now) as I learned at Next that we could do this. However, the problem was that there is no easy way to restart each ABL Application (even though there is a 'reload' button).

Hello Jeff,

Here is additional information in case you need it.

When working with PASOE, there are two types of applications: Web Application and ABL Web Application.

You can configure Agent and SessMgr parameters at the ABL WebApp level. For example, you can have separate databases and PROPATH settings for each ABL WebApps.

You can see the help info for the pasman / tcman.sh deploy command to see how to specify it:

   proenv>bin/tcman.sh help deploy

   usage: tcman deploy  [options] -u <uid:pwd> [-l] [-a <app-name>] <war-file-path> [abl-app-name]

Example:

   bin/tcman.sh deploy -a webapp1 $DLC/servers/pasoe/extras/oeabl.war ablapp1

   bin/tcman.sh deploy -a webapp2 $DLC/servers/pasoe/extras/oeabl.war ablapp2

You can use the tcman.sh reload command to reload the Web Application.

Example:

 bin/tcman.sh reload -u <username>:<password> /ablapp2

I understand that this would reload the Web Application at the Tomcat level.

You can use ps -ef to see the AVM process associated with the ABL WebApp.

I hope this helps,

Edsel

I don't know if it helps, but this question came up three years ago and at the time there wasn't any way to control abl applications individually within pasoe.  The best management options are available via load-balancing.  If you have multiple applications, it is likely that you will also need high-availability via load-balancing.

See those comments here...

https://community.progress.com/community_groups/openedge_general/f/26/t/21847

... starting from ...

"How does one control the starting and stopping of one ABL application under a PAS instance.  Lets just say I want to add an additional ABL application while the instance is running... is this possible and can you then startup this ABL application without stopping and starting the PAS instance"

>> We need to empty persistent procedures that are running when they have been changed

For anything that is changing within the AVM, the "resetting" can be done at the *ABL-Session* level.  You don't need to mess with the ABL application as a whole, nor even the msagent process.  (Stopping the msagent process would do the trick but it is overkill.)  You would just need to clear away the individual, inactive ABL sessions that predate your changes.  You can do this via the oemanager interface with REST.  For example here is a similar thread where Irfan was helping me to trim away the ABL sessions which (because of -q) had not picked up a new deployment of r-code :

https://community.progress.com/community_groups/openedge_development/f/19/t/36461?pi20882=2

Again, the better option is to use load-balancing and migrate clients to another server, so you can have an entire server to yourself for maintenance purposes.  Hope this is clear.

+1

SHELLEY CHASE WROTE THE FOLLOWING POST AT 5-19-2019 19:56:Hi Jeff,

Hello Jeff,

Here is more feedback.

I used  PASOE 11.7.4 (Linux) and I was able to use the Reload option in OEM to stop the server process associated with the ABL WebApp.

I noticed that it performs a REST request of "reset" using PUT.

I was able to run the same command via curl without issues.

ps -ef | fgrep ablapp2

curl -v -b /tmp/cookiefile -X PUT -d 'username=<username>&password=<password>' localhost:9090/login.jsp

curl -X PUT -b /tmp/cookiefile -d '{"reset":"true"}' localhost:9090/.../reset

ps -ef | fgrep ablapp2

Considering that you can reproduce the issue consistently, there must something else in the environment.

I hope this helps.

Note:

First curl command should use "-c" to save the cookies in particular OEMSESSION_ID and use POST to perform FORM-based authentication:

curl -v -c /tmp/cookiefile -X POST -d 'username=admin&password=4admin' localhost:9090/login.jsp

The 2nd curl command uses -b to read the cookies.

curl -X PUT -b /tmp/cookiefile -d '{"reset":"true"}' localhost:9090/.../reset

This is using OE 12. I've not tried using OE 11.7. But this is not a "Web App". This is the "ABL Application" tab on OEM.

I think we'll just go back to using multiple instances. It works and it's seems easier to manage.

The OEM rest api does not require cookies.  It will use them for session management if present, but they are not required.  It falls back to HTTP basic if the cookie is not present and you provide the credentials on the command line.  You can use HTTP basic authentication by specifying the -u switch for curl.  You can simplify the example and skip the call to login.jsp and just call the reset service directly.

"The log messages about memory leaks are just noise."

The initial message I started this thread with was not in the log. It was appearing as a dialog in OEM.

We will try with tcman and see what the difference may be.

For those following along, thank you for all the input. I think it is resolved.

I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

Anyhow.. this time the default was left and our other ABL Applications were added and everything seems to work swimmingly.  Even the 'Reload' button the 'ABL Applications' tab OEM does what is expected.

[View:/cfs-file/__key/communityserver-discussions-components-files/26/stopAgent.zip:320:240]

Hi Jeff,

I have attached a script that basically terminates the agents for the given ABL Application. I have tested it on 12.0 and it works fine to me. It is very simple to run so it should not take a lot of time. Can you please give it a try and let us know if the functionality is working as expected ?

Here are the step you have to follow to execute this script

### Script to terminateagents for the given ABL Application ###

1. Copy all the scripts to the PASOE Instance /bin directory.
2. Run the below command
    stopAgent.bat <ABLAppname>
  Example:- stopAgent.bat oepas1
3. This should termiante the agents for the given ABL Application.
 

>>>>I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

I attempted that also and ran into issues as well.  OEM didn't appear to like it, but I don't believe I tried to restart an oeable application.

>I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

Jeff, we had all sorts of issues with using the same name. We gave up and just named the apps different.

Thank you for the script Irfan. We will give it a try today and let you know how it works. I'm not sure that we will need it but it will be good to have on hand. :)