How to Restart a PASOE ABL Application

Posted by Jeff Ledbetter on 17-May-2019 18:00

Hi.

On our test server, we are experimenting with a single PASOE instance with multiple ABL Applications. In this scenario, is it possible to restart one of the ABL Applications without bouncing the whole instance? 

Using OEM, there is a 'Reload' option on the 'ABL Applications' tab. This however throws an error when selecting the ABL Application and choosing 'Reload':

"Failed to reload ABL application. oadevel - Unable to stop agent KZG98v5YSqmPUwmNwzCaGA: Server returned 500 for localhost:8821/.../KZG98v5YSqmPUwmNwzCaGA with an unexpected content type of text/html. An response containing application/vnd.progress+json or application/json was expected.."

Is this supposed to work and this is a bug? Or, is this expected behavior?

Thanks.

Posted by Peter Judge on 17-May-2019 18:30

TO the best of my knowledge, there's no "restart ABL app" option. To do so, you need to
 
  1. Kill all the agents for that abl application . You can get the list of running  agents via the REST API and kill them that way too.
  2. Restart (reload) all the ABL webapps associated with that ABL application.
 
 
 

All Replies

Posted by Matt Baker on 17-May-2019 18:15

Looks like /oemanager webapp returned a 500 which means an internal error.  Check the PASOE logs and see if there is more details as to why the agent failed to stop properly.   This isn't an OEM issue, but rather a /oemanager issue.

The shutdown through OEM is a stop of all each agent for the abl application, and then a restart of each webapp that contributes to the ABL application (as listed in openedge.properties).  This is all done through the /oemanager interface REST API.

Posted by Jeff Ledbetter on 17-May-2019 18:25

Hi Matt.

There are errors in the catalina*.log but not sure what they mean exactly. We do not have any webapps deployed (not intentionally). We chose 'deploy', selected 'oeabl.war' in the 'Deploy ABL Application' dialog, and gave it a name. Afterwards, we set the propath and agent parameters.

2019-05-17T12:55:15.241-05:00 WARN  [thd-10] o.a.c.loader.WebappClassLoaderBase.clearReferencesThreads - The web application [oadevel] appears to have started a thread named [Camel (camel) thread #3 - ShutdownTask] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)

java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

java.lang.Thread.run(Thread.java:745)

Posted by Peter Judge on 17-May-2019 18:30

TO the best of my knowledge, there's no "restart ABL app" option. To do so, you need to
 
  1. Kill all the agents for that abl application . You can get the list of running  agents via the REST API and kill them that way too.
  2. Restart (reload) all the ABL webapps associated with that ABL application.
 
 
 

Posted by dbeavon on 17-May-2019 18:35

What would be the goal for restarting an ABL application?  

We use lots of distinct ABL applications within a single instance.  There is rarely a scenario where one of them needs to be managed independently of the others.  Most of our management is done by trimming ABL sessions (ie. within an application).  Or forcing the expiration of HTTP sessions.  Or restarting individual msagent processes (mproapsv).  Or restarting the entire instance.

There is a lot of configuration related to an ABL application, but changes to that will take effect dynamically.  Are you trying to restart to make a config change?

Another thing I've noticed is that PASOE is fairly resilient.  If you want to just terminate one of the mproapsv processes, you can do that at the OS level.  I've found that you can basically just kill it out of task manager and PASOE will start up another one to take its place as needed.  Of course this is a "quick-and-dirty" approach and Progress certainly wouldn't recommend it.

In production our preferred approach to do maintenance is by using load-balancing and the "gracefully" removing a server from the load.  Then after the server is inactive for a while, you can do any maintenance that is needed.

Posted by Jeff Ledbetter on 17-May-2019 18:43

"TO the best of my knowledge, there's no "restart ABL app" option."

Thanks Peter.

I wonder what the 'Reload' button on the ABL Application tab is supposed to do? I couldn't find much information about it on the Hub (just the WebApp reload button). It would be nice if it did work. :)

Posted by Jeff Ledbetter on 17-May-2019 18:45

"What would be the goal for restarting an ABL application?"

Testing enviroment. We need to empty persistent procedures that are running when they have been changed.

Posted by Jeff Ledbetter on 17-May-2019 19:15

FWIW, I get this lovely error when trying to get the list of agents (via OEM). This error does not occur on a PASOE instance that only has a single ABL Application. :-\

"Get agent sessions failed for application oadevel: Cannot construct instance of `com.progress.isq.ipqos.resources.plugins.pas.entities.PASAgentSessionWrapper$Result` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('''') at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: com.progress.isq.ipqos.resources.plugins.pas.entities.PASAgentSessionWrapper["result"])"

Posted by Matt Baker on 17-May-2019 19:31

Looks like /oemanager webapp fell over completely.  It isn't returning any valid JSON and OEM choked on the bad data.   Looks like /oemanager is returning an empty string.

Posted by Matt Baker on 17-May-2019 19:34

Jeff,

Those errors are from Tomcat leak detector valve.  They indicate some thread failed to be shutdown (as the message says).  They are harmless by themselves as they are warnings, but webapps should be playing nice and cleanup after themselves on shutdown.  

Posted by Jeff Ledbetter on 17-May-2019 19:45

Okay. If this is something that we are causing, I don't know how we are doing it.

Posted by Matt Baker on 17-May-2019 19:55

The left over thread warning from Tomcat isn't something you should be able to cause.  The recommended path is to contact tech support if you care about the noise in the log file and they'll ask you a bunch of questions, and see if they can reproduce it to log a bug.

As for the agent not shutting down...that is a PASOE question and the process is the same, call tech support so they can investigate.  

As for the OEM choking on the bad output. Some improvements are warranted there to handle the bad output better.

Posted by Jeff Ledbetter on 17-May-2019 20:02

I don't care about the noise. I wouldn't have noticed if you had told me to look at the log. :-)

I just want to be able to restart my ABL Application (or remove the agents as suggested) and I cannot do either one. Maybe we deployed it all wrong.

Posted by Matt Baker on 17-May-2019 20:14

PDSOE is supposed to be doing this behind the scenes for you via OEM.  You can replicate this using the REST API's for PASOE as long as the /oemanager and /manager webapps are installed.

1. fetch list of webapps (and correlate what is listed in <instance>/conf/openedge.properties if needed)

documentation.progress.com/.../index.html

2. fetch each agent for each webapp

documentation.progress.com/.../index.html

3. request stop for the each agent

documentation.progress.com/.../index.html

4.

The restart of the webapp can be done through Tomcat's /manager application, or you can use tcman from command line to stop it and then restart it (this just calls the /oemanager text API).

docs.progress.com/.../Stop-a-Web-application-disable.html

OEM is bundling this all together through the restart button for you.  PDSOE is using the REST API that OEM exposes to do this.

It looks like step 3 is failing based on the error OEM is spitting out.

Posted by Shelley Chase on 20-May-2019 00:56

Hi Jeff,


If you can reproduce, please log a bug. 
It shouldn’t be this hard. 

Thanks
-Shelley

On May 17, 2019, at 4:15 PM, Matt Baker <bounce-mbaker@community.progress.com> wrote:

Update from Progress Community
Matt Baker

PDSOE is supposed to be doing this behind the scenes for you via OEM.  You can replicate this using the REST API's for PASOE as long as the /oemanager and /manager webapps are installed.

1. fetch list of webapps (and correlate what is listed in <instance>/conf/openedge.properties if needed)

documentation.progress.com/.../index.html

2. fetch each agent for each webapp

documentation.progress.com/.../index.html

3. request stop for the each agent

documentation.progress.com/.../index.html

4.

The restart of the webapp can be done through Tomcat's /manager application, or you can use tcman from command line to stop it and then restart it (this just calls the /oemanager text API).

docs.progress.com/.../Stop-a-Web-application-disable.html

OEM is bundling this all together through the restart button for you.  PDSOE is using the REST API that OEM exposes to do this.

It looks like step 3 is failing based on the error OEM is spitting out.

View online

 

You received this notification because you subscribed to the forum.  To unsubscribe from only this thread, go here.

Flag this post as spam/abuse.

Posted by Jeff Ledbetter on 20-May-2019 13:05

Hi Shelly. I can duplicate consistently. Not sure if it's a bug or we are doing something wrong but I will log it.

This isn't using PDSOE. This is using OEM. We aren't deploying a .WAR file of our application but just modifying the PROPATH to point to our various testing deployments.

Our goal was to have multiple ABL Apps on a single PASOE instance on our testing server (instead of the 3-4 PASOE instances that we have now) as I learned at Next that we could do this. However, the problem was that there is no easy way to restart each ABL Application (even though there is a 'reload' button).

Posted by egarcia on 20-May-2019 13:45

Hello Jeff,

Here is additional information in case you need it.

When working with PASOE, there are two types of applications: Web Application and ABL Web Application.

You can configure Agent and SessMgr parameters at the ABL WebApp level. For example, you can have separate databases and PROPATH settings for each ABL WebApps.

You can see the help info for the pasman / tcman.sh deploy command to see how to specify it:

   proenv>bin/tcman.sh help deploy

   usage: tcman deploy  [options] -u <uid:pwd> [-l] [-a <app-name>] <war-file-path> [abl-app-name]

Example:

   bin/tcman.sh deploy -a webapp1 $DLC/servers/pasoe/extras/oeabl.war ablapp1

   bin/tcman.sh deploy -a webapp2 $DLC/servers/pasoe/extras/oeabl.war ablapp2

You can use the tcman.sh reload command to reload the Web Application.

Example:

 bin/tcman.sh reload -u <username>:<password> /ablapp2

I understand that this would reload the Web Application at the Tomcat level.

You can use ps -ef to see the AVM process associated with the ABL WebApp.

I hope this helps,

Edsel

Posted by dbeavon on 20-May-2019 13:49

I don't know if it helps, but this question came up three years ago and at the time there wasn't any way to control abl applications individually within pasoe.  The best management options are available via load-balancing.  If you have multiple applications, it is likely that you will also need high-availability via load-balancing.

See those comments here...

https://community.progress.com/community_groups/openedge_general/f/26/t/21847

... starting from ...

"How does one control the starting and stopping of one ABL application under a PAS instance.  Lets just say I want to add an additional ABL application while the instance is running... is this possible and can you then startup this ABL application without stopping and starting the PAS instance"

>> We need to empty persistent procedures that are running when they have been changed

For anything that is changing within the AVM, the "resetting" can be done at the *ABL-Session* level.  You don't need to mess with the ABL application as a whole, nor even the msagent process.  (Stopping the msagent process would do the trick but it is overkill.)  You would just need to clear away the individual, inactive ABL sessions that predate your changes.  You can do this via the oemanager interface with REST.  For example here is a similar thread where Irfan was helping me to trim away the ABL sessions which (because of -q) had not picked up a new deployment of r-code :

https://community.progress.com/community_groups/openedge_development/f/19/t/36461?pi20882=2

Again, the better option is to use load-balancing and migrate clients to another server, so you can have an entire server to yourself for maintenance purposes.  Hope this is clear.

Posted by Blake Stanford on 20-May-2019 13:52

+1

SHELLEY CHASE WROTE THE FOLLOWING POST AT 5-19-2019 19:56:Hi Jeff,

If you can reproduce, please log a bug. 
It shouldn’t be this hard. 
Thanks
-Shelley

Posted by egarcia on 20-May-2019 15:32

Hello Jeff,

Here is more feedback.

I used  PASOE 11.7.4 (Linux) and I was able to use the Reload option in OEM to stop the server process associated with the ABL WebApp.

I noticed that it performs a REST request of "reset" using PUT.

I was able to run the same command via curl without issues.

ps -ef | fgrep ablapp2

curl -v -b /tmp/cookiefile -X PUT -d 'username=<username>&password=<password>' localhost:9090/login.jsp

curl -X PUT -b /tmp/cookiefile -d '{"reset":"true"}' localhost:9090/.../reset

ps -ef | fgrep ablapp2

Considering that you can reproduce the issue consistently, there must something else in the environment.

I hope this helps.

Posted by egarcia on 20-May-2019 15:38

Note:

First curl command should use "-c" to save the cookies in particular OEMSESSION_ID and use POST to perform FORM-based authentication:

curl -v -c /tmp/cookiefile -X POST -d 'username=admin&password=4admin' localhost:9090/login.jsp

The 2nd curl command uses -b to read the cookies.

curl -X PUT -b /tmp/cookiefile -d '{"reset":"true"}' localhost:9090/.../reset

Posted by Jeff Ledbetter on 20-May-2019 15:48

This is using OE 12. I've not tried using OE 11.7. But this is not a "Web App". This is the "ABL Application" tab on OEM.

I think we'll just go back to using multiple instances. It works and it's seems easier to manage.

Posted by Matt Baker on 20-May-2019 16:31

The OEM rest api does not require cookies.  It will use them for session management if present, but they are not required.  It falls back to HTTP basic if the cookie is not present and you provide the credentials on the command line.  You can use HTTP basic authentication by specifying the -u switch for curl.  You can simplify the example and skip the call to login.jsp and just call the reset service directly.

Posted by David Cleary on 20-May-2019 16:42

The log messages about memory leaks are just noise. Tomcat’s class loader has issues shutting down singletons and reports them that way. I can assure you the process holds on to no memory after termination.
 
As I’ve seen others suggest, trying the steps manually through tcman instead of PDSOE or OEM would be a big help in identifying which component is causing the problem.
 
Dave
 

Posted by Jeff Ledbetter on 20-May-2019 17:04

"The log messages about memory leaks are just noise."

The initial message I started this thread with was not in the log. It was appearing as a dialog in OEM.

We will try with tcman and see what the difference may be.

Posted by Jeff Ledbetter on 20-May-2019 19:04

For those following along, thank you for all the input. I think it is resolved.

I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

Anyhow.. this time the default was left and our other ABL Applications were added and everything seems to work swimmingly.  Even the 'Reload' button the 'ABL Applications' tab OEM does what is expected.

Posted by Irfan on 20-May-2019 19:05

[View:/cfs-file/__key/communityserver-discussions-components-files/26/stopAgent.zip:320:240]

Hi Jeff,

I have attached a script that basically terminates the agents for the given ABL Application. I have tested it on 12.0 and it works fine to me. It is very simple to run so it should not take a lot of time. Can you please give it a try and let us know if the functionality is working as expected ?

Here are the step you have to follow to execute this script

### Script to terminateagents for the given ABL Application ###

1. Copy all the scripts to the PASOE Instance /bin directory.
2. Run the below command
    stopAgent.bat <ABLAppname>
  Example:- stopAgent.bat oepas1
3. This should termiante the agents for the given ABL Application.
 

Posted by David Cleary on 20-May-2019 19:14

I recently discovered an issue while I was working on my NEXT workshop with deleting all the oeabl.war based applications removes a section from openedge.properties that cause future deployments to fail. Sounds like that was what you were running into.
 
Dave
 

Posted by Blake Stanford on 20-May-2019 19:38

>>>>I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

I attempted that also and ran into issues as well.  OEM didn't appear to like it, but I don't believe I tried to restart an oeable application.

Posted by Roger Blanchard on 20-May-2019 20:12

>I *think* the problem was that we were undeploying the 'default' ABL Application (or is it a Web Application??); the one that is named identically as the instance since it was not being used. I *think* that the docs stated that was okay to do but maybe not.

Jeff, we had all sorts of issues with using the same name. We gave up and just named the apps different.

Posted by Jeff Ledbetter on 21-May-2019 11:31

Thank you for the script Irfan. We will give it a try today and let you know how it works. I'm not sure that we will need it but it will be good to have on hand. :)

This thread is closed