Any configuration to maintain health of "generic adapte

Posted by dbeavon on 04-Feb-2020 00:49

I have a "brokerconnect" adapter running on the same server as PASOE. It receives requests to interact with a remote (activemq) messaging server.

This past week one of the "Srvr" entries got stuck at "RUNNING" and never transitioned back out of that state (to "AVAILABLE").  The related ABL/PASOE session had long-since been trimmed, and even the msagent process that contained that session had long-since been killed.  In fact the entire PASOE instance had already been restarted.

On the other side of the story is the activemq server.  When we noticed a "Srvr" stuck at "RUNNING" we proceeded to disconnect all the "openwire" connections on the activemq side and hoped that it would release the "Srvr" entry stuck at "RUNNING".  But disconnecting on the activemq server side had no effect on the "brokerconnect" adapter either.

So the end result is a brokerconnect "Srvr" that has lost both its connection to the ABL client session *and* the connection to the activemq server.  It was a bridge from nothing on the ABL side to nothing on the activemq side.  Yet it is still in the "RUNNING" state.  

I'm hoping someone can tell me if there is some configuration that would clean up after we reach this state.  Perhaps there is some health polling that I'm not using.  Ideally the health polling would release Srvr's from the "RUNNING" state after a period of time.  If PASOE is running locally on the same server as the "brokerconnect" adapter then you would think it could detect when the process ID of the msagent is no longer living, and do cleanup work afterwards (... there is no reason for the "Srvr" to remain "RUNNING" after the PASOE process that initiated the remote messaging had already been terminated).

Here is the output from -query on the adapter.

OpenEdge Release 11.7.5 as of Fri Jun  7 09:06:03 EDT 2019
Found 1 broker  (8323)
Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280)
Searching for sonicMQ1 (8288)
Connecting to sonicMQ1  (8276)
Broker Name                    : sonicMQ1
Operating Mode                 : Stateless
Broker Status                  :  ACTIVE 
Broker Port                    : 3620
Broker PID                     : 5916
Active Servers                 : 20
Busy Servers                   : 1
Locked Servers                 : 0
Available Servers              : 19
Active Clients (now, peak)     : (1, 7)
Client Queue Depth (cur, max)  : (0, 0)
Total Requests                 : 450017
Rq Wait (max, avg)             : (10 ms, 0 ms)
Rq Duration (max, avg)         : (463369 ms, 11 ms)
Srvr# State     Port  nRq    nRcvd  nSent  Started          Last Change      
00001 AVAILABLE 00000 023394 076550 023394 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00002 AVAILABLE 00000 023356 080302 023356 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00003 AVAILABLE 00000 023379 079009 023379 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00004 AVAILABLE 00000 023342 076895 023342 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00005 AVAILABLE 00000 023225 076138 023225 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00006 AVAILABLE 00000 023188 074913 023188 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00007 RUNNING   00000 006572 054214 006571 Jan 24, 2020 01:29 Jan 27, 2020 10:12 
00008 AVAILABLE 00000 023377 074687 023377 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00009 AVAILABLE 00000 023366 071522 023366 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00010 AVAILABLE 00000 023194 078796 023194 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00011 AVAILABLE 00000 023397 079035 023397 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00012 AVAILABLE 00000 023216 075580 023216 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00013 AVAILABLE 00000 023372 084099 023372 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00014 AVAILABLE 00000 023404 079532 023404 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00015 AVAILABLE 00000 023417 074098 023417 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00016 AVAILABLE 00000 023396 076725 023396 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00017 AVAILABLE 00000 023443 076152 023443 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00018 AVAILABLE 00000 023277 075429 023277 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00019 AVAILABLE 00000 023337 073635 023337 Jan 24, 2020 01:29 Feb 3, 2020 12:27 
00020 AVAILABLE 00000 023367 077338 023367 Jan 24, 2020 01:30 Feb 3, 2020 12:27 
*** Adapter status: : ***
Client list:
JMS client 172.31.22.241::sonicMQ1::3620::c3ec5f34704b81ee:-166e9de6:16fd63e27ff:-6c5
    Startup Parameters:
    jmsServerName: 
    Point-To-Point     
    brokerURL: tcp://ActiveMQProd.ufpi.com:61616?jms.prefetchPolicy.queuePrefetch=1&jms.redeliveryPolicy.maximumRedeliveries=-1&soTimeout=60000&soWriteTimeout=60000
    user: APP_SqlTrans
    password: xxx
    clientID: null
    pingInterval: 
    transactedPublish: true
    transactedReceive: false
    singleMessageAck: false
    symbiontAdapter: false
    jmsDomain: false
Queued Messages: 
End Queued Messages: 
Listeners: 
End Listeners: 
End client list 
 


Any help would be appreciated.  I already have a support case open with Progress support.  But it is quite different than this and I don't want to muddy the water.. This scenario is different in that the entire PASOE environment has been cycled, yet the "brokerconnect" adapter is still holding onto a Srvr at the "RUNNING"  status.

Unfortunately there is nothing in the -query output above that would allow me to do my own cleanup either.   It might be nice if the "Client list" allowed me to include some custom, user-supplied identification information, like the PASOE process ID.  Then I could do my own health check that detected the PASOE process, and killed/started the adapter after the process no longer exists.

All Replies

This thread is closed