We are often experiencing problems on our Solaris server with OE 11.5.1 whereby agents are getting locked in "SENDING" state. These agents often take up 5 - 10% of CPU utilization so we are constantly having to monitor and kill these runaway processes. So far I've identified that this is typically caused by bad connections from the end users. The users get disconnected yet the appserver remains in "SENDING" state.
We have other customers running on Centos with the same version of OE and same source base yet they are not experiencing the same problem. The difference is that these servers are hosted on premise so line issues are not really a factor as they are connecting over the LAN. We have not ruled out coding issues but I would like to know if there is anything from a config point of view that could minimize or eliminate these "hanging" agents.
If AppServer is running in Stateless mode then there might be, 2 possible reasons the agent hangs in SENDING state:
1) Client disconnects from broker without notifying or sending proper disconnect message to broker due to high latency in network or any type of network interruption or client crashed/killed.
2) Programming fault may cause a lot more data to be returned than expected, via the broker, from the AppServer to the client (for example: Temp-tables or datasets that contain many more records than expected), that makes the agent hangs in SENDING state for long time.
If you are with the test environment then run AppServer with srvrLoggingLevel=3 in ubroker.properties allowed to see what procedures are running by the agents in the server log file by identifying the gent PID those are in SENDING state or we could use “pstack <agent_pid_in_sending_state>” to verify the system calls also.
What type of client are you using? And what is AppServer operating mode ? I think it is Stateless.
As you said after the user get disconnected appserver remains in SENDING state, if that is the case we need to verify the client log also to see it is sending proper disconnect request or not.
Will it be possible to provide all the logs for further analysis like client, broker and agent logs?
And I think there is no configuration that could minimize or eliminate these hangs, that needs analysis of log files, after that we could suggest something.
Hi Arun,
Thanks for your response. Our clients connect via Webclient to a Stateless appserver. We have opened a case with Progress. Below are all the details we where able to gather.
$ asbman -i rovic_p -q
OpenEdge Release 11.5.1 as of Wed May 6 19:00:27 EDT 2015
Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)
Searching for rovic_p (8288)
Connecting to rovic_p (8276)
Broker Name : rovic_p
Operating Mode : Stateless
Broker Status : ACTIVE
Broker Port : 17301
Broker PID : 23667
Active Servers : 4
Busy Servers : 1
Locked Servers : 0
Available Servers : 3
Active Clients (now, peak) : (6, 37)
Client Queue Depth (cur, max) : (0, 6)
Total Requests : 34391
Rq Wait (max, avg) : (22805 ms, 40 ms)
Rq Duration (max, avg) : (9608243 ms, 865 ms)
PID State Port nRq nRcvd nSent Started Last Change
24668 SENDING 17311 007359 009322 1271474725 Nov 23, 2015 22:06 Nov 25, 2015 07:09
24716 AVAILABLE 17314 011156 013693 022392 Nov 23, 2015 22:06 Nov 25, 2015 07:09
12153 AVAILABLE 17312 010351 012333 019824 Nov 24, 2015 08:23 Nov 25, 2015 07:08
25197 AVAILABLE 17313 000962 001067 001855 Nov 24, 2015 16:50 Nov 25, 2015 07:09
$ netstat -an | grep 17311
*.17311 *.* 0 0 49152 0 LISTEN
127.0.0.1.36820 127.0.0.1.17311 49152 0 49152 0 ESTABLISHED
127.0.0.1.17311 127.0.0.1.36820 49152 0 49152 0 ESTABLISHED
$ asbman -i rovic_p -agentdetail 24668
OpenEdge Release 11.5.1 as of Wed May 6 19:00:27 EDT 2015
Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)
Searching for rovic_p (8288)
Connecting to rovic_p (8276)
agent PID= 24668
agent port= 17311
connection state= SENDING
current procedure= getfranchisedata.p at line -1
** Persistent procedures/Classes **
Handle File Name
000000 /u8/rslive/rsserver/Legacy/getsyssequence.r
PROPATH= /u8/rslive/rsclient/rovic,/u8/rslive/rsserver/Rs/AutomateFramework,/u8/rslive/rsserver/Legacy/Rollback,/u8/rslive/rsserver/Rs,/u8/rslive/rsserver/Scl/SmartComponentLibrary,/u8/rslive/rsserver/Legacy,/u8/rslive/rsserver/Scl/SmartComponentLibrary/src,/u/progress/openedge/115/tty,/u/progress/openedge/115/tty/prodict.pl,/u/progress/openedge/115/tty/adecomp.pl,/u/progress/openedge/115/tty/OpenEdge.Core.pl,/u/progress/openedge/115/tty/dataadmin.pl,/u/progress/openedge/115/tty/adeshar.pl,/u/progress/openedge/115/tty/OpenEdge.BusinessLogic.pl,/u/progress/openedge/115/tty/adecomm.pl,/u/progress/openedge/115/tty/OpenEdge.ServerAdmin.pl,/u/progress/openedge/115/tty/adeedit.pl,/u/progress/openedge/115,/u/progress/openedge/115/bin
Stack Trace:
--> getfranchisedata.p at line -1
Persistent procedures:
...........
Databases(logical/type/physical):
dwp PROGRESS /u8/rslive/database/dwp
intdb PROGRESS /u8/EVOLVE/32bit/intdb
commondb PROGRESS /u8/EVOLVE/rovic/commondb
documentdb PROGRESS /u8/EVOLVE/rovic/documentdb
dwplocal PROGRESS /u8/EVOLVE/rovic/dwplocal
motordata PROGRESS /u8/EVOLVE/rovic/motordata
retaildb PROGRESS /u8/EVOLVE/rovic/retaildb
smartdb PROGRESS /u8/EVOLVE/rovic/smartdb
temp-db PROGRESS /u8/EVOLVE/rovic/temp-db
$ kill -SIGUSR1 24668
PROGRESS stack trace as of Tue Nov 24 16:51:58 2015
Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23
Command line arguments are
/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf
Startup parameters:
-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)
(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv
(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv
(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv
(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv
(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1
(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1
(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1
(8) pcsample() +0x20 from: /lib/sparcv9/libc.so.1
(9) poll() +0x88 from: /lib/sparcv9/libc.so.1
(10) sock_specific_poll() +0x268 from: /u/progress/openedge/115/bin/_proapsv
(11) npp_specific_poll() +0x34 from: /u/progress/openedge/115/bin/_proapsv
(12) csnet_poll_two() +0x28c from: /u/progress/openedge/115/bin/_proapsv
(13) csd_poll() +0x38 from: /u/progress/openedge/115/bin/_proapsv
(14) main() +0x6d0 from: /u/progress/openedge/115/bin/_proapsv
(15) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv
** ABL Stack Trace **
--> getfranchisedata.p at line 0 (/u8/rslive/rsserver/Legacy/getfranchisedata.r)
** Persistent procedures/Classes **
..............
** PROPATH **
/u8/rslive/rsclient/rovic,/u8/rslive/rsserver/Rs/AutomateFramework,/u8/rslive/rsserver/Legacy/Rollback,/u8/rslive/rsserver/Rs,/u8/rslive/rsserver/Scl/SmartComponentLibrary,/u8/rslive/rsserver/Legacy,/u8/rslive/rsserver/Scl/SmartComponentLibrary/src,/u/progress/openedge/115/tty,/u/progress/openedge/115/tty/prodict.pl,/u/progress/openedge/115/tty/adecomp.pl,/u/progress/openedge/115/tty/OpenEdge.Core.pl,/u/progress/openedge/115/tty/dataadmin.pl,/u/progress/openedge/115/tty/adeshar.pl,/u/progress/openedge/115/tty/OpenEdge.BusinessLogic.pl,/u/progress/openedge/115/tty/adecomm.pl,/u/progress/openedge/115/tty/OpenEdge.ServerAdmin.pl,/u/progress/openedge/115/tty/adeedit.pl,/u/progress/openedge/115,/u/progress/openedge/115/bin
** Databases (logical/type/physical) **
dwp PROGRESS /u8/rslive/database/dwp
intdb PROGRESS /u8/EVOLVE/32bit/intdb
commondb PROGRESS /u8/EVOLVE/rovic/commondb
documentdb PROGRESS /u8/EVOLVE/rovic/documentdb
dwplocal PROGRESS /u8/EVOLVE/rovic/dwplocal
motordata PROGRESS /u8/EVOLVE/rovic/motordata
retaildb PROGRESS /u8/EVOLVE/rovic/retaildb
smartdb PROGRESS /u8/EVOLVE/rovic/smartdb
temp-db PROGRESS /u8/EVOLVE/rovic/temp-db
** End of Protrace **
$ pstack 24668
24668: /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl
ffffffff7e1dce70 _write (8d, 108623b20, 20, 0, 104b74910, ec00) + c
00000001001b9660 sock_write_ipv4 (8d, 20, 108623b20, 0, ffffffff7fffe588, ffffffff7fffe584) + 258
00000001001f123c npp_send (8d, 20, 108623b20, 104b74910, ffffffff7fffe588, ffffffff7fffe584) + 34
00000001000fe94c csnet_send_msg (104b74730, ffffffff7fffed48, 0, 2fa9b518, 20, 108623b20) + 13c
00000001000efa9c ub_sendRsp (232c, 100d436ec, 0, 0, 2000, 0) + 874
00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0
00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c
00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794
00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c
$ prstat -v -p 24668
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
24668 root 7.9 8.1 0.0 0.0 0.0 0.0 84 0.0 79 149 .1M 0 _proapsv/1
Total: 2670 processes, 7857 lwps, load averages: 9.78, 18.12, 13.02
$ prstat -mL
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
23667 root 79 21 0.1 0.0 0.0 0.2 0.0 0.2 2K 882 .6M 0 java/37
20889 root 14 83 3.2 0.0 0.0 0.0 0.1 0.1 222 0 2K 0 _proapsv/1
23667 root 38 19 0.0 0.0 0.0 27 0.0 15 .1M 286 .1M 0 java/708
26476 root 23 3.5 1.9 0.0 0.0 0.0 71 0.0 89 102 6K 0 _proapsv/1
$ asbman -i rovic_p -agentstop 24668
OpenEdge Release 11.5.1 as of Wed May 6 19:00:27 EDT 2015
Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)
Searching for rovic_p (8288)
Connecting to rovic_p (8276)
Shut down 24668 (8277)
$ asbman -i rovic_p -q
OpenEdge Release 11.5.1 as of Wed May 6 19:00:27 EDT 2015
Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)
Searching for rovic_p (8288)
Connecting to rovic_p (8276)
Broker Name : rovic_p
Operating Mode : Stateless
Broker Status : ACTIVE
Broker Port : 17301
Broker PID : 23667
Active Servers : 4
Busy Servers : 0
Locked Servers : 0
Available Servers : 4
Active Clients (now, peak) : (6, 37)
Client Queue Depth (cur, max) : (0, 6)
Total Requests : 34645
Rq Wait (max, avg) : (22805 ms, 40 ms)
Rq Duration (max, avg) : (57975770 ms, 2536 ms)
PID State Port nRq nRcvd nSent Started Last Change
24716 AVAILABLE 17314 011239 013795 022580 Nov 23, 2015 22:06 Nov 25, 2015 07:16
12153 AVAILABLE 17312 010435 012442 020030 Nov 24, 2015 08:23 Nov 25, 2015 07:16
25197 AVAILABLE 17313 001047 001160 002025 Nov 24, 2015 16:50 Nov 25, 2015 07:16
10718 AVAILABLE 17315 000002 000002 000002 Nov 25, 2015 07:16 Nov 25, 2015 07:16
/u8/rslive/temp/sim
$ ps -ef | grep 24668
root 24668 23667 2 Nov 23 ? 181:49 /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/
root 14907 21896 0 07:16:40 pts/40 0:00 grep 24668
$ netstat -an | grep 17311
*.17311 *.* 0 0 49152 0 LISTEN
$ kill -SIGUSR1 24668
PROGRESS stack trace as of Wed Nov 25 07:26:10 2015
Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23
Command line arguments are
/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf
Startup parameters:
-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)
(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv
(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv
(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv
(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv
(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1
(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1
(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1
(8) ub_processRequest() +0x2ac from: /u/progress/openedge/115/bin/_proapsv
(9) csd_dispatch_message() +0x48c from: /u/progress/openedge/115/bin/_proapsv
(10) main() +0x794 from: /u/progress/openedge/115/bin/_proapsv
(11) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv
** ABL Stack Trace **
--> getfranchisedata.p at line -1
** Persistent procedures/Classes **
...............
** Databases (logical/type/physical) **
dwp PROGRESS /u8/rslive/database/dwp
intdb PROGRESS /u8/EVOLVE/32bit/intdb
commondb PROGRESS /u8/EVOLVE/rovic/commondb
documentdb PROGRESS /u8/EVOLVE/rovic/documentdb
dwplocal PROGRESS /u8/EVOLVE/rovic/dwplocal
motordata PROGRESS /u8/EVOLVE/rovic/motordata
retaildb PROGRESS /u8/EVOLVE/rovic/retaildb
smartdb PROGRESS /u8/EVOLVE/rovic/smartdb
temp-db PROGRESS /u8/EVOLVE/rovic/temp-db
** End of Protrace **
$ pstack 24668
24668: /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl
00000001000ee944 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2cc
00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c
00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794
00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c
$ prstat -v -p 24668
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
24668 root 100 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 95 0 0 _proapsv/1
Total: 2670 processes, 7867 lwps, load averages: 46.36, 28.12, 20.95
$ ls /u8/rslive/logs/rovic_p.server.00001*
/u8/rslive/logs/rovic_p.server.000010.log /u8/rslive/logs/rovic_p.server.000011.log /u8/rslive/logs/rovic_p.server.000012.log
$ fuser /u8/rslive/logs/rovic_p.server.00001*
/u8/rslive/logs/rovic_p.server.000010.log:
/u8/rslive/logs/rovic_p.server.000011.log: 24668o
/u8/rslive/logs/rovic_p.server.000012.log: 10718o 25197o 12153o 24716o
User Control: by user number for all tenants
Usr:Ten Name Domain Type Wait Table:Part Dbkey Trans PID Sem Srv Login Time IPV# Remote Address
0 root 0 BROK -- 0 6401 0 6750 0 0 11/23/15 22:01
1 root 0 SERV -- 0 0 0 6788 0 0 11/23/15 22:01
2 root 0 SERV -- 0 0 0 20630 1 0 11/23/15 22:17
3 root 0 SERV -- 0 0 0 2004 1 0 11/23/15 22:36
10 root 0 APW -- 0 3316 0 6826 5 0 11/23/15 22:01
11 root 0 BIW -- 0 3614 0 6825 5 0 11/23/15 22:01
12 root 0 APW -- 0 3315 0 6835 6 0 11/23/15 22:01
13 root -4 SELF/APSV -- 0 0 0 10718 6 0 11/25/15 07:16
14 root -4 SELF/APSV -- 7 64 0 24668 7 0 11/23/15 22:06
15 root -4 SELF/APSV -- 0 0 0 25197 7 0 11/24/15 16:50
16 root -4 SELF/APSV -- 102 56222 0 24716 8 0 11/23/15 22:06
17 root -4 SELF/APSV -- 17 4848704 0 12153 8 0 11/24/15 08:23
18 root 0 MON -- 0 0 0 24542 9 0 11/25/15 07:23
39 SYSTEM -3 REMC/APSV -- 0 0 0 0 19 2 11/23/15 22:17 IPV4 10.1.4.213
RETURN - repeat, U - continue uninterrupted, Q - quit:
$ kill 24668
/u8/rslive/temp/sim
$ ps -ef | grep 24668
root 24668 23667 3 Nov 23 ? 201:15 /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/
root 21031 21896 0 07:36:22 pts/40 0:00 grep 24668
/u8/rslive/temp/sim
$ pstack 24668
24668: /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl
ffffffff7d8065b8 getLogCtxFromID (0, 4, 8, 3, ffffffff7d8054b8, 100d47220) + b8
00000001000ec58c csoTraceMessage (20d3, 0, 6028, c54e6c, 20d3, 0) + 3c
00000001000ee1a4 open4GLRead (100dfa668, 2000, 0, 0, b9000, 100d413c0) + 17c
00000001000ef4cc ub_sendRsp (ffffffff7fffed48, 104b74730, 6c, 200, ffffffff7fffe930, 0) + 2a4
00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0
00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c
00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794
00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c
$ grep 24668 ../../logs/rovic_p.server.000011.log
[15/11/24@15:08:19.935+0200] P-024668 T-000001 1 AS -- (Procedure: 'loaddocimages.p' Line:198) ** "c:\temp\10RLInvoice05.htm" was not found. (293)
[15/11/25@07:35:53.572+0200] P-024668 T-000001 1 AS -- KILL signal received. (298)
$ prstat -v -p 24668
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
24668 root 100 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 82 0 0 _proapsv/1
$ pstack 24668
24668: /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl
0000000100137078 logMsgLogIt (20d3, 0, 6028, c54e6c, 20d3, 0) + 38
00000001000ee1a4 open4GLRead (100dfa668, 2000, 0, 0, b9000, 100d413c0) + 17c
00000001000ef4cc ub_sendRsp (ffffffff7fffed48, 104b74730, 6c, 200, ffffffff7fffe930, 0) + 2a4
00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0
00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c
00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794
00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c
$ kill -SIGUSR1 24668
PROGRESS stack trace as of Wed Nov 25 07:42:14 2015
Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23
Command line arguments are
/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf
Startup parameters:
-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)
(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv
(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv
(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv
(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv
(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1
(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1
(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1
(8) ub_processRequest() +0x27c from: /u/progress/openedge/115/bin/_proapsv
(9) csd_dispatch_message() +0x48c from: /u/progress/openedge/115/bin/_proapsv
(10) main() +0x794 from: /u/progress/openedge/115/bin/_proapsv
(11) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv
** ABL Stack Trace **
--> getfranchisedata.p at line -1
** Persistent procedures/Classes **
..............
** Databases (logical/type/physical) **
** End of Protrace **
$ kill -9 24668
$ ps -ef | grep 24668
root 10560 21896 0 07:44:31 pts/40 0:00 grep 24668
$ prstat -v
Total: 2668 processes, 7857 lwps, load averages: 11.35, 14.93, 18.06
What is an indicator that can be monitored to call attention to consistent send state? Above is mention if a client does not properly disconnect appserver will continue in a sending state. Does OpenEdge watch for this and preform house keeping to clean up the disconnected session? Or, do we need to monitor for this condition?