Appserver stuck in "SENDING" state

Posted by Gareth Vincent on 22-Nov-2015 23:27

We are often experiencing problems on our Solaris server with OE 11.5.1 whereby agents are getting locked in "SENDING" state.  These agents often take up 5 - 10% of CPU utilization so we are constantly having to monitor and kill these runaway processes.  So far I've identified that this is typically caused by bad connections from the end users.  The users get disconnected yet the appserver remains in "SENDING" state.  

We have other customers running on Centos with the same version of OE and same source base yet they are not experiencing the same problem.  The difference is that these servers are hosted on premise so line issues are not really a factor as they are connecting over the LAN.  We have not ruled out coding issues but I would like to know if there is anything from a config point of view that could minimize or eliminate these "hanging" agents.

All Replies

Posted by Arun Kumar Mohapatra on 25-Nov-2015 09:56

If AppServer is running in Stateless mode then there might be, 2 possible reasons the agent hangs in SENDING state:

1) Client disconnects from broker without notifying or sending proper disconnect message to broker due to high latency in network  or any type of network interruption or client crashed/killed.

2) Programming fault may cause a lot more data to be returned than expected, via the broker, from the AppServer to the client (for example: Temp-tables or datasets that contain many more records than expected), that makes the agent hangs in SENDING state for long time.

If you are with the test environment then run AppServer with srvrLoggingLevel=3 in ubroker.properties allowed to see what procedures are running by the agents in the server log file by identifying the gent PID those are in SENDING state or we could use “pstack <agent_pid_in_sending_state>” to verify the system calls also.

What type of client are you using? And what is AppServer operating mode ? I think it is Stateless.

As you said after the user get disconnected appserver remains in SENDING state, if that is the case we need to verify the client log also to see it is sending proper disconnect request or not.

Will it be possible to provide all the logs for further analysis like client, broker and agent logs?

And I think there is no configuration that could minimize or eliminate these hangs, that needs analysis of log files, after that we could suggest something.

Posted by Gareth Vincent on 26-Nov-2015 04:55

Hi Arun,

Thanks for your response.  Our clients connect via Webclient  to a Stateless appserver.  We have opened a case with Progress.  Below are all the details we where able to gather.  

$ asbman -i rovic_p -q

OpenEdge Release 11.5.1 as of Wed May  6 19:00:27 EDT 2015

Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)

Searching for rovic_p (8288)

Connecting to rovic_p  (8276)

Broker Name                    : rovic_p

Operating Mode                 : Stateless

Broker Status                  :  ACTIVE

Broker Port                    : 17301

Broker PID                     : 23667

Active Servers                 : 4

Busy Servers                   : 1

Locked Servers                 : 0

Available Servers              : 3

Active Clients (now, peak)     : (6, 37)

Client Queue Depth (cur, max)  : (0, 6)

Total Requests                 : 34391

Rq Wait (max, avg)             : (22805 ms, 40 ms)

Rq Duration (max, avg)         : (9608243 ms, 865 ms)

PID   State     Port  nRq    nRcvd  nSent  Started          Last Change

24668 SENDING   17311 007359 009322 1271474725 Nov 23, 2015 22:06 Nov 25, 2015 07:09

24716 AVAILABLE 17314 011156 013693 022392 Nov 23, 2015 22:06 Nov 25, 2015 07:09

12153 AVAILABLE 17312 010351 012333 019824 Nov 24, 2015 08:23 Nov 25, 2015 07:08

25197 AVAILABLE 17313 000962 001067 001855 Nov 24, 2015 16:50 Nov 25, 2015 07:09

$ netstat -an | grep 17311

     *.17311              *.*                0      0 49152      0 LISTEN

127.0.0.1.36820      127.0.0.1.17311      49152      0 49152      0 ESTABLISHED

127.0.0.1.17311      127.0.0.1.36820      49152      0 49152      0 ESTABLISHED

$ asbman -i rovic_p -agentdetail 24668

OpenEdge Release 11.5.1 as of Wed May  6 19:00:27 EDT 2015

Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)

Searching for rovic_p (8288)

Connecting to rovic_p  (8276)

agent PID= 24668

agent port= 17311

connection state= SENDING

current procedure=  getfranchisedata.p at line -1

** Persistent procedures/Classes **

Handle  File Name

000000  /u8/rslive/rsserver/Legacy/getsyssequence.r

PROPATH= /u8/rslive/rsclient/rovic,/u8/rslive/rsserver/Rs/AutomateFramework,/u8/rslive/rsserver/Legacy/Rollback,/u8/rslive/rsserver/Rs,/u8/rslive/rsserver/Scl/SmartComponentLibrary,/u8/rslive/rsserver/Legacy,/u8/rslive/rsserver/Scl/SmartComponentLibrary/src,/u/progress/openedge/115/tty,/u/progress/openedge/115/tty/prodict.pl,/u/progress/openedge/115/tty/adecomp.pl,/u/progress/openedge/115/tty/OpenEdge.Core.pl,/u/progress/openedge/115/tty/dataadmin.pl,/u/progress/openedge/115/tty/adeshar.pl,/u/progress/openedge/115/tty/OpenEdge.BusinessLogic.pl,/u/progress/openedge/115/tty/adecomm.pl,/u/progress/openedge/115/tty/OpenEdge.ServerAdmin.pl,/u/progress/openedge/115/tty/adeedit.pl,/u/progress/openedge/115,/u/progress/openedge/115/bin

Stack Trace:

  --> getfranchisedata.p at line -1

Persistent procedures:

...........

Databases(logical/type/physical):

  dwp                 PROGRESS  /u8/rslive/database/dwp

  intdb               PROGRESS  /u8/EVOLVE/32bit/intdb

  commondb            PROGRESS  /u8/EVOLVE/rovic/commondb

  documentdb          PROGRESS  /u8/EVOLVE/rovic/documentdb

  dwplocal            PROGRESS  /u8/EVOLVE/rovic/dwplocal

  motordata           PROGRESS  /u8/EVOLVE/rovic/motordata

  retaildb            PROGRESS  /u8/EVOLVE/rovic/retaildb

  smartdb             PROGRESS  /u8/EVOLVE/rovic/smartdb

  temp-db             PROGRESS  /u8/EVOLVE/rovic/temp-db

$ kill -SIGUSR1 24668

PROGRESS stack trace as of Tue Nov 24 16:51:58 2015

Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23

Command line arguments are

/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf

Startup parameters:

-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)

(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv

(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv

(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv

(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv

(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1

(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1

(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1

(8) pcsample() +0x20 from: /lib/sparcv9/libc.so.1

(9) poll() +0x88 from: /lib/sparcv9/libc.so.1

(10) sock_specific_poll() +0x268 from: /u/progress/openedge/115/bin/_proapsv

(11) npp_specific_poll() +0x34 from: /u/progress/openedge/115/bin/_proapsv

(12) csnet_poll_two() +0x28c from: /u/progress/openedge/115/bin/_proapsv

(13) csd_poll() +0x38 from: /u/progress/openedge/115/bin/_proapsv

(14) main() +0x6d0 from: /u/progress/openedge/115/bin/_proapsv

(15) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv

** ABL Stack Trace **

--> getfranchisedata.p at line 0  (/u8/rslive/rsserver/Legacy/getfranchisedata.r)

** Persistent procedures/Classes **

..............

** PROPATH **

/u8/rslive/rsclient/rovic,/u8/rslive/rsserver/Rs/AutomateFramework,/u8/rslive/rsserver/Legacy/Rollback,/u8/rslive/rsserver/Rs,/u8/rslive/rsserver/Scl/SmartComponentLibrary,/u8/rslive/rsserver/Legacy,/u8/rslive/rsserver/Scl/SmartComponentLibrary/src,/u/progress/openedge/115/tty,/u/progress/openedge/115/tty/prodict.pl,/u/progress/openedge/115/tty/adecomp.pl,/u/progress/openedge/115/tty/OpenEdge.Core.pl,/u/progress/openedge/115/tty/dataadmin.pl,/u/progress/openedge/115/tty/adeshar.pl,/u/progress/openedge/115/tty/OpenEdge.BusinessLogic.pl,/u/progress/openedge/115/tty/adecomm.pl,/u/progress/openedge/115/tty/OpenEdge.ServerAdmin.pl,/u/progress/openedge/115/tty/adeedit.pl,/u/progress/openedge/115,/u/progress/openedge/115/bin

** Databases (logical/type/physical) **

  dwp                 PROGRESS  /u8/rslive/database/dwp

  intdb               PROGRESS  /u8/EVOLVE/32bit/intdb

  commondb            PROGRESS  /u8/EVOLVE/rovic/commondb

  documentdb          PROGRESS  /u8/EVOLVE/rovic/documentdb

  dwplocal            PROGRESS  /u8/EVOLVE/rovic/dwplocal

  motordata           PROGRESS  /u8/EVOLVE/rovic/motordata

  retaildb            PROGRESS  /u8/EVOLVE/rovic/retaildb

  smartdb             PROGRESS  /u8/EVOLVE/rovic/smartdb

  temp-db             PROGRESS  /u8/EVOLVE/rovic/temp-db

** End of Protrace **

$ pstack 24668

24668:  /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl

ffffffff7e1dce70 _write (8d, 108623b20, 20, 0, 104b74910, ec00) + c

00000001001b9660 sock_write_ipv4 (8d, 20, 108623b20, 0, ffffffff7fffe588, ffffffff7fffe584) + 258

00000001001f123c npp_send (8d, 20, 108623b20, 104b74910, ffffffff7fffe588, ffffffff7fffe584) + 34

00000001000fe94c csnet_send_msg (104b74730, ffffffff7fffed48, 0, 2fa9b518, 20, 108623b20) + 13c

00000001000efa9c ub_sendRsp (232c, 100d436ec, 0, 0, 2000, 0) + 874

00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0

00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c

00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794

00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c

$ prstat -v -p 24668

  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP

24668 root     7.9 8.1 0.0 0.0 0.0 0.0  84 0.0  79 149 .1M   0 _proapsv/1

Total: 2670 processes, 7857 lwps, load averages: 9.78, 18.12, 13.02

$ prstat -mL

  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID

23667 root      79  21 0.1 0.0 0.0 0.2 0.0 0.2  2K 882 .6M   0 java/37

20889 root      14  83 3.2 0.0 0.0 0.0 0.1 0.1 222   0  2K   0 _proapsv/1

23667 root      38  19 0.0 0.0 0.0  27 0.0  15 .1M 286 .1M   0 java/708

26476 root      23 3.5 1.9 0.0 0.0 0.0  71 0.0  89 102  6K   0 _proapsv/1

$ asbman -i rovic_p -agentstop 24668

OpenEdge Release 11.5.1 as of Wed May  6 19:00:27 EDT 2015

Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)

Searching for rovic_p (8288)

Connecting to rovic_p  (8276)

Shut down 24668 (8277)

$ asbman -i rovic_p -q

OpenEdge Release 11.5.1 as of Wed May  6 19:00:27 EDT 2015

Connecting to Progress AdminServer using rmi://localhost:20935/Chimera (8280)

Searching for rovic_p (8288)

Connecting to rovic_p  (8276)

Broker Name                    : rovic_p

Operating Mode                 : Stateless

Broker Status                  :  ACTIVE

Broker Port                    : 17301

Broker PID                     : 23667

Active Servers                 : 4

Busy Servers                   : 0

Locked Servers                 : 0

Available Servers              : 4

Active Clients (now, peak)     : (6, 37)

Client Queue Depth (cur, max)  : (0, 6)

Total Requests                 : 34645

Rq Wait (max, avg)             : (22805 ms, 40 ms)

Rq Duration (max, avg)         : (57975770 ms, 2536 ms)

PID   State     Port  nRq    nRcvd  nSent  Started          Last Change

24716 AVAILABLE 17314 011239 013795 022580 Nov 23, 2015 22:06 Nov 25, 2015 07:16

12153 AVAILABLE 17312 010435 012442 020030 Nov 24, 2015 08:23 Nov 25, 2015 07:16

25197 AVAILABLE 17313 001047 001160 002025 Nov 24, 2015 16:50 Nov 25, 2015 07:16

10718 AVAILABLE 17315 000002 000002 000002 Nov 25, 2015 07:16 Nov 25, 2015 07:16

/u8/rslive/temp/sim

$ ps -ef | grep 24668

   root 24668 23667   2   Nov 23 ?         181:49 /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/

   root 14907 21896   0 07:16:40 pts/40      0:00 grep 24668

$ netstat -an | grep 17311

     *.17311              *.*                0      0 49152      0 LISTEN

$ kill -SIGUSR1 24668

PROGRESS stack trace as of Wed Nov 25 07:26:10 2015

Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23

Command line arguments are

/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf

Startup parameters:

-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)

(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv

(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv

(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv

(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv

(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1

(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1

(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1

(8) ub_processRequest() +0x2ac from: /u/progress/openedge/115/bin/_proapsv

(9) csd_dispatch_message() +0x48c from: /u/progress/openedge/115/bin/_proapsv

(10) main() +0x794 from: /u/progress/openedge/115/bin/_proapsv

(11) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv

** ABL Stack Trace **

--> getfranchisedata.p at line -1

** Persistent procedures/Classes **

...............

** Databases (logical/type/physical) **

  dwp                 PROGRESS  /u8/rslive/database/dwp

  intdb               PROGRESS  /u8/EVOLVE/32bit/intdb

  commondb            PROGRESS  /u8/EVOLVE/rovic/commondb

  documentdb          PROGRESS  /u8/EVOLVE/rovic/documentdb

  dwplocal            PROGRESS  /u8/EVOLVE/rovic/dwplocal

  motordata           PROGRESS  /u8/EVOLVE/rovic/motordata

  retaildb            PROGRESS  /u8/EVOLVE/rovic/retaildb

  smartdb             PROGRESS  /u8/EVOLVE/rovic/smartdb

  temp-db             PROGRESS  /u8/EVOLVE/rovic/temp-db

** End of Protrace **

$ pstack 24668

24668:  /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl

00000001000ee944 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2cc

00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c

00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794

00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c

$ prstat -v -p 24668

  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP

24668 root     100 0.0 0.0 0.0 0.0 0.0 0.0 0.0   0  95   0   0 _proapsv/1

Total: 2670 processes, 7867 lwps, load averages: 46.36, 28.12, 20.95

$ ls /u8/rslive/logs/rovic_p.server.00001*

/u8/rslive/logs/rovic_p.server.000010.log  /u8/rslive/logs/rovic_p.server.000011.log  /u8/rslive/logs/rovic_p.server.000012.log

$ fuser /u8/rslive/logs/rovic_p.server.00001*

/u8/rslive/logs/rovic_p.server.000010.log:

/u8/rslive/logs/rovic_p.server.000011.log:    24668o

/u8/rslive/logs/rovic_p.server.000012.log:    10718o   25197o   12153o   24716o

User Control: by user number for all tenants

 Usr:Ten   Name      Domain     Type      Wait  Table:Part               Dbkey     Trans      PID Sem Srv Login  Time    IPV# Remote Address

   0       root          0     BROK       --        0                     6401         0     6750   0   0 11/23/15 22:01

   1       root          0     SERV       --        0                        0         0     6788   0   0 11/23/15 22:01

   2       root          0     SERV       --        0                        0         0    20630   1   0 11/23/15 22:17

   3       root          0     SERV       --        0                        0         0     2004   1   0 11/23/15 22:36

  10       root          0     APW        --        0                     3316         0     6826   5   0 11/23/15 22:01

  11       root          0     BIW        --        0                     3614         0     6825   5   0 11/23/15 22:01

  12       root          0     APW        --        0                     3315         0     6835   6   0 11/23/15 22:01

  13       root         -4     SELF/APSV  --        0                        0         0    10718   6   0 11/25/15 07:16

  14       root         -4     SELF/APSV  --        7                       64         0    24668   7   0 11/23/15 22:06

  15       root         -4     SELF/APSV  --        0                        0         0    25197   7   0 11/24/15 16:50

  16       root         -4     SELF/APSV  --      102                    56222         0    24716   8   0 11/23/15 22:06

  17       root         -4     SELF/APSV  --       17                  4848704         0    12153   8   0 11/24/15 08:23

  18       root          0     MON        --        0                        0         0    24542   9   0 11/25/15 07:23

  39       SYSTEM       -3     REMC/APSV  --        0                        0         0        0  19   2 11/23/15 22:17 IPV4 10.1.4.213

RETURN - repeat, U - continue uninterrupted, Q - quit:

$ kill 24668

/u8/rslive/temp/sim

$ ps -ef | grep 24668

   root 24668 23667   3   Nov 23 ?         201:15 /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/

   root 21031 21896   0 07:36:22 pts/40      0:00 grep 24668

/u8/rslive/temp/sim

$ pstack 24668

24668:  /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl

ffffffff7d8065b8 getLogCtxFromID (0, 4, 8, 3, ffffffff7d8054b8, 100d47220) + b8

00000001000ec58c csoTraceMessage (20d3, 0, 6028, c54e6c, 20d3, 0) + 3c

00000001000ee1a4 open4GLRead (100dfa668, 2000, 0, 0, b9000, 100d413c0) + 17c

00000001000ef4cc ub_sendRsp (ffffffff7fffed48, 104b74730, 6c, 200, ffffffff7fffe930, 0) + 2a4

00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0

00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c

00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794

00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c

$ grep 24668 ../../logs/rovic_p.server.000011.log

[15/11/24@15:08:19.935+0200] P-024668 T-000001 1 AS -- (Procedure: 'loaddocimages.p' Line:198) ** "c:\temp\10RLInvoice05.htm" was not found. (293)

[15/11/25@07:35:53.572+0200] P-024668 T-000001 1 AS -- KILL signal received. (298)

$ prstat -v -p 24668

  PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP

24668 root     100 0.0 0.0 0.0 0.0 0.0 0.0 0.0   0  82   0   0 _proapsv/1

$ pstack 24668

24668:  /u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rsl

0000000100137078 logMsgLogIt (20d3, 0, 6028, c54e6c, 20d3, 0) + 38

00000001000ee1a4 open4GLRead (100dfa668, 2000, 0, 0, b9000, 100d413c0) + 17c

00000001000ef4cc ub_sendRsp (ffffffff7fffed48, 104b74730, 6c, 200, ffffffff7fffe930, 0) + 2a4

00000001000ee918 ub_processRequest (104b74730, ffffffff7fffed48, 100f737f0, 0, 0, 14) + 2a0

00000001000f7a7c csd_dispatch_message (100d43ce8, 46, ffffffff7fffed48, 0, 100d413c0, 6c00) + 48c

00000001000f50a4 main (0, 0, 0, ffffe4a6, 1, 100d413c0) + 794

00000001000e0d7c _start (0, 0, 0, 0, 0, 0) + 17c

$ kill -SIGUSR1 24668

PROGRESS stack trace as of Wed Nov 25 07:42:14 2015

Progress OpenEdge Release 11.5 build 1190 SP01 on SunOS Cronus 5.10 Generic_150400-23

Command line arguments are

/u/progress/openedge/115/bin/_proapsv -logginglevel 2 -logfile /u8/rslive/logs/rovic_p.server.000011.log -ubpid 23667 -Ms 1 -logname rovic_p -logentrytypes ASPlumbing,DB.Connects -logthreshold 500000 -numlogfiles 3 -ASID 1 -ubpropfile /u/progress/openedge/115/properties/ubroker.properties -ipver IPv4 -pf /u8/rslive/rsclient/rovic/appserver.pf

Startup parameters:

-pf /u/progress/openedge/115/startup.pf,-cpinternal ISO8859-1,-cpstream ISO8859-1,-cpcoll Basic,-cpcase Basic,-d dmy,-numsep 44,-numdec 46,(end .pf),-logginglevel 2,-logfile /u8/rslive/logs/rovic_p.server.000011.log,-ubpid 23667,-Ms 1,-logname rovic_p,-logentrytypes ASPlumbing,DB.Connects,-logthreshold 500000,-numlogfiles 3,-ASID 1,-ubpropfile /u/progress/openedge/115/properties/ubroker.properties,-ipver IPv4,-pf /u8/rslive/rsclient/rovic/appserver.pf,-pf /u8/rslive/rsclient/globalappserver.pf,-rereadnolock,-noincrwarn,-mmax 65534,-D 500,-s 128,-h 12,-l 1000,-lkwtmo 120,-yr4def,-q,-T /u9/rstmp,-NL,(end .pf),-db /u8/rslive/database/dwp,-db /u8/EVOLVE/32bit/intdb,-db /u8/EVOLVE/rovic/commondb,-db /u8/EVOLVE/rovic/documentdb,-db /u8/EVOLVE/rovic/dwplocal,-db /u8/EVOLVE/rovic/motordata,-db /u8/EVOLVE/rovic/retaildb,-db /u8/EVOLVE/rovic/smartdb,-db /u8/EVOLVE/rovic/temp-db,(end .pf)

(1) uttrace() +0x1c4 from: /u/progress/openedge/115/bin/_proapsv

(2) drsigs() +0x780 from: /u/progress/openedge/115/bin/_proapsv

(3) drExitOnTerm() +0x3e0 from: /u/progress/openedge/115/bin/_proapsv

(4) drExitOnTerm() +0x650 from: /u/progress/openedge/115/bin/_proapsv

(5) _thr_slot_offset() +0x5b4 from: /lib/sparcv9/libc.so.1

(6) sema_post() +0x628 from: /lib/sparcv9/libc.so.1

(7) sema_post() +0x834 from: /lib/sparcv9/libc.so.1

(8) ub_processRequest() +0x27c from: /u/progress/openedge/115/bin/_proapsv

(9) csd_dispatch_message() +0x48c from: /u/progress/openedge/115/bin/_proapsv

(10) main() +0x794 from: /u/progress/openedge/115/bin/_proapsv

(11) _start() +0x17c from: /u/progress/openedge/115/bin/_proapsv

** ABL Stack Trace **

--> getfranchisedata.p at line -1

** Persistent procedures/Classes **

..............

** Databases (logical/type/physical) **

** End of Protrace **

$ kill -9 24668

$ ps -ef | grep 24668

   root 10560 21896   0 07:44:31 pts/40      0:00 grep 24668

$ prstat -v

Total: 2668 processes, 7857 lwps, load averages: 11.35, 14.93, 18.06

Posted by bremmeyr on 01-Dec-2015 10:33

What is an indicator that can be monitored to call attention to consistent send state? Above is mention if a client does not properly disconnect appserver will continue in a sending state. Does OpenEdge watch for this and preform house keeping to clean up the disconnected session? Or, do we need to monitor for this condition?

This thread is closed