Question about thread and memory resources consumed by "

Posted by dbeavon on 01-May-2018 08:30

Am I reading this right?  The "statefree" appserver ("classic") consumes a thread for every remote connection, even when idle?!

 Glance 11.00.044               09:13:04      abc     ia64    Current  Avg  High
--------------------------------------------------------------------------------
CPU  Util   S   SU                     U                       | 54%   43%   64%
Disk Util   F  F                                               |  8%    7%   77%
Mem  Util   S    SU                    UFF                     | 59%   59%   59%
Networkil   U                           URR                    | 62%   61%   62%
--------------------------------------------------------------------------------
                                  PROCESS LIST                      Users=  763
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   (1600% max) Cnt  IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
java               12345 root          5.4   901   0.0   828.6mb   3.49gb SLEEP

See above there are 901 threads sleeping.  This is on HP-UX.  There is also significant memory usage.  Memory seems to grow by 1-2 MB per connected client.  Memory use is high, although not all the memory is resident at once (in RSS).

Note that "statefree" client connections are typically long-running connections from an application that last the life of the client process.  They don't disconnect until the user closes out of their applications.  You can see the list of client connections with the asbman "-listclients" option.  The clients aren't supposed to consume actual agent resources until they run a method.

I was aware that each connected client would consume some broker memory (swapped to disk most of the time) but I never imagined a dedicated OS thread for each one!  Even if it is sleeping most of the time, this cannot be a good use of resources!

Let me know if I'm reading this wrong.  I have been dabbling in "PASOE" and cannot imagine tomcat does anything like this (create a thread for every user who has an HTTP session).  

We've been getting "out-of-memory" java errors in our appserver brokers and it caused me to start investigating.  We allow those java broker processes a max heap of 2000mb (-Xmx parameter).  But even that amount of memory seems to be insufficient now after the "-listclients" gets above 1000 or so.  I think we are going to be forced to double the -Xmx. It seems like "classic" appserver is wasteful of both OS threads and heap memory.

Posted by dbeavon on 03-May-2018 13:38

Coincidentally, we just ran into big problems today related to the number of threads that "classic" appserver creates in HP-UX.  Our statefree appserver reached 1200 threads and then everything started crashing.  According to the kctune command, that appears to have been the limit.

There are at least a two related KB articles from Progress.

 

BROKER CRASHES WITH JAVA.LANG.OUTOFMEMORYERROR: UNABLE TO CREATE NEW NATIVE THREAD

https://knowledgebase.progress.com/articles/Article/P181418

 

 

HOW TO CONFIGURE MAX_THREAD_PROC AND AVOID JAVA OUT OF MEMORY ERROR ON HP-UX.

https://knowledgebase.progress.com/articles/Article/P26400

 

 

According to the KB’s, Progress seems to be aware of the fact that their "classic" appservers use too many threads.  It seems to be an extremely bad design decision to hold onto one dedicated thread for every app/user connection.  Instead of dedicating threads, most server processes I've worked with are supposed to be designed with thread “pools” that are shared as a common resource for multiple clients.

It may sound a bit negative but I seriously doubt that Progress is going to fix this.  It's just another great incentive to get their customers buy their other PASOE product as well.  

All Replies

Posted by dbeavon on 02-May-2018 09:49

Wow, it looks like this KB answers my questions about "classic" appserver.

https://knowledgebase.progress.com/articles/Article/000029407

Here are the relevant parts:

  • The increase in threads is not related to the number of agents running, but rather to the number of

clients that have connected to the broker.

  • The threads in this pool are not trimmed.

I don't suppose its possible for Progress to revisit this design?  It seems to scale up poorly and uses tons of unnecessary resources (not even counting all the redundant resources used by _proapsv's).

Posted by dbeavon on 03-May-2018 13:38

Coincidentally, we just ran into big problems today related to the number of threads that "classic" appserver creates in HP-UX.  Our statefree appserver reached 1200 threads and then everything started crashing.  According to the kctune command, that appears to have been the limit.

There are at least a two related KB articles from Progress.

 

BROKER CRASHES WITH JAVA.LANG.OUTOFMEMORYERROR: UNABLE TO CREATE NEW NATIVE THREAD

https://knowledgebase.progress.com/articles/Article/P181418

 

 

HOW TO CONFIGURE MAX_THREAD_PROC AND AVOID JAVA OUT OF MEMORY ERROR ON HP-UX.

https://knowledgebase.progress.com/articles/Article/P26400

 

 

According to the KB’s, Progress seems to be aware of the fact that their "classic" appservers use too many threads.  It seems to be an extremely bad design decision to hold onto one dedicated thread for every app/user connection.  Instead of dedicating threads, most server processes I've worked with are supposed to be designed with thread “pools” that are shared as a common resource for multiple clients.

It may sound a bit negative but I seriously doubt that Progress is going to fix this.  It's just another great incentive to get their customers buy their other PASOE product as well.  

This thread is closed