Am I reading this right? The "statefree" appserver ("classic") consumes a thread for every remote connection, even when idle?!
Glance 11.00.044 09:13:04 abc ia64 Current Avg High
--------------------------------------------------------------------------------
CPU Util S SU U | 54% 43% 64%
Disk Util F F | 8% 7% 77%
Mem Util S SU UFF | 59% 59% 59%
Networkil U URR | 62% 61% 62%
--------------------------------------------------------------------------------
PROCESS LIST Users= 763
User CPU % Thrd Disk Memory Block
Process Name PID Name (1600% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
java 12345 root 5.4 901 0.0 828.6mb 3.49gb SLEEP
See above there are 901 threads sleeping. This is on HP-UX. There is also significant memory usage. Memory seems to grow by 1-2 MB per connected client. Memory use is high, although not all the memory is resident at once (in RSS).
Note that "statefree" client connections are typically long-running connections from an application that last the life of the client process. They don't disconnect until the user closes out of their applications. You can see the list of client connections with the asbman "-listclients" option. The clients aren't supposed to consume actual agent resources until they run a method.
I was aware that each connected client would consume some broker memory (swapped to disk most of the time) but I never imagined a dedicated OS thread for each one! Even if it is sleeping most of the time, this cannot be a good use of resources!
Let me know if I'm reading this wrong. I have been dabbling in "PASOE" and cannot imagine tomcat does anything like this (create a thread for every user who has an HTTP session).
We've been getting "out-of-memory" java errors in our appserver brokers and it caused me to start investigating. We allow those java broker processes a max heap of 2000mb (-Xmx parameter). But even that amount of memory seems to be insufficient now after the "-listclients" gets above 1000 or so. I think we are going to be forced to double the -Xmx. It seems like "classic" appserver is wasteful of both OS threads and heap memory.
Coincidentally, we just ran into big problems today related to the number of threads that "classic" appserver creates in HP-UX. Our statefree appserver reached 1200 threads and then everything started crashing. According to the kctune command, that appears to have been the limit.
There are at least a two related KB articles from Progress.
BROKER CRASHES WITH JAVA.LANG.OUTOFMEMORYERROR: UNABLE TO CREATE NEW NATIVE THREAD
https://knowledgebase.progress.com/articles/Article/P181418
HOW TO CONFIGURE MAX_THREAD_PROC AND AVOID JAVA OUT OF MEMORY ERROR ON HP-UX.
https://knowledgebase.progress.com/articles/Article/P26400
According to the KB’s, Progress seems to be aware of the fact that their "classic" appservers use too many threads. It seems to be an extremely bad design decision to hold onto one dedicated thread for every app/user connection. Instead of dedicating threads, most server processes I've worked with are supposed to be designed with thread “pools” that are shared as a common resource for multiple clients.
It may sound a bit negative but I seriously doubt that Progress is going to fix this. It's just another great incentive to get their customers buy their other PASOE product as well.
Wow, it looks like this KB answers my questions about "classic" appserver.
https://knowledgebase.progress.com/articles/Article/000029407
Here are the relevant parts:
clients that have connected to the broker.
I don't suppose its possible for Progress to revisit this design? It seems to scale up poorly and uses tons of unnecessary resources (not even counting all the redundant resources used by _proapsv's).
Coincidentally, we just ran into big problems today related to the number of threads that "classic" appserver creates in HP-UX. Our statefree appserver reached 1200 threads and then everything started crashing. According to the kctune command, that appears to have been the limit.
There are at least a two related KB articles from Progress.
BROKER CRASHES WITH JAVA.LANG.OUTOFMEMORYERROR: UNABLE TO CREATE NEW NATIVE THREAD
https://knowledgebase.progress.com/articles/Article/P181418
HOW TO CONFIGURE MAX_THREAD_PROC AND AVOID JAVA OUT OF MEMORY ERROR ON HP-UX.
https://knowledgebase.progress.com/articles/Article/P26400
According to the KB’s, Progress seems to be aware of the fact that their "classic" appservers use too many threads. It seems to be an extremely bad design decision to hold onto one dedicated thread for every app/user connection. Instead of dedicating threads, most server processes I've worked with are supposed to be designed with thread “pools” that are shared as a common resource for multiple clients.
It may sound a bit negative but I seriously doubt that Progress is going to fix this. It's just another great incentive to get their customers buy their other PASOE product as well.