Wrongly cross-posted in development thread
Greetings
Working at a clients whose system (Solaris / 10.2A02 - yes, I know) looks like this:
dot net front end -- external appserver (state-free) -- internal appserver (stateless) -- database
Everything was working (quite) well until a week ago when we noticed that asbman -q -i internal was reporting Active Clients as 511/now and 512/peak and eventually nothing would connect to the database.
Restarted the internal appserver and all became calm again. We have noticed though that on a daily basis (we are currently restarting the internal nightly) Active climbs to between 20+ to 40+. Of the 35 available servers only a few are ever busy sending. As a perfect example, when we saw the 511/512 figures, all 35 servers were running "Available".
We started to debug the situation by running asbman -listclients, seeing that in this case it was conn handle 748 and feeding that in turn to "asbman -q -i internal -clientdetail 748". This gave us the agent pid which we used to send a SIGUSR2 to the process. Generated a protrace which had an empty ABL stack 
We then grabbed the connection id and started to search through the broker and server logs.
A relevant few lines are these:
[15/05/25@10:01:31.234+1000] P-013643 T-000001 2 AS AS Application Server connected with connection id: 10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-7e7. (8358)
[15/05/25@10:01:31.234+1000] P-013643 T-000001 3 AS AS ASK Protocol is disabled.
[15/05/25@10:01:31.234+1000] P-013643 T-000001 3 AS AS CSS_CONN before ACK, connection=10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-7e7 bound=0 changed=0
[15/05/25@10:01:31.236+1000] P-013810 T-000001 3 AS AS requestID= 0a356d900000023199143204602200003521580000021885
[15/05/25@10:01:31.236+1000] P-013810 T-000001 2 AS AS -- TRACE: Non-PERSISTENT Procedure 'eris/internal/ReceiptPayment.p' START. (5498)
[15/05/25@10:01:31.245+1000] P-013810 T-000001 2 AS 4GLTRACE Run CheckDiary in me/meperprc.p [Main Block - eris/internal/ReceiptPayment.p @ 775]
[15/05/25@10:01:31.245+1000] P-013810 T-000001 3 AS 4GLTRACE Return from CheckDiary [me/meperprc.p]
What I can't understand is that given the connection id I'm following is the one that ends with -7e7", it seems to disappear almost immediately.
I'm just about at my limit of appserver debugging - can anyone make any suggestions as to where I should go from here or even better point me at some half-decent documentation that might help?
TIA
N/
Added to this, can anyone advise what the following lines mean? Repeated.
[15/05/25@10:03:26.015+1000] P-013810 T-000001 2 AS AS Application Server connected with connection id: 10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-647. (8358)
[15/05/25@10:03:26.015+1000] P-013810 T-000001 3 AS AS ASK Protocol is disabled.
[15/05/25@10:03:26.015+1000] P-013810 T-000001 3 AS AS CSS_CONN before ACK, connection=10.53.109.173::ASLiveMRM::3107::6fe9ca3f6757ba30:3e9e189d:14d863c9654:-647 bound=0 changed=0
TIA
Nigel.
Nigel,
Connections to the broker (your max 512 in ubroker.properties) are not correlated with use of the agents. I don't know what the issue is that you saw with conn handle 748 but it doesn't sound like that's your problem.
(For the lurkers, clients of stateless/state-free AppServers never actually speak directly to _proapsv agents. They only connect to the java broker and the broker passes along their request to an available agent).
Processes are connecting to the broker and never disconnecting. lsof -P -p <java broker pid> | grep TCP will show you all the connections. You might see that some dotNet is always connecting and re-connecting and never disconnecting. That would be my first guess.
Thanks for the input mon vieux - turned out that the problem was a few program that were not compiled on the internal appserver. STOP was being raised every time. It's a monster of a system but this specific event only happens 40 - 50 times a day - made it very hard to spot. So busy that the appserver logs (between 10 and 20MB) are being rotated every minute or even more frequently. Keeps me humble.
I am not sure it's exactly right to say that clients (if that's what you are saying) never disconnect from the Broker in a stateless model. They do disconnect from the broker but the broker doesn't necessarily disconnect from the agents. If there is processing to be done on the disconnect of stateless connection (i.e. the disconnect procedure) this would happen when the client disconnects, you just wouldn't be able to guarantee which agent that procedure would be run on.
I am wondering about the effect of having State-free on the front-end connecting to the Stateless on the backend (or middle end). It sort of seems illogical to me (without thinking really indepthly about it). If you are letting the front end connect to the front side AppServer all loosey goosey (i.e. any of the agents on each susquent request, i.e. statefree) who is responsible for that agent making sure the connection to the stateless server cleans up after itself? Why wouldn't both AppServers be state-free? Wouldn't you need to maintain the same sort of state at the statefree appserver as you do the stateless server? (again not fully thinking it through).
Are they doing disconnects from the statefree appserver? It would be interesting to do some dynamic object logging on the statefree appserver to see if its cleaning up its connections to the stateless server or if they assume that everything is gone after the clients request goes away that that would clean up the appserver connection on statefree appserver. Without testing, I don't think that would be the case. More likely the stateless connection would be left around waiting for its date to get back from the bathroom, while little does it know its got a new date happening with another stateless appserver down the road, so to speak.
Just a thought. Of course I could be completely wrong. There is a first time for everything. ;-)
Good luck.
Hi Molly
I'm told that the reason for the state-free vs stateless was that the front end needed to be state-free to deal with the app that uses a web service to talk to the front-end appserver and programs (progress) that talk to the back end appserver and programs. Problem, like a lot of systems, is that it's too entrenched and complex to re-architecture at the point of time.
N/
Hi.
"I am not sure it's exactly right to say that clients never disconnect from the Broker in a stateless model.."
Actually, it's pretty right to say that. :) The connection and context between the client and the broker is maintained in a stateless model. From the guide:
"When a client connects to an AppServer that is running in stateless operating mode, the AppServer broker maintains the connection between itself and the client. The AppServer broker can then dispatch each remote procedure request from the client to any available AppServer agent to execute. Because any AppServer agent that executed a previous request from this client might not be available, the AppServer agent that handles subsequent requests might be different."
Hi.
"I am not sure it's exactly right to say that clients never disconnect from the Broker in a stateless model.."
Actually, it's pretty right to say that. :) The connection and context between the client and the broker is maintained in a stateless model. From the guide:
"When a client connects to an AppServer that is running in stateless operating mode, the AppServer broker maintains the connection between itself and the client. The AppServer broker can then dispatch each remote procedure request from the client to any available AppServer agent to execute. Because any AppServer agent that executed a previous request from this client might not be available, the AppServer agent that handles subsequent requests might be different."
Flag this post as spam/abuse.
The difference between "pretty right" and "exactly right" is in how the client is coded to connect/disconnect ;)
Stateless/state-free modes certainly *assume* that the connection is persisted, but you can still code the client to disconnect and re-connect for every group of requests if you're so inclined / if you're dealing with legacy code that was originally written for state-aware/state-reset modes.