(1154) No servers are available. Try again later.

Posted by Vienneau on 08-Jun-2010 16:25

This is my first post, so please forgive me if I do something wrong.
OpenEdge 10.2A02 on AIX 6.1.  CHUI environment.
The majority of the users connect directly to the application from the server.  We also added some web connections recently that I'm still learning how to manage/understand.  And we've had some long-time TCP connections.
Yesterday the TCP connection was failing and users got this error:
The server or the system has no more resources. Please contact Progress
The log file shows:
[2010/06/07@08:12:24.548-0400] P-1089760    T-1     I BROKER  0: (1154)  No servers are available.  Try again later.
The Progress "help" says:
In the shared memory model with networking, there are too many clients  
logged into the database, or there are too few servers for this database.
You can log in after another user logs out.  You might want to check the
-Mn PROGRESS parameter given to the PROGRESS controlling server or BROKER
(PROSERVE -Mn n) and raise the number of servers. 
                     
-Mn was set to 9.
-n is set to 100.
I disconnected all the web users via promon (lacking a better alternative) but that didn't fix it.  There were far less than 100 connections.
In the end, I had to shut down the DB (mid-day - the users loved it) and bring it back up and that cleared everything.
But I don't really know what happened, or where to look for solutions.  Is it a hardware (shared) memory issue?  Does it relate to the SQL broker we added to handle the web connections?  I'm a bit lost and I haven't found a good summary of how the -Mn and -Mi and so on all interact and should be set.
If anyone has suggestions/ideas, I'd be very appreciative.
Matt

All Replies

Posted by Admin on 09-Jun-2010 00:48

You really should try asking this question of Tech Support - you're likely to get a faster answer.

However:

-Mn sets the total number of servers the broker can support.  (You said you set this to 9 - fair enough.)

-Ma sets the maximum number of connections allowed per server.  Have you set this parameter?  [It defaults to ceil(-n/-Mn)]

-Mpb sets the maximum number of servers in this pool.  It defaults to -Mn.  Did you set this parameter?

The message you're getting is caused by the broker saying, 'Either the maximum connection count [-n] has been reached, or more likely, all the available servers of your type have reached their maximum user count, and I can't spawn another one because the maximum number of servers [min(-Mpb, -Mn)] has been reached.'

So, possible causes are:

* You have reached your global connection limit (unlikely, based on what you've said).

* You have a number of SQL servers started - each one of those uses a -Mn slot as well - and the total number of remote connections is exceeding -Ma x count(4GL servers).  (For example, if you have 6 SQL servers started, you could only start 3 4GL servers, each with a -Ma of 11 (12?), so you could only support 33 (36?) remote connections.)  To fix this, kill some _sqlsrv2 processes, or better, create a secondary server pool to manage the SQL connections (or 4GL connections if you prefer).

* You have clients which are failing to connect due to firewall issues, causing your server connection count to be exhausted with PENDING connections without actually having any actual connections.  (This can happen if you have a firewall porthole that passes the primary login port for the database, but not the range in which the server ports are allocated, -minport -> -maxport arguments.)  To fix this, restart the server with -PendConnTime argument e.g. 5 seconds (or perhaps 30 seconds).  By default, pending connections last forever, so using -PendConnTime by default is actually good practise.  You can diagnose this by using promon db, R&D, 1, 2 (list servers).  If any servers consistently have PEND > 0, this is a problem for you.

* The broker has reached its ulimit for the number of running processes.  This can occur if you've changed the permission of the _mprosrv binary to be no longer setuid root.

In general, remote connections should be reserved for connections from off-machine unless you have latch stability problems.  However you have AIX, which means that AIX's broken shared memory system for 32-bit processes means you might be using remote connections even for local sessions just so you can allocate enough memory to your databases.  (You can fix this by moving to a 64-bit port for AIX.)

However, I ramble.  Hope this helps.  Alternatively, paste in the entire section of the log following a multi-user session begin (i.e. show us all the startup messages) so we can see more information about your setup, which might help diagnose your problem.

- David Eddy.

Posted by Vienneau on 09-Jun-2010 11:50

Thank you very much David.

You've definitely given me some areas to investigate - I'm happy you rambled.  Pending connections seems like the red flag in this situation so we'll concentrate on that and if we're still having problems I will post them in the Tech Support forum.

Matt

This thread is closed