AppServer does not consistently start

Posted by jkrugerrd@gmail.com on 24-Jun-2015 13:04

Hi. I am have issues with AppServer on a Linux environment that does not consistently restart after a  a daily mid nightly backup.   The AppServer will sometime start normal. When the scripts in the cron fail to start the AppServer the manual will start in after the first or second try. I thought it could be a "port in use" issue but cannot find a message or port number that cause the failure. I have also tried the suggestion in the knowledgebase article “S000027673”, without success.  Attach please find my environment, scripts and log files. I  suspect my setup and script files since this happen at 3 different customers/server with the same setup. I would really appreciate some guidance.  Jan Kruger

Posted by TheMadDBA on 24-Jun-2015 14:13

Which one of these scripts are you running? There are several that start/ appservers... some of them have pauses between the admin,nameserver and appserver starts that might not be long enough.

Also I just have to ask... why exactly are you shutting down the appservers and/or database for a backup? probkup online works just fine without having to kick users off.

In general terms you need to make sure that none of your ports overlap (with other Progress processes or common Linux ones). I see that you have port ranges defined for the DB servers and the appservers... but not the secondary SQL servers.

And you need to upgrade to 10.2B07 at the least.

All Replies

Posted by David Cleary on 24-Jun-2015 13:23

Are you sure your databases are fully started when you try and start the AppServer?
 
Dave
 
[collapse]
From: jkrugerrd@gmail.com [mailto:bounce-jkrugerrdgmailcom@community.progress.com]
Sent: Wednesday, June 24, 2015 2:05 PM
To: TU.OE.General@community.progress.com
Subject: [Technical Users - OE General] AppServer does not consistently start
 
Thread created by jkrugerrd@gmail.com

Hi. I am have issues with AppServer on a Linux environment that does not consistently restart after a  a daily mid nightly backup.   The AppServer will sometime start normal. When the scripts in the cron fail to start the AppServer the manual will start in after the first or second try. I thought it could be a "port in use" issue but cannot find a message or port number that cause the failure. I have also tried the suggestion in the knowledgebase article “S000027673”, without success.  Attach please find my environment, scripts and log files. I  suspect my setup and script files since this happen at 3 different customers/server with the same setup. I would really appreciate some guidance.  Jan Kruger

Attachments:
appserver.zip
Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by jkrugerrd@gmail.com on 24-Jun-2015 13:36

Yes. The users is active from Client Networking session  while I do the AppServer restart. Especially when I execute the AppServer startup script manually.

Posted by TheMadDBA on 24-Jun-2015 14:13

Which one of these scripts are you running? There are several that start/ appservers... some of them have pauses between the admin,nameserver and appserver starts that might not be long enough.

Also I just have to ask... why exactly are you shutting down the appservers and/or database for a backup? probkup online works just fine without having to kick users off.

In general terms you need to make sure that none of your ports overlap (with other Progress processes or common Linux ones). I see that you have port ranges defined for the DB servers and the appservers... but not the secondary SQL servers.

And you need to upgrade to 10.2B07 at the least.

Posted by lecuyer on 24-Jun-2015 14:23

The -14 error in the agent log file indicates that after binding successfully to a listening port, the agent was unable to call the socket level listen() api successfully.  In order to determine the reason for this error, we need to increase the logging level on the agent.  To do this, *add* the 'UBNET' attribute (not including the quotes) to the srvrLogEntryTypes in ubroker.properties for the appropriate appserver.  Also, set the srvrLoggingLevel property to 4.  This should cause the errno value set by the failing listen() call to be logged to the agent logfile.

Posted by lecuyer on 24-Jun-2015 14:29

Another thing to consider is whether many of the ports in the agent listening range are defined in the /etc/services file (or equivalent).  In general the agent will not use a port that is defined in the services file, even if it is not actually in use at the time it is starting up.  This can cause the agent to skip over ports that are actually usable, and incorrectly conclude that no ports in the range are available.  From the agent log file provided, that doesn't seem to be the case here, but it's something to consider when assigning port ranges.

Posted by jkrugerrd@gmail.com on 24-Jun-2015 14:34

TheMadDBA: I use the "startProServer.sh" script, but tried the others with no success.  I will try to increase the pause to 360. And try again. I will try the online backup, but the startup script should still be dependable. I will also put port ranges on the SQL server. How would I determine which port / port ranges is save to use. Is there preferred / suggested port I should use. I will upgrade to 10.2B07 but it will take a while since the customer is remote and downtime will have to be scheduled.

Posted by jkrugerrd@gmail.com on 24-Jun-2015 14:36

lecuyer: I will implement your changes and suggestion and report back with a more verbose log.

Posted by TheMadDBA on 24-Jun-2015 14:40

For the port ranges you need do two things...

1) Look at the /etc/services file like lecuyer suggested.

2) Look at the service ports, minport and maxport settings in all of the scripts and ubroker.properties

Between the two of them you should be able to find an acceptable range. How wide of a range you need for each grouping will depend on how many brokers/agents you are starting.

I agree the scripts should be dependable but you are hurting performance and complicating things by restarting so often.Ideally you should only stop the DB when you are doing maintenance.

Posted by jkrugerrd@gmail.com on 24-Jun-2015 14:46

TheMadDBA: Thanks I will follow all your suggestions and report back.

Posted by Marek Bujnarowski on 24-Jun-2015 14:48


Have you tried the workaround starting one initial agent?

[collapse]
From: lecuyer <bounce-lecuyer@community.progress.com>
Reply-To: "TU.OE.General@community.progress.com" <TU.OE.General@community.progress.com>
Date: Wednesday, June 24, 2015 at 21:30
To: "TU.OE.General@community.progress.com" <TU.OE.General@community.progress.com>
Subject: RE: [Technical Users - OE General] AppServer does not consistently start

Reply by lecuyer

Another thing to consider is whether many of the ports in the agent listening range are defined in the /etc/services file (or equivalent).  In general the agent will not use a port that is defined in the services file, even if it is not actually in use at the time it is starting up.  This can cause the agent to skip over ports that are actually usable, and incorrectly conclude that no ports in the range are available.  From the agent log file provided, that doesn't seem to be the case here, but it's something to consider when assigning port ranges.

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by jkrugerrd@gmail.com on 02-Jul-2015 03:07

Thanks guys my AppServer now consistently start. I did take a shotgun approach and apply all your surgestions so I cannot say exactly wat the remedy was. Thanks again.

This thread is closed