AppServer Load/Performance Testing

Posted by Vaibhav Parab on 07-Nov-2014 05:23

I would like to benchmark my Progress appserver components (NameServer and AppBroker) based on number of transactions it can handle with minimal response times. Any help would be greatly appreciated.


1. Can i increase the number of agents in the broker (from default value of maxSrvrInstances=25 to say maxSrvrInstances=512).

2. How many brokers can a nameserver (with NameServer Load Balancing) handle.

3. What should be the memory allocated to NameServer, Broker

4. Any tools to monitor the performance of the Progress components?

5. If someone can share their experiences of Performance Testing on Progress AppServer.

TIA, Vaibhav

All Replies

Posted by Irfan on 07-Nov-2014 07:40

Hi Vaibhav,
 
Please find my comments inline
 
 
[collapse]
From: vaibhav_parab [mailto:bounce-vaibhav_parab@community.progress.com]
Sent: 07 November 2014 16:54
To: TU.OE.Development@community.progress.com
Subject: [Technical Users - OE Development] AppServer Load/Performance Testing
 
Thread created by vaibhav_parab

I would like to benchmark my Progress appserver components (NameServer and AppBroker) based on number of transactions it can handle with minimal response times. Any help would be greatly appreciated.


1. Can i increase the number of agents in the broker (from default value of maxSrvrInstances=25 to say maxSrvrInstances=512).

Yes you can. But you might need to look at your application needs. You might need to look at how fast your server can handle the requests and pass it to the appserver broker. If your server is capable to use the maximum of agents then yes you can add as many agents you want.

When you query your appserver, look for  “Active Clients(now,peak)”and check what is its peak value. It will give you an idea of how many clients are concurrently connected in peak. This will help you determine how many agents you might require for your application.

But remember that each process will take up some memory and CPU. So recommendation is to start your agent processes based on its utilization.

2. How many brokers can a nameserver (with NameServer Load Balancing) handle.

I haven’t tried it, but it should be able to balance as many you configure. I don’t think there is any limit here. If your need to just horizontally balance the load across multiple appserver brokers then you might need to first look at the answer to your first question and then balance across multiple brokers.

3. What should be the memory allocated to NameServer, Broker

I haven’t seen there is any need to increase the memory of the Nameserver. Broker memory is completely dependent on the application i.e the number of requests the broker handles and the payload that are sent to the appserver broker.

4. Any tools to monitor the performance of the Progress components?

For my Performance testing , I rely on the utilities that put lesser loads on the server, hence I rely on utilities like top and generate scripts on top of it to monitor memory and CPU of the OpenEdge components

But if you do not want to do that then you can check for tools like nagios

If you are using SOAP or REST related clients then you can use Jmeter.  Jmeter helps you to distribute your client load across multiple VM’s or client machines and then you can also monitor your server’s OS Resources from Jmeter Client by running a Jmeter Agent on the server.

5. If someone can share their experiences of Performance Testing on Progress AppServer.

TIA, Vaibhav

Stop receiving emails on this subject.

Flag this post as spam/abuse.

[/collapse]

Posted by TheMadDBA on 07-Nov-2014 11:17

Are you talking about the components themselves or your application? Most of your issues will be with the application, hardware, database and the network.

1) You can, but at some point you are putting too much risk/work in one broker. Use the nameserver load balancer to scale.

2) The maximum number of brokers I have used the the Nameserver load balancer was somewhere around 30.

3) Occasionally you need to tweak the java settings, but what you really need to worry about is how much memory you are giving the actual appserver processes for -TB,-mmax,etc.

4) Performance of the appserver/nameserver itself is usually going to be the least of your issues. The only thing to watch out for are having too many active connections per broker. (See the advice about asbman from Irfan).

5) Once we got past the number of appserver brokers to run we spent the vast majority of the time tuning the application,network, operating system and database.

We used OS tools to monitor the systems and networks; promon and the virtual system tables to monitor the database and the application.

This was for over 4000 concurrent users running on about 400-500 appserver agents. We had very little issues with the infrastructure itself once we enabled the nameserver load balancer.

Posted by Vaibhav Parab on 08-Nov-2014 05:31

Thanks a lot @Irfan @TheMadDBA.

Just to give a background of what i'm working on: We have set a target number of users, concurrent hits and transactions per second and now are running tests to size our infrastructure accordingly. So the idea is to add infra wherever we identify it to be less. We have analysed and fine tuned the application logic through initial tests keeping in mind programming constructs to be used.

@Irfan, @TheMadDBA, I was looking into the Active Clients and Client Queue Depth parameters shown by asbman to calculate the exact number of Progress components (AppBroker,NameServer) to be setup. As mentioned by you, I have started looking into identifying ideally what memory is used by each agent (through -TB, -mmax and other such parameters) so as to get a good idea of how much the current infra can support. I was looking into topas (aix7.1) and other OS tools to monitor server info but have not reached to a point where i can do this through scripts. Would like to explore more about Nagios. Would continue working on this.

Still there is a confusion in what is the load on AppBroker when number of agents are increased. And, similarly, what is the load on NameServer when number of AppBrokers are increased as TheMadDBA had quoted point 1 " You can, but at some point you are putting too much risk/work in one broker. Use the nameserver load balancer to scale. "

Right now we have 3 thought processing running to have an ideal setup.

We have two Progress AppServers which has a hardware Load Balancer above it. Each application server has one NameServer (with NS LB enabled) and two brokers each running now with 20 agents.

Thought 1 :

Increase the number of agents under each of the brokers to get more active processes, less clients in queue waiting for processing. If required increase the number of brokers later.

Thought 2 :

Increase the number of nameserver and brokers with the same configuration of 20 agents each. This way we can have each nameserver manage 2 AppBrokers each. Load Balancing between the nameservers can be handled by the dedicated resources of the H/W LB above the AppServer. This way, "IF" there is any risk/work/load on single nameserver/broker to handle load would be minimized.

Thought 3 :

Remove any NameServer and make the calls by directly connecting to the brokers (-DirectConnect). This way all the load balancing would be done by the H/W LB. But i guess this would bring us to a greater loss as we would miss out the pros of UDP connection to the NameServer and certain other stuff.

Requesting you to share your thoughts on this. 

Posted by TheMadDBA on 08-Nov-2014 15:11

Load on the name server and broker should be relatively small, unless your application is connecting/disconnecting from the appservers way too frequently. Talk to your developers... ideal situation is connect at application startup and disconnect at shutdown.

On AIX specifically we had issues with JAVA memory and broker hangs once we got past 60 agents on one broker. We "solved" this by using the name server balancer to have no single broker with more than 40 agents and just increased the number of brokers responding to the same appserver name. I believe the core issue that we ran into has been solved in newer versions of Progress though.

A variation on your thought 1 is my basic approach... set 30-40 agents per broker and add more brokers to the balance... so instead of 2 brokers per nameserver.. use 3,4,10, whatever works.

Don't increase the number of name servers unless you see some real reason to do this... and have it verified as the real issue by a consultant and/or PSC.

We had one name server handling 60+ different brokers (not all balanced) and didn't have any issues with performance related to the nameserver/brokers.

If you think you have adequately tuned the app and database.... do it again... and then again.  I promise once you get the brokers set up properly your performance/scaling issues will have very little to do with their setup and everything to do with network traffic, app doing too many calls, reading too many records, database IO,etc.

Posted by Vaibhav Parab on 08-Nov-2014 23:46

Thanks a lot TheMadDBA. We have a java app connecting with Progress App using stateless mode of connection.Hence constant connections to the nameservers would happen.Will go ahead and implement your advice regarding the limited agents and increased number of brokers.

Will keep you updated if i deduce something out of my tests.

Keeping this forum open for all to share their valuable thoughts.

Thanks a lot guys.

Posted by Irfan on 10-Nov-2014 05:11

Hi Vaibhav,

How many client connections do you expect your application to get in peak times ? Might be constructing your environment based on that might be a right idea while doing Performance testing and quantify your configuration.

Consider the fact that of appserver broker's memory and CPU utilization, when configuring more than one broker.It will be responsible for routing your concurrent requests and carrying their payload .

As for my knowledge the amount of OS Resources consumed by Nameserver are very little.

This thread is closed