Processing messages in batches - Community Groups - Forum

Posted by habin123 on 25-Jan-2012 07:48

Hi,

Please help me with your suggestions in the below mentioned issue.

Issue:

We have a job which publishes 3 lakhs messages in 15 minutes everyday. This is creating a flow control of the publisher as subscribers are slow. To come out of this problem we increased the listeners of subscriber and everything worked fine for a moment. Now, these subscribers are inserting messages to a database on a low performing server. The result is again a flow control. We don’t have an option to increase the server performance at this point.

Is there a way that sonic esb provides to publish these 3 lakhs messages in batches? So that we come out of the flow control situation and the slow performing database can insert that data between each delay of the batch.

I don’t know if this is a good solution or not, please suggest.

Thanks,

Habin

Community Groups - Forum

All Replies

Posted by Rob Fitzpatrick on 26-Jan-2012 16:55

I'm not familiar with the terminology "3 lakhs messages". Does that mean "3 x 10^5 messages"? Anyway, I can't help you with the Sonic side, but I am interested in your definition of "slow performing database". Is this an OpenEdge database? How do you quantify "slow performing"? What tools/metrics are you using to gauge database performance? Do you know where your current bottleneck is in your database transaction throughput, and if so, have you attempted to eliminate or lessen it? If your subscriber app is write-intensive you may have a before-image bottleneck, or an under-performing I/O sub-system. If the clients are remote, there could also be network constraints or server contention. What are your Progress version, OS, and hardware specs on the DB server?

On the client side, you said "subscribers are inserting messages to a database". What type of clients are these, 4GL or SQL? Do they run in-house code, or something third-party? Can you access the source, and if necessary, change it?

It may not be necessary to engineer a workaround on the Sonic side if you can eliminate or reduce the problem on the database side.

Posted by habin123 on 27-Jan-2012 00:53

Thanks for the reply Rob.

Yes your understanding is correct 3 lakhs = 3 * 10 ^ 5 .We are using sonic 8.0.1.

The bottleneck is happening at the DB server.It is not able process (insert) such high number of records in this sort time as database is running on a server with low configuration.Right now,we can't upgrade the hardware.

Subscribers are java clients running sql insert queries.

Thanks,

Habin

This thread is closed