ESBDB with a large result

Posted by l4ky on 29-Dec-2009 20:06

Hi.. all

I'd like to ask, if someone have an experience or knowledge for my problem in the Workbench.

My problem is, I have a process which using ESBDB service for querying a data from a DB which resulting around 50000 record.

And in the Workbench if I run the process, I will get a message in the log which said "..... message was rejected because it is larger than the maximum queue size....",

I tried to solve it by increase the memory siz for the queue.. up to 100000Kbyte.

After that the process can show the result, and no message found in the log file...

What I want to ask is, is someone have any other solution for this problem??

Thanks..

Uq

All Replies

Posted by Bill Wood on 30-Dec-2009 09:15

There is also a "maximum" message size in Sonic that clients are limited to-- by default 10M.  (This is the Advanced property CONNECTION_TUNING_PARAMETERS.MAX_MSG_SIZE)   Sonic can handle larger messages but there is this limit is there as a 'test' because typically you should not be using messaging for single messages this size.  If you are trying to transfer 100M+ at a time, you are better off using the large message support (which chunks the file) or an alternate backchannel like FTP.

This is really a question of architecture and appropriate technology.  Sonic can handle very large messages but you need to change buffers, memory footprint, etc otherwise you'll run into performance issues.

The Database Service also has a hidden parameter (if you export the serviceType you will see it). There is no parameter to set the maximum number of rows,  but there is hidden parameter available to set the maximum result set size.  The default is set to 10 MB, to match the size of the SonicMQ design limit. This hidden init parameter is  called xmlSerializationLimit.   If the size of the result set exceeds this limit then the DB  service throws an exception resulting in an RME message (but this will, or course, not actually send the message.

In your case, you probably FIRST ran into the error with the output Queue too small.   When you increased the queue size, then you probably ran into the xmlSerializationLimit.  If you could check the RME for some message you will probably see one there.

This thread is closed