Possible to consume all messages off of a sonicmq queue in o

Posted by Paul Radbone on 12-Dec-2014 10:40

Hi,

Is it possible to consume all messages on a queue in one go?  I'm currently doing the following:

RUN createMessageConsumer IN hPTPSession
(THIS-PROCEDURE, /* this procedure will handle it */
"pp_process_message", /* name of internal procedure */
OUTPUT hConsumer).
IF ERROR-STATUS:ERROR THEN DO:
MESSAGE ERROR-STATUS:GET-MESSAGE(1).
END.

RUN receiveFromQueue IN hPTPSession
("to_rest", /* name of queue */
?, /* no message selector */
hConsumer). /* handles incoming messages*/
IF ERROR-STATUS:ERROR THEN DO:
MESSAGE ERROR-STATUS:GET-MESSAGE(1).
END.

/* Start receiving messages */
RUN startReceiveMessages IN hPTPSession.
IF ERROR-STATUS:ERROR THEN DO:
MESSAGE ERROR-STATUS:GET-MESSAGE(1).
END.

with a wait for 0 to check if there are any messages.

RUN waitForMessages IN hPTPSession ("inWait", THIS-PROCEDURE, 0).

Now every time pp_process_message fires, I only get one message.  Is there a way to do this, so that I can consume all messages available in one go?

All Replies

Posted by Paul Connaughton on 12-Dec-2014 11:15

If there are truly messages in the queue, my guess is the InWait user defined function is returning false.

If that is not the case, can you provide all of the code?

Posted by TheMadDBA on 12-Dec-2014 11:20

You can try setting the prefetch count to actually grab more than one message at a time from the queue, but Progress will still process each message one by one.

Are you wanting to increase performance or do you really just want the messages consolidated into a single message?

Posted by Paul Connaughton on 12-Dec-2014 11:46

I just tested this out and I am seeing different behavior with waitForMessage set to 0. Sometimes it picks up 1, sometimes 5, sometimes all.

If I set the waitForMessages to 1 it always consumes all of the messages from the queue before stopping.

Posted by Paul Radbone on 12-Dec-2014 11:50

The inwait value is hard coded to true at the moment.  I'm wanting to poll without blocking as this process is running on an appserver and being accessed from a REST service.

Posted by Paul Radbone on 12-Dec-2014 11:56

If I set the wait for to 1 as you have above, I get more than one message at a time, but if there are no messages, this blocks the rest front end for 1 second, which I dont want to do.

Posted by Bill Wood on 12-Dec-2014 12:02

You have to remember that messaging is intended to be asynchronous, disconnected and distributed. The clients cannot and do not know if there are messages anywhere in the network of brokers that "might" go to them. The client only sees what has been delivered to it, so you need to either play with PREFETCH or you need to wait.

When you wait, you need to wait for the time you specify because there could be a message arriving. =

Posted by Paul Radbone on 12-Dec-2014 12:21

Even with the prefetch set to 10, if the wait-for is 0, I still noly get one message at a time.

Polling every 50ms works ok, but it would be much better if the prefetch worked for me.

Posted by Paul Connaughton on 12-Dec-2014 14:48

What I am seeing is that if I do anything time consuming in the message consumer, like a PAUSE(5) statement, the results are unpredictable in terms of how many messages are received before the waitForMessages returns.

However, if I do not do anything in the message consumer, it seems to consume all of the messages every time.

While what Bill is saying is true, if your messages are not too large and/or you can off-load your consumer logic, you should be able to get the behavior you are looking for.

Posted by Paul Radbone on 12-Dec-2014 14:58

Hi,

When you day do noting in the consumer, do you mean no wait-for?  I dont quite understand?  Bearing in mind the consumer is running on an appserver.

Thanks,

Paul.

Posted by Paul Radbone on 12-Dec-2014 15:00

[quote user="TheMadDBA"]

You can try setting the prefetch count to actually grab more than one message at a time from the queue, but Progress will still process each message one by one.

Are you wanting to increase performance or do you really just want the messages consolidated into a single message?

[/quote]

I just want the messages consolidated into a single message.

Paul.

Posted by Thomas Mercer-Hursh on 12-Dec-2014 15:10

This seems like a peculiar requirement.  If they are logically one message, then send them as one message.  If they aren't logically one message, then you would be losing the integrity of the message by consolidating them.   I suppose you could use some kind of message aggregation service to bundle them, but again it seems like you would lose integrity and responsiveness.

Now, what I can see wanting to do is to have a service which receives the message and then forks a thread to process the message and immediately returns to process the next message.   With a single threaded AVM that's a problem, but you can fake it with asynchronous AppServer calls to process the message.

Posted by Paul Radbone on 15-Dec-2014 03:45

I just need a simple solution for a rest service.  Maybe if I explain what I want from the REST clients point of view, it will be clearer to see what I'm trying to achieve?

The REST client is going to poll the REST service every 200ms (for example) to see if any responses are waiting for it.  I want to read all waiting messages off of a sonic queue and return them back to the client.

I dont want to block the client (So I can't have a wait for on the appserver) as we will have 200+ clients connected and polling at any one time.

Now at the moment I have a REST client set to poll the REST service every 50ms as the appserver code (as in my original post) is only returning one message at a time - even when there are 10 messages on the queue.

Posted by Bill Wood on 15-Dec-2014 04:09

WRT
>> I just need a simple solution for a rest service.  
 
The ‘problem’ is that the REST client requirements are at some level in opposition to the JMS specification.   The Queue is designed to separate one or more receivers on the queue from the mechanics of the client.  Though it may seem clear that the messages are all for one REST client, JMS puts architectural restrictions on the ability of a client to ‘KNOW’ what is waiting for it.
 
Changing the PREFETCH (and the PREFETCH THRESHOLD) are the only way to approach what you want, but this is not a guarantee that the queue will be cleared.  (Past discussions on this thread have talked about PREFETCH, but you do need to set the prefetchThreshold to < 1 otherwise you will find you get 1 message on every 2nd request (approximately).
 
The basic problem is that in JMS the specification does not allow/specify a way to ‘clear a queue of pending messages’ on a single call.   In SonicMQ it is even more complex when you consider what is involved when you have a cluster of brokers, or a network of Routing Nodes.   In these cases the messaging servers don’t even have a view of “what is every message that should go to a particular client”.   Messages for a single queue might be distributed across different brokers, for example.
 
From a REST client (or rather an HTTP Client) you can configure a direct connection to the SonicMQ broker to get the next message.  That will only be one message at a time, but should relieve you from having to go through the AppServer (if that is an issue).
 
Others may have a suggestion, but the only think you might otherwise do is have the appserver continuously move messages from Sonic into a Database table.   Then when a REST client wants something, you would query the messages in the table and bundle them up into a single REST response. 

 

Posted by pedromarcerodriguez on 15-Dec-2014 04:32

Hi,

I might be losing something, but from a REST point of view, using SonicMQ as the transport I think the correct approach to take would be to implement a request-response pattern (activemq.apache.org/how-should-i-implement-request-response-with-jms.html). I couldn't find an example for SonicMQ but should be same approach.

Now, I am not sure what is your REST client, or why does it need to poll, but I would say that if it can connect to SonicMQ, when it submits a request, it subscribes to a response (using correlationId, temporary queues, ...). Also that would allow you to start (if required several REST services in your appserver), every one would fetch one message and process its response, and so it would do the REST client. I think that would be much more responsive that trying to process several REST requests in same client.

Again, apologies if I got wrong your requirements, I am not sure to understand them correctly.

Regards,

This thread is closed