Asynchronous Messaging Recomendations

Posted by gmacnamara on 26-Aug-2011 04:25

I’ve had some conflicting advice around this (I’ll protect the names) so I’m bringing the question to the masses. The requirement is simple so fingers crossed so is the answer. Here goes.
We have a system that sends messages to a third party system. At a specific time the third party allows only read access to their data so the messages must be stored until such time as they allow update access. For what its worth the messages are extremely valuable. So the question is what is the best method for storing the messages? I’ve had these two suggestions.

1)    1) Store them in a database and create a new service to access these

2)    2) Use SonicMQ and persist the messages.

One of the worries or considerations around the SonicMQ side of things is the way that the broker falls over is the queue fills up.

Thoughts?

Cheers

Technology Partner - Forum

All Replies

Posted by sfritz on 26-Aug-2011 05:57

It depends... as always :-)

A messaging system is not designed to be used as a long term storage.

It provides "buffers" for messages until the client can process it.

"Long term" is the part where people have different opinions.

I personally would store the messags in a database.

Using a "scheduler" serviece or an activation daemon the mesages can then be pulled out of hte database and forwarded to the target system.

Storing it in queues might look like the easiest approach on short term.

What if the project requirements change in the near future?

What if the message volume increases a lot?

etc.

When you store them in a DB then you are on teh safe side and can adjust easily.

Posted by gmacnamara on 26-Aug-2011 07:26

Hi Stefan,
Thanks for the reply. The thing on my mind was that if we were to use JMS then we have a defined supported standard API for managing the messages. I would have thought that setting up a persistent queue with its persistence is a database would have been the best option but others do not agree. I see this as the best of both worlds as if we went with the option of a database with bespoke services to manage that we would be creating everything from scratch. Am i perhaps missing somethign fundamental around how persistant messages are handed when persisting them to a database? I really appreciate your feedback so if possible if you could expand on your thoughts that would be great.
Regards
Grant

Posted by pmeadows on 26-Aug-2011 08:00

Hi Grant,

Could provide a little more insight into system: for example, the volume of messaging to this third party system, i.e. typical message size and rate (though as Stefan warns this has a habit of increasing over time as systems become more heavily used); how often the third party system goes into the read-only mode and for how long (is it a regular pattern that the third party has committed to, or do you need to handle unexpected longer-term outages); the total volume of messages you anticipate needing to buffer during the read-only periods?

Is it the case that the third party system supports updates the majority of the time and only goes into read-only mode for relatively short periods (e.g. for internal maintenance)?

As an aside, if you're finding the broker falls over as a queue fills then that points to a (mis)configuration issue.

Thanks, Paul.

Posted by gmacnamara on 26-Aug-2011 08:20

Thanks Paul,
The main driver for this is that want to write a pattern for this type of event so even though I have exact requirements for this project I want to produce something that other unknown projects can follow if required. The third party I’m talking about is effectively closed for business from say 8pm to 6am (as an example). The messages will be rather small (a few k) and relatively small in numbers (perhaps a few hundred or so). The main point of this particular requirement is the value of the messages. Now will be mandating things like CAA etc but it’s the storage that’s at question here.
So with this is mind we still need to create a pattern for the store and forward of messages as this will be required for other projects round the corner. I’m happy to implement either option but I’m trying to understand the technical background.
Again, this is invaluable information and I very much appreciate the cooperation of everyone.
Cheers
Grant

Posted by pyin on 26-Aug-2011 09:41

Hi Grant.

Design pattern wise, queuing is for store-and-forward.  As Paul noted, a CAA broker should not fail over as a queue fills up - messagea are either pushed to the db or the sender would be blocked.  Also, with CAA, messages are replicated and hence persisted on both machines.  Using the other approach, you will reply on the db replication.

my 2 cents.

Perry

Posted by rrudis on 26-Aug-2011 10:03

The use case that you describe (limited down time, relatively small message rates and sizes) sounds appropriate for using SonicMQ queues or durable subscriptions.  Where we tend to guide people away from using the broker persistent store is in the case of long term storage of large amounts of data, both for performance reasons and due to the fact that there is no mechanism for doing online backups of the broker store.

Posted by gmacnamara on 30-Aug-2011 07:39

Now thats an interesting point that you make re backups. Excuse my ignorance but if the queue is persisted to a DB as its store then would the backup of the DB not be sufficient to mitigate this problem?

Cheers

Grant

Posted by rrudis on 06-Sep-2011 16:41

You would have to take the broker offline to do the backup, and you would need to back up both the message store and recovery log files.

This thread is closed