Cluster and storage

Posted by appmath on 15-Nov-2012 17:24

Hello,

We need to update the storage for a two-broker-cluster to point to faster devices.

The process I have in mind is simple: update them and restart one broker at a time (make sure that there is always 1 broker running)

.

  1. If one broker is taken down while it was delivering some messages, and let's assume that a couple of queues were half full. Are we going to experience any disruptions/lost messages?
  2. Is there a proven technique to minimize the impact in terms of performance and risks.
  3. Once the 2nd broker has been restarted, the first broker would have all the  connections, is there a way to force the 1st broker to “share” all those  connections with the 2nd broker?

Thanks,

Aziz

All Replies

Posted by pmeadows on 16-Nov-2012 04:23

Hi Aziz,

From your question I assume the brokers in the cluster are standalone brokers, not replicated (CAA) pairs?

1. Some messages may be stuck/trapped for the time the broker is down, but they shouldn't be lost.  They'll get delivered once the broker is back up.

2. I don't know about a proven technique, but I'd allow the broker to shut down cleanly, copy its recovery logs and SonicMQStore directory to the new location, update the broker configuration to point to the new recovery log and SonicMQStore locations, then restart the broker.

3. As things stand, you'd need to restart the clients or drop their connections to the 2nd broker (via the 'Connection's view in the Sonic Management Console).  The second option assumes the client applications are coded such that they handle connection drops cleanly.  If clients (or their connections) periodically terminate and get created then you could simply wait for things to balance out naturally over time.

Finally, make sure that the file system you plan to use for the broker storage is suitable.  We typically expect the broker storage to be on local disks, though I believe a few users have moved the files to a SAN.  A key point is that when we issue a 'sync' for a file we expect the sync call to block until we're guaranteed the changes are committed to the file system.  This isn't always the case for network-based storage; a true sync across a network can be expensive.  Without this guarantee there's some risk of lost data or corruption in the event of a crash.

Thanks, Paul.

Posted by appmath on 16-Nov-2012 17:35

Hello Paul,

Yes, you are correct in assuming that the brokers aren't replicated.

Thanks for clearly answering every question.

Aziz

This thread is closed