Evaluating a Message Broker

If you have the technical need for messaging and have done some research into products that are out there, you are surely aware of the multitude of options that one has when it comes to picking a solution that will fulfill your needs for years to come. A thorough evaluation process should probably cover:

  • Performance – Evaluating a product on performance can only be done one way: In a lab. Vendors will generally not publish performance numbers for fear of being called out. So the only way to find out is by testing them on your own hardware/network configuration. The topics that should be covered are: Throughput and latency for point-to-point and publish-subscribe messaging, the storage scheme for persistent messages, and performance of pub-sub for high fan-out. These tests should probably be repeated many times for different quality of service options such as durable subscribers, persistent messages, multiple topics, SSL, large payloads, and transactional messages.
  • High Availability & Fail-Over – Believe it or not, some of the most expensive and most advertised message brokers did not even have a solution for client-side fail-over and broker discovery until just recently (some still don’t). Validating that you do not have to rely on home-grown retry logic for fail-over and fail-back should probably be the first step. Secondly and more importantly is the fail-over capabilities of the messaging broker. Obviously this largely depends on your requirements for continuous availability but understanding the time it takes for a back-up broker to take over the responsibilities of a failed broker is fairly important for most folks. Furthermore you would want to study the behavior of the broker when it fails while handling transactions that are in-flight and see if message order is preserved during fail-over and fail-back operations.
  • Scalability – Testing a messaging infrastructure for scalability can be done in multiple places. It would probably make most sense to first test the scalability of one broker and then the scalability of the messaging infrastructure as a whole (as you add more brokers). At times people confuse performance and scalability testing and I generally like to think of it this way: For performance tests you are looking for numbers. e.g. peak throughput per second, or sustained throughput per second. For scalability testing you are looking for numbers AND behavior. e.g. maximum number of connections AND what happens when we exceed those numbers? How does the broker behave (cpu, memory, performance) as the number of subscribers grows?
  • Handling of Fault Conditions – One thing worth checking is to see how the broker behaves when it runs out of memory or disk space. Same goes for messaging: Is there a degradation in quality of service when you have slow subscribers or abnormal subscriber termination. Some high-profile messaging brokers are notorious for crashing when they receive too many messages that might have an invalid destination.
  • Administration & Monitoring – In the past many firms evaluated a messaging brokers admin capabilities based on how easy it was to set up queues and topics. Some open source brokers are now blazing a new trail and getting rid of interfaces and XML files for defining queues and topics all together. But having the ability to look into the belly of the broker to monitor its health statistics and error conditions is going to be important regardless. Whether that is done via APIs or JMX or SNMP or a web interface, is a matter of preference and should be evaluated accordingly.
  • Standards Compliance – Perhaps it is worth ensuring that your organization builds its interfaces according to the JMS specification and thus it would be important to validate the track record of the vendor when it comes to adoption of the specification. For some products the adoption of specifications has come as an afterthought and it becomes quite apparent in their inability to reliably implement durable subscribers or persistent messaging. For other products the specification is not important as they target specific business areas and try to solve niche problems.
  • Other areas that can be of potential importance: security (Access Control Lists on queues and topics, setting up access groups, authenticating clients using SSL), supported languages (both for clients as well the implementation of the broker), additional messaging features (hierarchical topics, flow control), and the reliability, and support provided by the vendor

So given all that what is the best messaging product out there? How much does it cost? I have my favorites but then again it all depends on the problem you’re trying to solve.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s