Enterprise Integration PatternsMessaging Patterns
HOME    PATTERNS    RAMBLINGS    ARTICLES    TALKS    DOWNLOAD    BOOKS    CONTACT

Competing ConsumersCompeting Consumers

Messaging Patterns

Previous Previous   Next Next

An application is using Messaging. However, it cannot process messages as fast as they’re being added to the channel.

How can a messaging client process multiple messages concurrently?

Create multiple Competing Consumers on a single channel so that the consumers can process multiple messages concurrently.

Competing Consumers are multiple consumers that are all created to receive messages from a single Point-to-Point Channel. When the channel delivers a message, any of the consumers could potentially receive it. The messaging system's implementation determines which consumer actually receives the message, but in effect the consumers compete with each other to be the receiver. Once a consumer receives a message, it can delegate to the rest of its application to help process the message. (This solution only works with Point-to-Point Channels; multiple consumers on a Publish-Subscribe Channel just create more copies of each message.)

...

Example: Apache Kafka (NEW)

Apache Kafka is a distributed streaming platform that allows clients to consume messages in both Publish-Subscribe Channel and Competing Consumers semantics. Kafka is optimized for high throughout and horizontal scalability and therefore tries to avoid the overhead that can be inherent in coordinating across multiple Competing Consumers. So while the client interface allows multiple consumers to consume messages off a topic in a competing fashion, guaranteeing that each message is only consumed by one of the consumers, the implementation actually relies on multiple partitions implemented as s:

In the example above, a Kafka topic consists of three partitions divided across two consumers. A Message Router distributes the messages across the three Point-to-Point Channels. This router is generally a that works off the message key. This allows the sender to control the distribution of messages, e.g. to assure related messages go onto the same partition. This can be necessary as in-order delivery across partitions is not guaranteed.

This design implies that the decision which consumer receives the message is made a priori, meaning consumers don't actually compete but rather read from independent channels. This limits the ability to load balance between consumers, e.g. if one channel runs empty, the corresponding consumer won't be able to aid other consumers whose channels still have messages. Also, as the number of partitions is set independently, not all consumers will service the same number of partitions, as illustrated in the example above.

However, as Kafka partitions are relatively inexpensive, there are usually lots of them, meaning that each consumer is in reality likely to read of many partitions and therefore channels. This highlights another trade-off: there can't be more competing consumers in a consumer group than there are partitions on that topic.

Related patterns: Event-Driven Consumer, Message, Message Channel, Message Dispatcher, Message Router, Messaging, Point-to-Point Channel, Polling Consumer, Publish-Subscribe Channel, Transactional Client


Want to keep up-to-date? Follow My Blog.
Want to read more in depth? Check out My Articles.
Want to see me live? See where I am speaking next.

Enterprise Integration Patterns Find the full description of this pattern in:
Enterprise Integration Patterns
Gregor Hohpe and Bobby Woolf
ISBN 0321200683
650 pages
Addison-Wesley

From Enterprise Integration to Enterprise Transformation:

My new book describes how architects can play a critical role in IT transformation by applying their technical, communication, and organizational skills with 37 episodes from large-scale enterprise IT.

DRM-free eBook on Leanpub.com

Print book on Amazon.com

Creative Commons Attribution License Parts of this page are made available under the Creative Commons Attribution license. You can reuse the pattern icon, the pattern name, the problem and solution statements (in bold), and the sketch under this license. Other portions of the text, such as text chapters or the full pattern text, are protected by copyright.


Table of Contents
Preface
Introduction
Solving Integration Problems using Patterns
Integration Styles
File Transfer
Shared Database
Remote Procedure Invocation
Messaging
Messaging Systems
Message Channel
Message
Pipes and Filters
Message Router
Message Translator
Message Endpoint
Messaging Channels
Point-to-Point Channel
Publish-Subscribe Channel
Datatype Channel
Invalid Message Channel
Dead Letter Channel
Guaranteed Delivery
Channel Adapter
Messaging Bridge
Message Bus
Message Construction
Command Message
Document Message
Event Message
Request-Reply
Return Address
Correlation Identifier
Message Sequence
Message Expiration
Format Indicator
Interlude: Simple Messaging
JMS Request/Reply Example
.NET Request/Reply Example
JMS Publish/Subscribe Example
Message Routing
Content-Based Router
Message Filter
Dynamic Router
Recipient List
Splitter
Aggregator
Resequencer
Composed Msg. Processor
Scatter-Gather
Routing Slip
Process Manager
Message Broker
Message Transformation
Envelope Wrapper
Content Enricher
Content Filter
Claim Check
Normalizer
Canonical Data Model
Interlude: Composed Messaging
Synchronous (Web Services)
Asynchronous (MSMQ)
Asynchronous (TIBCO)
Messaging Endpoints
Messaging Gateway
Messaging Mapper
Transactional Client
Polling Consumer
Event-Driven Consumer
Competing Consumers
Message Dispatcher
Selective Consumer
Durable Subscriber
Idempotent Receiver
Service Activator
System Management
Control Bus
Detour
Wire Tap
Message History
Message Store
Smart Proxy
Test Message
Channel Purger
Interlude: Systems Management Example
Instrumenting Loan Broker
Integration Patterns in Practice
Case Study: Bond Trading System
Concluding Remarks
Emerging Standards
Appendices
Bibliography
Revision History