Enterprise Integration Patterns
Messaging Patterns
HOME PATTERNS RAMBLINGS ARTICLES TALKS DOWNLOAD BOOKS CONTACT
Scatter-GatherScatter-GatherMessaging Patterns » Message Routing

In the order processing example introduced in the previous patterns , each order item that is not currently in stock could be supplied by one of multiple external suppliers. However, the suppliers may or may not have the respective item in stock themselves, they may charge a different price and may be able to supply the part by a different date. To fill the order in the best way possible, we should request quotes from all suppliers and decide which one provides us with the best term for the requested item.

How do you maintain the overall message flow when a message needs to be sent to multiple recipients, each of which may send a reply?

Use a Scatter-Gather that broadcasts a message to multiple recipients and re-aggregates the responses back into a single message

The Scatter-Gather routes a request message to the a number of recipients. It then uses an Aggregator to collect the responses and distill them into a single response message.

There are two variants of the Scatter-Gather that use different mechanisms to send the request messages to the intended recipients:

  • Distribution via a Recipient List allows the Scatter-Gather to control the list of recipients but requires the Scatter-Gather to be aware of each recipient's message channel.
  • Auction-style Scatter-Gathers use a Publish-Subscribe Channel to broadcast the request to any interested participant. This option allows the Scatter-Gather to use a single channel but at the same time relinquishes control.
...

Example: Serverless Loan Broker on AWSNEW

The serverless Loan Broker example includes a Scatter-Gather that routes requests for a loan quote to interested banks and selects the best offer from the incoming responses. The example implements both a a Recipient List using AWS Step Functions' Map state (see Part 2) and a Publish-Subscribe Channel using the Amazon Simple Queuing Service (SQS) (see Part 3).

Example: Mulesoft ESBNEW

Mulesoft's enterprise service bus includes a Scatter-Gather construct, described as Scatter-Gather Router and can be configured using the scatter-gather element.

Although each route is independently executed, the Scatter-Gather Router configures a pre-defined number of paths, so it doesn't broadcast to an unknown number of recipients. Knowing all recipients makes the completeness condition of "Wait for all" (see Aggregator) a natural choice. Route errors propagate to the Scatter-Gather, which in turn flags an error, passing on any results that were successfully passed by the individual routes. The aggregation strategy is concatenation, turning single variable values into a list of all route results.

Example: Azure Durable Functions (Fan out/fan in)NEW

Durable functions, which are used to orchestrate serverless Azure Functions, make it easy to implement Scatter-Gather, following the Fan out/fan in construct. which is supported by the WhenAll method, which waits for all functions to execute:

Fan out/fan in with Azure Durable Functions

def orchestrator_function(context: df.DurableOrchestrationContext):
    work_batch = yield context.call_activity("GetVendors", None)
    parallel_tasks = [ context.call_activity("Vendor", b) for b in work_batch ]
    outputs = yield context.task_all(parallel_tasks)
    total = sum(outputs)
  

The number of recipients can be calculated at run-time, and the completeness condition can support a variety of strategies, including "Wait for all" (Task.WhenAll), "First Best" (Task.WhenAny), or "Fixed Timeout" (Task.WaitAll).

Related patterns: Aggregator, Introduction to Composed Messaging Examples, Asynchronous Implementation with MSMQ, Asynchronous Implementation with TIBCO ActiveEnterprise, Composed Message Processor, Publish-Subscribe Channel, Recipient List, Return Address, Splitter


Table of Contents
Preface
Introduction
Solving Integration Problems using Patterns
Integration Styles
File Transfer
Shared Database
Remote Procedure Invocation
Messaging
Messaging Systems
Message Channel
Message
Pipes and Filters
Message Router
Message Translator
Message Endpoint
Messaging Channels
Point-to-Point Channel
Publish-Subscribe Channel
Datatype Channel
Invalid Message Channel
Dead Letter Channel
Guaranteed Delivery
Channel Adapter
Messaging Bridge
Message Bus
Message Construction
Command Message
Document Message
Event Message
Request-Reply
Return Address
Correlation Identifier
Message Sequence
Message Expiration
Format Indicator
Interlude: Simple Messaging
JMS Request/Reply Example
.NET Request/Reply Example
JMS Publish/Subscribe Example
Message Routing
Content-Based Router
Message Filter
Dynamic Router
Recipient List
Splitter
Aggregator
Resequencer
Composed Msg. Processor
Scatter-Gather
Routing Slip
Process Manager
Message Broker
Message Transformation
Envelope Wrapper
Content Enricher
Content Filter
Claim Check
Normalizer
Canonical Data Model
Interlude: Composed Messaging
Synchronous (Web Services)
Asynchronous (MSMQ)
Asynchronous (TIBCO)
Messaging Endpoints
Messaging Gateway
Messaging Mapper
Transactional Client
Polling Consumer
Event-Driven Consumer
Competing Consumers
Message Dispatcher
Selective Consumer
Durable Subscriber
Idempotent Receiver
Service Activator
System Management
Control Bus
Detour
Wire Tap
Message History
Message Store
Smart Proxy
Test Message
Channel Purger
Interlude: Systems Management Example
Instrumenting Loan Broker
Integration Patterns in Practice
Case Study: Bond Trading System
Concluding Remarks
Emerging Standards
Appendices
Bibliography
Revision History


Enterprise Integration Patterns

Find the full description of this pattern in:
Enterprise Integration Patterns
Gregor Hohpe and Bobby Woolf
ISBN 0321200683
650 pages
Addison-Wesley

From Enterprise Integration to Enterprise Transformation:

My new book describes how architects can play a critical role in IT transformation by applying their technical, communication, and organizational skills with 37 episodes from large-scale enterprise IT.

DRM-free eBook on Leanpub.com

Print book on Amazon.com

Creative Commons Attribution License

Parts of this page are made available under the Creative Commons Attribution license. You can reuse the pattern icon, the pattern name, the problem and solution statements (in bold), and the sketch under this license. Other portions of the text, such as text chapters or the full pattern text, are protected by copyright.