Patterns and Best Practices for Asynchronous Messaging
Today's applications rarely live in isolation. Users expect instant access across all functions, which may reside in disparate applications or third-party services. However, building distributed and integrated applications requires developers to tackle asynchrony, partial failures, incompatible data models, API drift, and much more.
This pattern language consisting of 65 integration patterns helps developers design and build distributed applications or integrate existing ones. The patterns provide a technology-independent vocabulary and visual notation harvested from proven solutions to recurring problems. They also share common "gotchas" and design considerations. Besides receiving numerous accolades, the patterns spurred the development of a whole generation of open source Enterprise Service Bus (ESB) implementations, such as Apache Camel, Mule, WSO2, Oracle Service Bus, Open ESB, SonicESB, or Fuse ServiceMix.
Buy the book Enterprise Integration Patterns or read a sample chapter first. Find the most recent content in my blog or articles.
Messaging Patterns in Today's World: Microservices and Serverless
When Bobby Woolf and I started to document the patterns 20 years ago, the key technologies for distributed applications were JMS, SOAP, MSMQ, and .NET WCF. Large-scale application integration was done with integration tools and platforms, such as IBM WebSphere MQ, TIBCO, WebMethods (now Software AG), or Microsoft BizTalk. Although technology has evolved, many of these products still form the backbone of modern enterprises.
Cloud platforms and deployment automation have laid the foundation for a new generation of distributed systems: microservices and serverless architectures. Those applications rely on a smooth interconnect between components, giving rise to Service Meshes, Serverless Orchestrators, and Event Buses. Amazingly, we find the same integration patterns in those systems! That's why this site contains many modern examples for integration patterns:
New Book: Platform Strategy
Almost every IT strategy these days includes some element of platform strategy. And that's no surprise: platforms appear to be able to rewrite the laws of IT physics. They boost innovation despite (or perhaps due to) standardizing; they speed up developers while assuring compliance; and they reduce cognitive load without restricting choice.
As one might guess, building such an in-house platform, or even deploying one, is far from easy. Many organizations end up with something that’s outdated by the time it’s launched, restricts rather than enables users, and faces a certain demise when its use is mandated in a last-ditch effort to make the economics work.
Harvested from a decade of building successful in-house platforms and enabling enterprises to utilize cloud platforms, this book provides a fresh look at what makes IT platforms work, why they are different from other IT services, and how you can use them to boost developer productivity. Get a paper copy on Amazon or the ebook from Leanpub (Kindle, PDF).
Book: The Software Architect Elevator
Modern architects must do more than draw UML diagrams and recite architectural styles. Rather than focus on technical decisions alone, architects and senior technologists need to combine organizational and technical knowledge to help transform their company’s structure and processes. That's why successful architects are those who connect the IT engine room, where the technical reality is defined, to the organization's penthouse, where the business strategy is set.
This books equips architects and IT leaders with the technical, communication, and organizational skills to be successfully in modern enterprises. Available now on Amazon and anywhere where books are sold.
Work-in-progress: Conversation Patterns
Asynchronous messaging is the foundation for most integration solution because its architectural style acknowledges the challenges of distributed communication, such as latency or partial failure. However, many interactions between systems extend beyond sending a single, stateless message: a request may expect a response; a handshake or authentication are needed first; a reservation is confirmed or expires. Such conversations, stateful exchanges between participants, present new design challenges and patterns. I therefore started documenting Conversation Patterns, which are the starting point for Enterprise Integration Patterns 2.
What Makes Integration Difficult?
Architecting integration solutions is a complex task. There are many conflicting drivers and even more possible 'right' solutions. Whether the architecture was in fact a good choice usually is not known until many months or even years later, when inevitable changes and additions put the original architecture to test. Unfortunately, there is no "cookbook" for enterprise integration solutions. Most integration vendors provide methodologies and best practices, but these instructions tend to be very much geared towards the vendor-provided tool set and often lack treatment of the bigger picture, including underlying guidelines, principles and best practices.
Asynchronous Messaging Architectures
Asynchronous messaging architectures have proven to be the best strategy for enterprise integration because they allow for a loosely coupled solution that overcomes the limitations of remote communication, such as latency and unreliability. That's why most EAI suites and ESB's are based on asynchronous messaging. Unfortunately, asynchronous messaging is not without pitfalls. Many of the assumptions that hold true when developing single, synchronous applications are no longer valid. Vendor-independent design guidance helps developers avoid these pitfalls so they can build robust integration architectures based on asynchronous messaging.
How can Patterns Help?
Patterns are a proven way to capture experts' knowledge where no simple “one size fits all” answers exist, for example in application architecture, object-oriented design, or message-oriented integration . Each pattern tackles a specific problem by discussing design considerations and presenting an elegant solution that balances often conflicting forces. The solution is not the first approach that comes to mind, but one that has evolved through actual use over time, capturing the experience that senior developers and architects have gained by repeatedly building solutions and learning from their mistakes.
What am I Reading Right Now?
Modern Software Engineering: Doing What Works to Build Better Software Faster, Dave Farley, 2021
The "father of continuous delivery" distills the concepts behind modern software delivery, such as modularity, feedback, coupling. A must-read for every modern software engineer.
Practical Process Automation: Orchestration and Integration in Microservices and Cloud Native Architectures, Ruecker, 2021
Process automation is no longer the domain of large-scale business processes but an integral part of modern application architecture. This books shows you where the two meet.
System Architecture: Strategy and Product Development for Complex Systems, Crawley, Cameron, Selva, Addison-Wesley 2015
A tour de force on designing complex systems with a special emphasis on decomposition and logical relationships, augmented by a list of System Architecture Principles. Most examples drawing on non-IT domains helps convey the concepts without getting lost in technology debates and makes it worth the price.