http://www.eaipatterns.com Gregor's Ramblings on Enterprise Integration en Copyright (C) Gregor Hohpe Wed, 02 Jan 2008 07:22:52 -0800 The end of the year is always the time to reflect on the past happenings. My friends in Japan often send a New Year's card with 12 pictures, each showing the significant event during the month. I am not sure something significant related to Enterprise Integration Patterns happened each month, but I think it's still nice to reflect a little bit on the history and current state of the patterns. After all, the patterns have matured from a draft paper to being adopted as the lingua franca for a number of open source messaging projects. http://www.eaipatterns.com/ramblings/66_fouryears.html Since Google's Developer Day, I have been promising to speak more regularly about Google products. This is a bit of a balancing act for me, as I want to avoid the slippery slope of becoming a corporate sock puppet. I also don't want to neglect my interest in connected systems, SOA, and asynchronous messaging. With the Internet turning into the largest connected system, the gap between Web-based technologies and connected systems design has become significantly smaller, allowing me to combine interests and Google developer products more naturally. Of course, the fact that Google is rolling out new developer products and APIs around the clock, does not hurt either. http://www.eaipatterns.com/ramblings/65_gears.html I am just returning from a trip to Montreal and Keystone, CO for the OOPSLA and Colorado Software Summit conferences. I spoke on SOA Patterns, workshopped my conversation pattern paper, and gave six talks on event-driven architectures and building mashups using Yahoo! Pipes and Google Mashup Editor. Phew! http://www.eaipatterns.com/ramblings/64_oopslacss.html What’s wrong with this picture? I crash a stylish party at a pricey venue (SF Moma). I have no business being there, nor do I have the required invitation. I nonchalantly talk my way in, and hit the crowd with two fellow geeks. Within a few seconds an attractive blonde with a low cut dress approaches us and engages us in a conversation. We grab a few free drinks, indulge on the tirelessly circulating horse d’oeuvres, and talk to the blonde. http://www.eaipatterns.com/ramblings/63_bubble20.html I attended the Enterprise Mashup Summit last Friday. It was a small-ish event, with about 50 people attending. We saw about 10 presentations, mostly by vendors plus an open forum. None of the presentations were too sales-ey, which gave the event a nice workshop feel. Given the limited size, it would have been nice to have more time for interaction. Here is a quick rundown of my impressions. http://www.eaipatterns.com/ramblings/62_mashupsummit.html The Internet is cruel. So you finally made the leap from EAI stone age to become a hip mashup developer. Now it turns out you are once again behind the curve, Unless you've written your won Facebook app and scored a million or so installs you are just a worm in the dust of the global data superhighway. Join the club. http://www.eaipatterns.com/ramblings/61_facebookgarage.html As I reported from Mashup Camp, an increasing number of vendors play in the mashup space. I am obviously not the only one who noticed. Dion Hincliffe recently discussed 17 different mashup tools. Maybe I have an MBA hidden inside me (or 10 years of consulting left some marks), but I was temped to conduct my own market segmentation. Gartner would be proud of me. Just be aware that my analysis is not meant to be comprehensive and all statements carry a 0.5 probability of being right :-) http://www.eaipatterns.com/ramblings/60_mashupmarket.html Mashups pull data from different sources, aggregate and transform the data to be used in different contexts. EAI solutions pull data from different sources, aggregate and transform the data to be used in different contexts. Huh? http://www.eaipatterns.com/ramblings/59_mashupeai.html I attended the first day of Mashup Camp today. The event took place at the Computer History Museum, which is actually walking distance from my office. Ironically, the only day in the year when something actually happens within walking distance in Silicon Valley it had to rain. Oh well. http://www.eaipatterns.com/ramblings/58_mashupcamp.html Last time I claimed that users like events. This time I want to show how I fulfilled my personal desire for events off the Web. I built two solutions to alert me to new book reviews on Amazon, one using a Python script, the other using Yahoo! Pipes. http://www.eaipatterns.com/ramblings/57_pipes.html Two weeks ago I presented a keynote at the DEBS (Distributed Event-based Systems) conference. The emphasis of the conference was on academic work in the field but event-based systems are rapidly gaining traction in commercial applications as CEP (Complex Event Processing) is becoming a household name like SOA (equally difficult to pin down). All this talk about events reminded me that applications on the Web are becoming increasingly event driven. http://www.eaipatterns.com/ramblings/56_events.html When graphical user interfaces became popular thrifty people had the idea that they could generate them automatically. Well, that bubble burst pretty quickly. When people say nothing is new with Web services, they are right in this case. Once again people want to generate the interface automatically. http://www.eaipatterns.com/ramblings/55_api.html Last week Google held a developer day in 10 cities around the world. Here are my take-aways. http://www.eaipatterns.com/ramblings/54_gdd.html You can only get away with joking about transactions for so long before the gods of transactions bring you to justice. So is was only a matter of time before Eric Newcomer would catch me waxing about coffee and transactional integrity... http://www.eaipatterns.com/ramblings/53_eric.html Beginning of this year I submitted an article to a renowned software publication who is running a special issue on software patterns. Sadly, my article did not make it into the magazine but I decided to publish an expanded version on my site. Hey, I got to make it worth your time following my ramblings! Am I bitter that my article was rejected? Not at all, but if I ever get a hold of these #^%#$& -bleep- of a -bleep- I'll... http://www.eaipatterns.com/ramblings/52_soapatterns.html JavaOne is one of the few geek conferences that take place in San Francisco. The content can be hit or miss but the proximity and the good parties always make it worth stopping by. I also had to fulfil my duty to recruit the last smart software engineer in the Bay Area to Google. http://www.eaipatterns.com/ramblings/51_javaone.html I have been pretty quiet recently, but I finally have some results to show. The current issue of IEEE Internet Computing (May/June 2007) contains my guest column for "Toward Integration". Also, I authored one chapter in the upcoming book "SOA Expertenwissen" (in German). http://www.eaipatterns.com/ramblings/50_publications.html Even though I have been working for the (self-declared) non-evil empire for a while now, I still have a lot of ties to the Microsoft community, even beyond drinking Weissbier with Christian Weyer. I am particularly proud that I have been able to maintain my MVP status, Microsoft's Most Valuable Professional award. Here my (brief) report from the MVP Summit in Seattle & Redmond. http://www.eaipatterns.com/ramblings/49_mvpsummit.html Erik Doernenburg and I have been presenting our talk on Software Validation a few times now and have received very positive feedback. In the talk we mention that beyond visualization you can use similar techniques to perform checks and validations against a system. However, none of our demos actually included an example. So I went out to correct that and added validations to my messaging visualizer. http://www.eaipatterns.com/ramblings/48_validation.html I am just riding the train back from Aarhus (the second largest city in Denmark in case your geography skills are as weak as mine were before my first trip here) to Copenhagen. Many speakers consistently rave about JAOO. The speaker roster, the logistics, the attendees – everything just seems to be very high quality and run very smoothly. This year I was nominated to be track host for the SOA track entitled "SOA – What's Left to Say?" http://www.eaipatterns.com/ramblings/47_jaoo.html This year I got to go to fooCamp! What is fooCamp? It is sort of like Burning Man for techno geeks. But just as Burning Man has gone pretty high tech fooCampers started to have more fun by burning things. It's difficult to put the experience into words... http://www.eaipatterns.com/ramblings/46_foo.html I confess. I am a groupie. When Tim O'Reilly invited me to be part of a workshop on Web 2.0 patterns I was quite excited. First, because I was going to meet the exalted one in person. Second, because I would have a chance to learn a lot about Web 2.0 as I am pretty much starting from zero. http://www.eaipatterns.com/ramblings/45_web20.html I just returned from a trip to Germany to attend a workshop on The Role of Business Processes in Service Oriented Architecture. Besides a guaranteed top score in buzzword bingo the workshop provided a unique opportunity to connect thought leaders from academia and industry over the course of a week. The attendee roster sported the usual industry heavyweights, i.e., Microsoft, IBM, SAP, making me proud to add Google to the list. http://www.eaipatterns.com/ramblings/44_dagstuhl.html Hanging out with my intellectual drinking buddies reminded me that our integration patterns have been embraced by a fair number of commercial as well as open source projects. In my eyes this is really the best indicator of success for a pattern language. Latin – a dead language used mostly by doctors to sound more knowledgeable – is not a good model for a pattern language. You want a language that is alive and actually used by people. Talking to a few folks who did embrace our language motivated me to take a quick survey of the places where our patterns pop up. http://www.eaipatterns.com/ramblings/43_wildpatterns.html I am so clever. In my very first professional project in the US over 10 years ago I used a DSL to solve an EAI problem. If those buzzwords had been around at the time I would have been pretty cool, too. I guess you can’t have everything. http://www.eaipatterns.com/ramblings/42_eaidsl.html Some of the most hated people in San Francisco must be the meter maids, the DPT people who drive around in golf carts and hand out tickets to anyone who overslept street cleaning or did not have enough quarters for the meter. On some projects, the most hated people are the metric maids, the people who go around and try to sum up a developer’s hard work and intellectual genius in a number between 1 and 10. http://www.eaipatterns.com/ramblings/41_metrics.html A lot of discussion these days relates to DSL's (Domain Specific Languages) that allow business models and rules to expressed in a more readable fashion in the source code. Ultimately one might be able to share the executable source of the system with the business users to validate that this is what they wanted. While Java is not necessarily the greatest language to "host" a DSL we can go a lot further than developers generally believe or care for. http://www.eaipatterns.com/ramblings/40_marchnan.html Computer science is known as the field where every problem can be solved by just one more level of indirection. Well, a message channel berween two components is such a level of indirection. As it turns out naming this additional element can have a more profound influence on the system design than might appear at first. http://www.eaipatterns.com/ramblings/39_namingchannels.html After joining the big G I put particular emphasis on staying in touch with the .Net community. For one thing, a fair number of my intellectual drinking buddies tend to congregate around the large software company in the Pacific Northwest. Second, as a believer in cross-platform integration, technology-neutral design patterns and peace on earth I want to make sure I switch the koolaid flavor every so often. One opportunity to do that presented itself at OOP in Munich where Ralf Westphal invited me to speak on distributed system architectures as part of the .Net track. During the proverbial hallway discussions we started talking about boxes and lines, but in a profound way... http://www.eaipatterns.com/ramblings/38_interaction.html Remember the people from high school who had strange hobbies and could never get a date? Well, they all grew up and they are running a conference now, the Object-Oriented Programming, Systems, Languages, and Applications conference, OOPSLA in short. While the conference logistics can be quirky at times the content is top notch. And it seems like most of them have managed to get a date after all. http://www.eaipatterns.com/ramblings/37_oopsla.html Two years ago you were not cool if you did not have a blog. Today you are not cool if you have not done a podcast. Of course I cannot allow myself to fall behind, so here is my podcast on layered applications: http://www.mariocardinal.com/podcast/Show/0007.html http://www.eaipatterns.com/ramblings/36_podcast.html (Professional) life can be a little ironic sometimes. Now that I work for Google I was able to go to Microsoft's Professional Developer Conference aka PDC for the first time. It was actually very nice to be able to stay in touch with the "other side". http://www.eaipatterns.com/ramblings/35_pdc.html Two weeks ago I attended a very small, but special conference on enterprise software development. Here is what I came away with.. http://www.eaipatterns.com/ramblings/34_cb.html One of the biggest perceived fears of authors is that someone publishes a book or an article on the same topic just ever so slightly ahead of you that you have done all the work but look like a copy cat if you still publish your work. In reality this actually does not happen very often (at least not for books) as each author has his or her unique style and view even on the same topic. This time, though, Ken Arnold stole a little bit of my show by publishing an excellent article in ACM Queue magazine called "Programmers are People, too". http://www.eaipatterns.com/ramblings/33_interface.html As many people have already heard, I changed employers. Before the rumor mill completes its work entirely, here a few comments from my end. http://www.eaipatterns.com/ramblings/32_google.html I just returned from Microsoft TechEd Europe in Amsterdam. I was lucky enough to present on event-driven architectures in front of almost 600 people. Since multiple people asked me for the source files, I figured I'd post a quick summary of the talk, including the source files. http://www.eaipatterns.com/ramblings/31_techededa.html Having been to a number of conferences and their attached expo floors recently it was not surprising that I was not able to escape the line "this tool does not require programming, everything is done through configuration". Hearing this tag line for the third or fourth time got me wondering, "what really is the difference between coding and configuring?" http://www.eaipatterns.com/ramblings/30_configure.html This year I actually missed putting in a proposal for TechEd. However, that has not kept me from actually presenting :-) So I am here fighting the Orlando Summer humidity together with 11000 other geeks and am actually having a good time. http://www.eaipatterns.com/ramblings/29_teched.html On a recent project we developed a Web services framework that supports asynchronous callbacks and asynchronous invocation with polling. The difficulties we had describing what we were trying to do to my fellow ThoughtWorkers on other projects prompted me to discuss the different forms of asynchrony in a little more detail. http://www.eaipatterns.com/ramblings/28_asynchrony.html When you build a distributed system you have to choose between latency and reliability. Sticking your head in the sand will not help (and you get sand in your ears...). http://www.eaipatterns.com/ramblings/27_latency.html In my perennial battle to free distributed system developers' minds from the perils of the call stack mentality I came across another reason why a call stack is not appropriate for distributed applications: the call stack is inherently asymmetric. However, when building distributed applications, that asymmetry really has no place. http://www.eaipatterns.com/ramblings/26_symmetry.html It seems like I spend all my time at conferences. I am speaking at the Patterns and Practices Summit and SD West this week. This one is setup by Keith Pleas in close collaboration with the Patterns and Practices group at Microsoft. So yes, I am still promiscuous when it comes to platforms. But it makes me feel better that Ted Neward seems to beat me in that category, though. http://www.eaipatterns.com/ramblings/25_pnpSummit.html A number of people have already blogged, er ranted, in detail about the TSSS but that can't stop me from occupying my fair share of the cyberspace with my impressions. http://www.eaipatterns.com/ramblings/24_tss2005.html OK, so I am officially the slowest blogger on the 'Net... last week I attended two conferences, the BTW in Karlsruhe, Germany and TheServerSide Symposium in Las Vegas. Too bad they are over 6000 miles apart :-( http://www.eaipatterns.com/ramblings/23_btw2005.html Today's rambling takes a look at the application side of application integration. We recently looked at a packaged application under development and looked at criteria and options for how this application can live in an evolutionary and connected environment. http://www.eaipatterns.com/ramblings/22_extensibility.html Happy New Year everybody! Over the holidays I tried to sort out some of my thoughts on Web services, SOA, architectural styles, coupling and all the other good stuff. I started to think about service-oriented architectures as an architectural style and how it compares to prior styles, such as distributed component architectures. Maybe it was the New Year's mood but I started to think that the evolution of architectural styles is a little bit like drunk driving... http://www.eaipatterns.com/ramblings/21_drunkdriving.html One of my favorite pastimes is to argue with people whether a solution is stateless, whether it should be stateless and what it means to be stateless in the first place. Ideally, the debate would involve alcoholic beverages and the other person would pick up the check. After "loosely coupled", "stateless" must be a close runner-up as the ultimate nirvana in buzzword-compliant architectures. It is also equally hotly debated. Today, I 'd like to share my view on state and lessness. If you disagree you are welcome to argue with me, but you are buying! :-) http://www.eaipatterns.com/ramblings/20_statelessness.html Loosely coupled architectures are often portrayed as the holy grail of architecture. But all too often these architectures turn into what Martin Fowler coined the "Architect's dream but the developer's nightmare". This time I want to share some of the challenges we have found working with loosely coupled architectures and how we addressed them. http://www.eaipatterns.com/ramblings/19_composing.html People often criticize asynchronous messaging solutions as too complicated and cumbersome. Or, they believe distributed solutions cannot be successful unless they include a distributed transaction model. There is little doubt that asynchronous solutions require us to think in new ways as we have to deal with concurrency, out-of-sequence issues, correlation and other. However, the real world is full of examples of asynchronous processes that deal successfully with exactly the same issues. We don't have to go further than the local coffee shop... http://www.eaipatterns.com/ramblings/18_starbucks.html I just attended the ACM Middleware conference in Toronto (I thought a 1/2 day tutorial in integration patterns). This conference is more of an academic events when compared to some of the purely commercial conferences like JavaOne. It was interesting to see what academia are working on these days in the areas of middleware and SOA. http://www.eaipatterns.com/ramblings/17_acmconference.html Last year's OOPSLA tutorial on Enterprise Integration Patterns was quite well received. The most frequent constructive criticism was that the whole tutorial was PowerPoint as opposed to running code. I took that feedback to heart and set out to develop a toolkit that demonstrates the routing and transformation patterns in our book. After a lot of deja-vu in Windows GDI programming I created a toolkit that contains each pattern as a small executable. The toolkit is a lot of fun and makes for great demos, but it also triggered a bigger question. Are patterns meant to be codified as components? http://www.eaipatterns.com/ramblings/16_patternscomponents.html I just attended the Indigo Strategic Design Review in Redmond. This means we got to interact with the product team and play with some pre-beta bits. Naturally, a lot of details are under NDA and are likely to evolve before the beta release. So here some rather general impressions -- I heard Internet connectivity in jail is really bad :-). http://www.eaipatterns.com/ramblings/15_indigo.html Our most recent project was to replace a batch-oriented stored procedure with a more flexible, maintainable and scaleable Java-based architecture. We chose an event-driven architectural style that processes events as they occur. For the initial roll-out we did not even have to distribute the solution so it all runs in single JVM (if you ignore clustering for the moment). Read on to learn more about the implementation and our experiences with intra-JVM EDA. http://www.eaipatterns.com/ramblings/14_edainjvm.html The notion of Dependency Injection or Inversion of Control has generated quite a bit of interest lately. To a large extend, the move towards this architectural style is driven by developer's dissatisfaction with strong dependencies of individual components on their run-time container (for example, J2EE). These dependencies make it difficult or impossible to test components outside of their container, throwing a major monkey wrench into test-driven development of these types of applications. Dependency Injection avoids these dependencies and therefore improves testability. On a recent project we developed an event-driven architecture (intra-JVM, not distributed) and found that we could realize many of the same benefits. http://www.eaipatterns.com/ramblings/13_ioc.html It is nearly impossible to keep track of all the Web Services (WS-*) specifications or proposed "standards". The matter is only worsened by the many competing proposals. Should you be using WS-Eventing or WS-Notification? How about WS-Reliability vs. WS-ReliableMessaging? Even if you choose the "right" specification, it still is likely to evolve over time. So are these standards useful at all for Web Services development today? I contend that they are, albeit in a somewhat unconventional way.... http://www.eaipatterns.com/ramblings/12_jeopardy.html Many of you have seen my rants on doodleware and top-down modeling. At the same time, anyone who has seen my book must believe me when I say that I am a very visual person. Is Gregor going schizophrenic (or becoming a hypocrite)? Not at all! I am a big fan of visual representations that are harvested from an actual running system and present the user with an accurate, big picture view of a working system. I felt it was time to put some money (time) where my mouth is and try to build such a solution and see what actually happens... http://www.eaipatterns.com/ramblings/11_dependencies.html For a change I spoke at a conference without traveling halfway around the world. At over 15,000 attendees, JavaOne is one of the largest developer conferences. Unfortunately, the signal-to-noise ratio is not always all that great, but it is still a good place to meet old and new friends. This year's conference was littered with talk and demos on SOA, ESB, Web Services, orchestration etc. Here my impressions... http://www.eaipatterns.com/ramblings/10_javaone.html When using asynchronous messaging, we can no longer rely on the linear, in-sequence execution of events. Therefore, the notion of correlating multiple messages is fundamental to designing robust messaging solutions. While correlation seems to be a simple concept at first glance, there are quite a few interesting nuances. http://www.eaipatterns.com/ramblings/09_correlation.html A number of integration books classify integration into styles such as database integration, functional integration and so on. However, on closer examination it turns out that there is a fair amount of confusion about what these styles really mean. Let me give my view. http://www.eaipatterns.com/ramblings/08_integrationstyles.html I had never thought about having guest appearances on my ramblings, but this e-mail response from my friend Hyon made me laugh hard enough that I think it would be a waste not to share it with the community. http://www.eaipatterns.com/ramblings/07_noescape.html Design decisions can be influenced by many factors. In order to make matters more predictable, many architects look for clear-cut borderlines. How complex does my screen navigation have to be in order to justify a Front Controller as opposed to a Page Controller? How complex does my integration problem have to be to justify use of an EAI suite instead of hand wiring the solution? Invariably, the answer is "it depends". Let's have a closer look at why that is so. http://www.eaipatterns.com/ramblings/06_hysteresis.html Encapsulation and abstraction are the holy grail of today's software development. One of the great successes of hiding the implementation details under a common, abstracted interface are TCP/IP stacks. But as so often, overwhelming success can also have a flip side. http://www.eaipatterns.com/ramblings/05_encapsulation.html When we assess the quality of a software engineer we don't look as much how long he has worked with a specific language but rather whether he or she has a good grasp of the underlying models and techniques, such as OO design, systems architecture. Still most EAI job offers sound like a software catalog: "x years TIBCO IntegrationManager", "y years Vitria Businessware" etc. etc. Some thoughts on what we should be really looking for... http://www.eaipatterns.com/ramblings/04_skills.html The Hub-and-Spoke concept manifests itself in many ways in integration solution. Like any popular concept, it can sometimes cause as much confusion as it provides help. http://www.eaipatterns.com/ramblings/03_hubandspoke.html Many integration tools offer graphical development tools. Do they really make integration easier? http://www.eaipatterns.com/ramblings/02_doodleware.html Integration appliances aim to make integration cheaper faster and more reliable. Is this technology ready for prime time? http://www.eaipatterns.com/ramblings/01_appliances.html