Making ESB pain visible

1 July 2009

At some point last year I was asked to review the architecture of the software behind a large and popular website. The resident architect explained how he had followed a modern approach, decoupling the web front-end from back-end services that provide content. To gain further flexibility he had put the front-end and the services on an ESB, mostly to cater for the possibility to make the content available to other consumers. In short, the architecture diagram looked a lot like many others: nothing to see here, move on.

Website Architecture

The diagram above only shows one of the content services, which for the sake of this article is a service that provides contact details for a person.

Based on conversations with the project sponsors I began to suspect that at least the introduction of the ESB was a case of RDD, ie. Resume-Driven Development, development in which key choices are made with only one question in mind: how good does it look on my CV? Talking to the developers I learned that the ESB had introduced “nothing but pain.” But how could something as simple as the architecture in the above diagram cause such pain to the developers? Was this really another case of architect’s dream, developer’s nightmare?

Often a picture says more than a thousand words, but at the same time traditional architecture diagrams simply do not provide enough information. (I’ve argued this before, here and here for example.) The answer to my questions, and the anxious project sponsors’ question whether the project would deliver on time, lay in a different picture.

The picture should not only show the layering of the system in more detail than the architecture diagram but it should also visualise the complexity introduced by the ESB. It’s often possible to produce more detailed diagrams automatically from source code, configuration, or runtime inspection but in this case I couldn’t find a way to do that, which is which I drew the following:

In the centre column this diagram shows the layers involved, from the Struts action that needs to look up details by an ID down to the back-end that provides a “RESTful” service to retrieve the details. The top box contains the front-end layers while the box in the middle represents the ESB. The layers labelled in italics are frameworks or infrastructure software, the rest is project-specific code.

Key to this diagram are obviously the coloured representations of data travelling up and down the stack. Primitive types are grey, Java objects are purple circles, XML documents are shown as turquoise hexagons, and HTTP messages as green blobs. Mule messages, which are Java objects, are shown in dark purple. If one representation is wrapped in another it’s displayed inside the wrapper.

The diagram highlights that it’s a simple piece of data that travels all the way down the stack (actually further until it hits a relational database at some point) and, equally, that the data needed by the action is available in the right form from the service. All that the intermediate layers are doing is wrapping and unwrapping the data in various ways. In fact, out of the four layers of project-specific code three layers do nothing but wrapping. No wonder the developers felt some pain.

As a side note: the diagram also makes it obvious that the wrapping is asymmetric between the WsToRestHandler and the RestServiceWrapperImpl, which makes the point of this layer boundary even more questionable.

This diagram made people more interested in the cost of what had just been a simple, innocuous line in the architecture diagram; which was the point of the exercise. So, what would this diagram look like without the ESB? Easy:

Apart for the obvious simplicity the diagram highlights that every layer has a purpose and there is only one wrapping, which is not undone either. Would it surprise you if I told you that the project went live without the ESB?

Footnotes

1) This article is by no means implying that any of the mentioned technologies are useless or harmful. Their use was harmful in this case.

2) The pattern of wrapping and unwrapping the same data through multiple layers is not new, it was also quite common in early 3-tier architectures. (In fact, some people called their architectures n-tier; I guess to show that the number of tiers was somewhat arbitrary but large.) We had a term to describe this pattern: the wormhole pattern.

3) This project is not an isolated incident of a Wormhole ESB. I have personally seen several in the past years.

19 comments

  1. David Dossot

    1 July 2009, 16:33

    Besides resume boasting, what was the rationale behind this architecture? Support for an hypothetical drastic back-end change?

    I am having hard times imagining why someone would introduce an ESB, which is no small piece of infrastructure, if no usage of its value added services is made?

  2. Twitted by dancingmango

    1 July 2009, 19:54

    [...] This post was Twitted by dancingmango [...]

  3. Sriram

    1 July 2009, 20:16

    Nice diagrams. What tool did you use?

  4. RequestStreamMap | Design Website Blog

    2 July 2009, 06:07

    [...] Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting “Summon Request Stream Map“. [...]

  5. John D'Emic

    2 July 2009, 06:35

    A while ago I was involved in implementing a similar architecture. The project in question was a webapp with a SOAP API. This architecture was adopted in an effort to make us “eat our own dogfood”: if our API was good enough for our customers, then it should be good enough for us. This led to the artificial restriction that the webapp had to proxy all service calls through the SOAP API. The result was a marshaling mess similar to the above and project was ultimately late and canned. Additionally, the resulting API probably would have been of little us to anything but our webapp.

    I’ve conversely, however, found such marshaling layers useful when integrating data from disparate sources. In these scenarios, the disparate data is marshaled to a canonical format (ie, XML in the MuleMessage using the example above) and passed to a single class that knows how to persist it. As more data providers are added, transformers are added to move the data to the canonical format. Client applications, of course, can just hit the datastore directly and not proxy through anything else.

  6. Erik Doernenburg

    2 July 2009, 11:17

    @Sriram

    The tool I used to draw the diagrams is OmniGraffle.

    @David / @John

    The reason for the introduction of the ESB was that the content would potentially be consumed by third parties. In fact, I now remember hearing the words “eating our own dogfood” in this context, too; for the same reason John mentions.

  7. Wille

    2 July 2009, 18:53

    I’ve written at length over and over again of the ills of modern “Enterprise Architecture” and “Service Oriented Architectures”.

    Rather than just keeping it simple a lot of developers and “architects” seem to think that the sole purpose of software is to shuffle ones and zeroes through as many layers of “architecture” as is possible, regardless of whether those layers actually do anything other than just pass through.

  8. Holger Hoffstätte

    2 July 2009, 20:55

    Erik, you have no idea how much I enjoyed reading this. :)

  9. Andrew Perepelytsya

    3 July 2009, 03:34

    I don’t understand why so much noise? What I see is a clear abuse of common sense, be it with or without esb. What, are there no examples in IT?

    Were I to give my recommendation on their diagram – use the simplest thing possible for UI-to-backend communication, and add WS for external clients when needed. But routing everything through a WS… So, what’s wrong with an ESB here? Too much power in wrong hands?

    Andrew

  10. Ross Mason

    3 July 2009, 06:42

    I think its pretty funny that the technology was blamed here and not the architect (a conflict of interest for a ThoughtWorker I guess). Common sense should tell any architect that there is no need add a Web Service layer only to un-marshal the soap and convert it to a REST call and then make the actual call (two network hops). Given that this is all HTTP request response, just use HTTP. Just to be clear, Mule did not add these layers, the user did.

  11. Ross Mason

    4 July 2009, 01:24

    Hi Erik, I just posted about how to select whether to use an ESB for a project. Hopefully it might help some people avoid the pitfals of RDD and other misguided technology selection: http://rossmason.blogspot.com/2009/07/to-esb-or-not-to-esb.html

  12. From the Mule’s mouth - Musings from the MuleSource Experts » Blog Archive » To ESB or not to ESB

    6 July 2009, 18:42

    [...] There is an interesting post from ThoughtWorker Erik Dörnenburg with the unfortunate title “Making the ESB pain Visible”. Erik provides a real-world example of when not to use an ESB citing that – Based on [...]

  13. Erik Doernenburg

    8 July 2009, 14:43

    @Ross

    Great checklist, and I couldn’t agree more; when used for the right reasons an ESB like Mule can drastically simplify a project. Unfortunately, though, we still see a lot of architectures that wouldn’t make it past even the first point on your checklist. And it was those that I had in mind when I wrote this article.

  14. Shan Kandaswamy

    15 July 2009, 15:18

    Great Article…. It clearly is that the Architects who simple choose to implement ESB with no proper analysis is RDD. I got a very good idea on why we should not use ESB. Analysis still going on why we should use ESB.. Pointers welcome…

    Article twittered by Shan using bluez25 in twitter.

  15. Kamal

    10 August 2009, 16:17

    An interesting article, people need to choose the technologies like ESBs only when required. If someone thinks of using ESB no matter what the project requirement is; then definitely there is a problem.

  16. Colin Jack

    16 August 2009, 08:22

    Interesting post, I didn’t quite understand the bit about the asymmetry around the WsToRestHandler though.

    Also agree that the diagram here makes the cost involved much clearer.

  17. Ramu

    29 March 2012, 19:36

    Nice post.

    Another use case that I’ve seen is implementing business logic using Mule ESB where simple Template pattern was turned into many Mule models. Not a perfect fit. The argument was that whole application (integration use cases and business uses) should use Mule.

    Good architecture (right tools for the job) vs. RDD.

  18. Vaibhav Tyagi

    12 December 2012, 21:15

    Hi Erik,

    Nice Diagram. I think that you published one side of the problem or understood wrong.

    We have to go one level up to think about ESB usage. ESB is not just for publishing the services. There are lot of things required at Enterprise Level like Governance, Scaliblity, Security, Flexiblity, Maintaineablity, Multi Application Integration, Messaging Framework so we have to consider other factors before jump to the conclusion.

  19. Vaibhav Tyagi

    12 December 2012, 21:16

    Sorry instead of understood wrong I want to type i understood wrong.

Leave a comment