ESB and the Road to Indigo

by David A. Chappell

Related link:

Richard Turner, a Program Manager at Microsoft, had recently posted some questions regarding the usefulness of an ESB once Indigo is fully baked, and the WS-* stack becomes fully mature. Unfortunately his blogging system doesn't allow any response to be posted after a certain amount of time has passed, so he and I thought it would be best for me to post my comments here and he could point to it from over there....

Hi Rich,

I see that others have jumped in on your blog to help answer your questions regarding ESBs. I have a few more clarifications to add -

<Rich> “David Chappell of Sonic Software has written a good, albeit very Java centric book on ESB which I guess not surprisingly leans heavily towards discussing the notion of ESB from the context of the technologies available from Sonic and the Java platform.” </Rich>

Thank you for the complement on the book being well written.

Regarding your comments on the book being Java-centric… It also happens that all ESB implementations that I know of to date are built on some kind of Java technology somewhere. That happens to be an implementation detail. The book outlines the architectural components that make up an ESB. In the intro of the book I state that any vendor may use this as a guide to build their own implementation. And that implementation can be built using Java, C#, Perl, or Python for all I care provided that it conforms to the architectural principals as outlined in the book. Even when using an ESB that is based on Java, it doesn’t mean that you need to be a Java programmer to use it, because an ESB is based on a philosophy that a SOA should be configured rather than coded, and when you do need to write code there are a variety of non-Java based technologies to use (more on this in a bit). An ESB can be used in an all-Microsoft environment if the associated deployment tools and management framework makes the presence of a JVM a byproduct of the installation process that is hidden from daily operations and use.

That being said – It’s a heterogeneous world out there, and it’s a fact of life that Java technology exists in many IT environments. Java represents a set of standard APIs and interfaces, many of which have reached popular adoption. If you want to communicate with things that are built in Java then it’s a darn good idea to support it if you are able to. I hope I get that across well in the book (I actually say it somewhere in there). As an example, JMS is a well established standard for messaging, with broad industry support. However, it is only one of the MANY ways to integrate applications together through an ESB. However in the ESB book I don’t mention it any more or any less than other standard technologies like XPath or XSLT. In fact, I have an entire chapter which generically discusses messaging concepts such as store-and-forward messaging. At the end of the chapter is a small section on JMS and another equal amount of ink devoted to WS-Reliability and WS-ReliableMessaging. WS-Rel* is a perfectly valid way of linking applications together through an ESB, and will be much more relevant as it becomes commercially available in other platforms (like yours).

While you mention the book (thanks!), I think your comments are more directed at the whole notion of ESB’s, so I’ll focus on addressing that. Gartner defines what is referred to as a “multi-protocol” ESB. In a multi-protocol ESB, interfaces that are implemented in Java, C++, COM+, or C# are also equally valid ways of getting on and off the bus. In fact, such an ESB can mediate between different protocols, API conventions, and invocation styles. An application written to use the .NET stack can participate in a SOA across the ESB by sending or receiving a message using WS-ReliableMessaging, or by calling an ESB-supplied C# client. Somewhere on another node of the ESB, a J2EE appserver may be plugged into the bus using WS-Rel*, or through a JMS interface using a MessageDrivenBean. Meanwhile, in another part of an organization, applications that are accustomed to doing nightly batch processing based on bulk dump, load, and transfer of data may be doing so through the ESB using an FTP or file based interface, and taking advantage of the ESB’s mediation capabilities to break up the bulk data into its constituent parts for consumption and use by other endpoints on the ESB (using the aforementioned protocols and interfaces.)

Also, I agree with your assertion’s that ESB’s should embrace the WS-* specifications. An ESB should be considered a highly sophisticated, intelligent implementation backbone for the WS-* stack of specs (more on this later.) BTW, the final chapter of the ESB book is about how ESB’s and the Web Services stack of specifications are destined to evolve together.

You also brought up the question of “How, for example, would I add mainframe nodes, COM+ nodes, Web Service nodes etc. running on different platforms and technologies on top of JBI/Sonic/ without the use of adaptors and translators?”

How indeed? I’m guessing that your question lies in a misconception that an ESB does not provide adapters. In Gartner’s definition of a “single-protocol ESB”, I suppose that’s true. However in my opinion I think an ESB that only supports one interface style is severely limited. In Sonic’s case (just as an example), we provide over 200 adapters for packaged apps such as SAP and PeopleSoft, and also for protocols such as FIX and SWIFT.

<Rich>To my mind, an ESB is smart-plumbing to which to attach dumb nodes.</Rich>

One of the benefits of using an ESB is that it can connect, mediate, and control the interaction between a variety of systems without forcing an organization to adopt a single interface style or technology. Applications may participate in a SOA through an ESB using a variety of interface technologies such as Web services and application adapters, and also bridge to not-so-modern transports such as WebsphereMQ, TIBCO RV, FTP, and SMTP. The applications that are brought into the ESB using the FTP or file based interfaces are blissfully unaware that they are now participating in a newer infrastructure, because they just continue to dump and load their files every night. That is a great example of your “Smart ESB, and dumb nodes” observation.

An ESB provides an abstraction layer which separates the business process definition from the underlying physical messaging layer, protocol transports, and service endpoint bindings. Because this abstraction layer is based on process definitions built using modeling tools and configuration via a centrally accessible remote management layer, an application may be brought into the bus using FTP or file based interface today, and migrated over to a more direct interface style such as Web services or specialized packaged application adapter tomorrow, without having to go back and re-tool the surrounding application endpoints that interact with that application, or having to redo the process definition, or in most cases without the need to recompile or redistribute any code changes. This allows applications to be brought into a broad-scale SOA at their own pace, in a phased approach that allows the flexible scheduling of the IT resources responsible for doing it. This is one of the many ways that an ESB supports incremental adoption of a SOA.

Lastly, you asked this really important question – “What value, therefore, is an ESB [When web services smarts are built into the endpoints]?

I think there has been enough already said here by me and others on this thread, but just in case its not clear let me summarize the main points –

- An ESB is configuration-driven. Process definitions, service configurations, endpoint bindings, routing channel definitions can be configured and re-configured without any coding, and without any downtime. This provides post-deployment adaptability with no disruption to operations, no need to take services off-line, and no need to reboot servers.
- Even when the endpoints being connected are "Smart Nodes", an ESB provides value to the service invocations and message processing by enabling scalability in all directions. An ESB provides high-throughput, low-latency message based communication channels. Through transparent message server clustering, you can achieve increased capacity to handle any type of load of message based application data sharing and service invocation traffic where necessary. An ESB provides a scalable set of cooperating message servers which can be deployed across multiple platforms and managed through a single common management layer. These message servers, which act in cooperating software-based clusters, can be securely locked down in the DMZ and seamlessly route message traffic and service invocations to other message servers (using SSL and HTTPS tunneling with mutual authentication) and direct services both internally and externally. This creates a series of managed routes throughout your many firewall boundaries. As WS-RM becomes the prevailing standard for messaging, these message servers will become a scalable set of WS-RM SOAP message routers. Also, the independently scalable service hosting and invocation layer in the ESB architecture lets you dynamically deploy additional service instances to handle increases in processing load by the application and service endpoints. This lets you deploy and scale what you need for services independently of one another where and when you need them (see Ed Daniel’s link to my “ESB MythBusters” article excerpt). If the scope of you SOA scales up to hundreds or thousands of application and service endpoints, would you want them all to open up their own holes in the firewall and talk to each other directly, or would you want them to communicate through a set of well-defined tunnels? (See Figure below)


- If done well, the messaging layer of an ESB can provide continuous availability of the communication channels by providing automatic failover on server and communications channels, without disrupting operations or enduring manual recovery processes, and without requiring hardware-based clustering techniques.
- Stateful orchestrations can be combined with stateless itinerary based routing to provide a flexible model for implementing and managing self-directed business processes that can be highly distributed and capable of operating independently of other parts of the ESB, without the need for centralized servers that could introduce processing bottlenecks and act as a single point of failure.
- Secure access with global reach, where distributed processes and services may be defined and deployed to any location, from any location. An ESB allows the local autonomy of control over service configurations and their interactions, including access rights using local security domains. Because the ESB provides a purposed well defined conduit between services, yet provides loose coupling across all applications and services that are globally accessible across the bus, IT organizations can build SOA domains at a departmental or business unit level, yet selectively open up channels with other domains which include other business units, and extending into B2B collaborations with business partners. This can work seamlessly and reliably across Internet, satellite or other WAN links.

As web services smarts are built into the endpoints, then this will make ESBs even more practical for use. As Indigo is rolled out, for example, applications built on that platform will be by their nature message driven and better suited for loose coupling into an ESB. An ESB will still provide the mediation between disparate data formats and invocation styles, and provide intermediary services such as data transformation and content based routing in an independently scalable service model

I hope this clears a few things up. Feel free to give me a shout if you have any further questions.

Comments are welcome here from others as well. Please keep the Java vs. MS bashing stuff to a minimum. This is really about architecture.


2005-04-23 07:48:07
The Future of the ESB Market
Please see my response:
The View From The Bow: The future of the ESB market
2005-04-28 10:25:13
The Future of the ESB Market
Anne brings up some interesting points. There is some overlap between ESB, Web Services Management, and pure-play Web Services Platforms. However, I don't think anyone should take this as spelling gloom and doom for ESB. She is pointing out that there is going to be some consolidation in functionality and shakeout in the number of vendors as the lines between these different product categories blur. I'm OK with her conclusion "In the long run, maybe 4-5 players will survive. My bets are on Sonic, Systinet, TIBCO, and webMethods." :-)

One place where I disagree with her and with Rich Turner is on the view that the addition of more robust WS-* support such as WS-Rel* and WSS into the application platforms will greatly diminish the value of an ESB. An ESB is based on a philosophy that the entire design center around a SOA is about the "whitespace" that is between the endpoints--not just the SOAP stack at the endpoints themselves.

Process coordination, routing of service invocation based on the content of the message that represents the service invocation, data transformation, auditing and process tracking are all integral parts of an ESB implementation. Having WS-Rel* and WSS built into some of the application endpoints doesn't necessarily give you all of that by default. An ESB will still provide value as the common infrastructure between the various implementations (and interpretations) of WS-* specifications.

In fact, an ESB can even provide a degree of insulation from the uncertainty that exists today with overlapping standards initiatives. Because an ESB is all based on standards, and is capable of evolving with standards as different ones become predominant, its a much safer bet.

And what of the application endpoints that aren't built in .NET or some other "superplatform"?
Can all IT departments out there safely say that they have a roadmap that shows all applications will migrate to .NET, WLS, SAP etc or else?