One of the engineers from Microsoft Anandeep Pannu, has written very excitedly about participating in Qpid.

In July Sam Ramji announced a sponsorship for the Apache Software Foundation. It kind of signaled about their intent to get their feet wet with playing an active role in open source. It seems that most large vendors have realized the importance of participating and promoting open source on one hand and also harnessing the power of open source on the other hand. IBM for instance played this strategy very well over the last decade or so. This is very good PR for the Qpid project and hopefully attract more participants and end users to the project. Definitely looking forward to working them in Qpid and the AMQP working group.

Without any further do here are the reasons in no particular order.

1. Difficulty in identifying and dealing with failures.
When communicating between processes failures can happen at many levels. Dealing with them is not trivial. Of course you rely on frameworks based on technologies like RMI, CORBA, COM, SOAP, AMQP, REST(is an architectural style not a standard) etc to handle these. But the fact remains that you still need to clearly think about these cases and handle these situations properly.

For example if we consider a simple interaction between two processes on different computers, the following failures can happen.

• Failures that occur within the process that initiates the communication (sending the message or invoking the RPC call).
• Failures between the time the process hands over the request to the OS and the OS writing it to the network.
• Network failures between the time it takes to transmit the packets from one computer to the other.
• Failures between the time the OS on the receiving end receives the packets and then handing it over to the recipient process.
• Failures that occur when the recipient process tried to process the request/message.

Sometimes the framework you use, is unable to/may not report all these error cases. Sometimes when the error is reported, it may not contain enough information to figure out at which level the error occurred.
Did it reach the remote computer? if so how far up the stack did it go?. If the receiving process got the request or message did the error occur before or after the request/message was processed?
In some cases where idempotency is built into the the receiving application or the framework/protocol (ex a message client that detects duplicate messages, or doing an HTTP GET) a simple retry maybe ok. In some cases Idempotency and retrying maybe expensive or difficult to implement. In such cases careful thought needs to be given on how these different errors are identified and handled.

2. Achieving consistency in data across processes.
One of the hardest problems in programming distributed systems is achieving a consistent view of data across the processes. When one processes updates some data, you need to replicate them across the other processes, so if any other process decides to operate on the same set of data, then it is doing so on the most current copy.
Lets look at two examples.

Assume a global banking application for ABC bank. A customer goes to a branch in New York, US and deposits money to an account. A few moments later his relative in London, UK does a withdraw on that account. Due to latency there is obviously a time lag before the process in London, UK sees the updated amount in the account.

In an online trading system, a user in NY places an item for sale. The transaction is updated on the closest data center which is in Boston. A few moments later another user in LA is searching for the exact same item and is served off a data center in Phoenix. The user in LA may or may not see the item due to the latency involved in replicating the data across

For example 1 strong consistency is required, while for example 2, you could get away with weak consistency, for example by setting an SLA that says data is valid within a 5 min time window.
This is not an easy problem to solve and this area itself is a subject on its own. Wener Vogels wrote a nice peice on this called Eventually Consistent which is worth reading.
Of course there are specialized frameworks/libraries that can handle this for you. But still there is no escape for you and you pretty much need to have an understanding of the pros and cons of various approaches, failure modes etc.

3. Heterogeneous nature of the components involved in the system.
A distributed system may contain components written in a variety of languages deployed across machines with different architectures and operating systems. Needless to say that this poses certain challenges (especially integration, interoperability issues) when implementing the system. A whole range of standards/technologies were presented to solve these issues, including but not limited to CORBA, SOAP, AMQP, REST (is an architectural style not a standard) and RPC based frameworks like ICE, Thrift, Etch etc. Anyone who has worked with these technologies knows that neither of these are trivial to use nor provide a complete solution in every situation.

If anybody has read the recent posts by Steve Vinoski and the discussions around it would realize the issues/challenges surrounding RPC. The following paper discuss the impedance mismatch problems when working with IDL based systems. The issues with type systems and data formats are not limited to RPC only. When using a message oriented approach like SOAP (doc lit style) or AMQP you will end up tunneling data thats not supported by the protocol as a string or a sequence of bytes. When using REST you would need to represent your resource in a format the requesting application understands/supports, which maybe quite different from the native format.

Again not an easy issue to deal with no matter what technology or framework is used. As an architect/developer you need to understand these issues and deal with them accordingly.

4. Testing a distributed system is quite difficult.
This is arguably one of the hardest aspects of developing a distributed system. Verification of the behavior and impact of your code in the system is not easy.
There are many aspects that needs to be tested, and doing so before every checkin is not a fun task at all. Running some of these tests before every checkin is not practical. But its a good idea to run them nightly and some tests during the weekend. Here are some of the areas that needs to be tested (I plan to write another blog entry elaborating on the testing aspects).

• Functionality testing (can be covered with well written unit testing)
• Integration testing - you need to test the distributed system as a whole with all the components involved
• Interoperability testing - this is crucial when heterogeneous components (different languages, OS) are involved, and is quite different from integration testing
• TCK compliance - If your system is based on standards/specifications, then you need to ensure that you haven’t broken anything w.r.t compliance
• Performance testing - to ensure that your changes haven’t accidentally caused a degradation in performance
• Stress testing - to ensure that your checkin hasn’t accidentally caused any stability issues - ex increased chance of deadlocks when the load increases
• Soak testing - to ensure that your checkin hasn’t caused any longevity issues - ex a memory leak thats manifested after a couple hours, days

Most often than not developers cut corners in their testing as running these tests are tedious and time consuming. Also these tests need to be run regularly to catch issues in a timely manner and the best way to tackle this issue is to automate as much testing as possible. There many options with continuous build systems like cruisecontrol or using a plain old cron job.
Functionality testing, TCK compliance, certain types of integration and interoperability tests can be run periodically.
In most organizations test machines are just lying around doing nothing during the night (unless around the clock testing is done with development centers in different time zones.). Instead of wasting computing cycles, you could automate test suites to run during the night. More time consuming integration and interoperability tests, performance, stress and soak testing can be done nightly, while more longer duration soak testing can be scheduled to run during the weekends.

While testing is a tough issue for any type of system, distributed systems have a lot more failure points which adds to the complexity.
Getting these tests right to cover these failure points and executing them needs a lot of careful thought and planning.

5. The technologies involved in distributed systems are not easy to understand .
Distributed system are not easy to understand. Neither are the myriad of technologies used in developing these systems.
Most folks find it difficult to grasp the concepts behind these technologies. If you look into the discussions and misconceptions surrounding REST you can understand what I am trying to get at. CORBA was not an easy spec to understand, so is WS-* or AMQP. While it is true that you don’t need to understand everything to develop using them, you still need at least a reasonable understanding to figure how to tackle some of the above mentioned issues. Frameworks based on these technologies are touted as the cure for these problems. Sure they could help, but it still does not shift the burden away from you.
To compound the issue all sorts of vendors keep touting their technology/framework as the next silver bullet. No matter what vendor you use, at the end of the day you are still responsible for getting it right. And it is not an easy task. You need to face the reality that distributed systems are hard and that you cannot hide every complexity behind some framework.

Watch the WSO2 blog space or the Axis2/C website for a tutorial/blog post from Danushka.

1. The current code would have reached a stage where it is impossible to do any more modifications without breaking something else
2. The requirements have changed so much that the current architecture/design cannot handle it without a redesign
3. The current application doesn’t/will not scale, perform well enough as it wasn’t designed to handle the current load/anticipated growth
4. The folks who worked on the code are no longer there and nobody knows what the code really does (or what it was supposed to do)
5. You don’t really like the current way it is implemented and think that there is a better way to do it using framework X or library Y

While sometimes restructuring could be done easily, but 90% of the time it is not a trivial task. When the frustration gets to you, you may have even entertained the idea of rewriting an application/module/section from scratch. Is this really a good idea? Sometimes this maybe the only option, but most of the time this may end up being a bad idea due to a variety of reasons.

• One of the biggest mistakes is to throw away the old code without any due consideration simply based on the assumption that the old code is bad and we are going to write much better code.

  Throwing away the old code (especially if it was in production) means, you are throwing away months (or years) of tested, battle hardned code that may have had fixes for bugs that you aren’t even aware of. If you don’t take this into account, the new code you write may end up showing the same bugs that are already fixed in the old code. This will waste a lot of time ,effort, knowledge gained over the years

• This method or class is ugly, lets throw it away and write again.

  That odd looking method or that badly written class may have some fix for a race condition or an optimization that one of your customers is depending upon. Discarding that code as crap without really understanding whats going on may end up with dire consequences for your team.

• Not paying enough attention to the existing unit tests/ test frameworks when building the new system.

  These tests were added for a reason, possibly in response to a bug or some sort of intermittent failure that was reported on a production system. These failures/errors may have been reported way before you joined the company. Sometimes none of the current team members are aware of all the issues. So discarding the test code means you are throwing away years of hard work and knowledge

• The current code is way behind all the cool technology we have today. The new framework X can do things a lot more elegantly, so lets re write.

  This is by far the worst mistake. If something is not broken then why try to fix it?. The code is ugly, or technology used is not cool are not good reasons. Simply bcos the style or structure of the code is not according to your personal preference is not a reason at all for unwanted restructuring .

I have been guilty of doing most the of the above mentioned points and through hard lessons I have realized that,

• The best approach for restructuring starts with taking stock of the existing code base and tests written against that code. This will help you understand the strengths and weaknesses in the existing code, so you could ensure that you preserve the strong points while avoiding the mistakes.
• It is best to reuse as much code as possible, bcos no matter how ugly the code is, it has been tested/reviewed etc.
• Incremental changes are better than one massive code change. Incremental changes allows you to gauge the impact on the system more easily through feedback from tests etc. It is not fun to see 80 test failures after you make a change and can lead to frustration/preasure that will in turn result in more bad decesions . A couple of test failures is easy to deal with and provides a more managable approach.
• After each iteration it is important to ensure that the existing tests pass. Analyze why the tests are failing, and make modifications/add new tests if nessacery if the existing tests are not sufficient enough to cover the changes you made. Failure to do so can result in a lot of pain down the road.
• Avoid the temptation to rewrite everything. Personal preferences and ego shouldn’t get in the way. If something works then don’t change it.
• Remember that humans make mistakes and restructing will always not garuntee that it will be atleast as good as the previous attempt. I have seen and have been part of several failed restructuring attempts

Having said all of that, sometimes you have no option but to rewrite from scratch. But IMHO that should be your last resort.

Documentation is broken into two sections.
It is important to read and understand the pros and cons for each option.

If you have any questions/suggestions please post on qpid-users@incubator.apache.org
And/Or open a JIRA at http://issues.apache.org/jira/browse/qpid
Your feedback is most welcomed.

1. Configuring the JMS transport using Apache Qpid M2.1 release.
   • This version is the latest stable release and supports the 0-9 version of the AMQP protocol.
   • You can download it from here
   • It is recomended that you use the Java broker.
   • This is the recomended production version.
2. Configuring the JMS transport using Apache Qpid trunk. (with the usual caveat that it is a less stable and a moving target).
   • The trunk supports the 0-10 version of the AMQP protocol and the next release is M3.
   • You can build it from source. https://svn.apache.org/repos/asf/incubator/qpid/trunk/qpid
   • Has the following enchancements in the JMS client over M2.1
     1. You can specify multiple binding keys per destination. i.e you could bind a queue to a particular exchange using several binding keys there by allowing you to listen from multiple sources.
     2. You can bind your queue to any exchange type using the binding URL.
     3. You can set the following options per JMS connection.
        • maxprefetch - The maximum number of pre-fetched messages per destination
        • sync_persistence - When true, a sync command is sent after every persistent message to guarantee that it has been received by the broker.
     4. A new blocking transport (as an alternative to MINA) that gives much better latency, memory and increased throughput in many cases.
     5. Currently only the c++ broker supports the 0-10 version and only works on linux.
     6. In the pipeline
        • Java Broker - expected to be in M3 release. Currently being refactored
        • C++ broker - solaris and windows port are in progress (likely to be in M3 release)

1.0 Configuring the JMS transport using Apache Qpid M2.1 release

• 1.1 Qpid provides a convinient properties file based JNDI mechanism.
  • For provider URL please specify the location of the jndi.properties file.
  • The following link provides documentation for the jndi.properties file. JNDI How To
  • The connection URL format is given here. Connection URL format

  
<parameter name="myTopicConnectionFactory">
<parameter name="java.naming.factory.initial">org.apache.qpid.jndi.PropertiesFileInitialContextFactory</parameter>
<parameter name="java.naming.provider.url">file:///opt/workspace/sandbox/axis2-1.4</parameter>
<parameter name="transport.jms.ConnectionFactoryJNDIName">TopicConnectionFactory</parameter>
</parameter>


• 1.2 If you didn’t specify a custom destination for your service via the “transport.jms.Destination” parameter then the JMS transport will create a queue with the service name as its name. In AMQP world the above translates as follows. A queue will be created with service name as its name and will be bound to the amq.direct exchange with service name as the routing key
• 1.3 If you specify a custom destination name via transport.jms.Destination in your services.xml then Qpid will look that up in the jndi.properties file and create queue based on the definition given in the file.
  The binding URL format is given here. Binding URL format

2.0 Configuring the JMS transport using Apache Qpid trunk
In addition to whats described above you can do the following.

• 2.1 Specifing multiple binding keys per destination
  
  destination.wetherServiceQueue=topic://amq.topic/WeatherQueue?bindingkey='weather.us.*'&bindingkey='weather.ca.*'
  
  The above will create a queue named WeatherQueue and will bind it to amq.topic with ‘weather.us.*’ and ‘weather.ca.*’ as the binding keys. The broker will then route any messages that matches the above patterns into WeatherQueue.
• 2.2 Binding your queue to any exchange type the broker supports by using the binding URL.
  
For example if you want to bind your queue to a fanout exchange
destination.top10StockQueue=fanout://amq.fanout/WeatherQueue

  
• 2.3 Setting prefetch limit and sync_persistence per JMS connection
  
  amqp://username:password@clientid/test?brokerlist='tcp://localhost:5672'&maxprefetch=2000&sync_persistence=true

  
• 2.4 Using the new transport
  
  Set -Dtransport=io when starting Axis2 or Synapse.

(more…)

Intent/Problem Space

• Both Synapse and Camel have similar intents as Integration, Mediation and Routing Engines.
• Synapse also positions itself as an ESB, treating your entire network as the bus.
• Camel/ServiceMix is advertised as an ESB.
• Both have routing,transformation,mediation,enrichment,validation, logging ..etc.
• Initially Synapse was more focused on Web Services mediation. Since then, it has moved out of the Web Services umbrella and is trying to position itself as a general purpose integration/mediation/routing engine.
• Synapse by design have a proper abstraction layer where you can adapt (implement) Synapse on top of any Environment. It is one of the least known aspects about Synapse. Most folks assume Synapse can only work on top of Axis2. I have implemented the Synapse environment interface inside the Apache Qpid broker to leverage its mediation/routing capabilities.
• Camel on the other hand was designed to work on different environments from the begining. Camel can be used inside ServiceMix, ActiveMQ etc.

I haven’t experimented enough with Camel to compare performance, flexibility,extensibility or robustness with Synapse. Perhaps that would be an interesting topic to touch sometime later

Messaging Model

• The Synapse messaging model is based on SOAP. The current implementation of Synapse is powered by Axis2 and takes any type of non SOAP message and creates a fictitious SOAP message and pumps it through the Synapse engine.
• Camel has a neutral messaging model which I think is nice.
• However you can implement your own Synapse MessageContext as well. I have created an AMQP based messaging model for Synapse for my work on embedding Synapse as an AMQP Exchange.
• Synapse could also use generics to make it’s messaging model neutral. (I hope to make a proposal to this effect after my AMQP work).

Synapse Configuration vs Camel Context

• Both use very similar models for representing a runtime instance of the rules/endpoints/registry ..etc.
• Synapse has a XML based DSL for configuration.
• Camel has a java and XML based DSL for configuration. You can use Spring to configure the Camel Context as well.
• You can also use Synapse programatically all though we haven’t provided a clear API as such to the user community. I think such an API would be a valuable addition to Synapse.
• There is support for configuring Axis2 with Spring, so I assume you can do the same for Synapse very soon (if not already).
• Both supports the concept of a Registry. The registry mechanism is pluggable in both projects.

Synapse Mediators vs Camel Processors

• A Synapse mediator, mediates a message as it passes through. It could be a transformation,validation, logging, audit ..etc.
• You can chain mediators into sequences and combine different sequences to create a processing engine.
• Camel processors does more or less the same.
• In both Synapse and Camel you can write you own mediator by implementing the Mediator or Processor interfaces respectively.

Synapse Script Mediator vs Camel Predicates/Expressions
Again they both provide pluggable scripting support. Not much to be said there.

Endpoints/Transports

• Both represents endpoints via URIs.
• Camel and Synapse supports more or less the same set of transports with the following exceptions.
  • Camel has additional support for JBI,MINA, XMPP.
  • Synapse supports Non blocking HTTP, AMQP (native not via JMS).
    James Strachenanswered : “Camel supports non blocking HTTP too”
  • In Synapse you could switch transport. Ex incoming is JMS and outgoing is HTTP (does Camel have similar support?).
    James Strachen answered : “you can use Camel to do protocol switching from any protocol to any protocol with whatever EIP patterns in between (e.g. Message Translator etc).”
  • Axis2 has support for XMPP, so if you really want to you could leverage that in Synapse.

Enterprise Integration Patterns

• Camel has support/good documentation for Enterprise Integration Patterns (based on Gregor Hopes book).
• All though Synapse doesn’t directly say so, you can easily do the same. In fact it does have some built in support similar to Camel for some patterns while other patterns can be done with a little bit of work. I may in the future compile an article on how to do so with Synapse.

Embedability

• Camel has a programmable API as described above and I quite like it. It would be nice if Synapse can do something similar.
• Camel can be used inside ServiceMix, ActiveMQ ..etc. I have managed to embed Synapse inside an AMQP broker albeit with some code changes.
• It would be nice if you could embed the mediation engine in Synapse in other projects in a trivial way and also provide some documentation on it.
• One nice thing about Synapse is that it can be deployed as a module inside Axis2 or as war file inside Tomcat. Couldn’t find any documentation on easy Camel integration with Apache CFX
  James Strachen answered : “its trivial to deploy Camel inside any Spring or WAR application or as an OSGi bundle, working great with Spring Dynamic Modules”

WS-* QoS Support

• Synapse has Built in support for WS-* QoS features (adding/removing security, reliability ..etc) when running on top of Axis2.
• Does Camel have a similar strategy with it’s Apache CXF support? ( I couldn’t find any information in the Camel site about this)
  James Strachen answered : yes Camel has a similar WS-* strategy; using CXF to support the SOAP/WS-* protocols on any endpoint

Throttling/Load balancing/Policy based access and failover support for endpoints

• Synapse has good support for applying throttling , failover support and load balancing to endpoints.
• Synapse also can apply policies to enforce restrictions for better manage your endpoints.
• Not sure if Camel has any support for this area.
  James Strachen answered : “yes Camel has support for load balancing, throttling, resequencing etc. See http://activemq.apache.org/camel/enterprise-integration-patterns.html”

Clustering support

• Synapse allows mediator state to be replicated across the cluster.This is based on the Axis2 clustering support.
• This is nice when you want to implement stateful mediators, like a session mediator.
• Again haven’t come across any documentation in Camel on this area.
  James Strachen answered : “for stateful replication, we rely on ActiveMQ’s Message Groups feature or using EJB3″

Payload Conversion

• Synapse can covert from one message format to another via built in support for POX, SOAP and JMS.
• Camel provides a Type Converter interface (and some default implementations) to convert message payloads from one type to another when routing.
• Similarly in Synapse, for other payload types you could write a custom mediator to do the job.
• Also using the transport switching mechanism you can do some trivial things like converting a Qpid/AMQP message to an ActiveMQ message using the JMS transport. Sort of like a message bridge.

Summary
Both projects seems interesting and has some overlapping features, but different direction/focus and thought process behind them. IMHO it is nice to have choices in open source. Each community can learn from each other and strive to provide a better experience to the end user.

One of the key points he mentioned was that people often ignored latency and tried to work around it instead of embracing the reality. Which is exactly the second fallacy of distributed computing - Latency is zero. Understanding this reality is important especially if you are involved in systems that are distributed geographically. Here are some of the tips I noted down.

• Try to serve users from where they are located.
  Move latency from users into your network. This will move the complexity into your system, but it will improve the user perceived performance
  Ex: If you serve your European customers from a data center in US, your customers will experience some delays. If you add a data center in Europe then it will improve the customer experience, but now it will add more complexity into your system as you would need to deal with consistency between the data centers ..etc
• Co-locating your services is a good strategy, but it may not be possible all the time. So always think about how you would architect your system or components so there are no issues if you move them apart from 10ms to 100ms.
• Don’t couple your system to the hardware or network topology. Upgrading might be a nightmare
• Trying to achieve global data consistency limits options
  Think about what your business can tolerate when it comes to inconsistent state? Trying to achieve global consistency can make things very complicated.
  Keep your users in one data center. Bouncing them between data centers will force your to maintain global consistency. Tell them that the data is valid within x secs.
• Prioritize according to user needs
  Here is a nice example given by Dan. SLA for seeing the payments due on an invoice is 5 mins and the customer never complains:)
  SLA for seeing money in a sellers account is 10 secs, or else the users get a bit upset.
• If you are recovering from a failure, reduce serviceability until your system gets back to normal state. This will prevent overloading the system and increasing latency.
• Use distributed transactions carefully
  Here is an example.
  • Component C in the first configuration is complicated as it needs to ensure both data bases are updated. DB2 can be in a different geographical location and updating it can add a lot to latency.
  • In configuration two, C is smaller and simple. We can introduce other components to process the data without impacting the user perceived performance for the client.
  • Another advantage is that C can do a transaction even when DB2 is down.