Making Agile SOA work

Over the past few years the acronym SOA has been associated with a
 number of promises: building large scale systems in a rigorous way, ease 
of integration through standards adoption, separation of concerns of
 processes through orchestration tools via declarative rules, even 
’programming’ from analysts not to mention leveraging legacy systems 
that get new life breathed into them because they can now play in a 
wider world. Big promises all. With all the hype there have been some
 casualties along the way. Certainly the road to a successful SOA project
 is filled with challenges, many of which lie around stakeholder buy-in,
 being able to apply governance both deep and wide through the life-cycle
 and across the enterprise, pure complexity, integration testing issues,
 the old chestnut of what exactly a transaction means in an enterprise 
setting! As technologists we often focus on the technical challenges 
around projects of this kind around standards and the methods we use to
 develop the software, the development of service patterns, applying
 different methods to formalize the software life-cycle (such as a RUP
 versus Agile debate for systems of this type).

The problem with promises is that they quickly have to have to show benefit on the balance sheet, this is especially true of large strategic projects; investment is often nervously given. Successful large projects only become so because they remain accountable from inception onwards and to ensure this rigorous software methods have to be employed because SOA means integration of either new or wrapped legacy systems so the complexity of the entire system is multiplied by the complexities of all it’s parts – plus the dependencies between those parts. Indeed the dependency issue can force a counter agile approach. The key design tenet is to supply self-contained sentient service components. They can wrap the desired functionality; supply both technical and business level monitoring information so that higher-level services can consume them. A greater level of abstraction often enables the main integration points to grow more organically rather than with up-front design.. Not to say this wider reaching design doesn’t or shouldn’t occur far from it but iterative design feeds into it rather being the recipient of it. From a technical viewpoint this enables development to begin earlier and thus testing plus it allows mapping to potentially happen internally through the abstraction levels which reduces the need for pan-enterprise standardization. Whilst this does not mean a common meta model is not required it does mean it can grow incrementally and/or can be mapped from/to by the service parts. You can see a common thread of agile design feeding wider designs here. The approach f building services from smaller components is often called Service Component Architecture. So now to governance of the design process: This is the absolute key to success. It is unlikely that all sub parts of the system have the same deadlines and aren’t subject to localized pressures of their own. At least by breaking service contracts into smaller self managed parts a more agile and thus more accountable gradual delivery can begin. This isn’t to say that the more traditional Service Inventory Analysis is given up, far from it but logical views of a large complex system are all too often too far removed from reality and analysis often gets diluted away when the technical designers pragmatically begin work on the system. So the only way to succeed is to iteratively take business process models into your raw service components via test driven development until you have a full service inventory. This allows individual deliverables to continue. We are starting to see design methods such as RUP/OpenUP adopt more agile practices and agile methods being applied in a wider enterprise setting. It should also be noted that wider enterprise architectures methods such as TOGAF (The Open Group Architecture Framework) are starting to encompass agile techniques. A mistake is often made that agile means ad-hoc, it does not. It actually enforces a higher degree of rigour because to accountability is much higher. Various software engineering practices have attempted to increase ROI through re-use, de-coupling, component coherence, abstraction and so on, we have talked about design applications and issues of control but support tools are always going to be required, not just developer tools but service discovery tools are required so that components can be found, understood and (re) used – all too often the wheel is reinvented sometimes through ignorance.

So to the future: Because there are few formal definitions for what SOA actually is it will always be a chaotic path for some developments. Whilst governance and standards are vital for any interchange and integration project a stepwise but formal iterative process mixed with the higher level views (either top down or bottom up) will provide a clearer path because of the frequent reset points. A final example of this is managing transactions in a SOA environment. Because a failure in a synchronous system usually involves a rollback operation undoing what was done during the operation, within a workflow system we may have operations paused for long periods of time then found in error much later so one needs to design skeletal compensation statements right at the beginning. I recently designed a system that controlled a number of physical devices some of which may respond with an error condition hours or even days later therefore a classic rollback is out of the question so a set of cleanup or compensation actions has to be set up. All of this can often be identified and delivered within these smaller components because you are local to the problem domain quite quickly over a few iterations.