Making Agile SOA work

Over the past few years the acronym SOA has been associated with a
 number of promises: building large scale systems in a rigorous way, ease 
of integration through standards adoption, separation of concerns of
 processes through orchestration tools via declarative rules, even 
’programming’ from analysts not to mention leveraging legacy systems 
that get new life breathed into them because they can now play in a 
wider world. Big promises all. With all the hype there have been some
 casualties along the way. Certainly the road to a successful SOA project
 is filled with challenges, many of which lie around stakeholder buy-in,
 being able to apply governance both deep and wide through the life-cycle
 and across the enterprise, pure complexity, integration testing issues,
 the old chestnut of what exactly a transaction means in an enterprise 
setting! As technologists we often focus on the technical challenges 
around projects of this kind around standards and the methods we use to
 develop the software, the development of service patterns, applying
 different methods to formalize the software life-cycle (such as a RUP
 versus Agile debate for systems of this type).

The problem with promises is that they quickly have to have to show benefit on the balance sheet, this is especially true of large strategic projects; investment is often nervously given. Successful large projects only become so because they remain accountable from inception onwards and to ensure this rigorous software methods have to be employed because SOA means integration of either new or wrapped legacy systems so the complexity of the entire system is multiplied by the complexities of all it’s parts – plus the dependencies between those parts. Indeed the dependency issue can force a counter agile approach. The key design tenet is to supply self-contained sentient service components. They can wrap the desired functionality; supply both technical and business level monitoring information so that higher-level services can consume them. A greater level of abstraction often enables the main integration points to grow more organically rather than with up-front design.. Not to say this wider reaching design doesn’t or shouldn’t occur far from it but iterative design feeds into it rather being the recipient of it. From a technical viewpoint this enables development to begin earlier and thus testing plus it allows mapping to potentially happen internally through the abstraction levels which reduces the need for pan-enterprise standardization. Whilst this does not mean a common meta model is not required it does mean it can grow incrementally and/or can be mapped from/to by the service parts. You can see a common thread of agile design feeding wider designs here. The approach f building services from smaller components is often called Service Component Architecture. So now to governance of the design process: This is the absolute key to success. It is unlikely that all sub parts of the system have the same deadlines and aren’t subject to localized pressures of their own. At least by breaking service contracts into smaller self managed parts a more agile and thus more accountable gradual delivery can begin. This isn’t to say that the more traditional Service Inventory Analysis is given up, far from it but logical views of a large complex system are all too often too far removed from reality and analysis often gets diluted away when the technical designers pragmatically begin work on the system. So the only way to succeed is to iteratively take business process models into your raw service components via test driven development until you have a full service inventory. This allows individual deliverables to continue. We are starting to see design methods such as RUP/OpenUP adopt more agile practices and agile methods being applied in a wider enterprise setting. It should also be noted that wider enterprise architectures methods such as TOGAF (The Open Group Architecture Framework) are starting to encompass agile techniques. A mistake is often made that agile means ad-hoc, it does not. It actually enforces a higher degree of rigour because to accountability is much higher. Various software engineering practices have attempted to increase ROI through re-use, de-coupling, component coherence, abstraction and so on, we have talked about design applications and issues of control but support tools are always going to be required, not just developer tools but service discovery tools are required so that components can be found, understood and (re) used – all too often the wheel is reinvented sometimes through ignorance.

So to the future: Because there are few formal definitions for what SOA actually is it will always be a chaotic path for some developments. Whilst governance and standards are vital for any interchange and integration project a stepwise but formal iterative process mixed with the higher level views (either top down or bottom up) will provide a clearer path because of the frequent reset points. A final example of this is managing transactions in a SOA environment. Because a failure in a synchronous system usually involves a rollback operation undoing what was done during the operation, within a workflow system we may have operations paused for long periods of time then found in error much later so one needs to design skeletal compensation statements right at the beginning. I recently designed a system that controlled a number of physical devices some of which may respond with an error condition hours or even days later therefore a classic rollback is out of the question so a set of cleanup or compensation actions has to be set up. All of this can often be identified and delivered within these smaller components because you are local to the problem domain quite quickly over a few iterations.

SOA method melee

Much of the talk about SOA revolves around undelivered promises or companies reticent to commit. Here I present a way of working that allows a layered approach for those who need to at least assess the risk that may lay ahead. One of the greatest challenges that faces SOA adoption is what to do first; A recent report suggested that the majority of companies that thought about SOA adoption simply weren’t moving yet just because they want to watch the market. This is largely because with the best will in the world people don’t know what to look for, we hear about bottom-up, top-down, WS-*, registries and so on and so forth. In actual fact SOA itself does not have a formal definition, it’s interesting asking different people what they think often each assumption has a valid counter. The more complex SOA projects are often strategic, pan enterprise and involve issues of Geography and time. The only way to begin that SOA process is to get enough information at the top level to convince your CIO to invest and to do that you need enough understanding of each prospective part even at a high level.

This brings us to how you go about the process, what method do we use? This is usually when the wars begin! I have found from experience (not just in SOA – but going right back to CASE days) that one always to pull bits from different methods and make them work for you. A common mistake is to underestimate complexity with any size integration problem as that is all SOA really is – integration and maybe control processes over that. Most development methods where meant for a single project or at least for where you have the same level of control of all aspects of the development. The first point that I always encounter is lack of a common meaning or even agreement on meaning of terms, the truth is you need to begin with a taxonomy at the same time as you start to define services which is often people’s starting point or service discovery. This is totally compliant with Design by Contract.

Often when left to a bottom up approach this facet is missing and often discovered too late. This stage is your metamodel and it can be done incrementally. I does not have to be full blown MOF comliant design but it pays in spades to have a common description of terms. I like to do this step in an agile way informally identifying legacy systems to wrap or new services using the Zachman Framework iteratively (going down in layers through design iterations) as this is an excellent communication tool when it comes to reporting your initial findings back to the stakeholders. It’s important to have four things beginning at this point, as mentioned your early service discovery, your metamodel description (so each system can agree on types or how to transform them), developing a CIM with the business analysts and most importantly your error conditions. At this stage it’s vital to identify your exception conditions as asynchronous interconnects and transactions do not work like their synchronous relatives. One needs to design skeletal compensation statements right at the beginning. Let me give you an example, I recently designed a system that controlled a number of physical devices some of which may respond with an error condition hours or even days later therefore a classic rollback is out of the question so a set of cleanup or compensation actions has to be set up. All of this can often be done quite quickly over a few iterations and presented in a container this is a useful and simple container for the rest of the lifecycle although you may wish to choose a deeper method later, for example I would happily use an XP process to define the initial services which are held in and communicated through Zachman, then within each service moving into some MDA techniques such as defining a PIM and feed this iteratively into your effort to get the meta model off the ground (MOF), so within Zachman you can define what, where, how, who, when, why iteratively through layers of abstraction/detail with a common vocabulary so out of this comes your highest level service definitions using a common typing system, but that’s the easy bit because two important aspects can be added at this time your SLA definitions and your compensation procedures for asynchronous error conditions, and then later your KPI’s. So from a few simple iterations you have a framework that you can apply level by level per iteration to identify services, semantic types, SLA’s and error handling. So the next iteration might define a security model for example. It can be seen that I have only taken those parts of various methods to suit the deliverable at hand, one needs to be able to treat an integration project this way i.e. with agility because many aspects of your system can and will change in time. This is one of SOA’s promises – agility and this will not happen unless one can deliver in an agile way.

 

So where is RUP or Agile method X?  You will notice that the above draws on using EA and MDA techniques, the downside is there may be a learning curve for some but most EA/EAI will cope some care needs to be exerted as it doesn’t follow any method process flow. This is largely because this doesn’t exist for SOA and it doesn’t exist because every part or subsystem can be entirely different thus most methods don’t fit the distributed nature a large SOA because as previously stated they were designed for single projects but there is certainly a place for these methods per service definition, and indeed different sub systems may require different methods – it all depends.

 

Depends on what? Governance. This is the ability to control any aspect of the system, it’s design, it’s deployment, manage it when it’s up and running, upgrades – everything. Large SOA projects are nearly always never centrally controlled but this has to be your aim, in short the project will fail at worst or at best be compromised if this is not the case. Governance is the key, establish this at the outset and constantly work on it.

 

This brings us to tool support, this the area, not surprisingly that is most lacking there are many tools one might need from packet sniffers to workflow authoring tools

 

Looking forward to this coming year much talk is about cloud computing and virtualization, whilst these are interesting subjects they are more of a deployment and runtime governance issues and don’t really effect (too much) your design time choices