SOA - Transaction/compensation/cordination
I have a challange.
I have multiple services that are located on different server setups. They are build stateless N-tier architectures based on single call remoting.
The services do not support the flow of transactions. This would also couple the services an go against the very idea of service orientation.
But ..nevertheless, one needs to perform business operations that span multiple services.
How do I best support this. Should I have the programmers do custom compensation code very time (clean-up) and accept that if something crashes the data is inconsistant? or should I try to find/implement some kind of common cordination?
What are my choices?
[1486 byte] By [
Siteadm] at [2007-12-31]
If you have related services that need to be atomic, your choices are transactions or compensation. As you said, distributed transactions pretty much eliminate loose coupling. Reliable, trnasactional messaging can dramatically increase the probablility of the services succeeding but it's still possible the all the services will have to bereversed and because rollback isn't really an option in a loosely coupled system, compensation is your best option. I have some ideas on the topic here:
http://blogs.msdn.com/rogerwolterblog/archive/2006/05/24/606184.aspx and here:
http://msdn2.microsoft.com/en-us/library/aa964144.aspxThe fact that you have applications "located on different server setups" doesn't automatically make them services. You are right in staying away from atomic/2pc transactions that cross service boundaries.
If you find that you need to do quite a lot of compensating transactions beween services, I'd be open to the possibility of a less-than-optimal service decomposition. The loose coupling between services is first-and-foremost on the business level. Look for things that have a human involved in the workflow - that often delineates a service boundary, as does a long time-interval between operations.
My advice is to take a step back, rather than digging in and solving the technical problem.
Hi Roger
Thank you for answering my post.
Do know where I can find an example of an implementation of transactional messaging?
I'm guessing that you are saying that I should look into using some queuing and maybe also some kind of work flow engine?
I would like the solution to be as simple as possible because we need the functionallity very soon.
Sitedm
I agree with Udi,
if you find that you need the transactional behaviour alot then the business logic of the services is coupled even if you don't technologically couple the services by introducing distributed transactions.
Arnon
Hi Siteadm;
Can you give us more hints about your application. As the Udi and Arnon say, you do not to have to go for complex transactional system and implement distributed transactional system or Queued components. I would suggest few things to evaluate before making your decision...
- Is your application one logical entity that spans its operations to multiple servers? Or is it a trure Service Oriented Application?
- In case it is SOA you will need complex transactional system that can simply be provided by WS-AT in Windows Communication Foundation.
- In case that your application has code that executes on multiple servers, you might consider COM+ which provide you with couple of advantages...
- You can implement Remote transactions using COM+, this is fast and reliable
- You might need to implement Queued components that provide you with 100% guaranteed message delivery.
- COM+ will allow you to implement compensation for operations that cannot be rolled back automatically like writting to file system
- Evaluate the physical communication reliability between your servers.
- In case all your servers are in the same LAN, you go for transactional system using COM+ or WCF
- In case you do not have reliable connections, use compensation techniques.
- Does all the components that are located on different servers work on the Same Database or not?
I'm sure you have other things to evaluate like your current system and existing code
. Be careful about SOA as the term is sometimes misleading. If your application needs to expose functionality to other systems from a business prespective, then you have a true SOA.
I hope that was useful,
Thank you Roger
Thank you for the great links. I'm registering for a virtual lab tonight so that I can check it out. I know nothing about Service Broker :(
I will mark your posts as answers. (and also the others with relevant input). Im just afraid to mark them as answers yet - and then not getting anymore feedback. I'm not sure I have found the right solution for my case yet.
Hi Siteadm;
According to MSDN, your current technology does not fit your current requirement. Read From MSDN: "It is important to note, however, that neither the ASP.NET Web services nor the .NET Remoting plumbing supports propagating a declarative transaction, so it is impossible for either sort of endpoint to inherit a declarative transaction via a cross-process call.".
Reliability of connectivity between systems should make thinking about compensation comes in a second place provided that your transactions are short, optimized and independent from user input.
Now, I can suggest that you use WCF. WCF Supports many communication protocols and some models for distributed transactions including WS-AT (Web services atomic transactions), which fits your requirements. I'm glad to see the service and proxy layers in your architecture which means that migrating to WCF should have mimimum impact. You can find many resources on that including MSDN as a primary source of samples, see this link. Note that WS-AT requires special handling in terms of configuration.
I have a recent article about WS-AT and distributed transactions located here and it might give you some insight on how distributed transactions work. The article is conceptual and does not introduce WCF in depth, but it explains how WS-AT works.
I hope you find that useful,
"I have multiple applications that expose multiple services"
That's another unfortunate overloading of the term Service. Same with Web Service.
When talking about services in the context of SOA, then a service is not exposed by an application. If anything, the service IS the application. Other software interacts with the service by sending it messages - calling a "webmethod" on the service (with or without parameters) is just another way to send it a message. The webmethod is not the service.
On the topic of stateless-ness, just because you don't keep objects in memory between webmethod invocations, that doesn't make a service stateless - state is still maintained in the DB. The fact that several webmethods need to be called in a specific order, passing the state received from one to the invocation of another, in order to get any business functionality done points to a problem with the granularity of the messages your services process.
For more information on entity aggregation in the context of SOA, take a look at my article "Autonomous Services & Enterprise Entity Aggregation" from the Microsoft Architecture Journal, issue 8.
Hi Ahmed
I had first thought of using WCF and WS-AT. The problem is that it would take require locks in the databases that might be envolved.
I'm quite sure that this would give me all kinds of deadlocks - since each "applications" services are also exposed to external developers.
As I see it I have the same problem as if the services belonged to different organizations. There is no way they would let anyone flow a transaction into their systems.
I think I need to find the best way to do compensation instead of atomic transactions. Right?
Hi Siteadm,
As you say in one of your previous posts the services are 'fine grained' from the business process you are trying to model.
It sounds to me as though you do indeed need to use a compensation approach to your problem and this would be hosted and controlled from a higher level service ('Super Service' - awful name :)) that is utilising your 'fine grained' services you describe in the url's provided in your previous post - create another layer of abstract on top of the already existing services.
Obivously to support compensation in any of your 'fined grained' services will require funcrtionality to negate an operation supported, what I mean by this is if you had an operation to 'CreateOrder' you should also have functionality to 'DeleteOrder' for the same set of parameters - thus allowing you to 'rollback' any data you might have created\modified.
Now from my experience this all sounds quite simple in the abstract but it can get very complicated very quickly especially if you don't control the development or API's of the 'fined grained' services you wish to use compensation on.
HTH
Ollie Riches
Hi Siteadm;
Services exposed by the service layer should not be fine grained but rather should be coarse grained. The reason behind this is to minimize the number of possible calls cross application boundaries. So, at this point all your you can make your services stateless and always one call is enough to do the job. Always remember that cross process calls are expensive and you should always aim to minimize the round-trips between servers.
Back to your point about compensation, my advice is to use transactions where possible when all calls are under the control of your applications and no calls to external application is done. Even more, if an external application is calling ServiceA.ProcessRequest and this call triggers calls to other applications in your domain, you can still use transactions to start at the point where your application was first called. Avoid using transactions with systems external to your reliable network boundries (even if they support it). I do not see any reason that does not allow you to mix both techniques(compensation and transaction) where each one is applicable.
Compensation adds complexity to your solution so try to avoid it as possible or minimize its use. Think about using compensation when the following is true...
- Operation is lengthy and locking might involve performance problems.
- You understand and can evaluate the impact of having inconsistent data accross the applications during the time where certain request is being processed
- You are calling external application that is not reliable in terms of connectivity, responsivness or availability.
I hope you find this helpful,
Hi Udi
Thank you for answering my post. I'm sorry it has taken a few days to reply to it. I had to find time to read our article before I could reply. The article is great by the way.
There is a good vocabulary in your article that can help me describe the services to you.
I like the distinction of Process, Activity and Entity service that you mention.
The services that the applications expose are often entity services, but there is nothing stopping me from making one service that exposed all the services . I’m actually able to do transactions on the much related business calls for services inside the applications. Like in the diagram from the article (link below). I could make a service that exposes all functionallty of the CRM system - Much like the order processing service in your diagram. I am able to do distributed-transactions over multiple entity/activity services. There are messages that is sent today that include "Messages" with multiple entity types included. So they all get updated/created or rolled back.
http://msdn2.microsoft.com/en-us/arcjournal/bb245672.jour8autonomous04l(en-us,msdn.10).gif
Where my problem lies is in communicating with two services (In my world two appliactions). A CRM system and a ERP system. - also others.
I need to find a way for front ends to do "business transactions" that might span multiple services.
As I have come to see it. I believe that I should look at the applications/services as if they were located in different organizations - who would never allow the flow of atomic transactions - and then "garrantee" roll back by some kind compensation - maybe with a work flow engine with compensating work flows or SQL Broker . ..now I just have to figur out how to do that?
I would very much like any further comments you might have or link to articles that might push me in the right direction.
Thanks,
Siteadm
.gif){kind=link}