Any framework, samples or best practices for batch processing in .NET?
Hi,
Does any one have any pointers to good samples or frameworks that will help design and build a .NET batch processing solution (picking up requests, queuing them, processing, parallel processing, loading/reading from files/XMLs, loading tables (ETL), monitoring and tracking status of process and jobs, exception handling/logging, retries - all the usual set of stuff required in any typical batch processing scenario - know that Java has a Spring Batch framework - anything similar in .NET? any help in this area would be truly appreciated.
[560 byte] By [
jk67] at [2008-1-6]
I do not know such implementation, however for the similar requirements we use solution based on WF. Other thing that WF does not provide ETL functionality out of the box.
Microsoft also has BizTalk... through all options it also have ETL. Of course, it far from framework...
Hi
if you are working on sql 2005, you might consider using SSIS, as it provides a good starterpoint for ETL.
all the rest you can do by implementing a windows service which
- check a table for inserted requests
- starts SSIS packages
- polls for status of a given package
- logs errors and so on..
Hope this helps you out
Thanks. Well actually, the data sources in my case are primarily DB2 and Oracle. SQL Server 2005 may or may not be in the technology stack. Was looking at the documentation of the Spring Batch Framework (Java) - wish there was something on similar lines in .NET. If SSIS is not an option, do I need to build the whole infrastructure ground up in .NET - if there are any starter samples out there that can do some of the low level work, this would give me a good heads up.
Unfortunately, I can't say that I've found anything either, but just like you I have a requirement for such a beast, and just like you I came up with nada when investigating whether or not Spring.NET implements this as Spring does.
Please post here if you find a satisfactory answer (as will I).
Thanks,
--E
There is a framework built by Microsoft's partly-owned company called Avanade. The asset is called ACA Batch and I just did presentation at my current engagement where Spring Batch was the other alternative. ACA Batch won the deal and will be adopted as the standard going forward for batch development.
From your perspective, Chances are one in million of Avanade being involved at your customer but if they are you can probably explore that route.
--Shabab
I wonder about something.
You mention that you want your application to track jobs but that the jobs are primarily involved in data manipulation tasks (ETL type things). There are a LOT of packaged solutions on the market that range from inexpensive (and more suitable to departmental or small business solutions) to expensive (and more appropriate for high performance situations).
Oracle and IBM both sell solutions that work well in their respective database environments.
So if you are working in an environment where SQL Server is not an option, use .Net to talk to Oracle instead. The interface works fine and you can simply configure a high performance system rather than building your own.
I view this as a choice between tuning a race car and building my own go-kart. If I want to have fun, I might build a go-kart, but if I want to win a race...
Good luck,
Nick
