out of memory issue in Biztalk 2006
Hi All,
I am calling a web service inside my orchestration. When the data return from the web service is around 19MB, it doesnot give any problem. But when it is around 31MB, I am getting the following exception when receiving the data in the web service port:
InvalidOperationException: There was an error generating the XML document.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
Is there any way to solve the out of memory problem in Biztalk-2006?
Thanks,
Bikram
are you using any custom pipeline component that uses XmlDocument to load the doc in memory.
Or are you using variable of type XmlDocument inside your Orchestration
Since your message size are relatively big you need to pay attention not to load the document in memory, instead practice and use stream based reading.
Regards,
Saravana Kumar
http://www.biztalk247.com/v1/
http://www.digitaldeposit.net/blog
[Please mark the response as "Answer" if it solves your problem.]
No i am not using any custom pipelines.
For webservice port, we are using Default Passthru pipelines. Well after too much R&D and observation what
we have found is that Biztalk is unable to handle Huge Data (300 MB)Response that is coming into Biztalk.
Do we need to create custom pipeline to handle this OOM exception ?
We have used Expressions and we are capturing the flow in orchestration by writing into the event log. like "Before Calling of
Webesrvice", "After receiving Webservice Response".
For this huge response "Before calling webservice" has been logged into Eventlog and before logging next message we are
getting this OOM exception.
I have another doubt which I want to clear from you. Does Biztalk write the receive data from the web services into the MessageBOX database? If Yes, then what will happen if the response from the web service is huge (300MB). Are there any setting associated with this?
Is there any cache settings or memory settings in Biztalk to avoid this problem. since we have delivery next couple of days
your help will be greatly appreciated.
-Thanks in advance.
Bikram
You don't need a custom pipeline in your case.
can you post the piece of code you have inside the expression shape (including your logging calls)?
BizTalk will ofcourse write the response back to MsgBox with context properties, so that it can route it to different subscribers (Send ports), BizTalk is designed to handle large messages (300 mb is not that big in BizTalk world). You don't need to change any settings.
--
Regards,
Saravana Kumar
http://www.biztalk247.com/v1/
http://www.digitaldeposit.net/blog
[Please mark the response as "Answer" if it solves your problem.]
Hi,
Thanks for the reply.
Let me explain what I did in my orchestration.
I called a web service using Send and Receive shape inside my orchestration. I am receiving the error when the web service is sending the response. So the orchestration stop at that point. This problem didnt happen when the response from the web service is below 300MB. It works fine and I am getting the response. I use my expression shape for logging only, so that I can track the flow.
Do you want to look into my complete code?
Hope to get back from you soon.
Thanks,
Bikram
Bikram - mate!!! 300MB from a Webservice....sensational!! (are you using R2 as we could handle the 300MB using other transport formats - e.g. bindings)
Ok - let's talk a little about what's going on here.
You orch - goes through a Send/receive logical port to your WS, during sending through this logical port, your request will *always go through the messagebox DB* and the same with the return message.
BTS is very good at dealing with large messages and 300MB is considered to be a large message (over 100MB is the default setting) - your WS return message will be streamed through the receive pipeline (passthru I'm assuming).
- from this point on, the BTSNTSvc.exe will poll the messagebox db (calls a stored proc - 'dequeue_messages..') are there any messages for me to process?
- 300MB msg(s) will come over the wire and be streamed to your Orchestration. (let's assume I have 10 of these Orchs running)
Do you need all of 300MB in the orchestration at once? usually what we do is to stream these out to a SQL table (using for e.g. SQLBulkLoad) and make sql queries etc. to work with the data.
- when mapping within BTS06 the team have optimised this for working with large messages (bts04 loaded the messages in memory).
So my bet is that somewhere you're loading the entire 300MB message into memory - if I have 10 of these Orchs running that 3000MB consumed just by a single message. For a .NET application (BTSNTSVC.exe is essentially one) memory consumption should be no more than 700MB of virtual memory - too long for this thread to explain.
You mentioned that you track, trace - do something on the message. Take the following lines of code:
xdoc = new System.Xml.XmlDocument();
xdoc = msgWSReturn; //this is the 300MB message
helper.WriteMessageOutToTrace(xdoc);
The 2nd line above should be avoided at all costs!!!!! XmlDocuments load the entire message into memory (great for small messages, not great for large ones)
If you do need to manipulate the entire document at once then I recommend using the Microsoft.Xlangs.Core.XLangMessage (or XLangPart) and within these classes there is the method RetrieveAs(stream) - so you can deal with the message as a stream and not load entire 300MBs at once.
Look in the BTS06 SDK - and chase up the VirtualStream class (great class that uses temp files when messages get over a threshold. Otherwise it uses memory.
If this is *NOT* your problem then (you must be doing something massive 
- consider creating additional BizTalk Hosts/Host Instances and sharing the load of running those Orchestrations across a couple of BTSNTSvc.exe processes.
HTH
