BizTalk 2006 SOAP Adapter Issue
At this point I am not sure if it is something I am doing wrong, or a problem with the adapter itself (although I find that hard to believe).
I have an orchestration that makes a web service call and gets a response back. I have the orchestration setup so that it interacts with the web service through a logical "specify later" request-response port. The physical port is configured to use a web service proxy class generated through visual studio.
Both the request and the response work perfectly fine and the orchestration instances are executing through completely. However, the Process Memory Usage counter in perfmon for the host instance that is running the SOAP adapter is steadily increaing as requests are made and never comes back down. When I say never, I mean I can shut off all incomming messages and wait an hour and the memory usage counter stays the same.
The obvious problem here is that once the memory usage hits the threshold my entire solution goes into a throttling state and I get a backlog that is essentially never going to process.
So, does anyone have experience with a scenario like this? It seems like a memory leak, but I would think the adapter would be releasing whatever resources it uses. Further, the host instances for the receive locations as well as the orchestrations stay pretty static in regards to memory usage. All I know is that my only solution right now is to restart the SOAP Adapter's host instance. I don't like that.
[1520 byte] By [
drogin] at [2008-1-7]
Ok, further debugging/testing:
I setup a messaging only scenario using the same web service proxy assembly. It consists of a solict-response send port that makes the web service call, a one-way send port that has a subscription for the response message (and simply dumps it to the local file system), and a receive port that picks up XML file that is the web service call's argument.
After sending several hundred requests I see the send host's "process memory usage" counter in perfmon grow. When the requests are done the counter's value stays static and doesn't go back down. If I do a full stop on the application the counter stays the same. If I then restart the ports the memory usage stays the same. I finally restart the send host instance and the counter drops back to "normal". I've tried several permutations of the above and only one thing seems true: until I restart the application and restart the instance, the memory usage stays at its peak.
What is holding references to these objects? The adapter? The endpoint instance? The auto-generated web service proxy? Further, why would it be doing this? The request-response cycle is complete and the message is in the message box.
We have not seen this issue in our stress/performance runs. What does your send pipeline do? Does it create/hold on to something and not release it?
It's not clear what objects are staying 'alive'. Can you attach a debugger (windbg or cdb) to the SOAP send host instance and run the following commands:
.loadby sos mscorwks
!dumpheap -stat
!dumpheap
The last two commands (especially the last one) will tell you what objects are staying alive (along with their count and total size o the heap). There are three possibilities:
1) Very large number of small objects are staying alive
2) Small number of very large objects are staying alive
3) COM RCW objects are staying alive (and they hold on to the other objects in the native/C++ code)
You can check the column in the above output and see which is the case.
!dumpheap -mt <MT i.e. method table column from the above output>
will print the list of objects that are staying alive
!gcroot <object-address>
will tell you who is holding on to it
Hopefully, you will be able to track down this problem.
I will try what you've suggested, but since this was a production issue I went with the following workaround:
Call the web service directly from an expression shape.
For whatever reason, doing this prevents memory usage from increasing with each request, it pretty much holds static.
As an aside, I beleive the issue is related to XmlSerializer, which I am assuming is being used by the adapter to make the web service call (since I am using a proxy assembly rather than a generated orchestration). I know that XmlSerializer only caches the generated serialization assembly if you use certain constructors. Since you can't unload an assembly from an AppDomain, I think that is the issue.
I'll do my best to confirm it.
