Urgent: SslState throwing exception deep inside HttpRequest at random
We have a class in our website that calls Fedex's web servers to receive quotes using an XML interface. (This is part of our check out processes). Over the past few days we've started receiving Disposed Object exceptions randomly during our HttpRequest. The class that is throwing the exception is actually in System.Net.Security.SslState, which is deep inside Microsoft's framework library and not called directly. We have never had problems like this before, nor can we find references to it on the Internet. Fedex recently upgraded their systems, but claims we have done something to break our code, whereas the only change we've made in recent weeks was moving from .NET 1.1 to .NET 2.0. We've been running on this code for over 2 years and haven't had a problem until this week!
The exception is below, and a snippet of code is at the bottom. You can see we just write to the request, and then call BeginGetRequestStream to start the request in async fashion.
If someone could find a solution to this it would be great, we lost $15,000 alone today in orders due to this problem, and probably $50,000 or more over the past week.
message=Cannot access a disposed object.
Object name: 'SslStream'.
stack=
at System.Net.Security.SslState.ValidateCreateContext(Boolean isServer, String targetHost, SslProtocols enabledSslProtocols, X509Certificate serverCertificate, X509CertificateCollection clientCertificates, Boolean remoteCertRequired, Boolean checkCertRevocationStatus, Boolean checkCertName)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Atacomm.Shipping.Fedex.FedexRateQuote.HttpRequestCallback(IAsyncResult ar)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.CompleteCallback(Object state)
at System.Threading.ExecutionContext.runTryCode(Object userData)
at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.HttpWebRequest.InvokeGetRequestStreamCallback()
at System.Net.HttpWebRequest.EndWriteHeaders_Part2()
at System.Net.HttpWebRequest.EndWriteHeaders_Part2Wrapper(Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
state.httpRequest
= (HttpWebRequest)System.Net.WebRequest.Create("https://gateway.fedex.com:443/GatewayDC");
state.httpRequest.Method = "POST";
state.httpRequest.ContentType = "application/x-www-form-urlencoded";
....
state.httpRequest.Timeout = 15000; // 15 seconds in milleseconds
state.httpRequest.ContentLength = outstr.Length;
state.httpRequestData = outstr;
IAsyncResult ar = state.httpRequest.BeginGetRequestStream(new AsyncCallback(HttpRequestCallback),state);
public static void HttpRequestCallback(IAsyncResult ar)
{
FedexHttpState httpRequestState = (FedexHttpState) ar.AsyncState;
System.IO.Stream streamRequest = null;
try{
streamRequest = httpRequestState.httpRequest.EndGetRequestStream(ar);
streamRequest.Write(System.Text.UTF8Encoding.UTF8.GetBytes(
httpRequestState.httpRequestData),0,httpRequestState.httpRequestData.Length);
httpRequestState.httpResponse = (System.Net.HttpWebResponse)httpRequestState.httpRequest.GetResponse();
}
catch(Exception e)
{
EventLog Log = new EventLog();
Log.Source = "ResvNet Fedex RateQuote";
Log.WriteEntry(e.Message, EventLogEntryType.Error);
if (streamRequest!=null)
streamRequest.Close();
return;
}
streamRequest.Close();
.....
}
Above is our Async request/response handler. You can see we grab the exception and throw it into our event log.
We found a glitch we removed which was causing our page to freeze on the exception (we called the request twice, once for ground quotes, the other for express) and then waited for both to finish async. When the exception was thrown it was failing to clear the variable. So atleast people are now able to proceed with checkout.
But
We are still receiving the exception, we never experienced it before .NET 2.0. Fedex did maintanance with their system almost simultaniously with our switch to 2.0, so we don't know who is causing the problem. We've been trying to figure it out with their techs for 2 weeks now. An examination of packets on the network seem to point towards their disconnecting of the TCP connection but there is disagreement about it.
Its not every time we call it; it is usually one out of every 4 or 5 requests, and its during transmission of the request, not while reading the response.
I could try to make a better snippet of source code available if you'd like.
Biggest thing is, there is no documentation of HttpRequest throwing SslState exceptions. I don't see how we could be the only one and having it happen so repeatidly. We have customers who are still receiving no shipping quotes, so obviously its an important fix.
Derek,
Good news. We were discussing the code this afternoon, and I asked if anyone was experiencing quote problems in our Windows.Forms application; surprisingly I received the answer of no. We were however having failures on average one out of every 5 calls on our web server. There was no difference what so ever in the code being used to call fedex, its an external library that we developed which is used by both our ASP.NET webstore and by the Windows.Forms application. We already verified with Fedex that the rate requests were valid, so I stepped back and asked myself what is different between ASP.NET's enviroment and a Windows.Forms application.
The first thing that popped into my head is I've seen people complain about odds and ends with worker threads; which of course using an ASYNC HttpWebRequest, we would be using a worker thread. Note we never actually saw a threading related exception, but figured it was worth a try.
Changing the code over to execute the rate queries in-thread (With Fedex, we have to place two rate requests, one for ground, and the other for express....which slows down our ecommerce site, we prefer to send them side by side in different threads) seems to work. I have ran about 50 rate requests in a row on our beta site without a single SslState disposed message.
Short answer: Dropping ASYNC HttpRequest methods fixed our problems.
Long answer: We didn't change a single line between .NET 1.1 and .NET 2.0, and all of a sudden this frustrating glitch popped in. We are using ASP.NET default settings; and much prefer a multi-threaded approach to tackling the rate quotes. If anyone has any insight as to why I'm receiving these messages from .NET instead of something thread related, and what possible work arounds exist that would allow us to continue running multiple threads, that would really be great and I'd have to buy everyone beers.