I understand that the WF runtime will just queue the instances up and will persist them if the runtime is stopped cleanly but that this could lead to large memory usage and long stop times. So, how should I manage this situation in a reliable manner? We are likely to want reliable messaging to the WCF service but should I be using some queuing mechanisms of my own etc within my service?
For reliablility in WCF you have two options
1. Use Reliable Sessions for scenarios where the client and Server will be online and connected. To enable relieable session,
annotate your service interface with [ServiceContract(SessionMode=SessionMode.Required)] attribute.
2. Use Queued Messaging ( via NetMsmqBinding) for relieable and durable messaging. Ideal for scenarios where the client and server need not be online.
I think you are talking about how to manage concurrent requests at the service so that you can scale, and be resilient to failure. Many factors influence this:
If you want to guarantee one-way messages from the client, that they are never lost, and you want to guarantee that clients are never rejected (i.e., their messages don't queue and then timeout because they aren't processed) then MSMQ is the best bet. THis is not interoperable and it adds complexity to the client because they have to support MSMQ as well.
I'm guessing you are trying to receive messages and manage your throttles so that enough concurrent requests can be processed to maintain a level of throughput that will service all clients. I use the benchmark that 25 thread pool threads for ASP.NET will service from 350-500 requests per second on a high-end server machine. This is not guaranteed for your scenario, but I'm definitely seeing requests under 1 second that include transactions, security, reliable messaging with WCF calls.
What you really need to do is get a handle on how long it takes to process each request, and do some load testing to help you configure the throttles for number of concurrent requests, number of requests to allow queued, and so on. You can use the WCF performance counters to help you with that.
You also want to make sure you have tolerance if/when the windows service is recycled. Reliable messaging (RM) can help with that at the client, if the time it takes to recycle the service is less than the time it takes the client to retry sending the next message. RM is good for transient interruptions. But, it would be even better to provide some form of load balancing across multiple services so that if one is down, another gets the request. The most robust load balancing is by physical device (router), but you can look at NLB as well. On the same machine, you could roll your own for this, or take you chances with the recycle process on the service depending on who your clients are and if they can tolerate the risk of interruption. Most can't.
