For a LoadTest's "Run Settings" properties,
1. whats the difference between using ConnectionPool and ConnectionPerUser?
2. Which setting "DOES NOT" cause any bottle necks on the Agents.
Jesudhas
With ConnectionPerUser, each virtual user of a web test uses a dedicated connection to the web server, and that connection is closed when the web test session completes.
With a ConnnectionPool, the number of connections is specified by the Connection Pool Size setting, and if this is less than the number of web test users, then it is possible that web test requests will be queued while waiting for a web test request for another user to be executed.
So, if you need to accurately simulate the connection load placed on your web server, use ConnectionPerUser. The drawback to ConnectionPerUser is that each connection takes up a fair amount of memory on the agent, but web test requests for one user should never block a different user.
If you don't particularly care that the number of web server connections is accurate (because you are load testing an app rather than the web server), and you have a web test where the think times between requests are larger than the response times, you will probably get better performance out of the agent by using a connection pool if you make it large enough. For example, if you have resonse times that are 1-2 seconds and think times that are 5-10 seconds, the Connection Pool Size could probably be 50% of the number of users and the number of times that one request waits for another is likely to be small.
Hi,
I am running a load test with one web test present and i have various think times of between 5 and 20 seconds between requests. I have a constant load of a 100 users and am using connectionperuser. My problem is that my requests/sec starts off at approx 40 and over 30 minutes more or less linearly reduces to approx 1-2 requests/sec. If i change to connectionpool my request/sec stays more or less constant. The load on the server is approx 30-40 %. Is there some checks i can do on the client side to see why my request/sec is reducing or is there something obvious i am missing. Is it possibly that tcp resources on the client side are not being released. Also the client is not being overloaded in anyway either regarding processor or memory.
Regards
Hi
I have also experienced some problems while using ConnectionPerUser:
I use a loadtest with only 1 scenario, user load that increases from 1 to 60 during the first 30 minutes,
and remains on 60 untill the loadtest is complete.
The problems I have encountered are:
1. I get much slower response times while using ConnectionPerUser.
2. ConnectionPerUser somehow causes a memory leak in the agent ( joining homer jay's issue: "Is it possibly that tcp resources on the client side are not being released")
3. While using ConnectionPool, I get some timeout errors for overqueued requests, even when I define a connection pool much greater then the amount of my users.
Any thoughts?
Regards,
Nanneq.
If you are running on a multi-processor machine make sure you configure the "Server GC" (the server version of the garbage collector) for the load test agent process. Not setting this on a multi-processor machine could potentially be the source of all of these problems (see details at the bottom of my post).
Regarding problem #3, what is the exact error type and message for your "timeout errors for overqueued requests"?
Here's a copy of the item from the readme file that describes how to configure the Server GC:
Recommendation: Configure the agent process to use the Server GC on multi-processor computers.
Recommendation: The load test runtime engine will perform best if the agent process is configured to use the Server GC (the server vsion of the .Net runtime garbage collector) when running on a multi-processor computer. This is also true when running a load test locally if the local computer is a multi-processor.
For example, on one particular load test the throughput in requests per seconds was 365% higher when using the Server GC and the memory usage was 60% lower.
How to: Add the gcServer tag with an enabled attribute set to true in the runtime section of the appropriate configuration file.
<configuration>
<runtime>
<gcServer enabled="true" />
</runtime>
</configuration>
The configuration file for local load test runs is <Program Files>\Microsoft Visual Studio 8\Common7\IDE\VSTestHost.exe.config
The configuration file for a run using one or more agents is <Program Files>\Microsoft Visual Studio 2005 Team Test Load Agent\LoadTest\QTAgent.exe.config
Hi
The exact error message i got was:
"Timeout expired.
The timeout period elapsed prior to obtaining a connection from the pool.
This may have occurred because all pooled connections were in use
and max pool size was reached."
My load agent is not a multi-processor machine, but it does support Hyper threading.
I will try to configure the server GC anyway, and see if it makes any difference.
10X, Nanneq.
Hi again :)
Configuring the GC still does'nt help - The memory leak persists when using the "Connection per user" Method.
Any more ideas?
Regarding the timeout error on the connection pool, that error is actually not related to the load test connection pool and is not an error generated by the load test runtime engine. Instead, it is an error related to the SQL connection pool which I presume is being used by the ASP.NET app that you are testing. If you perform an Internet search on that error text you will find many references to this SQL connection pool error. For example:
http://www.dotnet247.com/247reference/msgs/31/158758.aspx
So I don't think the timeout error is caused by your load test connection pool setting at all, and I think I would suggest investigating why you are running out of SQL connections. Is the user load larger than the SQL connection pool size?
Regarding the memory leak when using connection per user, what exactly are you seeing? Is the virtual byte size of the QTAgent process increasing even though the user load is steady? By how much?
The connection pool error issue has been resolved internally,
thank anyway for your comment on that issue.
Regarding the memory leak:
The user load goes from 1 to 60 in the first half hour, and remains 60 untill the end.
On the other hand, The private bytes size of VSTestHost process keeps increasing throughout the whole test duration. It starts at approx. 100 MB, and goes up to 1 GB - that's when my machine crashes... (It takes about 18 hours to get this result in the current load configuration)
I have a few more questions to help get to the bottom of this:
1. Your original post implied that the memory leak only occurs if you use a connection per user, and not if you use a connection pool; is that still true after correcting the error?
2. What type of Web test(s) are included in the load test? (Declarative, C# coded web tests, or VB coded web tests?)
3. Is think time enabled or disabled for the load test?
4. How busy is the CPU on the machine where VSTestHost is running?
Thanks, Bill
alright:
1.yes, it's still true.
2. C# coded web tests .
3.Think time is enabled for the load test.
4. The CPU on the machine that runs VSTestHost is not busy at all - about 10% .
I'm trying to see if I can reproduce this problem, but so far I have not been able to reproduce it. I started a load test that includes a simple C# coded web test with a user load that ramps up to 200 users and with connection per user mode. The load test has been running for just over 3 hours, and so far the memory usage for VSTestHost looks flat (since the load ramped up to 200) at about 300MB for private bytes. I'll let this run overnight, but I don't expect the memory usage will go up.
I think there are two possibilities for why it is leaking for you and not for my test:
1. It is possible there is a memory leak in your coded web test. Other users have had this problem. If you use any static objects to store data across web tests, it is possible for your code to leak. If your code is doing anything like this, I suggest you check for your web tests for leaks.
2. There may be a leak in the VSTS load test or web test runtime code that only occurs when the coded web test uses certain options that your test is using, but my test is not. Can you either send me the source to your coded web test(s) or describe in details which WebTest and WebTestRequest properties your code tests use? If you want to email me your coded web test source, let me know and I'll provide my email address.
Bill
p.s. Just to make sure on my last question #1, there is no memory leak if you use a connection pool now right? Are there any problems when you run with a connection pool?
That is correct, there is no leak when I use Connection pool.
1. I will review my code for any static objects, but shouldn't that suppose to cause a memory leak also in connection pool method?
2. If no such objects will be found, I will ask for your E-mail address...
Thanks,
Nanneq
Yes, I would think that if your coded web tests were leaking memory that would also occur with a connection pool (which is why I wanted to make sure). My overnight test did not show any signs of a leak. My email alias is "billbar" (domain: microsoft.com).
Maximizing Load Generator Memory Utilization During a VSTT Load Test
When a VSTT web test is recorded, by default VSTT sets both the "Parse Dependent Requests" and the "Record Results" properties of the request to "True."
With these settings, perfmon counters and TTFB and TTLB numbers for each request, as well as for dependent requests, are recorded to VSTT's LoadTest database for later retrieval by the reporting tools of VSTT. This default behavior makes sense, especially for a web test where there is not a huge volume of requests being sent out.
However, when webtests are added to a LoadTest ,VSTT continues to keep track of all the perfmon counter information for all of the requests and the dependent requests, and does so repeatedly, each time the web test is executed by a new virtual user in the load test. If you have several complex web tests in your load test, as many people will when then are doing end-to-end testing on a nearly finished application, and if you are trying to use a large number of users (again, as many people will want to do, because if you can show results with high numbers of users to management, it is a lot easier to explain) -- all these requests are a lot of information for VSTS to hang onto. The load generator will start to loose available memory very rapidly. Any load testing tool will exhibit the same behavior if their logging is on.
VSTT has some workarounds for the memory loss on the load generator:
1. Start by making backup copies of all your working web tests (you might need some information from them later on).
2. Set the "Parse Dependent Requests" and the "Record Results" properties of all your requests to "False." If you have already created a complex load test you can use Visual Studio's "Find and Replace" feature to find and replace all instances of [ParseDependentRequests="True" ] and replace them with [ParseDependentRequests="FALSE" ]. You can do the same with the "Record Results" property -- change [RecordResult="True" ] to [RecordResult="False" ]. It is true that setting ParseDependentRequests to false will prevent image and java scripts from downloading, but these requests don't create a lot of processor utilization on IIS anyway. (see below for more information).
In the web application I was testing I had to test a search page which returned a lot of images (like 60 per page). On each search request the images changed. So not very far into the test I was tracking information on thousands of images. These fell under the category "other urls" and this is the sort of thing which produces the "The number of unique requests which can be tracked has been exceeded" error message.
3. Eliminate Unnecessary Transactions. VSTT also keeps statistics on each transaction you set. Each transaction chews up memory during a load test. If you have already created transactions around many requests, the best bet is to start by removing all the transactions and reset them, this time only wrapping transactions around the requests where it is absolutely necessary. You can remove the transactions without disturbing your test by dragging the request itself outside of the transaction and dropping it on the top node of the web test. Do this sequentially and your requests will be in the proper order when you finish. You can then delete the empty transaction.
Run your load test and verify that at the end of the test no URL's or images are reported in VSTT's "counter" node in the upper left.
At this point you are almost ready. However, if when you test runs, it has a lot of exceptions, these too, will chew up memory.
4. Fix Errors in your Web Test Scripts. If you are using parameters which produce errors in your loadtest, you must correct those errors in your web test script. Look at the number of failed tests, and if these are high, tweak you parameters or record a better script which does not fail as often.
5. Follow the instructions about deleting the threshold counter at: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=482304&SiteID=1
6. Set VSTS to use the server garbage collection as described in http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=210575&SiteID=1
With all these changes you still may see VSTT use quite a bit of memory. However, with these changes, VSTT will still be quite functional even when available memory becomes low.
At this point you will have turned VSTT into a pure load generator, and memory loss will be tremendously reduced during load tests. You will be able to get much higher user loads. However, VSTT will not report on any execution time or any other information at the end of the test.
7. Judiciously re-set some of the RecordResult properties to "True," remembering that for each request you set to ture will chew up a chunk of available memory. If you need to run a 2,4, or 6 hour test with high numbers of users, you must to set RecordResult to true very sparingly. (If you really need to record information for many requests, one alternative is to get more load generators.)
In the final testing of my application I found that I had many different paths: purchase path, search path, signed in users, not signed in users, etc., for which I was tracking response time information. Many of the requests in these paths, especially signing in, I was tracking two, three and four times without any really need to do so.
If you really need information about the response time of *all* the requests in all your paths, or what the response times look like when you are downloading all of the dependent requests, you could can run duplicate copies of your web tests: one set of tests configured as described above, a second set, at a much lower number of users, set with all the transactions and all the dependent requests.
Remember that as your load test pushes you webserver processor closer to 100%, an application in the development stages will likely start to produce a lot of failed responses.
