experiences with MS software testing?
hi,
just wondering has anyone done microsoft ISV software tests? i'm a one-person ISV and i'm interested to hear from others who have been through the testing process. i've been on the ISV empower program for the last 18 months and i'll be taking my tests shortly.
i intend to apply under the "platform" tests, doing the windows client test, and web services + .net framework. i read the instructions from the testing company VeriTest and it sounds very basic to be honest... don't crash the system, don't delete files without permission etc. etc. i was expecting my source code to be FxCop'd but it seems to be not so thorough.
i'll be interested to hear opinions or experiences on the topic
cheers
tim
i'm really surprised there isn't tons of discussion on this topic of software testing. it's by far the biggest part of my our of the bargain in terms of the ISV agreement, and i'm very interested to know more about it, i.e. not the official blurb from the VeriTest guys.
basically i've been making good use of my $$$$$ worth of MS licenses and i have a good content management system developed. i think i'm ready for it to go through the tests now.
i wonder is there some confidentiality thing that prevents people talking about the tests. hmmmmmm...
Tim,
What questions do you have about the available tests? I'd be more than happy to discuss these with you.
Charlie
I have a number of questions about the Platform Test that ISVs must take be a certified partner with ISV competency:
1) The test specification says: "Applications … must perform their primary functions as expected". How strict are they on this, and how do they decide what a "primary function" is. Do we give them some walkthrough test that demonstrates what we consider to be the "primary functions" of the software? Or do they just install it, hit the home page of the web application, and call it good? Or something else? What triggers failure? An uncaught exception?
2) As part of the Windows Server part of the test, the specification requires that the software being tested "Use Windows resources (heaps, locks, and handles) appropriately". What tool or procedure will be used to determine if my application is compliant?
3) For the Managed Code criteria, what mechanism will be used to verify that all assemblies comply with the specification?Thanks!
I think this is interesting ground too. We have an application that we're building that interacts with managed code and I'm interested in knowing the best ways of handling this with respect to successful certification. Do you have to try to identify every potential problem and deal with the consequences? Are there guidelines out there, case studies? It really helps to have a clear idea of what is involved.
I'm interested in this topic as well. We have just started our project and registered with Empower. Our goal is to become a certified ISV and get our application tested.
The application is primarily a Windows Client entirely based on .NET 2.0. There is a server part, which can reside locally on a Windows XP Professional client for single users or on a Windows Server 2003 for multi users and teams. In either case the server part requires an SQL 2005 server. For single users and very small teams SQL Server 2005 Express will be good enough.
Do we need to provide web services? Conceptually, for easy Web access of some data and information we have included some web services. However, due to the limited personnel and financial resources we don't expect to get that part finished by the time we will get the application tested.
I'm interested to take this tests into consideration right from the beginning. I would like to have an application that can be tested at any time and doesn't need any rework to get ready for the tests when it is time.
Peter
 Charlie_MS wrote: |
| What questions do you have about the available tests? I'd be more than happy to discuss these with you. |
|
My application is a typical client/server application consting of a client, a server and a SQL Server component. The client cannot be run (tested) without the other two. The application as a whole fits best into any of the following categories:
-
Operating system
-
Windows Server
-
Windows Client
-
Other components:
-
SQL Server component
-
Managed code component
Usually the client part will be installed on a Windows XP client and the server part will be installed as a service on a Windows 2003 server. The server part needs access to a SQL Server 2005. Both, client and server are 100% .NET managed code.
In its simplest configuration the whole application (client and server) can be installed on a Windows XP client with SQL Server 2005 Express and .NET Framework 2.0. This is a typical demo installation, so we will prepare an install script for this special case anyway. In this configuration the application may not perform very well (2.8 GHz, 1GB RAM is o.k). Can we use this configuration for the tests?
What are the tests I have to take? Are there any extra costs involved?
Peter
Peter,
In this case you may qualify for up to 4 components of the platform test (http://www.veritest.com/certification/ms/platformtest.asp). The configuration you suggest would be acceptable for the tests however it would limit you to 3 components (Windows XP, SQL Server, & Managed Code) rather than 4. You would gain an additional 10points but taking the 4th test, however it is another $400 for that component. Each component of the test is $400. The details of the test requirements can be found on the Veritest website.
I would recommend registerring for the tests through the Veritest website and they will work with you on the details of your test configuration. They have detailed instructions on how to register.
The Platform tests you qualify for are solely for the purpose of gaining partner points within the Microsoft Partner Program. If you are interested in promoting your application with a Microsoft Logo I would suggest you take either the Designed for Windows XP Logo test or the Certified for Windows Server 2003 Test. Each of these tests are more rigorous than the platform test and successfully passing the requirements grants you the use of a Logo to associate with your product. Because they are more rigourous these tests are also more expensive. For detailed information on the price of these tests please contact Veritest at dfwxp@veritest.com or WindowsCert@veritest.com.
I hope this post was helpful. Please let me know if you have any further questions.
Thanks,
Charlie
Peter,
In order to achieve the ISV competency you only need to pass 2 components of the Platform test: at least 1 OS component, and 1 elective component. In this case it sounds like you qualify for the Windows Client component and the Managed Code component. Passing these 2 tests and providing the customer references will gain you the required points to meet the competency requirements.
Thanks,
Charlie
J.
Here's some information for you.
1. You are encouraged to provide an overview of the primary functionality of the application and/or test cases for those functions. Veritest engineers are validating that a user can walk the primary functionality of the application without experiencing a crash. If the application hangs and does not recover, or crashes this is a failure. However there are caveats:
A failure within a client component or tool will not be considered a crash if it meets all three of the following conditions:
a) does not cause loss of data,
b) displays information that would allow a typical user to understand what wentwrong and how to avoid the problem in the future
c) allows the user to continue running the application or close it.
2. Veritest uses the Microsoft Application Verifier to test for this requirement. You can find this tool on msdn here (http://www.microsoft.com/downloads/details.aspx?FamilyID=bd02c19c-1250-433c-8c1b-2619bd93b3a2&displaylang=en).
3. For managed code criteria Veritest uses FxCop to validate that your binaries are managed code. They do not apply any rules to the application, but simply confirm that your binaries load into FxCop successfully. FxCop can be downloaded from the following website (http://www.gotdotnet.com/team/fxcop/)
I hope this post helps. Please let me know if you have any further questions.
Thanks,
Charlie
Charlie,
Thanks for your answers. I already read the various documents available from Veritest and the MS partner site but it was not clear to me what it means in regard to my situation. I was a little bit confused (somebody, by referring to the Veritest site, told me that it even will be possible to take a client/server test).
Now it's clear and I think we are in a good position. If we continue that way, we will be able to just take the tests without having to care anymore, since as part of our development process we are continuousely looking for to fulfill the requirements.
Peter
