Deploying a new website or application?

You may have thought about general testing, but have you thought about all the other things that can make your product or service a success?

This list of questions covers the types of testing that must be considered to ensure a quality product:

1. Design Validation - Statements regarding coverage of the feature design - including both specification and development documents. Will testing review design? Is design an issue on this release? How much concern does testing have regarding design, etc. etc..

2. Data Validation - What types of data will require validation? What parts of the feature will use what types of data? What are the data types that test cases will address? Etc.

3. API Testing - What level of API testing will be performed? What is justification for taking this approach (only if none is being taken)?

4. Content Testing - Is your area/feature/product content based? What is the nature of the content? What strategies will be employed in your feature/area to address content related issues?

5. Low-Resource Testing - What resources does your feature use? Which are used most, and are most likely to cause problems? What tools/methods will be used in testing to cover low resource (memory, disk, etc.) issues?

6. Setup Testing - How is your feature affected by setup? What are the necessary requirements for a successful setup of your feature? What is the testing approach that will be employed to confirm valid setup of the feature?

7. Modes and Runtime Operations - What are the different run time modes the tool can be in? Are there views that can be turned off and on? Controls that toggle visibility states? Are there options a user can set which will affect the run of the tool? List here the different run time states and options the tool has available. It may be worthwhile to indicate here which ones demonstrate a need for more testing focus.

8. Interoperability - How will this product interact with other products? What level of knowledge does it need to have about other tools/software -- “good neighbor”, software cognizant, software interaction, fundamental system changes? What methods will be used to verify these capabilities?

9. Integration Testing - Go through each area in the product and determine how it might interact with other aspects of the project. Start with the ones that are obviously connected, but try every area to some degree. There may be subtle connections you do not think about until you start using the features together. The test cases created with this approach may duplicate the modes and objects approaches, but there are some areas which do not fit in those categories and might be missed if you do not check each area.

10. Compatibility: Browsers - Is your feature a server based component that interacts properly with browsers? Is there a standard protocol that many browsers are expected to use? How many and which browsers are expected to use your feature? How will you approach testing browser compatibility? Is your server suited to handle ill-behaved browsers? Are there subtleties in the interpretation of standard protocols that might cause incompatibilities? Are there non-standard, but widely practiced use of your protocols that might cause incompatibilities?

11. Compatibility: Operating Systems - Is your feature a client based component that interacts with operating system? Is there a standard protocol supported by many servers that your client speaks? How many different operating systems will your client need to support? How will you approach testing server compatibility? Is your client suited to handle ill-behaved or non-standard operating systems? Are there subtleties in the interpretation of standard protocols that might cause incompatibilities? Are there non-standard, but widely practiced use of protocols that might cause incompatibilities?

12. Beta Testing - What is the beta schedule? What is the distribution scale of the beta? What is the entry criteria for beta? How is testing planning on utilizing the beta for feedback on this feature? What problems do you anticipate discovering in the beta? Who is coordinating the beta, and how?

13. Environment/System: General - Are there issues regarding the environment, system, or platform that should get special attention in the test plan? What are the run time modes and options in the environment that may cause difference in the feature? List the components of critical concern here. Are there platform or system specific compliance issues that must be maintained?

14. Configuration - Are there configuration issues regarding hardware and software in the environment that may get special attention in the test plan? Some of the classical issues are machine and bios types, printers, modems, video cards and drivers, special or popular TSR’s, memory managers, networks, etc. List those types of configurations that will need special attention.

15. User Interface - List the items in the feature that explicitly require a user interface. Is the user interface designed such that a user will be able to use the feature satisfactorily? Which part of the user interface is most likely to have bugs? How will the interface testing be approached?

16. Performance and Capacity Testing - How fast and how much can the feature do? Does it do enough fast enough? What testing methodology will be used to determine this information? What criterion will be used to indicate acceptable performance? If modifications of an existing product, what are the current metrics? What are the expected major bottlenecks and performance problem areas on this feature?

17. Scalability - Is the ability to scale and expand this feature a major requirement? What parts of the feature are most likely to have scalability problems? What approach will testing use to define the scalability issues in the feature?

18. Stress Testing - How does the feature do when pushed beyond its performance and capacity limits? How is its recovery? What is its breakpoint? What is the user experience when this occurs? What is the expected behavior when the client reaches stress levels? What testing methodology will be used to determine this information? What area is expected to have the most stress related problems?

19. Volume Testing - Volume testing differs from performance and stress testing in so much as it focuses on doing volumes of work in realistic environments, durations, and configurations. Run the software as expected user will - with certain other components running, or for so many hours, or with data sets of a certain size, or with certain expected number of repetitions.

20. International Issues - Confirm localized functionality, that strings are localized and that code pages are mapped properly. Assure tool works properly on localized builds, and that international settings in the tool and environment do not break functionality. How is localization and internationalization being done on this project? List those parts of the feature that are most likely to be affected by localization. State methodology used to verify International sufficiency and localization.

21. Robustness - How stable is the code base? Does it break easily? Are there memory leaks? Are there portions of code prone to crash, save failure, or data corruption? How good is the tool’s recovery when these problems occur? How is the user affected when the tool behaves incorrectly? What is the testing approach to find these problem areas? What is the overall robustness goal and criteria?

22. Error Testing - How does the tool handle error conditions? List the possible error conditions. What testing methodology will be used to evoke and determine proper behavior for error conditions? What feedback mechanism is being given to the user, and is it sufficient? What criteria will be used to define sufficient error recovery?

23. Usability - What are the major usability issues on the feature? What is testing’s approach to discover more problems? What sorts of usability tests and studies have been performed, or will be performed? What is the usability goal and criteria for this feature?

24. Accessibility - Is the feature designed in compliance with accessibility guidelines? Could a user with special accessibility requirements still be able to utilize this feature? What is the criteria for acceptance on accessibility issues on this feature? What is the testing approach to discover problems and issues? Are there particular parts of the feature that are more problematic than others?

25. User Scenarios - What real world user activities are you going to try to mimic? What classes of users (i.e. secretaries, artist, writers, animators, construction worker, airline pilot, shoemaker, etc.) are expected to use this tool, and doing which activities? How will you attempt to mimic these key scenarios? Are there special niche markets that your product is aimed at (intentionally or unintentionally) where mimic real user scenarios is critical?

26. Boundaries & Limits - Are there particular boundaries and limits inherent in the feature or area that deserve special mention here? What is the testing methodology to discover problems handling these boundaries and limits?

27. Operational Issues - If deployed in a data center, or as part of a customer's operational facility, then testing must, in the very least, mimic the user scenario of performing basic operational tasks with the software.