How we plan to cover the product so as to develop an adequate assessment of quality.
A good test strategy is:
- Specific
- Practical
- Justified
The purpose of a test strategy is to clarify the major tasks and challenges of the test project.
Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy.
Example of a poorly stated (and probably poorly conceived) test strategy:
"We will use black box testing, cause-effect graphing, boundary testing, and white box testing to test this product against its specification."
Creating a Test Strategy
The test strategy is a formal description of how a software product will be tested. A test strategy is developed for all levels of testing, as required. The test team analyzes the requirements, writes the test strategy and reviews the plan with the project team. The test plan may include test cases, conditions, the test environment, a list of related tasks, pass/fail criteria and risk assessment.
Inputs for this process:- A description of the required hardware and software components, including test tools. This information comes from the test environment, including test tool data.
- A description of roles and responsibilities of the resources required for the test and schedule constraints. This information comes from man-hours and schedules.
- Testing methodology. This is based on known standards.
- Functional and technical requirements of the application. This information comes from requirements, change request, technical and functional design documents.
- Requirements that the system can not provide, e.g. system limitations.
- An approved and signed off test strategy document, test plan, including test cases.
- Testing issues requiring resolution. Usually this requires additional negotiation at the project management level.
Defining a Test Strategy
A solid testing strategy provides the framework necessary to implement your testing methodology. A separate strategy should be developed for each system being developed taking into account the development methodology being used and the specific application architecture.
The heart of any testing strategy is the master testing strategy document. It aggregates all the information from the requirements, system design and acceptance criteria into a detailed plan for testing. A detailed master strategy should cover the following:
Project Scope
Restate the business objective of the application and define the scope of the testing. The statement should be a list of activities that will be in scope or out of scope. A sample list would include:
* List of software to be tested
* Software configurations to be tested
* Documentation to be validated
* Hardware to be tested
Test Objectives
The system under test should be measured by its compliance to the requirements and the user acceptance criteria. Each requirement and acceptance criteria must be mapped to specific test plans that validate and measure the expected results for each test being performed. The objectives should be listed in order of importance and weighted by Risk.
Features and Functions to be Tested
Every feature and function must be listed for test inclusion or exclusion, along with a description of the exceptions. Some features may not be testable due to a lack of hardware or lack of control etc. The list should be grouped by functional area to add clarity. The following is a basic list of functional areas:
* Backup and recovery
* Workflow
* Interface design
* Installation
* Procedures (users, operational, installation)
* Requirements and design
* Messaging
* Notifications
* Error handling
* System exceptions and third-party application faults
Testing Approach
The approach provides the detail necessary to describe the levels and types of testing. The basic V-Model shows what types of testing are needed to validate the system.
More specific test types include functionality, performance testing, backup and recovery, security testing, environmental testing, conversion testing, usability testing, installation and regression testing. The specific testing methodology should be described and the entry/exit criteria for each phase noted in a matrix by phase. A project plan that list the resources and schedule for each testing cycle should also be created that maps the specific testing task to the overall development project plan.
Testing Process and Procedures
The order of test execution and the steps necessary to perform each type of test should be described in sufficient detail to provide clear input into the creation of test plans and test cases. Procedures should include how test data is created, managed and loaded. Test cycles should be planned and scheduled based on system availability and deliverable dates from development. All application and environmental dependencies should be identified along with the procedures necessary to gain access to all the dependent systems.
Test Compliance
Every level of testing must have a defined set of entry/exit criteria which is used to validate that all prerequisites for a valid test have been met. All mainstream software testing methodologies provide an extensive list of entry/exit criteria and checklist. In addition to the standard list, additional items should be added based on specific testing needs. Some common additions are, environmental availability, data availability, and validated code which is ready to be tested.
Each level of testing should define specific pass/fail acceptance criteria, to ensure to ensure that all quality gates have been validated and that the test plan focuses on developing test that validate the specific criteria defined by the user acceptance plan.
Testing Tools
All testing tools should be identified and their use, ownership and dependencies defined. The tools category includes manual tools, such as templates in spreadsheets and documents as well as automated tools for test management, defect tracking, regression testing and performance/load testing. Any specific skill sets should be identified and compared against the existing skills identified for the project to highlight any training needs.
Defect Resolution
A plan to address the resolution of failed tests needs to be created that lists the escalation procedures to seek correction and retest of the failed tests along with a risk mitigation plan for high-risk test. Defect tracking should include basic metrics for compliance based on number and type of defect found.
Roles and Responsibilities
A matrix listing the roles and responsibilities of everyone involved in the testing activities, along with the anticipated amount of their time allocated to the project, must be prepared.
Process Improvement
The entire testing process should be focused on process improvement. The strategy should list ways to monitor progress and provide constant feedback. This feedback can serve to enhance the process, deliverables and metrics used in the testing. Root cause analysis should be performed on all reported defects to help isolate the true nature of the problem and prevent unnecessary repeat offenses.
Deliverables
All deliverables should be defined and their location specified. Common deliverables are test plans, test cases, test scripts, test matrix and a defect log.
Schedule
All testing activities should be combined into one master testing schedule. The schedule should include an estimate of time for each task and the dependences for each. Testing resources should be assigned to each task and quality gates should be listed to insure oversight of the entire process.
Environmental Needs
All the requirements of the testing environment need to be listed. Common ones include a description of the environment's use, management, hardware and software, specific tools needed, data loading and security requirements.
Resource Management
The skills of all personnel involved in the testing effort need to be assessed and the gaps noted so that a comprehensive training program can be designed. Specialty skills that will not be filled with in-house staff will require job descriptions and budgeting.
Risk and Contingencies
Planning for risk in advance and ways to mitigate it are essential for a robust strategy. A risk assessment that is prioritized by severity of risk and covers technology, resource, schedule and environmental issues should feed a detailed plan to mitigate each red flag.
Approvals and Workflow
All items on the critical path must go through an approval cycle. The procedures for approval and escalation must be well defined and assigned to resources prior to the start of the testing.
The above covers the main sections of a well-drafted and documented testing strategy. The more detail that you include in the strategy document, the less ambiguity and chance for deviation there will be throughout the project
The completion of the strategy signals the beginning of the test planning phase. For each type of testing identified in the master test strategy there should be a test plan identifying the components to be tested, the location of the test data, the test environment needs, the test procedures, resources required, and the tests schedule. For each plan a series of test conditions should be identified so that test cases with expected results can be generated for later execution. -------
What to include in the Testing Strategy
During the analysis phase, you gather and validate the business requirements for the solution. It makes sense that the Testing Strategy is completed during this phase as well. In a sense, you are defining the overall testing requirements.
The purpose of the Testing Strategy is to define the overall context for the entire testing process. The process is different depending on the specific characteristics of your solution. In many respects, this is the most important part of the testing process, since all future testing decisions will be made within the context of the strategy. Here are the basic parts of the testing strategy:
- Project Overview: You can copy this from the Project Definition.
- Business Risks: These are high-level risks of the project that will affect the overall testing strategy. For instance, the risk of doing business on the Internet may drive the need for rigorous system tests of firewalls, technical architecture, and security. The risks can be classified as high, medium, and low, depending on the nature and impact of the problem. For each high and medium risk, identify what elements in the overall testing approach will help ensure that the potential problem does not occur.
- Testing Milestones: This section gives the reader a preliminary overview of the testing timelines. Obviously, since this document is created in the analysis phase, these dates are subject to later revision.
- Testing Approach: This describes the testing process at a high level, including how you will conduct unit testing, integration testing, system testing, and acceptance testing. (If your project is large enough, each of these might be its own section.) This is where you make fundamental decisions regarding the type of testing that makes sense for your project. For instance, if you are implementing a packaged solution, the approach may start in system testing, with the vendor providing close support. If you are doing iterative development cycles, the testing approach will reflect this overall development life cycle. For system testing, define the major testing events, such as stress testing, security testing, disaster recovery testing, usability testing, and response time testing.
- Testing Environment: Think through the technologies and facilities needed for the testing process. If the overall testing environment needs are understood up front, it will be easier to break out the specific activities required to put the environment in place. In addition, you may need to plan for and acquire some parts of the environment well in advance.
Depending on your project, there may be other high-level sections to include, such as testing objectives, testing assumptions, testing organization, and testing tools, along with effort and cost estimates.
-------
Key points to remember
The developer’s role here is not trivial. As you contribute to this strategy document, keep these three points in mind:
- If you are working on a large project, you need to formulate an overall Testing Strategy during the analysis phase. The Testing Strategy defines the overall approach to testing and describes how the testing process will ensure that the solution has the appropriate level of quality and reliability.
- The Testing Strategy provides the overall guidelines from which all future testing decisions are made. A well-crafted Testing Strategy allows the rest of the testing process to be defined more effectively.
- The Testing Strategy needs to be understood and approved by the sponsor. If the strategy is accepted, there is a much better likelihood that the final solution will meet the customer’s expectations.