Extensive Hands On Experience
C/J SYSTEM SOLUTIONS, INC.
Consulting and Training for Computer Professionals
___________________________________
Planning for more effective testing
Does Testing require a plan? That may sound like a silly question, but many organizations leave the testing to the end of the project, and whatever time is left is the time that will be spent on testing. But that philosophy brings up many questions. What is the purpose of any test plan, what is to be accomplished with the testing, how do we allocate the resources for testing, how much time should be spent on testing, and of course how much testing is enough? In the event that defects are uncovered with they be resolved prior to shipment? Will the client be made aware of these problems? Have resources been allocated to resolve the critical defects? Many people assume that testing will continue until the product ships, but there must be a plan on how to do the required testing and how much resources will be available and when will they be necessary. Testing like any other part of the project must have a plan on how to accomplish the objectives. Budgets, people, time, required testing, all have to be identified up front and scheduled during the project cycle.
What is the purpose of the test plan? The test plan identifies what needs to be tested, it is essential to identify what needs to be tested, and prioritize those tasks before we can begin to allocate resources, such as budget, time, people, tools, etc... So how do we communicate the client’s needs to the test group? It should be the same way we identify all of our resources, by understanding the requirements. Requirements are identified as the must-haves. Clients may request more than what the project is capable of delivering, so we must separate the “needs from the wants.” This must be done prior to the beginning of any design or coding. Is there a clear understand of requirements prior to the beginning of the project? The requirements gathering process should eliminate most of the scope creep associated with the project. Keep in mind that the overall purpose of testing is to demonstrate that the product will meet the client’s needs, and not just to demonstrate the program runs. If these needs are not clearly identified prior to the design phase time may be spent on less important features and miss the required functions. If we fail to do this then the entire project is a waste of time. The objective of any project should be customer satisfaction. They only way this can be achieved is if everyone is in agreement with what the final outcome of the project should be.
Testing must be done in parallel with development. Programs should be designed modularly so they can be tested as the project moves along, not just at the end. The best time to test is immediately after the programmer is finished with the module. It is still fresh in their heads and any problems encountered can be readily addressed and resolved. Additionally, if there was a misunderstanding in the requirements they need to be resolved now and then shared with the other project members. It is also highly likely that the programmer will be removed from the project when his/her work is done and it is often difficult to get these resources back in the event a problem happens. This same problem will be prevented from happening again if it is uncovered early in the project. It is much cheaper to do it correctly the first time than to redo it later on. This is part of the “lessons learned” during the project. It is also very likely that some of the original staff will be reassigned during the project cycle and if they are who will be resolving the problems? Where will these resources come from?
We all too often hear the phrase, “It works on my machine.” The program may be right, that is, it runs as designed, but does it meet the user’s needs? Is it the right program? This must be demonstrated early in the project, not only at the end. If it is necessary to fix problems later on, how much of the additional coding and testing will have to be retested, regression testing, to demonstrate that nothing has been negatively affected. Regression testing is rarely done on a full system since there is never enough time to retest. But if it is touched, it must be retested, and also any other features affected by the change. Regression testing must be done throughout the project cycle, so there are no surprises at the end.
What are Scripts and Test Cases? These two are often confused as being the same thing. The script is the scenario of what events will happen, and the test case is the parameter that is passed to the script. Scripts are reusable and the test case will verify the program does or does not work. There will always be a minimum of two test cases, one positive and one negative. The same script can be used with the negative and positive parameter passed to the script. Again it is not the quantity of testing but the quality that counts. The number of test cases will be determined by the importance of the particular feature and to the ingenuity of the tester. Test scripts and test cases should be saved in a folder that is labeled by function, so they can be reused on other projects that are testing for the same functionality. It is not always a good idea to bury scripts and test cases in the test plan as they will be very difficult to recall when needed. Having them reside in a folders makes them more easily accessible and reusable. This will save time and money especially when enhancements or maintenance is required. Regression testing will use these files as well. Test specifications and not the test plan will be used to develop test cases and test scripts. The test specification will have define the functionality in greater detail. Any testing that is done needs to be documented so that in the event of a defect or enhancement, the recertification of the system will be much easier.
Critical Success Factors are those features or functions that are absolutely necessary for the success of the program. They should be identified as early as possible and prioritized and scheduled to be done. The Critical Success Factors are most important and need to be coded and tested as soon as possible. If they are not done early, and time runs out, they may be skipped. All people in the project will have their own set of Critical Success Factors. They can be identified by the clients, programmers, project managers, etc... And each must be identified and addressed. This is very difficult to do when the project team is not communicating well. Again the project must work together as a team and include the test group as a part of that team.
How much testing is enough? Quality and quantity do not necessarily go hand in hand. Have you not seen people work for eight hours and do only four hours of work? It goes the other way also. Many people can do the same amount of work in less time than others. This is one of the main reasons for the test plan. Have we achieved the objectives of the test plan? Will the client be pleased with the product? Will it meet the client’s needs? All of the critical success factors must be demonstrated.
The System Test will be performed by the black box test team. This test is often referred to as the Systems and Integration Testing Cycle. However, integration testing should have been done all along the project cycle and not left to the end. If it fails at this time, there is little time to fix it before the product ships. This testing should be an accumulation of all the module testing, but now done as a complete system. The level of confidence of the system should be high prior to the beginning of system test. No critical success factor errors should be uncovered at system test time. If there are any critical factor errors uncovered then there is something wrong with the process. There should be no surprises during this testing. “What you see is what you get”, is the motto of the system test cycle. The system test is a dry run of the Acceptance Test.
In additional to the standard Black Box Testing, that testing done by the test team, there will be additional testing beyond the scope of the test team. The scope of this testing will be determined by the needs of each organization. These are some of the additional testing that may be done on the product:
Alpha testing – This testing will be done by people outside of the test group. This may include the Project Managers, Business Analysts, Support Center, and internal users. This additional testing allows others to review and test according to their understanding of the system. Since these people are external to the test group, it will provide a different set of eyes and possible different test scripts and test cases. These test cases should be saved for future use by the test team.
Beta Testing – This testing is done at the client’s site and environment. Often referred to as Pilot testing. This testing will help to verify that the product will work in the client’s environment. Sometimes, the client will have features or functions not fully identified prior to the start of the project and may present problems unanticipated by the team. These problems need to be identified and addressed prior to delivery. The Beta Testing should also have a test plan to ensure that testing is complete and applicable.
Parallel Testing – This testing compares old versus new. Does the new system still have the features of the old and if not, was that the intent? If not, these discrepancies need to be addressed prior to the delivery of the system. Rarely will the new work like the old. This is why the system was upgraded. Have some of the previous features stopped working? Was that the plan or did something go wrong? Was a particular feature inadvertently changed?
All of the test scripts and test cases used during these three testing phases should be captured and added to the test bed repository. Since these tests will be done by other than testers, they should add valuable input on how to improve the succeeding test cycles. This will enhance the data base and in future applications, these errors will be prevented from making it through to these cycles. Lessons learned.
Acceptance Testing – The Acceptance test should have been identified prior to the beginning of development and testing. This test will be constantly be referred to during the project cycle to ensure that the overall objective will be achieved and there will be no surprises. The Acceptance test, sometimes referred to as the UAT, will be performed by the client with the cooperation of the Business Analyst.
There is a need to demonstrate that what used to work still works during each and every one of these test cycles. Too often testing is concentrated on the changes and does not include recertification that what used to work still does. Many projects fail because the fix caused a different function to stop working. This is why it is so important to have a library of regression tests.
There must be Test Plans for each one of these testing cycles. It is not the responsibility of the test group to design all these test plans, but each of these testing groups must design their own. These plans should also include Scripts and Test Cases. Testing by the seat of the pants will invariably end up with a product that is not going to meet the client’s needs.
There is a lot more to testing that meets the eye. It is a process that can be reused and will be constantly modified and enhanced. The test plan should be a template that can more easily be accessed and customized to a specific project. Good test planning will result in cost savings not only during the project cycle, but will greatly reduce ongoing maintenance costs.
C/J SYSTEM SOLUTIONS, INC.
450 West Hollis Street Unit A
NASHUA, NH 03062
Phone (603) 521-8544
Email: jyorknh@cjsysnh.com | 
