Test plan・Test Goal (Part 2) Process Quality, Work Quality and Test Management
Test goals should also include process quality, work quality and control
In the previous article, we introduced the important test goals, delivery time goals, cost goals, and product quality goals . In addition to these three goals, there are also test process quality goals, test work quality goals, and test management goals , so this article will introduce these three goals.
Test process quality goals focus on reviews and types of bugs found
The quality of the test process is a little difficult to understand, but in short , the quality of the process itself and the result of executing the process are good, whether it is a procedure for designing a good test and executing the test . Quality. For these as well, if you set quantitative goals, you can use them for test management. Let’s take a closer look, in turn.
(1) Review frequency of test design
In test design, specific test items and procedures are designed, and test procedures and test instructions are created by writing them down in the actual test method . These test procedures and test instructions are the deliverables of the test design, so we will review the test design to check if they are done properly and to identify any problems . How often you review your test design is an indicator of the quality of your test design process.
As a way of showing of reviewers frequency, review the number / number of page test manual procedures and review the number of times / refurbishment source code KLOC , etc., There are many in the way of taking the denominator used for normalization. It doesn’t matter how you express it, so if you set the frequency of views as a numerical goal , you can manage whether the review of the intensity that you thought at the time of planning the test is actually done.
(2) Pointing rate in test design review
A review of the test design will reveal some indications. By normalizing the number of indications and expressing it by the total number of indications / the number of pages of the test procedure and the indication rate of the total number of indications / KLOC of the repair source code, it is possible to set it as a target for the result of the test process called review. I can do it.
However, it is a little difficult to manage the test process using the point rate because it is not possible to say unconditionally whether the point rate of the review should be large or small . This is because the number of points pointed out in the review is influenced by the quality of the test specifications and test procedure manuals to be reviewed and the skill of the reviewer who reviews . Poor test specifications and test runbooks should increase the number of reviews pointed out, but if the reviewer does not have sufficient knowledge and experience, the number of points pointed out will decrease.
Therefore, when managing the review process quality by setting the review point rate as a target , keep the level of the reviewer’s knowledge and experience as constant as possible, or use the reviewer’s knowledge and experience to control the review point rate. It is necessary to think of a mechanism to adjust the judgment criteria of.
(3) Distribution of defects found in the test ( ratio of simple defects to complex defects and very complicated defects)
When you run a test, you will find defects, but some are easy to find and some are very difficult to find. As the test progresses, the defects that are easily found are fixed and reduced, so there are still some defects that are difficult to find at the final stage of the test. One of the quality goals of the test process is to classify the found defects into the following three types according to the ease of finding the defects and set the ratio as the target .
- Simple defect : The condition where the defect occurs is only one
- Complex bug : A bug that occurs when two or more conditions occur.
- Very complicated bug : When there are two or more occurrence conditions and one of them is a stochastic condition (it occurs once when repeated several times, etc.)
If the test period is sufficient and a good test design is made, very complicated defects will be found at a certain rate . Conversely, a very low percentage of complex defects can lead to problems in test duration and test design.
Quality goals for testing work use efficiency and defect recall
Once the test procedure and test instructions are completed, the actual test work will proceed according to them, but you will be able to manage the test by setting some quality goals for this test work as well.
(1) Test work efficiency (how many items are confirmed in one work)
The items to be confirmed in the test work are not limited to one for each test work. In some cases, one test can perform two or three checks at the same time. Of course, it is more efficient to be able to perform multiple checks in one test operation than to perform individual test operations one by one. Therefore, the work efficiency of the test can be expressed numerically by the number of items confirmed in all tests / the number of work in all tests .
Do Komu put a confirmation of what items to one of the test work, come determined by the stage of test design , so the target value of the work efficiency of the test when you end the test design is determined will. If the test goes according to plan, the work efficiency of the test is the same as the planned value, but the actual test often does not go according to plan.
For example, when I tested it, the result was NG , so I may run the same test again on the software that corrected it . In this case, since one test item is tested twice, the test efficiency will decrease . In another analogy, the plan was to perform three checks in one test, but in some cases some equipment may not be ready in time and one check may have to be redone later . At this time, the plan was to perform three confirmations in one test, but three confirmations will be performed in two tests, which also causes a decrease in test efficiency .
In this way, by targeting test efficiency at the planning stage, you can control whether the test work is proceeding as planned .
(2) Test automation rate (what percentage of the confirmation items can be confirmed by the automatic test)
As the scale of software grows, so does the scale of testing, so there are increasing attempts to automate testing as much as possible . If your company has some test automation in place, then the rate of test automation is also a quality goal. You can calculate what percentage of confirmations can be automated by the number of items to be confirmed by automation / the number of items to be confirmed in all tests .
At the time of planning, if you set the target value of the automation rate based on the existing automation test and the automation test newly developed this time, how many automation tests could be made in the actual test work? You can manage it.
(3) Defect recall rate (degree of defect recall depending on the environment and procedure)
It is important to be able to reproduce the same defect that was detected in the test in order to investigate the cause and take countermeasures afterwards . In order to reproduce the defect, it is important to accurately record the conditions for causing the defect, and if this is insufficient, the defect found will not be reproduced and the cause of the defect will be overlooked. Things also happen. Therefore, one of the test quality goals is the defect recall rate, which can be calculated from the number of defects that can be reproduced / the number of defects found .
Ideally, the recall rate of the defect is 100%, but it cannot be reproduced because it cannot be reproduced without some test data, such as a defect that occurs only once after trying tens of thousands of times. There are some problems that can not be reproduced in reality . I think that it can be estimated to some extent from the experience of the test so far that there is a defect that cannot be reproduced at what ratio, so it is better to set the target of the defect recall rate from that value.
Goals for test management are schedule maintenance and deliverable completion rate
Finally, there are goals for test management. Software testing is often invisible, so management is important, but it is necessary to monitor and manage whether test management itself is done properly, so this is the goal for that.
(1) Maintenance rate of test schedule ( maintenance rate of Miles Stone)
Test schedule management is the most frequently used item in test management. If the test is not completed by the scheduled release date, it will be a big problem, so it is necessary to manage the schedule firmly. To quantitatively indicate whether the test schedule is proceeding as planned, use the schedule maintenance rate calculated by the number of milestones completed by the planned date / the number of milestones planned so far. Is good.
For example, if you have set 10 test milestones to date, 7 of which have been achieved by the planned date, but 3 have been achieved later than the planned date, as of today. The schedule maintenance rate is 7/10, which is 70%. Since the target of the maintenance rate of this schedule is 100% , it is possible to judge the quality of schedule management by monitoring the actual results.
(2) Completion rate of test products ( completion rate of test results)
As with the test schedule, test deliverables are also an important item for test management. Test deliverables are test results and test reports that analyze and organize the test results . These test results are also scheduled at the time of planning to produce those results. Therefore, the completion rate of deliverables calculated by the number of test deliverables completed to date / the number of deliverables planned to be completed to date can also be used as a goal for managing tests.
For example, if you have a plan to get 300 test results to date, and 250 test results are actually obtained, the completion rate of the deliverable is 250/300, which is 83%. Since the target for the completion rate of test products is 100% , it can be used to judge the quality of test management by monitoring the results.
(3) Average number of days to deal with defects (average number of days from finding a defect to completing the response)
Another commonly used test management goal is the average number of days it takes to fix a defect. By counting the number of days from the day when the defect is detected to the day when the response is completed for each defect and calculating the average number of days, the average number of days for defect response can be found.
The average number of defect response days depends on the accuracy of the defect reproduction method recorded by the test team, the investigation / countermeasure ability of the design team that investigates / countermeasures the defect, and the spare capacity of the test team that retests the software that has been addressed. Will be done. Therefore, it is quite difficult to set the target value of the average number of days to deal with defects , but from the experience of software development and software testing so far, we searched for software development of similar scale and difficulty. I think it is better to aim for the actual value at that time.