Test report・Write an evaluation of test efficiency in the evaluation of the test plan

06/08/202121/10/2021Test qualityy

The efficiency of the test work is to test the evaluation of about also written evaluation

Write the test result and test evaluation in the test report of the software. The status of the bug detected in the test is the test result, and the good or bad of the test plan and implementation is the test evaluation . Of the test evaluations, the evaluation of the test content and the evaluation of the test schedule were introduced in the previous two articles, so let’s introduce the evaluation of the test efficiency concretely in this article .

Evaluation of test efficiency is measured by the number of test work items or the number of test confirmation items.

Test efficiency is the cost performance of a test obtained by dividing the test result by the test cost, but various test efficiencies can be considered depending on how the test results are counted. Are the following four typical ones?

1. Number of test work items / test cost : Cost performance for test work volume
2. Number of test confirmation items / test cost : Cost performance for test confirmation contents
3. Number of detected bugs / test cost　　: Cost performance for bugs detected in the test
4. Number of fixed bugs / test cost　　: Cost performance for bugs detected and fixed in the test

Since the main purpose of the test is to improve the quality by fixing bugs, it is better to use 4 fixed bugs / test cost or 3 detected bugs / test cost as the test efficiency, but the number of bugs found in the test is It depends on the quality of the software under test . A lot of bugs can be found in poor quality software with many bugs, but it is natural that only a few bugs can be found in good quality software with few bugs.

But test efficiency purpose of that order to calculate the improvement of the work efficiency of the test work itself , so, and the value of the test efficiency is dependent on the quality of the original software, obtained by calculation difficult to use the value of the test efficiency is It will end up. Therefore, Father Gutara used 1 test work item number / test cost or 2 test confirmation item number / test cost as the test efficiency.

The number of test work items is simple, but it is easy to see the efficiency of test preparation etc.

Test work number of items when calculating the test efficiency, which is calculated in the test work item number / test costs, ultimately of whether the test was carried out work of what item, test work the number of items of the total is. For example, if the test work called Function 1 Confirmation Test-A is performed once for the first version software and twice for the bug-fixed version software, a total of three times, the test work for Function 1 Confirmation Test-A will be performed. Since it has been carried out a total of 3 times, it is counted as 3 times as the number of test work items.

The test efficiency obtained by counting and summing the number of test work items for all tests by this counting method and dividing by the test cost incurred is an index showing how efficiently the test work was carried out. increase. In addition to the time during the actual test, the test work also requires work such as preparing the test environment and cleaning up after the test. The time before and after such a test can be saved by performing tests that use the same test environment at once.

The value of the test efficiency calculated by the number of test work items represents the efficiency of the test work including such test preparation, so for example, it is possible to reduce the total test work by devising how much compared to the previous test. It is useful for evaluation such as whether it was done.

The number of test confirmation items can be used to judge the status of test management for software quality shortages.

In terms of the number of test confirmation items / test cost, contrary to the previous one, the same test item is counted as one item no matter how many times it is repeated . Function 1 Confirmation test-Even if A is performed three times including the first time, only the result of the last test is meaningful when finally evaluating the quality of the software. The results of the previous two tests will be used for debugging work in the middle, but it is whether the result of the final test is OK or NG that shows the quality of the final software .

In order to judge the quality of the final software, it is not the number of test operations performed, but the number of test items tested, so the test efficiency is calculated from that perspective. What we do is the number of test confirmation items / test cost. Therefore, in order to calculate the number of test confirmation items / test cost, count the number of test items for which the execution was completed and the OK / NG result was obtained at the end of the test , and the test cost multiplied by that value is used. Divide.

The number of test confirmation items / test cost value is 　greatly affected by ( 1 ) the man-hours required for the test work and (2) how many times the same test was repeated . In particular, if the same test in (2) is repeated many times, the value of the number of test confirmation items / test cost will decrease. In other words, when the number of test confirmation items / test cost value is too small compared to other tests so far, it can be considered that the same test is repeated a lot .

In such a case, the quality of the software to be tested is often poor and it is difficult to remove the bugs. Normally, the test execution is interrupted and the software to be tested is designed and implemented by the department. Test management actions are often required to revert to. The value of test efficiency based on the number of test confirmation items can also be used to judge the quality of test management actions against such lack of software quality, so it is better to calculate it if possible.

The test cost is mainly the labor cost for designing and executing the test, and the equipment cost and equipment cost are added if necessary.

What is the best test cost to calculate test efficiency? Test costs include labor costs required for test design and execution, equipment costs required for test environment maintenance, and equipment costs　such as test equipment and data . In general, the labor cost is the largest cost, so I think it is good to use the labor cost for designing and executing the test as the test cost.

However, in the case of an organization where test automation is progressing, the recovery of development investment and capital investment for automation (equipment amortization cost) may be more costly than the labor cost . Alternatively, a large amount of test data preparation costs may be required. In such a case, it is better to calculate the test efficiency by including the equipment cost and the preparation cost such as data in the test cost.

Test automation aligns the development investment required for efficiency improvement and automation by automation

When evaluating test efficiency, it is also necessary to evaluate test automation . The higher the rate of test automation, the more efficient the test may seem, but in reality, automating tests requires development costs for automation and maintenance costs for automation .

The development costs of automation, for automation of test preparation tools software or, necessary for the tool software or the setting called, cost of the initial development is. The maintenance cost of automation is the cost to readjust the automation software and tool settings once developed according to the test contents and small changes of the test target .

Test automation is useful for improving test efficiency, but it requires a large amount of development cost at the beginning and a considerable maintenance cost depending on the construction , so it is necessary to repeat the same test over and over again. In some cases, it may not be possible to contribute to reducing test costs by exceeding development costs. If test automation is incorporated to some extent, the test cost that can be reduced by automation and the development cost + maintenance cost required for the introduction of automation are listed together, and it can really be effective in reducing the test cost. It is necessary to evaluate whether or not.