Release judgment others・Special recruitment is judged by test status, latent bugs and residual bugs

16/12/202020/10/2021Release Judgment

The reason for the special adoption of software is either untested, the risk of potential bugs, or the status of residual bugs.

In the previous article, I introduced that there is a possibility of releasing software by special adoption, but in fact, Gutara’s father approved the release of software by special adoption in any of the following 3 cases.

1. Some of the planned tests have not been completed yet (insufficient process quality)
2. The identification of potential bugs has not been completed yet (product quality)
3. The impact of potential bugs on service provision remains (product quality)

So, let’s introduce them a little more concretely in order.

Special recruitment due to partial incompleteness of planned tests

It 's impossible to release a part of the planned test if you think that it is incomplete . It’s too violent to release it before it’s been tested. However, there are tatemae and real intentions in the world.

For example, most of the tests have already been completed, but due to various reasons, some tests are still being tested at this time, and the tests have not been completed . Judging from the test results of the part that has already been completed, the risk of discovering new problems even in the test being conducted now is quite low . But the fact remains that the test isn’t over. However, the release date has come, which cannot be delayed in terms of business. What should I do? ?? ?? What would you do in such a case?　

Let’s assume a more specific case. For example, let’s say you have a test item called a 30-day energization test to confirm stable operation for a long time after starting up. And suppose that the test hasn’t been completed until the 27th as of today due to the delay in starting the test .　

Normally, if you do not test for another 3 days, you will not pass the 30-day energization test. Therefore, even if all the other tests have been completed, the test is incomplete . However, sales say that the software must be released tomorrow.　

In another example, suppose you plan to collaborate with a customer in a test environment close to production prepared by the customer as a test to confirm validity . The test parts shared by the company have been executed, but the test parts that the customer is in charge of have completed up to 98%, but the test implementation has been delayed for the remaining 2%, and the completion is 3 It will be a day later .　　

The unfinished test parts have been completed in the test using the in- house test environment and there are no problems, but the test plan has not been completed yet. But I have to release the software tomorrow.

At what time is it okay if some tests are not completed?

In this way, even if the stability confirmation test, which requires a long time, is not completed only for a part of the last period , or if the test is not completed due to a test delay of only a few days, there are other cases. If you 've tested your environment and settings and haven’t had any issues , it 's reasonable to assume that the risk of missing potential bugs due to incomplete testing is fairly low .

In such a situation, if you really have to release it today for business purposes, Father Gutara allowed the release as a special recruit only this time. The software will be released by special adoption in a state of insufficient quality that the test is not completed for several days until the test is completed.　

Of course, if a few days have passed since the release and all the tests are completed and there are no problems with the test results, then the specially adopted release will end and it will switch to the normal release.

Special adoption because the identification of potential bugs has not been completed yet

Along with the incompleteness of some tests, the occasional release is a special adoption when the potential bugs have not been fully identified . Father Gutara uses a tool to estimate the reliability growth curve to determine whether the potential bugs have been sufficiently identified, and the bug detection rate indicates what percentage of all bugs are currently detected. Was estimated and calculated , and this value was used for judgment.

For example, an estimated bug detection rate of 91% means that 91% of all bugs in the software have been detected in previous tests, and another 9% of bugs have not been found yet. Here is the estimation result that there will be. It’s an estimated calculation, so there’s no way to tell if there’s really another 9% of bugs lurking . Even so, by using the estimated bug detection rate, the status of potential bugs at the time of release can be used as a criterion for determining whether or not a bug can be released.

Using an estimate of the bug detection rate makes it easier to determine the release, but it also puts annoying situations . For example , suppose you have decided that a bug detection rate of 99% or higher is the pass criteria for soft releases . And suppose that the value of the bug detection rate estimated at the time of lily of a certain software is 98.9%, which is 0.1% short . According to the regulations, the bug detection rate has not reached the acceptance criteria, so it will be judged that it cannot be released. But is that all right?

Since it is a fact that the number of bugs that have already been found and the bugs that have not been fixed in time are true , there are various points to consider when deciding whether or not to release based on this, such as how to determine the judgment standard value. The methodology is correct.　

But when judging the release based on the estimated calculated value of the bug detection rate , is it correct to judge by the scoop ruler that the release is not possible because the specified value was not reached a little? Is the estimated bug detection rate so reliable ?

The accuracy of the estimated value used for judgment is not high, so let’s cover it with experience and intuition.

It is important to adhere to the acceptance criteria set by the rules . That’s right, but in fact, the method of estimating the bug detection rate based on this reliability growth curve is quite rough . It seems as if the numerical value is output, but it is not based on a solid theory. Based on the rule of thumb that bug detection will be in the form of a graph like this, we just apply the curve of the graph to the actual value of bug detection and estimate that bugs of this degree will be found in the future . Therefore, the bug detection rate obtained from it is not based on a solid theory, so it is not very accurate .

Since such an estimate did not reach the 0.1% acceptance criteria, it seems a little strange to judge that it cannot be released. So, at such times, Gutara’s father allowed the release of software by special adoption in some cases .

In some cases, it is possible to judge that there is no problem with the quality of the software by looking at various indicators other than the bug detection rate . At the time of software release judgment, there are various items to check, such as whether the test is sufficient in quantity and quality, and whether there is a problem with the pointed out status of design review and code review. If those various indicators indicate that there is no problem with the quality of the software, even if the bug detection rate does not meet the acceptance criteria a little, the release of the software was permitted by special adoption. ..

Of course, if the software is specially adopted and the release permission is given to the rejected product because the bug detection rate does not meet the standard, the test is continued after the release and the bug detection rate is waited for to decrease . The bug detection rate will gradually improve as long as new bugs are not found even after testing, so it will eventually meet the acceptance criteria. At that time, the specially adopted release will switch to the normal release, so it is necessary to continue testing until that time.