Release judgment criteria・See the status of bug detection for evaluate of testing
- 1. The status of bug detection and bug fixing is also important for test quality in release judgment
- 2. Look at the number of bugs found in the test with a normalized detection rate
- 3. Check the bug detection rate in the middle of the test
- 4. How was it simply compared to before?
- 5. What can you tell from such a sloppy thing?
- 6. Check the content of the found bug at the end of the test
- 7. Be careful if there are few stability bugs late in the test
- 8. Bugs found in the test can read various information indicating the quality of the test.
The status of bug detection and bug fixing is also important for test quality in release judgment
As the implementation status of the test progress of the test to make sure of the following is a test Jo bug has been detected and the situation, found the situation of bug fixes is. It’s annoying that you can’t find any bugs when you test, but it’s also annoying that you can’t find any bugs. So, do you good if it finds any degree of bug ?
The number of bugs found and the content of the bugs depend on the quantity and quality of the tests, as well as the potential for the software under test . This area is written in a separate article as an annoying relationship between testing and bugs, so if you are interested, please refer to that as well.
In order to judge the quality of the test from the status of the bugs found in the test, Father Gutara was careful about the number of bugs found and the content of the bugs found . Although it is qualitative, the quality of the test was judged from the following viewpoints.
Look at the number of bugs found in the test with a normalized detection rate
Regarding the number of bugs found in the test , the absolute number depends on the scale of the software and the scale of the test, so the validity cannot be judged unless it is normalized by something .
If you can use the number of test items as the denominator to normalize , this value is good because you can use the bug detection rate calculated by XX detection bugs / YY test items . There are automatic test and manual test as the test execution method, but the number of test items can be used for both. However, it is a little troublesome because it requires a mechanism to aggregate the number of test items. If the test is mainly manual testing, you may calculate the bug detection rate with XX detection bugs / YY man-hours of test effort.
Father Gutara was looking at the number of detected bugs in XX detection bugs / test man-hours YY man-days . This is because all software development projects did not have a mechanism to count the number of test items, so we used the manual test man-hours that can be used in any project.
Check the bug detection rate in the middle of the test
Bugs are found when the test starts, but at the start of the test, the progress of the test is generally poor, and the number of bugs found varies.
At the start of the test, the test environment may not be stable enough and the test work itself may not proceed smoothly. Also, in anticipation of this, we may make a test plan so that we will concentrate on checking the very basic operation that will surely pass at the start of the test. The idea is to make fine adjustments to the test environment in the basic operation check test.
So, Father Gutara checked the bug detection rate in the middle of the test, when the test environment was stable enough and the test team became accustomed to testing the software .
How was it simply compared to before?
So what do you judge from the bug detection rate ? Actually, there is no good criterion to say this . The number of bugs detected depends largely on the number of potential bugs in the underlying source code and the quality of the test, so this value is not the correct answer. So what was Gutara’s father checking for bug detection rates?
If the quality of the source code and the quality of the tests are about the same, the bug detection rate should be about the same. So, I was checking if the bug detection rate for this software was higher or lower than the previous similar development .
In the case of Japanese organizations, both the software development team and the test team are often fixed even though they have an organization . Looking at the quality of many software over the years, we can somehow understand the competence of each development team and test team. As an experience , I can understand that this development team will improve the quality to this extent by the start of system testing, and that this test team will find bugs of this degree .
Regarding the software test this time, if it matches the experience value , it is very rough and unfounded that there is no problem with the quality of the test, but I can judge.
What can you tell from such a sloppy thing?
I can’t say anything because there is little evidence, but when the bug detection rate is high , the quality of software modules installed from outside the company is often poor.
There are various sources, such as purchased functional modules and newly opened open source software, but if there is software that was brought in for the first time from outside and the quality is poor, the bug detection rate Is clearly larger.
On the other hand, when the bug detection rate is low , the number of skilled testers is often reduced due to the replacement of test team personnel .
Testing work, like software design / implementation , requires a high degree of expertise . Along with the desk study to know various methods as knowledge, the test skill that can be trained in many years of practice has a great influence on the bug detection ability. With fewer skilled testers from the test team, bug detection rates are lower, no matter how much past testing experience you have used.
When it comes to the purpose of judging the quality of a test, the decrease in the number of skilled personnel from the latter test team has a great influence on the quality of the test, so when judging the release, we will go into the content of the test and check it carefully. I need it.
Check the content of the found bug at the end of the test
We will check the number of bugs detected as well as the content of the bugs found, but this is mainly checked at the end of the test . By the end of the test, you’ll find stability bugs , such as complex conditions and infrequent occurrences .
These stability bugs are less noticeable until mid-test. Until the middle stage, you will find many bugs that are relatively easy to find , such as bugs related to normal function execution and bugs in a large-scale operating environment . In a situation where there are many such bugs, the system often does not operate stably. Therefore, it is difficult to find stability bugs because it is not possible to sufficiently test the operation under various conditions and the operation when multiple processes are in conflict.
At the end of the test , when a set of bugs are found and fixed, the system will be stable . When this happens, tests to confirm stability, such as competing systems and long-term operation under heavy load , will begin in earnest, and bugs found in those tests will become more noticeable.
Be careful if there are few stability bugs late in the test
From a different point of view , be aware that if you don’t find many stability bugs late in the test, you may have some problems with your test.
Is the quality of the software not good enough to confirm the stability even though the test is nearing the end, or is there a lack of test items to confirm the stability in the test design? Either is happening.
In the former case, the risk of overlooking the problem is low because the situation of quality shortage can be seen from the number of bugs and the estimation of the number of potential bugs by the psychological ad growth curve.
However, the latter test design problem is difficult to find from other indicators and should be detected here. Specifically, it is necessary to go back to check whether the test items related to stability are fully considered at the level of the test design document .
Bugs found in the test can read various information indicating the quality of the test.
So far, we have introduced how to view the bug detection status and bug fix status as one of the means to check the test implementation status. However, the information about the bugs found can tell you a lot, even as information to judge whether the quality of the test is good or bad. In the next article, I’ll show you how you can use the detected bugs as information to judge the quality of the test. ..