Release judgment others・Annoying relationship between testing and bugs
- 1. When looking at the status of bugs in the release judgment, it is a prerequisite that the quantity and quality of the test can be confirmed.
- 2. Tests cannot find all bugs
- 3. Zero bugs if you don’t test
- 4. Zero bugs found in low-performance tests
- 5. If you want to measure the quality of software by the bug detection rate (bug / KLOC)
When looking at the status of bugs in the release judgment, it is a prerequisite that the quantity and quality of the test can be confirmed.
The most intuitive and easy-to-understand way to judge the quality of software at the time of release judgment is whether there are many or few bugs . Software with many bugs has poor quality, and software with few bugs has good quality, which is a very easy-to-understand judgment method. By the way, how do you count bugs? You usually count the number of bugs you find by testing.
So, is the number of bugs found in the test the number of bugs in the software? This cannot be said to be the case. You can count the number of bugs you’ve found in the tests you’ve done so far, but not the number of bugs you might see in these tests you plan to do. In addition, I don’t know how many bugs are lurking somewhere, although I can’t find them in the tests I’m planning to do.
Tests cannot find all bugs
The bug sneaks into the code when you write the source code. No one knows how many bugs are lurking in all areas of the source code. For such latent bugs, (1) prepare various environments and check the actual machine to execute the software after satisfying various conditions, and (2) the designer reads and analyzes the source code. The software test work is to find out by performing the work of desk examination such as analysis using tools. However, there are some bugs that cannot be found in these tasks.
Well, in short, the general term for bugs is " the number of bugs found in the test “, not " the number of all bugs in the software " . And this emerges as an annoying relationship between testing and bugs at the time of release decision. One of the important indicators to judge the quality of software at the time of release judgment is the number of bugs, but since the number of all bugs is not known, it is not possible to judge whether all the bugs have been found, and the quality of the software This is because it becomes difficult to judge whether it is good or bad.
Zero bugs if you don’t test
For example, if you have software A and software B of the same scale, and the number of bugs in A is 100 and the number of bugs in B is 200, is A better in quality? It’s annoying that I can’t say that until I compare the quality and quantity of the tests.
If software A has 1,000 tests and software B has 3,000 tests, and the result is the number of bugs mentioned earlier, which is better, A or B? When calculating the bug detection rate per test, software A has 0.1 cases / test software B has 0.07 cases / test, and from the viewpoint of bug detection rate, software B seems to have better quality.
Well, in short, the number of bugs depends on the number of tests performed, which is a very simple story. Therefore, the quality is classified as follows according to the amount of tests and the number of bugs. There is no problem with the judgment of ② and ③, but it is difficult to judge whether the quality of ② and ④ is good or bad, so be careful.
The table below shows how the quality of the software changes depending on the amount of tests and the number of bugs.
| Relationship between test volume, number of bugs, and quality | Found a large number of bugs | Found a small number of bugs |
| Conducted a large number of tests | ① Is the quality bad? |
② Excellent |
| Perform a small amount of testing |
③ Oh my God! |
④ Is the quality good? |
Zero bugs found in low-performance tests
By the way, if you don’t find many bugs as a result of a lot of tests, is it safe to say that the quality of the software is good? Actually, there is a pitfall here as well. Tests find bugs when you run tests that are highly capable of finding bugs . For example, if you have an input item with an input range of 1 to 100, you can probably not find a bug by performing an input test of 1, 2, 3, …, 100 and 100 items. But if you enter a value like -1 or 999999999, you may find a bug.
In short, it’s no wonder that you can’t find potential bugs, no matter how many tests you increase, unless you have a good test with a good ability to find bugs. The table below shows how the quality of software depends on the amount of tests and the number of bugs.
| Relationship between test quality, number of bugs and quality | Found a large number of bugs | Found a small number of bugs |
| Perform good quality tests | ① Is the quality bad? | ② Excellent |
| Conduct a bad test | ③ Oh my God! | ④ Is the quality good? |
If you want to measure the quality of software by the bug detection rate (bug / KLOC)
After all, you’ve just taken it for granted that test quality and test quantity have a significant impact on the number of bugs you see in your tests. When judging the quality of software by the number of bugs or the bug detection rate
① Test density (number of tests / KLOC) and
( 2 ) Looking at the test quality (which cannot be measured directly)
It means that we have to judge the quality as a whole. Since there is no way to judge that this is correct, it is quite annoying to have to look at the test status and bug status of many software and get an intuition about the quantity and quality of the test. It was a story about the annoying relationship between testing and bugs.