Test report・Bugsmust must be prioritized and tracked
Write the distribution of detected bugs in the software test report
Write the test result and test evaluation in the test report of the software . Simply put, the status of the bug detected in the test is the test result, and the quality of the test work is the test evaluation. I write the status of the bug as a test result, but if the number of bugs is large, if I do not analyze and write it well, it will be difficult to use it to judge whether the quality is good or bad at the time of release judgment.
Find out so that you can provide information to judge the quality of the software at the time of release judgment, such as what kind of function had many bugs, what kind of test found many bugs , how many serious bugs were found , etc. Write down the analysis result of the bug in the test report. Let me introduce a few concrete examples based on the example that Father Gutara has seen to see what kind of analysis method is available.
Analyze the number and density of bugs by software function and test type
The first is an analysis from the perspective of where a large number of bugs were detected. When analyzing, the absolute number of bugs detected is also important, but it is natural that the number of bugs in the part where many new developments and fixes are made seems to be large, so normalize with the number of KLOCs in the newly / fixed source code. It is often the case that the uneven distribution of bugs can be correctly expressed by analyzing using the bug density (XX cases / KLOC) and the bug density normalized by the number of test items (XX cases / number of tests), so the absolute number. It is good to write the bug density together.
There are two ways to analyze where a bug was detected: analysis by software function and analysis by test type, so let’s introduce them in order.
(1) Bug detection status for each software function
The easiest way to analyze bugs is to classify the bugs accepted in this test according to which function of the software they are related to, and calculate the number of bugs and the bug density for each function. It is a method of arranging numbers such as 5 cases (0.2 cases / KLOC) for function A and 12 cases (0.5 cases / KLOC) for function B. By doing this , you can see which function has detected many bugs and which function has high bug density .
Anyone can be relieved that the structure of the software is complicated, the processing timing is difficult, and the bug density is high due to the functions that have difficulty in designing and implementing the software. But if it’s a relatively simple feature and the bug density is high , it’s a good idea to check again what caused it. If a lot of bugs due to careless mistakes are found because the person in charge of design / implementation is inexperienced, it is necessary to take supplementary measures such as reviewing the code developed by the person in charge again. maybe.
(2) Bug detection status for each test type
The second method of analysis is analysis for each type of test. I wrote the tests in another article, but they are planned and implemented by classifying them into tests with various names according to various purposes. If you classify the number of bugs detected by this test name , you can see the tendency of what was weak in terms of software quality . In the case of this classification, it is better to use the bug density (XX cases / number of tests) normalized by the number of test items.
For example, if many bugs in normal tests are detected, it should have been done before this test. It can be seen that bugs that should be found in unit tests have leaked to this test. I can do it. It’s possible that the unit testing method was bad, that you didn’t have enough time to do the unit testing, or that there was something wrong with the development process .
If many bugs are found in stability tests such as large-scale tests and high-load tests , this development may have affected the stability of the system . If you see too many bugs, you may have potential bugs that haven’t been found yet, so you might want to postpone the release a bit and test for more stability.
When there is a quality concern about stability etc., if possible, the bugs for each type of test with the software of the previous version of the same product are also listed, and if you look at the difference in bug density, It will also be a material to judge whether it is a problem peculiar to this software development.
Analyze bug density by focusing on the likelihood and importance of bugs
Next to the analysis of where the bug was detected is the analysis from the perspective of what kind of bug was detected. In this analysis, we classify the detected bugs into several categories and analyze what percentage of the total number of bugs were bugs in that category. There are various ways to classify, but the substitutes are the susceptibility to bugs and the importance of bugs, so let’s take a closer look.
(1) Classification according to the likelihood of bugs
A bug causes a bug state when some conditions are met. Bugs that are easy to meet the conditions are easy to occur, and bugs with complicated conditions are unlikely to occur, so the following classification can be made based on the ease of bugs.
- Simple condition bug : A bug that always causes a problem when one condition is met
- Compound condition bug : A bug that always causes a problem when two or more conditions are met at the same time.
- Probability occurrence bug : A bug that causes a problem by establishing once every few times after the occurrence conditions are met.
Probability occurrence bugs are when there is some timing dependency in the occurrence condition . For example, it is a bug that causes a problem when the normal condition that a specific value is input and the condition that depends on the timing when the termination process of communication with the outside is started overlap. In this case, assuming that the probability that the timing to start the communication process with the outside occurs is once in 100 times, if the process of inputting a specific value is executed 100 times, a problem will occur in one of them. Become.
In terms of the difficulty of detecting bugs by testing, or the cost of detecting them (labor, equipment, and time), the order is c> b> a . That is, if the ratio of bugs of b and c is high among the bugs found in the test, it means that the test was able to be carried out so much, and it can be judged that the quality of the test was good. .. On the contrary, if there are only bugs in a, it seems better to review the quality of the test.
(2) Classification by the importance of bugs
It is also important to classify bugs according to the importance of the content of the bug caused by it. A bug that causes a display error that is rarely used and a bug that greatly affects the function of the device are the same bugs, but the latter is a more important bug. An example of classifying bugs from that perspective is as follows.
Minor bug : A bug that does not affect the basic functions of the device, such as a difference in display (Minor Bug)
Normal bug : A bug that causes a slight effect on some basic functions or makes it impossible to use auxiliary functions (Major Bug)
Severe bug : A bug that makes some of the basic functions of the device unusable (Critical Bug)
Urgent bug : Many functions of the device cannot be used (Show Stopper)
If you compare it with game software, a minor bug is a bug that the design display of the character is partially changed. Is it a normal bug that some of the conditions for invoking a specific command are incorrect? Severe bugs have different effects, such as not firing a specific command. Is the emergency bug unable to log in or the game scene does not progress?
If a serious or urgent bug is found just before the end of the test, you will not have time to fix it and your software release plan will be disrupted . Therefore, if the ratio of severe bugs and urgent bugs is high , the risk as a software development project is high, so caution and improvement are required in various aspects. Perhaps we needed a development plan that would put more manpower and time into quality assurance activities in upstream processes such as unit tests and design reviews.
It’s also important to consider bugs that weren’t found.
Although it is essential to analyze the bugs found in the test in the test report, it is also very important to guess the bugs that have not been found yet as information for judging the quality of the software. In the next article, I’ll show you what you can write about bugs you haven’t found yet.