Release judgment criteria・The first type of test is abnormal and semi-normal

12/11/202020/10/2021Release Judgment

The type of test confirms the presence or absence of the test from five viewpoints

Of the four criteria used when determining software release ( test quality , residual bugs , product quality , process quality ), this article introduces how to determine whether the test quality is good or bad. Gutara father is the quality of the test amount of testing and individual types of tests and test the implementation of the bug situation was watching from four perspectives of. In this article we will look at the second type of test.

It is the type of test that is important after the amount of tests from the viewpoint of judging the quality of the test . You cannot judge the quality of a test by looking only at the quantity of the test without seeing whether you are performing the right type of test.

So, kind of test do you want to determine the bad settle for doing what I. Since it is difficult to quantify like the amount of tests, it is a qualitative judgment , but whether or not Father Gutara is conducting the necessary types of tests from the following five viewpoints regarding the types of tests performed. I have judged. (The type of test is sometimes called the test category)

　Ratio of abnormal  test and semi-normal test

   Status of stability and robustness tests

   Implementation status of RAS functional test

   Implementation status of tests related to the version upgrade function

   Confirmation status of past defects

I will introduce the specifics in order, but in this article, I will first introduce the abnormal  test and the semi-normal test . In addition, please refer to another article for details on abnormal system and quasi-normal system tests .

Normal test, semi-normal test and abnormal test

The meaning of the terms normal , semi-normal, and abnormal may also differ slightly from person to person and from organization to organization. Here, a test that confirms that the software operates correctly when the correct input and correct operation are executed in the correct operating environment is called a normal test .

Among the tests other than the normal system , the test that confirms the behavior corresponding to the abnormal state (error state) that is designed / implemented in advance in the software is called the semi-normal test, and the abnormality that is not implemented in the software. The test to confirm the operation when such a state occurs is called an abnormal test .　

For example, the processing when an error is detected in communication or when an incorrect value is input from the terminal is usually designed / implemented in software, so checking the operation in such a case is a semi-normal test. However, the behavior of the software that occurs as a result of the control variables in the memory being garbled to strange values ​​due to foreign is is called an abnormal test because it is not a pre-installed process.

Percentage of required semi-normal and abnormal tests

With this definition, what is the percentage of semi-normal and abnormal tests in all tests in your organization ? It’s a general story, but in embedded software, it is said that 20% of the source code is normal and 80% is written for processing when an error occurs . In fact, even in software module design, most of the design is about the content and response of error handling when an abnormality is detected. Therefore, it is a reasonable idea to think that 20 % of the test item items confirm the operation of the normal test, and most of the remaining 80% confirm the semi-normal test.

By the way, the processing when an error occurs is further divided into two, the first is the processing when a defined / expected error occurs , and the second is an unexpected error. It is the processing of time. In the former, the corresponding processing is implemented in the software for each type of expected error. Since this is a semi-normal process , the semi-normal test corresponds to this.

And when an unexpected error occurs that you did not think of at the design stage, if the default processing to be executed when no conditions are met is implemented in the software, that processing will be effective. Since this is anomalous processing , it is abnomalous processing that corresponds to this. By the way, if the default processing is not implemented, it means that what happens is not controlled by the software, so you do not know what will actually happen, you will be in a state with problems such as hangup. There are also things.

By the way, the processing of the semi-normal test and the processing of the abnormal test are combined to be the processing when an error of 80% occurs, but what is the ratio of the semi-normal test to the abnormal test ? There is no correct answer because it depends on the software configuration.

However, looking at the various products that Father Gutara has seen so far, 70% of the 80% of the processing when an error occurs is a semi-normal test and 10% is an abnormal test, which is a sensory ratio. ..

Percentage of required abnormal tests

Then, when judging the quality of the source code by the release judgment, what should be the ratio of abnormal tests as the quality of the test ? There is no value that must be this, or there is no theoretical background, but Father Gutara considers that the lowest line of the abnormal test ratio is about 60% as the lowest line, and has made a release judgment. I did.

Of course, it is better that the ratio of abnormal tests is about 80%, but if it reaches 80% for all products, it may be difficult to achieve in some cases, so set about 60% as the minimum line. I am. Do you think this value is low or high?

Common sense in Japan and common sense in the world

In fact, the Japanese company and talk to people that he is doing software development and testing in, by combining the semi-normal test and abnormal test, a percentage of the whole of the test is 60% but if less Do not – that You will get a response. On the other hand, when I talk to people from overseas companies , about 10% of tests are non -normal, what are abnormal tests and semi-normal tests? There are many cases where the reaction is very normal, and it can be a little confusing. I think this is a difference in the definition or perception of software quality.

Overseas, quality means that the functions described in the specifications work . And quality assurance means that what is written in the specifications is inspected by a test that it works. The purpose of the test is what is written in the specifications confirmation of that move , so, the normal test will be most test items.

On the other hand, the quality in Japan, features that were written in the specification is no case , even in the stable is to operate. Therefore, in Japan, it is necessary to confirm that the software works properly and guarantee the quality even in an abnormal state such as an incorrect input or a noisy environment , so a semi-normal test and an abnormal test will become necessary.

At the end, the circumstances of adults

Of course, there are companies that think about quality in the same way as in Japan from a global perspective, but many of the overseas companies that Gutara’s father has been working with are still thinking about testing, focusing on normal tests. It was.

If you outsource software development to testing to such a company, it is necessary to give various guidance on the test contents and have them carry out abnormal tests firmly, and we will also give guidance. However, even if you do your best to give guidance, it is often very difficult to bring the ratio of abnormal tests to 80% . Due to such adult circumstances , Father Gutara set the minimum line of 60% for the combined semi-normal test and abnormal test.

Test type Next is stability and robustness

Looking at the types of tests performed when judging the quality of the test, the stability test and robustness test are next to the abnormal test, semi-normal test, and abnormal tests. It is introduced in the next article, so please have a look if you are interested.

Next : Release judgment criteria・The second type of test is stability and robustness