Release judgment criteria・The second type of test is stability and robustness

12/11/202020/10/2021Release Judgment

The second type of test is the implementation status of stability test and robustness test.

This series of articles introduces the types of tests from the perspective of what kind of test you are doing when determining the quality of the test. Previous article was introduced in the normal system and the quasi-normal system and the different atmospheric system is that, easiest if you think about the quality of the test how divided the test is. However, this is a bit lacking in information to judge the quality of the test, is there anything else? Gutara father, software stability and robustness whether that test for the purpose of to make sure has been fully carried out, also in terms of determining the quality of the test had been viewed as information.

What kind of test is a test to confirm the stability of software ? Since it is stable, it is a test to confirm that the software can provide the functions described in the specifications at any time . To give a little concrete example, it’s a test to see if it always works even if the power is left on 24 hours a day, 365 days a year.

How stable is the embedded software required?

The stability of working 24 hours a day, 365 days a year is nothing special in Japan today. For example, for a TV with a recording function that you use at home, even if you turn off the power with the power button on the remote control, the software is actually still working on the TV, and when the reserved time comes, the reserved program is automatically set. Will record. In other words , the software on the TV is always running stably 24 hours a day, 365 days a year, so you can make recording reservations. If the software hangs up and you cannot make a reservation for the recording you have reserved, you will be in trouble.

To prevent this from happening, it is necessary to make sure that the stability of the software is tested at the time of software release and that the test results are correct. Therefore, when judging the quality of a test, whether or not a stability test is performed is an important confirmation point for embedded software. However, if the software can only be released after being tested 365 days after it is completed, it cannot keep up with the speed of development competition these days. What should I do?

How do you determine if the stability test is sufficient?

Father Gutara judged the status of the stability test from two perspectives . The first is the acceleration test . For example, in the case of a device in which the user looks at the screen and performs some operation, the amount of operation per day is estimated assuming the typical operation method of the user. If you perform 365 times the amount of operation, you can consider that you have performed 365 days of testing for user operations, although it is not perfect.　It 's an accelerated test of user operation .

By conducting several types of accelerated tests from this perspective, is it possible to carry out the stability confirmation test required by the time of release judgment ? That is the first confirmation item.

And the second is to check if there is an implementation plan of the test to confirm that it actually operates stably for 365 days or more . Even if you cannot finish checking stable operation for 365 days or more by the time of release judgment, what will you know if you start the test and continue the test after the release judgment?　

If your software has a latent bug that occurs after 365 days of operation, you can expect it to occur in internal testing before users encounter it in the market. increase. Unfortunately, after a bug has hit the market, you’ll have time to detect it before it’s discovered in the market and take steps to minimize the impact of any problems that may occur on the market . ..

Is it a bit exaggerated to check the stability so far?

Some people may wonder if such a thing is really necessary. However, bugs caused by resource leaks such as memory and bugs caused by timer and counter roll-up processing often occur only after a long period of actual operation. And most of the time, bugs caused by leaks and timer counters will occur in all products . As a result, the scope of the bug often spreads to all products in the market, which is very annoying. Father Gutara called this kind of bug a time bomb bug and was the most careful.

Time bomb bugs can be found in acceleration tests, but unfortunately not all cases can be accelerated, so there is a risk of overlooking the bug. As long as you can overlook a time bomb bug, it’s worth the effort if you can find it in-house, even if it’s only a few days, before the bug is revealed in the market . ..

What does the robustness test, which is checked next to stability, check ?

After confirming the stability test status, the next step is to confirm the robustness test status. What is the robustness of software in the first place? There are various definitions, but if you forcibly put them together, it can be said that the software will continue to run even if something unusual happens .　

For example, in situations where software is used, such as when the value entered by the user is incorrect, when a large number of characters such as 100 characters are entered in the 10-character input location, or when the network connection environment suddenly cuts off and is restored immediately. Various abnormal situations occur. At that time, if the software hangs or makes a mistake, it will be a problem. Even in such an abnormal situation, the software that can display a nifty error message without panic and continue to operate is robust software .

So, generally speaking , if you are doing an operation check test of an abnormal system , you can say that you are checking the robustness. However, with this, it is difficult to tell whether the quality of the robustness test is good or bad as a release judgment. Father Gutara was looking to see if several types of tests were being performed to determine the quality of the robustness tests .

• Operation test with repeated signal disconnection / recovery
• Operation test in a noisy environment
• Operation test at maximum configuration and high load

Operation test in signal disconnection / recovery state

The first is an operation test with repeated signal disconnection / recovery . Computers, including embedded systems, basically input some information as a signal, process it, and output it. For example, the smartphone you are using inputs article information from the Internet and outputs it to the screen of the terminal.　

This input signal is interrupted or connected due to various reasons . Even if the input signal is interrupted repeatedly in this way, it is necessary to continue processing without hanging . So, forcibly create an environment for such signal disconnection / recovery, and check if the test is performed to confirm that the software continues to operate stably in such an environment . ..

It seems to be difficult to write in general sentences, but in reality it is not such a big deal. For example, imagine a wireless LAN router in your home . With this wireless LAN router, if the other party’s computer or smartphone is in a room where radio waves are hard to reach, or if the microwave oven that causes noise is switched on, the wireless connection will be cut off or connected again.

So, when releasing the software of the wireless LAN router, for example, bring the terminal of the communication partner to the point where the radio wave reaches or does not reach, and forcibly create a state where the wireless connection is connected or disconnected. We check to see if we are testing whether normal functions work properly in such a state .

Operation test in a noisy environment

Next is the operation test in a noise environment , which is similar to the operation test in the repeated state of signal disconnection / recovery, but it is a little different. Noise can cause signal disconnection / recovery, but it also causes more things in the hardware .　

The easiest thing to understand is the occurrence of incorrect interrupt processing . Software often uses interrupt handling to achieve various functions. The source of this interrupt treatment is a hardware interrupt signal. This interrupt signal is quite sensitive , so even a short amount of noise can pick it up and cause interrupt processing.　

In embedded systems, it is called a noise interrupt . If the software is not designed to correctly ignore this noise interrupt when it is input, various problems will occur. In order to confirm that the software continues to operate properly even if a little noise is generated, it is necessary to confirm from the viewpoint of whether or not the operation check is tested in the environment where noise is generated.

This also seems to be difficult in general sentences, but what if you think about the wireless LAN router above? Wireless LAN routers are usually used at home, so if you look for something that makes noise in your home, you’ll find vacuum cleaners, microwave ovens, dryers, and old-fashioned glow lamp fluorescent lights.　

Therefore, you can check the quality of the robustness test by checking whether the wireless LAN router is tested to work properly next to it while moving the device that generates these noises . In reality, recent home appliances are less likely to make noise, but there are still some home appliances that generate a lot of old noise. So sometimes I made a successful bid for an old home appliance at an online auction and used it for testing.

Old appliances are good for testing wireless functions

Eh ~ Is that so okay? You might think, but that’s fine.　At the design stage , various measurements are required, and expensive ones are often used for dedicated measuring devices . The work scene is also something professional and a little cool.

However, at the testing stage, it is most important to reproduce the environment in which the product is actually used as much as possible and test it there. Therefore, if it is a wireless LAN router for home use, it is important to create a test environment by arranging old and new home appliances that are considered to be common in ordinary homes. And, in fact, old home appliances are quite important for testing wireless functions.

Imagine a house where your grandparents live in the countryside. Home appliances that have been used for a long time are still alive and well, rather than the latest digital home appliances . Fluorescent lamps do not have the current dimming function, but may be the type in which the growth bulb crackles to generate high voltage and activate the fluorescent lamp. (Young people today may not even have seen a fluorescent glow starter)　

The noise environment of electronic devices is considerably wild compared to the environment where the latest digital home appliances in the city are located. Robustness is that the wireless LAN router works properly even in such an environment without hanging .　

Let’s take a look at the maximum configuration and the operation test at high load.

Although the viewpoint is slightly different from robustness, we often see the maximum configuration and operation test under high load at the same time as confirming stability / robustness. It is not so difficult to check the implementation status of the maximum configuration and high load test. The specifications of the developed product describe the maximum number of peripheral devices and higher-level devices that can be connected , and the maximum performance . All you have to do is make sure that you have tested it for its maximum configuration and maximum performance.　

If you write it like this, it looks natural and made. Yes, it’s natural. But in reality, are you sure you can test it in any case? In fact, it can be difficult. Even if it is the maximum configuration of the wireless LAN router in the home that was used in the example earlier, for example, up to 32 wireless slave units can be connected. With this number of units, the devices are actually prepared and tested. It’s not too hard to do.

By the way, what is on the Internet side to which the wireless LAN router is connected? It may go through some devices, but there is a device on the station building side to which the wireless LAN router is connected . It works with one unit connected to hundreds to thousands of wireless LAN routers.　

This means that if the specifications of this station building side device are that the maximum number of wireless LAN routers that can be connected is 4000, at least 4,000 wireless LAN routers will be connected and operate normally when the software is released. I have to confirm. It’s quite difficult because it requires a lot of equipment, space, and electricity.

Is it difficult to test before shipping because such equipment cannot be prepared ? But unfortunately software bugs prefer to live in such areas . Murphy’s Law has not lost its effect in the world of software bugs.　

As a device vendor, it is still necessary to prepare a large-scale test environment and test it. In that case, if it is difficult to prepare the actual machine, we may prepare an alternative measure such as using a simulator. However, when using the simulator, there are some parts that are slightly different from the production environment, so it can not be said that it is a complete test, so it may be hard to say that it was a perfect test. There are places like that, so it’s quite difficult to test it.

Type of test, the next test RAS function is

When judging whether the test quality is good or bad, we look at the types of tests performed. Next to the stability test and reliability test is the RAS functional test. It is introduced in the next article, so please have a look if you are interested.

Next : Release judgment criteria・The third type of test is the RAS function