Release judgment criteria・The fourth type of test is the version upgrade function

13/11/202020/10/2021Release Judgment

The fourth type of test is the test status of the version upgrade function.

This series of articles introduces the types of tests from the perspective of what kind of test you are doing when determining the quality of the test. It is also a part of the Serviceability function of the RAS functions introduced in the previous article , but since the version upgrade function is especially important for the software installed in the product, we carefully check the contents of the test. I did.

In checking the test status of the version upgrade function, the weather is slightly different from other tests so far , and the function is stable, asking whether the test is prepared for a bad operating environment or repeated a sufficient number of times . We give specific items to confirm the sex and confirm the test contents in detail .

The reason why we pay special attention only to the version upgrade function and go into the test contents to check it is because the version upgrade function is the last bastion when a software defect occurs in the market .

What is the last fort and why do you give it special attention?

What should the manufacturer’s software development team do when a bug is found in a software embedded in a product that is in production on the market and a major problem occurs?

The first thing to do is to provide an urgent workaround . Problems that are occurring in the market, more so that it does not grow, Toka how to use Toka setting method, to provide suppressing workaround the occurrence of problems in the means that can be right now, to be due to a problem that to minimize the harm Is the highest priority.

If you can stop the spread of damage with an urgent workaround, the next thing to do in a hurry is to investigate the bug that caused the bug and develop a revised version of the software that fixes the bug . Then, the completed revised software will be applied to tens of thousands and hundreds of thousands of products operating in the market to alleviate market problems .　

The revised version of the software can be applied to products running on the market because it has a version upgrade function, and thanks to this, it is not necessary to recall all products from the market due to software defects. You can converge the problem. If the software cannot be upgraded, all products will need to be collected, so the final bastion in the fight against market defects to prevent this is the version upgrade function.

If you think about it, it’s a simple story, but when a bug causes a problem in the market, if the version upgrade function does not work properly, we, the device manufacturers themselves, will be in great trouble. That’s why I pay special attention when making a release decision.

Check point of version upgrade function # 1: Test in bad operating environment

So what should we check for testing the upgrade function? There are various points of view, but Father Gutara paid attention to two points and confirmed the contents of the test. The first is whether the version upgrade function is tested in a bad operating environment .

Bad operating environment is a bit confusing, so let’s restate it a little more concretely. There are various ways to send software to the device for version upgrade, but let’s consider the case of sending it via a network, for example. In this case, the version upgrade function consists of a server that sends out the software, a communication line, and a terminal that receives and rewrites the software.

In a good operating environment, there will be no delay in the communication line, no noise, the server will be responsive, and the terminal will be operating stably . Of course, it is necessary to confirm by testing that the version upgrade function works in such a good operating environment. But that alone is not enough.

The production environment has unexpected adverse conditions

In the actual version upgrade work, there are some areas where the communication line is full of noise and the delay is large . If the version upgrades of many devices overlap, the responsiveness of the server may slow down and a timeout may occur. The operation of the terminal or the power adapter may be unstable and the power may be turned off while writing the software to the flash memory .

When such a thing happens, do you give up even if the version upgrade function does not work properly? If you have a big problem in the market and want to replace it with a countermeasure version of the software, you can’t say such a lenient thing. Even if there are some bad conditions, I want the version upgrade function to have the tenacity to try again without giving up easily.　

It is best to confirm by a test to guarantee that it will work reliably even in an operating environment where the version upgrade function is poor . So, Gutara’s father always checked at the time of release judgment whether he was testing even in an operating environment where the version upgrade function was bad.

Confirmation point of version upgrade function # 2: Many tests

The second check item is the number of times the version upgrade test is executed . Things are relatively simple, and Gutara’s father confirms that he has conducted tests thousands of times or more for terminal equipment installed in the house and hundreds of times for collective equipment installed in the station building. I did.

Why do we need to run the same test hundreds or thousands of times? Do you feel that doing it a few times is enough? Certainly, if small software always executes operations in the same order and timing, it does not need to be executed hundreds of times.

However, recent software is generally made with multitasking and multithreading . And some tasks and threads have fluctuations in execution timing due to various uses .

The source of fluctuations in execution timing is, for example, tasks and threads that communicate with the outside . The timing of execution of these will fluctuate depending on the timing of communication from the outside. Another source of fluctuation is tasks and threads that have access to flash memory .　

The access speed of flash memory is much slower than that of normal RAM , and even if the flash memory is the same, the speed will change depending on the deterioration state of the memory element of the flash memory . As a result, tasks and threads that have access to flash memory also experience fluctuations in execution timing.

These tasks and threads work together with other tasks and threads to realize the functions of the entire system. Therefore, software created with multitasking or multithreading has minute fluctuations in the order and timing of execution each time, even if the effective source code is the same .

Timing fluctuations are a hotbed of bugs

And bugs like this kind of timing fluctuation . It works fine at normal timing, but at the timing of processing that occurs very rarely, some processing changes a little and as a result unexpected processing is executed, and a bug occurs after that. I often say that I am laughing .

Fluctuations in processing timing in a multitasking or multithreading environment are almost impossible to identify through desk studies such as design reviews, code reviews, and simulations . The only way to do this is to run the same process on the actual machine and check the actual machine test to see if anything strange happens due to the fluctuations that occur there.

So how many times should I test? Unfortunately, no one knows the correct answer . Since it can’t be helped, Gutara’s father decided a few thousand times for the terminal in the house and hundreds of times for the collective device of the station building, and requested the development department to carry out the test for that number of times only for the version upgrade function. was doing.

Is it really effective to do so many tests ?　When asked, I can only answer that I can’t say anything, because it is not the number of times that there is a basis in the first place.　

However, in my father’s experience, there are more bugs found in advance with this hundreds or thousands of upgrades than you can count with both hands. It is difficult to judge from the viewpoint of whether the test is good or bad in terms of cost effectiveness. However, if you think of the version upgrade function as the last bastion of software bug fixes, you can judge that it was effective.

The type of test, next is the confirmation status of past defects

When judging whether the test quality is good or bad, we look at the types of tests performed, but next to the test of the version upgrade function is the status of checking past defects in the test. Please see the following article if you are interested.

Next : Release judgment criteria・The fifth type of test is confirmation of past defects