Enterprise IT - Analyzing Technology Performance, Integration and Compatibility Before Deployment

Overview

Headquartered in Tokyo, Japan, NEC Personal Computers, Ltd., manufactures 24 different PCs for consumers and businesses. The products have a 64-bit operating system and two kinds of preinstalled applications running on the OS; one produced by NEC Personal Computers and the other by independent software vendors (ISVs).

NEC Personal Computers’ Software Integration Department is tasked with analyzing any technology performance, integration or compatibility issues and finding the root cause of these problems before shipping decisions are made. To do this, the company had been employing a mix of manual testing and limited automation, however, the approach was extremely resource-intensive and also failed to account for how customers use the technology. As a result, NEC Personal Computers began a search for a modern test automation solution that also tests how users interact with the technology.

A Shift to Automated User Experience Testing

The Software Integration Department’s mission is to deliver products to NEC Personal Computers’ customers without any OS problems. Led by Ichiro Mori, PM Group Manager, the team analyzes any defects found by the development or test departments and reproduces and fixes these issues prior to shipping. Mori states, “The problems include fatal errors that have an occurrence rate of less than .1 percent. Therefore, it’s necessary to repeat the same operation more than 1,000 times to reproduce a rare and fatal problem, resulting in an investigation of the cause taking a significant amount of employee hours.”

NEC Personal Computers was able to automate some of this testing via its proprietary testing tool, but the technology lacked the ability to handle the graphical user interface (GUI) and use keyboard inputs. Mori elaborates, “For example, if any user input like sign-in after a restart causes an error, the tool itself cannot reproduce it, resulting in the need for manual intervention and many employee hours of reproduction tests.”

Mori knew there was a better approach that could automate this critical user experience testing while also maximizing human testers’ time and began searching for a test automation partner to deliver these benefits.

More Comprehensive Testing with Fewer Resources

NEC Personal Computers evaluated multiple vendors before selecting Eggplant. The company was particularly impressed with Eggplant’s ease of use, its ability to test across a range of devices and applications and its intelligent image recognition technology.

“When capturing which button the tester pressed and replaying it, the method of using the coordinates on the screen does not allow the same script to be used between devices. However, by using intelligent image recognition it can locate the same icon to click, and by using text recognition it can search for a string to find the same button to click. Therefore, both recognition methods can not only use the same script between devices with a different resolution but can also determine whether the test result has displayed a successful or unsuccessful message on the screen.” Ichiro Mori, PM Group Manager

In addition to this enhanced testing precision, NEC Personal Computers also realized significant resource and efficiency benefits from its Eggplant investment. For example, the company’s legacy approach to functional testing of its ISV applications required approximately 525 employee hours. Deploying Eggplant enabled them to reduce this workload to just 85 hours of human testers’ time.