Service Provider Yields Improvement by Automating the Validation and Test Cycle

Case Overview

This case study shows how a large Service Provider (SP) took advantage of the years of experience Keysight has gained in automating the validation process for different Network Equipment Manufacturer (NEM) products before deploying into production networks. This required the setup and integration of network testing tools, as well as interacting with complex systems and devices under test (DUTs) manufactured by various NEMs.

Advanced networking architectures involve testing and tuning a large set of parameters across the environment, collecting the results and analyzing the data to create a feedback loop that allows network specialists to pinpoint where issues might arise and resolve bottlenecks before they affect production networks. This is what makes deployment of new devices and software updates utterly complex and takes a long time to test and re-test until all test cases are validated. A lot of these test scenarios are still performed manually and don’t scale. Here a complete automated test solution will help to improve test cycle time and reduce costs.

The Challenge: Manual testing results in prolonged test cycles, requiring external resources, thus baring additional risks

The Service Provider offers a complete portfolio of network services to its customers ranging from fixed and mobile to unified communications, Internet of Things, cloud security, 5G for business and cyber security, etc.

Many of their services have Service Level Agreements (SLAs), and are constantly monitored by their customers. Each glitch in the network infrastructure, especially unforeseen downtime, can mean loss of revenue and loss of business.

The SP is using hardware from different NEMs in order to build their network infrastructure. To perform all the required validation testing, they have a separate staging lab to undergo all the necessary use case scenarios, topologies and configurations for the new equipment. In addition, pre-existing devices need to get regular software updates, which sometimes is time critical due to bug fixes and security vulnerabilities.

Any software update or upgrade in production involves a tedious process of qualifying the software release in the staging lab first. They need to manually run hundreds of different test cases to rule out errors and loss of service that may happen in the production environment.

The biggest factor the SP needs to consider is the required level of manual testing when sizing up the effort (time and money) required for NEM software qualification tests. These days most NEMs have at least two release cycles every year, which has a significant impact on the SP’s resource planning to ensure that testing goes smoothly. In addition, the SP faces the challenge to have resources available on stand-by for unplanned critical security and bug fix updates.

Solution: Keysight’s Custom Automation Test Solution

Keysight Services was able to provide this large Service Provider with a customized turnkey solution that was developed from the ground up for the SP’s requirements and needs. After initial discussions, the customer was confident that Keysight Services had the ability and knowledge to take on such a complex project. All customer requirements were carefully considered when proposing process improvements and technical solutions to maximize the use of Keysight’s knowledge on test automation and test tool integration.

The constant interaction between the technical and business teams created a healthy collaboration between the Service Provider and Keysight Services. The project included all customer specifications, such as the usage of open source tools to create a truly agile CI/CD process by leveraging the Jenkins automation server, GitLab, Robot Framework, Python, NETCONF, YANG models, as well as Keysight’s own tester tools. The complete testing framework was based on REST APIs, which were made for easy integration of test tools, but also for allowing future enhancements.