5 Truths About AI-Driven Software Testing

Here’s the hard truth about traditional software test automation: it requires significant manual effort. It seems counterintuitive, but it’s true. There is a considerable amount of human intervention involved in what are considered to be “automated” software test processes.

In the report from O’Reilly Media, AI-Driven Testing: Bridging the Software Automation Gap, author and chief scientist, Tariq King, describes the current “gap” between manual and automated testing. King argues the best way to narrow that gap is to make use of artificial intelligence (AI) in your approach to test generation and execution.

Below, I’ve summarized some of my top five lessons learned to close the gap between manual and automated testing. For access to the full report, you can download it here.

1. Testing is more than checking.

A simple statement, but I think this is a really important point to make. When we think of test automation, what we often think of is the test execution. However, there’s more to it than that. Testing, when done right, consists of learning, experimenting, troubleshooting, trial-and-error, observation, extrapolation – you get the picture. These are complex, highly cognitive tasks that would typically require a human tester to manually perform.

This is where AI comes in. Advances in AI and machine learning (ML) technologies have made it possible to optimize repetitive tasks, perform bug hunting, and monitor change – which brings me to the next takeaway…

2. AI can test for things we previously thought subjective.

A common roadblock on the path to automation is the inability for machines to handle some of the nuances of human perception. For user interface (UI) design in particular, most of the developer requirements are qualitative in nature. Attributes like usability, accessibility, and trustworthiness can all fall under this category.

However, AI has shown us that machines are able to model patterns, workflows, and tasks, including UI design testing. AI can test user interfaces, services, and lower-level components and can evaluate the functionality, performance, design, accessibility, and trustworthiness of applications.

3. AI-driven test automation requires very little maintenance for visual updates and redesigns.

The purpose of functional UI testing is just how it sounds – to confirm the functionality of an application's UI. For most web and mobile applications, functional UI testing can be challenging when using testing frameworks that rely on the document object model (DOM). DOM-based element selectors make tests susceptible to breaking because the structure and behavior of the UI changes with each update.

AI, and more specifically a branch of AI known as intelligent computer vision, gives us the newfound capability to perceive and test anything with a screen. Using image-based analysis, AI bots recognize what appears on an application screen independently of how it is implemented. Without the need for DOM-based analysis, UI design changes do not result in excessive test script maintenance.

4. AI-driven test automation will increase both the level of test coverage and test speed.

Traditional approaches to test automation consist of manually creating test cases for each new feature or application. This can take weeks or even months to complete, and there is a high level of risk with manual test case creation. As King states in the report, “Over time the test coverage required to validate the quality of your software product diverges from the engineering team’s ability to design and write test scripts for it.” Essentially, the software’s complexity is increasing faster than test automation can keep up.

AI-driven testing essentially narrows that gap between software complexity and test automation. This is especially beneficial in the enterprise space, where modern business applications need end-to-end testing and where time-to-market cycles are continuously shrinking.

5. AI-driven test automation is not a thing of the future; it’s already here.

We once thought tasks such as voice and image recognition, driving, and even musical composition were impossible for a machine to accomplish. But we’re already seeing this type of automation today. So it shouldn’t surprise you to know we are currently training AI bots to perceive, explore, model, and test software functionality.

With all of the activity and buzz around AI for software testing, we are at the beginning of a new era of test automation. AI testing is enabling testers, developers, and all software professionals to tackle challenges that were once thought to be insurmountable.

Screenshot of O'Reilly AI Testing Report
For a complete look at the expanding role that AI plays in software test automation, I highly recommend downloading the report from O’Reilly, AI-Driven Testing: Bridging the Software Automation Gap.