Using AI in Software Testing
I recently caught up with Paul Gerrard, the internationally renowned, award-winning software engineering consultant, author, and coach. We got to talking about where and how Gen AI can add value to the world of testing. We drew out three areas where we already see it helping – and one intriguing possibility for the future.
Gen AI Can Do the Heavy Lifting of Test Design
The real value of testers is our ability to think critically and design the tests that get to the heart of the software and what it needs to do. Of course, this test design process is where the heavy lifting really happens. It’s also one of the areas where Gen AI can add the most value.
Let’s take this example to understand how.
Imagine you need to test braking specifications for a car. The specifications are just a tiny segment of the tests that need to be run for the vehicle and its overall function. At the same time, the specifications are also safety-critical, so it isn’t an area you can skim and hope for the best.
Gen AI has the capability and the capacity to do all your thinking in a fraction of the time. It can develop a model with all the required axioms and delve into a level and combination of incredibly powerful detail. It can look at every scenario for heat, ice, rain, and more, individually and in combination. It can factor in road material and assess how tolerances will change.
This ability to interrogate every nuance has the potential to save incredible amounts of time in test design. Its comprehensiveness brings another benefit too, which I’ll look at next.
Gen AI’s Lack of Wisdom Can Be a Benefit
You can think of Gen AI as knowing everything and knowing nothing. That’s because it lacks the wisdom that we bring to a problem as human testers.
To take the braking specification scenario again, Gen AI will give you every single possible permutation. Of those, some will be physically impossible or simply not reflective of the real world. Your wisdom brings the necessary X factor to the process – you’ll be able to see which tests don’t make sense or won’t be needed.
At the same time, Gen AI’s comprehensiveness can add value. As humans, we all have cognitive biases, which can lead to problems in software testing.
There’s a story, probably an urban myth, about the F-16 Fighting Falcon fighter aircraft. The story goes that it flew perfectly well in the northern hemisphere – but it flipped 180 degrees the moment it crossed the equator. Why? Because cognitive biases meant the northern hemisphere developers and testers hadn’t considered the southern hemisphere in their thinking.
It’s an extreme example, but it’s the sort of thing that happens all the time. Software designed for the 5G and full fibre connectivity that every city enjoys but which hasn’t yet reached rural communities. Technology that works in flat Illinois but can’t cope with mountainous Colorado.
In each of these cases – and countless others – the developers' and testers' cognitive biases created issues. Gen AI helps eliminate those cognitive biases because it lacks the wisdom to have them in the first place.
AI Can Help Eliminate Misunderstanding
Software development almost invariably involves multiple stakeholders. Each stakeholder brings a slightly different understanding of the subject. These multiple understandings bring value, but they also raise the possibility of confusion that could cause problems.
Let me share a very simple example we came across recently when we were working with Keysight’s Eggplant Generative AI (GAI).
To give you a quick bit of background, GAI is a highly tuned large language model trained on ISO/IEEE/BSI/ISTQB® testing material from trusted sources. It’s also trained on industry verticals such as healthcare, telecommunications, or aerospace and defense to offer industry-specific insight. You can fine-tune this base model to your precise requirements with complete confidence because GAI is offline and therefore 100% secure.
GAI had been fed a new source of data that contained a definition that contradicted a definition in another document (In the new document, PO was Purchase Order; in the original document, PO was Product Owner.) GAI did as we would expect – it queried Purchase Order, said the definition contradicted its understanding of the term and knew it felt wrong.
As testers, it’s our job to inspect documents and ask questions about terms and definitions to eliminate the ambiguities and conflicts that arise when different stakeholders from different disciplines collaborate. With Gen AI on our side, we enhance our ability to do this. It can help us iron out more difficulties more quickly.
Where Can AI Go Next?
In my recent blog Why Test? [link], I highlighted the value of the tester in highlighting competing and conflicting requirements. I pointed out that the earlier we shift left, the earlier we can help to pick up problems and stop them from happening. I absolutely believe this. I also absolutely believe that the reality of human communication makes this easier said than done.
Imagine your organization is developing a new piece of software. The business development team meets with your lead developer to discuss progress and fine-tune requirements. The next day, the lead developer is away, so your mid-weight developer attends the weekly finance team meeting where the software is discussed as an agenda item and a new requirement is discussed.
In today’s world, there’s every chance the development process would continue for quite a while before it filtered through that the finance team’s new requirement conflicted with one of the business development team’s requirements.
In the future, could we use Gen AI to parse the transcripts of all meetings and take the necessary action? We could ask it to build the model of requirements for us – and also to highlight when requirements change or conflict.
We’re not there yet, but it’s an exciting vision of how Gen AI could develop in the future, isn’t it?
Explore This Topic Further
If you’d like to explore these ideas in more depth, watch the recording of my LinkedIn livestream with Paul: A New Model for Testing 2.0. How Gen AI can add value was just one of the topics we discussed.