Describe It, Don't Select It: AI Element Location in Keysight Eggplant
Why test maintenance keeps eating your team’s time, and what changes when you stop telling the machine where to look and start telling it what to find.
Ask a QA director where their automation budget goes and you’ll rarely hear “running tests.” Execution is cheap and getting cheaper. The money goes somewhere less visible: keeping the tests alive. A button moves a few pixels. A release ships a new theme. A form grows a tenth field. None of it changes what the application does, and none of it changes what the test is supposed to check, yet each one can break the script, and someone must stop building coverage to go fix it.
That work is mostly invisible because it doesn’t show up as a line item. It shows up as engineers who were supposed to extend coverage spending their week repairing it instead. Across a mature suite, over enough releases, it becomes one of the larger unbudgeted drains on QA capacity. And it compounds quietly, because every release adds more tests that will eventually need the same care.
Here’s the part worth sitting with: that cost isn’t a flaw in any one tool. It’s a consequence of how we’ve been forced to describe what we want tested.
We’ve been encoding intent as coordinates
Locating an element on screen has traditionally meant one of two things. If the application exposes its structure, a web page with a clean DOM, you can target an element by its underlying identifier. But a large share of enterprise software doesn’t cooperate: packaged applications, core banking screens, electronic health record systems, legacy desktop tools, anything rendered inside a remote session or a graphics canvas. There’s no clean structure to grab onto.
For those interfaces, image-based matching has long been the answer, and a good one. You capture a reference image of the element, and the tool finds it on the live screen by visual comparison. It’s precise and deterministic: when the screen matches the reference, it matches exactly. Keysight Eggplant has done this well for a long time, and for a large class of work (where pixel-level precision is the whole point) it remains the right instrument.
But that precision carries a tax. The reference image is pinned to how the element looked the day you captured it. Change the resolution, the theme, the color, or the position, and the match fails, not because the test was wrong, but because the screen no longer matches the photo you took of it. So, you maintain image libraries. You re-capture. You re-tune. You do it again next release. The tool is doing exactly what you asked; the limit is what you were able to ask for. You could only describe the element by what it looked like at one moment, not by what it is.
A different unit of work
This is the shift behind Find by Description, the AI capability arriving in Keysight Eggplant. Instead of capturing a reference image, an engineer describes the element in plain language like “the ticket price field,” “the blue Submit button at the bottom of the form.” A computer vision model analyzes the live screen and locates the element that matches that description, using visual context (label, type, color, position) rather than pixel-level comparison.
The mechanism is worth being precise about, because it’s easy to describe wrongly. Find by Description does not detect a change and repair the script. It was never pinned to a fixed image in the first place. Every time the test runs, it locates the element fresh from your description against whatever is on screen at that moment. A cosmetic change, a restyled button, a shifted layout, a new theme, simply isn’t a break, because there was never a pixel signature to invalidate. The test is anchored to intent, not to an old screenshot.
That is a real change in the unit of work. You’re no longer telling the machine where something is and then maintaining that answer forever. You’re telling it what you’re looking for, the way you’d describe it to a colleague, and letting it find that each run.
What it does to the numbers
The effect is easiest to see on the kind of task that has always been disproportionately expensive: reading a dynamically changing value out of a visually complex screen, the live price of a train ticket between two stations on a given date, for example. A person does it in seconds. Automating it the traditional way does not.
In a survey of 40 automation testing professionals conducted in May 2026, more than half (55%) reported that this class of task takes them between one and four hours to automate with traditional methods. Nearly a third (31%) said four hours or more. Not for lack of skill, but because capturing, tuning, and maintaining the match is genuinely that involved.
In a worked example of that same train-booking task, Find by Description completed it in under 15 minutes and used 92% fewer lines of script, replacing a long sequence of pixel matches and coordinate math with a handful of plain-language instructions.
Two different measures, worth keeping separate: the hours come from what practitioners report about today’s methods; the under-15-minutes and the 92% come from one measured example of the new approach. Together they mark out the size of the gap. Your own applications will land somewhere along that range, but the direction isn’t subtle.
This isn’t a replacement. It’s a choice.
Find by Description doesn’t retire image matching or OCR, and it shouldn’t. Where you need pixel-level precision, confirming that a rendered output is correct down to the pixel; deterministic image matching is still the right tool. Find by Description sits alongside it as an optional capability you switch on where it earns its place: interfaces that change often, applications with no accessible structure, localization testing where one description holds across languages, cross-resolution work where you’d otherwise maintain a separate image set per environment.
The judgment call moves to the engineer, which is where it belongs. Precision when you need precision; resilience and speed when the maintenance tax is the thing actually costing you.
The real story
It’s tempting to file this under “AI feature, now faster.” That undersells it. The interesting part isn’t the speed, it’s that you can finally describe a test target the way a tester already thinks about it. Not as a region of pixels to be guarded against change, but as a thing on a screen with a name and a purpose. The hours your team loses to maintenance were never really about the tests running. They were about the distance between how people understand an interface and how they’ve been forced to encode it. Find by Description closes some of that distance, and that, more than any single number, is why it’s worth your attention.
See it run live in our launch webinar, AI Element Location: Finding UI Components by Natural Language Description, Not Selectors.
Register to watch Find by Description locate and act on real interface elements from a plain-English description, no reference images required.
Frequently Asked Questions about AI Element Locators
What is AI element location in Keysight Eggplant?
AI element location, branded Find by Description, lets a test engineer locate a UI element by describing it in plain English, "the blue Submit button at the bottom of the form," instead of capturing a reference image or maintaining a selector. A computer vision model analyzes the live screen and identifies the matching element by its visual context: label, type, color, and position.
Can you automate testing for applications that don't expose a DOM?
Yes. Because AI element location works only from what is visible on screen, it tests interfaces that have no accessible structure to target, packaged enterprise software, core banking and EHR systems, legacy desktop tools, and applications rendered inside a remote session or graphics canvas. No DOM access, application hooks, or instrumentation are required.
How is describing an element different from image-based matching?
Image-based matching pins a test to a reference screenshot, so a change in resolution, theme, or layout breaks the match and must be re-captured. AI element location locates the element fresh from your description each time the test runs, so a cosmetic change isn't a break. There was never a fixed pixel signature to invalidate. Both approaches coexist in Keysight Eggplant; pixel-precise image matching remains the right choice where exact rendering is what you're verifying.
Does AI element location automatically fix or heal broken tests?
No. It does not detect a change and repair a script. It re-locates the element from your plain-English description on every run, against whatever is on screen at that moment, so cosmetic UI changes never register as failures in the first place. The test is anchored to intent, not to a stored screenshot.
How much time does it save compared with traditional methods?
In a survey of 40 automation testing professionals (May 2026), 55% reported that automating a task such as reading a changing value from a complex screen takes one to four hours with traditional methods, and 31% reported four hours or more. In a separate worked example of one such task, Find by Description completed it in under 15 minutes and used 92% fewer lines of script. (The hours are reported practitioner times; the 15-minute and 92% figures come from one measured example.)
Can one test work across different resolutions, themes, and languages?
Yes. Because the test is anchored to a natural-language description rather than a fixed image, a single test holds across screen resolutions, DPI settings, OS themes, and minor redesigns. For localization, the same semantic description can locate an element regardless of the display language, avoiding a separate image set per environment.
Is the AI capability optional?
Yes. Find by Description is a capability you switch on per command where it earns its place; existing image-matching and OCR commands continue to work unchanged. The choice of method stays with the engineer — precision-critical steps can keep using image matching while change-prone interfaces use description-based location.