Find by Description

Software testing teams face a problem that rarely appears in project plans but reliably consumes a significant share of QA capacity: the ongoing cost of maintaining automation scripts. Every UI update, theme change, or minor redesign can invalidate pixel-matched image references and hard-coded coordinates, forcing engineers to spend hours — sometimes the better part of a working day — on a single task that a human tester would complete in seconds.

A May 2026 survey of 40 automation testing professionals put numbers to this reality. For a representative task — locating and reading a dynamically changing value from a visually complex UI — more than half of respondents reported taking between one and four hours. Nearly a third said four hours or more. Multiplied across test suites, release cycles, and teams, this becomes one of the largest unbudgeted drains on QA capacity: engineers maintaining scripts rather than extending coverage.

Find by Description, part of Keysight Eggplant, addresses this directly. Instead of capturing a reference screenshot and relying on pixel-level comparison, engineers write a plain-English description of the element they want to interact with — "the ticket price field," "the blue Submit button at the bottom of the form." A computer vision model analyzes the live screen and locates the matching element using semantic understanding, not pixel matching. It identifies elements by visual context, label, type, color, and position — and it adapts when the interface changes, because there are no pixel signatures or coordinates to invalidate.

In a real-world example built around a train booking workflow, Find by Description reduced script volume by 92% — from a multi-step sequence of pixel matches and coordinates to a small number of natural language instructions. The same task that occupied the majority of surveyed engineers for one to four hours was completed in under 15 minutes.

The capability works alongside Eggplant's existing OCR and image matching tools rather than replacing them. OCR reads and verifies on-screen text; image matching locates elements by pixel comparison; Find by Description extends these with contextual visual understanding. It can distinguish a "Cancel" button in a modal dialog from one in the main navigation — the kind of distinction that matters in practice but defeats pattern matching.

Because tests are anchored to natural language descriptions rather than fixed pixel signatures, they remain valid across UI iterations, theming changes, screen resolutions, and localization. A cosmetic change never becomes a script failure, because nothing was ever pinned to a pixel. The maintenance overhead that typically accompanies each release is dramatically reduced without any change to how tests are deployed or executed.

Find by Description requires no application-side instrumentation. It works purely from what is visible on screen — the same view a real user sees — making it applicable to the full range of applications automation teams test: web applications that undergo frequent UI updates, legacy desktop and embedded applications where DOM selectors are unavailable, multi-language deployments where element descriptions hold regardless of display language, and cross-device environments where maintaining per-resolution image sets is impractical.

The practical returns are straightforward. Hours previously absorbed by script maintenance are returned to engineers each sprint. Tests survive the routine UI changes that previously triggered maintenance cycles. Test logic written in natural language is self-documenting and readable by anyone reviewing it, making handover and collaboration easier. And a single description-based test works reliably across environments that previously required maintaining separate image variants.

For teams whose automation programs have grown large and mature, the compounding effect is significant: less time spent on break-fix after each release, more capacity directed toward coverage that hasn't yet been automated. For teams still scaling their automation, it means starting with scripts that won't require constant upkeep as the application evolves.

Find by Description is available now as part of Keysight Eggplant, deployable in cloud or on-premise environments using an organization's existing AI provider credentials.