The Missing Link in AI Validation
Automotive AI Series Blog 3
Why Validation Results Must Become Connected Evidence
A Good AI Model Is Not the Same as a Safe AI System
Blog 1 introduced the idea that trustworthy AI requires more than performance metrics alone. An AI model can produce a plausible result and still contribute to an unsafe system decision. Blog 2 extended this challenge beyond development, showing that confidence in AI systems can erode as systems, environments, and assumptions evolve over time.
Together, these observations raise an important question: What exactly are we validating when we validate AI?
Many validation activities focus on individual AI models. Accuracy, robustness, bias assessments, and explainability analyses can provide valuable insight into model behavior. However, a model is only one part of a larger AI-enabled system. A sensor perception output becomes meaningful only when it is used by the vehicle system. Object detections may feed sensor fusion, motion prediction, vehicle control, and safety functions. What appears to be a minor perception issue at the function level, such as the e-scooter example introduced in Blog 1, can therefore influence how the vehicle interprets and responds to a situation.
As we saw in the e-scooter example from Blog 1, the system correctly detected the road user but incorrectly interpreted the user's movement. The issue was not a missed detection. Rather, an incorrect interpretation of the situation contributed to an inappropriate vehicle response. Once that interpretation enters the vehicle system, it can influence motion prediction, vehicle behavior, and ultimately how the vehicle responds to the situation.
This is why validating an individual AI model is not the same as validating the complete AI-enabled system. A model can achieve strong performance metrics while important risks remain hidden in system interactions or in the conditions under which the system is ultimately deployed.
Using the e-scooter scenario introduced in Blog 1, the following three validation perspectives can be viewed as answering different questions about the same AI-enabled behavior.
Function, system, and domain are not competing perspectives. They answer different questions about the same AI-enabled behavior. A more detailed discussion of these validation layers is available in our white paper Rethinking AI Validation.
Validation Evidence Exists Across Multiple Perspectives
Even when AI validation considers function, system, and domain perspectives, the resulting evidence is often generated in separate activities throughout the AI lifecycle. Dataset-based validation focuses on coverage, quality, and bias. Model-based validation evaluates performance, robustness, and explainability. Operational evidence emerges later through updates, monitoring, incidents, and real-world system behavior.
Figure 2. Validation evidence is often generated separately for datasets, models, and real-world operations
As a result, evidence is frequently generated in isolation. Teams may know that a dataset contains coverage gaps, that a model performs well on a reference benchmark, or that an operational issue occurred after deployment. What is often missing is a clear connection between these findings and their impact on the overall AI-enabled system.
Development Evidence Is Only Part of the Story
While these validation perspectives are all important, they are not always treated equally throughout the AI lifecycle. Dataset-based and model-based validation activities are well established during development. Engineering teams routinely assess data quality, coverage, performance metrics, robustness, and explainability before deployment.
The situation is often different once systems enter operation. As discussed in Blog 2, confidence in an AI-enabled system cannot depend solely on evidence generated during development. Once vehicles enter operation, new evidence continuously emerges through real-world inference, software updates, operational monitoring, incidents, and changing environments. Inference refers to the process by which a trained AI model applies what it has learned to new inputs during operation. Yet these operational perspectives frequently receive less attention or remain outside the scope of traditional validation activities.
As a result, validation can become heavily development centric. Teams may know that a dataset contains coverage gaps or that a model performs well on a reference benchmark. What is often less clear is how the deployed system behaves as operating conditions change, new scenarios emerge, software evolves, and assumptions age over time.
Operational monitoring is therefore essential for comparing the behavior established during development with the behavior observed in real-world operation. Deployment-oriented inference testing and operational monitoring help reveal how systems behave under conditions that were not fully represented during development, or whose real-world significance was not fully understood at the time of validation.
Without this operational perspective, important risks may remain hidden until deployment. These risks do not necessarily originate from an obviously defective model. They may arise from:
- interactions between the AI function and the surrounding vehicle system
- insufficient adaptation to the intended ODD
- differences between training conditions and real-world inference
- changing data distributions or previously unseen edge cases
- assumptions that no longer hold after software or system changes
A model can pass dataset and performance checks while system-level and domain-level risks remain undiscovered until deployment. Development, deployment, and operation all generate valuable evidence. Each activity provides a different perspective on the AI-enabled system, but no single activity provides the complete picture. Taken together, these perspectives provide a broader understanding of the AI system, as illustrated in Figure 3.
Figure 3. Understanding AI behavior requires connecting dataset-based, model-based, and inference-based evidence across the lifecycle
Dataset-based evidence helps explain what information the model learned from. Model-based evidence helps explain how the model behaves. Inference-based evidence helps explain how the deployed system performs under real-world conditions.
Each perspective answers different questions. Together, they help engineering teams understand whether confidence in the system remains justified as the vehicle, software, operating environment, and assumptions evolve over time.
This is why fragmentation is not only a tool problem. It is a lifecycle problem. Evidence generated during development, system integration, deployment, and operation must remain connected if organizations want to understand how the behavior of AI-enabled systems changes over time.
The Cost of Fragmented Validation
The blind spots often appear at the boundaries: when a dataset finding is not linked to a model limitation, when a model result is not traced to a downstream system effect, or when a system test is not connected to the domain assumptions that make the result meaningful.
Introducing another specialized validation tool can generate additional information. But it also creates another source of evidence that must be interpreted, reviewed, documented, and connected to everything else.
As organizations expand their AI validation activities, evidence becomes distributed across dataset tools, model-analysis platforms, explainability methods, simulation environments, system-testing frameworks, and separate monitoring solutions.
The burden of connecting these results often falls on engineering teams: Reports are compared manually. Results are transferred between tools. Teams exchange spreadsheets and presentations. Assumptions are documented separately from the evidence they support. Relationships between findings must be reconstructed across different formats, owners, and development stages.
This process requires time, engineering resources, and repeated coordination. It also creates significant uncertainty. Even after teams invest the effort to bring the available evidence together, important gaps may remain:
- A dataset issue may never be connected to a model limitation.
- A model limitation may not be traced to its downstream system impact.
- A successful system test may not reveal insufficient coverage of the intended ODD.
- A real-world inference deviation may remain disconnected from the development assumption that caused it.
- An operational issue may be identified without a clear link to the model, dataset, software version, or system configuration involved.
Many validation activities generate valuable evidence. Yet that evidence is often fragmented across datasets, models, vehicle systems, operational monitoring, and lifecycle stages. As a result, engineering teams are left to reconstruct the relationships manually.
Fragmented validation creates fragmented evidence, high integration effort, and persistent blind spots.
A more holistic approach must extend beyond dataset checks and model performance evaluation. It must connect evidence across the AI function, the surrounding vehicle system, the intended application domain, and real-world operation.
Only then can engineering teams understand not just how an individual model performed under controlled conditions, but how the complete AI-enabled system behaves in its intended context and how that behavior changes throughout deployment.
Generating evidence is only one part of AI validation. Equally important is understanding how the available evidence relates to the behavior of the overall AI-enabled system.
From Fragmented Checks to a Connected Approach
This blog showed why evidence generated through isolated validation activities can still leave important gaps between the AI function, the vehicle system, the application domain, and the stages of development and operation.
Dataset-based, model-based, and inference-based evidence each provide valuable insights. Yet understanding how these different perspectives relate to one another throughout the lifecycle remains difficult. As AI systems evolve, confidence depends not only on the existence of evidence, but also on understanding how that evidence relates to the behavior of the overall system.
This leads toward a broader discipline: AI Assurance. The next blog explores how AI Assurance helps organizations maintain context, trace relationships, and support confidence in AI-enabled systems throughout the lifecycle.