Beyond Performance - Why Automotive AI Needs More Than Accuracy
Automotive AI Series Blog 1
How do we establish confidence in systems whose behavior is learned from data rather than explicitly programmed?
More Than a Detection Problem
Let’s take a look at the exemplary scenario below. At first glance, this looks like a like a typical urban traffic scene. A vehicle approaches an intersection. Pedestrians are crossing the street. A person on an e-scooter is moving through the city.
Nothing seems unusual. In fact, scenarios like this occur thousands of times every day in cities around the world. E-scooters and other new forms of urban mobility have become part of everyday traffic. Yet they also create situations that automotive perception systems need to interpret correctly. A person standing on an e-scooter may look similar to a person standing or walking, while moving at a very different speed.
In this illustrative scenario, the system correctly detects a person. However, it confidently interprets the road user as moving at walking speed, while the actual situation involves much faster movement. Without an effective plausibility check, this incorrect interpretation can propagate into subsequent system decisions.
The problem is therefore not simply that the system failed to detect something.
A seemingly plausible perception result led to an incorrect interpretation of the situation. The system recognized the road user, but misunderstood how that road user was moving. And that distinction matters. In AI-enabled advanced driver assistance systems (ADAS), perception results influence downstream decisions. An incorrect interpretation of the situation can affect motion prediction, vehicle behavior, and ultimately how the vehicle responds to the world around it.
A model can produce a seemingly valid output and still contribute to an unsafe system decision.
This example illustrates a fundamental challenge for automotive machine learning (ML): As ML models become increasingly embedded in perception systems, sensor fusion, driver monitoring, and automated driving functions, engineering teams must answer a question that goes far beyond accuracy:
How do we justify trust in systems whose behavior is learned from data rather than explicitly programmed?
Why Automotive AI Is Different
Conventional automotive software is typically based on explicitly defined logic. Engineers specify requirements, implement rules, and verify whether the software behaves according to those rules. The intended logic is represented in requirements, architecture, code, and tests.
Machine learning (ML) systems are different. Rather than manually defining every decision rule, engineers train models using data. The resulting behavior emerges from statistical patterns learned during training. This enables ML models to recognize complex relationships that would be difficult to express through conventional rules, making them highly valuable for perception, prediction, and other automotive AI applications.
But this capability introduces a different type of engineering challenge. Model behavior can depend on:
- the composition and quality of the training data
- the selected model architecture
- the objective used during training
- the conditions under which the model is evaluated
- the environment in which the model ultimately operates
A model can perform well on one dataset and behave differently when the inputs, context, or operating conditions change. This is what makes ML powerful. It is also what makes ML difficult to validate.
The question is no longer only whether the system works. Engineering teams also need to understand why the system behaves as it does, where its limitations are, and under which conditions the available conclusions remain valid.
Accuracy Is Only Part of the Story
Accuracy, precision, recall, and other performance metrics remain essential. They help engineering teams evaluate whether a model meets defined performance objectives.
But the e-scooter example shows why performance metrics cannot answer every safety-relevant and corner-case question.
An object may be detected correctly while its motion or context is still misunderstood. A high-performing model can behave unexpectedly when the conditions encountered in the real world differ from those represented in its training data. The model can only make decisions based on the patterns and relationships it learned during training, even when those patterns no longer fully reflect the situation at hand.
Trustworthy AI therefore requires more than good performance. It requires confidence that the system behaves appropriately within its intended context.
Trustworthy AI is described through different, often overlapping principles across standards, regulatory initiatives, and organizational frameworks. In the context of automotive AI, several system qualities are particularly relevant to building and justifying that confidence:
Other principles, including fairness and privacy, remain important dimensions of trustworthy AI. The focus here is on the qualities most directly connected to validating and justifying confidence in automotive AI behavior. These expectations are increasingly visible in customer requirements, internal governance processes, and emerging automotive standards such as ISO/PAS 8800:2024 [1].
Defining the principles of trustworthy AI is only the starting point. The real engineering challenge is determining whether an AI-enabled system actually exhibits those qualities in practice. To do that, organizations need a deeper understanding of how a system behaves, where its limitations lie, and under which conditions confidence in its behavior is justified.
Why Explainability Matters, But Is Not Enough
One important part of this discussion is Explainable AI, commonly referred to as XAI. Explainability describes methods that help engineers understand how or why an ML model produced a particular result. Depending on the method and engineering question, explainability can reveal which inputs influenced a decision, which patterns the model appears to use, or where unexpected correlations may exist.
This can help engineering teams investigate questions such as:
- Why did the model produce this result?
- Which features influenced the decision?
- Is the model relying on the intended information?
- Where might the model’s limitations or blind spots exist?
These insights are valuable. They can expose behavior that performance metrics alone may not reveal.
But explainability is not the same as trustworthiness. An explanation does not automatically demonstrate that the underlying dataset was sufficiently representative. It does not show that the system interaction is safe, that the relevant operational design domain (ODD) has been covered, or that behavior will remain appropriate after the system changes.
Explainability contributes important information. It does not establish trust on its own.
From Validation Results to Safety Evidence
As AI systems become more complex, organizations naturally react by generating increasing amounts of information.
Test results. Simulation results. Performance metrics. Dataset analytics. Explainability reports.
Each activity provides valuable insight into a specific aspect of the system. But trustworthy AI depends on more than isolated results. Trustworthiness cannot be demonstrated through individual tests alone.
Instead, organizations increasingly need a structured body of information that helps explain how an AI system behaves, where its limitations exist, and why confidence in its performance is justified. This body of information is often referred to as safety evidence.
Safety evidence can originate from many activities throughout the AI lifecycle, including testing, simulation, dataset analysis, robustness assessments, and explainability techniques.
Viewed in isolation, each activity provides only part of the picture. Together, they help build the understanding needed to evaluate AI systems in safety-critical applications.
As automotive AI becomes more widespread, the importance of safety evidence continues to grow.
We will revisit this concept throughout the remainder of this series.
Looking Ahead
Building confidence in an AI system is already challenging. Maintaining that confidence over years of operation is even harder. Vehicles are updated. Data changes. Operating environments evolve. Assumptions that were valid during development may no longer hold later in the lifecycle.
If confidence depends on evidence today, how do organizations know when that evidence is no longer sufficient tomorrow? The next blog explores this challenge in The Aging AI Problem: Why AI Validation Becomes a Lifecycle Challenge.
References:
[1] ISO/PAS 8800:2024 Road vehicles - Safety and artificial intelligence https://www.iso.org/standard/83303.html