Attacking AI with Fault Injection, Advancing FI with AI Assistance

Artificial intelligence is changing the Fault Injection (FI) landscape in two distinct ways. As AI increasingly moves from the cloud to edge devices, the hardware responsible for running AI models is becoming a new target for physical attacks. At the same time, AI itself, particularly Large Language Models (LLMs), is beginning to offer new ways to support and potentially improve Fault Injection testing. This webinar explores both developments through ongoing research into attacking edge AI inference and using AI to assist FI workflows.

The first part of the session examines what happens when Fault Injection is applied to AI workloads running on dedicated Neural Processing Units (NPUs). Edge AI systems are increasingly used in applications where devices need to make decisions locally and in real time, including automotive systems, drones, robotics, cameras, and other embedded platforms. These systems often rely on object detection models to identify and classify their surroundings.

The research presented in the webinar investigates the impact of Fault Injection on YOLOv5 object detection running on a commercial NPU platform. Rather than focusing only on whether an attack causes a system to crash or produces an obvious computational error, the researchers look at how faults can influence the output of the AI model itself.

The results demonstrate several types of effects that are particularly relevant to AI-based perception. Faults can cause confidence values associated with detected objects to change, potentially reducing the certainty of an otherwise correct detection. In other cases, objects may no longer be detected at all. Faults can also introduce detections that were not present in the original input, creating so-called phantom objects.

These effects are important because the output of an object detector is often only one component within a larger perception pipeline. A temporary error at the inference stage may be filtered out by downstream processing, but it may also influence subsequent system behavior. For example, tracking algorithms can associate detections across multiple frames. Under certain conditions, an erroneous detection could therefore become part of the system's ongoing representation of its environment rather than remaining an isolated error.

The webinar introduces a useful way to consider this propagation through the concepts of filter, latch, and amplify. Some errors may be filtered out before they have a meaningful effect. Others may be latched into a persistent state by subsequent processing. In more significant cases, downstream components may amplify the effect of the original fault. This highlights why evaluating the security of AI-enabled products requires looking beyond the accuracy of an individual model and considering how faults can propagate through the complete system.

The session also discusses the practical FI test setup and the initial results obtained from the NPU platform. These experiments demonstrate how established hardware security testing techniques can be applied to a new class of targets as AI accelerators become increasingly common in embedded devices. The research raises important questions for product security teams about how the physical resilience of AI hardware should be evaluated alongside traditional considerations such as model accuracy, software security, and adversarial machine learning.

The second part of the webinar turns the relationship between AI and Fault Injection around. Instead of attacking AI with FI, the researchers explore how AI can assist the FI testing process itself.

Fault Injection campaigns can involve large numbers of parameters, experiments, observations, and iterative decisions. Testers need to determine where and how to inject faults, interpret results, identify interesting behavior, and decide which experiments to perform next. The session investigates how LLMs and GPT-based tools could assist with parts of this workflow.

Through a demonstration of a GPT-assisted FI workflow, the webinar explores how an LLM can support activities such as campaign planning, interacting with test information, interpreting results, and helping guide subsequent testing. The objective is not to replace the security expert, but to investigate where AI assistance could make complex testing workflows more efficient or help researchers interact with test systems and results in new ways.

The initial findings illustrate both the opportunities and the questions that remain. AI assistance may help streamline certain repetitive or analytical tasks, but effective Fault Injection testing still depends heavily on expert knowledge, appropriate test design, and careful validation of results.

Together, the two research directions show how AI is beginning to intersect with physical security testing from both sides. AI-enabled hardware introduces new targets and new types of security-relevant failures, while AI tools may also become increasingly useful in how those systems are tested. The work presented in the webinar is preliminary and ongoing, with further research needed across additional hardware platforms, AI models, and deployment scenarios.