When Edge AI Lies: Fault Injection and False State in Live Perception Pipelines

Edge AI systems increasingly make real-time decisions in cameras, robotics, vehicles, and industrial equipment. These systems often rely on neural processing units, graphics processors, or other hardware accelerators to perform inference efficiently. However, continued operation does not necessarily mean that the system’s perception remains trustworthy. The white paper examines a particularly dangerous failure mode: an edge AI system that remains active and appears healthy while silently accepting an incorrect view of its environment.

The research focuses on fault injection against a commercial edge AI perception pipeline. The evaluated system uses a Rockchip RK3568 platform running a YOLOv5s_ReLU object detection model on its neural processing unit. The complete pipeline includes sensor input, CPU preprocessing, NPU-based object detection, CPU-based object tracking, and downstream decision or actuation logic. Because the system operates locally without assumed cloud synchronization, the integrity of each on-device inference directly affects the state used by the rest of the application.

Unlike attacks that modify firmware, software configurations, model weights, or input images, the study introduces short, precisely timed voltage glitches into the NPU power-delivery rail. Electromagnetic measurements are used to map the model’s execution timeline and identify sensitive stages of inference. The most effective attack window appears near the end of the model’s execution, during the neck and head stages where visual features are combined and converted into bounding boxes, object classes, and confidence scores.

The attack scenario assumes physical access and controlled laboratory conditions. An attacker might introduce a small hardware implant during supply-chain handling, repair, field service, or another period of access. The paper does not present the technique as an inexpensive remote exploit or claim that a fully self-contained field implant has already been demonstrated. Instead, it evaluates the integrity consequences once a precisely timed fault can be delivered to the target hardware.

More than 28,000 fault-injection attempts were performed. Many faults caused no observable change, while others affected confidence scores without producing an obvious visual failure. More significant outcomes included missed detections, displaced bounding boxes, phantom objects, and rare high-confidence misclassifications. In several cases, the platform continued to return valid-looking detector output even though the semantic content was wrong.

Representative examples included a bus whose detection confidence fell substantially, a bus that disappeared from the detector output entirely, and unrelated objects that appeared in the scene despite not being present. Under stronger fault conditions, the detector produced dense fields of hallucinated objects. In another example, the processed pipeline reported a truck with high confidence on an entirely black input frame. The paper notes that this result demonstrates a pipeline-level integrity failure, but does not prove which internal mechanism produced it. Possible contributors could include corrupted computation, stale tensor buffers, output-transfer errors, post-processing corruption, or memory reuse.

A focused set of approximately 4,700 experiments was then conducted around a productive fault setting. Within this selected region, confidence drift was the dominant result, followed by object hallucinations. The hallucinated outputs were not distributed evenly across object classes, suggesting that some semantic outputs may be easier to induce than others under a particular scene and fault condition. These focused results are not treated as campaign-wide success rates because they were collected after identifying a particularly effective attack region.

The paper’s central contribution is not limited to individual detector errors. It examines what happens when corrupted detections are passed into a stateful object tracker. The authors describe three possible downstream behaviors: filter, latch, and amplify.

Filtering occurs when an isolated or inconsistent false detection disappears before it can become stable tracker state. Latching occurs when repeated false detections appear consistently enough across frames to be accepted as tracked objects. Amplification occurs when bounded detector corruption leads to denser or longer-lasting false state in the tracker output.

Using a DeepSORT-style tracker configuration, the experiments show examples of all three behaviors. Short bursts of false detections could be rejected before meeting the tracker’s confirmation requirement. However, repeated errors across several frames could be promoted into apparent phantom tracks. Once accepted, these tracks could remain visible after the original fault stopped because the tracker preserves objects temporarily before removing them. Under heavier corruption, the displayed tracker output expanded into dense clutter that persisted beyond the detector errors themselves.

The study does not directly evaluate downstream actuation, and it does not claim that the observed tracker behavior automatically causes a specific physical outcome. Nevertheless, once a false object or missed detection becomes system state, it can be consumed by alerting logic, scene interpretation, planning software, operator displays, or automated controls. The risk therefore changes from a single incorrect inference to persistent false state that may influence later decisions while the hardware and software appear to be operating normally.

For product teams, the paper argues that resilience must mean more than uptime, crash recovery, or model accuracy under normal conditions. Evaluation should consider whether faults are rejected, accepted into state, or amplified over time. Safeguards may need to operate across several layers: detecting implausible track persistence or sudden track growth at the system level, identifying unusual confidence or class behavior at the model level, and treating the NPU and its power-delivery path as part of the hardware attack surface.

The findings are intentionally limited to one platform, model, tracker, and test environment. The reported rates should therefore not be generalized directly to every edge AI product. However, the study establishes an important failure path: fault injection can corrupt live inference without crashing the platform, and those errors can survive long enough to become persistent perception state. In edge AI, the most dangerous failure may not be a system that stops working, but one that continues operating while quietly accepting the wrong version of reality.