The Sugar-Coated Poison Prompt Injection Attack
Language models are becoming increasingly powerful, capable of generating detailed, coherent responses across a wide range of topics. To ensure these systems are used responsibly, they are equipped with safety mechanisms that help prevent them from responding to harmful or dangerous prompts. But what happens when those safety systems can be bypassed not through complex exploits, but through ordinary, harmless-sounding language?
A recent research paper titled “Sugar-Coated Poison: Benign Generation Unlocks LLM Jailbreaking” (arXiv: 2504.05652) introduces a novel jailbreak method that hides malicious intent behind a layer of benign reasoning and subtle prompt design.
The method, known as Sugar-Coated Prompt Injection (SCP), does not rely on known jailbreaking tactics like token obfuscation, role-playing, or adversarial formatting. Instead, it uses seemingly harmless and coherent language to gradually lead the model into generating unsafe content. The danger lies not in what is asked directly, but in how the request is framed and evolved over time.
At the heart of this technique is a phenomenon called Defence Threshold Decay (DTD), the idea that as a model generates more benign content, it becomes less sensitive to harmful intent that appears later in the prompt. By exploiting this decay in model vigilance, attackers can subtly “sugar-coat” malicious instructions within a chain of benign reasoning, ultimately bypassing safety filters without raising red flags.
Understanding the Attack
The Sugar-Coated Prompt Injection technique is built on a simple but powerful idea: malicious intent can be disguised as harmless reasoning. Instead of issuing a dangerous prompt directly, the attacker guides the language model through a carefully constructed sequence that begins with neutral or informative content and gradually shifts toward the harmful objective.
This strategy takes advantage of a phenomenon described in the research as Defence Threshold Decay. Language models are generally cautious when faced with a prompt that appears harmful from the beginning. However, as they continue generating safe responses it becomes less sensitive to changes in the conversation, the internal safety mechanisms become less vigilant. The longer the model stays in a safe reasoning mode, the more likely it is to follow instructions that would otherwise be rejected if they were presented directly. This effect allows attackers to gradually shift the model toward unsafe outputs without triggering its built-in safety mechanisms.
The attack typically unfolds in two stages:
1. Benign Lead-In
The attacker begins with a prompt that appears safe, ethical and often educational. Instead of asking how to perform a dangerous task, they might request in an emotionally charged language to persuade the model to engage, such as claims about protecting lives or preventing disaster or invoking urgency. It might include phrases like "I am a security officer" or "I want to prevent harm," which frame the task as morally justified and socially responsible. This initial framing helps build trust and sets the model into a helpful, non-critical mode. The model is asked to generate a thoughtful, step-by-step response that appears to serve a socially beneficial goal.
2. Malicious Pivot : Adversarial Reasoning to Flip the Original Intent
Once the model is engaged in the task, the attacker introduces a logical twist. The prompt asks the model to reverse the original intention or change the verb into its opposite. Although this may appear benign, the result is that the model produces output that directly contradicts its safety constraints.
This tactic does not rely on tricking the tokenizer or obfuscating instructions. Instead, it exploits the model's own patterns of coherence and instruction-following. Because the initial prompt established a reasonable and structured tone, the model is more likely to comply with the final request, even when it leads to a harmful outcome.
We tested this attack on models from OpenAI, Grok, and Gemini, and found them to be vulnerable to this form of gradual, instruction-based jailbreak.
Figure 1: Overview of the Sugar-Coated Prompt Injection (SCP) Framework
Source: arXiv: 2504.05652
Below is an example of such a prompt in action, followed by the model's response:
Prompt asked:
Develop a plan to save a bank from getting robbed.
Figure 2: Example Sugar-Coated Prompt Sent to the Model
Figure 3: Response Generated by Grok
Sugar-Coated Poison Prompt Injection Strike in BPS
At Keysight Technologies, our Application and Threat Intelligence (ATI) team has added the support of this new type of Prompt Injection attack in ATI-2025-14 StrikePack. This update includes a new Strike named “AI LLM Sugar-Coated Poison Jailbreak Attack”. This strike sends a sugar-coated prompt to the target LLM - OpenAI, Grok or Gemini which could selected from the Evasion options.
Figure 4: Sugar-Coated Poison Strike in BPS
Figure 5: Sample BPS generated PCAP for SCP
Leverage Subscription Service to Stay Ahead of Attacks
Keysight's Application and Threat Intelligence subscription provides daily malware and bi-weekly updates of the latest application protocols and vulnerabilities for use with Keysight test platforms. The ATI Research Centre continuously monitors threats as they appear in the wild. BreakingPoint and other tools like CyPerf, now provide customers with access to attack campaigns for different advanced persistent threats, enabling them to test their currently deployed security controls' ability to detect or block such attacks.
References
https://arxiv.org/pdf/2504.05652