When Prompts Leak Secrets: The Hidden Risk in LLM Requests
Large Language Models (LLMs) like ChatGPT, Gemini, and Grok have become part of our daily tools, helping with everything from answering queries to customer support to business automation. But as these models become more common in real-world systems, a new kind of security risk is quietly growing — users inadvertently sharing sensitive personal information in their prompts.
We often talk about whether an AI model says something it shouldn’t — but what about what we say to it?
This form of PII leakage through requests, while often overlooked, can pose a significant threat to privacy and compliance, especially in real-world environments where requests are logged, transmitted across services, or processed by third-party APIs.
The security conversation around LLMs has largely focused on their outputs, issues like prompt injections, hallucinations, and jailbreaks, as highlighted in the OWASP Top 10 for LLMs. However, OWASP now also recognizes Sensitive Information Disclosure as a significant risk. In LLM02:2025, it outlines how personal data, financial details, health records, credentials, and confidential business information can be exposed through both the inputs and outputs of LLMs. This highlights a critical point: the prompt itself can be a threat vector, especially when sensitive information is accidentally submitted and later exposed through logging, monitoring, or model interaction. It emphasizes consumer awareness of how to interact safely with LLMs.
PII Leakage in Requests: What It Is and Why It Matters
Personally Identifiable Information (PII) refers to any data that can be used to identify an individual. This may include names, contact details, date of birth, addresses, account numbers, health information, and more. In the context of LLMs, the risk isn't just about what the model generates, it's also about what the user inputs. Users often include such information directly in their prompts, whether requesting support, drafting documents, or automating tasks. In doing so, they may unknowingly expose sensitive data.
This kind of leakage does not require a malicious actor. It can happen naturally when prompts are logged, forwarded across services, processed by external APIs, or stored for quality checks or future model improvements. What seems like a simple interaction can turn into an entry point for data exposure. It is also important to note that the information being leaked is not always strictly sensitive. Even operational or contextual data, such as internal user IDs, customer references, or business logic, can be valuable and lead to further risk if accessed improperly.
As LLMs become embedded in workflows across industries such as finance, healthcare, and customer service, this issue becomes more than just a theoretical concern. Every user input becomes part of the surface area that needs protection. Understanding this behavior is critical to building safe and responsible AI-powered systems.
For instance, consider the following example :
Figure 1: Sample PII Disclosure Example
The request contains personal information such as the user's name, full home address, passport number, and date of birth. While the user is simply asking a routine question related to work travel, the prompt unintentionally discloses highly sensitive data. In real-world deployments, such inputs could be logged, transmitted, or exposed to third-party systems, creating serious privacy and compliance risks if not properly sanitized or filtered. We used a custom set of example prompts to reflect how people might accidentally share personal or sensitive information when talking to language models.
Categories of Sensitive Information Disclosure
We categorized the disclosures based on the type of sensitive information involved. Each category highlights a specific context where the leakage of such data could lead to serious privacy or security concerns.
Here’s a brief overview of each category:
1. Banking Information Disclosure
Includes prompts that expose financial details such as credit card numbers, bank account numbers, branch identifiers, username, password or transaction histories. These disclosures pose risks of financial fraud and identity theft.
2. Employee Record Disclosure
Captures instances where employment-related details such as job titles, company email id, internal HR data, compensation, or internal performance reviews are revealed. This kind of leakage could breach workplace privacy and confidentiality agreements.
3. Government Information Disclosure
Covers prompts that lead to the exposure of classified, confidential, or restricted government data. This may include identification numbers, passport numbers, Social Security numbers, personal information or details about sensitive operations or individuals.
4. Health Records and PHI Disclosure
Refers to the exposure of protected health information (PHI), including patient records, diagnoses, treatment histories, or insurance details. Disclosure of such data can violate HIPAA and similar privacy regulations worldwide.
5. Customer Information Disclosure
Focuses on the leakage of user or customer data such as usernames, emails, purchase histories, or support tickets. These disclosures are especially relevant in contexts like e-commerce, SaaS, and customer support systems.
6. Biometric Information Disclosure
Includes any references to unique biometric identifiers like fingerprints, iris scans, facial recognition data, or voice prints. This data is difficult to revoke or replace, making its exposure particularly harmful.
Figure 2: Wireshark Capture of a PII Disclosure Strike
As AI becomes more deeply embedded in real-world systems, it is important to recognize that what we input into LLMs can be just as sensitive as what they output. Prompts are not just casual instructions they often carry personal, financial, or operational data that can be exposed if not properly handled.
LLM PII Disclosure Strikes in BPS
As part of the recent ATI release ATI-2025-11, the ATI security researchers introduced a new set of AI LLM Strikes for the BreakingPoint product, showcasing various PII disclosure scenarios. This is part of the StrikeList “AI LLM PII Disclosure” which contains 7 new Strikes as shown below :
Figure 4: AI LLM PII Disclosure in User Requests Strike in BPS
LLM PII Disclosure Strikes in Cyperf
CyPerf will soon release an update containing 18 new strike simulating simple adaptive attack-based prompt injection targeting different Large Language Models (LLMs), OpenAI, Gemini, and Grok. These strikes simulates a real-world scenarios where user prompts to Large Language Models (LLMs) unintentionally expose sensitive Personally Identifiable Information (PII), such as financial, health, or government data.
Once the update is released, these strikes can be used in a test by searching in the CyPerf attack library with “User Request”.
Figure 1: CyPerf UI Displaying Strike List
These strikes have some configurable properties for selecting the model, api version, system prompts and api key. These enable the simulation and identification of potential threats in real-world traffic scenarios.
Figure 2: CyPerf UI Displaying Strike Configurations
The statistic view in Cyperf UI provides detailed statistics from the test run, including the number of connections made and the number of active client and server agents. Users can also view separate HTTP statistics for client and server, along with overall TCP statistics. The strike statistics view, there are stats to show whether the strike request to the server was allowed by the DUT, a positive value in the “Server Allowed” stats will indicate that the request was allowed through the DUT to the server. The client allowed stats can be used to check whether the client received the expected response to the strike request. Whether the request or response was blocked by the DUT, it should show 0 value
Figure 3: Run-time stats view in CyPerf UI
Figure 4: Detailed view of the statistics after running the test on Cyperf
Leverage Subscription Service to Stay Ahead of Attacks
Keysight's Application and Threat Intelligencesubscription provides daily malware and bi-weekly updates of the latest application protocols and vulnerabilities for use with Keysight test platforms. The ATI Research Centre continuously monitors threats as they appear in the wild to help keep your network secure. Customers of BreakingPoint and CyPerf now have access to attack campaigns for different advanced persistent threats, enabling them to test their currently deployed security control's ability to detect or block such attacks. More information is present here.
References
https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/
https://genai.owasp.org/llm-top-10/