Sensitive Information Disclosure by an LLM

The LLM feature reveals data the user should not see, such as other people's personal data, confidential records or credentials, from its training data, context or tools.

SeverityMedium
CVSS 3.14.3 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N Open in the calculator
CVSS 4.05.3 CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:L/VI:N/VA:N/SC:N/SI:N/SA:N Open in the calculator
CWECWE-201: Insertion of Sensitive Information Into Sent Data
OWASP Top 10A01:2025 Broken Access Control
CategoryInformation Disclosure

Description

A large language model (LLM) can only reveal what it has been given, but applications often give it more than the user in front of it should see. Sensitive information reaches the model's responses through several routes: personal or confidential data included in training or fine-tuning data, which the model may reproduce; data placed in its context, such as the system prompt, conversation memory shared between users, retrieved documents, or tool results that contain more fields than the task needs; and faults in session handling or caching that show one user's conversation to another.

Through any of these routes, a user, or an attacker with an ordinary account, may obtain other people's personal data, confidential business information or credentials by asking the model for them, or may receive them unprompted. The attacker cannot always choose what is revealed, particularly for memorised training data, which limits the impact in many cases. It is greater where the model's context or tools have broad access to customer records, or where credentials disclosed by the model give access to other systems, and exposure of personal data may need to be handled as a personal data breach.

The weakness is identified by asking for data belonging to other users or to the organisation, by probing for memorised training content, for example by giving the start of a record and asking the model to continue it, and by examining the raw API responses and tool results behind the conversation, which often contain more than the interface displays. Responses that contain personal data, credentials, internal identifiers or confidential documents the test account is not entitled to see confirm the finding. Retrieval that ignores document permissions, and leakage of the system prompt itself, are separate findings; this one covers the wider exposure of sensitive data through the model.

Remediation

Control what reaches the model, and treat everything it can see as visible to the user:

  • Minimise the data used for training and fine-tuning, and remove or pseudonymise personal and confidential data before use.
  • Apply the user's permissions before data enters the context: retrieve and pass only records the user is entitled to, and return only the fields the task needs from tools and APIs.
  • Keep credentials and secrets out of prompts and context entirely; tools should authenticate in the back end.
  • Isolate conversation history, memory and caches per user and per tenant, and test session handling for leakage between users.
  • Screen model output for personal data, payment card numbers and secrets with a data loss prevention filter before it is displayed, as defence in depth.
  • Review the model provider's data handling terms, including retention and any use of inputs for training, configure the service accordingly, and record the processing in the organisation's privacy documentation.

Confirm the fix by repeating the original requests from a low-privileged test account, and by checking the raw API responses as well as the displayed answers.

References


Add This Finding to Your Library

In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.

Free trial  ·  No card required

Only need the reporting? Our reporting tool is free and open source, with the same library.


More AI and LLM Applications Findings

Every AI and LLM Applications finding