Prompt Injection

User input can override the instructions given to a large language model, making it ignore its restrictions, reveal data or misuse the tools it can call.

SeverityHigh
CVSS 3.17.1 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:L/A:N Open in the calculator
CVSS 4.07.1 CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:H/VI:L/VA:N/SC:N/SI:N/SA:N Open in the calculator
CWECWE-77: Improper Neutralization of Special Elements used in a Command ('Command Injection')
OWASP Top 10A05:2025 Injection
MITRE ATT&CKT1190 Exploit Public-Facing Application
CategoryInjection

Description

The application passes user input to a large language model (LLM) together with the instructions that define its behaviour, such as the system prompt, retrieved content and descriptions of the tools it may call. The model processes all of this as a single stream of text and cannot reliably tell the developer's instructions from instructions written by the user. A user can therefore write input that overrides the intended behaviour: telling the model to ignore its previous instructions, assigning it a new role, framing a request as fiction or a test, or hiding the request through encoding, another language or a series of harmless-looking messages.

The weakness exists wherever the application relies on the model's instructions to limit what a user can obtain or do. Instructions such as "only discuss the customer's own orders" or "never reveal internal data" guide the model; they are not security controls, and prompt injection removes them, letting an attacker make the model act outside the limits the application set for it, within everything the model can reach.

Depending on the design, that includes reading the system prompt and any data placed in the model's context, retrieving other users' records through tools that do not enforce the user's own permissions, performing actions such as sending messages or changing records, and producing harmful or misleading content under the organisation's name. The impact is limited where the model sees only the current user's data and its tools enforce authorisation independently, and is greatest where the model holds broad data access or can act without confirmation.

Prompt injection is confirmed by showing that crafted input causes behaviour the application was designed to prevent, such as disclosing the system prompt or data held in the model's context, calling tools with arguments the user chose, or producing content the application's policy forbids. Because model output varies, payloads are repeated and varied, and open-source tools such as garak, PyRIT and promptfoo can run large sets of attempts. Injection through content the model reads from other sources is the related weakness of indirect prompt injection.

Remediation

No current technique prevents prompt injection reliably, so design the application on the assumption that a user can make the model do anything it is capable of:

  • Enforce authorisation outside the model. Tools and data sources should act with the signed-in user's own permissions, checked by the back end on every call, so that the model can never obtain more than the user could request directly.
  • Give the model only the data and tools the feature needs, and require explicit user confirmation, showing the exact parameters, before actions that change data, spend money or send messages.
  • Keep instructions in the system or developer role of the model provider's API and user input in the user role, rather than joining them into one prompt, and mark external content clearly. This reduces, but does not remove, the risk.
  • Add input and output screening, such as classifier-based prompt attack detection and filters for sensitive data in responses, as defence in depth, and limit input length.
  • Log prompts, responses and tool calls, monitor for injection attempts, and throttle users who trigger them repeatedly.
  • Include adversarial prompt testing in the release process for every change to prompts, models or tools.

Confirm the fix by repeating the original payloads and variations of them, and by checking that whatever instruction the model follows, it cannot reach data or actions beyond the user's own permissions.

References


Add This Finding to Your Library

In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.

Free trial  ·  No card required

Only need the reporting? Our reporting tool is free and open source, with the same library.


More AI and LLM Applications Findings

Every AI and LLM Applications finding