Improper Handling of LLM Output

LLM output is passed to browsers, databases, shells or other interpreters without validation or encoding, so manipulated output becomes an injection attack.

SeverityHigh
CVSS 3.18.1 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N Open in the calculator
CVSS 4.08.6 CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:H/VI:H/VA:N/SC:N/SI:N/SA:N Open in the calculator
CWECWE-1426: Improper Validation of Generative AI Output
MITRE ATT&CKT1190 Exploit Public-Facing Application, T1059 Command and Scripting Interpreter
CategoryInput Validation

Description

The application treats text generated by a large language model (LLM) as trusted and passes it, without validation or encoding, to components that interpret it. Because anyone who influences the model's input, through direct or indirect prompt injection, can steer its output, this turns the model into a route for classic injection attacks against those components, and the impact is that of the injection the output reaches. A signed-in user usually has that influence directly, and an outside attacker may have it through content the model reads.

Common cases include model output rendered as HTML or Markdown in a web interface, leading to cross-site scripting, in which script runs in the session of the user viewing the chat interface and can steal their data or act as them, or to data leaving through images and links that the browser loads; generated SQL or other query languages run directly against a database, and output passed to a shell, eval, a code interpreter or a template engine, both of which run with the application's privileges and can expose or alter the whole database or run code on the server; output used to build file paths or URLs that the server then opens or fetches, leading to path traversal or server-side request forgery that can reach internal services; and output placed into emails or messages sent to other people. The impact is lower where output only reaches a browser with strict encoding and a restrictive Content Security Policy.

The weakness is identified by tracing where model output goes in the application, and confirmed by prompting the model to produce a payload suited to each destination, such as an HTML element with an event handler, a query with an extra clause or a URL pointing at an internal address, and showing that the payload takes effect. Reviewing the Markdown renderer's configuration, including whether it allows raw HTML, and the Content Security Policy of the chat interface shows the browser-side exposure.

Remediation

Treat model output as untrusted input, and handle it as the application would any other user-supplied data:

  • Encode output for the context it is placed in. Render Markdown with raw HTML disabled, sanitise any HTML with a maintained library such as DOMPurify, restrict images and links to approved domains, and apply a restrictive Content Security Policy to the interface.
  • Never pass output to eval, a shell or a template engine. Where generated code must run, run it in an isolated, short-lived sandbox without access to secrets or internal networks.
  • Rather than running generated SQL, have the model return structured parameters, validated against a JSON schema, which the application places into fixed, parameterised queries. Where free-form queries are a requirement, run them with a read-only database account limited to the permitted tables and rows.
  • Validate file names, paths and URLs taken from output against strict allowlists before use.
  • Use the model provider's structured output features to constrain responses to the expected format, and reject output that does not match it.

Confirm the fix by repeating the payload tests for each destination and checking that the payloads are displayed or rejected as data rather than interpreted.

References


Add This Finding to Your Library

In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.

Free trial  ·  No card required

Only need the reporting? Our reporting tool is free and open source, with the same library.


More AI and LLM Applications Findings

Every AI and LLM Applications finding