Excessive Agency Granted to an LLM

An LLM is given more tools, permissions or freedom to act than its task needs, so manipulated or mistaken output can cause damaging actions.

SeverityHigh
CVSS 3.18.1 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N Open in the calculator
CVSS 4.08.6 CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:H/VI:H/VA:N/SC:N/SI:N/SA:N Open in the calculator
CWECWE-250: Execution with Unnecessary Privileges
CategoryInsecure Design

Description

The application lets a large language model (LLM) take actions by calling tools, functions, plugins or Model Context Protocol (MCP) servers, and the model has more agency than its purpose requires. The weakness takes three common forms. Excessive functionality: tools offer operations the feature does not need, such as a mailbox tool that can send and delete as well as read, or general-purpose shell, SQL or HTTP request tools. Excessive permissions: tools connect to other systems with a privileged service account rather than the identity of the user being served, often with access to every user's data. Excessive autonomy: actions that change data, spend money or send messages run without anyone confirming them.

On its own, excessive agency is a design flaw. It becomes exploitable when the model's behaviour is manipulated through direct or indirect prompt injection, or simply when the model misunderstands a request. An attacker who can influence the model, directly or by planting content it reads, can then make it carry out any action its tools allow, with the tools' privileges: depending on the tools, reading or changing other users' data, sending messages or emails in the organisation's name, deleting records, making purchases or running commands on the systems the tools reach. The impact is bounded by the tools and permissions granted, and is limited where every tool acts with the current user's own permissions and high-impact actions need the user's confirmation.

The weakness is identified by enumerating the tools available to the model, from their definitions in API traffic or configuration or by asking the model to describe them; by establishing which identity and permissions each tool uses in the systems it reaches; and by attempting, through crafted prompts, to make the model use tools outside their intended purpose, for example against another user's records, while checking whether any confirmation step intervenes.

Remediation

Limit what the model can do to the minimum the feature needs, and enforce those limits outside the model:

  • Remove tools and operations the feature does not use, and prefer narrow, purpose-built tools, such as one that looks up an order belonging to the current user, to general shell, SQL, file system or HTTP request tools.
  • Run tools with the identity of the user being served, through delegated authorisation such as OAuth with narrowly scoped tokens, and enforce authorisation in the systems the tools call, never through instructions to the model. Where a service account is unavoidable, give it least-privilege, preferably read-only, access.
  • Require explicit user confirmation, showing the exact action and parameters, before any action that changes data, spends money, sends messages or reaches outside the organisation.
  • Validate tool arguments against strict schemas and allowlists, apply rate and spending limits to actions, and run any code execution in an isolated sandbox without network access to internal systems.
  • Log every tool call with the identity, the parameters and the preceding conversation, and alert on unusual patterns.
  • Review the permissions of each MCP server and plugin, and the scopes of the tokens given to them, before connecting them.

Confirm the fix by repeating the attempts to misuse each tool, checking that out-of-scope calls fail in the back end and that consequential actions wait for the user's confirmation.

References


Add This Finding to Your Library

In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.

Free trial  ·  No card required

Only need the reporting? Our reporting tool is free and open source, with the same library.


More AI and LLM Applications Findings

Every AI and LLM Applications finding