Sensitive Paths Disclosed in robots.txt

The robots.txt file lists paths the organisation would rather keep hidden, such as administrative or internal areas, pointing anyone who reads it straight to them.

SeverityInformational
CVSS 3.10 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N Open in the calculator
CVSS 4.00 CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:N/SC:N/SI:N/SA:N Open in the calculator
CWECWE-538: Insertion of Sensitive Information into Externally-Accessible File or Directory
OWASP Top 10A01:2025 Broken Access Control
MITRE ATT&CKT1594 Search Victim-Owned Websites
CategoryInformation Disclosure

Description

The site's robots.txt file, which tells search engine crawlers which paths not to visit, lists paths that reveal sensitive or unadvertised areas, such as administrative interfaces, staging or test areas, backups, internal tools or APIs. The file is public by design and is one of the first resources an attacker or tester requests, so a Disallow entry draws attention to a path rather than hiding it. It gives an attacker no access by itself, but it saves them effort in finding administrative, test and other unadvertised areas of the site, which they can then target for weak authentication, default credentials or unpatched software.

The file is a request to well-behaved crawlers, not an access control: it does not stop anyone, or any crawler that chooses to ignore it, from requesting the listed paths, and a disallowed page that is linked from elsewhere may still appear in search results by its address. The impact therefore depends on whether the listed paths are properly protected. Each listed path should be requested to establish whether it is reachable and whether it requires authentication; a listed path that exposes content without authentication is a separate and more serious finding.

Remediation

Do not use robots.txt to hide sensitive areas:

  • Remove entries that name administrative, internal, test or backup paths, keeping only those needed to manage how search engines crawl public content.
  • Protect sensitive areas with authentication and authorisation, and restrict administrative interfaces to trusted networks where possible, so that it does not matter whether their paths are known.
  • Remove test, staging and backup content from production servers altogether.
  • To keep public pages out of search results, use a noindex directive in a meta element or in the X-Robots-Tag response header; crawlers only see it if the page is not disallowed in robots.txt.

Confirm the fix by reviewing the published robots.txt and requesting each previously listed path to check that it has been removed or is protected.

References


Add This Finding to Your Library

In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.

Free trial  ·  No card required

Only need the reporting? Our reporting tool is free and open source, with the same library.


More Web Applications Findings

Every Web Applications finding