Sensitive Paths Disclosed in robots.txt
The robots.txt file lists paths the organisation would rather keep hidden, such as administrative or internal areas, pointing anyone who reads it straight to them.
| Severity | Informational |
| CVSS 3.1 | 0 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N Open in the calculator |
| CVSS 4.0 | 0 CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:N/SC:N/SI:N/SA:N Open in the calculator |
| CWE | CWE-538: Insertion of Sensitive Information into Externally-Accessible File or Directory |
| OWASP Top 10 | A01:2025 Broken Access Control |
| MITRE ATT&CK | T1594 Search Victim-Owned Websites |
| Category | Information Disclosure |
Description
The site's robots.txt file, which tells search engine crawlers which paths not to visit, lists paths that reveal sensitive or unadvertised areas, such as administrative interfaces, staging or test areas, backups, internal tools or APIs. The file is public by design and is one of the first resources an attacker or tester requests, so a Disallow entry draws attention to a path rather than hiding it. It gives an attacker no access by itself, but it saves them effort in finding administrative, test and other unadvertised areas of the site, which they can then target for weak authentication, default credentials or unpatched software.
The file is a request to well-behaved crawlers, not an access control: it does not stop anyone, or any crawler that chooses to ignore it, from requesting the listed paths, and a disallowed page that is linked from elsewhere may still appear in search results by its address. The impact therefore depends on whether the listed paths are properly protected. Each listed path should be requested to establish whether it is reachable and whether it requires authentication; a listed path that exposes content without authentication is a separate and more serious finding.
Remediation
Do not use robots.txt to hide sensitive areas:
- Remove entries that name administrative, internal, test or backup paths, keeping only those needed to manage how search engines crawl public content.
- Protect sensitive areas with authentication and authorisation, and restrict administrative interfaces to trusted networks where possible, so that it does not matter whether their paths are known.
- Remove test, staging and backup content from production servers altogether.
- To keep public pages out of search results, use a
noindexdirective in ametaelement or in theX-Robots-Tagresponse header; crawlers only see it if the page is not disallowed inrobots.txt.
Confirm the fix by reviewing the published robots.txt and requesting each previously listed path to check that it has been removed or is protected.
References
Add This Finding to Your Library
In Pental this write-up is in Settings, Library. Add to my library copies it into your own, rated in your firm's CVSS version, ready to put in your own words.
Free trialYour trial starts when your portal goes live, not when you sign up, so the time you spend setting up does not count. · No card required
Only need the reporting? Our reporting tool is free and open source, with the same library.