How to fix: robots.txt Present - Review Manually
The response body looks like a robots.txt file. This check only confirms the file's format (User-agent/Disallow directives); it does not inspect which paths are listed.
Why it matters
If any Disallow entries point at genuinely sensitive areas, robots.txt effectively creates a map of them for attackers to probe.
Robots.txt files are publicly accessible. Manually review the Disallow entries for sensitive-looking paths (admin, backup, internal, etc.); a well-formed robots.txt with only public paths is not itself a finding.
How to fix it
- 1Manually review the listed Disallow paths for anything sensitive.
- 2Ensure all sensitive endpoints require proper authentication.
- 3Use auth and authorization rather than path hiding.
Code examples
Secure robots.txt
User-agent: *
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xmlReferences
Detected automatically by VulnRadar. The check id is robots-txt-exposure, stable across every scan so you can filter or diff on it.
More content checks
- MediumPotential Open Redirect Parameters
- HighDirectory Listing Appears Enabled
- MediumSensitive File References Detected
- HighPotentially Outdated JavaScript Libraries
- InfoCMS / Technology Fingerprinting
- MediumPotentially Dangerous Inline JavaScript
- LowServer version disclosed in response header
- MediumSensitive Keywords in HTML Comments
Questions
- How severe is robots.txt Present - Review Manually?
- VulnRadar rates this info severity in the content category. The severity is fixed per check, not scored per site, so the same finding is the same severity on every scan.
- Does VulnRadar check for this automatically?
- Yes. It runs as part of every scan under the check id robots-txt-exposure, which stays the same between runs so you can filter, diff, or gate CI on it.
- Which standards does robots.txt Present - Review Manually map to?
- It maps to CWE-200. In the OWASP Top 10 it falls under A05:2021.
Missing a check?
The detection set is open. If you know a misconfiguration this does not catch, propose it: the issue template asks for the threat model, how to detect it, the false-positive risk, and a test fixture, which is the same thing a maintainer would have to work out anyway.