Skip to main content
InfoContentOWASP A05:2021CWE-200

How to fix: robots.txt Present - Review Manually

The response body looks like a robots.txt file. This check only confirms the file's format (User-agent/Disallow directives); it does not inspect which paths are listed.

Why it matters

If any Disallow entries point at genuinely sensitive areas, robots.txt effectively creates a map of them for attackers to probe.

Robots.txt files are publicly accessible. Manually review the Disallow entries for sensitive-looking paths (admin, backup, internal, etc.); a well-formed robots.txt with only public paths is not itself a finding.

How to fix it

  1. 1Manually review the listed Disallow paths for anything sensitive.
  2. 2Ensure all sensitive endpoints require proper authentication.
  3. 3Use auth and authorization rather than path hiding.

Code examples

Secure robots.txt

text
User-agent: *
Disallow: /api/
Sitemap: https://yourdomain.com/sitemap.xml

References

Detected automatically by VulnRadar. The check id is robots-txt-exposure, stable across every scan so you can filter or diff on it.

See every content check

Questions

How severe is robots.txt Present - Review Manually?
VulnRadar rates this info severity in the content category. The severity is fixed per check, not scored per site, so the same finding is the same severity on every scan.
Does VulnRadar check for this automatically?
Yes. It runs as part of every scan under the check id robots-txt-exposure, which stays the same between runs so you can filter, diff, or gate CI on it.
Which standards does robots.txt Present - Review Manually map to?
It maps to CWE-200. In the OWASP Top 10 it falls under A05:2021.

Missing a check?

The detection set is open. If you know a misconfiguration this does not catch, propose it: the issue template asks for the threat model, how to detect it, the false-positive risk, and a test fixture, which is the same thing a maintainer would have to work out anyway.

Propose a checkHow a check is built

Scan your own site for robots.txt present - review manually

VulnRadar runs this check and hundreds more against a URL in seconds. No agent to install, just paste a URL.

Or scan this site as a demoRead the docs

How to fix: robots.txt Present - Review Manually | VulnRadar