Robots.txt Checker
Inspect the live robots.txt file for any domain without manually reading every directive.
Inspect robots.txt crawl rules
The audit reports availability, HTTP status, user-agent groups, Allow and Disallow rule counts, declared Sitemap URLs, and the raw file.
Interpret the result
The result helps you confirm which crawl instructions and sitemap locations are publicly exposed at the site root.
Catch common problems
A missing file, non-200 response, broad Disallow rule, malformed grouping, or outdated Sitemap entry can disrupt crawler discovery.
How to interpret robots.txt crawl rules
robots.txt controls crawling, not guaranteed index removal; this parser does not simulate every crawler’s precedence rules.
Common problems and next steps
A missing file, non-200 response, broad Disallow rule, malformed grouping, or outdated Sitemap entry can disrupt crawler discovery.
Frequently asked questions
What does the robots.txt crawl rules check cover?
The audit reports availability, HTTP status, user-agent groups, Allow and Disallow rule counts, declared Sitemap URLs, and the raw file.
How should I use the result?
Review the raw rules in user-agent context, narrow accidental blocks, keep Sitemap URLs current, and retest the published file.
What are the limits of this tool?
robots.txt controls crawling, not guaranteed index removal; this parser does not simulate every crawler’s precedence rules.