Robots.txt URL Tester
See whether a specific crawler is allowed to fetch a path and identify the exact matching rule.
Inspect robots.txt access for one URL
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
Interpret the result
Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
Prioritize fixes
The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
How to interpret robots.txt access for one URL
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties. The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
What the tool evaluates
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
- The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
- Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
- The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
Problems and next steps
Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
Frequently asked questions
What does this tool check?
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
How should I use the result?
Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
What are the limits?
The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
Related URL tools and guides
Robots.txt Checker
Fetch robots.txt and summarize user-agent groups, rules and sitemap declarations.
Learn more →Googlebot URL Checker
Test the response served to a Googlebot user-agent.
Learn more →Noindex Checker
Check meta robots and X-Robots-Tag directives that control search indexing.
Learn more →