Robots.txt URL Tester
See whether a specific crawler is allowed to fetch a path and identify the exact matching rule.
Inspect robots.txt access for one URL
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
Interpret the result
Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
Prioritize fixes
The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
How to interpret robots.txt access for one URL
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
Limits of this check
The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
Frequently asked questions
What does this tool check?
The tester loads the origin robots.txt, selects the most specific matching user-agent group and applies longest-rule matching with Allow winning equal-length ties.
How should I use the result?
Review the matched group and rule before narrowing an accidental block or documenting an intentional exclusion.
What are the limits?
The result evaluates published robots.txt syntax only; firewalls, authentication and crawler-specific behavior may still prevent a request.
Related URL tools and guides
Robots.txt Generator
Build a valid robots.txt with blocked paths, sitemap links and opt-outs for AI crawlers like GPTBot.
Learn moreRobots.txt Checker
Fetch robots.txt and summarize user-agent groups, rules and sitemap declarations.
Learn moreGooglebot URL Checker
Test the response served to a Googlebot user-agent.
Learn more