Rules, sitemaps and AI crawlers

Robots.txt Generator

Choose what crawlers may access, see warnings for risky rules and download the file ready to upload.

Runs in your browserNothing is uploaded

Input

Try an example
Default for all crawlers

One path per line, e.g. /admin/

AI crawlers to block

Model training

AI search and assistants

Output

robots.txt
User-agent: *
Disallow:

Save it as robots.txt in the root of your domain, e.g. https://example.com/robots.txt

Checks

  • Tip: add your sitemap URL so crawlers can find it
Test URLs against your rules with the Robots.txt URL Tester
Free to use No account required Runs in your browser

Inspect robots.txt crawl rules

The file is assembled with valid User-agent groups, and paths without a leading slash, blocked CSS or JavaScript, relative sitemaps and risky blanket blocks are flagged.

Interpret the result

Upload the file to the root of your domain, then test important URLs against it before relying on the new rules.

Prioritize fixes

robots.txt is a request, not access control: compliant crawlers follow it, but it does not hide or remove pages that are already indexed.

How to interpret robots.txt crawl rules

The file is assembled with valid User-agent groups, and paths without a leading slash, blocked CSS or JavaScript, relative sitemaps and risky blanket blocks are flagged.

Limits of this check

robots.txt is a request, not access control: compliant crawlers follow it, but it does not hide or remove pages that are already indexed.

Frequently asked questions

What does this tool check?

The file is assembled with valid User-agent groups, and paths without a leading slash, blocked CSS or JavaScript, relative sitemaps and risky blanket blocks are flagged.

How should I use the result?

Upload the file to the root of your domain, then test important URLs against it before relying on the new rules.

What are the limits?

robots.txt is a request, not access control: compliant crawlers follow it, but it does not hide or remove pages that are already indexed.

Related URL tools and guides

Robots.txt URL Tester

Test a URL against robots.txt rules for Googlebot, Bingbot and major AI crawlers.

Learn more

Robots.txt Checker

Fetch robots.txt and summarize user-agent groups, rules and sitemap declarations.

Learn more

Sitemap URL Checker

Discover a sitemap and check status codes for up to 1,000 URLs.

Learn more
urlscheck