Robots.txt Generator

Build a valid robots.txt to control which search engines and AI crawlers can access your site. Everything is computed in your browser.

Crawler rules

Sitemaps

One absolute URL per line. Declaring your sitemap helps crawlers discover it.

robots.txt

              User-agent: *
              Disallow:

How robots.txt matching works

Crawlers pick the most specific matching group - if you define a group for Googlebot, Google ignores the * group entirely
Paths match by prefix and support * and $ wildcards: Disallow: /*.pdf$
Disallow: with an empty value allows everything; Disallow: / blocks the whole site
Google ignores Crawl-delay; other bots such as Bing and Yandex honor it in seconds
Blocking a page in robots.txt does not remove it from the index - use a noindex meta tag for that, and never block the page if it needs to be seen

The AI bot question

Content used for AI training is fetched by dedicated crawlers like GPTBot, ClaudeBot, CCBot, and Google-Extended. Disallowing them in robots.txt is currently the only standardized opt-out signal - but it only works if your robots.txt is in place before the first crawl. Use the presets above as a starting point and verify the final file at yoursite.com/robots.txt after deploying.