CodeKitHubWhat Is This Tool?
robots.txt is a plain-text file at your site's root that tells web crawlers which parts of the site they may or may not access. Search engines like Google and Bing respect it for indexing; in 2026 it's also the standard way sites opt out of having their content used to train AI models, via specific bot names like GPTBot, CCBot and Google-Extended.
Getting the syntax exactly right matters — a wrong Disallow path can accidentally block search engines from your whole site, or an AI-blocking rule placed incorrectly can fail to actually block anything. This generator produces syntactically correct output for each scenario.
Why Use It?
- A dedicated AI-crawler-blocking preset covering GPTBot, ChatGPT-User, CCBot, Google-Extended, ClaudeBot, Bytespider and anthropic-ai — while explicitly keeping search engine access.
- Custom disallow paths for content you want hidden from all crawlers.
- Sitemap URL field — a small addition that helps search engines discover your pages faster.
- Correct syntax every time — no risk of an accidental typo blocking your whole site.
- Free, instant, download the file directly or copy the text.
How to Use
- Pick a preset: allow all, block all, or block AI crawlers only.
- Optionally add custom disallowed paths, one per line (e.g. /admin/).
- Optionally add your sitemap URL.
- Click Copy or Download, then upload the file to your site's root as /robots.txt.
Example
Input
Preset: Block AI crawlers, keep search engines. Sitemap: https://example.com/sitemap.xmlOutput
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
...
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlEach AI crawler gets its own explicit block, while the wildcard rule at the end keeps normal search engine access open.
Frequently Asked Questions
Does robots.txt actually stop AI from using my content?
It works on an honor system — reputable AI companies' crawlers do respect it (that's why GPTBot, CCBot etc. exist as distinct, documented user-agents specifically so sites can opt out). It does not stop crawlers that ignore the standard, and it has no effect on content already scraped before you added the block.
Will blocking AI crawlers hurt my Google/Bing search ranking?
No — the AI-crawler preset specifically blocks only the named AI bots and leaves a wildcard "Allow: /" for everyone else, so Googlebot and Bingbot are unaffected.
Where do I put the robots.txt file?
At the root of your domain, so it's reachable at yourdomain.com/robots.txt exactly — not in a subfolder. Most website hosts and static site generators have a designated public/root folder for this.
What's the sitemap line for?
It points crawlers to your sitemap.xml, which lists all your indexable pages. It's optional but commonly recommended — one extra line that can speed up how quickly new pages get discovered.
Can I combine custom disallow paths with the AI-blocking preset?
Yes — the AI-blocking preset adds AI bot blocks first, then applies your custom disallow paths (if any) under the general wildcard rule that still allows search engines.