AI Crawler Robots.txt Builder
Choose which AI crawlers to allow or block — GPTBot, ClaudeBot, Google-Extended, and more — and generate the robots.txt rules.
Decide how AI uses your content
A new generation of crawlers scrapes the web to train large language models and to power AI answer engines. Most of the major ones — OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, PerplexityBot, Common Crawl's CCBot — respect robots.txt, so you can decide site-wide whether each one may use your content. This builder turns that into a simple checklist: toggle a bot off to block it, on to allow it.
There's a real trade-off. Blocking guards your work against uncredited training use; allowing can earn citations
and visibility inside AI-generated answers. Choose per bot to match your strategy, then copy the rules into your
robots.txt.
Private and instant
Generated entirely in your browser — nothing is uploaded.
Frequently Asked Questions
Can robots.txt block AI crawlers?
Reputable AI crawlers like GPTBot, ClaudeBot, and Google-Extended honor robots.txt. Adding a Disallow for their user-agent asks them not to use your content. Less reputable bots may ignore it.
What is Google-Extended?
Google-Extended is a separate token that controls whether your content is used to train and ground Google’s AI (Gemini), without affecting your normal Google Search crawling and ranking.
Should I block all AI crawlers?
It depends on your goals. Blocking protects content from training use; allowing can increase visibility in AI answers that cite sources. This builder lets you choose per bot.
Is this tool free and private?
Yes. The file is generated in your browser — nothing is uploaded or stored.
Want a strategy for AI search visibility? Let’s talk.