Robots.txt AI-Crawler Tester
Check which AI crawlers your robots.txt allows or blocks (GPTBot, ClaudeBot, Google-Extended, and more), then generate the exact rules you want.
Free to use. Enter your email once to copy or download. Runs entirely in your browser.
Paste your robots.txt
Paste the contents of your robots.txt file. We parse it and show which known AI crawlers are allowed or blocked. Nothing is uploaded: this runs entirely in your browser.
Crawler analysis
Result for each known AI crawler. A crawler is Blocked when its rules containDisallow: / (a full-site block).
| Crawler | Status | Rule applied |
|---|
This checks the common full-block case (Disallow: /) and treats "no matching rules" as allowed. It is a quick check, not full RFC parsing.
Choose AI crawlers to block
Tick the AI crawlers you want to block. The robots.txt rules update live on the right. Nothing is uploaded: this runs entirely in your browser.
Your robots.txt rules
Paste these lines into the robots.txt file at the root of your domain (for example https://yoursite.com/robots.txt).
What are AI crawlers?
AI crawlers are bots that fetch your pages for AI systems. Some collect text to train large language models (GPTBot, Google-Extended, ClaudeBot, CCBot). Others fetch a page on demand when a user asks a question in a chatbot or an AI search engine (ChatGPT-User, PerplexityBot, OAI-SearchBot). They identify themselves with a user-agent string, which is what your robots.txt matches against.
How to block AI crawlers in robots.txt
To block an AI crawler, add a group for its user-agent with a full-site disallow. To block GPTBot, OpenAI's training crawler, you add:
User-agent: GPTBot Disallow: /
Repeat one group per crawler you want to block. The generator above builds this for you. An AI bot blocker is only as good as the user-agent list behind it, so this tool tracks the crawlers that matter today.
Should you block them? The tradeoff
Blocking AI scrapers is a real choice, not an obvious win. There are two different questions hiding inside it.
The first is training. Blocking GPTBot, Google-Extended, ClaudeBot, or CCBot tells those companies not to use your content to train their models. If your concern is that your writing or images get absorbed into a model without credit, blocking the training crawlers is a reasonable stance.
The second is visibility. Many of these same systems now answer questions and cite sources. If you block the crawlers that feed AI answers, you can reduce your presence in those answers. Blocking Google-Extended or GPTBot can quietly remove you from places where buyers are now researching. That is the heart of GEO, or Generative Engine Optimization: staying visible where AI engines describe and recommend businesses.
A common middle path is to allow the on-demand and search crawlers that surface you in AI answers, while blocking the pure training crawlers. There is no single right answer. It depends on whether you value control over your content or reach in AI search more, and that is worth thinking through deliberately.
Copy or download your result
Enter your email to export. You'll also get the occasional new free tool and first access when we open our next product to test users. No spam, unsubscribe anytime.
Enter your email below to unlock copy and download.
You're unlocked. Copy and download are enabled, and we'll be in touch when there's something worth your time.
Want this done for your whole site, and tracked over time?
Blocking or allowing AI crawlers is one lever. Deciding which bots to feed for AI search visibility and which to block, across your whole site, is a strategy. On a discovery call we map that for you.
Book a Discovery Call