robots.txt tester for Google and AI crawlers
Check whether Googlebot, GPTBot, ClaudeBot, PerplexityBot and other crawlers may fetch a page, and see the exact rule that decides it. Test your live robots.txt or paste a new version before you publish it.
How the result is decided
The tester follows the same rules Google documents for robots.txt. A crawler uses the group with the most specific matching user-agent, so a User-agent: GPTBot group overrides User-agent: * for GPTBot. Within that group the longest matching rule wins, and when an Allow and a Disallow rule are equally long, Allow wins. The wildcards * and $ are supported.
robots.txt controls crawling, not indexing. A blocked page can still show up in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag and let crawlers fetch the page.
Questions
Which AI crawlers should I test?
GPTBot collects data for OpenAI model training, OAI-SearchBot builds the index for ChatGPT search, and ChatGPT-User fetches pages when a user asks. Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User in the same way. Google-Extended controls use of your content for Gemini, separately from Googlebot.
Does every AI crawler respect robots.txt?
The well-known crawlers from OpenAI, Anthropic, Google and Apple say they follow robots.txt. Fetches that a user triggers directly can be treated differently by some providers, so check their documentation.
Can I test a robots.txt before I publish it?
Yes. Paste the new file in the robots.txt field and the tester uses it instead of the live version.