robots.txt tester for Google and AI crawlers
Check whether Googlebot, GPTBot, ClaudeBot, PerplexityBot and other crawlers may fetch a page, and see the exact rule that decides it. Test your live robots.txt or paste a new version before you publish it.
Or ask your AI assistant
This check is the robots_txt · test tool in the free OnPage.dev MCP server. Connect it to Claude, ChatGPT or Cursor once, then ask in your own words.
May GPTBot and ClaudeBot fetch example.com/blog? Check it with OnPage.dev.
robots_txt · testGPTBot blocked by Disallow: /blogrobots_txt · ai_policyRules for 13 AI crawlers, ready to paste
How the result is decided
The tester follows the same rules Google documents for robots.txt. A crawler uses the group with the most specific matching user-agent, so a User-agent: GPTBot group overrides User-agent: * for GPTBot. Within that group the longest matching rule wins, and when an Allow and a Disallow rule are equally long, Allow wins. The wildcards * and $ are supported.
robots.txt controls crawling, not indexing. A blocked page can still show up in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag and let crawlers fetch the page.
Questions
Which AI crawlers should I test?
GPTBot collects data for OpenAI model training, OAI-SearchBot builds the index for ChatGPT search, and ChatGPT-User fetches pages when a user asks. Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User in the same way. Google-Extended controls use of your content for Gemini, separately from Googlebot.
Does every AI crawler respect robots.txt?
The well-known crawlers from OpenAI, Anthropic, Google and Apple say they follow robots.txt. Fetches that a user triggers directly can be treated differently by some providers, so check their documentation.
Can I test a robots.txt before I publish it?
Yes. Paste the new file in the robots.txt field and the tester uses it instead of the live version.
More free tools
llms_txt · generateGoogle snippet checkerCheck if your title tag and meta description fit in Google on desktop and mobile, measured in pixels, and see where they get cut off.snippet · searchRedirect checkerCheck status codes and full redirect chains for up to 20 URLs at once, and verify each one ends at the right page.check_links · urlsOpen Graph checkerCheck how a page looks when it is shared on Facebook, LinkedIn, X, WhatsApp and Slack: every OG and Twitter tag, plus the real size and file type of the share image.snippet · socialSchema markup validatorValidate JSON-LD structured data before you publish: syntax, @context and @type, ISO dates, absolute URLs and Google’s required fields for rich results.schema · validatellms.txt validatorCheck your llms.txt against the llmstxt.org format: title, summary, sections, link format, size and whether the linked pages load.llms_txt · validateEntity graph checkerSee how your structured data tells AI and Google who is behind a page: Organization, author, publisher and sameAs profile links, with the fixes as JSON-LD.entity · graphRedirect map generatorPaste your old and new URLs and get a 301 redirect map with a confidence score per match, as _redirects, nginx, Apache or Next.js rules.plan · redirectsMobile first-screen checkerLoad a page in a real browser on a phone, tablet and desktop and see what visitors get first: H1, call to action, pop-ups, tap targets, faint text, layout shift and what crawlers without JavaScript miss.layout · renderScreen reader checkerSee your page the way a screen reader and a browsing AI agent read it: landmarks, headings and every link, button and image with the name it gets, plus what is missing.accessibility · screen_readerSEO audit to Google SheetsScan a page and get every SEO finding as a Google Sheet with one formula, plus a ready Slack message and one task per issue for Notion, Linear, Jira or Asana.share · export