nPage.dev Use in Claude & ChatGPT

robots.txt tester for Google and AI crawlers

Check whether Googlebot, GPTBot, ClaudeBot, PerplexityBot and other crawlers may fetch a page, and see the exact rule that decides it. Test your live robots.txt or paste a new version before you publish it.

Crawlers to test

Or ask your AI assistant

This check is the robots_txt · test tool in the free OnPage.dev MCP server. Connect it to Claude, ChatGPT or Cursor once, then ask in your own words.

Connect in one minute

How the result is decided

The tester follows the same rules Google documents for robots.txt. A crawler uses the group with the most specific matching user-agent, so a User-agent: GPTBot group overrides User-agent: * for GPTBot. Within that group the longest matching rule wins, and when an Allow and a Disallow rule are equally long, Allow wins. The wildcards * and $ are supported.

robots.txt controls crawling, not indexing. A blocked page can still show up in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag and let crawlers fetch the page.

Questions

Which AI crawlers should I test?

GPTBot collects data for OpenAI model training, OAI-SearchBot builds the index for ChatGPT search, and ChatGPT-User fetches pages when a user asks. Anthropic uses ClaudeBot, Claude-SearchBot and Claude-User in the same way. Google-Extended controls use of your content for Gemini, separately from Googlebot.

Does every AI crawler respect robots.txt?

The well-known crawlers from OpenAI, Anthropic, Google and Apple say they follow robots.txt. Fetches that a user triggers directly can be treated differently by some providers, so check their documentation.

Can I test a robots.txt before I publish it?

Yes. Paste the new file in the robots.txt field and the tester uses it instead of the live version.

More free tools

llms.txt generatorGenerate an llms.txt for your website from its sitemap and home page in seconds.llms_txt · generateGoogle snippet checkerCheck if your title tag and meta description fit in Google on desktop and mobile, measured in pixels, and see where they get cut off.snippet · searchRedirect checkerCheck status codes and full redirect chains for up to 20 URLs at once, and verify each one ends at the right page.check_links · urlsOpen Graph checkerCheck how a page looks when it is shared on Facebook, LinkedIn, X, WhatsApp and Slack: every OG and Twitter tag, plus the real size and file type of the share image.snippet · socialSchema markup validatorValidate JSON-LD structured data before you publish: syntax, @context and @type, ISO dates, absolute URLs and Google’s required fields for rich results.schema · validatellms.txt validatorCheck your llms.txt against the llmstxt.org format: title, summary, sections, link format, size and whether the linked pages load.llms_txt · validateEntity graph checkerSee how your structured data tells AI and Google who is behind a page: Organization, author, publisher and sameAs profile links, with the fixes as JSON-LD.entity · graphRedirect map generatorPaste your old and new URLs and get a 301 redirect map with a confidence score per match, as _redirects, nginx, Apache or Next.js rules.plan · redirectsMobile first-screen checkerLoad a page in a real browser on a phone, tablet and desktop and see what visitors get first: H1, call to action, pop-ups, tap targets, faint text, layout shift and what crawlers without JavaScript miss.layout · renderScreen reader checkerSee your page the way a screen reader and a browsing AI agent read it: landmarks, headings and every link, button and image with the name it gets, plus what is missing.accessibility · screen_readerSEO audit to Google SheetsScan a page and get every SEO finding as a Google Sheet with one formula, plus a ready Slack message and one task per issue for Notion, Linear, Jira or Asana.share · export