Free tool · No sign-up
AI crawler checker
An AI crawler checker reads a site’s robots.txt and tells you, bot by bot, whether the crawlers behind ChatGPT, Claude, Perplexity, Google and other AI tools may crawl it. Ours covers 16 crawlers and shows the exact line that blocks each one. Type a domain below to try it, free.
7 days free. No card required.
Placton tracks
ChatGPT
Claude
Google AI
Perplexity
Grok
Check AI crawler access for any site
We read the site’s robots.txt the way each crawler must and show the line that lets it in or keeps it out.
Free and without sign-up. We read robots.txt only, never your pages.
GPTBot checker: is my site blocked from ChatGPT?
OpenAI uses three user agents, and blocking one of them is not the same as blocking ChatGPT. Check all three above; this is what each result means.
OAI-SearchBot
ChatGPT search
If you block it: Your pages stop appearing in ChatGPT search answers, although OpenAI says they can still show up as plain navigation links.
ChatGPT-User
Pages a user asks for
If you block it: ChatGPT may not open your page when a user asks about it. OpenAI notes that robots.txt may not apply here, since a person made the request.
GPTBot
Model training
If you block it: Your content is left out of what OpenAI may use to train its models. It does not remove you from ChatGPT search.
Three robots.txt patterns that catch people out
User-agent: * Disallow: /
Closes the site to every crawler that has no group of its own, OAI-SearchBot and Googlebot included. It is often left over from a staging site.
User-agent: GPTBot Disallow: /
Opts out of training only. ChatGPT search can still find and cite you through OAI-SearchBot.
User-agent: * Disallow: /checkout/ User-agent: OAI-SearchBot Allow: /
A crawler with its own group ignores the rules for all crawlers. Here OAI-SearchBot may enter /checkout/: copy the lines it should keep into its own group.
OpenAI says a robots.txt change can take about 24 hours to show in ChatGPT search (see its crawler documentation (opens in a new tab)). Access is only the start: our guide How to rank in ChatGPT covers the rest.
What each bot in the AI crawler checker does
16 user agents from 9 companies. Each description comes from the company’s own documentation, checked on October 24, 2026; ByteDance publishes none for Bytespider.
Scroll sideways to see every column.
| User agent | Used for | What it does | To appear in AI answers |
|---|---|---|---|
| OAI-SearchBotOpenAI documentation (opens in a new tab) | AI search | Finds pages to show and cite in ChatGPT search answers. Sites that block it do not appear in those answers. | Allow it to appear in its answers. |
| ChatGPT-UserOpenAI documentation (opens in a new tab) | User request | Opens a page when a ChatGPT user, or a custom GPT, needs it to answer a question.OpenAI says robots.txt rules may not apply, because a person asked for the page. | Allow it, so the AI can open your page when asked. |
| GPTBotOpenAI documentation (opens in a new tab) | Model training | Collects content that may be used to train OpenAI’s models. Blocking it does not remove you from ChatGPT search. | Your choice. It does not decide whether AI search cites you. |
| Claude-SearchBotAnthropic documentation (opens in a new tab) | AI search | Indexes pages so Claude can find them when it searches the web. Blocking it may reduce your visibility in Claude’s answers. | Allow it to appear in its answers. |
| Claude-UserAnthropic documentation (opens in a new tab) | User request | Fetches a page when a Claude user asks a question that needs it. | Allow it, so the AI can open your page when asked. |
| ClaudeBotAnthropic documentation (opens in a new tab) | Model training | Collects public web content that could be used to train Anthropic’s models. | Your choice. It does not decide whether AI search cites you. |
| PerplexityBotPerplexity documentation (opens in a new tab) | AI search | Finds and links pages in Perplexity’s answers. Perplexity says it is not used to train AI models. | Allow it to appear in its answers. |
| Perplexity-UserPerplexity documentation (opens in a new tab) | User request | Visits a page when a Perplexity user’s question needs it.Perplexity says it generally ignores robots.txt, because a person asked for the page. | Allow it, so the AI can open your page when asked. |
| GooglebotGoogle documentation (opens in a new tab) | AI search | Google’s search crawler. AI Overviews are part of Google Search, so they draw on what Googlebot can crawl. | Allow it. Blocking it also removes you from Google Search. |
| Google-ExtendedGoogle documentation (opens in a new tab) | Model training | Not a separate crawler but a robots.txt switch: it decides whether Google may use your pages for its AI models outside Search. It does not affect Search or AI Overviews. | Your choice. It does not decide whether AI search cites you. |
| ApplebotApple documentation (opens in a new tab) | AI search | Crawls the pages that power search in Spotlight, Siri and Safari.Apple says that when robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. | Allow it to appear in its answers. |
| Applebot-ExtendedApple documentation (opens in a new tab) | Model training | A robots.txt switch, not a crawler: it decides whether Apple may use what Applebot crawls to train its generative AI models. | Your choice. It does not decide whether AI search cites you. |
| Meta-ExternalAgentMeta documentation (opens in a new tab) | Model training | Crawls the web to train Meta’s AI models and to index content for its products. | Your choice. It does not decide whether AI search cites you. |
| DuckAssistBotDuckDuckGo documentation (opens in a new tab) | AI search | Fetches pages for the AI-assisted answers in DuckDuckGo, which cite their sources. DuckDuckGo says the data is not used to train AI models. | Allow it to appear in its answers. |
| CCBotCommon Crawl documentation (opens in a new tab) | Model training | Builds Common Crawl, a free public archive of the web that many AI models have been trained on. | Your choice. It does not decide whether AI search cites you. |
| BytespiderByteDance | Model training | ByteDance’s crawler, widely reported to collect data for its AI models. | Your choice. It does not decide whether AI search cites you. |
Which AI crawlers to allow
To appear in AI answers, let the search and user crawlers in. They are the ones that read your pages when ChatGPT, Claude or Perplexity answer with web search. For Google AI Overviews, what counts is Googlebot.
Training crawlers are a business decision. Blocking them keeps your content out of future training data and does not remove you from AI search. The trade-off: a model that has never read about you may know less about you when it answers without searching.
Each crawler follows only its most specific group. If you give a bot a group of its own, repeat there any Disallow lines it should still respect.
robots.txt
# AI search and pages users ask for: allowed User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Model training: opted out (optional) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Disallow: /
Allowed in is not the same as recommended
A crawler that may read your site still chooses what to cite. When a customer asks which brand to pick, the answer can name a competitor or cite a review site instead of you, and robots.txt cannot show that.
Placton asks ChatGPT, Google AI Overviews, Claude, Perplexity and Grok the questions your customers ask, and shows who appears, in what position and which pages each answer cites. For the rest of your site’s readiness, run the free 18-point GEO audit.
What the trial includes
Robots.txt decides who gets in. The trial shows what five AI platforms say once they have read your site.
7 days free, no card. Sign up with your email or Google. When the trial ends, tracking stops and nothing is charged.
Up to 10 prompts. We suggest prompts and competitors from your website, and you edit the list before the first run.
All five AI platforms on day one. ChatGPT, Google AI Overviews, Claude, Perplexity and Grok answer your prompts as soon as your email is confirmed.
ChatGPT and Google AI again every day while you keep visiting your dashboard. Claude, Perplexity and Grok are asked once, at sign-up.
One “Run now” and the 18-point site audit. Seven AI crawlers are part of it, and the full audit is also free, with no account, in our GEO audit tool.
On paid plans only: team invites and the weekly email summary with alerts.
* On paid plans. ChatGPT is read on chatgpt.com in every country Placton tracks except Venezuela and Puerto Rico, where it is asked weekly through its API.
Placton Starter
Track 20 prompts on 5 AI platforms
$59/ month (USD)
Or $602 billed yearly · save $106
Start your 7-day free trial7 days free. No card required.
- Up to 20 prompts, about 1,440 answers a month
- ChatGPT, Google AI Overviews, Claude, Perplexity, Grok
- ChatGPT and Google AI every day*, the other three every week
- Competitor ranking (up to 25 rivals) and every cited URL
- Weekly insights, plus an email summary with alerts
- 18-point AI-readiness site audit
- Up to 2 team members
Need more prompts? Growth is $129 a month for 75 and Scale $299 for 250. Compare all plans
AI crawler checker FAQ
About Placton itself? Read the full FAQs or see every plan and price.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects content that may be used to train OpenAI’s models. ChatGPT search finds and cites pages through OAI-SearchBot, and ChatGPT-User opens pages when a user asks. Block GPTBot and allow OAI-SearchBot if you want to opt out of training and stay in ChatGPT search.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search, which Googlebot crawls. Blocking Googlebot, on the other hand, would take you out of both.
Do AI crawlers respect robots.txt?
The documented ones say they follow it when they crawl on their own. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. Some crawlers have been reported to ignore robots.txt altogether; for those, only a firewall rule works.
My robots.txt allows everything. Why can AI tools still not read my site?
Something other than robots.txt may be in the way: a CDN or firewall that blocks bots (some offer a one-click setting to block AI crawlers), a login wall, or text that only appears after JavaScript runs, which many AI crawlers do not do.
How long do robots.txt changes take to work?
Each company decides. OpenAI says about 24 hours for ChatGPT search, and DuckDuckGo says changes for DuckAssistBot take effect after 72 hours. Run the checker after you publish to confirm the new rules read the way you meant.
What does the line number in the result mean?
It is the line of your robots.txt that decides for that crawler: the most specific Allow or Disallow rule that matches the homepage, in the group that names the crawler or, if none does, in the group for all crawlers (User-agent: *). That is how RFC 9309, the robots.txt standard, says crawlers must read the file.
Is the AI crawler checker free?
Yes. There is no account to create, and any public site can be checked. The limit is 10 checks an hour and 30 a day per connection; robots.txt is read, shown and forgotten.
The crawlers can get in. Now see what they say about you
Add your website, review the prompts and competitors we suggest, and read what all five AI platforms answer on day one.
7 days free. No card required.