AI crawler checker
Blanc shows which AI crawlers can reach your page, including the ones behind ChatGPT Search, Claude, Perplexity, Google AI Overviews and Bing Copilot. It reads your robots.txt rules for 12 crawlers, looks for firewalls and noindex tags, and checks whether your text is in the HTML. It’s part of the free Blanc audit: no sign-up, results in a few seconds.
You get the full Blanc audit, the same as on the homepage. For the crawler results, open “Which AI crawlers can read this page” under More details. Anything that blocks a crawler also shows at the top, in Fixes, in order.
What does the AI crawler checker look at?
- robots.txt rules
- Whether each of the 12 crawlers below may fetch the exact page you enter, read the way crawlers read them (the longest matching rule wins), and whether robots.txt itself loads. A robots.txt that returns a server error tells most crawlers to stay away from the whole site.
- Firewalls and bot protection
- Signs that Cloudflare, Akamai, Imperva, Sucuri, AWS WAF, DDoS-Guard or a CAPTCHA stopped the check, with what that means and what to change.
- noindex and nosnippet
- Tags and headers that tell search engines, and the AI answers built on them, not to show or quote the page.
- Text without JavaScript
- How much of your text is in the HTML your server sends, since most AI crawlers don’t run scripts, and whether your links can be followed without them.
Which AI crawlers does it check?
12 crawlers from 5 companies: OpenAI, Anthropic, Perplexity, Google and Microsoft. 5 find pages for AI search, 3 open a page when a person asks, 3 are for training and 1 reviews ChatGPT ads. Only the search and on-request crawlers decide whether AI search can find and quote you. The one exception is Google-Extended, which also controls Gemini Apps answers.
| Crawler | What it does | If you block it |
|---|---|---|
| OAI-SearchBot OpenAI | AI search: ChatGPT Search | Your pages can drop out of this company’s AI search answers. |
| ChatGPT-User OpenAI | Opens a page on request: ChatGPT (user-requested) | The assistant can’t open your page when someone asks it to. |
| GPTBot OpenAI | Model training: OpenAI model training | Your content stays out of future model training. AI search answers are not affected. |
| Claude-SearchBot Anthropic | AI search: Claude search | Your pages can drop out of this company’s AI search answers. |
| Claude-User Anthropic | Opens a page on request: Claude (user-requested) | The assistant can’t open your page when someone asks it to. |
| ClaudeBot Anthropic | Model training: Anthropic model training | Your content stays out of future model training. AI search answers are not affected. |
| PerplexityBot Perplexity | AI search: Perplexity search | Your pages can drop out of this company’s AI search answers. |
| Perplexity-User Perplexity | Opens a page on request: Perplexity (user-requested) | The assistant can’t open your page when someone asks it to. |
| Googlebot | AI search: Google Search & AI Overviews | Your pages drop out of Google Search, AI Overviews included. |
| Google-Extended | Model training: Gemini training and grounding in Gemini Apps (control token, not a crawler) | Your content stays out of Gemini training, and your pages stop being used in Gemini Apps answers. Google Search and AI Overviews are not affected. |
| Bingbot Microsoft | AI search: Bing & Copilot | Your pages can drop out of this company’s AI search answers. |
| OAI-AdsBot OpenAI | Ad review: ChatGPT Ads landing-page review | OpenAI can’t review your ChatGPT ad landing pages, so the ads can’t run. |
Is GPTBot the crawler for ChatGPT Search?
No. OpenAI uses 3 crawlers for different jobs. OAI-SearchBot finds pages for ChatGPT Search, ChatGPT-User opens a page when a person asks ChatGPT to, and GPTBot collects data for training OpenAI’s models. Blocking GPTBot keeps your content out of training without removing it from ChatGPT Search. A checker that tests only GPTBot answers the training question, not the search one.
Does blocking AI training crawlers hurt my AI search visibility?
No. GPTBot, ClaudeBot and Google-Extended, the 3 training crawlers in the table, collect data for future models, so blocking them keeps your content out of training while AI search keeps working. Whether to block them is your choice. The one exception is Google-Extended: blocking it also stops your pages from being used in Gemini Apps answers. Blanc shows training crawlers for your information and never takes points off for blocking them.
Can AI crawlers read pages built with JavaScript?
Mostly no. Most AI crawlers read the HTML your server sends and don’t run scripts. If your text only appears after JavaScript runs, as on many React, Vite and AI-builder apps, those crawlers can receive an almost empty page. The checker reads your page the same way and tells you how much text is in the HTML.
Does Blanc pretend to be an AI crawler?
No. Blanc’s scanner always identifies itself as BlancBot. Your robots.txt rules are read exactly as written, so those results are certain. Firewalls can treat real AI crawlers differently from our scanner, so when one stops the check, the result says what it found and what it can’t prove.
How much does crawler access count in the Blanc score?
25 of the 100 points. Search crawlers carry 17: OAI-SearchBot 5, Googlebot 4, PerplexityBot 4, Claude-SearchBot 2 and Bingbot 2. The 3 on-request crawlers carry 4, and a page that can be indexed carries 4. Training crawlers and OAI-AdsBot carry 0.
Is the AI crawler checker free?
Yes. It is part of the free Blanc audit: up to 3 sites, 1 page per site, with no sign-up, no email and no card. Re-checking a page you’ve already audited is always free.
Where can I learn more?
- What blocks AI crawlers from your site: the 6 most common causes and how to fix each.
- robots.txt for AI crawlers: a template that keeps AI search open and blocks training if you want.
- How AI crawlers find your pages: links, sitemaps and JavaScript.
- ChatGPT Ads landing page check: whether OAI-AdsBot can review an ad landing page.
- About our crawler: how Blanc’s scanner identifies itself and what it fetches.