AI crawler checker

Blanc shows which AI crawlers can reach your page, including the ones behind ChatGPT Search, Claude, Perplexity, Google AI Overviews and Bing Copilot. It reads your robots.txt rules for 12 crawlers, looks for firewalls and noindex tags, and checks whether your text is in the HTML. It’s part of the free Blanc audit: no sign-up, results in a few seconds.

You get the full Blanc audit, the same as on the homepage. For the crawler results, open “Which AI crawlers can read this page” under More details. Anything that blocks a crawler also shows at the top, in Fixes, in order.

What does the AI crawler checker look at?

robots.txt rules
Whether each of the 12 crawlers below may fetch the exact page you enter, read the way crawlers read them (the longest matching rule wins), and whether robots.txt itself loads. A robots.txt that returns a server error tells most crawlers to stay away from the whole site.
Firewalls and bot protection
Signs that Cloudflare, Akamai, Imperva, Sucuri, AWS WAF, DDoS-Guard or a CAPTCHA stopped the check, with what that means and what to change.
noindex and nosnippet
Tags and headers that tell search engines, and the AI answers built on them, not to show or quote the page.
Text without JavaScript
How much of your text is in the HTML your server sends, since most AI crawlers don’t run scripts, and whether your links can be followed without them.

Which AI crawlers does it check?

12 crawlers from 5 companies: OpenAI, Anthropic, Perplexity, Google and Microsoft. 5 find pages for AI search, 3 open a page when a person asks, 3 are for training and 1 reviews ChatGPT ads. Only the search and on-request crawlers decide whether AI search can find and quote you. The one exception is Google-Extended, which also controls Gemini Apps answers.

CrawlerWhat it doesIf you block it
OAI-SearchBot
OpenAI
AI search: ChatGPT SearchYour pages can drop out of this company’s AI search answers.
ChatGPT-User
OpenAI
Opens a page on request: ChatGPT (user-requested)The assistant can’t open your page when someone asks it to.
GPTBot
OpenAI
Model training: OpenAI model trainingYour content stays out of future model training. AI search answers are not affected.
Claude-SearchBot
Anthropic
AI search: Claude searchYour pages can drop out of this company’s AI search answers.
Claude-User
Anthropic
Opens a page on request: Claude (user-requested)The assistant can’t open your page when someone asks it to.
ClaudeBot
Anthropic
Model training: Anthropic model trainingYour content stays out of future model training. AI search answers are not affected.
PerplexityBot
Perplexity
AI search: Perplexity searchYour pages can drop out of this company’s AI search answers.
Perplexity-User
Perplexity
Opens a page on request: Perplexity (user-requested)The assistant can’t open your page when someone asks it to.
Googlebot
Google
AI search: Google Search & AI OverviewsYour pages drop out of Google Search, AI Overviews included.
Google-Extended
Google
Model training: Gemini training and grounding in Gemini Apps (control token, not a crawler)Your content stays out of Gemini training, and your pages stop being used in Gemini Apps answers. Google Search and AI Overviews are not affected.
Bingbot
Microsoft
AI search: Bing & CopilotYour pages can drop out of this company’s AI search answers.
OAI-AdsBot
OpenAI
Ad review: ChatGPT Ads landing-page reviewOpenAI can’t review your ChatGPT ad landing pages, so the ads can’t run.

Is GPTBot the crawler for ChatGPT Search?

No. OpenAI uses 3 crawlers for different jobs. OAI-SearchBot finds pages for ChatGPT Search, ChatGPT-User opens a page when a person asks ChatGPT to, and GPTBot collects data for training OpenAI’s models. Blocking GPTBot keeps your content out of training without removing it from ChatGPT Search. A checker that tests only GPTBot answers the training question, not the search one.

Does blocking AI training crawlers hurt my AI search visibility?

No. GPTBot, ClaudeBot and Google-Extended, the 3 training crawlers in the table, collect data for future models, so blocking them keeps your content out of training while AI search keeps working. Whether to block them is your choice. The one exception is Google-Extended: blocking it also stops your pages from being used in Gemini Apps answers. Blanc shows training crawlers for your information and never takes points off for blocking them.

Can AI crawlers read pages built with JavaScript?

Mostly no. Most AI crawlers read the HTML your server sends and don’t run scripts. If your text only appears after JavaScript runs, as on many React, Vite and AI-builder apps, those crawlers can receive an almost empty page. The checker reads your page the same way and tells you how much text is in the HTML.

Does Blanc pretend to be an AI crawler?

No. Blanc’s scanner always identifies itself as BlancBot. Your robots.txt rules are read exactly as written, so those results are certain. Firewalls can treat real AI crawlers differently from our scanner, so when one stops the check, the result says what it found and what it can’t prove.

How much does crawler access count in the Blanc score?

25 of the 100 points. Search crawlers carry 17: OAI-SearchBot 5, Googlebot 4, PerplexityBot 4, Claude-SearchBot 2 and Bingbot 2. The 3 on-request crawlers carry 4, and a page that can be indexed carries 4. Training crawlers and OAI-AdsBot carry 0.

Is the AI crawler checker free?

Yes. It is part of the free Blanc audit: up to 3 sites, 1 page per site, with no sign-up, no email and no card. Re-checking a page you’ve already audited is always free.

Where can I learn more?