What blocks AI crawlers from your site, and how to check

Six things most often stop AI assistants from reading a page: a robots.txt rule, a firewall or bot-protection setting, text that only appears after JavaScript runs, a noindex or nosnippet tag, a robots.txt file that returns an error, and a login wall or CAPTCHA. None of them show up for human visitors, which is why they go unnoticed for months.

The fastest way to check all six is to run a free audit on the page. If you would rather check by hand, each section below explains how. Running ChatGPT ads? Landing pages have extra requirements, covered in our OAI-AdsBot checklist.

The crawlers that matter

AI companies send different crawlers for different jobs, and blocking each one has a different effect.

The pattern to remember: search and user crawlers decide whether you can appear in AI answers. Training crawlers decide whether your content trains future models, and blocking them is a business choice. Google-Extended is the exception worth knowing: one rule covers both Gemini training and Gemini Apps answers that cite the web, so blocking it also removes you from those answers, though not from Google Search or AI Overviews.

1. A robots.txt rule

robots.txt sits at the root of your domain and tells crawlers which paths they may visit. One old line, or one added by a security plugin, can shut out a crawler for the whole site:

User-agent: *
Disallow: /

Two details trip people up. A crawler named in its own group ignores the rules under User-agent: *. And the most specific matching rule wins, so Allow: /blog/ can override Disallow: / for that folder.

How to check: open yoursite.com/robots.txt and look for groups that name the crawlers above, then for a Disallow that covers your page. Our robots.txt guide for AI crawlers has a full template. To allow ChatGPT Search explicitly:

User-agent: OAI-SearchBot
Allow: /

If you block private folders for everyone, copy those Disallow lines into this group too.

2. A firewall or bot-protection setting

CDNs and security plugins can block AI crawlers before robots.txt is even read. Some offer a single switch to block AI bots, and it is easy to turn on without realising it also blocks search crawlers.

How to check: in your CDN or security plugin, look for bot management, AI crawler controls and rate limits. Your server logs will show whether crawlers such as OAI-SearchBot receive 403 or 429 responses.

3. Text that only appears after JavaScript

Google renders JavaScript before reading a page. Most AI crawlers read the HTML your server sends and stop there. A page that looks complete in a browser, and ranks well on Google, can look almost empty to them.

How to check: open the page, choose View page source, and search for a sentence from your main content. If it is missing, turn on server-side rendering or static pre-rendering for that page.

For a worked example, read how we fixed a React and Vite directory that sent crawlers 9 words.

4. A noindex or nosnippet tag

noindex asks search engines to leave the page out. nosnippet stops them from quoting it, and Google documents that it also keeps text out of AI Overviews. Both can sit in a meta tag or in an X-Robots-Tag response header, where they are easy to miss.

How to check: search the page source for noindex and nosnippet, and ask your developer whether the server sends an X-Robots-Tag header.

5. A robots.txt that returns an error

Under the robots.txt standard, a missing file (404) means everything is allowed. A server error (5xx) or a timeout means crawlers treat the whole site as blocked until the file loads again.

How to check: open yoursite.com/robots.txt. It should load quickly with status 200, or return 404 if you have no rules.

6. A login wall, CAPTCHA or very slow page

Crawlers cannot sign in or solve CAPTCHAs. Pages that respond slowly may be dropped before they finish loading.

How to check: open the page in a private window. If you see a sign-in screen or a human-verification check before the content, crawlers see it too.

What is not a blocker

Two things are often presented as problems when they are not:

• Blocking training crawlers. GPTBot and ClaudeBot affect model training, not AI search. Google-Extended also covers Gemini Apps answers, so treat it as a choice about Gemini, not about Google Search.
• A missing llms.txt. It is a proposed format that no major AI engine has confirmed it uses. It is cheap to add, but its absence blocks nothing.

Check a page in five minutes

1. Open yoursite.com/robots.txt and read every group that names an AI crawler or uses *.
2. View the page source and confirm the main text is there.
3. Search the source for noindex and nosnippet.
4. Open the page in a private window to check for sign-in or CAPTCHA screens.
5. Ask whoever manages your CDN or security plugin whether AI crawlers are blocked.

Once nothing blocks your page, the next step is making its facts easy to quote: see LocalBusiness schema for AI answers and how AEO differs from SEO. Removing these blockers makes a page readable to AI assistants. Whether they cite it also depends on what other sites, reviews and roundups say about you, which no on-site fix can control.