robots.txt for AI crawlers: which bots to allow, and which you can block

If you want to appear in AI answers, allow the search crawlers and the crawlers that fetch a page when someone asks about it. Training crawlers are a separate decision: blocking them keeps your content out of future models without removing you from AI search. The template below does both, and it fits on one screen.

Three kinds of AI crawler

Two tokens need a note:

• Google-Extended is a control token, not a crawler. Blocking it keeps your content out of Gemini training and out of answers Gemini Apps ground in web sources. It does not affect Google Search or AI Overviews, which follow Googlebot.
• OAI-AdsBot only visits landing pages submitted as ChatGPT ads. If you run ChatGPT ads, it must be allowed. See our OAI-AdsBot checklist.

A template you can paste

This version allows AI search and user-requested fetches, blocks training, and keeps a private folder private for everyone:

# AI search and user-requested fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin/

# Model training: blocked (a business choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

Change /admin/ to your own private paths, or remove those lines if you have none. If you are happy for your content to be used in training, delete the middle group.

Four matching rules that trip people up

1. A named group replaces the * group. A crawler that finds a group naming it ignores everything under User-agent: *. That is why the template repeats Disallow: /admin/ in the first group. Forgetting it is the most common way private folders end up open to AI crawlers.
2. Several User-agent lines can share one group. The rules below them apply to every crawler listed.
3. The longest matching path wins. Allow: /blog/ beats Disallow: / for pages in /blog/. When an Allow and a Disallow are equally long, Allow wins.
4. * and $ are wildcards. Disallow: /*.pdf$ blocks every URL ending in .pdf.

Mistakes to check for

• User-agent: * with Disallow: /. Left over from a staging site, this blocks every crawler, AI and Google alike.
• A plugin or CDN adding its own blocks. Some security tools block AI bots at the firewall, which robots.txt cannot override. Check both.
• A robots.txt that returns an error. A 5xx response or timeout tells crawlers to treat the whole site as blocked.
• Treating robots.txt as security. It is a public, voluntary request. OpenAI also notes that robots.txt rules may not apply to ChatGPT-User, because a person asked for the page.

robots.txt is one of six common blockers. The others, from firewalls to JavaScript-only pages, are in what blocks AI crawlers from your site.

Check yours

Open yoursite.com/robots.txt and compare it with the template, or run a free audit to see which of the 12 crawlers can reach a specific page, with the exact rule that decided it.