Guide
How AI crawlers find your pages, and the ones they never reach
AI crawlers find your pages in two ways: they follow links written in the HTML of pages they already know, and they read the pages listed in your sitemap. Most of them don't run JavaScript, so a link that only appears after your scripts run leads nowhere for them. A page with no HTML link pointing to it and no place in your sitemap is never found at all.
This guide shows how discovery works, what we saw AI crawlers do on our own site in its first days, and how to check which of your pages they can reach. If a crawler finds a page but still can't read it, start with what blocks AI crawlers instead.
How does a crawler find a page it has never seen?
A crawler finds a new page through a link from a page it already knows, or through a list of pages you give it. Google calls this step "URL discovery" and describes three sources: pages it has visited before, links it extracts from known pages, and sitemaps that site owners submit.
OpenAI and Anthropic document what their crawlers are for and how to allow or block them in robots.txt. Neither describes how their crawlers discover pages. What every crawler has to work with is the same, though: the HTML your server sends, your robots.txt and the sitemap it points to.
What does a link need for a crawler to follow it?
A link needs to be an <a> element with an href attribute, written in the HTML. Google says it can generally only crawl a link in that form, and that it can't reliably extract addresses from <a> tags without href or from other elements that act as links through script events.
<!-- A crawler can follow this -->
<a href="/pricing">Pricing</a>
<!-- These lead nowhere for a crawler -->
<a onclick="goTo('/pricing')">Pricing</a>
<button onclick="location.href='/pricing'">Pricing</button>
<span class="link" data-to="/pricing">Pricing</span>
All four look the same to a person with a mouse. Only the first one is a link to a crawler.
Why do JavaScript links matter more for AI crawlers than for Google?
Google renders pages with a recent version of Chrome and runs their JavaScript, so it can see links that a script adds after the page loads. Most AI crawlers skip that step. A December 2024 study by Vercel and MERJ found that none of the major AI crawlers it measured rendered JavaScript, including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot.
The same study found that these crawlers do download JavaScript files (11.50% of ChatGPT's requests and 23.84% of Claude's) without running them. So a menu that a React or Vue app builds in the browser can be fetched as code and still give the crawler no links to follow.
The same applies to anything that loads on demand: "Load more" buttons, infinite scroll, tabs filled in by a script, and results that only appear after someone types in a search box. If the address of a page only exists after a click or a script, a crawler that reads HTML has no way to reach it.
What did AI crawlers do on our own site in its first days?
Within two days of launch, crawlers from Google, Anthropic and OpenAI were visiting every day, and none of their requests failed. heyblanc.com went live on November 4, 2026. Cloudflare's AI Crawl Control dashboard showed these totals for two 24-hour windows, read on the mornings of November 6 and November 7 (Manila time):
| 24 hours ending | Nov 6 | Nov 7 |
|---|---|---|
| Requests from crawlers Cloudflare lists as AI crawlers | 57 | 67 |
| Requests allowed | 57 | 67 |
| Unsuccessful requests | 0 | 0 |
| Answered with HTTP 200 | 56 | 57 |
| Most requested address | /sitemap.xml (17) | / (16) |
| Google (Googlebot and one more) | 22 | 43 |
| Anthropic (ClaudeBot and two more) | 30 | 19 |
| OpenAI (GPTBot and two more) | 2 | 2 |
Three things stood out to us.
- The sitemap was the busiest address on day two. On November 6, sitemap.xml got more successful requests than any page on the site, 17 in 24 hours. Our robots.txt names it on its last line.
- Anthropic's crawlers asked for a lot of images. The November 6 summary says they downloaded 365.88 kB of images. That matches the Vercel and MERJ study, which found images were 35.17% of Claude's fetches. Your site icon is one image worth keeping reachable: AI answers show it next to your name, as our guide to your favicon in AI answers explains.
- OpenAI visited rarely. Two requests a day, both allowed. A new site with few links from other sites gives crawlers few ways in, so links from other sites are what we are working on next.
The remaining 10 requests on November 7 weren't errors, and the summary doesn't show which status they received. Cloudflare's list also includes Googlebot, which serves Google Search as well as Google's AI features. These numbers cover two days on one small site. They show that crawlers found and fetched our pages. They say nothing about how often any AI tool will quote us.
Which pages do crawlers never reach?
Crawlers never reach pages that nothing points to. The common cases:
- Pages linked only from a JavaScript menu or from buttons, as in the examples above.
- Pages linked only from ads, email or social posts. A campaign landing page can get plenty of visitors and still have no link on the site itself. If it isn't in the sitemap either, crawlers have no route to it.
- Old pages that lost their last link after a redesign, while still being live.
- Pages behind a form. Search results, filtered lists and anything that needs a login.
- Pages your robots.txt blocks. The crawler knows the address and doesn't fetch it. Our robots.txt guide for AI crawlers shows how to check the rules for each crawler.
Two related problems look like discovery working when it isn't. On a single-page app that serves the same HTML at every address, a crawler can follow every link and still read your homepage each time. Our scanner saw exactly this on NearMe Kids PH, our directory of kid-friendly places in Metro Manila, until we pre-rendered all 144 routes. Its host, Lovable, serves verified crawlers a rendered version of each route, so real AI crawlers were most likely spared, but every tool that reads the raw HTML wasn't. The second is the soft 404: an address that doesn't exist answers with HTTP 200 and the homepage. NearMe Kids PH still does this for made-up addresses, and we decided to leave it, since a real 404 there would also break routes that aren't pre-rendered yet. A crawler can then treat endless copies of one page as separate pages.
How do I check which of my pages crawlers can reach?
Most of these checks take a few minutes and need only a browser.
- Read your links in the page source. Open a page, choose View page source and search for
href="/. Every page you care about should appear in at least one link on a page that is already reachable, starting with your homepage menu. - Find your sitemap. Open
yoursite.com/robots.txtand look for aSitemap:line. If there isn't one, tryyoursite.com/sitemap.xml. - Compare the sitemap with your real pages. Every page you want found should be listed, including landing pages that the site itself doesn't link to.
- Test a made-up address. Open
yoursite.com/this-page-does-not-exist. It should show a 404 page. If your homepage appears instead, you have a soft 404. - Run an audit. The free Blanc audit flags links a crawler can't follow, reads your robots.txt and sitemap, and on single-page apps checks whether other pages send their own content and whether a made-up address returns your homepage.
- Look at real crawler traffic. If your site runs through Cloudflare, the AI Crawl Control dashboard shows which crawlers visited, the addresses they requested and the status codes they got.
Does being found mean AI tools will quote a page?
No. Being found means a crawler can fetch the page. Whether any AI tool quotes it or sends you visitors is a separate question. What happens next depends on whether the crawler can read the page's text and facts, and on whether the page answers the questions people ask. A page that can't be found doesn't get that chance, which is why discovery is the first thing to check.
Questions
Do AI crawlers use my sitemap?
They can. OpenAI and Anthropic don't publish how their crawlers discover pages, but on heyblanc.com sitemap.xml was the single most requested address in one 24-hour window, with 17 successful requests. List every page you want found, and add a Sitemap: line to robots.txt.
Will Google find links that AI crawlers miss?
Often, yes. Google says it renders pages with a recent version of Chrome and runs their JavaScript. A 2024 study by Vercel and MERJ found that none of the major AI crawlers it measured rendered JavaScript, so a link created by a script can reach Google and still be invisible to them.
If every page is linked, do I still need a sitemap?
It still helps. Google says a well-linked site can usually be discovered without one, and that a sitemap can improve crawling of larger or more complex sites. It costs little, and it is the one route that does not depend on your links.
Sources
- Google Search Central: In-depth guide to how Google Search works (last updated December 18, 2025)
- Google Search Central: SEO link best practices, make your links crawlable (last updated December 10, 2025)
- Google Search Central: What is a sitemap (last updated December 10, 2025)
- The rise of the AI crawler, Vercel and MERJ, December 2024
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
Checked November 12, 2026. Tell us at hello@heyblanc.com if something has changed.