Is your site readable by AI crawlers? A 5-minute check

In about five minutes you can check the three things that decide whether AI crawlers can reach and read your pages: crawler access (robots.txt), rendering (is your content in the HTML?), and discovery (your sitemap). The most common mistake is confusing the two kinds of AI crawler — so we'll start there.

First, the distinction that trips everyone up

AI crawlers come in two kinds. Search and retrieval crawlers fetch your page to answer a user and cite you — these decide your AI visibility. Training crawlers collect content to train future models. Each AI company documents its own crawlers and states they're controlled separately in robots.txt.

Check 1 — Crawler access (robots.txt)

Visit yoursite.com/robots.txt and confirm none of the search crawlers are disallowed. Watch the wildcard trap: if you explicitly allow Googlebot and Bingbot but then set a catch-all User-agent: * with Disallow: /, every unnamed AI search crawler falls through and is blocked.

Check 2 — Rendering (is your content in the HTML?)

Googlebot renders JavaScript, but many AI search crawlers render little or none — so content that only appears after scripts run may be unreadable to them. Test it: right-click and choose View Page Source (not Inspect), then search the source for a sentence from your page. If it's there, you're fine; if the source is mostly an empty <div id="root">, your content is client-rendered and at risk.

Check 3 — Discovery (sitemap, and a note on llms.txt)

A current sitemap.xml at yoursite.com/sitemap.xml helps crawlers find all your pages. On llms.txt: it's an emerging convention, but as of 2026 the major AI engines aren't known to use it for retrieval, so add it if you like but don't rely on it to fix visibility.

What each result means

Once you've run the three checks, here's how to read them:

• Access — pass: no search crawler is disallowed. Fail: a search crawler (or a catch-all) is blocked. Do this: allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot and Googlebot explicitly.
• Rendering — pass: your main text appears in View Page Source. Fail: the source is an empty shell. Do this: server-render or pre-render your primary content so it's in the HTML.
• Discovery — pass: a current sitemap lists your live pages. Fail: no sitemap, or a stale one. Do this: publish an up-to-date sitemap.xml and reference it in robots.txt.

Common mistakes

• The wildcard trap — naming a few good bots, then a catch-all Disallow: / that blocks the rest.
• Blocking by accident at the server — aggressive WAF rules or rate limits that return 429s to crawlers, even though robots.txt allows them.
• JavaScript-only content — the page looks fine in a browser but is empty in the raw HTML.
• Blocking Googlebot — treating it as "an AI bot" and cutting yourself out of Google Search and AI Overviews at the same time.
• Assuming a blocked GPTBot means invisibility — it doesn't; that only affects training, not citations.

A quick reality check

Two honest caveats. Robots.txt is a request, not technical enforcement — it works because reputable crawlers choose to honour it. And being reachable is necessary but not sufficient: once a crawler can reach you, your content still has to be readable and quotable to actually get cited.

Do it automatically

The free Site Check at Sailiently runs all three of these against your URL and reports exactly what, if anything, is blocking AI access — with the evidence behind each result.

Sources

The crawler facts above come from each engine’s own documentation:

• OpenAI — Overview of OpenAI Crawlersofficial source
• Anthropic — Bots & robots.txtofficial source
• Perplexity — Bots guideofficial source
• Google — What is Googlebotofficial source
• Google — Google’s crawlersofficial source

Related: What is GEO? · Making pages quotable