robots.txt is one of the oldest, plainest files on the web — a handful of lines at yoursite.com/robots.txt that tell automated crawlers what they can and can't fetch. For twenty years, it mattered mainly for Googlebot and Bingbot. In 2026, the same file also decides whether ChatGPT, Claude, Perplexity, and a growing list of AI systems can read your site at all.
That's the part most businesses miss. Nobody sets out to block AI crawlers on purpose. It usually happens by accident: a blanket Disallow: / left over from a staging environment, a security plugin or WAF that treats any "bot" as a threat, or a wildcard rule written years ago to keep out spam scrapers that also happens to catch every AI crawler by name. The file sits there quietly doing its job — which, unintentionally, is keeping AI systems out.
What robots.txt actually controls
robots.txt doesn't hide a page from people, and it isn't a security measure — it's an instruction that well-behaved crawlers are expected to follow voluntarily. Search engines, and now most major AI companies, publish named crawlers that check this file before reading anything else on your domain. If a crawler's name appears next to a Disallow: / rule, it's expected to leave without reading a single page.
The AI crawlers worth knowing by name
Each major AI system uses its own named crawler, and each one needs to be allowed individually — allowing Googlebot doesn't automatically allow Google's AI systems, and allowing one AI crawler doesn't allow the others.
| Crawler | Operated by | What it's used for |
|---|---|---|
GPTBot | OpenAI | Training and browsing for ChatGPT |
ChatGPT-User | OpenAI | Live page reads when a user asks ChatGPT to check a specific site |
ClaudeBot | Anthropic | Training and retrieval for Claude |
PerplexityBot | Perplexity | Real-time answers and citations in Perplexity search |
Google-Extended | Gemini and AI Overviews specifically — separate from ordinary Googlebot indexing | |
CCBot | Common Crawl | The open dataset a large share of AI models are trained on |
Applebot-Extended | Apple | Apple Intelligence |
Amazonbot | Amazon | Alexa and Amazon's AI shopping features |
Meta-ExternalAgent | Meta | Meta AI training and retrieval |
Bytespider | ByteDance | ByteDance's AI systems |
A working example: InfinitusNow's own robots.txt
Rather than describe this in the abstract, here's our own configuration, live at infinitusnow.com/robots.txt. It explicitly allows GPTBot, ClaudeBot, PerplexityBot, CCBot, Applebot-Extended, Bytespider, Amazonbot, Google-Extended, and several other AI crawlers to read the site, while disallowing crawler access to private application routes — the client dashboard, admin tools, login, and checkout pages — that were never meant to be public in the first place.
That second half matters as much as the first. Allowing AI crawlers isn't the same as allowing them everywhere. A correctly configured robots.txt opens the marketing site, service pages, and articles that are meant to be found, while keeping account and billing routes closed — the same logic you'd already apply to Googlebot, just extended to the newer set of crawler names.
How to check your own site in two minutes
- Open
yoursite.com/robots.txtdirectly in a browser. If nothing loads, there's no file at all — most platforms allow everything by default in that case, but it's worth confirming rather than assuming. - Look for any line reading
Disallow: /sitting underUser-agent: *— that blocks every crawler, AI or otherwise, from the entire site. - Look for the specific AI crawler names above. If they appear under a
Disallow: /rule anywhere in the file, that crawler is blocked, even ifUser-agent: *looks fine. - Check whether a security plugin, CDN, or WAF is blocking these crawlers at the server level instead of in robots.txt — a clean robots.txt doesn't help if the hosting layer is silently returning errors to GPTBot or ClaudeBot before the request ever reaches your site.
Our free AI visibility audit checks robots.txt AI crawler access automatically, alongside title/H1 structure, schema markup, and llms.txt presence, and flags exactly which crawlers — if any — are being turned away.