AI Crawler Checker & Simulator.

Simulate and verify how AI citation bots (GPTBot, ClaudeBot, PerplexityBot) and search engines interpret your `robots.txt` governance directives.

Input robots.txt Content:

Presets:

Test Request URL Path:

Access Verdict on Path: /services/ai-visibility

RFC 9309 Prefix Matching

GPTBot(OpenAI)

ALLOWED

Used by OpenAI for web indexing, ChatGPT Search answers, and model improvements.

Purpose: AI Search & CitationRule: allow '/' (Line 10)

ChatGPT-User(OpenAI)

ALLOWED

Triggered in real-time when a user asks ChatGPT to browse a specific URL directly.

Purpose: AI Search & CitationRule: allow '/' (Line 13)

ClaudeBot(Anthropic)

ALLOWED

Anthropic web crawler used for Claude search grounding and web browsing.

Purpose: AI Search & CitationRule: allow '/' (Line 16)

PerplexityBot(Perplexity AI)

ALLOWED

Web crawler used by Perplexity to index pages and provide cited answers.

Purpose: AI Search & CitationRule: allow '/' (Line 19)

Google-Extended(Google)

DISALLOWED

Separate token controlling Gemini training without affecting Google Search ranking.

Purpose: AI Model TrainingRule: disallow '/' (Line 23)

Googlebot(Google)

ALLOWED

Core Google search crawler required for organic Google rankings and AI Overviews.

Purpose: Web Search EngineRule: allow '/' (Line 3)

Applebot-Extended(Apple)

DISALLOWED

Apple Intelligence training crawler token.

Purpose: AI Model TrainingRule: disallow '/' (Line 26)

CCBot(Common Crawl)

DISALLOWED

Massive open dataset crawler frequently ingested into foundation AI training sets.

Purpose: Data AggregationRule: disallow '/' (Line 29)

Bytespider(ByteDance)

DISALLOWED

High-frequency crawler used by ByteDance for search and model training.

Purpose: Data AggregationRule: disallow '/' (Line 32)

AI Crawler Governance: Strategic Guidelines for Singapore Businesses

Blocking all AI scrapers prevents commercial answer engines (ChatGPT Search, Perplexity) from learning about your business, while allowing uncontrolled scrapers may expose private content. Infinitus recommends a Balanced Commercial Policy: allow AI citation bots (`GPTBot`, `ClaudeBot`, `PerplexityBot`) on public service pages, while restricting sensitive endpoints (`/api/`, `/admin/`) and blocking non-attribution training harvesters.

Read our in-depth study on robots.txt and AI visibility