What does the robots.txt Validator check?

It checks for syntax errors, accidental Disallow: / rules that block all crawlers, missing Sitemap directives, crawl-delay values over 10 seconds, unknown directives, and AI bot coverage — GPTBot, ClaudeBot, PerplexityBot, Bytespider, and Google-Extended.

Does this tool send my robots.txt content anywhere?

In URL mode the tool fetches your live robots.txt from the public URL. In Paste mode all analysis happens in your browser. No data is stored or transmitted beyond the initial URL fetch.

What is the AI bot coverage check?

The tool checks whether your robots.txt explicitly addresses major AI training crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Bytespider (ByteDance), and Google-Extended. Explicit Allow directives improve GEO (Generative Engine Optimisation) — the likelihood your content is cited in AI search answers.

What does the robots.txt health score mean?

The health score (0-100) summarises all checks. 100 means no issues detected. Points are deducted for missing sitemap reference, crawl-delay over 10 seconds, conflicting Allow and Disallow rules, missing AI bot directives, and syntax errors.

robots.txt Validator — Audit Crawl Directives & AI Bot Coverage

ConfigClarity robots.txt Validator is a free browser-based tool that audits robots.txt files for crawl directive errors, AI bot coverage gaps, and sitemap health issues. It checks whether GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, PerplexityBot, and Bytespider are explicitly addressed. It flags Crawl-delay values above 10 seconds, missing sitemap declarations, Allow and Disallow conflicts, unknown directives, and wildcard path patterns. Results include a health score from 0 to 100 and copy-paste fix snippets for each issue. No signup required. All processing runs in the browser. Nothing is stored or transmitted.

🕐 Cron Builder & Visualiser Visualize job overlaps 🔒 SSL Checker Check cert expiry 🐳 Docker Auditor Audit compose files 🛡️ Firewall Auditor Audit UFW rules 🔀 Reverse Proxy Mapper Map routing & audit 🤖 robots.txt Audit crawl & AI bots

Enter a URL or paste your robots.txt, then run the audit.

Frequently Asked Questions

Does this fetch my robots.txt live?

Yes in URL mode. Nothing is stored — fetch happens in your browser.

Why does AI bot coverage matter?

GPTBot, ClaudeBot, and CCBot index content for AI training. If you haven't explicitly addressed them, they may crawl freely.

Does blocking GPTBot actually stop AI training?

It signals your preference to OpenAI's crawler. GPTBot has stated it respects robots.txt exclusions for training data. ClaudeBot and Google-Extended also honour explicit blocks. CCBot compliance is less consistent. Blocking in robots.txt is the standard opt-out mechanism but cannot be technically enforced.

What's the file size limit?

Google recommends robots.txt stay under 500KB. Larger files may be partially ignored.

Does Disallow: / block everything?

For User-agent: * yes. But individual bots can have their own rules that override the wildcard.

What are unknown directives?

Lines that aren't part of the robots.txt standard (RFC 9309). Some like Host: are common but not universally supported.

robots.txt Validator

Checks

AI Crawler Coverage

Frequently Asked Questions