What is this AI crawler access checker?
A free online tool that checks whether the major AI crawlers can reach your site. Enter a site URL and it fetches your robots.txt, parses the rules, and reports the Allowed / Blocked status for each AI bot — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended and Meta-ExternalAgent — against your key paths (default "/"). It highlights answer-engine bots (AI search) that look blocked by accident. Only the URL you enter is fetched by our server, needed to work around the browser's CORS restrictions.
Why it matters for AEO / GEO
Answer engines such as ChatGPT, Perplexity and Google AI Overviews can only cite content from sites their crawlers are allowed to fetch. A single over-broad Disallow in robots.txt can silently keep your pages out of AI answers. This checker makes those accidental blocks obvious before they cost you visibility.
How to use it
- Enter a site root (e.g. https://example.com). We add /robots.txt automatically.
- Optionally list your key paths (one per line), such as /blog/ or /products/. Defaults to /.
- Press "Check AI crawlers" to see each bot's Allowed / Blocked status per path.
- Review the warning banner for answer-engine bots that are blocked — those are the most likely mistakes.
Common use cases
- Confirming OAI-SearchBot, PerplexityBot and Google-Extended can reach your content for AI search.
- Verifying that you block training bots (e.g. GPTBot) while still allowing answer-engine bots.
- Auditing a robots.txt change before it accidentally hides key landing pages from AI answers.