What is this robots.txt checker?
A free online tool to inspect a site's robots.txt. Enter a site root or a robots.txt URL and it fetches the file, parses the rules, and shows the Allow / Disallow directives grouped by User-agent, any declared sitemaps as links, and any crawl-delay. You can also test a specific path (and optional user-agent) to see whether it would be Allowed or Disallowed, using Google-style longest-match rules. Only the URL you enter is fetched by our server — needed to work around the browser's CORS restrictions.
How to use it
- Enter a site root (e.g. https://example.com) or a robots.txt URL. For a site root, we add /robots.txt automatically.
- Press “Check robots.txt” to fetch and parse it.
- Review the rules grouped by User-agent and the declared sitemaps. Optionally, type a path (and user-agent) to test whether it's Allowed or Disallowed.
Common use cases
- Confirming an important page isn't accidentally blocked from crawling.
- Checking whether a specific URL is Allowed or Disallowed for Googlebot.
- Verifying that your sitemap is declared in robots.txt.