Technical SEO

Robots.txt Tester

Test if your robots.txt rules allow or block specific URLs.

How Does robots.txt Work?

robots.txt is a text file placed in the root of your website that tells search engine crawlers which pages they can and cannot access. It uses Allow and Disallow directives to control crawler behavior.

Common Patterns

  • Disallow: /admin/ — Block all pages under /admin/
  • Disallow: /*.pdf$ — Block all PDF files
  • Allow: /admin/public/ — Allow specific subdirectory
  • User-agent: Googlebot — Target specific crawler

FAQ

Does robots.txt prevent indexing?

No. robots.txt only controls crawling, not indexing. If a page is linked from elsewhere, it may still be indexed. Use noindex meta tags to prevent indexing.

Is robots.txt case-sensitive?

The path in Disallow/Allow directives is case-sensitive. /Admin/ is different from /admin/.