Skip to content
Toolzen
  • Image CompressorImage tools
  • Image ResizerImage tools
  • Image ConverterImage tools
  • Favicon GeneratorImage tools
  • Merge PDFPDF tools
  • Split PDFPDF tools
  • Remove PDF PagesPDF tools
  • Images to PDFPDF tools

Robots.txt Tester

Fetch a site's robots.txt and test whether Googlebot and other crawlers may crawl a URL.

Loading tool…

How to use the Robots.txt Tester

  1. 1

    Enter a website address to fetch its robots.txt.

  2. 2

    Choose a crawler and type a path or URL.

  3. 3

    See whether it is allowed and which rule decides it.

Frequently asked questions

Is my request private?

Our server sends a request to the site you enter and shows you what came back. Nothing is stored, and the site sees our server, not you.

Which rules are applied?

Google's rules: the crawler uses its most specific user-agent group, the longest matching rule wins, Allow wins a tie, and * and $ wildcards are supported.