Robots.txt Tester
Test whether a crawler would be allowed to fetch a given path under a robots.txt, using Google's own documented precedence algorithm.
The Robots.txt Generator builds and syntax-checks a file. This tool answers a different question: given a robots.txt, would a specific crawler be allowed to fetch a specific path? It implements Google's own publicly documented precedence algorithm — the group whose User-agent exactly matches wins over the * fallback group; among matching Allow/Disallow rules, the longest rule wins, and a tie between Allow and Disallow is won by Allow. It does not fetch any live robots.txt, and it is not a complete simulation of Googlebot — crawl-delay, sitemap discovery, and any non-standard extension are outside its scope.
Tested entirely in your browser — nothing you enter here is sent to a server, and no live site is checked.
Paste a robots.txt, pick a crawler name and a URL path, and see whether that path would be allowed or disallowed — implementing Google's own documented precedence rules (longest matching rule wins; an Allow/Disallow tie is won by Allow).
This does not fetch any live robots.txt and is not a complete Googlebot simulation — crawl-delay, sitemap discovery, and non-standard extensions aren't evaluated.