Robots.txt Tester
Test whether a crawler would be allowed to fetch a given path under a robots.txt, using Google's own documented precedence algorithm.
The Robots.txt Generator builds and syntax-checks a file. This tool answers a different question: given a robots.txt, would a specific crawler be allowed to fetch a specific path? It implements Google's own publicly documented precedence algorithm — the group whose User-agent token matches the crawler (ignoring case and any version, so Googlebot/2.1 counts as Googlebot) wins over the * fallback group, and Googlebot-Image, -News and -Video use the Googlebot group when they have none of their own; among matching Allow/Disallow rules, the longest rule wins, and a tie between Allow and Disallow is won by Allow. It does not fetch any live robots.txt, and it is not a complete simulation of Googlebot — crawl-delay, sitemap discovery, and any non-standard extension are outside its scope.
Tested entirely in your browser — nothing you enter here is sent to a server, and no live site is checked.
Paste a robots.txt, pick a crawler name and a URL path, and see whether that path would be allowed or disallowed — implementing Google's own documented precedence rules (longest matching rule wins; an Allow/Disallow tie is won by Allow).
This does not fetch any live robots.txt and is not a complete Googlebot simulation — crawl-delay, sitemap discovery, and non-standard extensions aren't evaluated.
How to use the Robots.txt Tester
- Paste your file into "robots.txt content" (a short sample is pre-filled).
- Enter the crawler name in "User-agent (crawler name)", such as Googlebot or Bingbot.
- Enter a site-relative path starting with / in "URL path to test", such as /blog/post?page=2.
- Click Test to see ALLOWED or DISALLOWED and the rule that decided it.
Example: A crawler-specific group replaces the * group
robots.txt with "User-agent: *" / "Disallow: /search" and a separate "User-agent: Bingbot" / "Disallow: /private" group. Testing Googlebot on /search?q=x returns DISALLOWED: Longest matching rule in the "*" group (no group for "Googlebot") is "Disallow: /search" (7 characters). Testing bingbot on the same path returns ALLOWED, because Bingbot uses only its own group, which has no rule for /search.
Questions about the Robots.txt Tester
- Can I paste a full browser user-agent string?
- No. Enter the crawler's robots.txt token, such as bingbot; a full string like Mozilla/5.0 (compatible; bingbot/2.0) gets an error asking for the token. Matching ignores case and any version, so a User-agent: Googlebot/2.1 line counts as Googlebot.
- Does Googlebot-Image follow the Googlebot rules?
- Yes, when the file has no Googlebot-Image group: Googlebot-Image, Googlebot-News and Googlebot-Video fall back to a Googlebot group, as Google documents, and the result says which group was used. Other crawlers fall back to the * group.
- Can I test a full URL like https://example.com/page?
- No. The path must start with /, otherwise you see an error asking for a site-relative path. Query strings are kept and matched; anything after # is ignored, because fragments are not sent to the server.
- Are paths case-sensitive?
- Yes. Disallow: /Admin does not block /admin. The User-agent name is the only part matched without regard to case.
- How are * and $ handled?
- * matches any run of characters and a trailing $ anchors the rule to the end of the path. Disallow: /*.pdf$ blocks /files/a.pdf but not /files/a.pdf?dl=1.