Robots.txt Generator
Build a robots.txt file from structured user-agent, Allow, Disallow, and Sitemap rule groups, or validate one you already have — entirely locally, no live site is fetched.
Crawl-delay support is crawler-dependent — some crawlers (e.g. Bing, Yandex) honor it, but Googlebot does not use the Crawl-delay directive at all. To control Google's crawl rate, use the crawl-rate setting in Google Search Console instead.
robots.txt is a request to well-behaved crawlers about which paths they should not request — it is not an access-control or privacy mechanism. The file itself is public, any crawler can choose to ignore it, and a "Disallow" line does not remove a URL that's already indexed by a search engine (use a noindex meta tag, or the search engine's removal tools, for that). Never rely on this to keep something private.
Generated and validated entirely in your browser — nothing you enter here is sent to a server, and no live website is fetched or checked.
Build a robots.txt file from structured rule groups — user-agent, Allow, Disallow, and an optional crawl-delay per group — plus one or more Sitemap URLs, instead of hand-writing the syntax in a plain textarea. Switch to "Validate existing robots.txt" to check a file you already have for common syntax problems, entirely locally.
This tool does not fetch any live website's robots.txt — there is no remote checker here — and it does not claim to simulate exactly how Googlebot or any other specific crawler will interpret the file, only whether the syntax itself is well-formed.