Robots.txt Generator
Build a robots.txt file from structured user-agent, Allow, Disallow, and Sitemap rule groups, or validate one you already have — entirely locally, no live site is fetched.
Crawl-delay support is crawler-dependent — some crawlers (e.g. Bing, Yandex) honor it, but Googlebot does not use the Crawl-delay directive at all. To control Google's crawl rate, use the crawl-rate setting in Google Search Console instead.
robots.txt is a request to well-behaved crawlers about which paths they should not request — it is not an access-control or privacy mechanism. The file itself is public, any crawler can choose to ignore it, and a "Disallow" line does not remove a URL that's already indexed by a search engine (use a noindex meta tag, or the search engine's removal tools, for that). Never rely on this to keep something private.
Generated and validated entirely in your browser — nothing you enter here is sent to a server, and no live website is fetched or checked.
Build a robots.txt file from structured rule groups — user-agent, Allow, Disallow, and an optional crawl-delay per group — plus one or more Sitemap URLs, instead of hand-writing the syntax in a plain textarea. Switch to "Validate existing robots.txt" to check a file you already have for common syntax problems, entirely locally.
This tool does not fetch any live website's robots.txt — there is no remote checker here — and it does not claim to simulate exactly how Googlebot or any other specific crawler will interpret the file, only whether the syntax itself is well-formed.
How to use the Robots.txt Generator
- In Generate mode, set the User-agent for Rule group 1 (* means all crawlers).
- Add Disallow and Allow paths, each starting with / or *; use Add Disallow line or Add Allow line for more.
- Optionally add another rule group and one or more Sitemap URLs.
- Click Generate, review any warnings, then Copy or Download .txt.
- To check an existing file, switch to Validate existing robots.txt, paste it, and click Validate.
Example: Block admin and cart, keep a help page open
With User-agent *, Disallow /admin/ and /cart, Allow /admin/help, and Sitemap https://example.com/sitemap.xml, Generate outputs: User-agent: *, Allow: /admin/help, Disallow: /admin/, Disallow: /cart, a blank line, then Sitemap: https://example.com/sitemap.xml.
Questions about the Robots.txt Generator
- Why does the default output contain only User-agent: *?
- Empty Disallow fields are skipped, so a group with no paths produces only its User-agent line plus a warning that it has no effect. A group with no rules places no restrictions on that crawler.
- Why are Allow lines written before Disallow lines?
- The generator always writes Allow lines first in each group. Under RFC 9309, crawlers pick the most specific (longest) matching rule, not the first one, so this order does not change the result.
- What does the validator flag?
- It reports errors for lines without a colon, Allow or Disallow before any User-agent, a non-numeric Crawl-delay, and Sitemap values that are not absolute http or https URLs. Unknown directives such as Noindex get a warning, each with its line number.
- Can I write a relative sitemap path like /sitemap.xml?
- No. Sitemap entries must be absolute URLs such as https://example.com/sitemap.xml, or Generate shows an error.