robots.txt generator

How to use it

1. Start from a preset or edit the default group: set the user-agent (use “*” for all crawlers), then add the paths you want to allow or disallow. The file updates as you type — there is no Generate button.
2. Add another group when different crawlers need different rules, for example allow everything for “*” and block AI crawlers such as GPTBot and CCBot. A Sitemap URL and a Crawl-delay are optional.
3. Copy the result or download “robots.txt”, then upload it to the site root (https://example.com/robots.txt). Each subdomain needs its own file. Verify the behaviour in Search Console or another robots.txt tester before relying on it.

Example

Group 1: * — Disallow /admin/
Group 2: GPTBot — Disallow /
Sitemap: https://example.com/sitemap.xml

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The groups are independent: “*” only blocks /admin/, while GPTBot is blocked from the whole site. The Sitemap line is not part of either group. The count reads “Groups: 2 · Rules: 2”.

Rules, limits and known issues

• robots.txt is a crawler convention, not access control: it cannot protect private pages. Use authentication or server rules for anything that must stay private.
• Google ignores Crawl-delay (it is not defined by RFC 9309); some other crawlers honor it. The field is optional and off by default, so nothing is added unless you type a value.
• The AI preset is a curated list of common training crawlers — GPTBot, ChatGPT-User, Google-Extended, CCBot, ClaudeBot, PerplexityBot, Bytespider and Applebot-Extended. Tokens change over time; check each crawler’s documentation and edit the groups as needed.
• Paths are case-sensitive and support “*” and “$”. The longest matching rule wins, not the first one written, so an Allow must be more specific than the Disallow it overrides.
• Non-ASCII paths are written as UTF-8. Older crawlers may require percent-encoded URLs, so the tool warns but never rewrites what you typed.
• Only one Sitemap URL is supported. To add more, copy the result and append extra “Sitemap:” lines. The tool does not crawl your site or check whether a URL is actually blocked — use an external tester for that.
• Nothing is uploaded or stored: the rules live only in this page. Presets replace the current groups but keep the Sitemap and Crawl-delay; Reset clears everything. There is no account, history or shareable configuration.

Related tools

What people usually need next, once the file is ready.