Robots.txt Generator

What it does

This generator writes a starter robots.txt file. It opens with a group for all crawlers (User-agent: *) containing the paths you want kept out of the crawl, adds a separate "Disallow: /" group for each AI crawler you tick, and ends with a Sitemap line. You can copy the result or download it as robots.txt. The file is built in your browser; the tool does not read or test your live robots.txt.

How to use it

1. Paste the full address of your XML sitemap into Sitemap URL.
2. Under "Block these crawlers", tick any of GPTBot, ClaudeBot, PerplexityBot, CCBot or Google-Extended that you want to keep out. The ClaudeBot box writes a group for ClaudeBot only, since anthropic-ai and Claude-Web are retired names.
3. List folders or pages to keep all crawlers away from in Additional Disallow Paths, one per line, each starting with a slash. Leave it empty and the file allows everything.
4. Click "Generate robots.txt", then Copy or Download and upload the file to the root of your host.

What the rules actually do

Disallow stops compliant crawlers from fetching a URL. It does not remove the URL from search results, and a blocked page can still be indexed from links pointing to it, so use noindex or authentication for pages that must stay out. The tool also passes your paths through unchecked: a path without a leading slash, or a typo, goes into the file as typed.

The AI checkboxes cover a short list. Many vendors run separate agents for search features and user-requested fetches (OpenAI's OAI-SearchBot and ChatGPT-User, for example), and blocking a training crawler does not block those. Google-Extended controls whether Google may use your content for Gemini models; it does not affect Googlebot or Google Search. Check each vendor's documentation for the current names before relying on the list.

Related guide: The Complete robots.txt Reference: Precedence, Wildcards, AI Bots & Real-World Receipts