Robots.txt Generator

robots.txt tells well-behaved crawlers which parts of your site they may request. It is not a security feature — blocked URLs can still be visited and can even appear in search results if other pages link to them. To keep a page out of results, use a noindex tag instead.

Preset

Sitemap URL

Disallow paths (one per line)

Allow paths (exceptions)

Also block common AI-training crawlers (GPTBot, CCBot, Google-Extended, ClaudeBot, Applebot-Extended, PerplexityBot, Bytespider)

robots.txt

Test a path

Path to test

Runs entirely in your browser. Your input and files are not uploaded to any server.

How to use

Pick a starting preset.
List the paths to disallow or allow, one per line, and add your sitemap URL.
Test a few paths, then download robots.txt and upload it to the root of your domain.

Examples

Block an admin area

Disallow /admin/ while leaving the rest of the site crawlable.

Opt out of AI training crawlers

Add rules for user agents such as GPTBot and CCBot that publish opt-out instructions.

Privacy

Your files and text stay in your browser while you use this tool, and ads or anonymous visit statistics do not receive them. Only your favorites and recent tools are saved on your device. Privacy policy

Want proof? Check it with your browser’s Network tab: How to check that a tool does not upload your files

Frequently asked questions

Where does robots.txt go?

At the root of the host: https://example.com/robots.txt. Each subdomain needs its own file.

Does Disallow remove a page from Google?

No. It stops crawling, not indexing. Use a noindex meta tag (and allow crawling) to remove a page from results.

How does the tester decide?

It follows the Robots Exclusion Protocol (RFC 9309): the longest matching rule wins, Allow wins a tie, and * and $ wildcards are supported. It tests the group for “*”.

Related tools

All SEO Tools

XML Sitemap GeneratorPaste a list of URLs and get a valid, escaped sitemap.xml with optional last-modified dates.

Meta Tag GeneratorGenerate clean HTML meta tags for a page: title, description, canonical, robots, Open Graph and Twitter Card tags, with length guidance.

Canonical URL HelperNormalise URLs to one consistent canonical form: HTTPS, www or non-www, trailing slash, tracking parameters removed.

Favicon GeneratorGenerate a complete favicon set from an image or a letter: favicon.ico, 16–512 px PNGs, Apple touch icon, web manifest and HTML tags, in one ZIP.

Slug GeneratorTurn titles into clean, lowercase URL slugs.

Open Graph PreviewPreview a link card from your og:title, og:description and og:image, check image size and text length, and copy the Open Graph tags.