XML Sitemap GeneratorPaste a list of URLs and get a valid, escaped sitemap.xml with optional last-modified dates.
Robots.txt Generator
robots.txt tells well-behaved crawlers which parts of your site they may request. It is not a security feature — blocked URLs can still be visited and can even appear in search results if other pages link to them. To keep a page out of results, use a noindex tag instead.
Test a path
Runs entirely in your browser. Your input and files are not uploaded to any server.
How to use
- Pick a starting preset.
- List the paths to disallow or allow, one per line, and add your sitemap URL.
- Test a few paths, then download robots.txt and upload it to the root of your domain.
Examples
Block an admin area
Disallow /admin/ while leaving the rest of the site crawlable.
Opt out of AI training crawlers
Add rules for user agents such as GPTBot and CCBot that publish opt-out instructions.
Privacy
Runs entirely in your browser. Your input and files are not uploaded to any server. See the privacy policy for what is stored in your browser (favorites and history only). Privacy policy
Check it yourself: open your browser’s developer tools, select the Network tab and use the tool. Your file never appears in an upload request. Once this page is saved for offline use, you can also disconnect from the internet and keep working. Ads and anonymous visit statistics, when enabled, make their own requests, but they never receive your files or text.
Frequently asked questions
Where does robots.txt go?
At the root of the host: https://example.com/robots.txt. Each subdomain needs its own file.
Does Disallow remove a page from Google?
No. It stops crawling, not indexing. Use a noindex meta tag (and allow crawling) to remove a page from results.
How does the tester decide?
It follows the Robots Exclusion Protocol (RFC 9309): the longest matching rule wins, Allow wins a tie, and * and $ wildcards are supported. It tests the group for “*”.
Related tools
All SEO ToolsMeta Tag GeneratorGenerate clean HTML meta tags for a page: title, description, canonical, robots, Open Graph and Twitter Card tags, with length guidance.
Canonical URL HelperNormalise URLs to one consistent canonical form: HTTPS, www or non-www, trailing slash, tracking parameters removed.
Favicon GeneratorGenerate a complete favicon set from an image or a letter: favicon.ico, 16–512 px PNGs, Apple touch icon, web manifest and HTML tags, in one ZIP.
Slug GeneratorTurn titles into clean, lowercase URL slugs.
Open Graph PreviewPreview a link card from your og:title, og:description and og:image, check image size and text length, and copy the Open Graph tags.