Generate a robots.txt file for your website
A robots.txt file tells search engine crawlers which parts of your site they may visit. Set a default rule, add disallowed paths, optional rules for specific bots and your sitemap URL, and copy or download a ready-to-use file.
Default rule for all crawlers
Block specific AI crawlers
robots.txt
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Upload this file to the root of your site, e.g. https://yourdomain.com/robots.txt
robots.txt Generator: 3 simple steps
- 1
Choose whether all bots are allowed and add any paths to block.
- 2
Add your sitemap URL and optional bot-specific rules.
- 3
Copy or download robots.txt and upload it to your site root.
Remember that robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag for pages that must stay out of search.
Built for speed and privacy
- Valid, well-formatted output
- Rules for specific bots such as GPTBot
- Sitemap line included
- Download as a file
robots.txt Generator FAQ
In the root of your domain, so it is reachable at https://yourdomain.com/robots.txt.
Not reliably. It stops crawling, but URLs can still be indexed from links. Use a noindex meta tag or password protection to keep pages out of search.
Add a separate rule with the crawler's user-agent, for example GPTBot or CCBot, and Disallow: /. Not every bot obeys robots.txt.
Yes. A Sitemap line helps crawlers discover your sitemap even if you have not submitted it in Search Console.
More free tools
Fix technical SEO issues for good
We audit crawlability, sitemaps, structured data and speed, then fix what's holding you back.