Control who crawls your site
Start from a preset — allow everything, WordPress, online store or block everything for staging sites — then add the paths to keep crawlers out of, like admin areas, carts, internal search results and filter URLs, plus any Allow exceptions.
One click adds groups that block AI training and answer crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more) or SEO tool crawlers (AhrefsBot, SemrushBot, MJ12bot). Add a crawl-delay for Bing and Yandex and your sitemap URLs, and copy or download the finished file.
How to create robots.txt
- 1
Choose a preset
WordPress, store or allow all.
- 2
Add disallowed paths
One per line.
- 3
Choose bots to block
AI crawlers or SEO tools.
- 4
Add your sitemap
Then download robots.txt.
Key features
Presets
WordPress, e-commerce, staging.
AI crawler blocking
GPTBot, ClaudeBot, Google-Extended and more.
Sitemap lines
Point every crawler to your sitemap.
Live preview
Copy or download instantly.
Common uses
- New websites: Start with a sensible robots.txt.
- Opting out of AI training: Block model-training crawlers.
- Reducing crawl waste: Keep bots out of carts and search pages.
Tips and good to know
- robots.txt stops crawling, not indexing. To keep a page out of Google, use a noindex tag and let Google crawl it.
- Test the finished file with the Robots.txt Tester.