Test robots.txt like Google does
Fetch a site's live robots.txt or paste your own draft, then enter URLs or paths and pick a crawler: Googlebot, Googlebot-Image, Bingbot, GPTBot, ClaudeBot, Applebot or any other bot. Each URL is marked Allowed or Blocked with the exact line and rule that matched.
Matching follows Google's documented rules: the most specific user-agent group applies, the longest matching path wins, Allow wins a tie, and * and $ wildcards work. Warnings flag files that block the entire site or hide CSS and JavaScript from crawlers.
How to test robots.txt
- 1
Fetch or paste robots.txt
Edit it freely to try changes.
- 2
Enter URLs or paths
One per line.
- 3
Choose the crawler
Googlebot, Bingbot, AI bots…
- 4
Read each verdict
With the matching rule.
Key features
Google-accurate matching
Longest match, Allow on ties, wildcards.
Many crawlers
Search engines and AI bots.
Edit and retest
Try rules before publishing.
Safety warnings
Whole-site blocks and blocked assets.
Common uses
- Pages missing from Google: Rule out robots.txt blocks.
- Before deploying changes: Test new rules safely.
- AI crawler policy: Confirm GPTBot or ClaudeBot is blocked.
Tips and good to know
- Google Search Console's robots.txt report shows the version Google last fetched and any parse errors.
- Build a new file with the Robots.txt Generator.
Frequently asked questions
Which rule wins when Allow and Disallow both match?
The one with the longest path. If they're the same length, Allow wins.
Does Google follow Crawl-delay?
No. Bing and Yandex do; Google adjusts its crawl rate automatically.
Can a blocked page still appear in Google?
Yes, as a bare URL if other sites link to it. Use noindex (and allow crawling) to remove it.