Free tool
Robots.txt tester
Paste a robots.txt file and test any URL for Googlebot, Bingbot or AI crawlers. Shows the exact matching rule using Google's longest-match logic.
How Google reads robots.txt
Google picks the group whose User-agent line matches the crawler most specifically, falling back to *. Within that group the longest matching path rule wins; if an Allow and a Disallow match with the same length, Allow wins. * matches any characters and $ marks the end of the URL.
Blocking AI crawlers
You can block crawlers such as GPTBot or ClaudeBot by name. Google-Extended controls whether Google uses your content for some Gemini features; it doesn't affect Search rankings. Google's AI Overviews and AI Mode use the normal Googlebot index, and Search Console added a separate Search generative AI control in 2026.
robots.txt is not noindex
Blocking a URL stops crawling, but the URL can still appear in results if other pages link to it. To keep a page out of Google, allow crawling and add a noindex tag.