Is your site blocking ChatGPT, Claude, or Perplexity?
This free tool checks your website’s robots.txt file to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, and Bytespider are allowed or blocked. If an AI crawler is blocked, that engine cannot read your pages — which means it cannot mention or cite your brand in its answers.
Why this matters
AI engines like ChatGPT, Perplexity, and Gemini can only mention or cite your brand if their crawlers can actually read your pages. A single line in your robots.txt file can silently block them — even while your site ranks fine on Google. We cover this trade-off in detail in does blocking AI crawlers hurt brand visibility.
Frequently asked questions
What does it mean if an AI crawler is "blocked"?
Your robots.txt file has a rule telling that specific crawler not to fetch any pages on your site. The crawler will honor this and skip your site entirely — it cannot read your content, so that AI engine has no way to learn about or mention your brand from your own pages.
Is blocking AI crawlers bad for my business?
It depends on your goals. If you want ChatGPT, Perplexity, or Gemini to mention your brand when people ask relevant questions, blocking their crawlers removes that possibility. If you’re specifically trying to keep your content out of AI training data, blocking is intentional. Most businesses trying to grow visibility want these crawlers allowed.
Why is "Google-Extended" separate from Googlebot?
Googlebot handles regular Google Search indexing. Google-Extended is a separate control that only affects whether Google can use your content to improve Gemini and AI Overviews. You can block Google-Extended without affecting your normal Google Search rankings at all.
How do I unblock an AI crawler?
Edit your robots.txt file (usually at yoursite.com/robots.txt) and remove or narrow the Disallow rule under that crawler’s User-agent line. Most CMS platforms (WordPress, Webflow, Ghost) let you edit this file from your site settings without touching code.
Does this tool check anything other than robots.txt?
No — this free tool only checks crawler access rules in robots.txt. It does not check whether AI engines actually mention or cite your brand. For that, Lead Rescue runs daily AI visibility scans across ChatGPT, Perplexity, and Gemini.