AI Crawler Checker
Is your site blocking ChatGPT, Claude, or Perplexity?
This free tool checks whether GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers can access your robots.txt. A blocked crawler cannot mention or cite your brand.
Why use the AI Crawler Checker?
It checks every major AI crawler individually instead of a single pass or fail, and translates robots.txt syntax into plain language. You will know exactly which AI engines can read your site, and exactly what to change if they cannot.
Checks every AI crawler
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more, each one checked individually, not just a single pass or fail.
Plain-language results
No robots.txt syntax to decode. See exactly which engines are blocked and exactly what to change to fix it.
Free, instant, no signup
Paste a URL and get results in seconds. Nothing to install, no account, no credit card.
How does the AI Crawler Checker work?
Paste your URL and the tool fetches your robots.txt file, checks the access rules for each AI crawler, and returns a plain-language result in seconds. No signup and no technical knowledge required.
Paste your website URL
Enter any public website address. No signup, no login, nothing to install.
We read your robots.txt
The tool fetches your robots.txt file and checks the access rules for each major AI crawler individually.
See what’s blocked and fix it
Get a plain-language result showing which AI crawlers are allowed, blocked, or partially restricted, and what to change.
Common issues & solutions
⚠The checker says a crawler is blocked, but I already added an Allow rule.
✓robots.txt rules are read in order within each user-agent group. If a broader "Disallow: /" rule for that crawler appears above your Allow rule, some crawlers apply the first matching rule instead of the most specific one. Move the Allow rule above any conflicting Disallow, or remove the conflicting rule entirely.
⚠I updated robots.txt, but the checker still shows the old result.
✓The checker reads your live robots.txt on every check, so there is no caching on our end. Hosting providers and CDNs (Cloudflare, Vercel, Netlify) often cache robots.txt for minutes to hours. Purge your CDN cache, or wait about 15 minutes, then check again.
⚠My site has no robots.txt file at all.
✓A missing robots.txt means every crawler is allowed by default, which is why the checker shows every AI crawler as "Allowed." This is a valid, safe setup for most SaaS marketing sites. Only add a robots.txt if you need to block specific paths or specific crawlers.
⚠Some AI crawlers show "Partially restricted" instead of a clear pass or fail.
✓This means the crawler is blocked from certain paths, like /admin or /checkout, but allowed everywhere else. That is normal and usually intentional. Check that your public marketing pages, especially blog posts and product pages, are not among the restricted paths.
⚠I want to block GPTBot but still show up in ChatGPT’s live answers.
✓GPTBot and OAI-SearchBot are separate crawlers. Blocking GPTBot only stops OpenAI from using your content for future model training. It does not affect ChatGPT’s live web search. Keep OAI-SearchBot allowed if you want to appear in real-time ChatGPT answers.
When should you use the AI Crawler Checker?
Run it any time crawl access could be the silent reason you are invisible to AI engines: before a launch, after a redesign, during a client audit, or when a brand still is not showing up in AI answers despite good content.
Who is the AI Crawler Checker for?
Anyone whose growth depends on AI engines being able to find and cite their content: indie SaaS founders checking their own site, marketers running GEO audits, and agencies screening client sites for a hidden crawl-access problem.
Indie SaaS founders
Make sure your own site is visible to the AI engines your customers ask for recommendations, without needing an SEO background.
SaaS marketers
Add an AI-crawler check to your GEO and SEO audits so crawlability never becomes the silent reason you’re invisible.
SEO & GEO agencies
Run a quick, shareable check across client sites to spot robots.txt rules blocking AI engines before they cost visibility.
Why does AI crawler access matter for your brand?
AI engines like ChatGPT, Perplexity, and Gemini can only mention or cite your brand if their crawlers can actually read your pages. A single blocked line in robots.txt makes you invisible to that engine, even while your site ranks fine on Google. We cover this trade-off in detail in does blocking AI crawlers hurt brand visibility.
Frequently asked questions
What does it mean if an AI crawler is "blocked"?
Your robots.txt file has a rule telling that specific crawler not to fetch any pages on your site. The crawler will honor this and skip your site entirely. It cannot read your content, so that AI engine has no way to learn about or mention your brand from your own pages.
Is blocking AI crawlers bad for my business?
It depends on your goals. If you want ChatGPT, Perplexity, or Gemini to mention your brand when people ask relevant questions, blocking their crawlers removes that possibility. If you’re specifically trying to keep your content out of AI training data, blocking is intentional. Most businesses trying to grow visibility want these crawlers allowed.
Why is "Google-Extended" separate from Googlebot?
Googlebot handles regular Google Search indexing. Google-Extended is a separate control that only affects whether Google can use your content to improve Gemini and AI Overviews. You can block Google-Extended without affecting your normal Google Search rankings at all.
How do I unblock an AI crawler?
Edit your robots.txt file (usually at yoursite.com/robots.txt) and remove or narrow the Disallow rule under that crawler’s User-agent line. Most CMS platforms (WordPress, Webflow, Ghost) let you edit this file from your site settings without touching code.
Does this tool check anything other than robots.txt?
No, this free tool only checks crawler access rules in robots.txt. It does not check whether AI engines actually mention or cite your brand. For that, Lead Rescue runs daily AI visibility scans across ChatGPT, Perplexity, and Gemini.
Crawlable is step one. Cited is the goal.
Once AI engines can read your site, Lead Rescue tracks whether they actually mention and cite your brand across ChatGPT, Perplexity, and Gemini, every day, in plain language.
Check your AI visibility free →