AI Crawler Checker
Is your site blocking ChatGPT, Claude, or Perplexity?
This free tool sends a live request to your site using each AI crawler’s real user-agent, so you see exactly how your server responds, not what a config file claims.
Why use the AI Crawler Checker?
It sends a live request to your site as every major AI crawler, individually, and translates the response into plain language. You will know exactly which AI engines can read your site, and exactly what to change if they cannot.
Tests real crawler behavior
Sends a live request using each AI crawler’s actual identity, not a static config check, so you see exactly what your server does.
Plain-language results
No HTTP status codes to decode. See exactly which engines are blocked and what that likely means.
Free, instant, no signup
Paste a URL and get results in seconds. Nothing to install, no account, no credit card.
How does the AI Crawler Checker work?
Paste your URL and the tool sends a live request to it using each AI crawler’s real user-agent, then returns a plain-language result in seconds. No signup and no technical knowledge required.
Paste your website URL
Enter any public website address. No signup, no login, nothing to install.
We send a live request as each crawler
The tool sends a real HTTP request to your site using each AI crawler’s own user-agent, all at once.
See exactly what each crawler got back
Get a plain-language result for each crawler: allowed, blocked, rate limited, or unreachable.
Which crawlers does it test?
This tool tests 10 real AI crawlers: GPTBot, OAI-SearchBot, and ChatGPT-User (all OpenAI, powering ChatGPT), ClaudeBot (Anthropic, powers Claude), PerplexityBot (Perplexity), GrokBot (xAI, powers Grok), Googlebot and GoogleOther (Google, feed Gemini and AI Overviews), CCBot (Common Crawl, used to train many AI models), and Bytespider (ByteDance, feeds TikTok’s AI).
Common issues & solutions
⚠The live test shows a crawler is blocked, but I never set up any bot-blocking rules.
✓Many hosting providers and CDNs (Cloudflare, Vercel, AWS) include automatic bot-management features that block known AI crawler user-agents by default, even if you never configured anything yourself. Check your hosting provider or CDN’s security or bot-protection settings for a rule matching that crawler’s name.
⚠The live test shows "Could not connect" for a crawler.
✓This usually means the request timed out or the connection was dropped before a response came back, which can happen with a slow server, a firewall that silently drops traffic instead of returning an error, or a temporary outage. Try checking again in a minute.
⚠A crawler shows "Rate limited (429)."
✓Your server, or a service in front of it like Cloudflare, is telling that crawler to slow down, usually because of how much traffic it or similar requests have sent recently. This is not necessarily a permanent block. Running the check again later may show a different result.
⚠The result for a crawler changes between checks.
✓Server responses can vary run to run because of caching, load balancers, or bot-management services that only start blocking after repeated requests from the same source. If a result looks unexpected, run the check again.
⚠I want to block GPTBot but still show up in ChatGPT’s live answers.
✓GPTBot and OAI-SearchBot are separate crawlers. Blocking GPTBot only stops OpenAI from using your content for future model training. It does not affect ChatGPT’s live web search. Keep OAI-SearchBot allowed if you want to appear in real-time ChatGPT answers.
When should you use the AI Crawler Checker?
Run it any time crawl access could be the silent reason you are invisible to AI engines: before a launch, after a redesign, during a client audit, or when a brand still is not showing up in AI answers despite good content.
Who is the AI Crawler Checker for?
Anyone whose growth depends on AI engines being able to find and cite their content: indie SaaS founders checking their own site, marketers running GEO audits, and agencies screening client sites for a hidden crawl-access problem.
Indie SaaS founders
Make sure your own site is visible to the AI engines your customers ask for recommendations, without needing an SEO background.
SaaS marketers
Add an AI-crawler check to your GEO and SEO audits so crawlability never becomes the silent reason you’re invisible.
SEO & GEO agencies
Run a quick, shareable check across client sites to spot AI crawlers being blocked before they cost visibility.
Why does AI crawler access matter for your brand?
AI engines like ChatGPT, Perplexity, and Gemini can only mention or cite your brand if their crawlers can actually read your pages. A firewall or bot-management rule can silently block a crawler without you ever noticing, even while your site ranks fine on Google. We cover this trade-off in detail in does blocking AI crawlers hurt brand visibility.
Frequently asked questions
Which AI crawlers does this tool test?
It tests 10 real AI crawlers: GPTBot, OAI-SearchBot, and ChatGPT-User (all OpenAI, powering ChatGPT), ClaudeBot (Anthropic, powers Claude), PerplexityBot (Perplexity), GrokBot (xAI, powers Grok), Googlebot and GoogleOther (Google, feed Gemini and AI Overviews), CCBot (Common Crawl), and Bytespider (ByteDance, feeds TikTok’s AI).
What does it mean if an AI crawler is "blocked"?
The crawler sent a real request to your site using its own identity, and your server responded with an error status, like 403 Forbidden, instead of the page. That crawler could not read your content, so the AI engine it powers has no way to learn about or mention your brand from that page.
Is blocking AI crawlers bad for my business?
It depends on your goals. If you want ChatGPT, Perplexity, or Gemini to mention your brand when people ask relevant questions, blocking their crawlers removes that possibility. If you’re specifically trying to keep your content out of AI training data, blocking is intentional. Most businesses trying to grow visibility want these crawlers allowed.
Why does the result for GrokBot come with a warning?
xAI documents a GrokBot user-agent, but Grok is widely reported to also crawl with disguised browser user-agents and rotating IPs. So a "GrokBot allowed" result confirms the documented bot got through, but it does not guarantee you can actually keep Grok out entirely. Treat GrokBot as a weaker signal than the other crawlers.
How do I unblock an AI crawler the live test shows as blocked?
Check your hosting provider, CDN, or firewall’s bot-management or security settings for a rule matching that crawler’s user-agent, like "GPTBot" or "ClaudeBot", and allow it. Most managed platforms (Cloudflare, Vercel, WordPress security plugins) let you adjust this without touching code.
Does this tool check my robots.txt file?
No. This tool only sends live requests using each crawler’s real user-agent to see how your server actually responds. robots.txt is a separate, voluntary file that a crawler may or may not follow, and it can’t tell you what a firewall or bot-management layer will actually do. We test the real thing instead.
Does this tool check whether AI engines mention or cite my brand?
No. This free tool only checks whether each crawler can reach your site. It does not check whether AI engines actually mention or cite your brand in their answers. For that, Lead Rescue runs daily AI visibility scans across ChatGPT, Perplexity, and Gemini.
Crawlable is step one. Cited is the goal.
Once AI engines can read your site, Lead Rescue tracks whether they actually mention and cite your brand across ChatGPT, Perplexity, and Gemini, every day, in plain language.
Check your AI visibility free →