Free GEO toolsAI crawler checker
Can AI crawlers read your site?
Enter a page. We read your live robots.txt and test it against 22 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta and others, then tell you which line to change.
What it checks
- AI search crawlers, which decide what can be cited in answers
- Fetchers that open your page when someone asks an assistant about it
- Training crawlers, kept apart because blocking them is your call
- Server errors, HTML answers and typos that silently change the rules
Why it matters
Each AI company runs separate bots for search, for user requests and for training. They are switched on and off separately: OpenAI documents that blocking GPTBot (training) does not stop OAI-SearchBot, the crawler behind ChatGPT search. A single Disallow: / under the wrong user agent can take you out of AI answers without anyone noticing.
Questions
About the AI crawler checker
If you want to be quoted in AI answers, allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and the rest of the “AI search” rows) and the user-triggered fetchers. Training crawlers are a separate business decision.
Not from ChatGPT search, according to OpenAI’s crawler documentation: each setting is independent. GPTBot is for training; OAI-SearchBot decides whether ChatGPT search can show and cite your pages.
They use pages that Googlebot crawls and indexes, so Googlebot is the one that matters. Google-Extended controls Gemini training and grounding, and Google says it doesn’t affect Search (AI features and your website).
robots.txt is only the door. A CDN or firewall can block AI crawlers before they read it, content that needs JavaScript can be invisible to them, and being readable doesn’t make an engine recommend you. Our free AI visibility checker shows who the engines name instead.
Following RFC 9309, the robots.txt standard: a crawler follows the group that names it, otherwise the “*” group; the longest matching rule wins and Allow wins a tie. A missing file means no limits; a server error means “stay out”.
We checked every company’s own documentation on October 31, 2026, and each row in the result links to it. Companies add bots from time to time, and we update the list when they do.
More free tools
Readable is step one. Does ChatGPT recommend you?
We ask ChatGPT, Gemini and Google AI Overviews three questions your buyers ask and show who gets named, you or a competitor. Free, no signup.
Run the free AI visibility check