Your robots.txt is quietly blocking the AI crawlers you want

If your business is invisible in ChatGPT, Perplexity, and Google's AI answers, your robots.txt is probably the reason. One stray line tells the exact bots that feed those answers to leave, and they listen.

Why doesn't my site show up in ChatGPT or Perplexity?

Because you told the crawlers to go away. AI answer engines don't invent your site out of thin air. They send a bot to read your pages, and if that bot hits a Disallow, you're not in the answer. Simple as that.

The gut-punch: most of these blocks are accidents. Somebody flipped a “block AI bots” toggle. Somebody copied a robots.txt off a template. A plugin added a rule nobody reviewed. Now the door's shut and nobody remembers shutting it. Frank doesn't guess at what's blocked. He reads the file.

What's the difference between a training crawler and an answer crawler?

This is the part most people get wrong, and it's the whole game. Two different jobs hide under the “AI bot” label, and you get to choose them separately.

Training crawlers grab your content to help build future models. OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, and Google's Google-Extended token all sit here. Block these and your words won't feed model training. That's a fair choice. Nobody's forcing you to donate your content.

Answer crawlers are different animals. They fetch your page so the AI can cite you in an answer, with a link back. OpenAI's OAI-SearchBot surfaces you in ChatGPT's search. Anthropic's Claude-SearchBot indexes you for Claude. PerplexityBot puts you in Perplexity's results. Block these and you vanish from the answer, link and all.

OpenAI says it flat out. A webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot so crawled content isn't used for training their foundation models. Two switches. Not one.

So why does “block AI bots” break the thing I want?

Because most blanket blocks don't know the difference. The toggle in your CMS, the firewall rule, the template robots.txt: they see “AI” in the name and swat everything. Training bot, answer bot, all gone. You wanted to stop the scrapers. You also deleted yourself from ChatGPT and Perplexity.

Anthropic spells out the cost. Disable Claude-SearchBot and you prevent their system “from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.” That's your visibility, gone, over one line you probably never read.

Which AI crawlers should I actually allow?

If you want to show up in AI answers, allow the answer bots. Here's the short list, checked against each company's own docs:

• OAI-SearchBot: ChatGPT search. Allowing it is how your site shows up in ChatGPT's search results, per OpenAI's docs.
• ChatGPT-User: fires when a ChatGPT user's request pulls your page.
• Claude-SearchBot: Claude's search index. Blocking it drops your visibility in Claude's answers.
• Claude-User: fires when a Claude user's question pulls your page.
• PerplexityBot: Perplexity's search results. Perplexity recommends allowing it in robots.txt.

Training is your call. Want your content out of model training but still in the answers? Disallow GPTBot, ClaudeBot, CCBot, and Google-Extended, and leave the answer bots alone. Note that Google-Extended only controls Gemini training. It does not touch Google Search or AI Overviews, so blocking it won't hide you from Google's AI answers. Blocking plain Googlebot will.

What does a clean robots.txt look like?

robots.txt allows everything by default. The damage comes from a Disallow you didn't mean to ship. First, kill the blanket block if it's there by accident:

# The line that makes you invisible. Delete it if you didn't mean it.
User-agent: *
Disallow: /

Then make the answer bots explicit, so no future template rule buries them:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

Want out of training but still in the answers? Add this block and stop there:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

One warning. OpenAI says search changes can take about 24 hours to register after you edit robots.txt. Fix it, then give it a day before you panic.

It's not just robots.txt: is Cloudflare blocking them too?

Maybe. robots.txt is a polite request, not a wall. A firewall or bot-management rule is the wall, and it doesn't care what your robots.txt says.

Perplexity says it straight: if you run a Web Application Firewall, you may need to explicitly whitelist their bots so they can reach your content. Cloudflare ships a one-click “block AI bots” control, and plenty of operators flip it on without reading the fine print. It blocks the answer bots right alongside the scrapers. Same accident, different switch.

And know the limits of the file itself. Some fetchers ignore it. Perplexity states that Perplexity-User “generally ignores robots.txt rules” because a live user asked for the page. Anthropic notes that blocking their bots by IP address may not work and can even stop them from reading your robots.txt at all. The file is your first lever, not your only one. Check the firewall too.

What this costs you

Here's what it means if you run the business. AI search is where buyers are starting now. A prospect asks ChatGPT “who does this in my city,” the model goes looking, and your site said “no bots allowed.” You're not in the running. You didn't lose that deal on price or product. You lost it on a text file most owners have never opened.

That's the cheapest fix in your entire funnel, and almost nobody is doing it. Your competitors included.

Open your robots.txt. Read every Disallow out loud. If you can't say why it's there, it's costing you answers. Frank doesn't block the leads walking up to the door. He lets them in.

Want us to audit yours and wire your site into AI answers the right way? Start the brief.