Cloudflare was silently blocking AI crawlers from this site. Here's the fix.

If you want your site to show up in AI answers, check one Cloudflare setting today. There is a toggle that returns a 403 blocked to the exact AI crawlers you want reading your site, and it overrides your robots.txt without telling you.

We found it on our own site. Real browsers loaded fine. Google loaded fine. But when we tested the way an AI crawler hits the page, we got blocked. Here is how we caught it and fixed it, so you can check yours in about two minutes.

How to tell if it is happening to you

Pretend to be an AI crawler and see what the site returns. From a terminal, send a request with an AI bot's user-agent and look at the status code.

curl -A "GPTBot" -I https://yourdomain.com
curl -A "ClaudeBot" -I https://yourdomain.com

GPTBot is OpenAI's crawler, and its user-agent is documented in OpenAI's docs. ClaudeBot is Anthropic's, documented here. If your browser gets 200 but either bot comes back 403, something in front of your site is blocking them.

Why robots.txt does not save you

Because this block happens at the edge, before robots.txt is ever read. robots.txt is a polite request a bot chooses to honor. An edge block is a wall. It returns 403 no matter what your robots.txt says.

So you can have a perfectly welcoming robots.txt, your sitemap submitted, everything done right, and still be invisible to AI search, because the request never gets far enough to see any of it.

The fix

In Cloudflare, find the setting that blocks AI scrapers and crawlers and turn it off, or set the specific bots you want to Allow. It lives under the bot and security controls. One toggle.

Cloudflare added this as a one-click way to keep AI companies from training on your content, which is a fair thing to want. But it is blunt. Block AI bots also blocks the crawlers that put you in AI answers, and the ones that fetch a page live when a user asks an assistant about you. If being cited by ChatGPT or Perplexity matters to you, that trade is backwards.

Flip it, then re-run the curl test. You want 200 for GPTBot, ClaudeBot, and the rest. We did exactly this on j450n.ai, and every AI crawler went from 403 to 200.

Your robots.txt can say come on in while your CDN says get lost. Test the door, do not just read the sign.

Want your site to actually show up when someone asks an AI about your work? Tell us what you are building and we will make sure the machines can find it. Questions first? Reach us at [email protected].