Check 4 of 23 · Can AI reach your site?

Your server answers an AI crawler

A firewall can turn away AI crawlers even when robots.txt welcomes them. We knock as a crawler, as a browser, and as a plain visitor, and compare the answers.

  • Who fixes itYou, or your web person
  • TimeAbout 20 minutes
  • Worth7 of 100 points
  • How sure we areLikely

What this is

Your robots.txt is only a request. Your firewall or security plugin does not read it, and it decides on its own who gets in.

Every visitor says who it is. A browser says “I am Chrome.” ChatGPT’s search crawler says “I am OAI-SearchBot.” Some firewalls let the browser in and shut the door on anything else. Crawlers are not browsers.

So we request your homepage three times, each time saying something different:

  1. As OAI-SearchBot, the crawler ChatGPT search uses.
  2. As a normal browser.
  3. As a plain unnamed visitor that claims to be nobody special.

Comparing the three answers tells us what the firewall is doing.

Why it matters to you

This is a meaningful check, 7 points. If a firewall refuses crawlers, everything in AI index crawlers are allowed in is moot, because the crawler never reaches the page. A full block caps your whole score at 40.

Most sites pass, and one thing that looks like a block is not. Some services, Cloudflare for one, challenge any visitor that claims to be a crawler but does not arrive from that crawler’s own network. We fail that challenge because we are not OpenAI. The real crawler passes it because it is. We do not mark you down for that.

What we look for

  • Pass: your homepage answers our OAI-SearchBot request with a normal page (HTTP 200).
  • Warn (half points): the crawler request is refused, but the plain unnamed visitor gets in. That is a verification challenge, not a block. You also get a warn if nothing answered cleanly at all.
  • Fail (0 points): the crawler request is refused, the plain unnamed visitor is refused, and only the browser gets a page. Something in front of your site turns away everything that is not a browser.

Whether you can fix this yourself

Probably not without a web person, unless you know your way around your security plugin. The setting lives in a firewall, a CDN, or a WordPress security plugin, and each one names it differently.

Squarespace, Wix, Shopify and SimplePractice run the firewall for you and do not block these crawlers by default. A fail there is almost always a setting someone turned on, or a Cloudflare rule in front of your site.

How to fix it

If you got a warn

Probably nothing. To be certain, ask whoever runs your CDN or host to look in the logs for OAI-SearchBot, Claude-SearchBot and PerplexityBot getting 200 responses.

If you got a fail

  1. WordPress security plugins first. Wordfence, Sucuri, All-In-One Security and iThemes each have a “block fake or unknown bots” style option. Turn it off, or add the AI crawlers to its allow list.
  2. Cloudflare. In Security, then Bots, look for a rule that blocks or challenges all bots. Turn on the “verified bots” allowance instead, or add a rule allowing the AI crawler names.
  3. Other firewalls (Sucuri WAF, Fastly, Akamai). Search their help for “allow bot user agent” and add the crawlers by name.

How to check it worked

Open your site in a normal browser and confirm it loads. Then run the free check again. The result tells you exactly which of the three requests got through.

Do not worry about

A warn on this check when your site uses Cloudflare. That is the verification challenge described above, and the real crawlers pass it.