AI index crawlers are allowed in

What this is

When someone asks ChatGPT or Perplexity “who does couples counseling near me,” the assistant does not read the whole internet on the spot. It looks in an index. An index is a big list of pages it read earlier.

Robots build that index. Each company has its own. We check eight of them:

• OAI-SearchBot (ChatGPT search)
• Claude-SearchBot (Claude)
• PerplexityBot (Perplexity)
• Googlebot (Google, including AI Overviews and AI Mode)
• Bingbot (Bing and Copilot)
• DuckAssistBot (DuckDuckGo)
• Applebot (Siri and Apple search)
• Amazonbot (Alexa)

Your robots.txt file can tell any of these to stay out. See Your robots.txt file works for what that file is.

Why it matters to you

This is the biggest check on the whole page, 13 points. If you block the robot that builds an index, you are not in that index. Nothing else on your site can make up for it.

Blocking any of the five big ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) also caps your whole score at 40, because everything else depends on this.

There is a second kind of robot, a training crawler: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent. Those collect text to teach future models. Blocking them is a separate choice, a licensing choice, and it is yours to make. It does not remove you from AI answers. We list them in your results but never score them.

What we look for

• Pass: all 8 index crawlers may read your homepage.
• Warn: 1 or 2 of the 8 are blocked. You keep most of the points.
• Fail: 3 or more are blocked, or we could not read your robots.txt at all (robots treat an unreadable file as “stay out”).

A crawler that is not named in your file counts as allowed.

Whether you can fix this yourself

Yes. It is a few lines in a text file. Most blocks come from a copied “block all AI” list that also blocks the search robots. If you would rather not, Setup ($350, 3 business days) does it for you.

Squarespace, Wix and Shopify write robots.txt for you and do not block these by default. Their “AI crawler” switch is meant for training crawlers. Check it is not blocking the index ones.

How to fix it

Open your robots.txt and remove any Disallow: / line that sits under one of the eight names above. If you would rather be explicit, add this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

WordPress

Yoast SEO and Rank Math both have a robots.txt editor under Tools. If a security plugin (Wordfence, Sucuri, AIOS) has a “block AI bots” feature, un-tick the eight names above.

Squarespace or Wix

Search your platform’s help for “AI crawlers”. Make sure the setting only blocks training crawlers, not search ones.

Shopify

Online Store, then Themes, then Edit code, then open robots.txt.liquid. Remove any block for the names above.

Your own site or a developer

Edit robots.txt in the top folder of the site. Keep training blocks if you want them. Remove index blocks.

How to check it worked

Go to your site address with /robots.txt on the end. Read down the list. None of the eight names should be followed by Disallow: /. Then run the free check again.

Do not worry about

Blocking GPTBot, ClaudeBot or CCBot. Those are training crawlers. Keeping them out is fine and does not change this check.