Check 3 of 23 · Can AI reach your site?

AI live-fetch agents are allowed in

When a person asks an assistant about you by name, it may open your page right then. Your robots.txt has to let that happen.

  • Who fixes itYou can do this
  • TimeAbout 10 minutes
  • Worth9 of 100 points
  • How sure we areProven

What this is

Some AI robots do not build an index. They go and look at one page, right now, because a person just asked. We call them live-fetch agents.

Say a client types “look at yourbusiness.com and tell me their hours” into ChatGPT. ChatGPT sends a robot to your site at that moment. If your robots.txt says that robot may not enter, the assistant tells the person it could not read your page.

We check three of them:

  • ChatGPT-User (ChatGPT)
  • Claude-User (Claude)
  • Perplexity-User (Perplexity)

These are different robots from the ones that build the index. Those are covered in AI index crawlers are allowed in.

Why it matters to you

This is a big check, 9 points. The person has already found you. They have your name. They are asking the assistant to read your page and tell them about you. This is the best moment a website gets. Blocking the live-fetch agent turns that moment into “I could not access that site.”

Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent) are a separate matter. They gather text to train future models. Blocking them is a choice about licensing, not about being found, and it does not affect AI answers or this check.

What we look for

  • Pass: all 3 live-fetch agents may read your homepage.
  • Fail: any of the 3 is blocked, or we could not read your robots.txt at all. Each blocked agent costs 3 of the 9 points, so one blocked agent scores 6 of 9 but still shows as a fail.

An agent that is not named in your file counts as allowed.

Whether you can fix this yourself

Yes. It is a text file edit. The usual cause is a copied “block all AI” list that lumps these in with training crawlers. If you would rather not, Setup ($350, 3 business days) does it for you.

Squarespace, Wix and Shopify write robots.txt for you. Their “AI crawler” switches are aimed at training bots, but check what they cover before you turn one on.

How to fix it

Find any User-agent: line naming one of the three agents and remove the Disallow: / under it. Or add this, which says yes out loud:

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

WordPress

Yoast SEO and Rank Math have a robots.txt editor under Tools. If your security plugin has a bot block list, un-tick these three names.

Squarespace or Wix

Search your platform’s help for “AI crawlers” and read which robots the setting blocks.

Shopify

Online Store, then Themes, then Edit code, then robots.txt.liquid. Remove any block for these three.

Your own site or a developer

Edit robots.txt at the top of the site and remove the block.

How to check it worked

Open your site address with /robots.txt on the end. None of the three names should sit above a Disallow: /. Then run the free check again. For a real-world test, ask ChatGPT to “read yourbusiness.com and summarize it.”

Do not worry about

GPTBot and ClaudeBot. Blocking those stops model training only. This check does not look at them.