Check 1 of 23 · Can AI reach your site?

Your robots.txt file works

robots.txt is a tiny note on your site that tells robots what they may visit. It has to load without an error, or robots think the whole site is off limits.

  • Who fixes itYou can do this
  • TimeAbout 10 minutes
  • Worth5 of 100 points
  • How sure we areProven

What this is

Every website can have a small text file called robots.txt. It lives at the front door of your site, like this:

https://yourbusiness.com/robots.txt

It is a note for robots. AI crawlers, and Google, read it first, before anything else. It says which parts of your site they may look at.

You do not have to have one. If it is missing, robots take that as “come on in.”

Why it matters to you

There is one trap. If the file is there but broken, or your server answers with an error, robots follow a rule that says: treat that as “stay out of the whole site.”

So a broken robots.txt can hide your entire site from AI. That is why this check exists. Most sites pass it.

What we look for

  • Pass: the file loads and we can read it.
  • Warn: there is no file. That is fine. You lose a couple of points because you cannot point robots at your sitemap without one.
  • Fail: we asked for the file and got an error, or no answer at all.

Whether you can fix this yourself

Yes, usually. It is a plain text file with a few lines in it. No code needed.

The one time you cannot is when your website vendor owns that file for you. Wix, Squarespace and most all-in-one platforms write robots.txt for you, and it already works. If you are on one of those and this check failed, the problem is somewhere else, most often a firewall. See Your server answers an AI crawler.

How to fix it

If you got a warning (no file)

Add one. Here is a complete file that says “everyone is welcome, and here is my map”:

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

Swap in your own domain. Then:

  • WordPress: install or open Yoast SEO or Rank Math. Both have a “robots.txt editor” under their Tools page. Paste the text and save.
  • Squarespace or Wix: you already have one, so there is nothing to do.
  • Shopify: Online Store, then Themes, then Edit code, then “Add a new template” and pick robots.txt. Shopify writes a good default for you.
  • Your own site or a developer: save the text above as robots.txt in the top folder of the site, next to your homepage.

If you got a fail (error)

Something is answering that address with an error. In order of how often we see it:

  1. A security plugin or firewall is blocking us. On WordPress, look in Wordfence, Sucuri or iThemes for a “block unknown bots” setting. Turn it off, or allow the AI crawlers by name.
  2. The file has a typo that makes the server choke. Open it, delete everything, paste the simple file above, save.
  3. The site is down or moving. Load your homepage in a browser. If that fails too, call your host.

How to check it worked

Open a browser and go to your site address with /robots.txt on the end. You should see plain text, not an error page. Then run the free check again.

Do not worry about

Your robots.txt does not need to name any AI crawler. A crawler that is not mentioned is allowed. Some other audit tools show that as a wall of red warnings. It is not a problem.