Is Google allowed to read your site?

Worth knowing

robots.txt controls crawling, not indexing.

This is the mistake that catches most people. Blocking a page in robots.txt stops Google reading it, but Google can still list the address if other pages link to it, usually with no description. And because Google cannot read the page, it never sees a noindex tag on it either.

So: to keep a page out of search results, leave it crawlable and add a noindex tag. Use robots.txt for things that should not be fetched at all, like admin areas, endless filter pages and internal search results.

How the rules are read

• Each crawler follows one group. The group that names it most specifically, or the User-agent: * group if none does. A crawler with its own group ignores the * rules entirely.
• The longest matching rule wins. Allow: /shop/sale/ beats Disallow: /shop/ for addresses under /shop/sale/. If two rules are the same length, Allow wins.
• * matches anything, $ marks the end. Disallow: /*.pdf$ blocks every PDF.
• Disallow: with nothing after it allows everything. Disallow: / blocks everything. One character apart.
• Paths are case-sensitive. /Private/ and /private/ are different rules.

Search engines and AI crawlers are separate choices

You can stay in Google and Bing while asking AI companies not to train on your pages. These lines do that, and leave search alone:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Two things to weigh first. These are requests: reputable companies honour them, but nothing enforces them. And blocking the search crawlers of AI services, such as OAI-SearchBot or PerplexityBot, can keep you out of the answers those services give, which is where some customers now look for local businesses.

Where to edit robots.txt

• WordPress: WordPress makes a basic one itself. To change it, use your SEO plugin’s robots.txt or file editor (Yoast and Rank Math both have one), or upload your own file to the site’s root folder.
• Wix: in the SEO settings there is a robots.txt editor. Wix’s default is sensible; change it only for a reason.
• Squarespace: the file cannot be edited. There is a setting to ask AI crawlers to stay out, under the site’s crawler settings.
• Shopify: edit the robots.txt.liquid template in your theme’s code. Shopify’s default already blocks checkout and cart pages, which is what you want.

After changing it, test it here, then use the indexability checker on an important page to confirm nothing else stands in the way.