Try it yourself
This is the whole of one site's robots.txt. The site has been live for eight months and Google has indexed the homepage only.
Which line is doing the damage?
- User-agent: *
-
Keeping the admin area out of search is normal and sensible. It is not why the rest of the site is missing.
-
Found it. A bare slash means "every page on this site". One character wide, and it blocks the lot. It usually arrives with a staging site and is never taken out.
-
Pointing at the sitemap is exactly right. It just names a list of pages that crawlers have already been told to ignore.
Pick a line. Every one of them has an answer.
Show me the answer
Disallow: /
Found it. A bare slash means "every page on this site". One character wide, and it blocks the lot. It usually arrives with a staging site and is never taken out.
The written version
robots.txt: the file that lets search engines in →
How to check your own site for this, what to change, and how to confirm it worked.
Now the real thing
Run this check on your own site.
The same check, plus twenty-nine others, against your actual pages. Free, no account, about a minute.