Fix library · Technical
robots.txt: the file that lets search engines in
robots.txt is a plain text file at the root of your site with a few lines in it, and it is read before anything else on your domain. Most of the time it is uneventful. Occasionally it contains two words that remove an entire business from Google, left behind from the week the site was being built.
robots.txt is blocking your whole site
What SEOPert detected
Your robots.txt tells every search engine not to crawl any page on the site.
What this means
Your robots.txt tells all search engines not to crawl any page. Unless the site is meant to be private, this keeps you out of Google.
How to fix it
- Open yourdomain.com/robots.txt and look for “Disallow: /” under “User-agent: *”.
- Remove that line (or change it to “Disallow:” with nothing after it). Re-run your audit to confirm.
On WordPress
- Go to Settings → Reading and untick “Discourage search engines from indexing this site”.
- If the block remains, edit robots.txt with the Yoast SEO or Rank Math file editor, or remove a physical robots.txt file from your hosting account.
On Shopify
- Online Store → Preferences: check the store is not password-protected. If your theme has a custom robots.txt.liquid, ask your developer to remove the global block.
On Wix
- Go to your SEO settings, open the robots.txt editor, and remove the “Disallow: /” rule.
If you are handing this to a developer
Remove the global "Disallow: /" for User-agent: * in robots.txt.
Check it worked. Open yoursite.com/robots.txt: there should be no “Disallow: /” under “User-agent: *”.
No robots.txt file found
What SEOPert detected
No robots.txt file was found at the root of your domain.
What this means
robots.txt is a small file that tells search engines how to crawl your site and where your sitemap is.
How to fix it
- Ask your host or developer to add a robots.txt file at the root of your site that allows crawling and lists your sitemap address.
On WordPress
- WordPress normally serves a robots.txt automatically. If it is missing, Yoast SEO and Rank Math both include a robots.txt editor.
On Shopify, Wix, Squarespace and Webflow
- Your platform normally provides robots.txt automatically. If this still appears, contact the platform’s support.
If you are handing this to a developer
Serve /robots.txt with "User-agent: *", "Allow: /" and a "Sitemap:" line.
Check it worked. Open yoursite.com/robots.txt in a browser: it should show a short text file.
Related
Check your own site
See which of these apply to you.
The free audit checks up to 10 pages and tells you exactly what it found, in the same plain language as these guides. No account, no card.