A robots.txt file is a small text file that tells search engines which parts of your website they may visit, and one wrong line in it can hide your whole site from Google. It is a set of polite instructions, not a lock: good crawlers obey it, but it cannot keep a page secret or remove it from search results.
This robots.txt seo guide explains the file in plain English, the accidents that quietly cost small businesses traffic, and a safe starter file you can copy.
Robots.txt SEO Guide: What the File Actually Does
Your robots.txt file lives at the top of your website: add /robots.txt to your domain and read it yourself. It speaks to crawlers using just a few instructions: User-agent says which crawler the rule is for, Disallow says which paths it may not fetch, Allow makes an exception, and Sitemap points to your XML sitemap.
The key distinction: crawling means fetching a page, indexing means storing it in search results, and ranking means deciding where it appears. Robots.txt controls only the first. Google’s own introduction to robots.txt puts it bluntly: the file is not a mechanism for keeping a web page out of Google. A blocked page that other sites link to can still show up in results, usually with no description — which looks worse than either outcome you wanted.
Three limits matter. The file is public, so never list a secret folder in it. It is voluntary — Google and Bing follow it, but scrapers often ignore it, so it is no substitute for a password. And it covers one host only: a subdomain needs its own file. Google supports only four fields — user-agent, allow, disallow and sitemap — so old tricks like crawl-delay are ignored, and a noindex line inside robots.txt has not worked for Google for years. Robots.txt and your sitemap do opposite jobs — one holds Google back, the other invites it in — as our sitemap guide explains.
The One Line That Hides Your Whole Website
Ask any SEO consultant for their favourite horror story and you will hear this one: a business launches a beautiful new website — and then nothing. No calls from Google, no enquiries, no traffic. Weeks pass before someone types the domain followed by /robots.txt and finds this:
- User-agent: *
- Disallow: /
Those two lines tell every crawler to stay away from everything. They belong on a staging copy of a site, never on the live one — but they travel with the site at launch more often than anyone admits, because the developer forgets to remove them and the owner has no reason to look. If your site has ever gone strangely quiet after a redesign or a move to a new host, this is the first place to look, before you spend a penny on anything else.
The fix takes seconds: replace Disallow: / with Disallow: left blank, which means nothing is off limits. Just do not expect traffic back the same afternoon. Google caches your robots.txt file for up to 24 hours, and your pages still need to be crawled again after that — so give it days, and recheck the file after every launch or host move.
Five Everyday Accidents (and the Fix for Each)
Beyond the whole-site block, smaller mistakes quietly eat away at rankings. Here are the five we see most on small business sites.
1. Blocking a folder to hide it
A solicitor blocks /client-documents/ hoping to keep files private — but the folder name is now public, and linked documents inside can still be indexed without a description. Fix: password-protect anything private.
2. Blocking a page and noindexing it at the same time
You add a noindex tag to a page you want out of search, then also block it in robots.txt for good measure. Google then cannot fetch the page, so it never sees the noindex tag. Fix: choose one method and leave the page crawlable.
3. The prefix trap
Robots.txt matches from the start of the path, so Disallow: /blog blocks /blog/ and also /blog-tips and /blogger.html. Fix: use the trailing slash — Disallow: /blog/ — when you mean the folder only.
4. Blocking the files Google needs to see your pages properly
Google renders pages like a browser. Block /wp-content/ or /assets/ and it may not load your stylesheets or scripts, so it judges a broken version of your content. Fix: never block theme, CSS and JavaScript folders.
5. Wildcards that catch too much
Disallow: /*? blocks every URL containing a question mark — killing endless shop filter combinations, but also useful paginated pages. Fix: aim narrowly, for example Disallow: /*?sort=, and test real URLs first.
What Is Worth Blocking, and What You Should Never Block
What should a small business actually block? Internal search results (like /?s= on WordPress) can generate unlimited thin pages. Endless filter combinations on a shop waste crawl effort while your real category pages wait. Cart, checkout and account pages are useless in search and personal by nature. Admin and login areas are fine to block too, as long as they are also password-protected, because robots.txt alone secures nothing.
And what should you never block? Anything a customer or Google needs: your CSS, JavaScript and images; pages you want removed from search (use noindex instead); and pages carrying a canonical tag pointing elsewhere, because Google must crawl the duplicate to see where it should point. If your site has already vanished, our guide to why a website stops showing on Google covers the other usual suspects.
A Safe Starter File You Can Copy
For a typical small business site, this is a sensible starting point — adjust the paths and delete lines that do not apply:
- User-agent: *
- Disallow: /search
- Disallow: /cart/
- Disallow: /checkout/
- Disallow: /account/
- Sitemap: https://yourbusiness.co.uk/sitemap.xml
Notice what is not in there: no block on your content folders, no crawl-delay, no noindex line, and no Disallow: /. On WordPress, a virtual robots.txt already blocks /wp-admin/ while allowing admin-ajax.php, so you usually need no physical file — and your SEO plugin’s Tools section usually offers a robots.txt editor. One more WordPress trap: under Settings, then Reading, the “Discourage search engines from indexing this site” checkbox is meant for sites under construction. On a live site it must be unchecked — ticked, it tells every crawler to go away. Our plain-English guide to what technical SEO actually means for business owners puts robots.txt among the other foundations deciding whether Google can read your site.
AI Crawlers in 2026: A Separate Decision
AI companies now run several crawlers with different jobs: training-data collectors, index builders for assistants’ answers, and fetchers that load one page because a customer pasted your link into a chat. Blocking one does not block the others, and blocking the wrong one can cost you visibility in AI answers.
The classic mistake is blocking Googlebot to stay out of AI training — but Googlebot is what puts you in Google Search, so blocking it removes you from search entirely. Google’s AI token is separate, Google-Extended, and Google says blocking it does not affect Search inclusion. So allow the search and answer crawlers, and set training rules with their specific tokens, rather than copying an old block list. This is a business decision about where you want to be seen, not an SEO fix.
Test Before You Publish, and What Recovery Really Takes
Before any change goes live, run this 60-second check. List the URLs that must stay crawlable: home page, key service pages, top posts, and their CSS and JavaScript files. Test each against the new rules — the longest matching rule wins, and paths are case-sensitive, so /Admin/ and /admin/ differ. And confirm /robots.txt itself returns a proper response, not a server error, which can pause crawling for hours.
After publishing, open the robots.txt report in Search Console to see the version Google fetched, then use URL Inspection on one or two key pages to confirm Googlebot can reach them. If you just fixed a bad block, ask for a recrawl there. And be honest about timing: the cache clears within about a day, but pages are recrawled on Google’s schedule, so recovery takes days or weeks — never a guaranteed date.
Conclusion
Robots.txt is the smallest file on your website and one of the easiest to get wrong. Open yourdomain.com/robots.txt today: if you see Disallow: / on a live site, fix it now. If the file looks sensible, leave it alone — a short, correct file beats a long, clever one.
FAQs
Where can I find a reliable robots.txt seo guide?
A reliable robots.txt seo guide starts with Google’s own Search Central documentation, which explains exactly how Google reads the file. Supplement the official guide with a plain-English walkthrough written for small businesses, and ignore any guide that promises ranking boosts from robots.txt alone.
Will blocking a page in robots.txt remove it from Google?
Blocking a page in robots.txt will not reliably remove the page from Google, because a blocked URL can still be indexed from links pointing to it. The blocked page usually appears in search results without a description, which looks unprofessional. Use a noindex tag or password protection instead.
How do I check my own robots.txt file?
Checking a robots.txt file takes under a minute: type your domain followed by /robots.txt into a browser and read the rules. Look for Disallow: / under User-agent: *, which blocks the whole site, and check that your home page and service pages are not covered by any rule.
Does every small business website need a robots.txt file?
Not every small business website needs a robots.txt file, because without one Google assumes it may crawl everything. A robots.txt file becomes useful when a site has low-value URLs worth keeping crawlers away from, such as internal search results or checkout pages.
How long does a robots.txt fix take to work?
A robots.txt fix takes at least a day to be picked up, because Google usually caches the file for up to 24 hours. After the cache clears, affected pages still need to be recrawled, so visible recovery typically takes days or weeks rather than hours.
A reliable robots.txt seo guide starts with Google’s own Search Central documentation, which explains exactly how Google reads the file. Supplement the official guide with a plain-English walkthrough written for small businesses, and ignore any guide that promises ranking boosts from robots.txt alone.
Blocking a page in robots.txt will not reliably remove the page from Google, because a blocked URL can still be indexed from links pointing to it. The blocked page usually appears in search results without a description, which looks unprofessional. Use a noindex tag or password protection instead.
Checking a robots.txt file takes under a minute: type your domain followed by /robots.txt into a browser and read the rules. Look for Disallow: / under User-agent: *, which blocks the whole site, and check that your home page and service pages are not covered by any rule.
Not every small business website needs a robots.txt file, because without one Google assumes it may crawl everything. A robots.txt file becomes useful when a site has low-value URLs worth keeping crawlers away from, such as internal search results or checkout pages.
A robots.txt fix takes at least a day to be picked up, because Google usually caches the file for up to 24 hours. After the cache clears, affected pages still need to be recrawled, so visible recovery typically takes days or weeks rather than hours.