Crawl control · Intermediate · 15 minutes, validated
How to write a robots txt — It Governs Crawling. It Does Not Govern Indexing, and It Hides Nothing
You want to keep crawlers off the URLs that waste their time, without discovering three weeks later that you blocked the stylesheet Google renders the page with.
The short answer
robots.txt controls crawling and nothing else: it cannot remove a page from the index, it cannot hide anything from a browser, and a URL blocked in it can still be listed if other sites link to it. If you want a page out of the index, that is a noindex directive on the page or an X-Robots-Tag header, and the page has to stay crawlable for either to be read.
The useful job the file does is protecting crawl capacity — keeping requests away from faceted parameter URLs, internal search results and cart paths that generate infinite variations. The dangerous job it can do is block the CSS and JavaScript Google needs to render your pages, which makes a working site look broken to the crawler.
By the HostingFast team · Reviewed 24 August 2026
Intermediate
Level assumed
5
Stages start to finish
Free
What support costs
Proven
Measured on our own stack
Short file, long consequences. This assumes you can read an access log to find out where crawl requests are actually going, because writing rules without that is guessing at which paths are wasting anything.
Two rules govern everything below. Never disallow a path that must be crawled to be understood, and never rely on this file for anything you would describe as privacy. Both mistakes are common and neither announces itself.
What the file can and cannot do
A Disallow line is a request that a well-behaved crawler skips a path. It is not authentication, not authorisation and not a barrier of any kind: the URL is still reachable by anyone who types it, and the file itself is public, which means it doubles as a helpful index of the directories you would rather not advertise.
It also cannot deindex. Blocking a URL prevents the crawler reading it, which prevents it reading the noindex you put on the page — so a blocked page with a noindex stays in the index indefinitely. Unblock it, let the noindex be read, and block it afterwards if you still want to.
Never block the assets that render the page
Google renders pages with a real browser engine, fetching your CSS and JavaScript to do it. Disallowing /wp-content/ or a build directory means the crawler evaluates your page without its stylesheet or its scripts, and it judges what it sees. Pages that look correct to you can look empty or broken to it.
Prove it rather than assume it: the URL Inspection tool renders the page as the crawler sees it and lists the resources it could not load. That screenshot is the definitive answer to whether a rule is costing you, and it takes about thirty seconds to get.
Spend the file on genuine crawl waste
Start almost empty — allow everything, name the sitemap — and add exclusions only where you can point at the waste. The candidates are consistent: internal search result URLs, faceted filter parameters that multiply combinatorially, cart and checkout paths, and print or tracking variants of pages that already exist.
Find them in the access log rather than by intuition. Filter for the crawler's user agent, group by path, and the pattern of wasted requests is normally obvious in the first screen of output. A rule you can justify from a log is worth ten rules copied from a template.
Know which file is actually being served
WordPress generates a virtual robots.txt when no physical file exists. Upload a real one and it takes precedence, which is how sites end up editing a plugin setting for a week with no effect on the file the crawler is reading. Load /robots.txt in a browser and compare it against what you think you configured.
Then validate. A syntax error fails silently and it fails toward over-blocking, so use the tester in Search Console on a handful of real URLs — one you expect allowed, one you expect blocked. Keep a copy of the file in your repository and in the daily backup, because a bad edit to a two-line file can be the most expensive change on the site.
Where the file lives on our platform
The file is served straight off the document root by the web server, so it is a File Manager or SSH edit away and there is nothing to purge afterwards. Raw access logs are in cPanel, which is where the evidence for every rule you write should come from.
Your existing site moved across by our engineers, free of charge — including the server configuration files most migrations quietly leave behind.
- Raw access logs in the panel, for evidence rather than guesswork
- File Manager and SSH, so a two-line file is a two-line edit
- Daily backups on every plan, with restores you run yourself
- Free migration that brings the server-level files across too
Why HostingFast
Standard on every plan
The distinction stated up front
Crawling versus indexing versus access control, because conflating them is where every expensive robots.txt mistake begins.
Rendering protected
CSS and JavaScript stay crawlable, and you verify it with the rendered screenshot rather than trusting the rule.
Rules with evidence behind them
Every exclusion is justified from the access log, so the file matches your site rather than somebody's template.
The virtual-file trap named
WordPress serves a generated file until a real one exists, which explains a week of edits with no visible effect.
Validated on real URLs
One path you expect allowed and one you expect blocked, checked in the tester, because syntax errors fail quietly.
Reversible in seconds
The file sits in the document root with a daily backup behind it, so a bad edit is a one-minute recovery.
Quick Start
From order to online
- 1
Find out which file is being served
Load /robots.txt in a browser. If WordPress is serving a virtual file, a plugin setting is your editor; if a physical file exists, it wins and the plugin does nothing.
- 2
Start from allow-everything plus the sitemap
Most sites need very little. Permit the whole site, add the Sitemap line, and treat every further rule as something you have to justify.
- 3
Pull the crawl waste out of the log
Filter the access log for the crawler's user agent, group by path, and look for parameter explosions and internal search URLs. Write rules against what you find.
- 4
Confirm nothing that renders is blocked
Run URL Inspection and read the rendered screenshot and the blocked-resources list. A stylesheet behind a Disallow makes a good page look broken.
- 5
Validate, then keep a copy
Test one allowed URL and one blocked URL in the tester. Commit the file to your repository so the next bad edit has something to be compared against.
Built In
Loaded onto every plan
- cPanel — the control panel most of the industry already runs
- SSH, Git and Composer on the developer-focused plans
- Daily backups on every plan, with restores you run yourself from the panel
- Your existing site moved across by our engineers, free of charge
- NVMe SSD storage on every tier, including the entry plan
- LiteSpeed caching compiled into the server, not bolted on by plugin
- Per-site PHP version switching from the control panel
- Staging environments for rehearsing a change before it ships
- Free SSL on every plan, reissued automatically before it can expire
- Human support on duty every hour of every day
Frequently Asked
What people ask us most often
Does a physical robots.txt override the one WordPress generates?
Yes. WordPress only serves its virtual file when no real file exists in the document root, so the moment you upload one, plugin settings that claim to edit robots.txt stop having any effect. If your changes are not showing up, load /robots.txt in a browser and then look in File Manager for a physical file somebody added months ago.
How do I actually stop a page being indexed, if robots.txt cannot?
A noindex meta tag on the page, or an X-Robots-Tag header for non-HTML files. Crucially the page must stay crawlable for either to be read — blocking it in robots.txt guarantees the noindex is never seen, which is why blocked pages sometimes sit in the index for months with no snippet.
Which crawl traps show up first in an access log?
Faceted filter parameters, because they multiply combinatorially and a crawler will happily explore thousands of them. After that: internal search result URLs, calendar or archive pagination with no end, and tracking parameters creating a duplicate of every page. Group the log by path and the pattern is usually visible immediately.
Is Crawl-delay worth putting in the file?
Google ignores it, so it does nothing for the crawler most people are writing the file for. Some other crawlers honour it. If Googlebot's request rate is genuinely a load problem, the answer is the crawl-rate setting in Search Console and a faster origin — a cached response that never enters PHP costs you almost nothing to serve.
Keep reading
How to Reduce Server Response Time
Find where the pause before the first byte is actually spent, then remove it layer by layer — intermediate, roughly an afternoon.
How to Change Nameservers
Hand an entire zone to different nameservers without dropping a record, with both clocks watched — beginner, roughly 5 minutes plus propagation.
TXT Record (Glossary)
The record type behind verification, SPF and a good deal of quiet DNS plumbing.
Joomla Hosting
Joomla in one click, with your existing site migrated across free of charge.
Web Hosting
cPanel hosting on NVMe behind a LiteSpeed cache — SSL, migration and a year-one domain included.
Changing hosts? Run through our checklist first.
A straightforward sequence for a switch your visitors never feel: which files move first, how to shift email across without losing a single message, the right moment to repoint DNS, and the two mistakes behind almost all the downtime we get asked to rescue.
Edit it where the logs are too.
cPanel with raw access logs, SSH on developer plans, daily backups you restore yourself, and free migration from wherever you are now.
View Joomla Hosting plans