Search engines crawl billions of pages every day, but not all of them get equal attention. Crawl resources are finite, and sites that don’t actively manage how bots move through their content often end up with crawlers spending time on junk URLs while genuinely important pages wait.
Robots.txt is one of the oldest tools in technical SEO — over thirty years old at this point — and it’s still doing useful work. A small text file sitting at the root of a domain, with a few lines of directives, and search engines have clear instructions about where to go and where not to bother.
This guide covers how robots.txt actually works, why it still matters, and how to avoid the mistakes that turn a simple file into a serious SEO problem.
What Is A Robots.txt File?
A robots.txt file lives in the root directory of a website and tells search engine crawlers which pages they’re welcome to access and which they should skip.
The building analogy holds up: it’s less like a locked door and more like a sign at the entrance directing visitors where to go. A site might let crawlers roam freely through blog posts and product pages while steering them away from account areas and checkout flows.
Here’s what a basic file looks like:
User-agent: *
Disallow: /account/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
Every crawler gets the same message: most of the site is accessible, account and checkout pages aren’t, and the sitemap is here.
Why Robots.txt Matters For SEO
Every website gets a finite amount of crawling attention from search engines. This is the crawl budget — and how that budget gets spent has real consequences for how quickly pages get discovered and indexed.
Bots that spend time crawling duplicate URLs, parameter-generated filter pages, and tracking variants are bots not crawling the content that actually matters. Robots.txt redirects that attention.
What proper robots.txt usage does in practice:
- points crawlers toward high-value pages instead of low-value noise
- keeps duplicate and thin content out of the crawl
- reduces unnecessary server load from bot traffic
- improves overall crawling efficiency
For large ecommerce or content-heavy sites, the difference in indexing speed can be significant. A new product page or article getting indexed in hours rather than days is often a direct result of crawl budget being managed well.
Key Robots.txt Directives Explained
The file relies on a small set of directives. Each one does something specific.
User-Agent
Specifies which crawler a rule applies to. A rule can target one bot specifically:
User-agent: Googlebot
Disallow: /private/
Or apply to every crawler at once:
User-agent: *
Disallow
Tells crawlers to stay away from a specific page or directory:
User-agent: *
Disallow: /internal-search/
Internal search result pages generate enormous numbers of low-value URL variations. This keeps them out of the crawl entirely.
Allow
Carves out an exception inside a broader restriction. If an entire directory is blocked but one page within it needs to stay accessible:
User-agent: *
Disallow: /members/
Allow: /members/login
Members area blocked, login page open.
Sitemap
Points crawlers directly to the XML sitemap:
Sitemap: https://www.example.com/sitemap.xml
Including this line means crawlers don’t have to discover important pages through links alone — they have a direct map.
Common Ways Robots.txt Is Used
Blocking Internal Search Pages
Search functions can generate thousands of URL variations from a single interface. A photography marketplace produces URLs like:
example.com/search?query=portrait+lens
Multiplied across every search term any user has ever typed, that’s an enormous amount of crawlable noise with almost no value in search results. One rule blocks all of it:
User-agent: *
Disallow: /search
Blocking Filter And Parameter URLs
Ecommerce filtering creates URL variants constantly:
example.com/shoes?color=red
example.com/shoes?sort=price
Near-identical content, different URLs, wasted crawl budget. Blocking specific parameters is the fix:
User-agent: *
Disallow: *color=
Disallow: *sort=
Blocking User Account Areas
Login pages, dashboards, order history — none of these belong in search results. Keeping them out of the crawl is straightforward:
User-agent: *
Disallow: /account/
Blocking File Types
Sites that want to keep downloadable files out of the crawl can block by file extension:
User-agent: *
Disallow: /*.pdf$
Whether PDFs have organic search value worth keeping depends on the site.
Blocking AI Crawlers And Scrapers
An increasingly relevant use case: blocking bots that scrape content for AI model training. Specific crawlers can be targeted individually:
User-agent: GPTBot
Disallow: /
It’s worth thinking through carefully, though — blocking every AI crawler may reduce visibility in AI-powered search experiences. The right decision depends on what the site is trying to accomplish, and there’s no single right answer that applies everywhere.
Common Robots.txt Mistakes To Avoid
The file is simple enough that mistakes happen easily. Some of them are significant.
Blocking Your Entire Website
User-agent: *
Disallow: /
This blocks every crawler from the entire site. It’s sometimes used on staging environments — and it’s catastrophic on a live site. Any time a site moves from staging to production, checking this is worth adding to the launch checklist.
Blocking Important Resources
Search engines need access to CSS, JavaScript, and image files to render pages and understand what’s on them. Blocking these — even by accident, as part of a broader directory block — prevents accurate page interpretation. The ranking effects tend to show up quietly over time rather than all at once.
Confusing Blocking With Removing From Index
Robots.txt prevents crawling. It does not prevent indexing. A blocked page that’s linked to from external sites can still appear in search results — search engines know the URL exists; they just can’t read the content. Getting a page out of results entirely requires a noindex tag, not a robots.txt rule. This is one of the most commonly misunderstood distinctions in technical SEO.
Forgetting Subdomain Robots.txt Files
Search engines treat subdomains as separate websites. The robots.txt at example.com does nothing for blog.example.com or shop.example.com. Each subdomain needs its own file. On larger sites with multiple subdomains, this gets missed more often than it should.
How To Test A Robots.txt File
Testing before publishing changes is not optional. A misconfigured file can quietly block crawlers from entire sections of a site, and the consequences often don’t show up immediately.
Tools worth using:
- Google Search Console robots.txt tester — tests specific rules against specific URLs
- Screaming Frog — crawls the site the way a search engine would and surfaces what’s blocked
- Online robots.txt validators — quick syntax checks before anything goes live
The goal is simple: confirm that pages that should be accessible are, and pages that should be blocked actually are.
Final Thoughts
Robots.txt has been around for thirty years because it solves a real problem simply. Managed well, it keeps crawl budget focused on the pages that deserve attention and away from the ones that don’t.
The principle to keep in mind: block what’s genuinely low-value, leave valuable content fully accessible, and never restrict the resources crawlers need to render pages correctly. Sites that handle crawl behaviour carefully tend to index new content faster and hold organic visibility more consistently over time.
Frequently Asked Questions
What does robots.txt do?
It gives search engine crawlers instructions about which parts of a site they should and shouldn’t access.
Does robots.txt prevent indexing?
No — only crawling. A blocked page can still be indexed if other sites link to it. A noindex tag is what removes a page from search results.
Where does robots.txt need to be located?
The root directory of the domain: example.com/robots.txt. It won’t be recognised anywhere else.
Should every website have one?
Yes. Even simple sites benefit—at minimum to point crawlers to the sitemap and provide basic direction.
Can robots.txt stop malicious bots?
Not reliably. Legitimate crawlers follow the rules. Malicious bots typically don’t.
