After this lesson you can read any robots.txt file, write a safe one for a small site, and explain clearly what it cannot do. It is a tiny file with an outsized reputation for breaking sites.
What it is
robots.txt is a plain text file at the root of a host: https://example.com/robots.txt. Before crawling, well-behaved crawlers fetch it and read the rules addressed to them. Per Google Search Central, its main purpose is managing crawler traffic: keeping crawlers off unimportant or heavy URLs, so your server is not overloaded and crawling goes to the pages that matter.
Each host needs its own: blog.example.com does not use the file from example.com.
The syntax
User-agent: *
Disallow: /admin/
Disallow: /api/
Sitemap: https://example.com/sitemap.xmlUser-agentnames the crawler a group of rules applies to;*means all.Disallowlists a path prefix not to fetch. An emptyDisallow:allows everything.Allowmakes an exception inside a disallowed path.Sitemapgives the full address of a sitemap. See XML sitemaps.- Lines starting with
#are comments, which we skip in examples here to keep them clean.
Paths are matched from the start, so Disallow: /admin/ covers /admin/users/. Google also understands * as a wildcard and $ as an end-of-URL marker in paths. Details are in Google's robots.txt specification; test unusual patterns before relying on them.
This site's real file is short: it disallows /admin/ and /api/ for all crawlers and names its sitemap index. Those are areas of no value to search, and the file leaves everything else open.
A group for a particular crawler
User-agent: Googlebot
Disallow: /internal-search/
User-agent: *
Disallow: /admin/A crawler follows the most specific group that names it, not both. That surprises people: a rule in the * group does not also apply to Googlebot if Googlebot has its own group.
The important limitation
Google's documentation states plainly that robots.txt is not a mechanism for keeping a web page out of Google. If other sites link to a blocked URL, Google may still index the address and show it, often without a description, because it was never allowed to read the page.
Which tool for which goal?
| Goal | Right tool |
|---|---|
| Reduce crawling of low-value URLs | robots.txt |
| Keep a page out of search results | noindex meta tag or header on a crawlable page |
| Keep content private | Authentication (a login) |
| Choose a preferred duplicate | canonical or redirect |
The trap: blocking a page in robots.txt and adding noindex to it. The crawler cannot fetch the page, so it never reads the noindex. See rendering and indexing.
Not a security tool
The file is public, so listing /secret-reports/ advertises it. Obeying it is voluntary: reputable crawlers do, bad ones may not. Never rely on it to protect anything.
Do not block what Google needs
Google can need CSS, JavaScript and images to render a page. Documentation allows blocking resource files only when the page still works without them. Blocking the folder that holds your scripts can leave Google with an incomplete view of the page; see JavaScript and SEO.
Launch-day disaster
Staging sites are often given Disallow: / to keep them out of crawlers. The file is then copied to production. The result: a live site that crawlers are asked not to fetch. Always check the live /robots.txt after a launch or migration, and set a reminder to recheck it. Search Console's robots.txt report (see Search Console basics) shows the file Google last read and any parse issues.
Status of the file itself
The file's HTTP status matters. Google documents that a missing file (a 404) is treated as no restrictions, while server errors on the file can cause Google to pause crawling the site for a while. So keep the file reachable and fast.
Common mistakes
Disallow: /left over from staging.- Using it to hide pages from search, instead of noindex.
- Blocking CSS and JavaScript the page needs.
- Putting the file in a sub-folder where no crawler looks.
- Expecting a change to apply instantly; crawlers cache the file.
- Listing secret URLs in it.
- Misspelling directives so they are ignored.
Exercise
Write a robots.txt for a fictional shop at https://shop.example.com. Block the cart and checkout paths, keep everything else open, and reference a sitemap. Then say what you would use to keep an internal search results page out of search.
Show a sample
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://shop.example.com/sitemap.xmlTo keep internal search results out of search, add a noindex robots meta tag to those pages and leave them crawlable so the tag can be read, or avoid linking to them. Blocking only in robots.txt would not guarantee they never appear.
Key points
- robots.txt sits at the root of a host and tells well-behaved crawlers which paths they may fetch.
- It is for managing crawler traffic; Google states it is not a mechanism for keeping a page out of search.
- Use noindex (on a crawlable page) or authentication to keep content out of results.
- A stray Disallow: / can block a whole site; always test changes.