Himanshu SharmaSoftware engineer
Let’s talk

SEO Basics · Lesson 16 of 32

robots.txt: controlling what crawlers fetch

Learn robots.txt syntax, what it can and cannot do, how it differs from noindex, and the classic mistakes that block a whole site from crawling.

11 minTechnical SEOUpdated 14 November 2026

Awaiting technical review. This lesson is readable but not yet listed in search engines.

By the end of this lessonRead and write a simple robots.txt file, and explain why it controls crawling and not indexing.

After this lesson you can read any robots.txt file, write a safe one for a small site, and explain clearly what it cannot do. It is a tiny file with an outsized reputation for breaking sites.

What it is

robots.txt is a plain text file at the root of a host: https://example.com/robots.txt. Before crawling, well-behaved crawlers fetch it and read the rules addressed to them. Per Google Search Central, its main purpose is managing crawler traffic: keeping crawlers off unimportant or heavy URLs, so your server is not overloaded and crawling goes to the pages that matter.

Each host needs its own: blog.example.com does not use the file from example.com.

The syntax

text
User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml
  • User-agent names the crawler a group of rules applies to; * means all.
  • Disallow lists a path prefix not to fetch. An empty Disallow: allows everything.
  • Allow makes an exception inside a disallowed path.
  • Sitemap gives the full address of a sitemap. See XML sitemaps.
  • Lines starting with # are comments, which we skip in examples here to keep them clean.

Paths are matched from the start, so Disallow: /admin/ covers /admin/users/. Google also understands * as a wildcard and $ as an end-of-URL marker in paths. Details are in Google's robots.txt specification; test unusual patterns before relying on them.

This site's real file is short: it disallows /admin/ and /api/ for all crawlers and names its sitemap index. Those are areas of no value to search, and the file leaves everything else open.

A group for a particular crawler

text
User-agent: Googlebot
Disallow: /internal-search/

User-agent: *
Disallow: /admin/

A crawler follows the most specific group that names it, not both. That surprises people: a rule in the * group does not also apply to Googlebot if Googlebot has its own group.

The important limitation

Google's documentation states plainly that robots.txt is not a mechanism for keeping a web page out of Google. If other sites link to a blocked URL, Google may still index the address and show it, often without a description, because it was never allowed to read the page.

Which tool for which goal?

Goal Right tool
Reduce crawling of low-value URLs robots.txt
Keep a page out of search results noindex meta tag or header on a crawlable page
Keep content private Authentication (a login)
Choose a preferred duplicate canonical or redirect

The trap: blocking a page in robots.txt and adding noindex to it. The crawler cannot fetch the page, so it never reads the noindex. See rendering and indexing.

Not a security tool

The file is public, so listing /secret-reports/ advertises it. Obeying it is voluntary: reputable crawlers do, bad ones may not. Never rely on it to protect anything.

Do not block what Google needs

Google can need CSS, JavaScript and images to render a page. Documentation allows blocking resource files only when the page still works without them. Blocking the folder that holds your scripts can leave Google with an incomplete view of the page; see JavaScript and SEO.

Launch-day disaster

Staging sites are often given Disallow: / to keep them out of crawlers. The file is then copied to production. The result: a live site that crawlers are asked not to fetch. Always check the live /robots.txt after a launch or migration, and set a reminder to recheck it. Search Console's robots.txt report (see Search Console basics) shows the file Google last read and any parse issues.

Status of the file itself

The file's HTTP status matters. Google documents that a missing file (a 404) is treated as no restrictions, while server errors on the file can cause Google to pause crawling the site for a while. So keep the file reachable and fast.

Common mistakes

  • Disallow: / left over from staging.
  • Using it to hide pages from search, instead of noindex.
  • Blocking CSS and JavaScript the page needs.
  • Putting the file in a sub-folder where no crawler looks.
  • Expecting a change to apply instantly; crawlers cache the file.
  • Listing secret URLs in it.
  • Misspelling directives so they are ignored.

Exercise

Write a robots.txt for a fictional shop at https://shop.example.com. Block the cart and checkout paths, keep everything else open, and reference a sitemap. Then say what you would use to keep an internal search results page out of search.

Show a sample
text
User-agent: *
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://shop.example.com/sitemap.xml

To keep internal search results out of search, add a noindex robots meta tag to those pages and leave them crawlable so the tag can be read, or avoid linking to them. Blocking only in robots.txt would not guarantee they never appear.

Key points

  • robots.txt sits at the root of a host and tells well-behaved crawlers which paths they may fetch.
  • It is for managing crawler traffic; Google states it is not a mechanism for keeping a page out of search.
  • Use noindex (on a crawlable page) or authentication to keep content out of results.
  • A stray Disallow: / can block a whole site; always test changes.

Check yourself

1. Where must robots.txt live?
2. What does Disallow: / under User-agent: * do?
3. You block /thank-you/ in robots.txt and it still appears in results with no description. Why?
4. Can robots.txt secure private data?