Control starts at one file
After this lesson you can write and explain a robots.txt that treats search visibility and AI training as two separate decisions.
The robots.txt file lives at the root of a host (for example /robots.txt). It lists groups of rules, each addressed to a crawler by a user-agent token, with Allow and Disallow paths. Well-behaved crawlers read it before fetching. It is public and voluntary: it is not a lock. Private material needs a login, not a robots.txt line. Also, a disallowed URL can still appear in results as a bare link if others link to it; use noindex (which requires the page to be crawlable so the tag is seen) when you need removal.
Purpose matters: three kinds of fetching
Vendors increasingly separate:
- Search crawling, to build an index or surface pages in search answers.
- Training crawling, to collect content that may be used to train models.
- User-triggered fetching, when a person asks an assistant to open a page.
Different tokens, different consequences. Below is what each vendor documents, read in 2026-11. Vendors change tokens and behaviour, so check the pages before editing.
What vendors document
Google. Googlebot is the crawler for Search; Google's documentation says its common crawlers always respect robots.txt for automatic crawls. Google-Extended is a robots.txt token only, with no user agent string of its own. Per Google, it controls whether crawled content may be used to train future Gemini models and for grounding in Gemini apps and certain developer products, is not a ranking signal, and does not affect inclusion in Google Search. To limit Search features, use the snippet and noindex controls instead (lesson What Google documents about AI Overviews and AI Mode).
OpenAI. Its bots documentation lists OAI-SearchBot (for ChatGPT's search features), GPTBot (content that may be used to train foundation models) and ChatGPT-User (visits when a user asks, which OpenAI notes may not be governed by robots.txt). OpenAI says each setting is independent, and that changes to robots.txt for search can take about 24 hours to take effect.
Anthropic. Its crawler help article describes three tokens: ClaudeBot (training data collection), Claude-User (fetches on a user's request) and Claude-SearchBot (improving search results). Each can be allowed or blocked separately. I read this through search-result summaries of the official article, not the full page, so confirm the details there. Older tokens such as anthropic-ai and Claude-Web are reported as no longer listed.
Microsoft Bing. Bing's crawler is bingbot; its documentation is on the Bing Webmaster side and I did not retrieve token details for AI features, so check it before writing rules.
Example files
Allow everything except admin areas and keep AI search access, but opt out of OpenAI and Anthropic training crawlers:
User-agent: *
Disallow: /admin/
Disallow: /api/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlBecause the search and user tokens (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User) have no group of their own, they fall under the * group and are allowed except for the two disallowed folders.
Keep Gemini from using the site for training and grounding without touching Search:
User-agent: Google-Extended
Disallow: /Block one folder for all crawlers while allowing one file inside:
User-agent: *
Disallow: /private/
Allow: /private/public-note.htmlImportant syntax facts (established, from the Robots Exclusion Protocol as published): a crawler obeys the most specific group that matches its token; groups are separated by blank lines; paths are case-sensitive; an empty Disallow: allows everything. A crawler matching a named group ignores the * group, so repeat shared rules in each named group.
Is blocking training wise?
That is a business decision, not a technical one. Considerations: do you want your writing used to train models, is your content paid or licensed, and do you care about attribution? Blocking training crawlers does not, per the vendors' documentation, remove you from search features, but it cannot undo past collection. Blocking search crawlers does mean those products' search answers will not feature your pages (OpenAI says sites that opt out will not appear in ChatGPT search answers, though navigational links may still appear).
Do not forget the network layer
Google's AI features page says to allow crawling not only in robots.txt but in any CDN or hosting infrastructure. A firewall rule or bot-protection setting that blocks crawlers can silently make a site invisible even though robots.txt is open. Test by requesting a page with a crawler's user agent string, and check server logs. Note: user-agent strings can be faked, so verify important crawlers using the vendor's published IP ranges or reverse DNS where they provide them.
This site
This site's robots.txt, as of 2026-11-14, contains User-agent: *, Disallow: /admin/, Disallow: /api/ and a Sitemap line. It names no AI crawlers. Whether to opt out of any training crawler is Himanshu's decision; mark it for confirmation. Do not paste the examples above into production without that decision.
Common mistakes
- Using robots.txt to hide private pages.
- Disallowing
/for*while testing and forgetting it, which removes you from everything. - Blocking the search token when you meant to block the training token.
- Adding a named group and forgetting that it replaces the
*rules for that crawler. - Copying a long "block all AI bots" list from a blog without checking which tokens still exist.
Exercise
- Write a robots.txt for a fictional bookshop that wants ChatGPT and Claude search access but opts out of their training crawlers and Gemini training.
- Explain in one sentence each what you chose and what you did not change.
- Find your own site's robots.txt and add a comment-free list of the tokens it names. Then check each token against the vendor page.
Show a sample answer
Groups: GPTBot, ClaudeBot and Google-Extended each with Disallow: /; no group for OAI-SearchBot, Claude-SearchBot or Googlebot, so they follow *. Remember to verify the tokens on each vendor's current page before publishing.
Previous: Product and LocalBusiness markup. Next: llms.txt and machine-readable files.
Key points
- robots.txt tells crawlers what they may fetch; it is a voluntary convention, not security.
- Vendors separate purposes: OpenAI documents OAI-SearchBot (search), GPTBot (training) and ChatGPT-User (user requests).
- Google's Google-Extended is a control token for Gemini training and grounding, not a ranking signal and not a Search crawler.
- Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot, each with its own token (as of 2026-11, check the article).
- Blocking a search crawler removes you from that product's search answers; blocking a training crawler is a different choice.