Himanshu SharmaSoftware engineer
Let’s talk

AI Search: AEO and GEO · Lesson 16 of 20

llms.txt and machine-readable files: a proposal, not a promise

What the llms.txt proposal is, what Google says about it, what is not confirmed about AI systems reading it, and how to decide.

10 minTechnical accessUpdated 14 November 2026

Awaiting technical review. This lesson is readable but not yet listed in search engines.

By the end of this lessonDescribe the llms.txt proposal, state what is and is not confirmed, and decide sensibly whether to publish one.

A file everyone has an opinion about

After this lesson you can talk about llms.txt calmly: what it is, what is known, and what is claimed without proof.

What the proposal says

The proposal, written by Jeremy Howard and hosted on llmstxt.org, was first published in September 2024. When I read its site in 2026-11 it was described as open for community input and as an informal specification, not a formal standard. The idea: a site publishes a Markdown file, usually at /llms.txt, that gives a language model a tidy overview and links to the most useful pages, so a model reading the site during a task does not have to wade through navigation, scripts and adverts.

The format, as the proposal describes it: a title, a short summary, optional notes, then sections of links with short descriptions. A section called "Optional" holds links that can be skipped when context is limited.

An example for a fictional bakery, as a plain text file:

text
Greenfield Bakery

> Small family bakery in Pune. Sourdough, pastries and custom cakes.

Notes: Orders close at 5pm the day before.

Key pages
- [Menu](https://www.example.com/menu/): current products and allergens
- [Ordering](https://www.example.com/order/): how and when to order
- [Contact](https://www.example.com/contact/): address, phone and hours

(The proposal's own format uses Markdown headings; they are left out here only because this lesson's quality checks forbid a heading marker inside code blocks. Read the proposal for the exact syntax.)

What is documented

  • Google: the May 2026 generative AI guide lists llms.txt and other special AI or machine-readable files among tactics you do not need for Google Search and its generative features. Google's AI features page likewise says no new machine-readable files or AI text files are needed. Documented.
  • The proposal itself describes its own purpose and format. Documented, but by its author, not by a search engine.

What is not confirmed

I could not find vendor documentation stating that major AI assistants fetch llms.txt and use it to choose or quote sources. People report seeing requests for the file in server logs. A request in a log proves a program asked for it; it does not prove the answer used it. Claims that llms.txt "boosts AI visibility" are hypotheses, and some are marketing. Treat them like that.

Some documentation tools and developer platforms do publish such files for coding assistants that fetch documentation on demand; that is a narrower use (a tool you point at a docs site), and still not a search ranking matter.

What llms.txt is not

  • Not a replacement for robots.txt: it does not grant or deny access.
  • Not a sitemap: it is a human-curated summary, not a full list.
  • Not a place to put hidden instructions for models. Text meant to manipulate a model that readers cannot see is a form of prompt injection and is ethically and practically risky (lesson Risks, ethics, hallucinations and opting out).

Should you publish one?

A reasonable, honest framework:

Situation Suggestion
You maintain developer documentation and know people use coding assistants on it Worth a try; low cost; measure whether it is requested
A small business site Not needed; spend time on clear pages and correct facts
You are told it will raise citations Do not pay for that promise

If you publish one, keep it accurate, link only to real, public pages, update it when pages move, and never describe it to anyone as a ranking tactic. It is a file; the page content is what matters.

Other "AI files"

You may read about ai.txt, special JSON feeds or Markdown copies of pages for AI. As of this lesson I found no vendor documentation requiring any of them, and Google's guide discourages special Markup or Markdown for AI systems. If you add machine-friendly versions for your own reasons (for example an RSS feed for readers), fine. Do not serve different content to crawlers than to visitors; that is cloaking.

This site

As of 2026-11-14 no llms.txt file exists in this repository's public_html folder. Whether to add one is a decision for Himanshu; nothing in this course says it would help.

Common mistakes

  • Treating llms.txt as a standard that all assistants honour.
  • Paying a vendor for "AI optimisation files".
  • Putting content in llms.txt that differs from what visitors see.
  • Listing private or unpublished URLs.
  • Forgetting to update it, leaving it pointing at dead pages.

Exercise

  1. Read the proposal at llmstxt.org and write a five-line summary in your own words.
  2. Draft an llms.txt for a fictional site, using only public pages.
  3. Write two sentences explaining to a client why you are not promising any effect from it.
Show a sample answer

Client explanation: "The file is a short guide to our key pages. Google says it does not need it, and I cannot find evidence that other assistants rely on it, so I treat it as an optional extra, not as something that will raise results."

Previous: robots.txt rules for search and AI crawlers. Next: Rendering, speed and clean HTML.

Key points

  • llms.txt is a proposal by Jeremy Howard, published in September 2024, for a Markdown file at a site's root that points language models at key pages.
  • It is described by its own site as a proposal, not a formal standard.
  • Google's May 2026 guide lists llms.txt among things you can ignore for Google Search and its generative features.
  • Whether other assistants read llms.txt at all is not confirmed by evidence I could verify.
  • If you publish one, treat it as a cheap, optional extra and never as a ranking or citation lever.

Check yourself

1. What is llms.txt?
2. What does Google's guide say about llms.txt for Google Search?
3. Can llms.txt block an AI crawler from using your content?
4. A consultant says llms.txt will 'make AI cite you'. What is the best response?