llms.txt: what it does and what it doesn’t

The proposal was published by Jeremy Howard on 3 September 2024 at llmstxt.org. It suggests adding a markdown file at /llms.txt that describes the site and links to its most useful pages, preferably to clean markdown versions of them. The stated reason is practical: context windows are still too small to hold most websites whole, and turning HTML pages full of navigation and scripts back into clean text is difficult and imprecise.

Three details in the proposal are often lost when the idea is repeated:

• It is aimed at inference, the moment an agent needs information while helping a user. The authors expected it to be mainly useful there, not for training.
• It is not an access control. robots.txt says what crawlers may fetch; llms.txt only offers a reading list. It doesn’t replace a sitemap either, which lists every indexable page.
• It is a proposal. The site describes the spec as open for community input. No standards body has adopted it.

The file is plain markdown with a fixed order (spec):

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote with a short summary.
3. Optional paragraphs or lists with more detail, but no headings.
4. Optional H2 sections, each a list of links. A link can be followed by a colon and a note about what the page contains.

A section titled Optional has a special meaning: it holds links an agent can skip when it needs a shorter context. A small product’s file might look like this:

# Acme Invoices

> Invoicing app for freelance designers. Sends invoices, tracks
> payments and chases late ones. Web and iOS.

Plans start with a free tier limited to three clients.

## Docs

- [Getting started](https://acme.example/docs/start.md): Create an account and send a first invoice
- [API reference](https://acme.example/docs/api.md): REST endpoints and authentication

## Product

- [Pricing](https://acme.example/pricing.md): Plans and limits
- [Changelog](https://acme.example/changelog.md)

## Optional

- [About](https://acme.example/about.md)

The proposal also suggests serving a markdown version of each page at the same URL with .md appended, so the links lead to text a model can read without stripping HTML.

Google Search

Google is the clearest. Its page on AI features says you don’t need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode (source). Its AI optimisation guide names llms.txt and says that creating one will neither harm nor help a site’s visibility or rankings in Google Search, because Google Search ignores them (source).

John Mueller of Google’s Search team wrote on Bluesky on 17 June 2025, “FWIW no AI system currently uses llms.txt”, and added that this is obvious from server logs, as reported by Search Engine Roundtable. That statement is over a year old and was one person’s observation, so read it as a dated data point.

Chrome

A different Google team takes a softer line. Lighthouse, Chrome’s auditing tool, has an llms.txt check in its agentic browsing category. The documentation calls the file an “emerging convention” and says that without it agents may spend more time crawling to understand a site’s structure. A missing file is marked not applicable, since the file is optional; only a server error fails the audit (Lighthouse docs). Search Engine Journal noted the gap between the two teams in May 2026. The Lighthouse category is about browser agents, and says nothing about search rankings or citations.

OpenAI, Anthropic and Perplexity

The crawler documentation from OpenAI, Anthropic and Perplexity explains user agents and robots.txt. None of the three says its assistant reads a website’s llms.txt to find, choose or rank sources for an answer. We found no such statement elsewhere on their sites either. That is an absence of evidence, and it could change, but it is the honest state of things as of November 2026.

What these companies do is publish llms.txt files for their own developer documentation. OpenAI’s and Perplexity’s docs pages point readers to a documentation index at /llms.txt, and Anthropic’s Claude Code docs have one too. That tells you where the file has found a use.

The working use case is documentation read by coding agents. A developer asks an agent to integrate your API. The agent, or the developer, fetches one file that lists every docs page with a one-line description, then fetches only the markdown pages it needs. That is faster and cleaner than crawling a docs site built for browsers, and it works today because the person or agent requests the file on purpose. OpenAI’s developer index is a good model: a title, a summary and a list of links to markdown pages.

Documentation platforms have made this nearly free. The proposal’s site lists Mintlify, GitBook, Yoast SEO, AIOSEO and Wix among the tools that can generate the file. Mintlify, for example, hosts one automatically for every docs project (Mintlify’s docs). These vendors benefit when the format is seen as necessary, so take their descriptions of what AI tools do with it as marketing unless they cite the AI company.

It is worth having if any of these is true:

• You have an API, SDK, CLI or plugin that developers will set up with an agent’s help.
• You have help docs long enough that a clean index saves a reader time.
• Your site exposes something agents can act on, and you want one place to explain how.

IndieTide publishes one at /llms.txt for the third reason: it lists the week’s launches as markdown and explains how an agent can prepare a submission.

1. Check whether you already have one. Open yoursite.com/llms.txt. Your docs platform or an SEO plugin may be generating it.
2. Generate it, don’t hand-write it. Build the file from the same data that feeds your sitemap or docs navigation, so it can’t drift out of date. A stale index that links to deleted pages is worse than none.
3. Start with the H1 and the blockquote. Use the same one-sentence description as your homepage. Then link ten to thirty pages that matter, each with a short note. Leave out legal pages and marketing filler, or put them under Optional.
4. Link to markdown where you can. If your framework can serve a .md version of a page, link that. If it can’t, link the normal pages. Don’t build a parallel markdown site just for this.
5. Serve it as plain text with a UTF-8 charset, at the root, returning a 200 status.
6. Say only true things. The file is public and anyone can read it. Stuffing it with keywords or claims that aren’t on your site gains nothing with systems that ignore it and looks bad to a person who opens it.

An hour is a reasonable budget. If it is taking a day, stop.

• Better rankings or AI Overview citations in Google. Google says the file is ignored.
• A mention in ChatGPT, Claude or Perplexity answers. No documentation from those companies connects the file to what their assistants say about a product.
• Control over crawling or training. That is robots.txt’s job. See Should you block AI crawlers?
• A replacement for clear pages. Assistants that search read your normal HTML pages, so the product description on the page itself does the work. How to get your product mentioned by ChatGPT and Perplexity covers what is documented there.

You can test the claim yourself. Search your server logs for requests to /llms.txt and look at the user agents. If documented AI fetchers request it, you’ll see them; if only SEO audit tools do, you’ll see that instead. How to test whether AI assistants know your product lists the user agents to look for.