llms.txt: what it is, who reads it and when it's worth it

llms.txt is a plain text file at the root of a site that sums up what the site is and points to the pages an AI agent should read. It’s written in Markdown, a format with light marks for headings and links. The format is an open proposal, not an official standard. Google says you don’t need one to show up in AI features in Search (Google Search Central, AI features and your website). It’s worth publishing when agents look up your content while they’re helping someone, as with software docs, and when the file is built from your content instead of written by hand.

What the file is

The idea comes from Jeremy Howard. He published it on September 3, 2024 and revised it in August 2026 (llmstxt.org, the proposal). The reason is practical. A page built for people comes wrapped in menus, ads and JavaScript, and an agent burns context and time digging the text out.

The format is short and follows a fixed order, according to the spec:

a level 1 heading with the site’s name, the only required part;
a blockquote with a summary of what the site is;
free text, with no headings, that helps read the rest;
sections under level 2 headings, each with a list of links in the [name](url): note format.

By convention, a section called Optional holds links an agent can skip when it needs to save context.

Version 2 brought three changes, listed on the proposal’s changes page. The file can live in a subpath, like /docs/llms.txt, and covers the pages below it. Each page can have a Markdown version at the same address, ending in .md. And the HTML can point to both with rel="alternate" and rel="describedby".

Who reads it

The proposal itself says who the file is for: an agent that needs information right when it’s helping someone. The most common use is software documentation, where a coding agent looks for the right API reference. The author expected it to be used more for answering than for training models, and says that’s how it turned out.

An agent, here, is a system that pursues a goal by taking actions, checking its progress and deciding its own next step. The definition comes from Caleb Sponheim at Nielsen Norman Group, NN/g. He names search agents and coding agents as the most visible examples in early 2026.

Reading a site is expensive for this kind of reader. Sarah Gibbons and Kate Moran, at NN/g, describe three ways an agent uses a page: looking at a screenshot, reading the accessibility tree or talking straight to an API. A single screenshot takes tens of thousands of tokens, the units of text a model processes. The accessibility tree costs a few thousand.

Evan Sunwall’s test of ChatGPT’s agent mode, at NN/g, shows how big the search gets. To find a restaurant for a business lunch, the agent spent 6 minutes researching, with 10 searches and 96 sources.

AI labs publish the file for their own docs. There’s OpenAI’s, Anthropic’s and the one for the Gemini API. The proposal also lists tools that generate the file on their own, like Mintlify, GitBook, Wix and the Yoast SEO plugin for WordPress.

The adoption figure comes from the author. He talks about thousands of sites with the file, with no independent count behind it (llmstxt.org, version 2 changes).

What it doesn’t do

It doesn’t control access. robots.txt is what says what can be crawled, and even that is a request, not a lock. The robots.txt standard, RFC 9309, says its rules “are not a form of access authorization.”
It doesn’t keep a page out of Search. To limit what Google shows, AI features included, you use robots.txt and the nosnippet and noindex controls (Google Search Central).
It doesn’t replace the sitemap. The sitemap protocol lists the pages a search engine can index. The proposal wants the opposite: a short selection that fits in a model’s context.
Google doesn’t require it. Google’s page on AI features says you don’t need to create “AI text files” to show up in AI Overviews or AI Mode.

Where the sources disagree

The most curious split happens inside Google itself. Search Central says the file isn’t needed. Lighthouse, Chrome’s auditing tool, checks whether one exists at the domain root. That check sits in an experimental agentic browsing category, which needs Chrome 150 or later (Chrome for Developers, agentic browsing scoring).

Even there the file is optional. According to the audit page, Lighthouse only flags a problem when the server errors out while serving the file. If the address returns a 404, the audit is marked not applicable.

The way I read it, the two pages are talking about different readers. Search builds its index with a crawler and doesn’t need a shortcut. An agent that opens a site in the middle of a conversation has little time and little context, and that’s who the file is for.

NN/g’s article on agents as users doesn’t even mention a special file. To serve people and agents at the same time, it recommends accessibility basics: clear names on every element, predictable patterns, semantic HTML and a logical hierarchy. The post on semantic HTML covers that part.

When it’s not worth publishing

Not every site wants to shorten an agent’s path. The same NN/g article points to businesses where the visit is the product, like ad-supported sites. There, an agent that takes the answer without the visit is a problem. The article also mentions cases where friction exists for regulatory reasons, as in finance and health. In those cases, handing an agent a shortcut is a business decision, not just a technical one.

How to build one

Start with the title and the summary. The site’s name as the heading and, right below, one or two sentences on what it is and who it’s for.
Pick pages, don’t list them all. Each link gets a short note on what the page holds. That note is what helps an agent decide whether to open it.
Point to readable text. The proposal asks for links to model-friendly content, like a page’s Markdown version when there is one.
Generate it from your content. A hand-written file goes stale with the first post published after it.
Make sure the address responds. A server error is the only case Lighthouse marks as a failure.
Test it with an agent. The proposal suggests asking an agent questions about the site when all it’s been given is the file.

How it shows up in my projects

On kahue.net, the llms.txt is generated at build time, the step that puts the site together before it goes live, from the content. It has the name, the profile summary, the main pages, the list of posts with each one’s description and the address of all three languages. A new post gets in on its own at the next build. I left the reason in a code comment: a fixed file in the public folder would go stale with the first new post and start saying something false about the site.

The site doesn’t publish a Markdown version of each page. The link goes to the static HTML, which already arrives with the full text and doesn’t depend on JavaScript to show it.

On the Vexor site, the product where I build AI assistants for WhatsApp, the llms.txt (in Portuguese) went in as one piece of the SEO and GEO layer I wrote. GEO is the work of getting found and cited by AI answer engines. The file sits next to the questions answered on the page and the offer and price in structured data. To see how those engines pick what to cite, read the post on AI answer engines.

The common mistake

The common mistake is publishing llms.txt and expecting to climb in Google or get cited by an AI. Google says the usual SEO practices still apply, and the file isn’t part of that. What it does is shorten an agent’s path to the right pages. If those pages are weak, the file just gets the agent to them faster.