llms.txt is a markdown file you put at your site root to hand AI agents a curated map of your most important pages. Google now checks for it inside Chrome Lighthouse, which is why site owners suddenly want one. Here is the catch the hype skips: most large language model crawlers still do not read the file, and Google Search says outright you do not need it to show up in AI Overviews. This guide gives you the exact spec, a copy-paste template, and a straight verdict on whether it earns your time today.
What is llms.txt, in one paragraph
llms.txt is a single plain-text file, written in Markdown, served at https://yourdomain.com/llms.txt. It gives language models and AI agents a fast, structured summary of your site so they do not have to crawl every page to learn what matters. Jeremy Howard of Answer.AI proposed the format on September 3, 2024, and the specification lives at llmstxt.org. The pitch is a curated index a model reads in one request, not a sprawling sitemap built for search spiders.
The name echoes robots.txt on purpose, but the job is different. robots.txt tells crawlers where they may not go. llms.txt tells AI systems where the good content is and how it connects. One is a fence. The other is a guide. A sitemap enumerates every URL for a machine that crawls exhaustively. llms.txt hands a model the few pages a thoughtful human would point to first.
Lighthouse now audits llms.txt
In 2026 Google started grading your site on a file it also says you do not need, and it did so from inside Chrome itself. The check landed in Chrome Lighthouse under a new experimental category called Agentic Browsing. When a tool that ships with Chrome DevTools starts scoring your site for something, owners notice fast. The category is experimental, built on proposed standards, and testing it requires Chrome 150 or later.
The Agentic Browsing audit grades four areas, not just llms.txt. It checks for an llms.txt file, support for the WebMCP protocol, the quality of your accessibility tree, and cumulative layout shift. Together those four signals answer one question: can an AI agent read your site, understand its structure, and act on it? llms.txt covers the reading map. WebMCP covers the actions an agent can take. The accessibility tree covers how cleanly the page parses. Layout shift covers whether the page is stable enough to touch. The file sits next to accessibility metrics, not SEO ones, for that reason.
The Lighthouse llms.txt check is optional by design. If the file returns a 404, the audit is marked Not Applicable, not failed. Lighthouse only flags a problem when retrieving the file triggers a genuine server error, which signals something is broken rather than simply absent.
The honest catch: Google grades it, then tells you to skip it
For search visibility, the crawlers largely do not read it. Google Search has stated separately that llms.txt is not required to appear in AI Overviews or AI Mode. Its own AI Search optimization guidance groups llms.txt with tactics you can skip. At a Search Central event, Google staff confirmed the company is not pursuing the format, and John Mueller earlier compared it to the deprecated keywords meta tag that no AI services ended up using.
Google built a tool that grades your site on a file Google Search also tells you to skip. Both teams are right, because they are measuring different things.
Here is the nuance most thin guides miss. Search visibility and agent readiness are two different problems. One helps a page get found by a generative search feature. The other helps an autonomous agent understand and use your site once it arrives. Lighthouse audits the second case. Google Search comments on the first. So both statements hold at once: you do not need llms.txt for AI Overviews, and Lighthouse can still nudge you to add one for agentic browsing.
So does anything honor the file? A growing set of documentation tools and a handful of AI products consume it. The broad LLM crawlers do not treat it as authoritative. Google Search ignores it for ranking. That mixed picture is why the verdict is not a simple yes or no.
Adding the file is cheap and harmless. Expecting it to move your traffic this quarter is not realistic.
The spec: what the file actually contains
The llmstxt.org spec defines a Markdown file with a small, fixed shape. An H1 carrying the site or project name is the only strictly required element. After that you add an optional blockquote summary, optional free-form details, and one or more sections of Markdown links to your key pages. The format leans on Markdown because models and agents are the intended readers. An H1 signals the subject. A blockquote signals the summary. A list of links signals the navigable surface, all in syntax a model already parses fluently.
The required and optional parts
- H1 title (required): a single top-level heading with your project or site name, for example # MemX.
- Blockquote summary (optional but recommended): a one-line > description giving an agent the key context it needs.
- Details (optional): a short paragraph or list expanding on the summary, with no special heading.
- Section headings (optional): H2 headings like ## Docs or ## Guides that group related links.
- Markdown links (the substance): list items in the form - [Page title](https://example.com/page): short note.
Keep llms.txt curated, not exhaustive. A sitemap lists every URL. An llms.txt should list only the pages you would point a smart assistant to first. Quality of selection is the entire value.
Copy-paste template
Save the following as llms.txt, swap the example lines for your own pages, and serve it from your site root. The structure follows the llmstxt.org spec exactly: H1, blockquote, optional detail line, then grouped sections of Markdown links.
- # Your Site Name
- > One sentence describing what your site does and who it serves.
- Optional follow-up line with any context an agent should know before reading the links.
- ## Core pages
- - [Home](https://yourdomain.com/): what the site is.
- - [Pricing](https://yourdomain.com/pricing): plans and costs.
- ## Documentation
- - [Getting started](https://yourdomain.com/docs/start): first steps.
- - [API reference](https://yourdomain.com/docs/api): endpoints and auth.
The single most common reason the Lighthouse audit fails: plain-text links. Lighthouse parses llms.txt as Markdown and only counts links written as [text](url). Plain lines like Home: / are ignored, and Lighthouse grades the file as containing no links.
One published walkthrough showed the audit score jump from 0.67 to 1.0 after the only change was converting plain-text descriptions into Markdown bracket-paren link syntax. The trap is that the signals point opposite ways. The file extension is .txt and the server returns text/plain, yet the parser demands Markdown. Wrap every link in [text](url) and the same file that scored as empty suddenly counts every entry.
How to add it, step by step
- Draft the file: start with your H1, add a blockquote summary, then list your most important pages as Markdown links grouped under H2 sections.
- Use absolute URLs: write full https links rather than relative paths so any agent can resolve them.
- Serve it at the root: the file must be reachable at https://yourdomain.com/llms.txt, not in a subfolder.
- Verify the response: load the URL in a browser; you should see raw Markdown text and a 200 status, not a 404 or a redirect.
- Run Lighthouse: open Chrome 150 or later, run the Agentic Browsing audit, and confirm llms.txt passes rather than reporting no links.
- Keep it current: prune dead links and add new key pages when your site structure changes, just as you would a sitemap.
llms.txt vs robots.txt vs sitemap.xml
| Trait | llms.txt | robots.txt / sitemap.xml |
|---|---|---|
| Audience | LLMs and AI agents | Search engine crawlers |
| Format | Markdown, human-readable | Plain directives / XML |
| Purpose | Curated map of key content | Crawl rules and full URL inventory |
| Status | Proposed, optional, experimental | Long-established, widely honored |
| Honored by Google Search | No, not used for ranking | Yes, core to indexing |
Should you add llms.txt right now
Add it in an hour, skip it as a traffic play. The longer version: add it if you publish documentation, a knowledge base, or content you want AI assistants to summarize accurately, and if writing the file costs an hour or less. The downside is minimal, and the Lighthouse audit gives you a concrete pass-or-fail target. Skip it as a traffic strategy. It will not lift your search rankings, and Google Search has said as much. Treat llms.txt as low-cost future-proofing for agentic browsing, not a growth lever.
Where personal AI memory fits
llms.txt helps agents understand a public website. The mirror-image problem is helping AI understand you: your own documents, photos, notes, and chats scattered across apps. That is what MemX does. It is a consumer AI memory layer that sits over your personal content on Android, iOS, and WhatsApp, so an assistant can recall what you actually saved instead of re-reading everything each time. MemX is private by architecture, with per-user keys, encryption at rest, and an on-device first pass, so your memory stays yours rather than becoming training fuel. Same principle as llms.txt, pointed inward: give the model a clean, curated map instead of raw sprawl.
01What is llms.txt and where does it go?
It is a Markdown file served at your site root, yourdomain.com/llms.txt, that gives AI agents a curated map of your most important pages. It starts with an H1 title, an optional blockquote summary, and lists of Markdown links to key content.
02Does Google use llms.txt for AI Overviews?
No. Google Search has stated llms.txt is not required to appear in AI Overviews or AI Mode, and its AI Search guidance lists the file among tactics you can skip. It is not a ranking signal.
03Is llms.txt the same as robots.txt?
No. They serve different audiences. robots.txt fences crawlers out of areas they may not visit, while llms.txt guides AI agents toward your best pages. One restricts access; the other curates a map. The names rhyme, but the jobs are opposite.
04Does a missing llms.txt hurt my Lighthouse score?
No. If the file returns a 404, the Agentic Browsing audit is marked Not Applicable because the file is optional. Lighthouse only flags a problem when retrieving the file triggers a genuine server error.
05Do AI crawlers actually read llms.txt yet?
Some documentation tools and AI products consume it, but broad LLM crawlers do not treat it as authoritative, and Google Search ignores it for ranking. Adding it is cheap future-proofing, not a near-term traffic strategy.
