Does llms.txt Work? What Is Known, and What Is Not
There is no good evidence that llms.txt helps a site appear in AI answers. Google says you do not need it, the largest study we found showed no benefit, and we could find no major AI company promising to use it. It takes minutes to add and does no known harm, so the sensible position is that it is optional and low priority.
What is llms.txt?
llms.txt is a plain Markdown file, usually at the root of a site, that lists a site’s most useful pages with short descriptions. The proposal, published by Jeremy Howard in September 2024, describes it as a way “to provide information to help agents use a website”.
The idea is reasonable. Web pages are full of menus, scripts and layout that a language model has to wade through, and a clean index could save it the trouble. Whether anyone’s systems actually behave that way is a separate question.
It is often compared to robots.txt, but the two do different jobs. robots.txt gives crawlers rules. llms.txt offers a reading list and enforces nothing, so it cannot control access. Whether a bot can reach your site at all depends on robots.txt and your CDN, which is covered in our guide to which AI crawlers to allow.
What each source says
| Source | What it says | What it tells you |
|---|---|---|
| Google Search Central | “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” | Clear for AI Overviews and AI Mode. Says nothing about other assistants |
| John Mueller of Google, June 2025 | “FWIW no AI system currently uses llms.txt.” | One person’s view, more than a year old |
| SE Ranking study | Nearly 300,000 domains. 10.13% had the file, and it did not improve the model’s prediction of citations | Correlation data from a vendor, and the strongest study we found |
| Ahrefs log study, May 2026 | 137,210 domains. 97% of llms.txt files received no requests that month, and most bots reading the files were not AI tools | Measures fetches. It did not measure citations, or whether bots used what they fetched |
| Crawler docs from OpenAI, Anthropic and Perplexity | None commits to using other sites’ llms.txt files. OpenAI and Perplexity do point readers to an llms.txt for their own documentation | Publishing one for your docs is a different thing from reading everyone else’s |
| The llms.txt proposal | A way to help agents use a website | A statement of intent, with no evidence of effect |
What we know
- Google does not need it for its AI search features. That comes from Google’s own documentation.
- It is not an agreed standard. It remains a proposal.
- Plenty of sites have tried it: about one in ten in SE Ranking’s sample, and 28% in Ahrefs’, whose customers skew technical.
- In that sample, having one did not help predict how often a domain was cited.
- In Ahrefs’ logs, most files were not requested at all in May 2026.
What nobody outside these companies knows
- Whether any model weighs it at all. Companies do not publish how sources are selected, so a claim in either direction is an inference.
- Whether crawlers fetch it for a reason. A log entry shows a bot requested the file. It cannot show the contents affected an answer.
- Whether it helps once an agent is on a site. Mueller, reported by Search Engine Journal, said it may help there, but that it cannot help systems choose between sites. Our inference is that a coding assistant reading a library’s docs is the likeliest use, and a chatbot picking an agency the least likely.
- Whether this changes. Vendors could adopt it next year. Any verdict here has a short shelf life, and it is worth checking again.
Should you add one?
| Your site | Suggestion |
|---|---|
| Software or API with public docs | Reasonable. Developers and coding tools look the likeliest audience |
| Large site with clear documentation or help content | Reasonable, if it can be kept up to date automatically |
| Small business or local service | Optional. Spend the time on service pages first |
| Ecommerce store | Low priority. Product data and crawl access matter more |
| Anyone being charged extra for it | Ask what evidence supports the fee |
How to make one
If you want one, it takes minutes. A minimal version looks like this:
# Example Plumbing Ltd
> Emergency plumbers covering Sheffield and Rotherham.
## Services
- [Boiler repair](https://www.example.co.uk/boiler-repair/): Gas Safe engineers, same-day visits
- [Bathroom fitting](https://www.example.co.uk/bathrooms/): Full installs and prices
## About
- [About us](https://www.example.co.uk/about/): Who we are and where we workPut it at yourdomain.co.uk/llms.txt. The proposal says it can sit at the root or at any path, covering the pages under that path. Keep to facts that match the site, link to pages you want read, and update it when services or prices change. Our own file is a working example of the format. Having one is not evidence that it works, and we would not claim otherwise.
What to do first
These come first, because each one affects whether you can be found or measured at all:
- Confirm the crawlers you want can actually reach you, including past your CDN. Our AI crawlers guide shows how to check.
- Write pages that answer real questions directly, then earn mentions elsewhere. Our guide to getting recommended by ChatGPT walks through this.
- Measure referrals, so you know whether anything is changing. See how to track AI traffic in GA4.
If you want this looked at for your own site, our AI search optimisation service starts with those three, and the free SEO audit is a straightforward first step.