Who llms.txt actually helps

llms.txt is often pitched as robots.txt for AI, and SEO plugins will now generate one in a click. HTTP Archive's 2025 Web Almanac found a valid llms.txt on 2.13% of desktop sites and 2.10% of mobile sites, and about 40% of the files it found were generated by the All in One SEO plugin.¹ Most of those files will never be read by anything that affects how a business is found. The file does have a use. It is a narrower one than the pitch suggests.

This post sets out who reads llms.txt, who does not, and when it is worth writing. How assistants find and cite sources in general is covered in the AI search guide.

Jeremy Howard proposed llms.txt in September 2024 as a Markdown file at /llms.txt that gives language models a clean summary of a site, with links to Markdown versions of its pages at the same URL plus .md. The proposal points out that an HTML page wraps its information in navigation, ads and JavaScript, and describes the file as a way to help agents use a website. Howard's own update says the expectation was that llms.txt would mainly be useful for inference, and that this is how it has been used, though training runs could take advantage of it too.²

Inference here means the moment of use: an assistant or agent reading the file because someone has asked it to do something on your site, or with your product, right now. That is a different job from a crawler collecting pages for training or for a search index.

Google's guide to its generative AI features says you do not need machine-readable files, AI text files or Markdown to appear in Search, and that creating llms.txt files will neither harm nor help visibility or rankings in Google Search, because Search ignores them.³ John Mueller of Google said in June 2025 that no AI system currently uses llms.txt, and that this is obvious from server logs.⁴ In 2026 he added that an AI system cannot trust what such a file says as a way of telling websites apart.⁵

The other providers are silent on it as a crawling signal. OpenAI's crawler documentation describes no use of llms.txt,⁶ and neither do Anthropic's or Perplexity's pages on their crawlers.⁷⁸ OpenAI and Perplexity do publish llms.txt files, for their own developer documentation, which says a lot about who the file is really for.

The citation studies agree. SE Ranking analysed 300,000 domains, found that 10.13% had an llms.txt file, and concluded that it does not affect how AI systems see or cite content.⁹ In a separate model of 216,524 pages cited by ChatGPT, it found llms.txt had a negligible effect on the likelihood of citation, and that removing it from the model made the predictions more accurate.¹⁰

Ahrefs looked at server logs for 137,210 domains on its web analytics product in May 2026. Up to 28% published an llms.txt file, and 97% of those files received no requests at all that month. Of the bots that did fetch them, GPTBot was the most frequent and Claude Code second, ahead of every AI search and assistant bot, and Ahrefs found the files were read far more by agentic tools than by anything else.¹¹ OtterlyAI's experiment on one site found 84 requests for /llms.txt among more than 62,100 AI bot visits over 90 days.¹²

So a training crawler does sometimes fetch the file. When it does, it gets a summary of pages it can crawl anyway. Mintlify, which generates these files for documentation sites, reported in 2025 that ChatGPT accounted for most visits to llms-full.txt across 25 of its customers, but its data did not separate training crawlers from other OpenAI traffic.¹³ In my view a file that repeats your site in Markdown adds very little to what a model learns about you. The control over training is robots.txt, covered in the AI crawlers guide.

The file earns its place when an agent is working on a task. Coding agents are the clearest case. Claude Code's documentation tells agents to fetch its llms.txt index to discover all available pages before exploring further,¹⁴ and Stripe's file is written for agents building with its API.¹⁵ Mintlify tested Claude Code and Codex on 20 documentation sites across 2,400 runs. On plain HTML the agents kept reaching for .md and /llms.txt versions of pages and hitting 404s, and linking an llms.txt cut dead URLs by about 90%. Accuracy stayed in the mid to high 90s whatever the format,¹⁶ so the gain was in wasted requests, which cost the agent time and tokens.

Browser agents are the other case, and this is where JavaScript-heavy apps come in. Vercel and MERJ found that none of the major AI crawlers rendered JavaScript, with Gemini (through Googlebot) and Applebot the exceptions.¹⁷ A tool that fetches the raw HTML of a single-page app sees an empty shell. An agent driving a real browser sees the rendered app, but still has to work out which screen does what and where each task lives. Google's Lighthouse now has an agentic browsing category that checks for llms.txt at the domain root, alongside its advice that agents rely on the accessibility tree and that semantic HTML and proper ARIA labelling come first.¹⁸

For that kind of site, a short llms.txt that says what the app does, where the main tasks are, and links to plain-text help pages gives an agent a map before it starts clicking. Mueller made a similar point: once someone is already on your website, some kind of automated helper may be useful.⁵ That is a real use, and it is a long way from ranking.

• Yes, for a web app or SaaS product whose interface is built in JavaScript and that people may ask an agent to use. Write it by hand: what the product does, the main tasks and the URLs where they happen, and links to help pages in Markdown.
• Yes, for developer documentation or an API. This is where agents read it most. Link every page in Markdown, and consider an llms-full.txt with the full text.
• Not as a visibility tactic for a local business, a shop or a content site. It will not change rankings or citations.³⁹ If a plugin generates one, it does no harm. Do not spend time on it.
• Whatever you decide, put your content in the HTML and use semantic markup and ARIA labels. Every crawler and agent benefits from that, with or without llms.txt.¹⁸