llms.txt is a plain-text Markdown file at yourdomain.com/llms.txt that lists a site's most useful pages with short descriptions, so AI tools can find the right content without reading every page. Most business websites do not need one in 2026. Google Search ignores it, and an Ahrefs study of 137,210 domains found 97% of llms.txt files received no requests in May 2026.
Most people asking what is llms.txt have been offered one as part of an "AI visibility" package. The file is cheap to make, so the question that matters is whether anything reads it. For a software company with API documentation, something does. For a local service business or a typical online store, the evidence says almost nothing does in 2026.
On this page
- What is llms.txt in plain terms?
- What does an llms.txt file look like?
- How is llms.txt different from robots.txt and sitemap.xml?
- Do AI crawlers read llms.txt?
- What do Google and OpenAI say about llms.txt?
- Does your website need an llms.txt file?
- How do you create an llms.txt file?
- How do you check whether any AI tool reads your llms.txt?
- Is there any risk in publishing an llms.txt file?
- What should you do instead for AI visibility?
What is llms.txt in plain terms?
llms.txt is a proposal, not an official standard. Jeremy Howard published it on llmstxt.org on September 3, 2024, as a proposed common file that helps AI agents use a website. No standards body has adopted it, and no AI company requires it.
Think of it as a short guide written for software. A web page carries menus, scripts, cookie banners and ads around its actual text. An AI tool with a limited context window wastes space on all of that. An llms.txt file gives it a hand-picked list of the pages worth reading, each with a one-line note, in a format language models handle well.
Two companion ideas come with the proposal. Each listed page can offer a clean Markdown copy at the same address with .md added, such as page.html.md. Some sites also publish llms-full.txt, a single file holding the full text of their documentation, so an agent can load everything in one request.
The spec says the file is used "on demand," when an agent needs information about a site. That is the main difference from a crawler file. Nothing visits llms.txt on a schedule unless a tool decides to.
What does an llms.txt file look like?
The format is ordinary Markdown in a fixed order. Only the first part is required:
- An H1 line with the name of the site or project. This is the only required section.
- A blockquote with a short summary of what the site is and who it serves.
- Optional paragraphs or lists with more detail, but no further headings.
- One or more H2 sections, each a list of links with a colon and a short note.
- An H2 named "Optional" for secondary pages an agent can skip when it is short on space.
A file for a hypothetical HVAC company in Columbus, Ohio would start like this:
# Example Heating and Cooling> Residential furnace and AC repair in Columbus and Franklin County, Ohio. Same-day service on calls before 2 p.m.## Services- [Furnace repair](https://www.example.com/furnace-repair/): diagnosis fee, typical repair costs, response times- [AC installation](https://www.example.com/ac-installation/): system sizes, price ranges, rebates## Optional- [Company history](https://www.example.com/about/)
It lives at the site root, though the spec also allows it at any path, where it covers the pages below that path.
How is llms.txt different from robots.txt and sitemap.xml?
The three files get mentioned together, but each does a different job, and llms.txt replaces neither of the others.
File | What it does | Who reads it today |
|---|---|---|
robots.txt | Tells crawlers which paths they may and may not fetch | Googlebot, Bingbot, OAI-SearchBot, GPTBot and most major crawlers |
sitemap.xml | Lists every page you want search engines to find and index | Google, Bing and other search engines |
llms.txt | Offers a short, hand-picked reading list for AI tools, with summaries | Mainly coding assistants and a small share of AI bots |
The llmstxt.org spec draws the same line: sitemaps list every page for search engines, while llms.txt gives language models a short, chosen overview. Access control stays with robots.txt. If you want ChatGPT search to show your pages, the rule that matters is whether robots.txt allows OAI-SearchBot, as OpenAI's crawler documentation explains.
Do AI crawlers read llms.txt?
Barely. The largest log study so far, Ahrefs' analysis of 137,210 domains published June 15, 2026, counted requests to llms.txt files in server logs for May 2026. Its findings:
- 28% of the domains published an llms.txt file, about 38,000 sites. Ahrefs calls this an upper bound, because its customers are more SEO-aware than the web overall.
- 97% of those files received zero requests that month. The other 3%, about 1,100 domains, took all the measured traffic.
- Bots made 96% of the requests. SEO audit tools sent 21.7% of them, more than any AI bot.
- AI bots sent 19.5% in total. GPTBot led at 4.51%, ClaudeBot sent 0.8% and OAI-SearchBot, the top retrieval bot, sent 0.74%.
- Claude-Code, Anthropic's coding assistant, ranked second among AI bots.
Googlebot fetched llms.txt files about 900 times in May, which Ahrefs puts down to routine crawling. A fetch also does not prove a bot used the file. The study did not measure citations, so it cannot say whether a file changed any AI answer.
The pattern is clear enough to act on. The tools that do read llms.txt are coding assistants loading developer documentation. The search and chat bots that decide which businesses get recommended almost never ask for it.
What do Google and OpenAI say about llms.txt?
Google has answered directly. Its guide to optimizing for generative AI features, updated July 10, 2026, says you do not need "machine readable files, AI text files, markup, or Markdown" to appear in Google Search, including AI Overviews and AI Mode. It adds that creating llms.txt "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
OpenAI sends a mixed signal. It publishes an llms.txt index for its own developer documentation, with Markdown copies of each docs page. Its guidance for site owners who want to appear in ChatGPT search, the publishers and developers FAQ, asks for OAI-SearchBot access and never mentions llms.txt.
That split explains the whole debate. AI companies publish llms.txt for developers who paste documentation into coding tools. None of the major search products asks businesses to publish one to be found.
Does your website need an llms.txt file?
Use this table to decide. The recommendation follows who reads the file today.
Type of site | Recommendation | Why |
|---|---|---|
Software product with public API or developer docs | Add llms.txt and Markdown copies of docs pages | Coding assistants read these files, and developers paste them into AI tools |
SaaS marketing site without docs | Optional, low priority | Little evidence that chat or search bots request it |
Online store | Skip; spend the time on product data | Shopping answers rely on product feeds and structured data |
Local service business | Skip | Recommendations come from search crawlers, listings and reviews |
Publisher or large content site | Optional, if a plugin builds it automatically | No harm in Google and near-zero upkeep when automated |
For example, say a 20-person software company has 300 API reference pages. A developer asks an AI coding tool to write an integration, and the tool loads the company's llms.txt to find the right endpoints. That saves the developer time and cuts wrong answers about the API. Now say a dental clinic in Phoenix has 14 pages. No coding tool will ever load its file, and patients asking ChatGPT for a dentist reach it through search, listings and reviews.
How do you create an llms.txt file?
If the table says yes, the build is small. Choose the route that matches your platform:
- On WordPress, turn on the llms.txt feature in Yoast SEO. Yoast's feature page says it is free on every plan, generated in one click and refreshed automatically.
- On a docs platform, check its settings first, because several documentation hosts generate the file for you.
- On a custom site, write the file by hand or generate it from your sitemap, keeping only the pages an AI tool should read first.
- Upload it to the site root so it loads at yourdomain.com/llms.txt as plain text.
- Add Markdown copies of the listed pages only if you have developer docs that agents will load.
- Open the file in a browser to confirm it displays as plain text, then click every link to make sure none returns an error page.
- Regenerate the file whenever you add, rename or remove a listed page.
The last step is where hand-made files fail. A stale llms.txt points agents to moved or deleted pages, which is worse than having no file. Automate it, or do not publish it.
How do you check whether any AI tool reads your llms.txt?
Your server logs hold the answer. Ask your host or developer for the last 30 days of access logs, then filter for requests to /llms.txt and group them by user agent:
- GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-Code for Anthropic, PerplexityBot for Perplexity.
- SEO audit tools, which in the Ahrefs data sent more requests than any AI bot.
- Unknown agents, which you can ignore unless the volume is large.
User agents can be faked, so check the source addresses too. OpenAI publishes the IP ranges for its crawlers, linked from its crawler documentation, and a request claiming to be GPTBot from another address is not OpenAI.
Hosting dashboards and CDN analytics such as Cloudflare can show the same data without raw logs. If three months pass with no AI requests, the file is doing nothing for you, and that is useful to know before paying anyone to maintain it.
Is there any risk in publishing an llms.txt file?
The risks are small and specific. The file is public, so anything you list is visible to anyone who opens it, including competitors comparing your service pages and prices. It offers no protection either: a page left out of llms.txt is still open to every crawler that robots.txt allows.
The bigger risk is decay. A file built once and forgotten keeps sending AI tools to old prices, retired services and broken links. If you publish one, put it on the same update schedule as your sitemap.
What should you do instead for AI visibility?
Put the hours where the readers are. These four jobs decide far more AI citations than llms.txt:
- Allow OAI-SearchBot and Googlebot in robots.txt, your CDN and your firewall. The guide to getting your business recommended by ChatGPT walks through the OpenAI side.
- Answer buyer questions in plain text near the top of each page, with prices, ranges and dates. How Google picks those passages is covered in the guide to showing up in Google AI Overviews.
- Keep schema markup accurate for the rich results Google still shows. The types worth adding in 2026 are listed in the explainer on what schema markup is and how to add it.
- Get named on review sites, directories and lists that AI tools cite.
The GEO guide lists the other AI add-ons Google says you can skip.
Open yourdomain.com/llms.txt today to see whether a plugin or vendor already published one, and check that every link in it still works. For help deciding where AI visibility work pays off on your site, talk to the Techno Mantra SEO team, which covers crawler access, page structure and schema. The AI development team builds llms.txt and Markdown docs for software products that need them. Send your site through the contact page or to info@technomantraa.com, and every inquiry gets a reply within one business day.
Frequently asked questions
No. It is a proposal Jeremy Howard published on llmstxt.org in September 2024. No standards body has adopted it, and no major AI search product requires it. Treat it as a community convention that some tools, mainly coding assistants, have chosen to read.
No. Google's guide to generative AI features says creating llms.txt will neither harm nor help your visibility or rankings in Google Search, because Google Search ignores the file. The only risk is a stale file that points AI tools to moved or deleted pages.
There is no evidence that it does. OpenAI's guidance for appearing in ChatGPT search asks for OAI-SearchBot access in robots.txt and does not mention llms.txt. In the Ahrefs study, OAI-SearchBot made 0.74% of the requests that llms.txt files received.
llms.txt is a short index of links with one-line notes. llms-full.txt is a single file holding the full text of a site's documentation, so an AI tool can load everything in one request. The full version only makes sense for documentation sites.
No. llms.txt is a reading list, not an access rule. To stop a crawler, use robots.txt rules for its user agent, such as GPTBot for OpenAI model training. Blocking OAI-SearchBot removes your pages from ChatGPT search answers, so decide crawler by crawler.