What Is llms.txt? The Honest 2026 Explainer

1. What llms.txt Actually Is

llms.txt is a plain-text Markdown file that a site publishes at its root — yoursite.com/llms.txt — containing a short, hand-written summary of the site: what it is, what it does, and a curated list of its most important pages. The idea is that a large language model reading the file gets a compact, accurate primer instead of having to piece one together from navigation menus, ads, and JavaScript-heavy HTML.

It is not a technical standard in the way robots.txt or sitemap.xml are. Nobody enforces it, no browser reads it, and as you'll see below, the companies whose products it's meant to help haven't committed to reading it either. It's best understood as a proposed convention — something a site owner can choose to publish, in the hope that some current or future AI system finds it useful.

It matters to site owners specifically because it sits at the intersection of two trends that are both very real: search traffic increasingly gets filtered through AI-generated answers instead of a list of blue links, and a genuine cottage industry of "AI SEO" and "GEO" consultants has sprung up promising to help sites win in that new landscape. llms.txt is one of the cheapest, most heavily promoted tactics in that space — which is exactly why it deserves a closer look than the average listicle gives it, rather than a reflexive "just add it, it can't hurt" shrug.

2. Where the Idea Came From

llms.txt was proposed on September 3, 2024 by Jeremy Howard, co-founder of the AI research lab Answer.AI and previously known for the fast.ai deep learning courses. The full specification and rationale live at llmstxt.org, the project's own site.

Howard's stated problem was a real one: language models have limited context windows, and a typical web page — full of navigation, ads, cookie banners, and script tags — wastes most of that budget on things that aren't the actual content. His proposal: give models a clean, curated shortcut. The related idea of appending .md to a page's URL to get a clean Markdown version (so yoursite.com/page also exists at yoursite.com/page.md) comes from the same proposal and is meant to work alongside the root llms.txt file.

The idea picked up genuine attention in the developer and SEO communities through 2025, and early support came from a handful of documentation-heavy tools and, according to reporting at the time, some interest from Perplexity and Anthropic. That's a meaningfully different claim from "AI crawlers use this file to rank or cite your site" — a distinction that a lot of 2026 marketing content blurs.

3. The Format: What Goes in the File

The spec is deliberately simple. A valid llms.txt has:

• An H1 with the site or project name (required)
• A blockquote with a one- to two-sentence summary (required)
• Optional plain paragraphs with extra context
• One or more H2 sections, each a short list of links with optional descriptions
• An optional final "Optional" H2 section for lower-priority links — the spec explicitly allows a model with limited context to skip this section first

In practice, a minimal file looks like this:

# Key For Riches

> Free SEO and AI tools for keyword research, site audits, rank tracking
> and content optimization. No signup required.

## Docs
- [All Tools](https://tools.keyforriches.com/): Full catalog of free tools
- [Blog](https://tools.keyforriches.com/blog/): SEO guides paired with tools

## Optional
- [Privacy Policy](https://tools.keyforriches.com/privacy-policy.php)

That's the whole shape of it — a name, a summary, and a short, deliberately curated set of links. The spec is on GitHub and documented in full at llmstxt.org if you want to read the original reasoning rather than a summary of it.

Notice what's absent from that structure: there's no validator any AI company officially runs against it, no required fields beyond the name and summary, and nothing stopping a site from publishing one that wildly oversells what it does. That's not a flaw unique to this proposal — plenty of useful web conventions started as unenforced suggestions — but it's worth being clear-eyed that "spec-compliant" and "actually read by anything" are two separate claims, and only the first one is guaranteed by following the format correctly.

4. llms.txt vs. robots.txt vs. sitemap.xml

These three files sit at the same address — your site's root — and get confused constantly, but they do genuinely different jobs:

• robots.txt is a permission file. It tells crawlers where they're allowed to go, and it's respected (imperfectly) by essentially every well-behaved bot, search or AI. Read our robots.txt and sitemaps explainer for the full mechanics.
• sitemap.xml is a discovery file. It lists every URL you want indexed, so crawlers can find pages they might otherwise miss. More is generally better here — completeness is the goal.
• llms.txt is a curation file. It deliberately lists only your best pages, written for something with a small context budget to skim. Completeness is explicitly not the goal — the spec asks you to leave things out.

The practical difference: robots.txt and sitemap.xml are consumed by crawlers that everyone agrees to support, and skipping them has measurable, well-documented consequences. llms.txt is consumed by — as the next section covers — almost nobody, yet.

5. Does Anyone Actually Read It? The 2026 Evidence

This is the part most "add llms.txt now" articles skip, and it's the most important part. Three independent data points, from three different angles, all point the same direction:

Google says no, directly. Google's own guide to optimizing for generative AI features states that you don't need special machine-readable files, AI text files, or Markdown to appear in Google Search or its AI features — Google Search doesn't use them. Search Advocate John Mueller went further in comments covered by Search Engine Land, comparing llms.txt to the old keywords meta tag: a claim a site owner makes about their own content, unverified and easy to game, which is exactly why search engines stopped trusting the keywords tag decades ago. He added that server logs show AI services largely don't even check for the file.

The crawl logs agree. Cloudflare, which sits in front of a meaningful share of the web and can see raw bot traffic, published data in its crawler behavior analysis showing GPTBot, ClaudeBot and the other major AI crawlers overwhelmingly request ordinary HTML pages — not curated Markdown summaries at a special path.

And the adoption data confirms it's mostly theater. Ahrefs analyzed the actual outcome at scale in its study "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" — a striking number that lines up with the smaller-scale crawl-log studies circulating elsewhere, which put llms.txt fetches at a tiny fraction of one percent of total AI bot traffic. Roughly one in ten sites now has a file, per separate industry tracking, but having one and being read are two very different things.

6. Where It Actually Works: Developer Docs

The one place llms.txt has real, measurable traction is developer documentation — and it's worth understanding why, because it clarifies exactly when the file is and isn't useful. In November 2024, the documentation-hosting platform Mintlify rolled out automatic llms.txt and llms-full.txt generation for every site it hosts, instantly giving thousands of companies' docs a compliant file with zero effort. That single rollout is why the documentation for Anthropic, Cursor, Coinbase, Pinecone and Windsurf all have one, according to Mintlify's own write-up of real-world examples — not because each company individually decided AI visibility was worth chasing.

The reason this context is different: AI coding assistants like Cursor and Claude Code actively fetch documentation while a developer is working, to answer "how do I call this API" in real time. That's a fundamentally different task from a general web-search AI deciding what to cite in an answer, and it's a case where a clean, complete Markdown dump of your docs (the llms-full.txt variant, also born out of Mintlify and Anthropic's collaboration) genuinely saves a coding tool from parsing your HTML. If you run developer-facing documentation on a platform that generates this automatically, there's a real, narrow case for it. That's a meaningfully different claim from "any website should have an llms.txt to get cited by ChatGPT" — the pitch most 2026 SEO content is actually making, and the one the evidence above doesn't support.

7. Should You Add One Anyway?

Given the evidence above, here's the balanced take: yes, probably, but for the right reason. A well-written llms.txt costs about five minutes and zero ongoing maintenance. There's no penalty for having one, and standards that start with near-zero adoption sometimes do get picked up later — llms.txt already has stated interest from Anthropic and Perplexity, even if it isn't load-bearing today for general search or AI-overview visibility. Think of it the way you'd think of registering a domain variant you might need eventually: cheap insurance, not a growth tactic.

What it should not do is replace the things that demonstrably do affect whether AI systems surface and describe your site accurately: clear, well-structured HTML content; genuine E-E-A-T signals like real authorship and cited sources; accurate schema markup; a correct robots.txt and sitemap; and pages that directly and clearly answer the questions people are actually asking. If you're choosing where to spend an afternoon, an AI visibility check on your existing content will teach you more than an llms.txt file will — and if you run product or API documentation rather than a content or marketing site, the calculus shifts further in favor of having one, for the developer-tooling reasons above rather than for search visibility.

8. How to Create Yours in Under 5 Minutes

If you've read all that and still want one — and there's no good reason not to, given how cheap it is — keep it genuinely curated rather than dumping your whole sitemap into it. Pick your name and a one-sentence summary, group a handful of your best pages under short headings, and put anything lower-priority (privacy policy, affiliate disclosure) in the Optional section at the end.

For the field-by-field walkthrough of the generator itself — what belongs in each section and where exactly to upload the finished file — see the full generator guide.

9. Quick Answers

Will llms.txt get me cited in ChatGPT or Google AI Overviews? There's no current evidence it does. Google explicitly says it doesn't use it; independent crawl-log data shows AI bots rarely request it at all.

Does it hurt to have one? No. It costs a few minutes to create and carries no known downside — the concern is only that it's frequently sold as more effective than the evidence supports.

Is it different from a sitemap? Yes — a sitemap aims for completeness so search engines can index everything; llms.txt aims for a short, curated shortlist for something with limited context to skim.

Is llms-full.txt the same thing? It's a companion variant from the same proposal — a single file containing your entire documentation in clean Markdown, rather than a short curated summary with links. It's the version Mintlify and Anthropic built together, and it's aimed squarely at the developer-docs use case in the previous section, not general site discovery.

Do I need developer skills to add one? No — it's a plain text file. If your site runs on a docs platform like Mintlify it may already be generated automatically; otherwise a generator can produce the file for you to upload the same way you'd upload robots.txt.

Should I revisit this later? Worth it. Adoption, crawler behavior, and official positions from AI companies are all still moving in 2026 — what's true today may not hold in a year.