What is llms.txt?

llms.txt is a proposed standard for AI discoverability. Concretely, it is a plain-text markdown file placed at the root of a website — for example https://yoursite.com/llms.txt — that tells AI search engines and AI agents what the site is about and which pages matter most. It was proposed by Answer.AI co-founder Jeremy Howard in September 2024 at llmstxt.org, and is often described as "robots.txt for AI" — except that where robots.txt tells crawlers what not to access, llms.txt tells them what the site actually provides.

What does an llms.txt file contain?

The format is deliberately simple: an H1 heading with the project name, a one-line summary in a blockquote, and H2 sections of markdown links with short descriptions:

# Acme Docs

> Acme Docs is a documentation site for the Acme API.

## Documentation

- [Getting started](https://acme.dev/docs/start): install the SDK in 5 minutes
- [API reference](https://acme.dev/docs/api): every endpoint and parameter

## Related

- [Sitemap](https://acme.dev/sitemap.xml): machine-readable URL index
- [llms-full.txt](https://acme.dev/llms-full.txt): self-contained full content

Why do AI search engines need it?

ChatGPT, Claude, Perplexity and Gemini answer questions by crawling the web, but crawling a site gives them thousands of URLs without telling them what the site is for. llms.txt solves this with an orientation file: an AI agent reads one short document and immediately knows your structure — documentation, pricing, blog — and can decide which pages deserve a deeper fetch. Studies of AI citation behavior (Victorino LLC, The Digital Bloom, Position Digital, Omniscient Digital) consistently show that structured, entity-rich, question-oriented documents are cited more often than unstructured pages. That is the premise behind the GEO score built into this generator.

What is llms-full.txt?

A companion file that inlines the full content of linked pages as one self-contained markdown document. Where llms.txt is a navigator (typically 1–50 KB of links), llms-full.txt can be 100 KB to several MB of content, letting an AI agent ingest an entire documentation corpus in a single HTTP request — ideal for RAG pipelines and AI support bots.

How does llms.txt differ from robots.txt and sitemap.xml?

robots.txt controls crawler access with allow/disallow rules; sitemap.xml lists every URL in machine-readable XML; llms.txt describes what those URLs mean in a human- and AI-readable summary. The three files are complementary — a complete AI setup publishes all three, and llms.txt links to the other two. See llms.txt vs robots.txt and llms.txt vs sitemap.xml for detailed comparisons.

Is llms.txt supported yet?

Adoption is early. Google has stated it does not officially support the file, and a 2026 Ahrefs study found most domains with a valid llms.txt received zero requests for it in a month. What that means in practice: publishing llms.txt costs nothing, takes minutes, and positions you early for a convention that AI tools are increasingly checking. The LLMsTxt Generator produces the file from your sitemap in seconds — including a GEO score that shows how citable your file is.