llms.txt: A Practical Implementation Guide

llms.txt is a plain-text Markdown file, placed at a website’s root, that gives AI systems a curated index of the site’s most important pages — a proposed convention from llmstxt.org for making content easy for large language models to find, parse, and ingest. It is not a ranking mechanism and not an official standard. It is a hint, and this guide covers exactly what it does, how to build one, and what the evidence says about whether it is worth your time.

We run llms.txt on this site and auto-generate it on every build, so the walkthrough below is what we actually operate — not a theoretical spec reading.

What llms.txt Is (and Is Not)

The idea behind llms.txt is simple. A large language model that wants to answer a question about your site has to work from whatever it can crawl, and most web pages are cluttered with navigation, scripts, and boilerplate that waste an AI’s limited context window. llms.txt offers a clean, prioritized map: here are the pages that matter, in plain Markdown, with one-line descriptions.

It helps to be precise about what it is not:

• Not a ranking factor. No major AI vendor has confirmed that publishing an llms.txt improves how often you are cited. Treat it as content hygiene, not an optimization lever.
• Not an official standard. As of mid-2026 llms.txt is a community convention with no backing from the IETF, W3C, or any recognized standards body.
• Not a replacement for good content or SEO. It points AI at your pages; it does not make weak pages authoritative.

Think of it as the AI-era analogue of a sitemap: a courtesy index that costs little to maintain and may help the systems that choose to read it.

The llms.txt Format

The llmstxt.org spec defines a small, strict Markdown structure so the file is readable by both humans and machines:

1. An H1 with the site or project name (the only required element).
2. A blockquote summary immediately after the H1 — a short paragraph describing what the site is and who it serves.
3. H2 sections, each grouping related pages as a Markdown link list, where each item is [Page title](URL): one-line description.

A minimal llms.txt example reads top to bottom like this (shown with each Markdown mark inline so you can see the exact syntax):

• The H1 site name: # Example Co
• A blockquote summary: > Example Co publishes independent guides on customer data platforms for marketing, data, and IT leaders.
• An H2 section heading: ## Guides
• One link per line beneath it: - [What Is a CDP?](https://example.com/what-is-a-cdp/): Definition, how it works, and how to choose one.
• Another link: - [CDP vs CRM](https://example.com/cdp-vs-crm/): Key differences and when you need each.
• A second H2 section: ## Glossary
• Its links: - [Identity Resolution](https://example.com/identity-resolution/): How profiles are stitched from many identifiers.

That is the entire format. Its restraint is deliberate — a file this predictable is trivial for a model to parse reliably.

llms.txt vs llms-full.txt vs robots.txt

These three files are often confused. They do different jobs and coexist without conflict.

robots.txt governs permission; llms.txt and llms-full.txt govern convenience. The split between the two llms files matters: llms.txt is a lightweight table of contents an AI reads to decide what is relevant, while llms-full.txt is the heavy payload — the full Markdown body of every page — that a model can ingest in a single pass without crawling each URL. Small sites may only need llms.txt; content-heavy sites benefit from offering both.

How to Create an llms.txt File

There are two approaches, and the right one depends on how often your content changes.

Manual Approach

For a small or slow-moving site, write the file by hand:

1. Create llms.txt in your site’s public root so it serves at https://yourdomain.com/llms.txt.
2. Add the H1 (site name) and a one-paragraph blockquote summary.
3. Group your most important pages under H2 headings, each as a [title](url): description list item.
4. Order by importance — put the pages you most want cited first.
5. Deploy it as a static file and re-edit whenever key pages change.

The risk with the manual approach is drift: the file goes stale the moment you publish new content and forget to update it.

Automated Approach

For any site that publishes regularly, generate the file from your content source so it can never fall out of sync. The pattern: read your content directory at build time, extract each page’s title and description from its frontmatter, and write out both llms.txt (index) and llms-full.txt (full bodies). Wire the generator in as the first step of your build so both files always reflect the current content.

How cdp.com Generates Its llms.txt

This site takes the automated approach, and the mechanics are worth showing because they are the part no generic guide can describe first-hand.

A Node script (scripts/generate-llms-txt.mjs) runs as the first step of the build. It walks the five content collections — basics, glossary, articles, industry studies, and podcasts — reads the title and description from each Markdown file’s frontmatter, and emits two files into the public root:

• llms.txt — a curated preamble (the CDP definition, the 3-stage evolution framing) followed by one link line per page, grouped by collection. This is the index.
• llms-full.txt — the same preamble followed by the complete Markdown body of every page, separated by dividers. This is the full corpus for ingestion.

Two design choices make it durable. First, because generation is tied to the build, the files regenerate on every deploy — a new glossary entry appears in both files automatically, with no manual step to forget. Second, the curated preamble (the parts that state cdp.com’s canonical framing) is maintained in the script itself, not derived from a page, so the taxonomy the site wants AI to absorb is stated once, authoritatively, at the top of both files. The per-page lines are mechanical; the framing is deliberate.

The build log confirms the output on each run — the page count and per-collection totals are printed so a stale or truncated file is visible immediately.

Verifying Your llms.txt

After deploying, confirm three things:

• It serves. Fetch https://yourdomain.com/llms.txt and check it returns the file as text/plain or text/markdown, not a 404 or an HTML error page.
• It parses. Confirm the H1, blockquote, and H2 link-list structure match the spec — malformed Markdown defeats the purpose.
• It stays current. If auto-generated, publish a new page and verify it appears after the next build. If manual, put a recurring reminder on your calendar.

Does It Actually Work? An Honest Adoption Status

This is where most guides overclaim. The honest picture in 2026:

• Google does not support it. Google’s Gary Illyes stated publicly at Search Central Live (July 2025) that Google does not use llms.txt and has no plans to. Gemini and Google-Extended behavior does not change in response to the file.
• Other engines are partial and mostly unconfirmed. Some retrieval-oriented AI products are reported to fetch llms.txt, but the major model vendors have not published clear commitments that they read it in production, and none has confirmed it as a citation or ranking signal.
• Adoption is early. An SE Ranking analysis of 300,000 domains found roughly a 10% adoption rate — about one site in ten — after more than a year of industry discussion.

The reasonable conclusion: publish an llms.txt if the cost is near zero (especially if you can auto-generate it), because it may help the systems that choose to read it and it does no harm. Do not expect it to move citation rates on its own, and do not treat it as a substitute for the entity consistency, structured data, and authoritative content that AI engines demonstrably do reward.

An llms.txt is only as good as the content it indexes. The same discipline that makes a page worth citing — authoritative, first-party, well-structured — is what makes the file worth publishing at all.

Related Articles

• GEO vs SEO: What Changes for AI Search — Where llms.txt fits in the broader shift to AI-answer optimization
• What Is LLM SEO? — The technical mechanics of how models find and cite content, of which llms.txt is one lever
• How AI Is Redefining the CDP — Why structured, unified data signals matter to AI systems
• Generative Engine Optimization (GEO) — The discipline of earning citations in AI-generated answers
• AI Use Cases That Are Reshaping Marketing in 2026 — How AI search and content optimization sit in the marketing stack

FAQ

Does ChatGPT read llms.txt?

There is no confirmation from OpenAI that ChatGPT reads llms.txt in production. Some observers report correlations in citation behavior for sites that publish the file, but OpenAI has not stated that it uses llms.txt, and it is not a confirmed ranking or citation signal. Publish one if it is cheap to maintain, but do not assume ChatGPT depends on it.

What is the difference between llms.txt and robots.txt?

robots.txt controls crawler access; llms.txt indexes content for AI ingestion. robots.txt tells user-agents which URLs they may or may not fetch — a permission layer. llms.txt does the opposite job: it hands AI systems a curated, prioritized Markdown list of your most valuable pages to make ingestion efficient. They serve different purposes and should both exist on a site that wants AI visibility.

Do I need llms-full.txt as well?

Only if your site is content-heavy. llms.txt is a lightweight index; llms-full.txt concatenates the full text of your pages into one file so a model can ingest everything in a single pass without crawling each URL. Small sites can rely on llms.txt alone. Content libraries with dozens or hundreds of pages benefit from also publishing llms-full.txt, ideally auto-generated so it stays current.

Is llms.txt an official standard?

No — llms.txt is a community convention, not an official standard. It originates from llmstxt.org and has no backing from the IETF, W3C, or any recognized standards body as of 2026. That does not make it useless, but it means support is voluntary and varies by vendor, and the format could change. Treat it as a low-cost convention rather than a guaranteed, standardized interface.