Skip to main content
All terms
AI Crawler Access & TechnicalUpdated

llms.txt

Also known as: llms txt, llm.txt, llms.txt file

In short

llms.txt is a proposed convention, introduced by Jeremy Howard in September 2024: a markdown file at your site root that lists your most important pages with short descriptions so a large language model can orient itself. No major AI engine has been shown to rely on it, and publishing one has no measured effect on whether you get cited.

What it proposes

The convention, set out at llmstxt.org, suggests publishing /llms.txt: a markdown file listing your most important pages with short descriptions, so a language model can orient itself without crawling everything. The proposal also suggests offering clean markdown versions of pages at the same URL with .md appended.

The format

Only the first line is required:

# Your site name

> One-sentence summary of what the site is.

## Guides
- [Getting started](https://example.com/start): what a new customer needs first
- [Pricing](https://example.com/pricing): plans and what each includes

## Optional
- [Changelog](https://example.com/changelog)
  1. An H1 with the site or project name — required.
  2. A blockquote summary — optional.
  3. H2 sections of links, each with a short description — optional.
  4. A section named Optional for secondary pages, by convention.

Why it is on the "does not work" list

Adoption by the engines is the whole question, and it has not happened. No major AI engine has documented reading the file, and analyses of citation behavior show no difference between sites that publish one and sites that do not. Publishing it changes nothing measurable.

We do publish one — citecycle.com/llms.txt — because it is generated automatically from the same content as our sitemap and costs nothing to keep current. We do not expect it to earn a single citation, and you should not either.

What to do instead

The technical work that does matter is duller:

  • Make sure AI crawlers can reach you at all. Check that GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot are not blocked in your robots.txt and not being turned away by your CDN's bot rules. This one is binary and it is checked far too rarely.
  • Keep cited pages fresh. Updated pages earn substantially more citations than stale ones.
  • Structure pages so passages can be lifted — a direct answer under a question-shaped heading.

We keep this entry because being wrong about llms.txt is common and cheap to correct, not because the idea is bad.

Frequently asked questions

What is an llms.txt file?
A plain markdown file served at /llms.txt that gives AI models a short map of your site: the site's name as a heading, an optional one-line summary, and lists of links to your key pages with a description of each. It was proposed by Jeremy Howard in September 2024, with a revised version 2 in August 2026.
Does llms.txt help with AI search visibility?
There is no evidence that it does. No major AI engine has documented using the file to choose sources, and comparisons of sites with and without one show no difference in citations. It is harmless and cheap, not a lever.
How do I create an llms.txt file?
Write a markdown file starting with an H1 of your site's name — the only required part — optionally followed by a blockquote summary, then H2 sections listing links to your important pages with a short note on each. Save it at the root of your domain as /llms.txt. Generating it from your sitemap or CMS keeps it current.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may and may not fetch, and AI crawlers do honor it. llms.txt is a reading guide with no access rules, and nothing obliges a crawler to read it.

Related terms

  • AI crawler

    An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.

  • GPTBot

    GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.

  • AI visibility

    AI visibility is how often AI answer engines name and cite your brand in response to the questions your customers ask. It is measured as two separate rates per engine — how often the prose names you, and how often the citations include your domain — tracked across repeated runs rather than single answers.

Related guides