Skip to main content
All terms
AI Crawler Access & TechnicalUpdated

llms.txt

Also known as: llms txt, llm.txt

In short

llms.txt is a proposed convention — a markdown file at your site root that summarizes your content for large language models. No major AI engine has been shown to read it, and publishing one has no measured effect on whether you get cited.

What it proposes

The convention suggests publishing /llms.txt: a markdown file listing your most important pages with short descriptions, so a language model can orient itself without crawling everything. The idea is reasonable and the format is pleasant to write.

Why it is on the "does not work" list

Adoption by the engines is the whole question, and it has not happened. No major AI engine has documented reading the file, and analyses of citation behavior show no difference between sites that publish one and sites that do not. Publishing it costs an hour and changes nothing measurable.

What to do instead

The technical work that does matter is duller:

  • Make sure AI crawlers can reach you at all. Check that GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot are not blocked in your robots.txt and not being turned away by your CDN's bot rules. This one is binary and it is checked far too rarely.
  • Keep cited pages fresh. Updated pages earn substantially more citations than stale ones.
  • Structure pages so passages can be lifted — a direct answer under a question-shaped heading.

We keep this entry because being wrong about llms.txt is common and cheap to correct, not because the idea is bad.

Related terms

  • AI crawler

    An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.

  • GPTBot

    GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.

  • AI visibility

    AI visibility is how often AI answer engines name and cite your brand in response to the questions your customers ask. It is measured as two separate rates per engine — how often the prose names you, and how often the citations include your domain — tracked across repeated runs rather than single answers.

Related guides