robots.txt
Also known as: robots txt, robots exclusion protocol, robots file
In short
robots.txt is a plain-text file at your site root that tells automated crawlers which paths they may fetch. Each block names a user agent and lists allow and disallow rules. It is advisory rather than enforced, and it controls crawling rather than indexing or access.
The group-matching rule that catches everyone
This is the single most consequential detail and it is counterintuitive: a crawler obeys only the most specific matching group, and that group replaces the wildcard group entirely.
User-agent: *
Disallow: /admin
User-agent: GPTBot
Allow: /
GPTBot is now permitted into /admin, because it reads its own group and
ignores * completely. Every named group must repeat the disallow rules you
want to keep.
What it controls, and what it does not
| Controls | Does not control |
|---|---|
| Whether a compliant crawler fetches a path | Whether a URL appears in an index |
| Crawl load on your server | Access by anything ignoring the file |
| Which named agents may fetch what | What a CDN or WAF does at the network layer |
That last row is where most AI-visibility problems actually live. A permissive robots.txt sitting behind a bot rule that returns 403 to non-browser user agents reads as "allowed" while being effectively blocked — see AI crawler for how to test it.
Stating an AI policy explicitly
Naming AI agents individually — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — is worth doing even when the wildcard already allows them. It documents a deliberate decision, and it makes the file legible to anyone auditing whether AI systems may read the site.
Frequently asked questions
- Does robots.txt keep a page out of search results?
- Not reliably. It asks crawlers not to fetch the page, but a URL that is linked from elsewhere can still appear in results without a description. To keep something out of an index, use a noindex directive on a page that is crawlable, or require authentication.
- Does robots.txt actually block anything?
- It blocks well-behaved crawlers that choose to honour it. It is a convention, not a security control, and it does not stop anything determined to ignore it. Never use it to protect sensitive paths — listing them there advertises exactly what you want hidden.
- How do I allow AI crawlers in robots.txt?
- Add a named group for each agent with Allow: /, repeating any disallow rules from your wildcard group. Naming them explicitly also documents your policy, which matters when someone later audits whether AI systems may read the site.
Related terms
- AI crawler
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
- GPTBot
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
- ClaudeBot
ClaudeBot is Anthropic's web crawler, used to gather publicly available content for Claude. A separate agent, Claude-User, fetches a specific page live when a user's request requires it. Both are identified by name in robots.txt and can be allowed or disallowed independently.
- Google-Extended
Google-Extended is a robots.txt token that controls whether Google may use your content to ground and improve its generative AI products, including Gemini and AI Overviews. It is not a crawler — Googlebot still fetches your pages for search regardless of what you set.
Related guides
- Generative Engine Optimization (GEO): the complete guide
GEO is the practice of getting your brand named and cited inside AI answers. What it is, how it differs from SEO, and the levers that measurably work.