ClaudeBot
Also known as: Anthropic crawler, Claude-User, Claude-SearchBot, ClaudeBot user agent
In short
ClaudeBot is Anthropic's web crawler, which collects publicly available web content for Claude. Anthropic runs two other agents: Claude-User, which fetches a page when a user's request needs it, and Claude-SearchBot, which indexes content for Claude's search. Each is named separately in robots.txt and can be allowed or blocked independently.
Three agents, three effects
Anthropic documents three user agents. They do different jobs, and blocking each has a different cost.
| Agent | What it does | Cost of blocking |
|---|---|---|
| ClaudeBot | Collects web content on Anthropic's schedule | Claude knows less about you generally |
| Claude-User | Fetches a page when a user's request needs it | The fetch fails for someone who asked for you |
| Claude-SearchBot | Analyzes content to improve Claude's search results | Your pages can drop out of Claude's search answers |
Blocking Claude-User is rarely intended and immediately visible to a real person who wanted your content. Blocking Claude-SearchBot matters most for visibility in answers, because it is the agent behind Claude's own search.
Copy-paste robots.txt rules
To allow all three while keeping private areas closed (repeat your own disallow lines in each group):
User-agent: ClaudeBot
Allow: /
Disallow: /admin
User-agent: Claude-User
Allow: /
Disallow: /admin
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
To block collection but stay reachable in search and for users:
User-agent: ClaudeBot
Disallow: /
Leaving Claude-User and Claude-SearchBot without a named group means
they follow your User-agent: * rules.
Writing the rule correctly
A named user-agent group replaces the wildcard group for that agent rather
than adding to it. If User-agent: * disallows /admin and you add a
ClaudeBot group containing only Allow: /, you have just permitted ClaudeBot
into /admin.
Repeat every disallow you want to keep inside each named group. This is the single most common robots.txt mistake when adding AI crawler rules.
robots.txt is only half of it
An allow directive states intent; it does not guarantee access. CDN and WAF bot rules block AI agents at the network layer with no robots.txt involvement, and that is the more frequent cause of invisibility. Verify with a differential fetch — normal user agent versus bot user agent — as described under AI crawler.
Anthropic's own reference for these agents, including its published IP ranges, is its help article on how Anthropic crawls the web. For the bigger picture of why crawler access is the first thing to check, see the guide to generative engine optimization.
Frequently asked questions
- How do I block ClaudeBot?
- Add a named group to robots.txt with User-agent: ClaudeBot and Disallow: /. A named group replaces the wildcard group for that agent, so repeat any other disallow rules you want to keep — and blocking ClaudeBot does not block Claude-User or Claude-SearchBot, which have their own user-agent names.
- Should I allow ClaudeBot?
- If you want Claude to be able to describe your business accurately, yes. Blocking removes you from what it can say about your category. The main reason to block is content you license or paywall and do not want collected.
- What is the difference between ClaudeBot, Claude-User and Claude-SearchBot?
- ClaudeBot collects web content on Anthropic's schedule. Claude-User fetches one page at the moment a user asks Claude about it. Claude-SearchBot analyzes content to improve the quality of Claude's search results. Blocking Claude-User means a person who explicitly asks Claude to read your page gets an error; blocking Claude-SearchBot can keep your pages out of Claude's search answers.
- Can I slow ClaudeBot down instead of blocking it?
- Yes. Anthropic supports the non-standard Crawl-delay directive, for example User-agent: ClaudeBot followed by Crawl-delay: 1. Most sites never need it.
- How do I know a request really came from Anthropic?
- Anthropic publishes the IP ranges its crawlers use. A request claiming to be ClaudeBot from an address outside those ranges is not Anthropic.
Related terms
- AI crawler
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
- GPTBot
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
- robots.txt
robots.txt is a plain-text file at your site root that tells automated crawlers which paths they may fetch. Each block names a user agent and lists allow and disallow rules. It is advisory rather than enforced, and it controls crawling rather than indexing or access.
- PerplexityBot
PerplexityBot is Perplexity's crawler, used to index pages for the search product behind its answers. Perplexity-User is the separate agent that fetches a page live when a user's question requires it. Both can be allowed or blocked by name in robots.txt.
Related guides
- Generative engine optimization (GEO): what it is and how it works
What generative engine optimization (GEO) is, how ChatGPT, Gemini, Claude and Google's AI Overviews pick the sources and brands they name, how GEO differs from SEO, and the levers that measurably work.