ClaudeBot
又称: Anthropic crawler, Claude-User, ClaudeBot user agent
一句话解释
ClaudeBot is Anthropic's web crawler, used to gather publicly available content for Claude. A separate agent, Claude-User, fetches a specific page live when a user's request requires it. Both are identified by name in robots.txt and can be allowed or disallowed independently.
Two agents, two effects
The distinction mirrors the pattern across AI companies and is worth getting right before editing robots.txt.
| Agent | When it runs | Cost of blocking |
|---|---|---|
| ClaudeBot | On Anthropic's crawl schedule | Claude knows less about you generally |
| Claude-User | When a user's request needs your page | The fetch fails for someone who asked for you |
Blocking the second is rarely intended and immediately visible to a real person who wanted your content.
Writing the rule correctly
A named user-agent group replaces the wildcard group for that agent rather
than adding to it. If User-agent: * disallows /admin and you add a
ClaudeBot group containing only Allow: /, you have just permitted ClaudeBot
into /admin.
Repeat every disallow you want to keep inside each named group. This is the single most common robots.txt mistake when adding AI crawler rules.
robots.txt is only half of it
An allow directive states intent; it does not guarantee access. CDN and WAF bot rules block AI agents at the network layer with no robots.txt involvement, and that is the more frequent cause of invisibility. Verify with a differential fetch — normal user agent versus bot user agent — as described under AI crawler.
常见问题
- How do I block ClaudeBot?
- Add a named group to robots.txt with User-agent: ClaudeBot and Disallow: /. Remember that a named group replaces the wildcard group for that agent, so repeat any disallow rules you want to keep — and note that blocking ClaudeBot does not block Claude-User.
- Should I allow ClaudeBot?
- If you want Claude to be able to describe your business accurately, yes. Blocking removes you from what it can say about your category. The main reason to block is content you license or paywall and do not want in a training corpus.
- What is the difference between ClaudeBot and Claude-User?
- ClaudeBot crawls broadly on Anthropic's schedule. Claude-User fetches one page at the moment a user asks about it. Blocking Claude-User means a person who explicitly asks Claude to read your page gets an error instead.
相关术语
- AI crawler
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
- GPTBot
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
- robots.txt
robots.txt is a plain-text file at your site root that tells automated crawlers which paths they may fetch. Each block names a user agent and lists allow and disallow rules. It is advisory rather than enforced, and it controls crawling rather than indexing or access.
- PerplexityBot
PerplexityBot is Perplexity's crawler, used to index pages for the search product behind its answers. Perplexity-User is the separate agent that fetches a page live when a user's question requires it. Both can be allowed or blocked by name in robots.txt.
相关指南
- Generative Engine Optimization (GEO): the complete guide
GEO is the practice of getting your brand named and cited inside AI answers. What it is, how it differs from SEO, and the levers that measurably work.