AI crawler
又称: AI bot, LLM crawler, AI scraper, AI user agent
一句话解释
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
Three jobs, three kinds of crawler
Not every AI bot does the same thing, and the distinction matters when you decide what to allow.
| Purpose | Example agents | What blocking costs you |
|---|---|---|
| Training corpus | GPTBot, CCBot | Absence from future model knowledge |
| Search index | OAI-SearchBot, PerplexityBot | Absence from that product's answers |
| Live user fetch | ChatGPT-User, Claude-User | The page fails when a user asks about it |
Blocking the third category is the most immediately damaging and the least often intended: it means that when a user explicitly asks an assistant to read your page, it cannot.
The failure mode nobody notices
Most access problems are not in robots.txt. They are in a CDN or WAF bot-management rule, switched on by whoever handles security, which returns a 403 to non-browser user agents. Nothing in your marketing stack reports it, your robots.txt looks permissive, and you are invisible to every AI product simultaneously.
The test that catches it is a differential fetch: request your homepage with a normal browser user agent, then with each crawler's, and compare the status codes. A normal 200 alongside a bot 403 is a block, wherever it originates.
Check it before anything else
Crawler access is binary and it gates everything downstream. There is no point optimizing passages, chasing citations or building profiles if the engines cannot fetch the page. It takes minutes to verify and is worth re-checking periodically, because CDN defaults change without anyone telling marketing.
常见问题
- Which AI crawlers should I allow?
- At minimum the ones tied to products your customers use: GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, ClaudeBot and Claude-User for Anthropic, PerplexityBot for Perplexity, and Google-Extended for Google's AI surfaces. CCBot feeds Common Crawl, which many systems draw on indirectly.
- How do I check if AI crawlers are blocked?
- Read your robots.txt for each user agent, then fetch your own homepage while sending that crawler's user-agent string. A robots.txt that allows them is not sufficient — CDN and WAF bot rules block AI crawlers at the network layer without any robots.txt entry, and that is the more common cause.
- Should I block AI crawlers to protect my content?
- It is a legitimate choice with a clear cost. Blocking removes you from the answers those products give, including answers about your own category. Publishers with paid content often block deliberately; brands that want to be recommended almost never should.
相关术语
- GPTBot
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
- llms.txt
llms.txt is a proposed convention — a markdown file at your site root that summarizes your content for large language models. No major AI engine has been shown to read it, and publishing one has no measured effect on whether you get cited.
- AI citation
An AI citation is a source that an answer engine links or attributes when producing an answer. Being cited means the engine used or referenced your page. It does not mean the answer recommended your brand — those are separate outcomes, and roughly six in ten citations never name the brand they cite.
- AI visibility
AI visibility is how often AI answer engines name and cite your brand in response to the questions your customers ask. It is measured as two separate rates per engine — how often the prose names you, and how often the citations include your domain — tracked across repeated runs rather than single answers.
相关指南
- Generative Engine Optimization (GEO): the complete guide
GEO is the practice of getting your brand named and cited inside AI answers. What it is, how it differs from SEO, and the levers that measurably work.