GPTBot
又称: OpenAI GPTBot, OAI-SearchBot, ChatGPT-User
一句话解释
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
The three agents, and why the distinction matters
Treating "OpenAI's crawler" as one thing leads to the wrong robots.txt.
- GPTBot — broad crawling for training data. Blocking affects what future models know about you in general.
- OAI-SearchBot — builds the search index ChatGPT draws on when answering. Blocking removes you from that retrieval pool, which is the one that decides whether you are cited today.
- ChatGPT-User — fetches a page live, because a user asked. Blocking makes your page fail on demand for someone who specifically wanted it.
A brand that blocks all three because it read one article about GPTBot has made a much larger decision than it realised.
Writing the robots.txt correctly
The trap is that a named user-agent group replaces the wildcard group for
that agent rather than adding to it. If your User-agent: * group disallows
/admin and you add a User-agent: GPTBot group that only says Allow: /,
GPTBot is now permitted into /admin.
Every named group must repeat the disallow rules you want to keep.
robots.txt is only half the check
An allow rule proves your intent, not your accessibility. CDN and WAF bot rules block AI agents at the network layer regardless of what robots.txt says, and that is the more common cause of invisibility. Verify by fetching your own pages with each agent's user-agent string and comparing against a normal browser request — see AI crawler for the method.
常见问题
- How do I block GPTBot?
- Add a named group to robots.txt with User-agent: GPTBot and Disallow: /. Note that a named group replaces the wildcard group entirely for that agent, so any rules you want to keep must be repeated inside it — and blocking GPTBot alone does not block OAI-SearchBot or ChatGPT-User.
- Should I allow GPTBot?
- If you want your brand to be known to and recommended by ChatGPT, yes. The clearest case for blocking is paywalled or licensed content you do not want in a training corpus. For most businesses the visibility cost of blocking outweighs the control it buys.
- What is the difference between GPTBot and ChatGPT-User?
- GPTBot crawls broadly on OpenAI's schedule to gather training data. ChatGPT-User fetches a specific page at the moment a user's question requires it. Blocking ChatGPT-User means that when someone explicitly asks ChatGPT to read your page, it cannot.
相关术语
- AI crawler
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
- ChatGPT SEO
ChatGPT SEO is the practice of getting a brand named and cited in ChatGPT's answers. Because ChatGPT searches the live web before answering most commercial questions, the practical target is the pages it retrieves and the third-party sources that name you — not the model's training data.
- llms.txt
llms.txt is a proposed convention — a markdown file at your site root that summarizes your content for large language models. No major AI engine has been shown to read it, and publishing one has no measured effect on whether you get cited.
- AI citation
An AI citation is a source that an answer engine links or attributes when producing an answer. Being cited means the engine used or referenced your page. It does not mean the answer recommended your brand — those are separate outcomes, and roughly six in ten citations never name the brand they cite.
相关指南
- Generative Engine Optimization (GEO): the complete guide
GEO is the practice of getting your brand named and cited inside AI answers. What it is, how it differs from SEO, and the levers that measurably work.