Google-Extended
Also known as: Google Extended, Gemini crawler token
In short
Google-Extended is a robots.txt token that controls whether Google may use your content to ground and improve its generative AI products, including Gemini and AI Overviews. It is not a crawler — Googlebot still fetches your pages for search regardless of what you set.
A permission, not a bot
Most robots.txt entries name something that will visit you. Google-Extended does not. It is a control token: Googlebot does the crawling, and the token tells Google what it may do with what it already fetched.
This is why you will never see Google-Extended in your access logs, and why "blocking" it does not reduce crawl load.
The trade-off, stated plainly
| Setting | Search | AI Overviews & Gemini |
|---|---|---|
| Allow | Unaffected | Eligible to be used and cited |
| Disallow | Unaffected | Excluded |
Disallowing does not reclaim the click that an AI Overview absorbed. The overview still appears; it is simply built from someone else. For most brands that is the worst of both outcomes — the traffic loss without the visibility.
The genuine case for blocking is a publisher or licensor whose content is the product, where being ingested is a commercial harm rather than a marketing opportunity. That is a real position, and it is not most businesses.
Check it is not set by accident
Google-Extended is occasionally disallowed by a template, a plugin default, or a decision made during an unrelated policy discussion years earlier. It is worth reading your live robots.txt rather than assuming, because nothing surfaces this in analytics — the absence of AI citations looks identical to never having earned any.
Frequently asked questions
- Does blocking Google-Extended affect my search rankings?
- No. It is a separate control from Googlebot, and disallowing it does not remove you from Google Search or change how you rank. What it does affect is whether your content can be used in Google's AI answers.
- Should I block Google-Extended?
- Only if you have a specific reason to keep content out of AI answers — licensed material, paywalled archives, a publisher strategy. For a brand that wants to be recommended, blocking removes you from AI Overviews and Gemini without returning the traffic those surfaces absorbed.
- Is Google-Extended a crawler?
- No, and this is the common misunderstanding. It is a permission token evaluated in robots.txt, not a user agent that fetches pages. Nothing identifies itself as Google-Extended in your server logs.
Related terms
- AI crawler
An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to build a training corpus, to maintain a search index, or to retrieve a page live while answering a user's question. Each is identified by a named user agent you can allow or block in robots.txt.
- robots.txt
robots.txt is a plain-text file at your site root that tells automated crawlers which paths they may fetch. Each block names a user agent and lists allow and disallow rules. It is advisory rather than enforced, and it controls crawling rather than indexing or access.
- AI Overviews
AI Overviews are AI-generated summaries Google places above the traditional results for many queries. They synthesize several sources into a short answer with links to what was used, pushing the classic blue links further down the page and often resolving the query without a click.
- GPTBot
GPTBot is OpenAI's crawler for gathering publicly available web content used in training its models. It is one of three OpenAI agents: GPTBot for training, OAI-SearchBot for the search index behind ChatGPT's answers, and ChatGPT-User for fetching a page live when a user's question requires it.
Related guides
- Generative Engine Optimization (GEO): the complete guide
GEO is the practice of getting your brand named and cited inside AI answers. What it is, how it differs from SEO, and the levers that measurably work.