AI crawler controls are not one switch

By SimonSeptember 16, 2026

Google-Extended, Googlebot and AI-service bots govern different uses. Map training, search and user-triggered retrieval before changing crawler rules.

Three freestanding model doorways with separate paper routes and different opening positions.

AI-generated conceptual illustration of purpose-specific crawler policies. Robots.txt communicates preferences; it is not a security boundary.

You may want readers to find your articles through search without allowing those articles to train AI models. That is a reasonable policy, but it does not translate into a single setting. You need to know which service each control affects.

I would not hand a web lead a ticket that just says "block AI". It could mean blocking training crawls, search indexing, retrieval for a specific user, or reuse of content already collected. Those are different requests. The provider documentation below describes different controls, and no two providers draw the boundaries quite the same way.

Write down which uses you want to allow and which you want to restrict. Then ask the web lead to check the current documentation and prepare a configuration for your site. Copying another publisher's rules could block discovery you meant to keep.

Google-Extended is not an AI Overviews switch

Google's crawler documentation describes Google-Extended as a standalone product token. It governs specified uses of crawled content: training future Gemini generations powering Gemini Apps and the Vertex AI API for Gemini, plus grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Here, grounding means giving a model content from the Search index at prompt time to improve factuality and relevance.

Calling Google-Extended a training switch misses the grounding uses. If you restrict it, you are making a decision about both.

Google says Google-Extended does not affect inclusion in Google Search and is not a Search ranking signal. It also has no separate HTTP request user-agent string: Google uses existing crawler user agents and applies the robots.txt token as a control. Don't expect to prove that preference is working by finding requests labelled Google-Extended in server logs.

AI Overviews and AI Mode belong to Search. In its guidance on AI features, Google says these features use Googlebot crawl controls and Search preview controls. To be considered for a supporting link, a page must be indexed and eligible to appear with a snippet. Blocking Google-Extended does not opt a page out of AI Overviews or AI Mode.

Google Search and specified Gemini uses shown as separate policy scopes — Conceptual scope map based on Google's documentation: Google-Extended covers specified Gemini training and grounding uses, not AI Overviews or AI Mode in Search.

What each provider says its bots do

The table below follows documentation retrieved September 15, 2026: Google's crawler guidance above, OpenAI's bot definitions and Anthropic's crawler controls. These are the providers' stated behaviors. I have not independently audited every request.

Control or agent: `Googlebot`

Documented purpose: Google Search crawling, including Search features.

Boundary a publisher should retain: Blocking it can obstruct ordinary discovery as well as AI features; it is not a training-only choice

Control or agent: `Google-Extended`

Documented purpose: Specified Gemini training and grounding uses.

Boundary a publisher should retain: Does not control Search inclusion or ranking; no separate HTTP user agent

Control or agent: `GPTBot`

Documented purpose: Crawls content that may be used to train OpenAI foundation models.

Boundary a publisher should retain: Independent from OpenAI's search preference

Control or agent: `OAI-SearchBot`

Documented purpose: Surfaces sites in ChatGPT search.

Boundary a publisher should retain: Opted-out sites are excluded from search answers but can still appear as navigational links.

Control or agent: `ChatGPT-User`

Documented purpose: Certain user-initiated visits in ChatGPT and custom GPTs.

Boundary a publisher should retain: Not automatic crawling or the Search opt-out control; robots.txt rules may not apply.

Control or agent: `ClaudeBot`

Documented purpose: Collects material potentially used for model training.

Boundary a publisher should retain: Restricting it signals exclusion of future materials from training datasets.

Control or agent: `Claude-SearchBot`

Documented purpose: Indexing and analysis for search quality.

Boundary a publisher should retain: Disabling it can reduce visibility and accuracy in Claude search results.

Control or agent: `Claude-User`

Documented purpose: Retrieval in response to a user's request.

Boundary a publisher should retain: Anthropic says site owners can control this access and its bots honor robots.txt.

OpenAI explicitly allows you to set GPTBot and OAI-SearchBot separately, so you can allow search while disallowing training use. Anthropic also separates training, search and user-request bots. But notice the difference on user requests: Anthropic says its bots honor robots.txt, while OpenAI says those rules "may not apply" to ChatGPT-User. You cannot carry one company's promise over to another.

Other agents exist, including OpenAI's OAI-AdsBot for pages submitted as ads on ChatGPT. Review those separately if they affect your site; the table focuses on training, search and user-requested retrieval.

Robots.txt cannot protect a private library

Google's robots.txt guidance explains the limitation: the file guides crawling, but it does not enforce access or reliably keep a page out of search. Crawlers can ignore it. Google can also index a URL without its contents if it discovers the URL elsewhere.

For confidential files, Google recommends password protection rather than robots.txt. If you run a paid library, ask the technical owner to test whether an unauthorised request can retrieve the protected content. A hidden link, a paywall overlay or a path listed in robots.txt does not prove that the content is protected.

Indexing controls are different again. Google documents noindex for excluding a page from Search and nosnippet, max-snippet and data-nosnippet for limiting what Search displays from it. Those choices have Search consequences; they are not interchangeable with a provider-specific training preference.

The crawler also needs to see the instruction. Google advises checking that preview controls are visible to Googlebot and allowing time for recrawling and processing. Blocking a page from crawling can prevent Google from seeing its noindex instruction. Review crawl and indexing rules together rather than assuming one restrictive setting reinforces another.

Write down the change before deploying it

Use a separate worksheet for each host or content category that needs different treatment. It should give the implementer enough detail to make the change and the approver enough detail to question it.

Decision field: Business purpose

What to write down: The specific use to permit or restrict, such as training rather than all AI activity

Decision field: Content scope

What to write down: Public articles, member pages, downloadable files and relevant hosts or subdomains

Decision field: Provider and token

What to write down: Exact documented agent or product token, source URL and review date

Decision field: Expected effect

What to write down: What should change and what must remain available

Decision field: Delivery layers

What to write down: Robots.txt, page directives, authentication and any CDN or firewall rules involved

Decision field: Test evidence

What to write down: Representative URLs, expected responses and checks for retained search access

Decision field: Accountability

What to write down: Approver, implementer, rollback procedure and next documentation review

Suppose a publisher wants its public articles discoverable in Google and ChatGPT, but wants to restrict the documented training uses. Its web lead would evaluate Google-Extended and GPTBot preferences while preserving the intended Search controls. There is still a business decision to make: is losing the Gemini grounding uses covered by Google-Extended acceptable? "Allow discovery, restrict training" describes what this hypothetical publisher wants. It does not mean every provider offers that exact split.

A deployment checklist connecting policy, scoped configuration, access tests and rollback — Operational review sequence. Test the intended restriction and the discovery paths that must continue working.

Test access, then watch for unintended effects

Save the current configuration and make sure you can roll back. Before and after the change, check representative URLs, including a public article and a protected page. Look at the actual responses. A settings screen only tells you what you asked the system to do.

Where provider documentation supplies verification methods or published IP ranges, use those rather than assuming a user-agent name proves identity. Google documents crawler IP ranges and hostname patterns, and OpenAI publishes IP lists alongside its bot definitions. A request log can help you inspect access behavior, but it does not by itself establish all downstream uses of collected content.

Monitor search access and errors after deployment. Use the publisher measurement worksheet for the business effects, with the change date recorded. A decline after a rules change deserves investigation; it doesn't establish the cause without checking other changes.

If you announce the change, say which provider and purpose it covers, which content it applies to and when the preference changed. Do not claim that previously collected material has been deleted or removed from a trained model. These controls do not establish either outcome.