Skip to Content
Platform Ontology

AI agent and crawler names

AI crawlers and browsing agents often announce themselves by name in the User-Agent header.

Gets silentlyWhen the page loads

Operators publish names such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Google-Extended. Many split them into training crawlers, search crawlers and fetchers that run when a person asks a question. Sites can allow or block them in robots.txt (RFC 9309). A name is only a claim unless the operator also publishes IP ranges or signs requests.

What it can and can't do

What the user does: Nothing. It is sent automatically

It can

  • Read the agent's name from the header
  • Apply robots.txt rules per product token
  • Separate training crawlers from user triggered fetchers
  • Check IP lists to confirm a claim

It cannot

  • Prove the name is true
  • Stop operators that ignore robots.txt
  • See the question the person asked
  • Cover agents that pose as a normal browser

How it differs between platforms: Many browsing agents run a normal Chrome and send a normal user agent, so names catch only the declared ones.

Browser support

Version notes
  • Chrome: Not browser behavior; sent by agent and crawler operators

How a site asks for it

APIUser-Agent request header; robots.txt product tokens (RFC 9309)
How access is grantednone
What the user seesNone
Needs a secure context (HTTPS)No
Needs a user gestureNo
How long it lastsNot listed
How to take it backNot listed
Inside an iframeNot listed

Typical flow

  1. Agent sends a User-Agent with its product name
  2. Site reads the header and robots.txt rules
  3. Site allows, limits or blocks by name
  4. Site may verify IP ranges or signatures

Data you get

Shape: header
How often: per request

FieldTypeUnitRangePrecision
User-AgentstringNoneproduct token and URL of the operatorNot listed
robots.txt ruleruleNoneallow or disallow for a product tokenNot listed

Technical limits

  • Claim only; verify by IP or signature
  • Agents using a normal browser look like people
  • Lists change often

The story

What it is

AI services often introduce themselves by name. Sites decide whether to let each one in.

For you

You never see it. When you ask an AI to read a page, a fetcher may visit with its own name, which the site can allow or block.

For makers

Decide per name and per purpose: training, search, user requested. Confirm claims with published IP ranges or signatures. Keep your list current because names and tokens change.

Where you see it

TypeExampleApps
newsA publisher blocks training crawlers but allows search crawlers.None named
shoppingA retailer allows user triggered fetchers so shoppers can compare prices through an assistant.None named
securityA firewall rule rate limits any client that claims an AI crawler name without matching IP ranges.Cloudflare
analyticsTraffic reports separate AI fetchers from human visits.None named
developerA docs site allows AI fetchers so developers get answers with citations.None named
educationA university site blocks training crawlers on course notes.None named

Risks

  • Names can be faked
  • Blocking may hide a site from AI search

What the platform says

Good uses and abuse

Legitimate uses

  • Content policy
  • Traffic reports

Known abuse

  • Blocking legitimate assistants

Scores

ScoreValueMeaning
Sensitivity1 of 5How revealing the data is
Utility4 of 5How useful it is to honest apps
Friction0 of 5How much the user has to do
Fingerprint bits0Rough identifying entropy, when known

History

  • 2022-09, rfc: Robots Exclusion Protocol published as RFC 9309
  • 2026-03, all: Google reported to list Google-Agent as a user triggered fetcher

Sources

How well checked: Inferred from related behavior.

On this page