AI agent and crawler names
AI crawlers and browsing agents often announce themselves by name in the User-Agent header.
Operators publish names such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Google-Extended. Many split them into training crawlers, search crawlers and fetchers that run when a person asks a question. Sites can allow or block them in robots.txt (RFC 9309). A name is only a claim unless the operator also publishes IP ranges or signs requests.
What it can and can't do
What the user does: Nothing. It is sent automatically
It can
- Read the agent's name from the header
- Apply robots.txt rules per product token
- Separate training crawlers from user triggered fetchers
- Check IP lists to confirm a claim
It cannot
- Prove the name is true
- Stop operators that ignore robots.txt
- See the question the person asked
- Cover agents that pose as a normal browser
How it differs between platforms: Many browsing agents run a normal Chrome and send a normal user agent, so names catch only the declared ones.
Browser support
Version notes
- Chrome: Not browser behavior; sent by agent and crawler operators
How a site asks for it
| API | User-Agent request header; robots.txt product tokens (RFC 9309) |
| How access is granted | none |
| What the user sees | None |
| Needs a secure context (HTTPS) | No |
| Needs a user gesture | No |
| How long it lasts | Not listed |
| How to take it back | Not listed |
| Inside an iframe | Not listed |
Typical flow
- Agent sends a User-Agent with its product name
- Site reads the header and robots.txt rules
- Site allows, limits or blocks by name
- Site may verify IP ranges or signatures
Data you get
Shape: header
How often: per request
| Field | Type | Unit | Range | Precision |
|---|---|---|---|---|
User-Agent | string | None | product token and URL of the operator | Not listed |
robots.txt rule | rule | None | allow or disallow for a product token | Not listed |
Technical limits
- Claim only; verify by IP or signature
- Agents using a normal browser look like people
- Lists change often
The story
What it is
AI services often introduce themselves by name. Sites decide whether to let each one in.
For you
You never see it. When you ask an AI to read a page, a fetcher may visit with its own name, which the site can allow or block.
For makers
Decide per name and per purpose: training, search, user requested. Confirm claims with published IP ranges or signatures. Keep your list current because names and tokens change.
Where you see it
| Type | Example | Apps |
|---|---|---|
| news | A publisher blocks training crawlers but allows search crawlers. | None named |
| shopping | A retailer allows user triggered fetchers so shoppers can compare prices through an assistant. | None named |
| security | A firewall rule rate limits any client that claims an AI crawler name without matching IP ranges. | Cloudflare |
| analytics | Traffic reports separate AI fetchers from human visits. | None named |
| developer | A docs site allows AI fetchers so developers get answers with citations. | None named |
| education | A university site blocks training crawlers on course notes. | None named |
Risks
- Names can be faked
- Blocking may hide a site from AI search
What the platform says
- OpenAI: Overview of OpenAI crawlers. Docs list the user agents and what each one does.
Good uses and abuse
Legitimate uses
- Content policy
- Traffic reports
Known abuse
- Blocking legitimate assistants
Scores
| Score | Value | Meaning |
|---|---|---|
| Sensitivity | 1 of 5 | How revealing the data is |
| Utility | 4 of 5 | How useful it is to honest apps |
| Friction | 0 of 5 | How much the user has to do |
| Fingerprint bits | 0 | Rough identifying entropy, when known |
History
- 2022-09, rfc: Robots Exclusion Protocol published as RFC 9309
- 2026-03, all: Google reported to list Google-Agent as a user triggered fetcher
Related capabilities
Related
- Signed agent requests (Web Bot Auth). Can prove which agent operator made a request
- Bot detection and CAPTCHA signals. Can combine many browser and network clues to guess if you are a bot
- Client IP address. Can see the public internet address your connection uses
- ASN, ISP and hosting type. Can tell which internet company you use and if you look like a home user
Sources
How well checked: Inferred from related behavior.