AI training

ClaudeBot

Collects web content potentially used for Claude model training.

Operated by Anthropic

See your agent traffic

PurposeAI training

HTTP identifierClaudeBot

IdentityVerify beyond the name

THE SIGNAL

What this visit tells you

Treat repeated crawls as content access, not live Claude users. A training visit does not establish inclusion in a model.

Crawl volume

Content paths

Response status

Your next move

Keep training access decisions separate from the ability of a person using Claude to read your public product pages.

The technical details

Markdown

How to identify it

Look for the ClaudeBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

How to check identity

Compare the request IP with Anthropic’s current crawler list linked at https://claude.com/robots. The User-Agent alone does not authenticate the request.

Access and robots.txt

Anthropic documents robots.txt controls for ClaudeBot. Apply rules on each host you manage; this setting is independent of its other crawler identities.

Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.

User-agent: ClaudeBot
Disallow: /

Seeing 403 or 404 responses?

403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.

Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.

SourcesReviewed 2026-09-14

Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.

APOSTL Pulse

See ClaudeBot in context

See their requests. Find the pages that matter.

See your agent traffic

What this visit tells you

Your next move

The technical details

See ClaudeBot in context

Different bots, different intent

Claude-SearchBot

Claude-User