Anthropic
ClaudeBot
Collects web content potentially used for Claude model training.
AI trainingAI search
Claude-SearchBot
Indexes web content to improve Claude search responses.
Operated by Anthropic
See your agent traffic PurposeAI search
HTTP identifierClaude-SearchBot
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
Search crawl coverage is a discoverability signal. It cannot establish whether Claude recommended your product.
Public-page coverage
Retrieval failures
Referrals separately
Your next move
Review failed retrievals on canonical public pages. Compare missing pages against navigation and sitemap coverage.
The technical details
MarkdownHow to identify it
Look for the Claude-SearchBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
Compare the request IP with Anthropic’s current crawler list linked at https://claude.com/robots. The User-Agent alone does not authenticate the request.
Access and robots.txt
Use the Claude-SearchBot robots.txt group to control this search-indexing path. Blocking it can limit discovery through Claude search; configure each host separately.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: Claude-SearchBot
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee Claude-SearchBot in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
Anthropic
Claude-User
Retrieves web content in response to Claude users.
User-requested fetch