OpenAI
GPTBot
Collects web content that may be used to train OpenAI foundation models.
AI trainingOpen web data
CCBot
Collects web pages for the Common Crawl open web dataset.
Operated by Common Crawl
See your agent traffic PurposeOpen web data
HTTP identifierCCBot
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
Open dataset collection is different from a visit by a named AI product. A later downstream use cannot be inferred from this request.
Crawl volume
Origin load
Response status
Your next move
Set a deliberate dataset collection policy. Track crawl load and successful retrieval separately from commercial agent activity.
The technical details
MarkdownHow to identify it
Look for the CCBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
Match the IP against Common Crawl’s published ccbot.json. IPv4 addresses can also be checked with forward-confirmed reverse DNS under crawl.commoncrawl.org; its IPv6 crawlers do not support reverse DNS.
Access and robots.txt
The operator documents robots.txt controls for this identifier.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: CCBot
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee CCBot in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
Anthropic
ClaudeBot
Collects web content potentially used for Claude model training.
AI training