# CCBot

Operator: Common Crawl
Purpose: Open web data
Reviewed: 2026-09-14
HTTP identifier: CCBot
robots.txt token: CCBot

## What is CCBot?

Collects web pages for the Common Crawl open web dataset.

## How to identify it

Look for the CCBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

## How to check identity

Match the IP against Common Crawl’s published ccbot.json. IPv4 addresses can also be checked with forward-confirmed reverse DNS under crawl.commoncrawl.org; its IPv6 crawlers do not support reverse DNS.

## Access and robots.txt

The operator documents robots.txt controls for this identifier.

Optional full-site opt-out; merge with existing rules only if intended. Not a security control.

```text
User-agent: CCBot
Disallow: /
```

## What it means in your analytics

Open dataset collection is different from a visit by a named AI product. A later downstream use cannot be inferred from this request.

## Useful measures

Crawl volume; Origin load; Response status.

## What to check next

Set a deliberate dataset collection policy. Track crawl load and successful retrieval separately from commercial agent activity.

## Sources

- [commoncrawl.org](https://commoncrawl.org/ccbot)

Operator facts are source-backed; interpretation and actions are Apostl guidance.

## Related profiles

- [GPTBot](https://apostl.dev/bots/gptbot)
- [ClaudeBot](https://apostl.dev/bots/claudebot)

[Start measuring agent traffic](https://apostl.dev/start/agent-analytics)