# Diffbot-User

Operator: Diffbot
Purpose: User-requested fetch
Reviewed: 2026-09-14
HTTP identifier: Diffbot-User
robots.txt token: Diffbot-User

## What is Diffbot-User?

Retrieves URLs on behalf of Diffbot users.

## How to identify it

Look for the Diffbot-User identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

## How to check identity

A User-Agent match identifies a claimed client. Check the source IP and any operator-published verification method before granting access.

## Access and robots.txt

Diffbot documents robots.txt and Crawl-delay compliance by default. A configured override can apply when a customer has an agreement with the site owner.

Optional full-site opt-out; merge with existing rules only if intended. Not a security control.

```text
User-agent: Diffbot-User
Disallow: /
```

## What it means in your analytics

A fetch can follow a person submitting a specific URL.

## Useful measures

Requested URLs; Response codes; Redirect chains.

## What to check next

Match failed retrievals to the submitted URL and check whether authentication is required.

## Sources

- [docs.diffbot.com](https://docs.diffbot.com/docs/does-crawl-respect-robotstxt)

Operator facts are source-backed; interpretation and actions are Apostl guidance.

## Related profiles

- [Diffbot](https://apostl.dev/bots/diffbot)
- [ChatGPT-User](https://apostl.dev/bots/chatgpt-user)
- [Claude-User](https://apostl.dev/bots/claude-user)

[Start measuring agent traffic](https://apostl.dev/start/agent-analytics)