Diffbot
Diffbot
Builds the search index behind Diffbot’s knowledge services.
AI searchUser-requested fetch
Diffbot-User
Retrieves URLs on behalf of Diffbot users.
Operated by Diffbot
See your agent traffic PurposeUser-requested fetch
HTTP identifierDiffbot-User
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
A fetch can follow a person submitting a specific URL.
Requested URLs
Response codes
Redirect chains
Your next move
Match failed retrievals to the submitted URL and check whether authentication is required.
The technical details
MarkdownHow to identify it
Look for the Diffbot-User identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
A User-Agent match identifies a claimed client. Check the source IP and any operator-published verification method before granting access.
Access and robots.txt
Diffbot documents robots.txt and Crawl-delay compliance by default. A configured override can apply when a customer has an agreement with the site owner.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: Diffbot-User
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee Diffbot-User in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
OpenAI
ChatGPT-User
Fetches pages for user actions in ChatGPT and custom GPTs.
User-requested fetch
Anthropic
Claude-User
Retrieves web content in response to Claude users.
User-requested fetch