Semrush
SemrushBot
Collects web and link data for Semrush analysis tools.
SEO analysisPurposeSEO analysis
HTTP identifierDataForSeoBot
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
This is data collection for SEO analysis. A visit is not evidence that an AI assistant recommended your business.
Linked URL failures
Crawl frequency
Origin load
Your next move
Separate link crawling from user-facing agent tasks. Investigate response errors on valuable linked pages before changing site-wide access.
The technical details
MarkdownHow to identify it
Look for the DataForSeoBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
Compare the source IP with DataForSEO’s published IPv4/IPv6 subnets. For reverse DNS under crawling-gateway-*.dataforseo.com, confirm that a forward lookup returns the original IP.
Access and robots.txt
Supports robots.txt and Crawl-delay.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: DataForSeoBot
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee DataForSeoBot in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
Ahrefs
AhrefsBot
Collects link and web data for Ahrefs and the Yep search engine.
SEO analysis