Ahrefs
AhrefsBot
Collects link and web data for Ahrefs and the Yep search engine.
SEO analysisSEO analysis
SemrushBot
Collects web and link data for Semrush analysis tools.
Operated by Semrush
See your agent traffic PurposeSEO analysis
HTTP identifierSemrushBot
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
A backlink crawler may revisit old linked URLs. That 404 can reflect a historical link rather than a broken current customer journey.
Linked URL failures
Crawl frequency
Origin load
Your next move
Check whether repeatedly requested missing URLs still have useful inbound links. Fix or redirect relevant destinations; do not redirect every unknown path.
The technical details
MarkdownHow to identify it
Look for the SemrushBot identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
For unexplained crawling, send the relevant URL and request evidence to bot@semrush.com. A matching name does not authenticate the source.
Access and robots.txt
Use a SemrushBot robots.txt group to control link-index crawling. Crawl-delay is supported up to 10 seconds; Semrush’s other tools have separate tokens.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: SemrushBot
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee SemrushBot in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
Majestic
MJ12bot
Builds web link data for Majestic using a distributed crawler.
SEO analysis