OpenAI
GPTBot
Collects web content that may be used to train OpenAI foundation models.
AI trainingFIELD GUIDE / BOT TRAFFIC
Every bot has
a different purpose
Know who’s visiting. Understand why. Decide what happens next.
The directory
115 profiles
Anthropic
ClaudeBot
Collects web content potentially used for Claude model training.
AI trainingPerplexity
PerplexityBot
Indexes pages for Perplexity search, not foundation-model training.
AI searchByteDance
Bytespider
ByteDance crawler listed by Cloudflare in its AI crawler reference.
Research & dataByteDance
TikTokSpider
TikTok-named crawler tracked by Cloudflare in its AI traffic research.
Research & dataMoonshot AI
KimiBot
Collects content that may train Kimi foundation models.
AI trainingMoonshot AI
Kimi-SearchBot
Builds the index for Kimi search.
AI searchMoonshot AI
Kimi-User
Retrieves pages in response to Kimi user actions.
User-requested fetchBaidu
Baiduspider
Crawls websites for Baidu Search.
Web searchSogou
Sogou spider
Crawls web pages for Sogou Search using its web-spider identity.
Web search
Huawei
PetalBot
Indexes websites for Petal Search and Huawei content recommendations.
Web search
360 Search
360Spider
Indexes web pages for 360 Search.
Web search
Naver
Yeti
Crawls web documents for Naver Search.
Web search
Microsoft
bingbot
Crawls pages for Bing Search.
Web search360 Search
360Spider-Image
Retrieves images for 360 Search.
Web search360 Search
360Spider-Video
Retrieves video for 360 Search.
Web search
Ahrefs
AhrefsBot
Collects link and web data for Ahrefs and the Yep search engine.
SEO analysisAhrefs
AhrefsSiteAudit
Runs site audits configured in Ahrefs.
SEO analysis
Ai2
AI2Bot
Collects web content for training open language models.
AI training
Amazon
Amazonbot
Collects web content for Amazon products and services, potentially including AI training.
AI trainingAmazon
Amzn-SearchBot
Crawls for Amazon search experiences, including Alexa, rather than generative-model training.
AI searchAmazon
Amzn-User
Retrieves current web information for user actions such as Alexa queries.
User-requested fetchAnthropic
Claude-SearchBot
Indexes web content to improve Claude search responses.
AI searchAnthropic
Claude-User
Retrieves web content in response to Claude users.
User-requested fetch
Apple
Applebot
Indexes content for Apple search experiences including Siri, Spotlight and Safari.
Web searchApple
Applebot-Extended
Controls whether Applebot-collected content can train Apple generative models.
Content-use policyPolicy control · no separate crawler
Atlassian
atlassian-bot
Indexes connected websites for Atlassian Rovo.
AI search
Babbar
Barkrowler
Builds the web-link graph used by Babbar SEO tools.
SEO analysis
Common Crawl
CCBot
Collects web pages for the Common Crawl open web dataset.
Open web dataDataForSEO
DataForSeoBot
Collects backlink data for DataForSEO.
SEO analysis
Diffbot
Diffbot
Builds the search index behind Diffbot’s knowledge services.
AI searchDiffbot
Diffbot-User
Retrieves URLs on behalf of Diffbot users.
User-requested fetch
DuckDuckGo
DuckAssistBot
Retrieves pages for DuckDuckGo’s AI-assisted answers.
AI searchDuckDuckGo
DuckDuckBot
Crawls websites for DuckDuckGo Search.
Web search
Exa
ExaSearchBot
Indexes public pages for Exa search and retrieval.
AI search
AdsBot-Google
Checks web page quality for Google Ads.
Ad reviewAdsBot-Google-Mobile
Checks mobile landing pages for Google Ads.
Ad reviewAPIs-Google
Delivers Google API push notifications.
User-requested fetchFeedFetcher-Google
Retrieves RSS and Atom feeds for Google services.
User-requested fetchGoogle-Agent
Navigates websites for user-requested actions on Google infrastructure.
User-requested fetchGoogle-CloudVertexBot
Crawls sites for owner-requested Vertex AI agents.
AI searchGoogle-CWS
Fetches URLs supplied in Chrome Web Store listings.
User-requested fetchGoogle-Extended
Controls content use for Gemini training and grounding; it does not control Google Search inclusion.
Content-use policyPolicy control · no separate crawlerGoogle-GeminiNotebook
Retrieves sources supplied to Gemini Notebook.
User-requested fetchGoogle-InspectionTool
Fetches pages for Google’s Search testing tools.
SEO analysisGoogle-Pinpoint
Retrieves document sources selected by Pinpoint users.
User-requested fetchGoogle-Read-Aloud
Retrieves pages for user-requested text-to-speech.
User-requested fetchGoogle-Site-Verification
Fetches Search Console ownership-verification tokens.
User-requested fetchGooglebot
Crawls pages for Google Search using desktop and smartphone variants.
Web searchGooglebot-Image
Fetches images for Google’s visual search features.
Web searchGooglebot-Video
Fetches video for Google’s video-related search features.
Web searchGoogleMessages
Builds previews for links sent through Google Messages.
Link previewsGoogleOther
Retrieves public content for Google’s internal projects.
Research & dataGoogleOther-Image
Retrieves public image URLs for GoogleOther.
Research & dataGoogleOther-Video
Retrieves public video URLs for GoogleOther.
Research & dataGoogleProducer
Retrieves feeds submitted to Google Publisher Center.
User-requested fetchMediapartners-Google
Reads participating pages to support AdSense ad selection.
Ad reviewStorebot-Google
Checks pages for Google Shopping.
Commerce
Klaviyo
KlaviyoAIBot
Indexes connected website content for Klaviyo’s customer agent.
AI search
Lyrenth
AIWebIndex
Builds Lyrenth’s index of public pages for API retrieval.
AI search
Majestic
MJ12bot
Builds web link data for Majestic using a distributed crawler.
SEO analysis
Meta
facebookexternalhit
Retrieves previews of links shared in Meta apps.
Link previewsMeta
meta-externalagent
Collects content for uses including Meta AI training.
AI trainingMeta
meta-externalfetcher
Fetches external pages for user actions in Meta products.
User-requested fetchMeta
meta-webindexer
Crawls pages to improve Meta AI search results.
AI searchMicrosoft
adidxbot
Checks landing pages for Microsoft advertising.
Ad reviewMicrosoft
MicrosoftPreview
Generates page previews for Microsoft products.
Link previews
Mistral AI
MistralAI-Index
Builds Mistral’s search index.
AI searchMistral AI
MistralAI-Training
Builds datasets for training Mistral generative models.
AI trainingMistral AI
MistralAI-User
Retrieves pages for user questions in Vibe.
User-requested fetch
Mojeek
MojeekBot
Builds the independent Mojeek search index.
Web searchMoz
DotBot
Collects links for Moz’s link index.
SEO analysisMoz
rogerbot
Audits websites in Moz Pro campaigns.
SEO analysis
Naget
NagetBot
Builds Naget’s search index from public HTML.
Web searchNaver
Ads-Naver
Collects page information for Naver advertising.
Ad reviewNaver
Blueno
Fetches link-preview information for Naver services.
Link previewsOpenAI
ChatGPT-User
Fetches pages for user actions in ChatGPT and custom GPTs.
User-requested fetchOpenAI
OAI-AdsBot
Reviews pages submitted as ChatGPT ads for safety and relevance; this collection is not for foundation-model training.
Ad reviewOpenAI
OAI-SearchBot
Discovers content for ChatGPT search results.
AI search
Parallel
ShapBot
Discovers and indexes websites for Parallel’s web APIs.
AI searchPerplexity
Perplexity-User
Fetches pages on behalf of Perplexity users.
User-requested fetch
Pinterestbot
Indexes content and refreshes the details behind Pins.
Web search
Qwant
Qwantbot
Builds Qwant’s web search index.
Web search
SB Intuitions
SBIntuitions-SearchBot
Retrieves relevant sites for Sarashina user questions.
User-requested fetchSB Intuitions
SBIntuitionsBot
Collects content for AI development and analysis.
Research & data
Semrush
SemrushBot
Collects web and link data for Semrush analysis tools.
SEO analysisSemrush
SemrushBot-BA
Crawls for Semrush Backlink Audit.
SEO analysisSemrush
SemrushBot-FT
Retrieves content for Semrush Plagiarism Checker.
SEO analysisSemrush
SemrushBot-OCOB
Crawls for Semrush Content Toolkit.
SEO analysisSemrush
SemrushBot-SI
Crawls for Semrush On Page SEO Checker.
SEO analysisSemrush
SemrushBot-SWA
Checks URL accessibility for SEO Writing Assistant.
SEO analysisSemrush
SiteAuditBot
Checks technical SEO in Semrush Site Audit.
SEO analysisSemrush
SplitSignalBot
Crawls pages for Semrush’s SplitSignal SEO tests.
SEO analysisSEOkicks
SEOkicks
Collects external links for the SEOkicks backlink database.
SEO analysis
Serpstat
serpstatbot
Collects backlink data for Serpstat.
SEO analysis
Slack
Slackbot-LinkExpanding
Retrieves shared links to build previews in Slack.
Link previews
UptimeRobot
UptimeRobot
Checks endpoints configured for availability monitoring.
Monitoring
Yandex
YandexAccessibilityBot
Checks page availability.
MonitoringYandex
YandexAdNet
Crawls for Yandex Advertising Network.
Ad reviewYandex
YandexBlogs
Indexes blog comments.
Web searchYandex
YandexBot
Indexes pages for Yandex Search.
Web searchYandex
YandexCalendar
Retrieves user-subscribed calendar files.
User-requested fetchYandex
YandexCheckBot
Checks page availability across regions.
MonitoringYandex
YandexDirect
Retrieves advertising-partner page content.
Ad reviewYandex
YandexFavicons
Retrieves icons for search results.
Link previewsYandex
YandexFeed
Retrieves owner-submitted structured feeds.
User-requested fetchYandex
YandexImageResizer
Retrieves images for mobile services.
Link previewsYandex
YandexImages
Indexes pictures for Yandex Images.
Web searchYandex
YandexMedia
Indexes multimedia content.
Web searchYandex
YandexMobileBot
Checks mobile-friendly page layouts.
SEO analysisYandex
YandexPagechecker
Fetches pages for markup validation.
SEO analysisYandex
YandexScreenshotBot
Captures page screenshots.
SEO analysisYandex
YandexSitelinks
Checks search sitelink availability.
MonitoringYandex
YandexVideo
Indexes video for Yandex search.
Web searchYandex
YandexWebmaster
Fetches pages for Yandex Webmaster.
SEO analysisNo matching profiles. Try another name or reset the filters.
APOSTL PulseMeet the agents visiting your website
See their requests. Find the pages that matter.
See your agent traffic