OpenAI
OAI-SearchBot
Discovers content for ChatGPT search results.
AI searchAI search
AIWebIndex
Builds Lyrenth’s index of public pages for API retrieval.
Operated by Lyrenth
See your agent traffic PurposeAI search
HTTP identifierAIWebIndex, AIWebIndex-Agent
IdentityVerify beyond the name
THE SIGNAL
What this visit tells you
AIWebIndex handles autonomous indexing; AIWebIndex-Agent handles on-demand retrieval. Separate the two identifiers when comparing crawl schedules with requested refreshes.
Fetched pages
Failed retrievals
Repeat fetches
Your next move
Review indexed public paths and confirm the operator’s request-verification signals.
The technical details
MarkdownHow to identify it
Look for the AIWebIndex, AIWebIndex-Agent identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.
How to check identity
Validate HTTP signatures using api.lyrenth.com/.well-known/http-message-signatures-directory, or use Lyrenth’s published IP ranges and forward-confirmed lyrenth.com DNS.
Access and robots.txt
Both HTTP identifiers follow the AIWebIndex robots.txt rules. Lyrenth applies these controls to autonomous and on-demand retrieval.
Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.
User-agent: AIWebIndex
Disallow: /Seeing 403 or 404 responses?
403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.
Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.
SourcesReviewed 2026-09-14
Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.
APOSTL PulseSee AIWebIndex in context
See their requests. Find the pages that matter.
See your agent trafficKEEP EXPLORING
Different bots, different intent
Anthropic
Claude-SearchBot
Indexes web content to improve Claude search responses.
AI search
Perplexity
PerplexityBot
Indexes pages for Perplexity search, not foundation-model training.
AI search