# AIWebIndex

Operator: Lyrenth
Purpose: AI search
Reviewed: 2026-09-14
HTTP identifier: AIWebIndex, AIWebIndex-Agent
robots.txt token: AIWebIndex

## What is AIWebIndex?

Builds Lyrenth’s index of public pages for API retrieval.

## How to identify it

Look for the AIWebIndex, AIWebIndex-Agent identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

## How to check identity

Validate HTTP signatures using api.lyrenth.com/.well-known/http-message-signatures-directory, or use Lyrenth’s published IP ranges and forward-confirmed lyrenth.com DNS.

## Access and robots.txt

Both HTTP identifiers follow the AIWebIndex robots.txt rules. Lyrenth applies these controls to autonomous and on-demand retrieval.

Optional full-site opt-out; merge with existing rules only if intended. Not a security control.

```text
User-agent: AIWebIndex
Disallow: /
```

## What it means in your analytics

AIWebIndex handles autonomous indexing; AIWebIndex-Agent handles on-demand retrieval. Separate the two identifiers when comparing crawl schedules with requested refreshes.

## Useful measures

Fetched pages; Failed retrievals; Repeat fetches.

## What to check next

Review indexed public paths and confirm the operator’s request-verification signals.

## Sources

- [lyrenth.com](https://lyrenth.com/crawler-policy)

Operator facts are source-backed; interpretation and actions are Apostl guidance.

## Related profiles

- [OAI-SearchBot](https://apostl.dev/bots/oai-searchbot)
- [Claude-SearchBot](https://apostl.dev/bots/claude-searchbot)
- [PerplexityBot](https://apostl.dev/bots/perplexitybot)

[Start measuring agent traffic](https://apostl.dev/start/agent-analytics)