# meta-webindexer

Operator: Meta
Purpose: AI search
Reviewed: 2026-09-14
HTTP identifier: meta-webindexer
robots.txt token: meta-webindexer

## What is meta-webindexer?

Crawls pages to improve Meta AI search results.

## How to identify it

Look for the meta-webindexer identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

## How to check identity

A User-Agent match identifies a claimed client. Check the source IP and any operator-published verification method before granting access.

## Access and robots.txt

Meta documents named robots.txt rules for this crawler; allow up to 24 hours for cached rules to refresh.

Optional full-site opt-out; merge with existing rules only if intended. Not a security control.

```text
User-agent: meta-webindexer
Disallow: /
```

## What it means in your analytics

Search collection is distinct from a user opening your website.

## Useful measures

Fetched pages; Failed retrievals; Repeat fetches.

## What to check next

Check public information pages for successful retrieval and measure referrals separately.

## Sources

- [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)

Operator facts are source-backed; interpretation and actions are Apostl guidance.

## Related profiles

- [facebookexternalhit](https://apostl.dev/bots/facebookexternalhit)
- [meta-externalagent](https://apostl.dev/bots/meta-externalagent)
- [meta-externalfetcher](https://apostl.dev/bots/meta-externalfetcher)

[Start measuring agent traffic](https://apostl.dev/start/agent-analytics)