APOSTL
AI search

meta-webindexer

Crawls pages to improve Meta AI search results.

Operated by Meta
See your agent traffic
PurposeAI search
HTTP identifiermeta-webindexer
IdentityVerify beyond the name

THE SIGNAL

What this visit tells you

Search collection is distinct from a user opening your website.

Fetched pages
Failed retrievals
Repeat fetches

Your next move

Check public information pages for successful retrieval and measure referrals separately.

The technical details

Markdown
How to identify it

Look for the meta-webindexer identifier in a request’s User-Agent. This is a name match, not identity verification. Version strings may change.

How to check identity

A User-Agent match identifies a claimed client. Check the source IP and any operator-published verification method before granting access.

Access and robots.txt

Meta documents named robots.txt rules for this crawler; allow up to 24 hours for cached rules to refresh.

Optional full-site opt-out. Merge with existing rules only if intended. robots.txt does not secure private content.

User-agent: meta-webindexer
Disallow: /
Seeing 403 or 404 responses?

403 means access was denied; 404 means the resource was not found. Compare the public path, request time, edge security event and origin response to find the cause.

Separate intended restrictions and secret-file probes from pages that should work. A User-Agent name alone does not justify allowing a request.

SourcesReviewed 2026-09-14

Identity and purpose are based on these sources. Analytics interpretation and suggested checks are Apostl guidance.

APOSTL Pulse

See meta-webindexer in context

See their requests. Find the pages that matter.

See your agent traffic

KEEP EXPLORING

Different bots, different intent

Meta

meta-externalfetcher

Fetches external pages for user actions in Meta products.

User-requested fetch