--- title: AI Crawler Analytics vs Agent Journey Analytics | Apostl description: Compare request-level AI crawler analytics with bounded agent journey estimates, including what each signal can support and what it cannot prove. canonical: https://apostl.dev/blog/ai-crawler-analytics-vs-agent-journey-analytics eyebrow: Crawler analytics vs journeys schema_type: BlogPosting date_published: 2026-08-28 date_modified: 2026-08-28 primary_cta_label: Inspect agent journeys with Pulse primary_cta_href: /pulse final_cta_title: Keep the raw request and add the bounded sequence final_cta_body: Install Pulse on the server, verify one real public request, and inspect which public resources machine clients appear to use together. final_cta_label: Explore Apostl Pulse final_cta_href: /pulse image: /assets/blog/ai-crawler-vs-agent-journey-analytics.webp image_alt: A crawler grid beside a multi-step agent route across public pages visual_kicker: Two lenses on one request stream visual_title: Counting crawls and estimating journeys answer different questions visual_left_title: Request-level crawler lens visual_left_items: Bot and operator evidence | Path, status, and bytes | Access and delivery decisions visual_axis: Same server facts visual_right_title: Correlated journey lens visual_right_items: Bounded related requests | Ordered public surfaces | Content-flow hypotheses visual_caption: Correlation adds sequence, not identity. Keep raw requests available beside every estimated journey. --- # AI crawler analytics vs agent journey analytics: what each can actually tell you AI crawler analytics and agent journey analytics often appear in the same dashboard pitch. They shouldn't be treated as interchangeable. A crawler report records which automated client requested a resource, when it arrived, how the server answered, and how much data moved. Journey reporting correlates public requests into estimated periods of machine activity. Request facts support access policy and infrastructure; correlated sequences help teams study agent-facing paths. Neither view reveals the user's prompt, proves a citation, identifies a person or company, or assigns a conversion. Choose according to the decision. ## The unit of measurement changes the answer Crawler analytics asks: **what automated traffic reached the site?** Its records include timestamp, path, status, bytes, User-Agent, detected crawler, and operator. Aggregation shows volume, popular paths, errors, and bandwidth. Journey analytics asks: **which public resources appear to have been used together?** It needs a correlation key and time boundary. Three hits to `/llms.txt`, `/docs/auth.md`, and `/openapi.json` may become one estimated research session. Ten requests can mean ten crawls, one agent collecting ten resources, or a mixture. Counting cannot settle that ambiguity. Correlation offers a bounded interpretation, but remains a model. Keep “request” and “estimated journey” separate in the data model and dashboard. Calling both “visits” invites claims the evidence cannot support. ## What request-level crawler reporting measures well For security, infrastructure, content operations, and robots policy, request-level reporting is the useful starting point. [Cloudflare AI Crawl Control][cloudflare-analysis] exposes total and allowed requests, data transfer, status-code distribution, content formats, popular paths, and grouping by crawler, category, operator, or host. Its access controls let a site owner allow or block individual crawlers. Those dimensions support concrete decisions: - A rise in 403 responses can reveal that a policy is blocking a crawler you meant to allow. - Repeated requests to large media files can explain bandwidth cost without implying product interest. - Heavy access to `/docs/*` and little access to `/api/*` can guide caching and content delivery work. - Requested and served content types can expose failed content negotiation for Markdown or JSON clients. Detection quality still depends on the plan and the signal. Cloudflare states that its free-plan detection uses User-Agent strings and therefore recognizes well-known, self-identifying crawlers. Eligible Bot Management customers can use detection IDs for a more thorough path. The [getting-started documentation][cloudflare-start] limits free-plan metrics to the past 24 hours. The honest output is “requests attributed to this crawler or category under this detection method.” It is not “people from this AI product researched our company.” ## A named bot still does not tell you the purpose of the request Official crawler documentation makes the ambiguity visible. [OpenAI documents][openai-bots] `OAI-SearchBot` for search, `GPTBot` for content that may be used in model training, and `ChatGPT-User` for certain user-triggered actions. OpenAI explicitly says `ChatGPT-User` is not an automatic web crawler. It also gives independent robots.txt controls to `OAI-SearchBot` and `GPTBot`. [Anthropic makes a similar distinction][anthropic-bots]: `ClaudeBot` can collect material for model development, `Claude-SearchBot` supports search quality, and `Claude-User` retrieves content at a user's direction. An operator total that combines these roles throws away information. A training crawl, a search-indexing request, and a user-directed fetch can have different implications for content policy and product demand even when the operator name is the same. Preserve at least four fields when the source makes them available: operator, agent string or detection ID, declared purpose, and confidence. Unknown should remain a valid value. A dashboard that turns every unidentified client into “other AI” looks complete but gives teams no reliable basis for action. ## What an agent journey model adds To recover a sequence from server-side evidence, [Pulse SDK][pulse-github] groups requests with the same project, trusted IP, and full User-Agent until a **30-minute inactivity window** expires. Its current public rule treats an exact `/llms.txt` request as immediate evidence. A generic non-browser client can also qualify after it reaches **two distinct machine-readable surfaces within ten minutes**, such as `openapi.json` followed by a Markdown page. This model adds three useful objects that a crawler chart usually lacks: - A bounded start and end for related activity. - An ordered list of public surfaces and paths. - A distinction between a single probe and a multi-resource sequence. Those objects show whether clients stop at the index or continue into auth and endpoint reference. They also expose broken sequences: an agent fetches `/llms.txt`, gets a 200, then receives a 404 on the first linked Markdown page. That is more actionable than “two requests from a bot.” Correlation does not upgrade evidence into identity. Shared egress IPs, proxies, changing User-Agents, retries, and parallel tasks can merge unrelated requests or split one task into several journeys. The model should publish its keys, windows, exclusions, and confidence rules so analysts can judge the estimate. ## Two lenses on the same server log One request stream can feed both analyses: ```text PUBLIC HTTP REQUESTS | +-- CRAWLER LENS --> classify each request --> aggregate by bot/path/status --> access and delivery decisions | +-- JOURNEY LENS --> correlate bounded requests --> order public surfaces --> content-flow hypotheses ``` The shared input matters: classify request facts once, then analyse them at both levels. | Question | Crawler analytics | Journey analytics | |---|---|---| | Which bots request the most bytes? | Strong | Secondary | | Which crawler gets blocked or fails? | Strong | Useful only within a sequence | | Which public pages are requested together? | Weak | Stronger, as an estimate | | Was access automatic or user-directed? | Sometimes, when the agent declares a role | Sometimes, from role plus sequence; still uncertain | | Who was the person or company? | Not proven | Not proven | | Did the model answer correctly or convert? | Not measured | Not measured | The final rows show that a sequence can improve operational interpretation without identifying the client. ## One sequence, three defensible readings Consider this public traffic over six minutes: ```text 09:12 GET /llms.txt 200 text/markdown 09:13 GET /docs/auth.md 200 text/markdown 09:15 GET /openapi.json 200 application/json 09:18 GET /api/mcp 401 application/json ``` A crawler report can state that four requests arrived, name the detected client when possible, and show the paths, formats, and status codes. It can flag the 401 and report the bytes served. Under the journey rule, matching correlation keys and timing make the four requests one estimated sequence. The result describes an index-to-auth-to-schema path and surfaces failed API access at the end. A product analyst may hypothesize that a machine client was evaluating integration requirements. That hypothesis is reasonable enough to prioritize a documentation check. It is not evidence of a specific prompt, company, purchasing stage, or lost conversion. The page should label the interpretation as a hypothesis and keep the raw request sequence available for review. ## Choose by the decision, not the dashboard name Use crawler analytics when the owner of the decision sits in infrastructure or content access: allow/block rules, robots policy, status failures, cache behaviour, transfer volume, and crawler-specific delivery. Add journey analytics when a team owns public agent experience: machine-readable navigation, docs architecture, API discovery, agent-facing examples, and the order in which surfaces are used. The sequence is especially useful when the next action is editorial or technical and can be verified after a change. Use both when a policy choice can affect a journey. Blocking a search crawler may reduce automated indexing while leaving some user-triggered retrieval possible, depending on the operator's documented agents and robots behaviour. A combined view lets the owner inspect access outcomes and the downstream sequence without collapsing the two into one metric. Do not add journey correlation merely to inflate a traffic number. It earns its place when someone will change a page, route, policy, or test based on the ordered evidence. ## Instrument request facts before tuning correlation Start with complete, server-side request facts for public surfaces. Record the canonical path, method, response status, response content type, timestamp, duration, a trusted proxy-derived client address when the deployment supports it, and the full User-Agent. Keep the trust boundary explicit: a forwarded IP header is useful only when the origin accepts traffic through the proxy that sets it. Classify the surface separately from the client. Useful surface types include `llms`, Markdown docs, OpenAPI, public API, MCP, pricing, and other public pages. Exclude assets, authenticated account routes, mutations, and irrelevant server failures according to a documented policy. Then add correlation with versioned rules. Store the correlation rule used for each journey so a later model change does not silently rewrite history. Monitor split and merge failure modes with synthetic sequences. Keep request totals available beside journey totals; one is not a replacement for the other. Pulse's documented collection boundary is narrow but still needs review by the site owner. The SDK sends the trusted client IP and User-Agent to Apostl and records the **canonical origin and path only**. Its [README source][pulse-readme] says query parameters, fragments, request bodies, cookies, and authorization headers are not sent. Public health and API GET/HEAD requests are included, while account routes, assets, mutations, and 5xx responses are excluded. ## Put the limits next to the metric Every journey dashboard should state what it cannot establish. At minimum: - A journey is a time-bounded correlation, not a verified person, company, or exact model identity. - A requested page does not prove that its content appeared in an answer. - A sequence does not expose the prompt or the agent's internal reasoning. - Server traffic alone does not measure answer quality, task completion, pipeline, or revenue. - Attribution requires a separate, explicit join to product or conversion evidence. These labels make the data more useful because they keep each team inside the decisions the signal can support. Request-level data can govern access and delivery. Correlated sequences can reveal likely public content paths and where they break. Outcome instrumentation can then test whether a change improved the task that matters. If your next question is “which public docs and API surfaces do machine clients use together?”, [install Pulse SDK][pulse-page] and inspect the first server-side journeys. Keep your crawler controls in place; correlation adds a second view without replacing them. [cloudflare-analysis]: https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/ [cloudflare-start]: https://developers.cloudflare.com/ai-crawl-control/get-started/ [openai-bots]: https://developers.openai.com/api/docs/bots [anthropic-bots]: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler [pulse-github]: https://github.com/apostl-dev/pulse-sdk [pulse-readme]: https://raw.githubusercontent.com/apostl-dev/pulse-sdk/main/README.md [pulse-page]: /pulse