LLM Observability vs Inbound Agent Traffic Analytics

LangSmith defines a trace as the collection of runs for one operation. A run might be an LLM call, prompt-formatting step, retrieval request, or output parser. Observations sit inside traces, and related traces can be grouped into sessions in the Langfuse data model.

Both products depend on instrumentation inside the application. An integration, SDK, decorator, or OpenTelemetry exporter records what the application did and sends that telemetry to the observability backend. Once instrumented, an AI engineering team can inspect questions such as:

• Which prompt and model produced this answer?
• Which retrieval or tool step failed?
• How many tokens did the run use, and where did latency accumulate?
• Did a new release change an evaluator score or error rate?

Those questions require privileged access to the execution. The observer sees internal steps because the application emits them. A Langfuse trace covers the request lifecycle across prompts, model responses, token use, latency, tools, retrieval, and custom logic, according to its tracing overview. An anonymous website owner cannot reconstruct that lifecycle from a later page fetch.

Inbound agent traffic analytics sees the other side of the interaction. A documentation site, API portal, or product page receives a request from a client it does not control. The server can observe the method, path, response status, timing, headers, and network metadata available at that boundary. It cannot see the private prompt or reasoning that caused the request.

Apostl Pulse collects at the public server boundary. The open-source Pulse SDK observes eligible public requests in the server runtime, then estimates related journeys using bounded evidence. Requests with the same project, trusted IP, and full User-Agent are grouped until a 30-minute inactivity window expires. An exact llms.txt visit is immediate evidence; a generic non-browser client can qualify after visiting two distinct machine-readable surfaces within ten minutes.

The distinction changes how the dashboard should be read. This is a classification of observed traffic, not a declaration of who or what sits behind it. A generic HTTP client may be acting for a coding agent, a scheduled script, or a person at a terminal. A browser-shaped request may come from automation. Keeping an unknown bucket is part of the model, not a data-quality embarrassment.

OpenTelemetry semantic conventions help standardize names and attributes inside telemetry systems. Its GenAI attribute registry documents model and agent fields for emitted telemetry; it does not turn an anonymous inbound HTTP request into a trace for an application you do not operate. Likewise, a server access signal does not replace the prompt and tool detail that application instrumentation can record.

Suppose your company runs an internal support agent and publishes developer documentation. The support agent's observability stack can trace every model call it makes. A customer using Claude Code, Codex, Cursor, or a private agent can still request those docs without touching your support application. Your LLM tracing project remains silent because no instrumented code ran.

The server receives useful but narrower evidence: /llms.txt, an API reference, a Markdown guide, perhaps a pricing page. This can answer which public resources draw machine requests and where clients receive a 404, 403, or 429. It cannot tell you whether the agent was comparing vendors, fixing an integration, or drafting a purchase recommendation.

That is why “agent traffic increased” should not be reported as “agent adoption increased.” Traffic measures served requests. Adoption needs product use or another explicit outcome. A request path can choose the next page to inspect; it cannot prove the visitor's intent.

Reverse the scenario. Your team owns an agent that calls a search tool, retrieves three documents, invokes a model twice, and returns a wrong answer. The public sites it contacted may each see an inbound request. None sees the full execution tree.

The debugging job belongs in LLM observability. A trace can show whether retrieval returned the wrong document, a tool timed out, the prompt dropped a constraint, or the model output failed an evaluator. Langfuse recommends representing model calls and tool calls as typed, nested observations because the structure supports debugging, dashboards, datasets, and evaluations; the recommendation appears in its tracing guidance. LangSmith supports integrations and manual instrumentation for the same reason: the application must emit the steps that matter.

Server analytics can contribute one fact: a particular public resource was requested and returned a particular response. How the agent used that response, and why its final answer failed, remain private to the agent run.

Some companies operate an AI product and also publish the public material that outside agents use. They need both systems, but the reports should remain distinct unless a real correlation key exists.

Use LLM observability to investigate owned execution: prompts, model calls, tool paths, retrieval, latency, cost, errors, and evaluation. Use inbound analytics to inspect public demand: requested documentation, machine-readable endpoints, response patterns, and estimated journeys. Add authenticated product analytics for principal identity and task completion.

A useful weekly review might produce three separate conclusions:

• Inbound traffic shows repeated requests for an outdated integration guide.
• Server logs show that the guide returns successfully but its linked schema returns 404.
• Product events show whether authenticated integrations succeed after the repair.

Only authenticated product events can support the product outcome. Inbound data chooses where to look, while server logs diagnose delivery and product events measure completion. Combining the conclusions in one slide is fine. Treating them as one event stream is not.

AI engineering usually owns tracing because it can change prompts, models, tool code, exporters, and evaluations. Platform or reliability teams may own the shared OpenTelemetry pipeline. Security and compliance teams set retention and redaction rules because traces can contain prompts, responses, retrieved content, and user identifiers.

Inbound analytics often belongs closer to the public surface. DevRel can see which guides agents request. Web and API platform teams can investigate status codes and routing. Product growth can compare public demand with authenticated activation without claiming causality. Security can separate allowed public reading from blocked mutations or account routes.

The data contract should follow that ownership. Pulse records the canonical origin and path; query parameters, fragments, request bodies, cookies, and authorization headers are not sent. The SDK includes eligible public health and API GET and HEAD responses while excluding auth and account routes, assets, mutations, and 5xx responses. It sends trusted client IP and full User-Agent data for classification, so proxy trust and privacy review still matter.

Keep APOSTL_PULSE_API_KEY in the server runtime. The Pulse SDK README explicitly warns against placing it in a client bundle, NEXT_PUBLIC_*, HTML, logs, or a public diagnostics response.

Start with the question your current telemetry cannot answer.

If an owned agent produces expensive, slow, or incorrect runs, instrument the application and inspect traces. If the team cannot tell whether outside agents request its docs, pricing, API reference, or llms.txt, add inbound server analytics. If the question is “which account completed the integration,” neither telemetry system is enough by itself; instrument the authenticated outcome.

After installation, prove the collection path before trusting a dashboard. LLM observability needs a known test run whose spans and expected attributes appear. Inbound analytics needs a real request to the deployed public origin, a successful flush, and accepted delivery counters. Keep the expected event, deployment, time window, and verification result together so a missing chart does not become a speculative debugging exercise.

Explore Apostl Pulse when the missing measurement is public inbound agent traffic. It provides a server-side baseline for the pages agents use while preserving the distinction between an observed request, an estimated journey, and an identified actor.