---
title: Custom Agentic Benchmarks for Developer Journeys | Apostl
description: Test whether AI agents can complete critical SDK, API, MCP, and documentation journeys with observable outcomes and reproducible evidence.
canonical: https://apostl.dev/solutions/custom-agentic-benchmarks
schema_type: TechArticle
date_modified: 2026-08-27
eyebrow: Agent Experience
primary_cta_label: Make Custom Benchmarks
primary_cta_href: https://cal.com/apostl/15min
---
# Custom Agentic Benchmarks

A custom benchmark tests whether an AI agent can complete the integration your product promises. It turns that question into a versioned task with a defined environment, an observable stop condition, and evidence your team can inspect.

Apostl builds benchmarks around real developer journeys through your public documentation, SDK, API, or MCP surface. We establish the baseline, show where the journey stops, and rerun selected paths when the product changes. See the current public format in [Agent Arena](/arena).

## Docs can be readable while integration stays unmeasured. {#why-benchmark}

Tempo made the problem concrete in [Introducing stable-bench-v1](https://tempo.xyz/developers/blog/introducing-stable-bench-v1), published on 21 July 2026. The team reported that agents drove over 70% of its documentation traffic on an average day in July. Tempo had already adopted Markdown conversion and `llms.txt`, but those interfaces did not show whether an agent could build a working integration.

Its answer was `stable-bench-v1`: a reproducible evaluation suite for stablecoin applications. Tempo measures whether an agent finishes the task, whether the implementation uses current primitives, and how many turns and tokens it consumes. The benchmark and run data are available in the [tempo-evals repository](https://github.com/tempoxyz/tempo-evals).

The important move is from document availability to task completion. An agent can find a guide and still choose an old package, miss a required funding step, stop at a plausible response, or produce code that never reaches the target system. Searchability and `llms.txt` help the agent enter the journey. A benchmark checks whether it exits with a result.

## Start with the job, then define proof. {#benchmark-contract}

Every Apostl benchmark begins with one user request and one terminal outcome. “Integrate our SDK” is too broad. “Submit a testnet transaction and return its verified receipt” can be tested. So can an accepted order, a deployed address, or a successful inference response with the required output.

The measurement contract fixes four things before a run:

- The task states what the agent must accomplish in language a developer would use.
- The environment defines the public guide, runtime, tools, and scoped test credentials available to the agent.
- The stop condition names the evidence that proves completion.
- The run record preserves the route, commands, responses, elapsed time, frictions, and any human interruption.

This is where a custom benchmark differs from a synthetic docs score. The evaluator does not reward code that looks reasonable or a setup command that exits cleanly. HTTP 200 is not terminal proof unless that response satisfies the stated stop condition.

Tempo uses a hidden oracle and an independent verifier for each versioned task. One published example asks an agent to create and fund an account, create a denylist policy, deploy a `TIP-20` stablecoin, submit transactions, and write the resulting addresses and transaction hashes to an artifact. Apostl does not copy that task or grading system. We apply the same discipline to the journey your product needs to verify.

## Measure the run your team can improve. {#measurement}

Apostl reports four comparable signals: verified outcome, time to result, unique frictions, and normalized model cost. Together they separate a completed journey from a fast-looking failure.

The verified outcome is the primary result. If the agent stops because a faucet is unavailable, a credential is missing, an SDK call fails, or the documentation points to an outdated path, the run remains blocked. The report assigns the finding to the relevant documentation, SDK, product, credential, external dependency, or environment boundary instead of turning every failure into a docs problem.

Time, frictions, and cost explain what happened around that result. A later run can show that the same task still completes with fewer recovery steps, or catch the opposite case when a release leaves the happy-path prose intact while breaking the actual integration.

The comparison remains bounded. Each result describes one dated journey and environment, not a universal product rating. Changing the task, credentials, runner, or acceptance criteria changes what the benchmark means, so those inputs stay visible beside the result.

## Turn one critical journey into a benchmark. {#engagement}

The first scope is intentionally narrow. Your team selects a journey that matters to activation, a partner launch, or a release. Together we define the user request, the target environment, and the evidence that would convince your technical owner that the task finished.

Apostl then runs the journey in a clean isolated environment and returns the reviewed result with the commands, responses, blocker context, and acceptance boundary. The baseline shows both the terminal outcome and the exact path taken to reach it. When the benchmark uncovers a problem, your team can review a scoped next action rather than sort through a generic audit backlog.

To start, we need:

- the public documentation, SDK, API, or MCP path the agent should use;
- a test environment and scoped credentials when the task requires them;
- the success criteria for the terminal outcome;
- one technical owner who can review the boundary and evidence.

After the baseline, selected critical paths can be rerun as the documentation, SDK, API, authentication flow, or test environment changes. The same contract turns an isolated finding into a regression check.

## Make the benchmark useful outside the report. {#use-the-result}

A benchmark earns its place when it changes a product decision. The trace shows Docs where an instruction sends the agent down the wrong path, gives SDK Engineering a reproducible failure, and tells the Developer Platform owner whether a release changed completion, recovery time, or cost. If the path still works, the run provides dated evidence instead of an assumption.

[Agent Arena](/arena) shows how Apostl presents reviewed public results: the task appears above the table, the runner and methodology stay visible, and blocked runs do not receive passing labels. A custom engagement begins with your own journey and acceptance criteria; any public comparison requires a scope that supports a fair, reviewable result.

If one agent journey affects activation, partner delivery, or release confidence, make it testable. [Make Custom Benchmarks](https://cal.com/apostl/15min) with Apostl, starting from one task and one outcome your team already cares about.