Skip to main content

Start an eval run

POST 
/api/v1/evals/:eval_id/runs

Runs the eval against its dataset, creating one real agent generation per item and scoring the outputs.

wait: true executes the run synchronously and returns it terminal, with its scores. The dataset is capped at 25 items for a synchronous run; a larger one is rejected with 400 rather than partially scored.

wait: false (the default) enqueues one task per item and returns immediately with status: "queued". A worker executes the items and the run settles itself; poll GET /evals/{eval_id}/runs/{eval_run_id} for the terminal status, or subscribe to the eval_run.completed webhook. There is no item cap on a queued run.

The whole run is pinned to one agent version, stamped on agent_version: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (or the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into a single score.

With baseline_run_id, the finished run's aggregate_scores.baseline carries per-scorer deltas against that run, computed over the items present and scorable in both runs, with the divergence counted. A delta over a shifted dataset is therefore never presented as a clean comparison.

Request​

Responses​

Eval run finished (wait: true) or queued (wait: false)