openapi: 3.0.3 info: title: SOAT Evaluations API version: 1.0.0 description: >- API for datasets, evals, and eval runs — the repeatable test suites that verify agent behavior before a change rolls out (Evaluations resource) contact: name: SOAT Team url: https://github.com/ttoss/soat servers: - url: '{baseUrl}' description: Base URL of your SOAT deployment (e.g. https://your-soat.com or http://localhost:5047) variables: baseUrl: description: The base URL of your SOAT deployment default: http://localhost:5047 tags: - name: Evaluations description: Datasets, evals, and eval runs security: - bearerAuth: [] paths: /api/v1/datasets: get: tags: - Evaluations summary: List datasets description: Returns the datasets defined in a project operationId: listDatasets parameters: - name: project_id in: query description: Project ID (required if not using project key auth) schema: type: string example: proj_V1StGXR8Z5jdHi6B - name: limit in: query required: false description: Maximum number of results to return schema: type: integer default: 50 - name: offset in: query required: false description: Number of results to skip schema: type: integer default: 0 responses: '200': description: List of datasets content: application/json: schema: type: object required: - data - total - limit - offset properties: data: type: array items: $ref: '#/components/schemas/Dataset' total: type: integer limit: type: integer offset: type: integer '401': description: Unauthorized '403': description: Forbidden '500': description: Internal server error post: tags: - Evaluations summary: Create a dataset description: >- Creates a project-scoped dataset — a named collection of test cases an eval runs an agent against. Names are unique per project. Datasets are operator-owned **fixtures**. The platform's content purge never deletes or mutates a dataset item, so erasing a generation cannot silently stop a test suite from being runnable. operationId: createDataset requestBody: required: true content: application/json: schema: type: object required: - name properties: project_id: x-soat-ref: projects type: string description: Project ID (required if not using project key auth) example: proj_V1StGXR8Z5jdHi6B name: type: string description: Unique name within the project example: billing-regressions description: type: string nullable: true description: What this suite covers example: Questions the billing agent regressed on in Q2 responses: '201': description: Dataset created successfully content: application/json: schema: $ref: '#/components/schemas/Dataset' '400': description: Bad request (missing or invalid name) '401': description: Unauthorized '403': description: Forbidden '409': description: A dataset with that name already exists in the project '500': description: Internal server error /api/v1/datasets/{dataset_id}: get: tags: - Evaluations summary: Get a dataset description: Returns a specific dataset operationId: getDataset parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B responses: '200': description: Dataset details content: application/json: schema: $ref: '#/components/schemas/Dataset' '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset not found put: tags: - Evaluations summary: Update a dataset description: Updates a dataset's name and/or description operationId: updateDataset parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object properties: name: type: string example: billing-regressions description: type: string nullable: true example: Questions the billing agent regressed on in Q2 responses: '200': description: Dataset updated successfully content: application/json: schema: $ref: '#/components/schemas/Dataset' '400': description: Bad request '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset not found '409': description: A dataset with that name already exists in the project delete: tags: - Evaluations summary: Delete a dataset description: >- Deletes a dataset, its items, and every eval bound to it. Results of runs that already scored those items keep their frozen copies of the input and expected output. operationId: deleteDataset parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B responses: '204': description: Dataset deleted successfully '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset not found /api/v1/datasets/{dataset_id}/items: get: tags: - Evaluations summary: List dataset items description: Returns the test cases in a dataset, oldest first operationId: listDatasetItems parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B - name: limit in: query required: false description: Maximum number of results to return schema: type: integer default: 50 - name: offset in: query required: false description: Number of results to skip schema: type: integer default: 0 responses: '200': description: List of dataset items content: application/json: schema: type: object required: - data - total - limit - offset properties: data: type: array items: $ref: '#/components/schemas/DatasetItem' total: type: integer limit: type: integer offset: type: integer '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset not found post: tags: - Evaluations summary: Add a dataset item description: >- Adds one test case. `input` is replayed verbatim as the generation's messages, so it must be a non-empty array of `{ role, content }`. operationId: createDatasetItem parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object required: - input properties: input: $ref: '#/components/schemas/DatasetItemInput' expected_output: type: string nullable: true description: Reference answer for exact_match / embedding_similarity / llm_judge scorers example: Your invoice is issued on the first of each month. metadata: type: object nullable: true additionalProperties: true description: Free-form tags, opaque to the platform example: topic: billing responses: '201': description: Dataset item created successfully content: application/json: schema: $ref: '#/components/schemas/DatasetItem' '400': description: Bad request (input is not message-shaped) '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset not found /api/v1/datasets/{dataset_id}/items/from-generation: post: tags: - Evaluations summary: Curate a dataset item from a generation description: >- Promotes a real, completed generation into a test case: its input messages become the item's `input`, and its own answer becomes `expected_output` unless you supply one. Use it to build an evaluation set out of production traffic rather than hand-authoring fixtures. The item is a **copy**, not a view. It keeps working after the source generation's content is purged, and `source_generation_id` goes null if that generation is deleted — a purge can never quietly stop a suite from being runnable. Requires both `evaluations:CreateDataset` and `generations:GetGeneration`: the call copies content out of a generation, so a principal that may not read that generation may not curate it either. Only a **completed** generation can be promoted (`409 GENERATION_NOT_COMPLETED`), and only while its content is still available: an agent or project running with `trace_content_mode: none` never stored the input, and a purged or expired generation no longer has it (`409 GENERATION_CONTENT_UNAVAILABLE`). Generations that predate input recording answer the same way. operationId: createDatasetItemFromGeneration parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object required: - generation_id properties: generation_id: type: string description: >- The completed generation to promote. Must belong to the same project as the dataset. example: gen_V1StGXR8Z5jdHi6B expected_output: type: string nullable: true description: >- Reference answer. Omit to use the generation's own answer; pass `null` to store the item with no reference answer. example: Your invoice is issued on the first of each month. metadata: type: object nullable: true additionalProperties: true description: Free-form tags, opaque to the platform example: topic: billing responses: '201': description: Dataset item created from the generation content: application/json: schema: $ref: '#/components/schemas/DatasetItem' '400': description: >- Bad request (generation_id missing, or the generation belongs to a different project than the dataset) '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset or generation not found '409': description: >- The generation has not completed, or its content was never stored or has been purged /api/v1/datasets/{dataset_id}/items/{item_id}: put: tags: - Evaluations summary: Update a dataset item description: >- Updates a test case. Runs that already scored it are unaffected — each result carries its own frozen copy of the input and expected output. operationId: updateDatasetItem parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B - name: item_id in: path required: true description: Dataset item ID schema: type: string example: dsit_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object properties: input: $ref: '#/components/schemas/DatasetItemInput' expected_output: type: string nullable: true example: Your invoice is issued on the first of each month. metadata: type: object nullable: true additionalProperties: true example: topic: billing responses: '200': description: Dataset item updated successfully content: application/json: schema: $ref: '#/components/schemas/DatasetItem' '400': description: Bad request '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset or item not found delete: tags: - Evaluations summary: Delete a dataset item description: >- Deletes a test case. Results of runs that already scored it stay readable; their `dataset_item_id` becomes null. operationId: deleteDatasetItem parameters: - name: dataset_id in: path required: true description: Dataset ID schema: type: string example: dset_V1StGXR8Z5jdHi6B - name: item_id in: path required: true description: Dataset item ID schema: type: string example: dsit_V1StGXR8Z5jdHi6B responses: '204': description: Dataset item deleted successfully '401': description: Unauthorized '403': description: Forbidden '404': description: Dataset or item not found /api/v1/evals: get: tags: - Evaluations summary: List evals description: Returns the evals defined in a project operationId: listEvals parameters: - name: project_id in: query description: Project ID (required if not using project key auth) schema: type: string example: proj_V1StGXR8Z5jdHi6B - name: limit in: query required: false description: Maximum number of results to return schema: type: integer default: 50 - name: offset in: query required: false description: Number of results to skip schema: type: integer default: 0 responses: '200': description: List of evals content: application/json: schema: type: object required: - data - total - limit - offset properties: data: type: array items: $ref: '#/components/schemas/Eval' total: type: integer limit: type: integer offset: type: integer '401': description: Unauthorized '403': description: Forbidden '500': description: Internal server error post: tags: - Evaluations summary: Create an eval description: >- Binds an agent under test to a dataset and a list of scorers. The agent and the dataset must belong to the same project as the eval; a cross-project reference is rejected with 400. Scorer config is frozen here rather than read from the agent at run time, so two runs of the same eval are always judged by the same criteria and their comparison measures the agent instead of the config drifting underneath it. Each scorer `type` may appear at most once. operationId: createEval requestBody: required: true content: application/json: schema: type: object required: - name - agent_id - dataset_id - scorers properties: project_id: x-soat-ref: projects type: string description: Project ID (required if not using project key auth) example: proj_V1StGXR8Z5jdHi6B name: type: string description: Unique name within the project example: billing-regression-suite agent_id: x-soat-ref: agents type: string description: The agent under test example: agent_V1StGXR8Z5jdHi6B dataset_id: type: string description: The dataset to run it against example: dset_V1StGXR8Z5jdHi6B scorers: $ref: '#/components/schemas/Scorers' pass_threshold: type: number nullable: true description: >- 0–1. The run passes iff its pass rate — passed items over non-errored items — is at least this. Null reports scores without gating on them. example: 0.8 responses: '201': description: Eval created successfully content: application/json: schema: $ref: '#/components/schemas/Eval' '400': description: Bad request (unknown scorer type, cross-project reference, invalid threshold) '401': description: Unauthorized '403': description: Forbidden '409': description: An eval with that name already exists in the project '500': description: Internal server error /api/v1/evals/{eval_id}: get: tags: - Evaluations summary: Get an eval description: Returns a specific eval operationId: getEval parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B responses: '200': description: Eval details content: application/json: schema: $ref: '#/components/schemas/Eval' '401': description: Unauthorized '403': description: Forbidden '404': description: Eval not found put: tags: - Evaluations summary: Update an eval description: >- Updates an eval. Changing `agent_id` re-validates the scorers against the new agent, since an `output_schema` scorer that was legal against the old one may not be. operationId: updateEval parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object properties: name: type: string example: billing-regression-suite agent_id: x-soat-ref: agents type: string example: agent_V1StGXR8Z5jdHi6B dataset_id: type: string example: dset_V1StGXR8Z5jdHi6B scorers: $ref: '#/components/schemas/Scorers' pass_threshold: type: number nullable: true example: 0.8 responses: '200': description: Eval updated successfully content: application/json: schema: $ref: '#/components/schemas/Eval' '400': description: Bad request '401': description: Unauthorized '403': description: Forbidden '404': description: Eval not found '409': description: An eval with that name already exists in the project delete: tags: - Evaluations summary: Delete an eval description: Deletes an eval, its runs, and their results operationId: deleteEval parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B responses: '204': description: Eval deleted successfully '401': description: Unauthorized '403': description: Forbidden '404': description: Eval not found /api/v1/evals/{eval_id}/runs: get: tags: - Evaluations summary: List eval runs description: Returns an eval's runs, newest first operationId: listEvalRuns parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B - name: limit in: query required: false description: Maximum number of results to return schema: type: integer default: 50 - name: offset in: query required: false description: Number of results to skip schema: type: integer default: 0 responses: '200': description: List of eval runs content: application/json: schema: type: object required: - data - total - limit - offset properties: data: type: array items: $ref: '#/components/schemas/EvalRun' total: type: integer limit: type: integer offset: type: integer '401': description: Unauthorized '403': description: Forbidden '404': description: Eval not found post: tags: - Evaluations summary: Start an eval run description: >- Runs the eval against its dataset, creating one real agent generation per item and scoring the outputs. `wait: true` executes the run synchronously and returns it terminal, with its scores. The dataset is capped at 25 items for a synchronous run; a larger one is rejected with 400 rather than partially scored. `wait: false` (the default) enqueues one task per item and returns immediately with `status: "queued"`. A worker executes the items and the run settles itself; poll `GET /evals/{eval_id}/runs/{eval_run_id}` for the terminal status, or subscribe to the `eval_run.completed` webhook. There is no item cap on a queued run. The whole run is pinned to **one** agent version, stamped on `agent_version`: pass one explicitly to evaluate a canary before promoting it, or omit it to use the active release's stable version (or the live draft when no release is in effect). Without the pin, release assignment would bucket each item independently and blend two configs into a single score. With `baseline_run_id`, the finished run's `aggregate_scores.baseline` carries per-scorer deltas against that run, computed over the items present and scorable in **both** runs, with the divergence counted. A delta over a shifted dataset is therefore never presented as a clean comparison. operationId: startEvalRun parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B requestBody: required: true content: application/json: schema: type: object properties: wait: type: boolean default: false description: >- True runs the eval synchronously (25-item cap) and returns a terminal run with its scores. False — the default — enqueues the items and returns a `queued` run immediately. example: true agent_version: type: integer nullable: true description: >- An archived agent version to evaluate. Defaults to the active release's stable version, or the live draft version when no release is in effect. example: 3 baseline_run_id: type: string nullable: true description: >- A terminal run of the same eval to compare against. The finished run's `aggregate_scores.baseline` reports per-scorer deltas over the item intersection. A run of a different eval is rejected with 400. example: evrun_V1StGXR8Z5jdHi6B metadata: type: object additionalProperties: true description: >- Caller-supplied key/value metadata attached to the run record for attribution — what this measurement was of (the commit or release candidate being scored, the CI job that asked for it). Round-trips verbatim on every read of the run, the list included. The bag is caller-owned and no key is reserved: everything the platform decides about a run (`status`, `agent_version`, `baseline_run_id`, `aggregate_scores`, `passed`, the counts) is a field of its own and cannot be written from here. Nothing in the scoring path reads it. A non-object is rejected with `400 VALIDATION_FAILED` and no run is created. example: commit_sha: 9f2c1ab ci_job: nightly-evals tool_context: type: object additionalProperties: type: string description: >- Key/value context forwarded to every item's generation, so an agent whose tools authorize through `tool_context` is scored against the configuration it runs in production rather than with an empty bag. Each key is forwarded as one `X-Soat-Context-` header and resolves any `{{context:}}` token in a bound tool's headers or `preset_parameters`. Stored on the run and re-read per item, since a queued run (the default) is driven by a worker with no request behind it. **Write-only**: no read of the run returns it, unlike `metadata` — a run is a report other people read, and a credential in it is not theirs to see. Cleared once the run reaches a terminal state. An eval generation has no session, so the reserved keys `session_id`, `actor_id` and `actor_external_id` are dropped (in any casing) rather than forwarded. Every other key becomes an HTTP header name and must match that grammar, or the request is rejected with `400 INVALID_TOOL_CONTEXT_KEY` and no run is created. example: ocaToken: eyJhbGciOiJIUzI1NiJ9.abc tenant: acme responses: '201': description: >- Eval run finished (`wait: true`) or queued (`wait: false`) content: application/json: schema: $ref: '#/components/schemas/EvalRun' '400': description: >- Bad request (non-boolean wait, dataset empty or over the synchronous cap, unknown agent_version, invalid baseline, scorers no longer valid against the agent, a `tool_context` key that cannot become a header) '401': description: Unauthorized '403': description: Forbidden '404': description: Eval not found '500': description: Internal server error /api/v1/evals/{eval_id}/runs/{eval_run_id}: get: tags: - Evaluations summary: Get an eval run description: Returns a run's status, counts, and aggregate scores operationId: getEvalRun parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B - name: eval_run_id in: path required: true description: Eval run ID schema: type: string example: evrun_V1StGXR8Z5jdHi6B responses: '200': description: Eval run details content: application/json: schema: $ref: '#/components/schemas/EvalRun' '401': description: Unauthorized '403': description: Forbidden '404': description: Eval or run not found /api/v1/evals/{eval_id}/runs/{eval_run_id}/results: get: tags: - Evaluations summary: List eval run results description: Returns the per-item results of a run, oldest first operationId: listEvalResults parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B - name: eval_run_id in: path required: true description: Eval run ID schema: type: string example: evrun_V1StGXR8Z5jdHi6B - name: limit in: query required: false description: Maximum number of results to return schema: type: integer default: 50 - name: offset in: query required: false description: Number of results to skip schema: type: integer default: 0 responses: '200': description: List of eval results content: application/json: schema: type: object required: - data - total - limit - offset properties: data: type: array items: $ref: '#/components/schemas/EvalResult' total: type: integer limit: type: integer offset: type: integer '401': description: Unauthorized '403': description: Forbidden '404': description: Eval or run not found /api/v1/evals/{eval_id}/runs/{eval_run_id}/cancel: post: tags: - Evaluations summary: Cancel an eval run description: >- Cancels a queued or running run: its outstanding item tasks are dropped so it stops consuming provider budget, and the run settles as `canceled`. Results already written are kept — they are real measurements of generations that were really paid for — and `completed_count` / `errored_count` report what ran. `aggregate_scores` is deliberately left null: a partial roll-up in the same field a completed run uses would read as a whole-dataset verdict. A run that has already finished is rejected with 400. operationId: cancelEvalRun parameters: - name: eval_id in: path required: true description: Eval ID schema: type: string example: eval_V1StGXR8Z5jdHi6B - name: eval_run_id in: path required: true description: Eval run ID schema: type: string example: evrun_V1StGXR8Z5jdHi6B responses: '200': description: Eval run canceled content: application/json: schema: $ref: '#/components/schemas/EvalRun' '400': description: The run has already finished '401': description: Unauthorized '403': description: Forbidden '404': description: Eval or run not found '500': description: Internal server error components: schemas: DatasetItemInput: type: array description: Messages replayed verbatim as the generation's input items: type: object required: - role - content properties: role: type: string example: user content: description: Message content — a string, or AI SDK content parts example: When is my invoice issued? Scorers: type: array description: >- Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. items: oneOf: - $ref: '#/components/schemas/ExactMatchScorer' - $ref: '#/components/schemas/ContainsScorer' - $ref: '#/components/schemas/JsonLogicScorer' - $ref: '#/components/schemas/OutputSchemaScorer' - $ref: '#/components/schemas/EmbeddingSimilarityScorer' - $ref: '#/components/schemas/LlmJudgeScorer' - $ref: '#/components/schemas/ToolScorer' ExactMatchScorer: type: object required: - type properties: type: type: string enum: [exact_match] ContainsScorer: type: object required: - type - value properties: type: type: string enum: [contains] value: type: string example: invoice case_sensitive: type: boolean default: false JsonLogicScorer: type: object required: - type - expression properties: type: type: string enum: [json_logic] expression: type: object additionalProperties: true description: A JSON Logic expression; a truthy result scores 1 OutputSchemaScorer: type: object required: - type properties: type: type: string enum: [output_schema] schema: type: object additionalProperties: true description: >- JSON Schema the structured output is validated against. Frozen here so two runs stay comparable; falls back to the agent's `output_schema` when omitted. EmbeddingSimilarityScorer: type: object required: - type - pass_threshold properties: type: type: string enum: [embedding_similarity] pass_threshold: type: number minimum: 0 maximum: 1 description: >- The item passes this scorer when the cosine similarity between the embeddings of the output text and `expected_output` is greater than or equal to this value. Required. example: 0.85 LlmJudgeScorer: type: object required: - type - prompt - pass_threshold properties: type: type: string enum: [llm_judge] prompt: type: string description: >- The judge prompt. `{{input}}`, `{{output}}` and `{{expected}}` are replaced with the item's input messages, the agent's output text, and the item's `expected_output`. Slots are filled in one pass, so a slot value that itself contains `{{output}}` is not re-expanded. The judge must answer with a JSON object carrying a numeric `score` between 0 and 1 and an optional `reasoning` string; a reply that does not marks the **item** errored, never the run failed and never a score of 0. example: >- Rate 0-1 how well the answer matches the reference. Answer with {"score": <0-1>, "reasoning": ""}. Question: {{input}} Answer: {{output}} Reference: {{expected}} pass_threshold: type: number minimum: 0 maximum: 1 description: >- The item passes this scorer when the judge's score is greater than or equal to this value. Required. example: 0.7 ai_provider_id: type: string nullable: true description: >- The AI provider that runs the judge; it must belong to the eval's project. Omit to use the project's default model route. example: aip_V1StGXR8Z5jdHi6B model: type: string nullable: true description: >- Overrides the provider's default model. Pinned per scorer, because deltas between runs judged by different models are not comparable. example: gpt-4o-mini ToolScorer: type: object description: >- A custom scoring algorithm — a server-callable project tool the engine invokes once per item. The tool receives the same variables a `json_logic` expression reads — `input`, `output`, `object` (when the agent has an `output_schema`), `expected`, and `item.metadata` — with `preset_parameters` merged in at the top level, and must answer with a JSON object carrying a numeric `score` between 0 and 1, an optional boolean `passed`, and an optional `reasoning` string. A malformed answer or a failed call marks the **item** errored, never the run failed and never a score of 0. required: - type - name - tool_id properties: type: type: string enum: [tool] name: type: string description: >- Keys this scorer's outcomes and aggregate scores, so it must be unique within the eval and must not shadow a built-in scorer type. Unlike the built-in types, several `tool` scorers may coexist under distinct names. example: toxicity tool_id: x-soat-ref: tools type: string description: >- The tool that scores each item. It must belong to the eval's project and be server-callable (`http`, `mcp`, `builtin`, or `pipeline` — a `client` tool pauses for a calling client an eval run does not have). example: tool_V1StGXR8Z5jdHi6B action: type: string nullable: true description: >- The operation to invoke; required when the tool type is `builtin` or `mcp`. example: score-toxicity preset_parameters: type: object nullable: true additionalProperties: true description: >- Fixed values merged into every call's input at the top level. The engine-injected keys (`input`, `output`, `object`, `expected`, `item`) are reserved and rejected. pass_threshold: type: number nullable: true minimum: 0 maximum: 1 description: >- Fallback verdict cutoff when the tool answers without a `passed` flag: the item passes this scorer when `score` is greater than or equal to this value. A tool-returned `passed` always wins. When the tool omits `passed` and no threshold is set, the item is recorded as errored — the scorer produced no verdict. example: 0.5 ScorerResult: type: object properties: scorer: type: string description: >- The scorer that produced this entry — the scorer type, or for a `tool` scorer its `name` example: contains score: type: number example: 1 passed: type: boolean reasoning: type: string description: >- The stated rationale; present for `llm_judge` and for `tool` scorers whose tool returned one Dataset: type: object properties: id: type: string example: dset_V1StGXR8Z5jdHi6B project_id: x-soat-ref: projects type: string name: type: string description: type: string nullable: true created_at: type: string format: date-time updated_at: type: string format: date-time DatasetItem: type: object properties: id: type: string example: dsit_V1StGXR8Z5jdHi6B dataset_id: type: string example: dset_V1StGXR8Z5jdHi6B input: $ref: '#/components/schemas/DatasetItemInput' expected_output: type: string nullable: true metadata: type: object nullable: true additionalProperties: true source_generation_id: type: string nullable: true description: >- The generation this item was curated from. A curated item is a deliberate fixture: erasing the source generation neither deletes nor mutates it. example: gen_V1StGXR8Z5jdHi6B created_at: type: string format: date-time updated_at: type: string format: date-time Eval: type: object properties: id: type: string example: eval_V1StGXR8Z5jdHi6B project_id: x-soat-ref: projects type: string name: type: string agent_id: x-soat-ref: agents type: string example: agent_V1StGXR8Z5jdHi6B dataset_id: type: string example: dset_V1StGXR8Z5jdHi6B scorers: $ref: '#/components/schemas/Scorers' pass_threshold: type: number nullable: true created_at: type: string format: date-time updated_at: type: string format: date-time AggregateScores: type: object nullable: true description: Per-scorer rollup plus the run-level pass rate; null until the run is terminal properties: scorers: type: object additionalProperties: type: object properties: mean: type: number pass_rate: type: number pass_rate: type: number nullable: true description: Passed items over non-errored items; null when nothing was scorable scored_item_count: type: integer description: Items that produced a score — errored items are excluded baseline: $ref: '#/components/schemas/BaselineComparison' BaselineComparison: type: object nullable: true description: >- Comparison against the run named by `baseline_run_id`; absent when the run named none. Every number here is computed over the **item intersection** — the dataset items present and scorable in both runs — because a delta only means something when both sides answered the same question. The compared/added/removed counts make any dataset drift visible instead of letting it read as agent regression. Positive deltas mean this run scored higher than the baseline. properties: run_id: type: string example: evrun_V1StGXR8Z5jdHi6B compared_item_count: type: integer description: Items scorable in both runs — the basis of every delta added_item_count: type: integer description: Scorable here but not in the baseline (added, or errored there) removed_item_count: type: integer description: Scorable in the baseline but not here (removed, or errored here) pass_rate_delta: type: number nullable: true description: >- Run-level pass-rate delta over the intersection; null when the two runs share no comparable item scorers: type: object description: >- Per-scorer deltas, keyed by scorer type. A scorer only one of the two runs ran is omitted rather than compared against nothing. additionalProperties: type: object properties: mean_delta: type: number pass_rate_delta: type: number EvalRun: type: object properties: id: type: string example: evrun_V1StGXR8Z5jdHi6B eval_id: type: string example: eval_V1StGXR8Z5jdHi6B agent_version: type: integer description: The one agent version every item in this run executed against example: 3 status: type: string enum: [queued, running, completed, failed, canceled] baseline_run_id: type: string nullable: true example: evrun_V1StGXR8Z5jdHi6B trigger_id: type: string nullable: true description: >- The trigger that started this run — set when a schedule (or a manual trigger fire) started it, null for a run started through this API. Kept even if the trigger is later deleted. example: trg_V1StGXR8Z5jdHi6B aggregate_scores: $ref: '#/components/schemas/AggregateScores' passed: type: boolean nullable: true description: >- Null when the eval declares no pass_threshold, and until the run is terminal item_count: type: integer completed_count: type: integer errored_count: type: integer metadata: type: object additionalProperties: true nullable: true description: >- The caller-owned key/value metadata supplied when the run was started, returned verbatim. Null when the run was started without any (a trigger-started run included — see `trigger_id` for that provenance). The server writes nothing here. example: commit_sha: 9f2c1ab ci_job: nightly-evals started_at: type: string format: date-time nullable: true finished_at: type: string format: date-time nullable: true created_at: type: string format: date-time EvalResult: type: object properties: id: type: string example: evres_V1StGXR8Z5jdHi6B eval_run_id: type: string example: evrun_V1StGXR8Z5jdHi6B dataset_item_id: type: string nullable: true description: Null once the dataset item has been deleted example: dsit_V1StGXR8Z5jdHi6B input: $ref: '#/components/schemas/DatasetItemInput' expected_output: type: string nullable: true generation_id: type: string nullable: true example: gen_V1StGXR8Z5jdHi6B output: type: string nullable: true description: >- The agent's final output text. Cleared when the linked generation's content is purged; the scores and the frozen input survive. scores: type: array items: $ref: '#/components/schemas/ScorerResult' passed: type: boolean description: AND over the per-scorer passed flags error: type: string nullable: true description: >- Item-level failure reason. A generation that did not complete — a `requires_action` pause, a provider failure — is recorded here and excluded from the aggregates rather than scored 0. created_at: type: string format: date-time securitySchemes: bearerAuth: type: http scheme: bearer description: JWT token or sk_ api key