# Evaluations

> MCP tools for the Evaluations module. See the [Evaluations module docs](/docs/modules/evaluations) for permissions and data model.

# Evaluations

MCP tools for the Evaluations module. See the [Evaluations module docs](/docs/modules/evaluations) for permissions and data model.

### `list-datasets`

List datasets

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `project_id` | `string` | no | Project ID (required if not using project key auth) |
| `limit` | `number` | no | Maximum number of results to return |
| `offset` | `number` | no | Number of results to skip |

### `create-dataset`

Create a dataset

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `project_id` | `string` | no | Project ID (required if not using project key auth) |
| `name` | `string` | yes | Unique name within the project |
| `description` | `string \| null` | no | What this suite covers |

### `get-dataset`

Get a dataset

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |

### `update-dataset`

Update a dataset

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `name` | `string` | no | — |
| `description` | `string \| null` | no | — |

### `delete-dataset`

Delete a dataset

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |

### `list-dataset-items`

List dataset items

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `limit` | `number` | no | Maximum number of results to return |
| `offset` | `number` | no | Number of results to skip |

### `create-dataset-item`

Add a dataset item

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `input` | `array<object>` | yes | Messages replayed verbatim as the generation's input |
| `expected_output` | `string \| null` | no | Reference answer for exact_match / embedding_similarity / llm_judge scorers |
| `metadata` | `object \| null` | no | Free-form tags, opaque to the platform |

### `create-dataset-item-from-generation`

Curate a dataset item from a generation

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `generation_id` | `string` | yes | The completed generation to promote. Must belong to the same project as the dataset. |
| `expected_output` | `string \| null` | no | Reference answer. Omit to use the generation's own answer; pass `null` to store the item with no reference answer. |
| `metadata` | `object \| null` | no | Free-form tags, opaque to the platform |

### `update-dataset-item`

Update a dataset item

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `item_id` | `string` | yes | Dataset item ID |
| `input` | `array<object>` | no | Messages replayed verbatim as the generation's input |
| `expected_output` | `string \| null` | no | — |
| `metadata` | `object \| null` | no | — |

### `delete-dataset-item`

Delete a dataset item

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `dataset_id` | `string` | yes | Dataset ID |
| `item_id` | `string` | yes | Dataset item ID |

### `list-evals`

List evals

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `project_id` | `string` | no | Project ID (required if not using project key auth) |
| `limit` | `number` | no | Maximum number of results to return |
| `offset` | `number` | no | Number of results to skip |

### `create-eval`

Create an eval

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `project_id` | `string` | no | Project ID (required if not using project key auth) |
| `name` | `string` | yes | Unique name within the project |
| `agent_id` | `string` | yes | The agent under test |
| `dataset_id` | `string` | yes | The dataset to run it against |
| `scorers` | `array<object \| object \| object \| object \| object \| object \| object>` | yes | Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. |
| `pass_threshold` | `number \| null` | no | 0–1. The run passes iff its pass rate — passed items over non-errored items — is at least this. Null reports scores without gating on them. |

### `get-eval`

Get an eval

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |

### `update-eval`

Update an eval

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `name` | `string` | no | — |
| `agent_id` | `string` | no | — |
| `dataset_id` | `string` | no | — |
| `scorers` | `array<object \| object \| object \| object \| object \| object \| object>` | no | Scorer configs, a discriminated union on `type`. Each type may appear at most once. Every scorer produces `{ score: 0–1, passed: boolean }`; binary scorers emit 0 or 1. `exact_match` compares the trimmed output text to `expected_output`. `contains` looks for `value` in the output text. `json_logic` evaluates `expression` over `{ input, output, object, expected, item.metadata }`, where `object` is the structured output (absent when the agent has no `output_schema`). `output_schema` validates the structured output against the scorer's own `schema`, falling back to the agent's; it requires the agent to carry an `output_schema`, because without one the platform emits no structured output and every item would score 0. `llm_judge` grades the output with a model completion, returning a continuous score plus its `reasoning`. Its `pass_threshold` is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. `embedding_similarity` embeds the output text and `expected_output` with the platform's configured embedding model (`EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its `pass_threshold` is required for the same reason as the judge's. An item without an `expected_output` scores 0; an embedding backend failure marks the **item** errored, never a score of 0. `tool` runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct `name` — outcomes and aggregates key on the name. |
| `pass_threshold` | `number \| null` | no | — |

### `delete-eval`

Delete an eval

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |

### `list-eval-runs`

List eval runs

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `limit` | `number` | no | Maximum number of results to return |
| `offset` | `number` | no | Number of results to skip |

### `start-eval-run`

Start an eval run

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `wait` | `boolean` | no | True runs the eval synchronously (25-item cap) and returns a terminal run with its scores. False — the default — enqueues the items and returns a `queued` run immediately. |
| `agent_version` | `number \| null` | no | An archived agent version to evaluate. Defaults to the active release's stable version, or the live draft version when no release is in effect. |
| `baseline_run_id` | `string \| null` | no | A terminal run of the same eval to compare against. The finished run's `aggregate_scores.baseline` reports per-scorer deltas over the item intersection. A run of a different eval is rejected with 400. |
| `metadata` | `object` | no | Caller-supplied key/value metadata attached to the run record for attribution — what this measurement was of (the commit or release candidate being scored, the CI job that asked for it). Round-trips verbatim on every read of the run, the list included. The bag is caller-owned and no key is reserved: everything the platform decides about a run (`status`, `agent_version`, `baseline_run_id`, `aggregate_scores`, `passed`, the counts) is a field of its own and cannot be written from here. Nothing in the scoring path reads it. A non-object is rejected with `400 VALIDATION_FAILED` and no run is created. |
| `tool_context` | `object` | no | Key/value context forwarded to every item's generation, so an agent whose tools authorize through `tool_context` is scored against the configuration it runs in production rather than with an empty bag. Each key is forwarded as one `X-Soat-Context-<key>` header and resolves any `{{context:<key>}}` token in a bound tool's headers or `preset_parameters`. Stored on the run and re-read per item, since a queued run (the default) is driven by a worker with no request behind it. **Write-only**: no read of the run returns it, unlike `metadata` — a run is a report other people read, and a credential in it is not theirs to see. Cleared once the run reaches a terminal state. An eval generation has no session, so the reserved keys `session_id`, `actor_id` and `actor_external_id` are dropped (in any casing) rather than forwarded. Every other key becomes an HTTP header name and must match that grammar, or the request is rejected with `400 INVALID_TOOL_CONTEXT_KEY` and no run is created. |

### `get-eval-run`

Get an eval run

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `eval_run_id` | `string` | yes | Eval run ID |

### `list-eval-results`

List eval run results

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `eval_run_id` | `string` | yes | Eval run ID |
| `limit` | `number` | no | Maximum number of results to return |
| `offset` | `number` | no | Number of results to skip |

### `cancel-eval-run`

Cancel an eval run

#### Arguments

| Argument | Type | Required | Description |
| -------- | ---- | -------- | ----------- |
| `eval_id` | `string` | yes | Eval ID |
| `eval_run_id` | `string` | yes | Eval run ID |