Evaluations

MCP tools for the Evaluations module. See the Evaluations module docs for permissions and data model.

list-datasets

List datasets

Arguments

Argument	Type	Required	Description
project_id	string	no	Project ID (required if not using project key auth)
limit	number	no	Maximum number of results to return
offset	number	no	Number of results to skip

create-dataset

Create a dataset

Arguments

Argument	Type	Required	Description
project_id	string	no	Project ID (required if not using project key auth)
name	string	yes	Unique name within the project
description	string | null	no	What this suite covers

get-dataset

Get a dataset

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID

update-dataset

Update a dataset

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
name	string	no	—
description	string | null	no	—

delete-dataset

Delete a dataset

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID

list-dataset-items

List dataset items

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
limit	number	no	Maximum number of results to return
offset	number	no	Number of results to skip

create-dataset-item

Add a dataset item

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
input	array<object>	yes	Messages replayed verbatim as the generation's input
expected_output	string | null	no	Reference answer for exact_match / embedding_similarity / llm_judge scorers
metadata	object | null	no	Free-form tags, opaque to the platform

create-dataset-item-from-generation

Curate a dataset item from a generation

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
generation_id	string	yes	The completed generation to promote. Must belong to the same project as the dataset.
expected_output	string | null	no	Reference answer. Omit to use the generation's own answer; pass null to store the item with no reference answer.
metadata	object | null	no	Free-form tags, opaque to the platform

update-dataset-item

Update a dataset item

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
item_id	string	yes	Dataset item ID
input	array<object>	no	Messages replayed verbatim as the generation's input
expected_output	string | null	no	—
metadata	object | null	no	—

delete-dataset-item

Delete a dataset item

Arguments

Argument	Type	Required	Description
dataset_id	string	yes	Dataset ID
item_id	string	yes	Dataset item ID

list-evals

List evals

Arguments

Argument	Type	Required	Description
project_id	string	no	Project ID (required if not using project key auth)
limit	number	no	Maximum number of results to return
offset	number	no	Number of results to skip

create-eval

Create an eval

Arguments

Argument	Type	Required	Description
project_id	string	no	Project ID (required if not using project key auth)
name	string	yes	Unique name within the project
agent_id	string	yes	The agent under test
dataset_id	string	yes	The dataset to run it against
scorers	array<object | object | object | object | object | object | object>	yes	Scorer configs, a discriminated union on type. Each type may appear at most once. Every scorer produces { score: 0–1, passed: boolean }; binary scorers emit 0 or 1. exact_match compares the trimmed output text to expected_output. contains looks for value in the output text. json_logic evaluates expression over { input, output, object, expected, item.metadata }, where object is the structured output (absent when the agent has no output_schema). output_schema validates the structured output against the scorer's own schema, falling back to the agent's; it requires the agent to carry an output_schema, because without one the platform emits no structured output and every item would score 0. llm_judge grades the output with a model completion, returning a continuous score plus its reasoning. Its pass_threshold is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. embedding_similarity embeds the output text and expected_output with the platform's configured embedding model (EMBEDDING_PROVIDER / EMBEDDING_MODEL — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its pass_threshold is required for the same reason as the judge's. An item without an expected_output scores 0; an embedding backend failure marks the item errored, never a score of 0. tool runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct name — outcomes and aggregates key on the name.
pass_threshold	number | null	no	0–1. The run passes iff its pass rate — passed items over non-errored items — is at least this. Null reports scores without gating on them.

get-eval

Get an eval

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID

update-eval

Update an eval

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
name	string	no	—
agent_id	string	no	—
dataset_id	string	no	—
scorers	array<object | object | object | object | object | object | object>	no	Scorer configs, a discriminated union on type. Each type may appear at most once. Every scorer produces { score: 0–1, passed: boolean }; binary scorers emit 0 or 1. exact_match compares the trimmed output text to expected_output. contains looks for value in the output text. json_logic evaluates expression over { input, output, object, expected, item.metadata }, where object is the structured output (absent when the agent has no output_schema). output_schema validates the structured output against the scorer's own schema, falling back to the agent's; it requires the agent to carry an output_schema, because without one the platform emits no structured output and every item would score 0. llm_judge grades the output with a model completion, returning a continuous score plus its reasoning. Its pass_threshold is required: a continuous score says nothing about where "good enough" is, and a defaulted cutoff would silently decide the gate. embedding_similarity embeds the output text and expected_output with the platform's configured embedding model (EMBEDDING_PROVIDER / EMBEDDING_MODEL — the same stack document ingestion uses) and scores their cosine similarity, clamped to 0-1. Its pass_threshold is required for the same reason as the judge's. An item without an expected_output scores 0; an embedding backend failure marks the item errored, never a score of 0. tool runs a custom scoring algorithm: a server-callable project tool the engine invokes once per item with the item's context. Unlike the built-in types it may appear several times, each under a distinct name — outcomes and aggregates key on the name.
pass_threshold	number | null	no	—

delete-eval

Delete an eval

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID

list-eval-runs

List eval runs

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
limit	number	no	Maximum number of results to return
offset	number	no	Number of results to skip

start-eval-run

Start an eval run

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
wait	boolean	no	True runs the eval synchronously (25-item cap) and returns a terminal run with its scores. False — the default — enqueues the items and returns a queued run immediately.
agent_version	number | null	no	An archived agent version to evaluate. Defaults to the active release's stable version, or the live draft version when no release is in effect.
baseline_run_id	string | null	no	A terminal run of the same eval to compare against. The finished run's aggregate_scores.baseline reports per-scorer deltas over the item intersection. A run of a different eval is rejected with 400.
metadata	object	no	Caller-supplied key/value metadata attached to the run record for attribution — what this measurement was of (the commit or release candidate being scored, the CI job that asked for it). Round-trips verbatim on every read of the run, the list included. The bag is caller-owned and no key is reserved: everything the platform decides about a run (status, agent_version, baseline_run_id, aggregate_scores, passed, the counts) is a field of its own and cannot be written from here. Nothing in the scoring path reads it. A non-object is rejected with 400 VALIDATION_FAILED and no run is created.
tool_context	object	no	Key/value context forwarded to every item's generation, so an agent whose tools authorize through tool_context is scored against the configuration it runs in production rather than with an empty bag. Each key is forwarded as one X-Soat-Context-<key> header and resolves any {{context:<key>}} token in a bound tool's headers or preset_parameters. Stored on the run and re-read per item, since a queued run (the default) is driven by a worker with no request behind it. Write-only: no read of the run returns it, unlike metadata — a run is a report other people read, and a credential in it is not theirs to see. Cleared once the run reaches a terminal state. An eval generation has no session, so the reserved keys session_id, actor_id and actor_external_id are dropped (in any casing) rather than forwarded. Every other key becomes an HTTP header name and must match that grammar, or the request is rejected with 400 INVALID_TOOL_CONTEXT_KEY and no run is created.

get-eval-run

Get an eval run

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
eval_run_id	string	yes	Eval run ID

list-eval-results

List eval run results

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
eval_run_id	string	yes	Eval run ID
limit	number	no	Maximum number of results to return
offset	number	no	Number of results to skip

cancel-eval-run

Cancel an eval run

Arguments

Argument	Type	Required	Description
eval_id	string	yes	Eval ID
eval_run_id	string	yes	Eval run ID