Eval

This page is auto-generated from the formations OpenAPI spec. Do not edit manually — run pnpm generate-formations-resource-docs to regenerate.

Binds an agent under test to a dataset and the scorers its outputs are judged by. pass_threshold is the pass rate a run must reach for its passed verdict — the gate an agent-version promotion consumes.

Syntax

type: eval
properties:
  name: String
  agent_id: String
  dataset_id: String
  scorers: Object[]
  pass_threshold: Number

Output

The physical resource ID is the public ID of the created resource. Reference it from other resources with a ref expression:

      some_field:
        ref: MyEval

Properties

name

Eval name, unique within the project

Required: Yes Type: String

---

agent_id

Public ID of the agent under test (or ref expression)

Required: Yes Type: String

---

dataset_id

Public ID of the dataset to run against (or ref expression)

Required: Yes Type: String

---

scorers

Scorer configs — exact_match, contains, json_logic, output_schema, embedding_similarity, llm_judge, or tool. Same shape as the evals REST contract.

Required: Yes Type: Array of Object

---

pass_threshold

0–1. A run passes when its pass rate over non-errored items reaches this. Omit for a run that reports scores without a verdict.

Required: No Type: Number Nullable: Yes

---