Skip to main content

Evaluate an Agent

Traces show what one run did; an evaluation shows whether the distribution of runs got better or worse after a prompt, model or tool change. A dataset holds test cases, an eval binds an agent to that dataset plus scorers, and a run executes the real agent against every case and scores the outputs. Build a small suite, run it, fix the prompt, and measure the fix against the first run as a baseline.

Scorers here are deterministic; no judge model. For open-ended answers, see Judge Open-Ended Answers.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

See Users.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create the agent under test​

A support agent with a deliberately vague prompt; the first run measures it, the second fixes it. See Projects, AI Providers, and Agents.

PROJECT_ID=$(soat create-project --name "Eval Workshop" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Billing Assistant" \
  --instructions "You are a billing support assistant. Answer in one short sentence." | jq -r '.id')

echo "AGENT_ID: $AGENT_ID"

Step 3 — Build a dataset​

A dataset item is one test case: input is an array of { role, content } messages replayed verbatim as the generation's input; metadata is a free-form bag the platform never interprets, readable from a json_logic scorer. Fields: Evaluations — Dataset item.

DATASET_ID=$(soat create-dataset \
  --project-id "$PROJECT_ID" \
  --name "billing-questions" \
  --description "Questions every release must still answer" | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"When is my invoice issued?"}]' \
  --expected-output "Your invoice is issued on the first of each month." \
  --metadata '{"topic":"invoicing"}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"How do I get a refund?"}]' \
  --expected-output "Open a refund request from the order page." \
  --metadata '{"topic":"refunds"}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"How do I cancel my plan?"}]' \
  --expected-output "Cancel from Billing then Subscription." \
  --metadata '{"topic":"cancellation"}' | jq -r '.id'

soat list-dataset-items --dataset-id "$DATASET_ID" | jq '.data | map({id, topic: .metadata.topic})'

A content purge never deletes or rewrites a dataset item.

Step 4 — Bind an eval with two scorers​

An eval freezes the criteria: agent under test, dataset, scorers, and the threshold the verdict gates on. Scorer config lives on the eval, not the agent, so two runs are judged the same way and their comparison measures the agent.

Two deterministic scorers:

ScorerAsks
json_logicDid the agent answer at all?
containsDid it include the mandated support hand-off?
EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "billing-regression" \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"json_logic","expression":{"!=":[{"var":"output"},""]}},{"type":"contains","value":"billing@example.com"}]' \
  --pass-threshold 0.67 | jq -r '.id')

soat get-eval --eval-id "$EVAL_ID" | jq '{name, pass_threshold, scorers}'

A json_logic expression is evaluated over input, output, object, expected, and item.metadata; other scorer types: Evaluations — Scorers.

Step 5 — Run it and read the verdict​

wait: true executes the items sequentially in-process and returns the run terminal, with scores. Capped at 25 items; larger suites use queued runs.

BASELINE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '{status, passed, agent_version, item_count, completed_count, errored_count, aggregate_scores}'

Expected shape (contains fails every item; the prompt never asks for a hand-off):

{
  "status": "completed",
  "passed": false,
  "agent_version": 1,
  "item_count": 3,
  "completed_count": 3,
  "errored_count": 0,
  "aggregate_scores": {
    "scorers": {
      "json_logic": { "mean": 1, "pass_rate": 1 },
      "contains": { "mean": 0, "pass_rate": 0 }
    },
    "pass_rate": 0,
    "scored_item_count": 3
  }
}

Per-item results say which case regressed:

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.data | map({input: .input[0].content, output, passed, scores: [.scores[] | {scorer, score}]})'

Each result carries generation_id, so any case opens as an ordinary generation with a readable trace.

Eval runs have real side effects

Every item is a real generation: an agent with a write-capable http or mcp tool performs N real writes per run. There is no tool-stub mode. Point an eval'd agent's tools at a staging target.

Step 6 — Fix the prompt, then measure the fix​

Add the missing instruction; this archives a new agent version, stamped on the next run.

soat update-agent --agent-id "$AGENT_ID" \
  --instructions "You are a billing support assistant. Answer in one short sentence, then add: For more help, contact billing@example.com" \
  --version-label "adds-handoff" | jq '{version}'

Re-run the same eval, naming the first run as the baseline:

CANDIDATE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true \
  --baseline-run-id "$BASELINE_RUN_ID" | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$CANDIDATE_RUN_ID" \
  | jq '{passed, agent_version, pass_rate: .aggregate_scores.pass_rate, baseline: .aggregate_scores.baseline}'

Expected shape (positive pass_rate_delta = improvement over the baseline):

{
  "passed": true,
  "agent_version": 2,
  "pass_rate": 1,
  "baseline": {
    "run_id": "evrun_stmdkHIXmu4KwTDl",
    "scorers": {
      "json_logic": { "mean_delta": 0, "pass_rate_delta": 0 },
      "contains": { "mean_delta": 1, "pass_rate_delta": 1 }
    },
    "pass_rate_delta": 1,
    "compared_item_count": 3,
    "added_item_count": 0,
    "removed_item_count": 0
  }
}
Your numbers will differ

qwen2.5:0.5b follows the instruction only some of the time, so pass_rate_delta may be 0.33 or 0.67 rather than 1. The direction must hold: the run told about the hand-off scores at least as well as the one that was not.

Every delta is computed over the item intersection (cases present and scorable in both runs); comparison rules: Evaluations.

Step 7 — Editing a case cannot rewrite history​

Every result carries its own frozen copy of the item's input and expected_output, taken at run time (Evaluations — Frozen inputs), so dataset items keep full CRUD without touching past runs.

ITEM_ID=$(soat list-dataset-items --dataset-id "$DATASET_ID" | jq -r '.data[0].id')

soat update-dataset-item --dataset-id "$DATASET_ID" --item-id "$ITEM_ID" \
  --input '[{"role":"user","content":"On what day is my invoice issued?"}]' \
  --expected-output "On the first of each month." | jq '{id, input}'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.data[0] | {frozen_input: .input[0].content, frozen_expected: .expected_output}'

A baseline delta therefore never reports dataset drift as agent regression. Deleting an item nulls dataset_item_id on past results and changes nothing else.

Step 8 — What the numbers mean​

Evaluations — Pass semantics: an item passes when all its scorers pass; the run's verdict gates on the pass rate against pass_threshold, never on a pooled mean.

Errors are not zeros

An item whose generation did not complete is an error: excluded from aggregate_scores, counted in errored_count, never scored 0. A run that scored nothing does not pass.

What's next​