Skip to main content

Evaluate an Agent

Every time you reword an instruction, swap a model, or add a tool, you ship a change whose effect you cannot see. Traces tell you what one run did. They cannot tell you whether the distribution of runs got better or worse — which is the only question that matters when the prompt you just edited serves production traffic.

An evaluation answers it. A dataset holds test cases, an eval binds an agent to that dataset plus a list of scorers, and a run executes the real agent against every case and scores the outputs. You will build a small suite, run it, fix the prompt, and measure the fix against the first run as a baseline.

Everything here is deterministic apart from the model's own wording — no judge model in the evaluation path. For grading open-ended answers, see Judge Open-Ended Answers.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide to bring the stack up with Docker Compose.
  • Ollama running locally with qwen2.5:0.5b available. This tutorial uses a local provider so it runs without external credentials — to connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.
  • New to SOAT? Read Key Concepts to understand projects, agents, and generations first.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • For production hardening (secrets, env vars), see Configuration.
  • Server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Admin is the built-in superuser role. See Users for authentication details.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create the agent under test​

A support agent with a vague prompt. The vagueness is the point: it is what the first run will measure and the second will fix. See Projects, AI Providers, and Agents for the resources it depends on.

PROJECT_ID=$(soat create-project --name "Eval Workshop" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Billing Assistant" \
  --instructions "You are a billing support assistant. Answer in one short sentence." | jq -r '.id')

echo "AGENT_ID: $AGENT_ID"

Step 3 — Build a dataset​

A dataset item is one test case. input is an array of { role, content } messages, replayed verbatim as the generation's input — so a case is exactly what a user would have sent. metadata is a free-form bag the platform never interprets, readable later from a json_logic scorer. See Evaluations — Dataset item for the full field list.

DATASET_ID=$(soat create-dataset \
  --project-id "$PROJECT_ID" \
  --name "billing-questions" \
  --description "Questions every release must still answer" | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"When is my invoice issued?"}]' \
  --expected-output "Your invoice is issued on the first of each month." \
  --metadata '{"topic":"invoicing"}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"How do I get a refund?"}]' \
  --expected-output "Open a refund request from the order page." \
  --metadata '{"topic":"refunds"}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"How do I cancel my plan?"}]' \
  --expected-output "Cancel from Billing then Subscription." \
  --metadata '{"topic":"cancellation"}' | jq -r '.id'

soat list-dataset-items --dataset-id "$DATASET_ID" | jq '.data | map({id, topic: .metadata.topic})'

Datasets are operator-owned fixtures. A content purge never deletes or rewrites a dataset item, so an unrelated erasure request cannot quietly stop your suite from being runnable.

Step 4 — Bind an eval with two scorers​

An eval freezes the criteria: the agent under test, the dataset, the scorers, and the threshold the run's verdict gates on. Scorer config lives here rather than being read off the agent at run time, so two runs of the same eval are always judged the same way and their comparison measures the agent instead of the criteria shifting underneath it.

Two deterministic scorers, each doing a different job:

ScorerAsks
json_logicDid the agent answer at all?
containsDid it include the mandated support hand-off?
EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "billing-regression" \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"json_logic","expression":{"!=":[{"var":"output"},""]}},{"type":"contains","value":"billing@example.com"}]' \
  --pass-threshold 0.67 | jq -r '.id')

soat get-eval --eval-id "$EVAL_ID" | jq '{name, pass_threshold, scorers}'

A json_logic expression is evaluated over input, output, object, expected, and item.metadata — see Evaluations — Scorers for the details and the other scorer types.

Step 5 — Run it and read the verdict​

wait: true executes the items sequentially in-process and returns the run terminal, with its scores. It is capped at 25 items — for anything larger, see queued runs.

BASELINE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '{status, passed, agent_version, item_count, completed_count, errored_count, aggregate_scores}'

Expected shape — the contains scorer fails every item, because nothing in the prompt asks for a hand-off:

{
  "status": "completed",
  "passed": false,
  "agent_version": 1,
  "item_count": 3,
  "completed_count": 3,
  "errored_count": 0,
  "aggregate_scores": {
    "scorers": {
      "json_logic": { "mean": 1, "pass_rate": 1 },
      "contains": { "mean": 0, "pass_rate": 0 }
    },
    "pass_rate": 0,
    "scored_item_count": 3
  }
}

Now look at the individual cases. This is where a failing suite becomes actionable — the run-level number says something regressed, the results say which case.

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.data | map({input: .input[0].content, output, passed, scores: [.scores[] | {scorer, score}]})'

Each result also carries generation_id, so any case can be opened as an ordinary generation and its trace read step by step. An eval run is real traffic, not a simulation.

Eval runs have real side effects

Every item is a real generation, so an agent with a write-capable http or mcp tool performs N real writes per run. There is no tool-stub mode, deliberately — running the real agent is what makes a score mean anything. Point an eval'd agent's tools at a staging target.

Step 6 — Fix the prompt, then measure the fix​

Add the missing instruction. This archives a new agent version, which the next run stamps on itself.

soat update-agent --agent-id "$AGENT_ID" \
  --instructions "You are a billing support assistant. Answer in one short sentence, then add: For more help, contact billing@example.com" \
  --version-label "adds-handoff" | jq '{version}'

Re-run the same eval, naming the first run as the baseline:

CANDIDATE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true \
  --baseline-run-id "$BASELINE_RUN_ID" | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$CANDIDATE_RUN_ID" \
  | jq '{passed, agent_version, pass_rate: .aggregate_scores.pass_rate, baseline: .aggregate_scores.baseline}'

Expected shape — a positive pass_rate_delta is the prompt change paying off:

{
  "passed": true,
  "agent_version": 2,
  "pass_rate": 1,
  "baseline": {
    "run_id": "evrun_stmdkHIXmu4KwTDl",
    "scorers": {
      "json_logic": { "mean_delta": 0, "pass_rate_delta": 0 },
      "contains": { "mean_delta": 1, "pass_rate_delta": 1 }
    },
    "pass_rate_delta": 1,
    "compared_item_count": 3,
    "added_item_count": 0,
    "removed_item_count": 0
  }
}
Your numbers will differ

qwen2.5:0.5b follows an instruction like this some of the time, not all of it, so your pass_rate_delta may be 0.33 or 0.67 rather than 1. That is the point of measuring instead of guessing — and the reason a suite is judged on its pass rate, not on one case. What must hold is the direction: the run that was told about the hand-off scores at least as well as the one that was not.

Positive deltas mean this run scored higher than the baseline. Every number is computed over the item intersection — the cases present and scorable in both runs; see Evaluations for the full comparison rules.

Step 7 — Editing a case cannot rewrite history​

Dataset items keep full CRUD. That is safe because every result carries its own frozen copy of the item's input and expected_output, taken at run time — see Evaluations — Frozen inputs. Edit a case and the runs that already scored it are untouched.

ITEM_ID=$(soat list-dataset-items --dataset-id "$DATASET_ID" | jq -r '.data[0].id')

soat update-dataset-item --dataset-id "$DATASET_ID" --item-id "$ITEM_ID" \
  --input '[{"role":"user","content":"On what day is my invoice issued?"}]' \
  --expected-output "On the first of each month." | jq '{id, input}'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.data[0] | {frozen_input: .input[0].content, frozen_expected: .expected_output}'

The old result still reads back the wording it was actually scored on, so a baseline delta can never report dataset drift as agent regression. Deleting an item nulls dataset_item_id on past results and changes nothing else.

Step 8 — What the numbers mean​

Per-scorer, per-item, and per-run pass rules are defined in Evaluations — Pass semantics. In short: an item passes when all its scorers pass, and the run's verdict gates on the pass rate against pass_threshold, never on a pooled mean.

Errors are not zeros

An item whose generation did not complete is recorded as an error: excluded from aggregate_scores, counted in errored_count, never scored 0. A run that scored nothing at all does not pass.

What's next​

Read next: Judge Open-Ended Answers for grading answers that have no single right string, Gate a Canary Promotion on an Eval to make a rollout wait for a green suite, and Evaluations for the full data model.