Skip to main content

Judge Open-Ended Answers

exact_match and contains work when the right answer is a known string. Most agent output is not like that: a summary, an explanation, a rewritten paragraph — all have many correct forms and no single one to match. Assert on the wording and you measure phrasing, not quality.

An llm_judge scorer grades the answer with a model completion instead, returning a continuous 0–1 score plus its reasoning. It is an ordinary, tool-less completion resolving through the same AI providers path.

You will bind an llm_judge scorer next to a deterministic one, run it and read each item's score and reasoning, see why an unparseable verdict is an error rather than a zero, run the same eval queued and poll for the verdict, and cancel a run mid-flight.

This tutorial assumes you have been through Evaluate an Agent.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide to bring the stack up with Docker Compose.
  • Ollama running locally with qwen2.5:0.5b available. This tutorial uses a local provider so it runs without external credentials — to connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.
  • New to SOAT? Read Key Concepts to understand projects, agents, and evaluations first.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • For production hardening (secrets, env vars), see Configuration.
  • Server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Admin is the built-in superuser role. See Users for authentication details.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — An agent with open-ended output​

A support-reply drafter. There is no single correct draft, which is exactly why a string matcher cannot grade it. See Agents and Evaluations — Dataset for the resources involved.

PROJECT_ID=$(soat create-project --name "Judge Workshop" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Reply Drafter" \
  --instructions "Draft a short, warm reply to the customer message. Two sentences at most." | jq -r '.id')

DATASET_ID=$(soat create-dataset --project-id "$PROJECT_ID" --name "reply-drafts" | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"My package arrived damaged. What now?"}]' \
  --expected-output "Apologize, then offer a replacement or a refund and ask for a photo of the damage." | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"I was charged twice this month."}]' \
  --expected-output "Apologize, confirm the duplicate charge will be refunded, and give the expected timeline." | jq -r '.id'

Step 3 — Bind the judge​

The judge's prompt carries three slots the platform fills per item — {{input}}, {{output}}, and {{expected}}; see Evaluations — LLM judge.

pass_threshold on the scorer is required, with no default — a judge emits a continuous score, so nothing about the number itself says where "good enough" is. Keep a deterministic scorer alongside the judge as a cheap structural floor.

EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "reply-quality" \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"json_logic","expression":{"!=":[{"var":"output"},""]}},{"type":"llm_judge","ai_provider_id":"'"$AI_PROVIDER_ID"'","prompt":"You grade customer support drafts. Reply with only JSON: {\"score\": <number 0-1>, \"reasoning\": \"<one sentence>\"}. Customer message: {{input}} Draft reply: {{output}} Reference answer: {{expected}}","pass_threshold":0.7}]' \
  --pass-threshold 0.5 | jq -r '.id')

soat get-eval --eval-id "$EVAL_ID" | jq '.scorers | map({type, pass_threshold})'

Slots are filled in one pass — a slot value containing {{output}} is never re-expanded, and an unrecognized {{…}} is left as written so a typo stays visible.

Step 4 — Run it and read the reasoning​

RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '{status, passed, completed_count, errored_count, aggregate_scores}'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '.data | map({output, error, scores})'

A judged item looks like this — the score gates, and reasoning is stored for audit:

{
  "output": "I am sorry your package arrived damaged. Send us a photo and we will ship a replacement right away.",
  "error": null,
  "scores": [
    { "scorer": "json_logic", "score": 1, "passed": true },
    {
      "scorer": "llm_judge",
      "score": 0.9,
      "passed": true,
      "reasoning": "Apologizes, asks for a photo, and offers a replacement."
    }
  ]
}
A small judge model may not answer in JSON — and that is instructive

The judge must reply with a JSON object carrying a numeric score between 0 and 1. qwen2.5:0.5b frequently ignores that contract, so on this local stack items often come back with error set and no scores. That is designed behavior: a judge that could not reach a verdict means the answer was never graded, so recording 0 would fabricate a regression; an out-of-range score is likewise rejected, not clamped. Errored items are excluded from aggregate_scores and counted in errored_count. Use a real judge model (see Connect Third-Party LLMs) for a suite you intend to gate on.

The judge model is pinned per scorer config — deltas between runs judged by different models are not comparable, so re-run the baseline when you change the judge. The judge's ai_provider_id must belong to the eval's project, and the project's default model route applies when the scorer pins none.

Step 5 — Run it queued instead of blocking​

A judged suite makes two provider calls per item — exactly the workload not to hold a request open for. wait selects the mode; both modes share one execution path, so their runs are directly comparable.

waitBehavior
trueExecutes items sequentially in-process, returns the run terminal with its scores. Capped at 25 items.
false (default)Enqueues one task per item, returns immediately with status: "queued". No item cap.
QUEUED_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false)
QUEUED_RUN_ID=$(printf '%s' "$QUEUED_RUN" | jq -r '.id')

printf '%s' "$QUEUED_RUN" | jq '{status, item_count, aggregate_scores}'

# → retry 180
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$QUEUED_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$QUEUED_RUN_ID" \
  | jq '{status, passed, completed_count, errored_count, pass_rate: .aggregate_scores.pass_rate}'

Each item becomes one queued task; the worker that drains the run's last task settles it and fires the eval_run.completed webhook exactly once. Poll the returned evrun_… id as above, or subscribe to the webhook — Gate a Canary Promotion on an Eval does the latter.

An empty dataset is rejected in both modes, and a wait: true run over more than 25 items is rejected naming the cap rather than scored partially — wait: false is the answer for a large suite.

Step 6 — Cancel a run mid-flight​

A queued run holds real provider budget. Cancelling drops its outstanding tasks, so it stops spending on the next tick, and settles it canceled — see Evaluations — Canceling a run.

DOOMED_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false | jq -r '.id')

soat cancel-eval-run --eval-id "$EVAL_ID" --eval-run-id "$DOOMED_RUN_ID" \
  | jq '{status, item_count, completed_count, errored_count, aggregate_scores, passed}'

Expected shape:

{
  "status": "canceled",
  "item_count": 2,
  "completed_count": 0,
  "errored_count": 0,
  "aggregate_scores": null,
  "passed": null
}

Results already written are kept and completed_count / errored_count report what ran. aggregate_scores is left null on purpose — a partial roll-up would read as a whole-dataset verdict. A canceled run fires no lifecycle event, and cancelling a run that already finished is a 400.

Next steps​

Eval spend is metered separately from production (source: "eval" / "eval_judge") — see Evaluations — Eval spend.

Read next: Gate a Canary Promotion on an Eval to make a rollout depend on a green suite, and Evaluations — LLM judge for the full contract.