Skip to main content

Judge Open-Ended Answers

exact_match and contains need a known right string; summaries, explanations and rewrites have many correct forms. An llm_judge scorer grades the answer with a tool-less model completion through the same AI providers path, returning a 0–1 score plus reasoning.

Bind an llm_judge scorer next to a deterministic one, read each item's score and reasoning, see why an unparseable verdict is an error rather than a zero, check the judge against your own grades, run the eval queued and poll, and cancel a run mid-flight. Assumes Evaluate an Agent.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

See Users.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — An agent with open-ended output​

A support-reply drafter: no single correct draft exists. See Agents and Evaluations — Dataset.

PROJECT_ID=$(soat create-project --name "Judge Workshop" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Reply Drafter" \
  --instructions "Draft a short, warm reply to the customer message. Two sentences at most." | jq -r '.id')

DATASET_ID=$(soat create-dataset --project-id "$PROJECT_ID" --name "reply-drafts" | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"My package arrived damaged. What now?"}]' \
  --expected-output "Apologize, then offer a replacement or a refund and ask for a photo of the damage." | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"I was charged twice this month."}]' \
  --expected-output "Apologize, confirm the duplicate charge will be refunded, and give the expected timeline." | jq -r '.id'

Step 3 — Bind the judge​

The judge's prompt has three per-item slots: {{input}}, {{output}}, {{expected}} (Evaluations — LLM judge). pass_threshold on the scorer is required, no default. Keep a deterministic scorer alongside as a structural floor.

model pins the judge, so a change to the provider's default model cannot move the scores. The prompt carries a rubric saying what each score means; a judge asked for "a score" drifts toward the top (Eval Design — Calibrating a judge).

EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "reply-quality" \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"json_logic","expression":{"!=":[{"var":"output"},""]}},{"type":"llm_judge","ai_provider_id":"'"$AI_PROVIDER_ID"'","model":"qwen2.5:0.5b","prompt":"You grade a customer support draft against a reference answer. Score 1 when the draft apologizes and offers the fix the reference names, 0.5 when it offers the fix without apologizing, 0 when it offers no fix. Reply with only JSON: {\"score\": <1, 0.5 or 0>, \"reasoning\": \"<one sentence>\"}. Customer message: {{input}} Draft reply: {{output}} Reference answer: {{expected}}","pass_threshold":0.7}]' \
  --pass-threshold 0.5 | jq -r '.id')

soat get-eval --eval-id "$EVAL_ID" | jq '.scorers | map({type, pass_threshold})'

Slots fill in one pass: a value containing {{output}} is never re-expanded, and an unrecognized {{…}} is left as written.

Step 4 — Run it and read the reasoning​

RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true | jq -r '.id')

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '{status, passed, completed_count, errored_count, aggregate_scores}'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '.data | map({output, error, scores})'

A judged item (the score gates; reasoning is stored for audit):

{
  "output": "I am sorry your package arrived damaged. Send us a photo and we will ship a replacement right away.",
  "error": null,
  "scores": [
    { "scorer": "json_logic", "score": 1, "passed": true },
    {
      "scorer": "llm_judge",
      "score": 0.9,
      "passed": true,
      "reasoning": "Apologizes, asks for a photo, and offers a replacement."
    }
  ]
}
A small judge model may not answer in JSON

The judge must reply with a JSON object carrying a numeric score between 0 and 1. qwen2.5:0.5b often ignores that, so items come back with error set and no scores. An ungraded answer is an error, never a 0; an out-of-range score is rejected, not clamped. Errored items are excluded from aggregate_scores and counted in errored_count. Use a real judge model (Connect Third-Party LLMs) for a suite you gate on.

Re-run the baseline when you change the judge's model. The judge's ai_provider_id must belong to the eval's project; the project's default model route applies when the scorer pins none.

Step 5 — Check the judge against your own grades​

A judge's pass_threshold means nothing until the judge agrees with your own verdicts (Eval Design — Calibrating a judge). Export each draft with the judge's verdict, record yours next to it, and count the agreement.

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '[.data[] | {output, judge_passed: ([.scores[]? | select(.scorer == "llm_judge") | .passed] | first), human_passed: null}]' \
  > judge-grades.json

jq '.[0].human_passed = true | .[1].human_passed = false' judge-grades.json > judge-graded.json

jq '{
  graded: map(select(.judge_passed != null)) | length,
  agreed: map(select(.judge_passed != null and .judge_passed == .human_passed)) | length,
  disagreements: map(select(.judge_passed != null and .judge_passed != .human_passed))
}' judge-graded.json
{
  "graded": 2,
  "agreed": 1,
  "disagreements": [
    {
      "output": "We apologize for any inconvenience and appreciate your understanding.",
      "judge_passed": true,
      "human_passed": false
    }
  ]
}

The second command stands in for your review: set human_passed on each draft to your own verdict. Items the judge could not grade (Step 4) carry judge_passed: null and are left out of the count. On each disagreement, read the judge's reasoning, then fix the rubric or move pass_threshold with update-eval and run again. Grade 20–30 drafts, borderline ones included, before gating on the judge.

Step 6 — Run it queued instead of blocking​

A judged suite makes two provider calls per item. wait selects the mode; both share one execution path, so runs are comparable.

waitBehavior
trueExecutes items sequentially in-process, returns the run terminal with its scores. Capped at 25 items.
false (default)Enqueues one task per item, returns immediately with status: "queued". No item cap.
QUEUED_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false)
QUEUED_RUN_ID=$(printf '%s' "$QUEUED_RUN" | jq -r '.id')

printf '%s' "$QUEUED_RUN" | jq '{status, item_count, aggregate_scores}'

# → retry 180
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$QUEUED_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$QUEUED_RUN_ID" \
  | jq '{status, passed, completed_count, errored_count, pass_rate: .aggregate_scores.pass_rate}'

Each item is one queued task; the worker draining the last task settles the run and fires the eval_runs.completed webhook once. Poll the evrun_… id or subscribe to the webhook (Gate a Canary Promotion on an Eval).

An empty dataset is rejected in both modes; a wait: true run over 25 items is rejected naming the cap.

Step 7 — Cancel a run mid-flight​

Cancelling drops the run's outstanding tasks and settles it cancelled (Evaluations — Canceling a run).

DOOMED_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false | jq -r '.id')

soat cancel-eval-run --eval-id "$EVAL_ID" --eval-run-id "$DOOMED_RUN_ID" \
  | jq '{status, item_count, completed_count, errored_count, aggregate_scores, passed}'

Expected shape:

{
  "status": "cancelled",
  "item_count": 2,
  "completed_count": 0,
  "errored_count": 0,
  "aggregate_scores": null,
  "passed": null
}

Written results are kept; completed_count / errored_count report what ran; aggregate_scores stays null. A cancelled run fires no lifecycle event; cancelling a finished run is a 400.

Next steps​