Skip to main content

Grade Structured Output with Your Own Scorer

An agent that returns an object is right when the object has the right shape, the right values, and obeys rules only your systems know. Three scorers check one each: output_schema for shape, json_logic against each item's metadata for values, and a tool scorer running your own code for the rule. Build a refund-triage suite, read its pass rate per kind of request, fix the agent, and measure the fix against the first run.

Assumes Evaluate an Agent.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

See Users.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — An agent that returns an object​

The agent classifies a customer message as refund or question and reads the amount asked for. Its output_schema constrains the model; the parsed object is what the scorers read as object (Agents — Structured Output).

PROJECT_ID=$(soat create-project --name "Refund Scoring" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Refund Triage" \
  --instructions "Classify the customer message. category is refund when the customer asks for money back, otherwise question. refund_amount is the amount the customer asks for, or 0." \
  --output-schema '{"type":"object","required":["category","refund_amount"],"properties":{"category":{"type":"string","enum":["refund","question"]},"refund_amount":{"type":"number"}}}' | jq -r '.id')

echo "AGENT_ID: $AGENT_ID"

Step 3 — Write the scorer service​

The business rule: a refund never exceeds what the customer paid. The price lives in your order system, not in the model's output, so the check is your code. The engine sends it { input, output, object, expected, item } and expects { score, passed?, reasoning? } back (Evaluations — Custom scorers).

The automated tutorial tests inject SOAT_SCORER_BASE_URL so the server container can reach the service. With the SOAT server in Docker and the service on the host, use http://host.docker.internal:8789 instead of localhost.

SCORER_URL="${SOAT_SCORER_BASE_URL:-http://localhost:8789}/score"

node -e '
const http = require("http");
const grade = ({ object, item }) => {
  if (object.category !== "refund") return { score: 1, passed: true, reasoning: "Not a refund." };
  const paid = item.metadata.order_total;
  const ok = object.refund_amount <= paid;
  return { score: ok ? 1 : 0, passed: ok, reasoning: "Refund " + object.refund_amount + " against " + paid + " paid." };
};
http
  .createServer((req, res) => {
    let body = "";
    req.on("data", (chunk) => (body += chunk));
    req.on("end", () => {
      res.setHeader("Content-Type", "application/json");
      if (req.method !== "POST") return res.end("{}");
      try {
        res.end(JSON.stringify(grade(JSON.parse(body))));
      } catch (error) {
        res.statusCode = 400;
        res.end(JSON.stringify({ error: String(error) }));
      }
    });
  })
  .listen(8789);
' > refund-scorer.log 2>&1 &
SCORER_PID=$!
echo "Scorer PID: $SCORER_PID"

# → retry 10
node -e 'require("http").get("http://localhost:8789/health", (r) => process.exit(r.statusCode === 200 ? 0 : 1)).on("error", () => process.exit(1))'

Step 4 — Register it as a tool​

An http tool sends its arguments as the JSON body of a POST (Tools — http). Calling it directly with a refund above the price paid shows the verdict the eval will record (Tools — Calling a Tool Directly).

SCORER_TOOL_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "refund-policy" \
  --type http \
  --description "Checks a refund against the price paid" \
  --execute '{"url":"'"$SCORER_URL"'","method":"POST"}' | jq -r '.id')

soat call-tool --tool-id "$SCORER_TOOL_ID" \
  --input '{"object":{"category":"refund","refund_amount":60},"item":{"metadata":{"order_total":25}}}'
{ "score": 0, "passed": false, "reasoning": "Refund 60 against 25 paid." }

Step 5 — A dataset labelled by kind​

Each item's metadata carries its kind, the category it should get, and the price paid. Scorers read it as item.metadata; the price never reaches the agent (Eval Design — Label each item's kind).

DATASET_ID=$(soat create-dataset --project-id "$PROJECT_ID" --name "refund-requests" | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"Order 1042 cost $40 and arrived broken. Please refund the $40."}]' \
  --metadata '{"kind":"refund","expected_category":"refund","order_total":40}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"I paid $25 for order 1182 and it was late. Refund me $60 for the trouble."}]' \
  --metadata '{"kind":"refund","expected_category":"refund","order_total":25}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"Do you ship to Canada?"}]' \
  --metadata '{"kind":"question","expected_category":"question","order_total":0}' | jq -r '.id'

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role":"user","content":"How long does delivery take?"}]' \
  --metadata '{"kind":"question","expected_category":"question","order_total":0}' | jq -r '.id'

No item carries an expected_output: every scorer here grades the object, not a reference string.

Step 6 — Bind three scorers​

ScorerAsks
output_schemaIs the object well formed? Its own schema adds minimum: 0, stricter than the agent's
json_logicIs category the one the item's metadata expects?
tool (refund-policy)Does the refund stay within the price paid?

An item passes only when all three pass (Evaluations — Pass semantics). The tool scorer's own passed decides its verdict, so it needs no pass_threshold. group_by: kind rolls every run up per metadata.kind too (Evaluations — Grouped aggregates).

EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "refund-triage" \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"output_schema","schema":{"type":"object","properties":{"refund_amount":{"type":"number","minimum":0}}}},{"type":"json_logic","expression":{"==":[{"var":"object.category"},{"var":"item.metadata.expected_category"}]}},{"type":"tool","name":"refund-policy","tool_id":"'"$SCORER_TOOL_ID"'"}]' \
  --pass-threshold 0.75 \
  --group-by kind | jq -r '.id')

soat get-eval --eval-id "$EVAL_ID" | jq '{scorers: .scorers | map(.name // .type), group_by}'

Step 7 — Run it and read each scorer​

The run is queued and polled to completed: four real generations plus a tool call per item can outlast one HTTP request (Sync and Async). Outcomes and aggregates key on the scorer type, or on name for a tool scorer. The tool's reasoning is stored on the result.

BASELINE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false | jq -r '.id')

# → retry 300
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '{passed, completed_count, errored_count, scorers: .aggregate_scores.scorers}'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.data | map({output, error, scores: [.scores[]? | {scorer, passed, reasoning}]})'

The over-limit request, read as asked:

{
  "output": "{\"category\":\"refund\",\"refund_amount\":60}",
  "error": null,
  "scores": [
    { "scorer": "output_schema", "passed": true, "reasoning": null },
    { "scorer": "json_logic", "passed": true, "reasoning": null },
    { "scorer": "refund-policy", "passed": false, "reasoning": "Refund 60 against 25 paid." }
  ]
}

A well-formed, correctly classified object still fails: only your rule knew the price. An object that breaks the agent's own output_schema fails the generation, so the item is an error, not a failure (Evaluations — Errors are not zeros).

Step 8 — Read the pass rate per kind​

The run's aggregate_scores.grouping holds one rollup per kind, computed like the run's own figures (Evaluations — Grouped aggregates).

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq '.aggregate_scores.grouping | {groups: (.groups | map_values({scored_item_count, pass_rate})), ungrouped_item_count}'
{
  "groups": {
    "question": { "scored_item_count": 2, "pass_rate": 0.5 },
    "refund": { "scored_item_count": 2, "pass_rate": 0.5 }
  },
  "ungrouped_item_count": 0
}

The two kinds fail for different reasons: a question read as a refund fails json_logic, an over-limit refund fails refund-policy. One overall pass rate cannot say which moved. When kinds need their own thresholds, give each its own dataset and eval.

Step 9 — Fix the agent and measure the fix​

Tell the agent the rule, then run against the first run as the baseline (Evaluations — Baseline deltas).

soat update-agent --agent-id "$AGENT_ID" \
  --instructions "Classify the customer message. category is refund when the customer asks for money back, otherwise question. refund_amount is the amount the customer asks for, never more than the price they say they paid, or 0." \
  --version-label "caps-refund" | jq '{version}'

CANDIDATE_RUN_ID=$(soat start-eval-run --eval-id "$EVAL_ID" --wait false \
  --baseline-run-id "$BASELINE_RUN_ID" | jq -r '.id')

# → retry 300
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$CANDIDATE_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$CANDIDATE_RUN_ID" \
  | jq '{passed, errored_count, baseline: .aggregate_scores.baseline}'

baseline.scorers carries a refund-policy entry with its own pass_rate_delta, so the rule's movement reads apart from shape and category. baseline.grouping carries the same deltas per kind, so a fix to refunds that broke questions shows as two groups moving apart. The delta can come back negative: qwen2.5:0.5b does not reliably follow the added instruction, and with four items one item is 0.25 of the pass rate. A delta is a fix only once its baseline.p_value says it is not noise; with four items no split can get there, so a real suite needs more (Eval Design — Noise before signal).

What's next​