Skip to main content

Grade an Eval with a Decider

A support agent drafts replies; a decider named reply-review asks three questions about each one: does it resolve the issue, does it respect policy, how warm is it. The same decider reviews a live reply before it is sent and grades every item of an eval as a decider scorer, so the eval measures exactly what production enforces.

The decider is answered by TypeSafe Jev, a model that returns calibrated probabilities instead of text, through a pipeline tool that bridges its answer shape to the decider contract.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047   # CLI, SDK, and curl — do NOT append /api/v1

# The default is the real endpoint; overridable so the tutorial also runs against a mock.
export TYPESAFE_BASE_URL="${TYPESAFE_BASE_URL:-https://api.typesafe.ai}"
export TYPESAFE_API_KEY="${TYPESAFE_API_KEY:-apikey_your-typesafe-key}"

Step 1 — Log in as admin​

See Users for authentication.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create a project​

PROJECT_ID=$(soat create-project --name "support-desk" | jq -r '.id')
echo "Project: $PROJECT_ID"

Step 3 — Reach Jev from a tool​

The key goes in a secret, and the http tool names it by reference, so no read of the tool returns it. model is pinned in preset_parameters: every call asks jev-latest.

SECRET_ID=$(soat create-secret --project-id "$PROJECT_ID" \
  --name typesafe-api-key --value "$TYPESAFE_API_KEY" | jq -r '.id')

JEV_TOOL_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name jev-systemone \
  --type http \
  --description "TypeSafe Jev evaluation endpoint" \
  --parameters '{"type":"object","properties":{"state":{},"questions":{"type":"object"}},"required":["state","questions"]}' \
  --execute '{
    "url": "'"$TYPESAFE_BASE_URL"'/v1/systemone",
    "method": "POST",
    "headers": {"Authorization": "Bearer {{secret:'"$SECRET_ID"'}}"}
  }' \
  --preset-parameters '{"model": "jev-latest"}' | jq -r '.id')
echo "Jev tool: $JEV_TOOL_ID"

Step 4 — Bridge Jev's answers to the decider contract​

A decider calls its tool with { input, questions } and reads back { answers: [...] }. Jev speaks a neighbouring shape, and a pipeline tool translates both ways (Bridging an engine with a pipeline):

JevDecider contractBridge
{ state, questions }, questions keyed by id{ input, questions }, questions an arraystate = input; each question read by position
noul question, answered as noul, a probabilitypredicate, answered as probabilitytype: "noul" in; probability = noul out
criteria, a list of level labelslevels, a list of { label, description }criteria = each level's label
score is the probability-weighted level (1.9)score is a level index (2)rounded: (s + 0.5) - ((s + 0.5) % 1)
probabilities keyed by level indexprobabilities, a list of { value, probability }one entry per level
legend and confidence beside every answernot readthe output names each field, which leaves them behind
BRIDGE_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name jev-reply-review \
  --type pipeline \
  --description "Answers reply-review through Jev" \
  --pipeline '{
    "steps": [{
      "id": "jev",
      "tool_id": "'"$JEV_TOOL_ID"'",
      "input": {
        "state": {"var": "input.input"},
        "questions": {
          "resolves_issue": {"type": "noul", "instructions": {"var": "input.questions.0.instructions"}},
          "respects_policy": {"type": "noul", "instructions": {"var": "input.questions.1.instructions"}},
          "tone": {
            "type": "score",
            "instructions": {"var": "input.questions.2.instructions"},
            "criteria": {"map": [{"var": "input.questions.2.levels"}, {"var": "label"}]}
          }
        }
      }
    }],
    "output": {
      "answers": [
        {"name": "resolves_issue", "probability": {"var": "steps.jev.answers.resolves_issue.noul"}},
        {"name": "respects_policy", "probability": {"var": "steps.jev.answers.respects_policy.noul"}},
        {
          "name": "tone",
          "score": {"-": [
            {"+": [{"var": "steps.jev.answers.tone.score"}, 0.5]},
            {"%": [{"+": [{"var": "steps.jev.answers.tone.score"}, 0.5]}, 1]}
          ]},
          "probabilities": [
            {"value": 0, "probability": {"var": "steps.jev.answers.tone.probabilities.0"}},
            {"value": 1, "probability": {"var": "steps.jev.answers.tone.probabilities.1"}},
            {"value": 2, "probability": {"var": "steps.jev.answers.tone.probabilities.2"}}
          ]
        }
      ]
    }
  }' | jq -r '.id')
echo "Bridge: $BRIDGE_ID"

The bridge is written for one question set: it reads each question by its position and answers each by name, and tone has three levels. A question added, moved or given another level count fails its decisions until the bridge follows.

Step 5 — Create the decider​

The questions are an array, in the order the bridge reads them. Jev accepts at most 10 levels on a score, fewer than a decider allows, so keep scores to 10 levels or less.

DECIDER_ID=$(soat create-decider \
  --project-id "$PROJECT_ID" \
  --name reply-review \
  --tool-id "$BRIDGE_ID" \
  --questions '[
    {
      "type": "predicate",
      "name": "resolves_issue",
      "instructions": "Does the `reply` resolve what the `customer` asked?"
    },
    {
      "type": "predicate",
      "name": "respects_policy",
      "instructions": "Does the `reply` avoid promising refunds, credits or dates the agent cannot grant?"
    },
    {
      "type": "score",
      "name": "tone",
      "instructions": "How warm is the `reply`?",
      "levels": [
        {"label": "Cold", "description": "Curt or dismissive."},
        {"label": "Neutral", "description": "Polite but impersonal."},
        {"label": "Warm", "description": "Friendly and empathetic."}
      ]
    }
  ]' | jq -r '.id')
echo "Decider: $DECIDER_ID"

Step 6 — Review a live reply​

In production the decider sits in the loop: before a reply is sent, ask it with POST /api/v1/deciders/{decider_id}/decisions. The input names customer and reply, the fields the instructions point at in backticks.

soat create-decision \
  --decider-id "$DECIDER_ID" \
  --input '{
    "customer": "I was charged twice for order 1042.",
    "reply": "Sorry about that! I have flagged the duplicate charge for our billing team, who will reverse it."
  }' \
  --metadata '{"ticket_id": "ZD-1042"}' \
  --wait true | jq '{status, decider_version, answers}'

Expected output against the mock (Jev's own figures vary):

{
  "status": "completed",
  "decider_version": 1,
  "answers": [
    { "type": "predicate", "name": "resolves_issue", "probability": 0.85 },
    { "type": "predicate", "name": "respects_policy", "probability": 0.85 },
    {
      "type": "score",
      "name": "tone",
      "score": 2,
      "probabilities": [
        { "value": 0, "label": "Cold", "probability": 0 },
        { "value": 1, "label": "Neutral", "probability": 0.1 },
        { "value": 2, "label": "Warm", "probability": 0.9 }
      ]
    }
  ]
}

tone.probabilities is Jev's distribution over the levels, labelled from the decider's own levels: the certainty a bare score: 2 drops. The same answers are keyed by name in answers_by_name.

Step 7 — The agent under test and its dataset​

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Support Replier" \
  --instructions "Reply to the customer in two sentences. Never promise a refund; say billing will review it." \
  | jq -r '.id')

DATASET_ID=$(soat create-dataset --project-id "$PROJECT_ID" --name support-replies | jq -r '.id')

soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role": "user", "content": "I was charged twice for order 1042."}]' > /dev/null
soat create-dataset-item --dataset-id "$DATASET_ID" \
  --input '[{"role": "user", "content": "My invoice shows the wrong company name."}]' > /dev/null
echo "Agent: $AGENT_ID  Dataset: $DATASET_ID"

Step 8 — Bind the decider as a scorer and run​

input maps each item into the shape the questions read; score reads the decision's answers by name. Averaging the two probability values keeps Jev's certainty in the score: a reply Jev is unsure about scores lower than one it is sure about, even when both predicates hold.

EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name reply-quality \
  --agent-id "$AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{
    "type": "decider",
    "name": "reply_review",
    "decider_id": "'"$DECIDER_ID"'",
    "input": {"customer": {"var": "input.0.content"}, "reply": {"var": "output"}},
    "score": {"/": [{"+": [
      {"var": "answers_by_name.resolves_issue.probability"},
      {"var": "answers_by_name.respects_policy.probability"}
    ]}, 2]},
    "pass_threshold": 0.7
  }]' \
  --pass-threshold 0.8 | jq -r '.id')

BASELINE_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true)
BASELINE_RUN_ID=$(echo "$BASELINE_RUN" | jq -r '.id')
echo "$BASELINE_RUN" | jq '{status, passed, decider_versions, reply_review: .aggregate_scores.scorers.reply_review}'

Expected output against the mock:

{
  "status": "completed",
  "passed": true,
  "decider_versions": { "reply_review": 1 },
  "reply_review": { "mean": 0.85, "pass_rate": 1 }
}

A failed decision, or a score outside 0–1, errors the item instead of scoring it 0 (errors are not zeros).

Step 9 — Read the decision behind an item​

Each score carries the decision_id that produced it; the decision's metadata points back at the eval, the run and the item.

ITEM_DECISION_ID=$(soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$BASELINE_RUN_ID" \
  | jq -r '.data[0].scores[0].decision_id')

soat get-decision --decision-id "$ITEM_DECISION_ID" \
  | jq '{decider_version, tone: .answers_by_name.tone.score, run: .metadata.eval_run_id}'

Step 10 — Change the criteria, see the pin​

Rewording a question archives a new decider version. A run grades every item under the version it started with and names it in decider_versions, so comparing against the baseline says which criteria each run used.

soat update-decider --decider-id "$DECIDER_ID" --questions '[
  {"type": "predicate", "name": "resolves_issue", "instructions": "Does the `reply` resolve what the `customer` asked, or name who will?"},
  {"type": "predicate", "name": "respects_policy", "instructions": "Does the `reply` avoid promising refunds, credits or dates the agent cannot grant?"},
  {"type": "score", "name": "tone", "instructions": "How warm is the `reply`?", "levels": [
    {"label": "Cold", "description": "Curt or dismissive."},
    {"label": "Neutral", "description": "Polite but impersonal."},
    {"label": "Warm", "description": "Friendly and empathetic."}
  ]}
]' | jq '{version}'

soat start-eval-run --eval-id "$EVAL_ID" --baseline-run-id "$BASELINE_RUN_ID" --wait true \
  | jq '{decider_versions, reply_review: .aggregate_scores.scorers.reply_review}'

Expected output against the mock:

{ "version": 2 }
{
  "decider_versions": { "reply_review": 2 },
  "reply_review": { "mean": 0.85, "pass_rate": 1 }
}

The baseline says reply_review: 1, the candidate reply_review: 2: a delta between them measures the agent and the criteria at once. Re-run the baseline under the new version before reading the delta as the agent's.

Next Steps​