Skip to main content

Gate a Canary Promotion on an Eval

A promotion gate names an eval on a canary release; promote-agent-release refuses until that eval has a run that finished completed, passed: true, pinned to the canary version.

You will deploy an agent and its suite as one formation, change the prompt through the template, start a gated canary release, see PROMOTION_GATE_UNMET, see a green run of the wrong version fail to open the gate, produce the run that does, promote, then add a nightly scheduled run and a webhook.

Assumes Evaluate an Agent.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

See Users and Projects.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

PROJECT_ID=$(soat create-project --name "Gated Rollout" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

Step 2 — Ship the agent and its suite in one template​

Datasets, dataset items, and evals are formation resource types, so the suite lives in the agent's template. Test cases are their own resource, not a list inside the dataset (same shape as Memories).

cat > release-notes-formation.json << 'EOF'
{
  "resources": {
    "provider": {
      "type": "ai_provider",
      "properties": {
        "name": "Local Ollama",
        "provider": "ollama",
        "default_model": "qwen2.5:0.5b"
      }
    },
    "agent": {
      "type": "agent",
      "properties": {
        "name": "Release Notes Writer",
        "ai_provider_id": { "ref": "provider" },
        "instructions": "Summarize the change in one sentence."
      }
    },
    "suite": {
      "type": "dataset",
      "properties": {
        "name": "release-notes-suite",
        "description": "Cases every release must still pass"
      }
    },
    "emptyFileCase": {
      "type": "dataset_item",
      "properties": {
        "dataset_id": { "ref": "suite" },
        "input": [{ "role": "user", "content": "We fixed a crash when uploading a 0-byte file." }],
        "expected_output": "Fixed a crash when uploading an empty file."
      }
    },
    "passkeyCase": {
      "type": "dataset_item",
      "properties": {
        "dataset_id": { "ref": "suite" },
        "input": [{ "role": "user", "content": "Login now supports passkeys." }],
        "expected_output": "Added passkey support to login."
      }
    },
    "gate": {
      "type": "eval",
      "properties": {
        "name": "release-notes-gate",
        "agent_id": { "ref": "agent" },
        "dataset_id": { "ref": "suite" },
        "scorers": [
          { "type": "json_logic", "expression": { "!=": [{ "var": "output" }, ""] } }
        ],
        "pass_threshold": 1
      }
    }
  },
  "outputs": {
    "agent_id": { "ref": "agent" },
    "eval_id": { "ref": "gate" },
    "dataset_id": { "ref": "suite" }
  }
}
EOF

TEMPLATE=$(cat release-notes-formation.json)

STACK=$(soat create-formation \
  --project-id "$PROJECT_ID" \
  --name "release-notes-stack" \
  --template "$TEMPLATE")

FORMATION_ID=$(printf '%s' "$STACK" | jq -r '.id')
AGENT_ID=$(printf '%s' "$STACK" | jq -r '.outputs.agent_id')
EVAL_ID=$(printf '%s' "$STACK" | jq -r '.outputs.eval_id')

printf '%s' "$STACK" | jq '{status, outputs}'

The only scorer asserts the agent answered at all, so the gate mechanism, not the 0.5B model's output, is under observation. A real gate uses the scorers from Evaluate an Agent and a judge from Judge Open-Ended Answers.

Step 3 — Change the prompt through the template​

A formation apply archives a version exactly as a PUT would (Agents — Versioning and Staged Rollout). An out-of-band update-agent on a formation-managed agent is drift the next apply undoes.

CANDIDATE_TEMPLATE=$(printf '%s' "$TEMPLATE" | jq \
  '.resources.agent.properties.instructions = "Summarize the change in one sentence, in past tense, starting with a verb."')

soat update-formation --formation-id "$FORMATION_ID" --template "$CANDIDATE_TEMPLATE" | jq '{status}'

soat list-agent-versions --agent-id "$AGENT_ID" | jq '.data | map({version, instructions: .config.instructions})'

Version 2 now exists in history. No traffic serves it yet.

Step 4 — Start a gated canary release​

An ordinary canary release (Agents — Staged Rollout), 20% of traffic on version 2, plus --promotion-gate: the eval that must go green before promotion.

soat set-agent-release --agent-id "$AGENT_ID" \
  --stable-version 1 --canary-version 2 --canary-percent 20 \
  --promotion-gate "$EVAL_ID" | jq '{active_release}'

Expected output:

{
  "active_release": {
    "stable_version": 1,
    "canary_version": 2,
    "canary_percent": 20,
    "promotion_gate": "eval_rgMz0oEpbT3oWEK9"
  }
}

Step 5 — Promotion is refused until there is evidence​

With no qualifying run, promote is a 409 and changes nothing (Agents — Eval-gated promotion).

# → expect-fail
soat promote-agent-release --agent-id "$AGENT_ID"

Expected output (the rollout keeps running):

{
  "code": "PROMOTION_GATE_UNMET",
  "meta": { "promotion_gate": "eval_rgMz0oEpbT3oWEK9", "agent_version": 2 }
}

Step 6 — A green run of the wrong version does not count​

Run the eval without agent_version. During an active release an unpinned run uses the stable version, so this measures version 1.

STABLE_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true)
printf '%s' "$STABLE_RUN" | jq '{agent_version, status, passed}'

# → expect-fail
soat promote-agent-release --agent-id "$AGENT_ID"

The run passes and the gate stays shut: a green run against another version is not evidence about the canary. A run resolves one version at start, stamps it on agent_version, and every item executes against it (Evaluations — Version pinning).

Step 7 — Produce the run that opens the gate​

Same eval, pinned to the canary version (Evaluations — Version pinning).

CANARY_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true --agent-version 2)
printf '%s' "$CANARY_RUN" | jq '{id, agent_version, status, passed}'

soat promote-agent-release --agent-id "$AGENT_ID" | jq '{version, active_release, instructions}'

soat list-agent-versions --agent-id "$AGENT_ID" | jq '.data | map({version, eval_run_id})'

Expected output (the promoted version records which run cleared it):

[
  { "version": 2, "eval_run_id": "evrun_b2vkVf4zeRG23Zic" },
  { "version": 1, "eval_run_id": null }
]

eval_run_id is the audit trail, with per-item scores behind it.

note

abort-agent-release is ungated: rolling back to the stable config is always allowed. Only promotion needs evidence.

Step 8 — Keep feeding the gate after you stop watching​

A trigger with target_type: "eval" runs the suite on a cadence. Declare it in the same template and subscribe a webhook to the verdict. Creating an eval-target trigger requires evaluations:RunEval on top of triggers:CreateTrigger.

WEBHOOK_ID=$(soat create-webhook --project-id "$PROJECT_ID" \
  --name "eval-verdicts" \
  --url "http://127.0.0.1:9/eval-verdicts" \
  --events '["eval_runs.completed","eval_runs.failed"]' | jq -r '.id')

NIGHTLY_TEMPLATE=$(printf '%s' "$CANDIDATE_TEMPLATE" | jq \
  '.resources.nightly = {"type":"trigger","properties":{"name":"nightly-release-notes-gate","type":"schedule","target_type":"eval","target_id":{"ref":"gate"},"cron":"0 3 * * *"}} | .outputs.trigger_id = {"ref":"nightly"}')

TRIGGER_ID=$(soat update-formation --formation-id "$FORMATION_ID" \
  --template "$NIGHTLY_TEMPLATE" | jq -r '.outputs.trigger_id')

echo "TRIGGER_ID: $TRIGGER_ID"

Fire it now. A firing always starts a queued run (sync vs async); result.result_id is the evrun_… to poll.

FIRING=$(soat fire-trigger --trigger-id "$TRIGGER_ID")
printf '%s' "$FIRING" | jq '{status, result}'

NIGHTLY_RUN_ID=$(printf '%s' "$FIRING" | jq -r '.result.result_id')

# → retry 180
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$NIGHTLY_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$NIGHTLY_RUN_ID" \
  | jq '{status, passed, agent_version, trigger_id}'

# → retry 30
soat list-webhook-deliveries --webhook-id "$WEBHOOK_ID" | jq -e '.data[0].event_type == "eval_runs.completed"'

soat list-webhook-deliveries --webhook-id "$WEBHOOK_ID" | jq '.data[0].payload.data'

The run records its origin in trigger_id, kept even if the trigger is deleted. The delivered payload is the eval run, verdict included (abridged):

{
  "id": "evrun_g5i4CdIFvcOvRZR8",
  "eval_id": "eval_rgMz0oEpbT3oWEK9",
  "status": "completed",
  "passed": true,
  "aggregate_scores": {
    "scorers": { "json_logic": { "mean": 1, "pass_rate": 1 } },
    "pass_rate": 1,
    "scored_item_count": 2
  }
}

Exactly one event fires per terminal run.

The URL above is unroutable

http://127.0.0.1:9/… cannot accept a POST, so delivery fails and retries, but the delivery record still carries the event type and full payload. Point it at a real endpoint and verify the signature; see Webhooks.

The trigger's input may also carry agent_version and baseline_run_id, passed to every run it starts and validated at fire time: a schedule naming a version that no longer exists fails the firing, with the reason on the firing record.

Next steps​