Skip to main content

Gate a Canary Promotion on an Eval

A promotion gate makes a canary rollout decision evidential: a release names an eval, and promote-agent-release refuses until that eval has a run that finished completed, with passed: true, pinned to the canary version.

You will:

  1. Deploy an agent and its test suite together as a formation — the suite ships with the thing it verifies.
  2. Change the prompt through the same template, archiving a canary candidate.
  3. Start a canary release that names the eval as its promotion_gate.
  4. Watch promotion be refused with PROMOTION_GATE_UNMET.
  5. Watch a green run of the wrong version fail to open the gate — the reason pinning exists.
  6. Produce the run that does open it, and promote.
  7. Add a nightly scheduled run and a webhook, so the gate keeps being fed after you stop watching.

This tutorial assumes you have been through Evaluate an Agent.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide to bring the stack up with Docker Compose.
  • Ollama running locally with qwen2.5:0.5b available. This tutorial uses a local provider so it runs without external credentials — to connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.
  • New to SOAT? Read Key Concepts to understand projects, agents, and evaluations first.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • For production hardening (secrets, env vars), see Configuration.
  • Server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Admin is the built-in superuser role. See Users for authentication details, and Projects for the container everything below lives in.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

PROJECT_ID=$(soat create-project --name "Gated Rollout" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

Step 2 — Ship the agent and its suite in one template​

Datasets, their items, and evals are all formation resource types, so the suite that verifies an agent lives in the same template as the agent. Test cases are their own resource rather than a list inside the dataset — see Memories for the same shape.

cat > release-notes-formation.json << 'EOF'
{
  "resources": {
    "provider": {
      "type": "ai_provider",
      "properties": {
        "name": "Local Ollama",
        "provider": "ollama",
        "default_model": "qwen2.5:0.5b"
      }
    },
    "agent": {
      "type": "agent",
      "properties": {
        "name": "Release Notes Writer",
        "ai_provider_id": { "ref": "provider" },
        "instructions": "Summarize the change in one sentence."
      }
    },
    "suite": {
      "type": "dataset",
      "properties": {
        "name": "release-notes-suite",
        "description": "Cases every release must still pass"
      }
    },
    "emptyFileCase": {
      "type": "dataset_item",
      "properties": {
        "dataset_id": { "ref": "suite" },
        "input": [{ "role": "user", "content": "We fixed a crash when uploading a 0-byte file." }],
        "expected_output": "Fixed a crash when uploading an empty file."
      }
    },
    "passkeyCase": {
      "type": "dataset_item",
      "properties": {
        "dataset_id": { "ref": "suite" },
        "input": [{ "role": "user", "content": "Login now supports passkeys." }],
        "expected_output": "Added passkey support to login."
      }
    },
    "gate": {
      "type": "eval",
      "properties": {
        "name": "release-notes-gate",
        "agent_id": { "ref": "agent" },
        "dataset_id": { "ref": "suite" },
        "scorers": [
          { "type": "json_logic", "expression": { "!=": [{ "var": "output" }, ""] } }
        ],
        "pass_threshold": 1
      }
    }
  },
  "outputs": {
    "agent_id": { "ref": "agent" },
    "eval_id": { "ref": "gate" },
    "dataset_id": { "ref": "suite" }
  }
}
EOF

TEMPLATE=$(cat release-notes-formation.json)

STACK=$(soat create-formation \
  --project-id "$PROJECT_ID" \
  --name "release-notes-stack" \
  --template "$TEMPLATE")

FORMATION_ID=$(printf '%s' "$STACK" | jq -r '.id')
AGENT_ID=$(printf '%s' "$STACK" | jq -r '.outputs.agent_id')
EVAL_ID=$(printf '%s' "$STACK" | jq -r '.outputs.eval_id')

printf '%s' "$STACK" | jq '{status, outputs}'

The gate's only scorer asserts the agent answered at all. That is deliberately weak for a tutorial — it keeps the gate's mechanism the thing under observation rather than what a 0.5B-parameter model happens to write. A real gate uses the scorers from Evaluate an Agent and a judge from Judge Open-Ended Answers.

Step 3 — Change the prompt through the template​

Version snapshots are written by the shared business-logic layer, not by the REST handlers, so a formation apply archives a version exactly as a PUT would — see Agents — Versioning and Staged Rollout. Editing through the template keeps the formation the source of truth — an out-of-band update-agent on a formation-managed agent is drift the next apply will undo.

CANDIDATE_TEMPLATE=$(printf '%s' "$TEMPLATE" | jq \
  '.resources.agent.properties.instructions = "Summarize the change in one sentence, in past tense, starting with a verb."')

soat update-formation --formation-id "$FORMATION_ID" --template "$CANDIDATE_TEMPLATE" | jq '{status}'

soat list-agent-versions --agent-id "$AGENT_ID" | jq '.data | map({version, instructions: .config.instructions})'

Version 2 now exists in history. No traffic serves it yet.

Step 4 — Start a gated canary release​

--promotion-gate is the only new part of an otherwise ordinary canary release (Agents — Staged Rollout): 20% of traffic on version 2, and the eval that must go green before version 2 can become everyone's.

soat set-agent-release --agent-id "$AGENT_ID" \
  --stable-version 1 --canary-version 2 --canary-percent 20 \
  --promotion-gate "$EVAL_ID" | jq '{active_release}'

Expected output:

{
  "active_release": {
    "stable_version": 1,
    "canary_version": 2,
    "canary_percent": 20,
    "promotion_gate": "eval_rgMz0oEpbT3oWEK9"
  }
}

Step 5 — Promotion is refused until there is evidence​

With no qualifying run on record, promote is a 409 and changes nothing. See Agents — Eval-gated promotion.

# → expect-fail
soat promote-agent-release --agent-id "$AGENT_ID"

Expected output — a 409, with the rollout left running untouched:

{
  "code": "PROMOTION_GATE_UNMET",
  "meta": { "promotion_gate": "eval_rgMz0oEpbT3oWEK9", "agent_version": 2 }
}

Step 6 — A green run of the wrong version does not count​

Run the eval without agent_version. During an active release an unpinned run uses the active release's stable version — so this measures version 1, the config you are trying to replace.

STABLE_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true)
printf '%s' "$STABLE_RUN" | jq '{agent_version, status, passed}'

# → expect-fail
soat promote-agent-release --agent-id "$AGENT_ID"

A passing run, and the gate stays shut: a green run against another version is not evidence about the canary. A run resolves exactly one version at start, stamps it on agent_version, and every item executes against it — see Evaluations — Version pinning for why unpinned runs resolve to the stable version.

Step 7 — Produce the run that opens the gate​

Same eval, same scorers — this time pinned to the canary version, which is what Evaluations — Version pinning exists for.

CANARY_RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true --agent-version 2)
printf '%s' "$CANARY_RUN" | jq '{id, agent_version, status, passed}'

soat promote-agent-release --agent-id "$AGENT_ID" | jq '{version, active_release, instructions}'

soat list-agent-versions --agent-id "$AGENT_ID" | jq '.data | map({version, eval_run_id})'

Expected output — the promoted version records which run cleared it:

[
  { "version": 2, "eval_run_id": "evrun_b2vkVf4zeRG23Zic" },
  { "version": 1, "eval_run_id": null }
]

That eval_run_id field is the audit trail: "why was version 2 promoted?" has an answer with per-item scores behind it.

The gate cannot be argued with, only satisfied

abort-agent-release is ungated — rolling back to the stable config is always allowed. Only promotion needs evidence, which is the asymmetry you want under pressure.

Step 8 — Keep feeding the gate after you stop watching​

A trigger with target_type: "eval" runs the suite on a cadence — the nightly regression nobody has to remember to start. Declare it in the same template, and subscribe a webhook to the verdict.

Creating an eval-target trigger requires evaluations:RunEval on top of triggers:CreateTrigger: a trigger may only start what its creator could start directly.

WEBHOOK_ID=$(soat create-webhook --project-id "$PROJECT_ID" \
  --name "eval-verdicts" \
  --url "http://127.0.0.1:9/eval-verdicts" \
  --events '["eval_run.completed","eval_run.failed"]' | jq -r '.id')

NIGHTLY_TEMPLATE=$(printf '%s' "$CANDIDATE_TEMPLATE" | jq \
  '.resources.nightly = {"type":"trigger","properties":{"name":"nightly-release-notes-gate","type":"schedule","target_type":"eval","target_id":{"ref":"gate"},"cron":"0 3 * * *"}} | .outputs.trigger_id = {"ref":"nightly"}')

TRIGGER_ID=$(soat update-formation --formation-id "$FORMATION_ID" \
  --template "$NIGHTLY_TEMPLATE" | jq -r '.outputs.trigger_id')

echo "TRIGGER_ID: $TRIGGER_ID"

Rather than waiting until 03:00, fire it now. A firing always starts a queued run (see sync vs async); its result.result_id is the evrun_… to poll.

FIRING=$(soat fire-trigger --trigger-id "$TRIGGER_ID")
printf '%s' "$FIRING" | jq '{status, result}'

NIGHTLY_RUN_ID=$(printf '%s' "$FIRING" | jq -r '.result.result_id')

# → retry 180
soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$NIGHTLY_RUN_ID" | jq -e '.status == "completed"'

soat get-eval-run --eval-id "$EVAL_ID" --eval-run-id "$NIGHTLY_RUN_ID" \
  | jq '{status, passed, agent_version, trigger_id}'

# → retry 30
soat list-webhook-deliveries --webhook-id "$WEBHOOK_ID" | jq -e '.data[0].event_type == "eval_run.completed"'

soat list-webhook-deliveries --webhook-id "$WEBHOOK_ID" | jq '.data[0].payload.data'

The run records where it came from in trigger_id, and keeps it even if that trigger is later deleted. The delivered payload carries the verdict inline:

{
  "eval_id": "eval_rgMz0oEpbT3oWEK9",
  "eval_run_id": "evrun_g5i4CdIFvcOvRZR8",
  "passed": true,
  "aggregate_scores": {
    "scorers": { "json_logic": { "mean": 1, "pass_rate": 1 } },
    "pass_rate": 1,
    "scored_item_count": 2
  }
}

Exactly one event fires per terminal run, so anything automating the next step can act on this payload alone.

The URL above is deliberately unroutable

http://127.0.0.1:9/… cannot accept a POST, so the delivery attempt fails and retries — which is exactly why it is useful here: the delivery record still carries the event type and full payload, so you can inspect what SOAT sent without standing up a listener. Point this at a real endpoint and verify the signature; see Webhooks.

The trigger's input may also carry agent_version and baseline_run_id, which are passed to every run it starts. Both are validated at fire time, so a nightly schedule naming a version that no longer exists fails the firing — with the reason on the firing record — instead of creating a run that could never execute.

Next steps​

Read next: Agent Versioning and Canary Rollout for the rollout mechanics this builds on, Formations for declarative stacks, and Evaluations for the module reference.