Skip to main content

Orchestrations

DAG-based pipeline definitions for chaining agents, tools, and knowledge lookups into repeatable pipelines.

Orchestration or workflow?

An orchestration is a pipeline that ends — a directed acyclic graph that starts, flows forward through its nodes, and terminates. A workflow is a state graph a task lives in — a long-lived entity that moves between named states over days or weeks, including backward.

You want…Use
A deterministic, forward-only sequence of steps that runs and completesOrchestration (this module)
Statuses, transitions, guards, a kanban board, or an entity that revisits statesWorkflows

The two compose: when a task enters a state, it may dispatch an orchestration (or an agent) to do that state's work. See Workflows & Tasks.

Overview

Orchestrations let you describe a directed acyclic graph (DAG) of nodes where each node performs a discrete operation. Nodes in the same execution round run in parallel; edges with activation groups control fan-in convergence.

Use orchestrations when you know the exact steps in advance and want deterministic, auditable execution — not when you need an LLM to decide which steps to take. An agent node inside an orchestration can still use LLM reasoning internally, but the orchestration graph itself is deterministic. See it end to end in Orchestrate a Sonnet - Step 6 (Create the orchestration graph).

See the Permissions Reference for the IAM action strings for this module.

An orchestration can also be declared as a Formation resource, letting you deploy a team of agents together with the flow that coordinates them as a single stack — see the Agent Squad example.

To run an orchestration automatically — on a cron schedule, in response to an inbound webhook, or on demand — bind it to a Trigger with target_type: orchestration.

Data Model

Orchestration

FieldTypeDescription
idstringPublic ID (orch_ prefix)
project_idstringOwning project
namestringHuman-readable name
descriptionstring | nullOptional description
nodesarrayOrdered list of node definitions
edgesarrayDirected connections between nodes
state_schemaobjectOptional JSON Schema describing the run state
input_schemaobjectOptional JSON Schema describing the run input
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

OrchestrationRun

FieldTypeDescription
idstringPublic ID (orch_run_ prefix)
orchestration_idstringParent orchestration
project_idstringOwning project
statusstringqueued | running | sleeping | awaiting_input | succeeded | failed | cancelled | expired
stateobjectCurrent mutable execution state
active_nodesarrayNode IDs awaiting input or a scheduled wake (populated when awaiting_input, or sleeping while parked on a delay/poll wait)
artifactsobjectOutputs keyed by node ID
errorobject | nullError details if failed
node_executionsarrayPer-node execution records (see Node Executions)
usageobjectToken/cost roll-up (total_input_tokens, total_output_tokens, total_cached_tokens, total_reasoning_tokens, total_cost_usd) summed across every metered generation the run produced (see Run usage). Present on the single-run read; omitted from run list responses
required_actionobject | nullPresent when status is awaiting_input (see Human Nodes)
trace_idstring | nullLinked observability trace, if any
inputobject | nullInitial input provided at run creation
outputobject | nullTerminal node artifact(s) when the run has succeeded
started_atstring | nullISO 8601 execution start timestamp
completed_atstring | nullISO 8601 terminal timestamp (succeeded/failed/cancelled/expired)
created_atstringISO 8601 creation timestamp
updated_atstringISO 8601 last-updated timestamp

NodeExecution

Each entry in a run's node_executions array records a single node execution, in chronological order. Together they form the execution trace of a run — the orchestration analogue of an LLM trace.

FieldTypeDescription
node_idstringID of the executed node
node_typestring | nullNode type (agent, transform, …)
attemptinteger1-based attempt number (a retried node yields one record per attempt)
statusstringrunning | completed | failed | requires_action | skipped (running is the transient pre-completion state of a side-effecting node)
inputobject | nullResolved input_mapping the node received
outputobject | nullOutput artifact the node produced (null when failed)
errorobject | null{ code, message } when status is failed
started_atstring | nullISO 8601 timestamp when the node began executing
completed_atstring | nullISO 8601 timestamp when the record was written
created_atstringISO 8601 creation timestamp

Key Concepts

Node Types

TypeDescription
agentInvokes a SOAT Agent with a prompt. Uses agent_id and prompt.
toolCalls a SOAT Tool. Uses tool_id and input_mapping. Its artifact is the tool's own result object — see Node artifacts. Gated by Guardrails at dispatch — see Guardrail interception.
transformEvaluates a JSON Logic rule against the current state. Uses expression.
knowledgeSearches a knowledge source via the Knowledge module. Uses input_mapping with query and optional memory_ids.
memory_writeWrites a Memory entry. Uses memory_id and input_mapping with content.
conditionEvaluates a JSON Logic rule and emits a string label. Downstream edges use condition: "<label>" to select the active branch.
humanPauses the run and waits for external input. The run enters awaiting_input status with required_action.
approvalProposes a guarded tool call and pauses for a human decision via the Approvals queue. Uses tool_id, arguments, and expires_in. See Approval Nodes.
loopIterates a state collection, running a sub-orchestration per item. Uses orchestration_id, collection, item_variable, and parallelism. See Loops.
pollCalls a tool on an interval until a JSON Logic exit condition on the response holds. Uses tool_id, exit_condition, and interval. See Polling.
delayWaits for a fixed duration, then continues. Accepts 5s/5m/2h/500ms or ISO 8601 (PT5S).
emit_eventEmits an internal event of type event_type carrying the input_mapping result as the event data. Any Webhook subscribed to that event type in the run's project delivers it — signed, retried, and tracked by the Webhooks module. The graph holds no URL or secret. Fire-and-forget: the run does not block on or fail from delivery. See Emitting events.
webhookPauses awaiting an inbound callback (mode: "receive"). The run enters awaiting_input with required_action.type: "webhook_receive"; resume it via human-input. (To send data out of a graph, use emit_event.)
sub_orchestrationRuns another orchestration as a single step. Uses orchestration_id. The node's artifact is the child run's output — i.e. { terminalNodeId: terminalArtifact }, the same shape used for output on OrchestrationRun and for each item in a loop node's results array — not a flattened value. state_mapping values are JSON Logic, whose var reader descends dot-paths, so {"var": "output.terminalNodeId.someField"} pulls a deep field directly — no extra transform node needed.

Node artifacts

Every completed node produces an artifact — the object that state_mapping reads as output and that downstream nodes read as nodes.<id>. The shape is per node type:

TypeArtifact
agent{ content }. With an output_schema, a response that parses as a JSON object becomes that object instead (so its own fields are read directly).
toolThe tool's result object itself, not a wrapper — a tool returning {"status":"ok"} yields {"status":"ok"}, read as {"var": "output.status"}. Only a non-object result (string, number) is wrapped as { result }. A guardrail-blocked call yields { status: "blocked", reason } instead — see Guardrail interception.
transform{ result } — the evaluated expression.
conditionNo artifact; the node emits a branch label. Its namespace entry is { label }, read as {"var": "nodes.<id>.label"}.
knowledge{ results } — the matched entries.
memory_write{ action } — e.g. "created".
human, webhook (mode: "receive")The payload submitted to submit-human-input, verbatim.
approval{ decision, approvalId, resolvedBy, reason, result, editedArgs } — see Approval Nodes.
loop{ results } — one entry per item, each the sub-run's output. See Loops.
poll{ result, attempts, conditionMet, timedOut }. See Polling.
delay{ waited } — the duration as declared.
emit_event{ emitted, eventType }. See Emitting events.
sub_orchestrationThe child run's output, i.e. { terminalNodeId: terminalArtifact }.

The common trap is tool: because the artifact is the tool's result verbatim, {"var": "output.result"} resolves to null for any tool returning a JSON object. Map the field the tool actually returns.

Tip: a state_mapping that writes null usually means the mapping read a field the artifact does not have. Every artifact is visible under state.nodes.<id> in get-orchestration-run, so check there for the real shape.

Guardrail interception on tool nodes

A tool node's call is classified by Guardrails at dispatch, the same single gating mechanism agent tool calls use. With no agent in scope the node composes the project + tool scopes only; the strictest action class is enacted in graph terms:

  • A / passing B — the tool runs with the (cleaned) input_mapping result.
  • C (human sign-off) — the run parks on the node (required_action.type: "approval") and files an ApprovalItem with the frozen arguments, exactly like an approval node. On approval the node re-dispatches the tool with the frozen (or edited) arguments and continues down its success edge; on rejection/expiry the tool never runs and only a matching condition: "rejected" / "expired" edge follows.
  • D / tripwire — a routable blocked outcome: the node records a { status, reason } artifact and emits a blocked (or tripwire) branch label, so an edge with condition: "blocked" routes to a fallback. An unlabeled success edge does not follow a blocked node, so the happy path never runs on a blocked call.

Guardrails attach to the referenced tool (or the run's project) via guardrail_ids; there is no per-node guardrail field.

Loops (collection iteration)

A loop node iterates an array in the run state and runs a sub-orchestration once per item. It is the collection counterpart to poll (which repeats until a condition).

FieldDefaultPurpose
orchestration_id— (required)Public ID of the orchestration to run for each item (same field the sub_orchestration node uses)
collectionstate.itemsState path to the array to iterate; a path without the state. prefix is normalised to one. A missing or non-array value yields zero iterations
item_variableitemEach element is passed as the sub-run's input under this key; run input is seeded under the input namespace, so the sub-graph reads it with {"var": "input.item"}
parallelism5Items are processed in batches of this size

The node completes with an artifact { results: [...] } — one entry per item, in order, holding that sub-run's output. A graph containing a loop node is exempt from cycle detection (loops introduce intentional cycles).

{
"id": "summarise_each",
"type": "loop",
"orchestration_id": "orch_summariseOne",
"collection": "state.documents",
"item_variable": "doc",
"parallelism": 3,
"state_mapping": { "state.summaries": { "var": "output.results" } }
}

Polling

A poll node repeatedly calls a Tool until a JSON Logic exit condition on its response is satisfied. It is the condition-based counterpart to loop (which iterates a known collection).

Each attempt:

  1. Calls toolId (resolving inputMapping against state, like a tool node).
  2. Evaluates exitCondition against an augmented context — the run state plus response (the latest tool result) and attempt (1-based count). A truthy result stops polling.
  3. Otherwise the interval becomes a scheduled resumption: the run is parked and the background scheduler drives the next attempt after interval, bounded by maxIterations (default 10, ceiling 1000). There is no wall-clock ceiling — the wait no longer holds an HTTP request open, so a poll can span hours or days.

The node completes with an artifact { result, attempts, conditionMet, timedOut }. On exhaustion it completes with conditionMet: false (branch on it downstream with a condition node) — unless failOnTimeout: true, which fails the run with ORCHESTRATION_POLL_EXHAUSTED.

{
"id": "wait_for_render",
"type": "poll",
"tool_id": "tool_renderStatus",
"input_mapping": { "id": { "var": "jobId" } },
"exit_condition": { "==": [{ "var": "response.status" }, "completed"] },
"interval": "5s",
"max_iterations": 60,
"state_mapping": { "state.render": { "var": "output.result" } }
}

Note: poll and delay waits are offloaded to the background scheduler (see Durable Background Execution). They do not hold an HTTP request open, and a run parked on a wait survives a server restart.

Emitting events

To send data out of a graph, an emit_event node emits an internal event — it does not call any URL itself. Delivery is entirely the Webhooks module's job: any webhook subscribed to the event type (in the run's project) delivers the event, already signed (X-Soat-Signature), retried, tracked as a WebhookDelivery, and policy-gated. The graph therefore holds no URL and no secret — auth and endpoints are managed once, centrally, on the webhook subscription.

  • event_type — the event type to emit, e.g. guardrail.exception. A subscriber listens with create-webhook --events "guardrail.exception" (or a pattern like guardrail.*).
  • input_mapping — resolved against run state to build the event data payload.

The node is reactive and fire-and-forget, exactly like the run's own lifecycle events: it completes as soon as the event is emitted, and the run neither blocks on nor fails from any subscriber's delivery outcome. Its artifact is { emitted: true, eventType: "<type>" }. (If a graph needs a synchronous call whose failure must fail the run, use an http tool node instead — that is a tool call, not a notification.)

{
"id": "alert",
"type": "emit_event",
"event_type": "guardrail.exception",
"input_mapping": { "reason": { "var": "state.exception" } }
}

The emitted event carries resource_type: "orchestration_run" and the run's id as resource_id, so subscribers (and webhook policies) can scope to orchestration output. See Delivery for the envelope and signature format.

Retry Policy

Any node can declare a retry policy. When the node throws a transient error and attempts remain, the run parks as sleeping and re-executes the node after a backoff delay (offloaded to the scheduler, exactly like poll/delay — so retries survive a restart and hold no worker). Absent, or max_attempts <= 1, is fail-fast (today's behaviour).

Retriable vs terminal. Unexpected/infrastructure errors (network, timeouts, provider SDK throws) and upstream 5xx errors are retriable. Deliberate 4xx business errors (validation, not found, conflict) are terminal and fail the run immediately without consuming attempts.

Attempt history. Each attempt writes its own node_executions record with an incrementing attempt — failed attempts 1..N-1 followed by a final completed (success) or failed (retries exhausted, run fails).

FieldTypeDescription
max_attemptsintegerTotal attempts including the first (default 1, ceiling 20).
backoff.strategystringfixed (constant delay_ms) or exponential (doubles per prior attempt). Default fixed.
backoff.delay_msintegerBase delay between attempts in ms (default 1000).
backoff.max_delay_msintegerCap on the computed backoff delay in ms (default 300000).
{
"id": "call_flaky_api",
"type": "tool",
"tool_id": "tool_upstream",
"retry": {
"max_attempts": 4,
"backoff": { "strategy": "exponential", "delay_ms": 1000, "max_delay_ms": 60000 }
}
}

Note: node execution of side-effecting nodes is idempotent across a retry redelivery or a reaper redrive — see Idempotency. A retry (a new attempt) is deliberately not deduped; a redelivery of the same attempt is.

Durable Background Execution

Runs execute in a queue-backed durable worker, detached from the HTTP request that starts them:

  • start-orchestration-run persists the run, enqueues a continue task, and returns immediately with status: "queued"no node executes inside the request. A worker claims the task and drives the run; observe progress with get-orchestration-run (which includes node_executions) or via run lifecycle webhook events. (The single-process default runs the worker loop inside the API process, so the run starts draining right away.)
  • delay and poll waits park the run as sleeping — it holds no worker and no memory, pure DB state. The wake time (wake_at) and how to continue are persisted with the run, and the scheduler enqueues a wake task when the wait is due — so a run containing delay: "2h" survives a restart and completes on schedule.
  • human and webhook (mode: receive) nodes park the run as awaiting_input (also pure DB state, no worker); satisfy the pause with submit-human-input, which applies the submitted payload, drives the run inline, and returns the settled result. resume-orchestration-run only re-drives an awaiting_input run from its last checkpoint — it carries no node_id or payload, so it cannot satisfy a pause and will simply re-park on the same node.

Queue driver. The queue is a run_tasks table claimed in batches with SELECT … FOR UPDATE SKIP LOCKED, so multiple workers never claim the same task and no new infrastructure is required. A claimed task holds a lease; if the worker fails to acknowledge it before the lease expires, the task is redelivered (at-least-once delivery). A task is minted only when there is work to pick up — a continue when a run starts or the reaper reclaims an orphan, a wake when a parked wait comes due. Parking itself holds no task.

Pluggable queue drivers. The queue is reached through a four-operation abstraction (enqueue / claim / ack / retry, plus a stats snapshot), so the backend is selected with ORCHESTRATION_QUEUE_DRIVER and nothing else in the engine, scheduler, or worker changes. Both drivers are held to one shared conformance suite, so they are interchangeable for at-least-once delivery, lease-based redelivery, delayed availability, and exclusive claim.

postgres (default)sqs
Backing storeorchestration_run_tasks tablean SQS queue
enqueueINSERT (a future available_at parks the task)SendMessage (DelaySeconds)
claimSELECT … FOR UPDATE SKIP LOCKED + leaseReceiveMessage; the visibility timeout is the lease
ackDELETE the rowDeleteMessage
retryclear the claim, set a new available_atChangeMessageVisibility
Repeated failureredelivered until ackedthe queue's redrive policy → DLQ
Delivery counterattempts columnApproximateReceiveCount
Per-project max_concurrent_runsenforced at claim timenot enforced
oldest_queued_age_seconds, per_project statsreportednull / empty

Postgres remains the default and needs no infrastructure beyond the database. Choose sqs when a deployment standardizes on a managed queue and accepts the two differences above: per-project concurrency limits are not evaluated (only the per-worker ORCHESTRATION_WORKER_CONCURRENCY cap applies), and the operator stats are limited to what GetQueueAttributes exposes. A backoff longer than SQS's 15-minute maximum delay becomes 15 minutes — the task simply becomes visible early, and the run's own persisted wake_at still decides whether there is anything to do.

ORCHESTRATION_QUEUE_DRIVER=sqs
ORCHESTRATION_QUEUE_SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/soat-orchestration-tasks

An unrecognized ORCHESTRATION_QUEUE_DRIVER, or sqs without a queue URL, fails loudly (QUEUE_DRIVER_MISCONFIGURED) rather than silently falling back to Postgres.

Separate worker process. The worker loop is an extractable module that runs inside the API process by default. node dist/worker.js (built from src/worker.ts) starts only the scheduler tick + worker loop — no HTTP listener — so the queue can be drained by a dedicated worker with the API tier running request-only (ORCHESTRATION_WORKER_DISABLED=true). On SIGTERM/SIGINT the worker shuts down gracefully: it stops claiming new tasks and waits for tasks it has already claimed to finish before exiting (tasks not finished before the timeout are left un-acked and redelivered, so no work is lost).

Worker fleet deployment. The published image ships both entrypoints, so a fleet is a second service off the same image:

server:
environment:
ORCHESTRATION_WORKER_DISABLED: 'true' # API tier stays request-only

worker:
command: ['node', 'packages/server/dist/worker.mjs']
environment:
ORCHESTRATION_WORKER_HEARTBEAT_FILE: /tmp/soat-orchestration-worker.heartbeat
healthcheck:
test: ['CMD', 'node', 'packages/server/dist/workerHealthcheck.mjs']

Worker healthcheck. A worker serves no HTTP, so it cannot answer the API's /health probe. Instead it republishes a heartbeat file after every successful queue claim, and workerHealthcheck.mjs exits 0 only while that heartbeat is younger than ORCHESTRATION_WORKER_HEARTBEAT_STALE_MS. Grading the last successful claim rather than the last timer tick is deliberate: a worker whose loop still fires but can no longer reach the queue goes unhealthy instead of looking alive. The API's GET /health is unchanged — still a bare {"status":"ok"}.

Crash recovery. While a run is running it holds a leaselease_expires_at is set when execution starts and refreshed after every completed round (every checkpoint). If the process driving a run crashes or is redeployed mid-execution, it stops refreshing the lease. A background reaper reclaims runs whose lease has expired and enqueues a continue task so a worker re-drives them from the last checkpoint, not from scratch: completed nodes are skipped and only the unfinished frontier re-executes. A healthy long run is never reclaimed because it refreshes its lease each round.

Idempotency of node execution

At-least-once delivery means a node executor must tolerate replay. Each side-effecting node execution (agent, tool, memory_write, emit_event, sub_orchestration, loop) is written with a run-scoped idempotency key {run_id}:{node_id}:{attempt}, where attempt is the node retry attempt (not the queue delivery counter). The keyed node_executions record is inserted running before the side effect runs and updated in place afterward:

  • A redelivery of the same task replays the same (run, node, attempt) → the key already exists as completed → the stored output is reused and the executor is not re-invoked. So a redeploy mid-run neither loses the run nor repeats a completed side effect.
  • A retry (attempt N failed; the policy schedules attempt N+1) is a new key → the executor runs for real, which is what a retry means.

The honest boundary: a worker that crashes between firing the side effect and marking the key completed leaves a running key; the redelivering worker re-executes under the same key. To let downstream services dedupe that window, an http tool node forwards its key verbatim as an Idempotency-Key request header. Pure nodes (condition, transform, delay, human, approval, webhook) have no external side effect and are unkeyed.

Synchronous (compatibility) mode. Pass wait: true to start-orchestration-run to block until the run reaches a terminal (succeeded/failed) or awaiting_input state, sleeping through any delay/poll waits in-process. This preserves the legacy behaviour for callers (and tests) that need the settled run in the response. Nested loop and sub_orchestration runs always execute synchronously so their output can be aggregated.

Lifecycle events. The following events are emitted through the Webhooks module so callers do not need to poll:

EventWhen
orchestration_runs.startedA run is created and begins executing
orchestration_runs.awaiting_inputA run pauses on a human/webhook node
orchestration_runs.succeededA run reaches succeeded
orchestration_runs.failedA run reaches failed

The scheduler tick (which enqueues wake tasks for due sleeping runs and continue tasks for orphaned running runs) and the queue worker loop are configurable:

Environment VariableRequiredDescription
ORCHESTRATION_SCHEDULER_INTERVAL_MSNoScheduler tick interval in ms (default 5000).
ORCHESTRATION_RUN_LEASE_TTL_MSNoHow long a running run's lease is valid before the reaper may reclaim it, in ms (default 600000). Must exceed the longest single round of node execution.
ORCHESTRATION_WORKER_INTERVAL_MSNoWorker loop tick interval in ms (default 5000) — how often the worker drains the queue.
ORCHESTRATION_TASK_LEASE_TTL_MSNoHow long a claimed queue task's lease is valid before it may be redelivered, in ms (default 60000).
ORCHESTRATION_WORKER_DISABLEDNoSet to true to stop the API process from running the in-process worker loop and its enqueue kicks, so a dedicated worker.js process owns draining.
ORCHESTRATION_WORKER_BATCHNoMaximum tasks a worker claims per tick (default 10).
ORCHESTRATION_WORKER_CONCURRENCYNoGlobal cap on simultaneously claimed, unacked tasks per worker process (unset = no cap, bounded only by the batch size). Each tick claims at most CONCURRENCY − in-flight. A fleet of P workers bounds global parallelism at P × CONCURRENCY. See Concurrency limits.
ORCHESTRATION_QUEUE_DRIVERNoQueue backend: postgres (default) or sqs. An unknown value is rejected at startup.
ORCHESTRATION_QUEUE_SQS_QUEUE_URLWith sqsThe SQS queue URL tasks are published to and received from.
ORCHESTRATION_QUEUE_SQS_REGIONNoRegion for the SQS client (falls back to AWS_REGION, then us-east-1).
ORCHESTRATION_QUEUE_SQS_ENDPOINTNoOverride the SQS endpoint (LocalStack / ElasticMQ). Credentials otherwise resolve through the standard AWS provider chain.
ORCHESTRATION_WORKER_HEARTBEAT_FILENoWhere a standalone worker publishes its liveness heartbeat. Unset (the default for the in-API worker) writes nothing.
ORCHESTRATION_WORKER_HEARTBEAT_STALE_MSNoHow old the heartbeat may be before the worker healthcheck fails (default 30000). Must exceed ORCHESTRATION_WORKER_INTERVAL_MS.

Concurrency limits

Parallelism is bounded on two axes so a busy tenant can't starve others and the fleet can't outrun a provider's rate limits.

  • Per project. A project's max_concurrent_runs caps how many of its runs may be actively driven at once. It is enforced at queue claim time: while the project already has that many runs holding a claimed, lease-valid task, its further tasks stay queued (they are never failed and never re-enqueued with a bumped delivery count) until a slot frees. null (the default) means unlimited.

    Only actively-driven runs occupy a slot — a run parked on a delay/poll wait (sleeping) or a human node (awaiting_input) holds no task and therefore no slot. A run never blocks on itself, so a multi-round run under max_concurrent_runs: 1 continues normally.

  • Global (per worker). ORCHESTRATION_WORKER_CONCURRENCY caps the tasks a single worker process holds claimed-and-unacked at any instant, across ticks (not merely per claim batch). ORCHESTRATION_WORKER_BATCH remains the per-tick claim size beneath it, so the effective claim each tick is min(BATCH, CONCURRENCY − in-flight).

Per-project limits are enforced by the Postgres driver only — its claim is a SQL join over tasks → runs → projects, evaluated in the same transaction that leases the task. Under ORCHESTRATION_QUEUE_DRIVER=sqs only the global per-worker cap applies.

Queue metrics

GET /api/v1/orchestrations/queue/stats returns a point-in-time snapshot of the run queue — waiting vs. claimed task counts, the oldest waiting task's age, recent claim-latency percentiles (computed in-process over a rolling 5-minute window, no external metrics stack), and a per-project breakdown. driver names the active backend; under sqs the depths come from GetQueueAttributes, and oldest_queued_age_seconds / per_project are null / empty because SQS exposes neither. It is guarded by the orchestrations:GetQueueStats action (intended for admin/operator policies); a project-scoped caller sees only their own projects under per_project. This is distinct from the unauthenticated GET /health liveness probe, which stays a bare {"status":"ok"}.

State and Mappings

Each node can define:

  • input_mapping — Maps node input keys to values resolved against the run state before execution. Each value is JSON Logic (see Input Mapping).

  • state_mapping — Projects a node's artifact into state after execution. Each key is a state write path and should start with the literal state. prefix (e.g. "state.summary"); a key without the prefix (e.g. "summary") is normalized to be state-relative, the same convention loop.collection already uses. Each value is JSON Logic evaluated against { "output": <the node's artifact>, "state": <run state> } — the same evaluator as input_mapping/transform/condition, just a different context. { "summary": { "var": "output.content" } } writes the artifact's content field to state.summary; a literal value (string, number, boolean) is written as-is. A dotted target such as "state.proposed.action_id" builds a nested object (state.proposed = { action_id: … }), so a later node reads it back with {"var": "proposed.action_id"} — the var reader descends dot-paths.

    { "id": "summarise", "type": "agent", "agent_id": "agent_xyz", "state_mapping": { "state.summary": { "var": "output.content" } } }

    Since it is JSON Logic, state_mapping can also compute derived values or read the artifact's own upstream state — e.g. { "state.count": { "+": [{ "var": "state.count" }, { "var": "output.delta" }] } } accumulates a running total across nodes.

The nodes.<id> namespace

Every completed node's full artifact is also recorded at state.nodes.<nodeId>, whether or not the node declares a state_mapping — giving orchestrations the same read-any-upstream-result ergonomics as a pipeline's steps.<id> (see Pipeline Tools). A downstream node reads it with { "var": "nodes.<nodeId>.<field>" } without any explicit wiring on the upstream node:

[
{ "id": "fetch", "type": "tool", "tool_id": "tool_abc" },
{
"id": "summarise",
"type": "agent",
"agent_id": "agent_xyz",
"input_mapping": { "prompt": { "var": "nodes.fetch.text" } }
}
]

nodes is a reserved top-level state key: the engine owns it exclusively, so static validation rejects a state_mapping write targeting it, and a { "var": "nodes.<id>..." } reference is checked the same way a state_mapping-declared key is — <id> must name an earlier (upstream) node in the graph. (An input_schema property named nodes is allowed: run input is seeded under state.input, so it cannot collide.) A condition node completes with a label rather than an artifact; its namespace entry is { "label": "<emitted label>" }, readable as { "var": "nodes.<id>.label" }. The field names available under nodes.<id> are the artifact's own — nodes.fetch.text above assumes the fetch tool returns a text field, since a tool node's artifact is its result object verbatim (see Node artifacts).

Evaluation scope

Every JSON Logic expression in a graph is evaluated against the run state, which is the single shared context: it holds the run input (see Run input) plus everything upstream nodes have written via state_mapping, plus every upstream node's raw artifact under nodes.<id>. What differs between node types is not the scope but what each node does with it:

  • transform and condition evaluate their expression against the full state directly.
  • agent, tool, knowledge, memory_write, human, webhook, sub_orchestration evaluate each input_mapping value against the full state and pass only that projected result to the node (an agent's prompt, a tool's input, etc.) — they do not receive the whole state.
  • poll evaluates its input_mapping against state, then its exit_condition against state augmented with response (the latest tool result) and attempt (see Polling).

Input Mapping (JSON Logic)

Each input_mapping value is evaluated as JSON Logic against the run state — the same evaluator used by transform and condition nodes. This gives one expression language across the whole platform: pass literals, read state, or compute derived values inline, without a dedicated transform node.

ValueBehaviour
String, number, booleanPassed through as a literal
A single-key object whose key names a JSON Logic operator (var, cat, if, >, arithmetic, …)Evaluated against state
Any other object or arrayPassed through as a literal, but recursed into — a JSON Logic marker nested inside it (at any depth) is still resolved
"input_mapping": {
"language": "pt-BR",
"threshold": 0.8,
"documentId": { "var": "temaDocumentId" },
"label": { "cat": ["Tema: ", { "var": "titulo" }] },
"isLong": { ">": [{ "var": "wordCount" }, 500] },
"data": { "title": { "var": "titulo" }, "theme": { "var": "tema" } }
}

Run input

Values passed to start-orchestration-run via input seed the initial state under an input namespace, read with {"var": "input.key"} — matching the pipeline/formation convention, so run input, pipeline input.*, and formation ${...} all read the same way.

Input keys round-trip verbatim (they are not case-transformed), so a key sent as cycle_task is read as {"var": "input.cycle_task"} — not {"var": "input.cycleTask"}. Because the input namespace is always seeded, a {"var": "input.<name>"} reference in an input_mapping satisfies static validation regardless of the declared input_schema; a flat {"var": "<name>"} reference is never satisfied by run input (only by an upstream node's own state_mapping write) — earlier releases also seeded run input flat across top-level state keys, but that alias has been removed.

To pass a literal object that happens to look like a JSON Logic expression — e.g. the JSON Logic object {"var": "x"} itself, as data rather than an expression to evaluate — wrap it in preserve, which returns its argument unevaluated: {"preserve": {"var": "x"}}.

Note: an input_mapping bare string is a literal value; use {"var": "key"} to read state.key. (Earlier releases treated a bare state.<key> string as a state path — migrate those to {"var": "key"}.)

Parallel Execution

All nodes that become active in the same round execute concurrently via Promise.all. After all complete, their outputs and state mutations are applied sequentially to avoid races. A single node with multiple outgoing edges activates all targets in parallel.

Activation Groups (Fan-In)

Edges can carry an activation_group name and an activation_condition to control when a downstream node runs:

activation_conditionBehaviour
all (default)The target node activates only after every edge in the group comes from a completed node.
anyThe target node activates as soon as any edge in the group comes from a completed node. Activated at most once per run.

Edges without an activation_group always pass through unconditionally.

Cycle Detection

A DFS-based cycle check runs both at create/update time (see Static Validation) and again before a run begins. Orchestrations that contain a loop node are exempt — loops introduce intentional cycles. If a cycle reaches execution anyway, the run is created, set to failed, and the error field contains code: "ORCHESTRATION_CYCLE_DETECTED".

Static Validation

Orchestration graphs are validated before they are persisted. create-orchestration and update-orchestration reject an invalid graph with HTTP 400 (code: "ORCHESTRATION_VALIDATION_FAILED"); the error.meta field carries the full errors and warnings arrays. The same checks are available without persisting through validate-orchestration, which returns a { valid, errors, warnings } result.

Errors (block create/update):

CheckExample
Node missing its required fieldan agent node without agent_id, a transform/condition node without expression
Duplicate node idtwo nodes share id: "a"
Dangling edgean edge whose from/to references a node that does not exist
Cycle (no loop node present)a → b → a
Unsatisfiable input_mapping referencea {"var": "x"} whose state.x is never written by an upstream node, in a graph that declares an input_schema — declaring x in the schema does not help, since run input is only readable as {"var": "input.x"}
Unsatisfiable nodes.<id> referencea {"var": "nodes.ghost..."} where ghost is not an earlier (upstream) node in the graph — checked regardless of input_schema, since nodes is never part of run input
Reserved nodes namespace writea state_mapping key (e.g. "state.nodes.x") targets the engine-owned nodes state key

Warnings (never block):

CheckExample
Conditional-branch state reada node reads {"var": "branch"} that an upstream node writes only on one side of a condition, so it may be undefined when the node runs

The input_mapping reachability check only treats an unwritten reference as an error when an input_schema is declared (a closed input contract). Without an input_schema the graph stays permissive — a parallel (non-upstream) node's state_mapping may legitimately write the key before the reader runs. The check walks the graph's edges to determine which nodes are upstream, and uses dominator analysis to distinguish a key that is guaranteed-written from one written only on a conditional branch. A {"var": "nodes.<id>..."} reference is the one exception: since nodes.<id> is written exclusively by the referenced node completing, an unwritten reference is always an error, open contract or not.

soat validate-orchestration \
--nodes '[{"id":"a","type":"transform","expression":1,"state_mapping": { "state.step1": { "var": "output.result" } }},
{"id":"b","type":"transform","expression":1,"input_mapping":{"val":{"var":"step1"}}}]' \
--edges '[{"from":"a","to":"b"}]'
# → { "valid": true, "errors": [], "warnings": [] }

Node Executions

Every time a node runs, the engine persists an entry in the run's node_executions array capturing the resolved input_mapping it received, the output artifact it produced, its status, and — on failure — the structured error. The record is written even when a node throws, so a failed run is fully debuggable: get-orchestration-run shows which node failed, what input it received, and why, instead of only the final state plus a single error message.

When a run completes, nodes that were never reached (because they were on an un-traversed condition branch or an activation group that never fired) are recorded with status: "skipped". Their input, output, started_at, and completed_at fields are all null. This makes every declared node visible in the execution trace regardless of which branches ran. Walk through it end to end in Conditional Branching in Orchestrations.

{
"status": "failed",
"error": { "code": "RESOURCE_NOT_FOUND", "message": "Agent 'agent_x' not found." },
"node_executions": [
{
"node_id": "fetch",
"node_type": "tool",
"status": "completed",
"input": { "url": "https://example.com" },
"output": { "result": "..." }
},
{
"node_id": "summarise",
"node_type": "agent",
"status": "failed",
"input": { "prompt": "..." },
"output": null,
"error": { "code": "RESOURCE_NOT_FOUND", "message": "Agent 'agent_x' not found." }
}
]
}

Records are returned by both get-orchestration-run and list-orchestration-runs, ordered oldest-first. A node that pauses the run for human input is recorded with status: "requires_action"; once submit-human-input satisfies the pause, that same record is updated to status: "completed" with output set to the submitted payload and completed_at set to the resume time — it is never left behind as requires_action in a finished run. A pause that is re-entered without being satisfied — by resume-orchestration-run, a reaper redrive, or a queue redelivery — reuses that same record rather than appending another, so a paused node stays exactly one record per attempt. A node that was never reached is recorded with status: "skipped" once the run completes. For a worked example of reading back the accumulated state and per-node output of a finished run, see Orchestrate a Sonnet - Step 9 (Inspect the run state).

Run usage

Every generation an agent node dispatches meters against the run: its usage event carries the run's run_id and the dispatching node_id. get-orchestration-run surfaces the roll-up inline as a usage object (total_input_tokens, total_output_tokens, total_cached_tokens, total_reasoning_tokens, total_cost_usd) summed across the run's generations — "one operating cycle → one action" cost, without a second request. For the full per-event breakdown (line items, price rows, by_meter_type split), fetch the run receipt at GET /api/v1/usage/receipt?run_id=… — see Receipts.

When a run is started by a trigger, the trigger id is propagated onto every in-run generation's usage event, so run spend also rolls up per trigger via the usage event list (?trigger_id=).

Note: usage events are metered as each generation settles, so the roll-up is read from get-orchestration-run, not from the start-orchestration-run response. Even with wait: true the start response can carry usage: null — the run has settled but its final usage events may not have landed yet. Read the run once more to get the totals.

Human Nodes

When a human node is reached, the run pauses and the GET run response includes a required_action object:

{
"required_action": {
"type": "human_input",
"node_id": "approval",
"prompt": "Please approve or reject."
}
}

required_action.type discriminates why the run paused: human_input for a human node, webhook_receive for a webhook node in mode: "receive". Both pause reasons are resumed the same way — POST /orchestration-runs/{id}/human-input with the paused node's node_id — there is currently no separate, independently-authenticated callback endpoint for webhook-receive nodes, so delivering the callback requires the same platform bearer token or API key as any other write to the run.

Approval Nodes

An approval node proposes a guarded tool call and pauses the run for a human decision. Unlike a human node — which is resumed directly via human-input — an approval node files an ApprovalItem at emit time and is resumed only by resolving that item through the Approvals queue (POST /approvals/{id}/approve or /reject), or by server-side expiry.

The run pauses with required_action.type: "approval", carrying the created item:

{
"required_action": {
"type": "approval",
"node_id": "gate",
"approval_id": "apr_x1y2z3a4b5c6d7e8",
"expires_at": "2026-07-15T16:00:00.000Z"
}
}

The node's arguments, reasoning, evidence, and predicted_impact mappings are resolved against run state and frozen onto the item at emit time. On resolution the decision (approved | rejected | expired) becomes the node's branch label:

  • Edges labeled condition: "approved" / "rejected" / "expired" route by the decision — the counterpart of a condition node's labels.
  • An unlabeled edge leaving an approval node follows only on approval; the rejection and expiry paths must be modeled with explicit labeled edges. If no edge matches a rejected/expired decision, the run ends at the node.

Expiry is enforced server-side (see Approvals — Expiry is a hard gate): an expired item can never execute, and the run routes down its expired edge.

Common Errors

CodeStatusCauseWhat to do
ORCHESTRATION_VALIDATION_FAILED400create-orchestration/update-orchestration rejected an invalid graphRead error.meta.errors, or call validate-orchestration first — see Static Validation
ORCHESTRATION_CYCLE_DETECTEDA cycle reached execution (graphs with a cycle are normally rejected at validation time)Remove the cycle, or use a loop node if the repetition is intentional — see Cycle Detection
ORCHESTRATION_NODE_FAILED422A node could not execute as declared — a missing required field (an agent node without agent_id, a delay without duration), or an unsupported result (an agent node whose response streamed)Inspect the failing node's entry in node_executions for the exact error — see Node Executions
the underlying codevariesA node threw while executing. The originating error propagates unchanged rather than being wrapped — a referenced agent_id/tool_id that no longer exists surfaces RESOURCE_NOT_FOUND, and a failing http tool surfaces that tool's own errorDo not key error handling on ORCHESTRATION_NODE_FAILED for these; read the failing node's error.code from node_executions — see Node Executions
ORCHESTRATION_POLL_EXHAUSTEDA poll node's max_iterations was reached with failOnTimeout: trueRaise max_iterations/interval, or handle conditionMet: false downstream instead of setting failOnTimeout — see Polling

Debugging a failed run beyond "check the trace": call get-orchestration-run and read node_executions — each entry has node_id, the resolved input the node received, and (on failure) the structured error, so you can see exactly which node failed, with what input, and why, without reconstructing state from the trace alone. See Node Executions.

A run appears stuck in a non-terminal state:

  • queued — the run is enqueued and waiting for a worker to claim its continue task; it advances to running on the next worker tick. Expected briefly after an async start-orchestration-run. If it never advances, confirm a worker is running (the API process runs one unless ORCHESTRATION_WORKER_DISABLED=true) — see Durable Background Execution.
  • sleeping — the run is parked on a delay/poll wait or a node's retry backoff and holds no worker; it resumes on its own once its scheduled wake (or the node's backoff delay) elapses. active_nodes names the node being waited on; the wake time itself is persisted with the run but is not exposed on the API, so use the node's declared duration/interval to know when to expect it. This is expected, not stuck — see Durable Background Execution.
  • awaiting_input — the run is parked on a human node or a webhook (mode: receive) node; it stays there until submit-human-input is called with the paused node's node_id — see Human Nodes.
  • running for far longer than expected — the process driving it may have crashed or been redeployed mid-execution. The background reaper reclaims any run whose lease (lease_expires_at) has expired and resumes it from the last checkpoint; a healthy run refreshes its lease every round, so this self-heals within ORCHESTRATION_RUN_LEASE_TTL_MS without intervention — see Durable Background Execution.

Examples

Create a sequential pipeline

The fetch node maps output.text because a tool node's artifact is its tool's result object verbatim — substitute whatever field your tool returns (see Node artifacts).

soat create-orchestration \
--project-id "$PROJECT_ID" \
--name "fetch-and-summarize" \
--nodes '[
{"id":"fetch","type":"tool","tool_id":"tool_abc","state_mapping": { "state.raw": { "var": "output.text" } }},
{"id":"summarise","type":"agent","agent_id":"agent_xyz","input_mapping":{"prompt":{"var":"raw"}},"state_mapping": { "state.summary": { "var": "output.content" } }}
]' \
--edges '[{"from":"fetch","to":"summarise"}]'

Start a run

Returns immediately with status: "queued"; a worker claims the run and drives it in the background, moving it to running. Add wait: true (--wait in the CLI) to block until the run settles (see Durable Background Execution).

# Async (default): returns a "running" run immediately
soat start-orchestration-run \
--orchestration-id orch_01 \
--input '{"query": "summarize Q1 revenue"}'

# Synchronous: block until the run completes or pauses
soat start-orchestration-run \
--orchestration-id orch_01 \
--input '{"query": "summarize Q1 revenue"}' \
--wait

Parallel fan-out

Both branch_a and branch_b run concurrently after start completes:

{
"nodes": [
{ "id": "start", "type": "transform", "expression": { "var": "query" } },
{ "id": "branch_a", "type": "agent", "agent_id": "agent_a", "state_mapping": { "state.a": { "var": "output.content" } } },
{ "id": "branch_b", "type": "agent", "agent_id": "agent_b", "state_mapping": { "state.b": { "var": "output.content" } } }
],
"edges": [
{ "from": "start", "to": "branch_a" },
{ "from": "start", "to": "branch_b" }
]
}

Fan-in with activation_condition: all

merge runs only after both branches complete:

{
"edges": [
{ "from": "branch_a", "to": "merge", "activation_group": "join", "activation_condition": "all" },
{ "from": "branch_b", "to": "merge", "activation_group": "join", "activation_condition": "all" }
]
}

Condition-based routing

A condition node emits a string label; edges carry condition: "<label>" to select the active branch. The unselected branch's nodes are recorded as skipped. For a runnable walkthrough, see Conditional Branching in Orchestrations.

{
"nodes": [
{
"id": "check",
"type": "condition",
"expression": { "if": [{ ">": [{ "var": "score" }, 0.8] }, "high", "low"] }
},
{ "id": "high_path", "type": "agent", "agent_id": "agent_high" },
{ "id": "low_path", "type": "agent", "agent_id": "agent_low" }
],
"edges": [
{ "from": "check", "to": "high_path", "condition": "high" },
{ "from": "check", "to": "low_path", "condition": "low" }
]
}

Agent Squad

A team of agents plus the flow that coordinates them can deploy as a single Formation stack, because an orchestration is itself a formation resource type. A node's agent_id uses a ref expression to bind to an agent created in the same template; SOAT resolves it to the physical agent_... ID before the orchestration is created. Node fields are written in snake_case (agent_id, input_mapping, state_mapping), exactly as in this module's REST contract. For a full step-by-step build, see Create an Agent Squad.

cat > squad.json << 'EOF'
{
"resources": {
"Provider": {
"type": "ai_provider",
"properties": { "name": "OpenAI", "provider": "openai", "default_model": "gpt-4o" }
},
"Writer": {
"type": "agent",
"properties": {
"name": "Writer",
"ai_provider_id": { "ref": "Provider" },
"instructions": "Draft a short article on the given topic."
}
},
"Reviewer": {
"type": "agent",
"properties": {
"name": "Reviewer",
"ai_provider_id": { "ref": "Provider" },
"instructions": "Tighten and fact-check the draft."
}
},
"ContentSquad": {
"type": "orchestration",
"properties": {
"name": "content-squad",
"input_schema": { "type": "object", "properties": { "topic": { "type": "string" } } },
"nodes": [
{
"id": "write",
"type": "agent",
"agent_id": { "ref": "Writer" },
"input_mapping": { "prompt": { "var": "topic" } },
"state_mapping": { "state.draft": { "var": "output.content" } }
},
{
"id": "review",
"type": "agent",
"agent_id": { "ref": "Reviewer" },
"input_mapping": { "prompt": { "var": "draft" } },
"state_mapping": { "state.final": { "var": "output.content" } }
}
],
"edges": [{ "from": "write", "to": "review" }]
}
}
},
"outputs": {
"squad_id": { "ref": "ContentSquad" }
}
}
EOF

FORMATION=$(soat create-formation \
--project-id "$PROJECT_ID" \
--name "content-squad" \
--template-file squad.json)

SQUAD_ID=$(printf '%s' "$FORMATION" | jq -r '.outputs.squad_id')

soat start-orchestration-run \
--orchestration-id "$SQUAD_ID" \
--input '{"topic": "agent squads"}' \
--wait

Deploying this template creates the provider, both agents, and the orchestration in dependency order. Running it with { "topic": "..." } as input drives write then review and leaves the final draft at state.final.