Debugging a Run

SOAT records what happened on several surfaces, each documented on its own module page. This page routes a symptom to a surface, draws the id graph joining them, and names the cases where a surface is empty by design.

For the field-level contrast between activity, the audit log and traces, see Activity vs. the audit log vs. traces.

Start from the symptom

Symptom	Surface	Open it with
The call came back an error	the error body, then the generation	A failed wait=true run answers GENERATION_FAILED, whose meta carries generation_id and trace_id
It answered 202 and nothing came back	the generation's status	GET /generations/{generation_id} — running, requires_action, completed or failed
It finished, but the answer was wrong	the transcript	GET /generations/{generation_id}/transcript — the turn read back step by step
It stopped early, or mid-sentence	stop_reason on the generation	The Stop Reason table — max_steps, depth_guard, chain_limit and length are four different bugs
A multi-agent run broke, and you don't know where	the trace tree	GET /traces/{trace_id}/tree from any trace in the run — the node carrying an error is the one that failed
The agent said it queued something, and nothing happened	the chain	GET /chains/{chain_id} — expired means an approval lapsed and nothing resumed the work
A run failed after exhausting its retries	the exceptions queue	GET /exceptions — filed for orchestration runs, tripwires, expired approvals and chain limits
A tool call never ran	the exception's guardrail_version	A guardrail_tripwire item names the exact guardrail document that stopped it
It cost more than you expected	usage events	Attribution runs to the project, agent, generation and action_id, so spend is grouped by the operation that caused it
Something changed and nobody says who	the audit log	The recorded action is the permission string that authorized it, and 403s are recorded too
"What have the agents been doing?"	the activity feed	One entry per autonomous execution — tool calls, schedule firings, approvals resolved

The first four rows are one investigation, in order: the error names the ids, the generation says how it ended, stop_reason says why, and the transcript says what it did.

The id graph

Every surface is reached from a generation, so a generation id is the one thing to capture at your own boundary.

Three of these are one call rather than a walk:

• The whole tree from any node. GET /traces/{trace_id}/tree resolves the root itself; no climb up parent_trace_id. Nodes carry no generations; read each node's with soat list-generations --trace-id <trace_id> (next bullet).
• Every turn of one run. GET /generations?trace_id= returns all generations on a trace; chain_id, orchestration_run_id and node_id slice the same list other ways.
• A run's work from its orchestration. A node execution record carries no generation id; the pointer runs from the generation's attribution columns. See Finding an orchestration run's generations.

The one join SOAT does not keep is session_id. A trace record has no session field, so capture (session_id, generation_id, trace_id) at your own boundary when the generation is created; it cannot be reconstructed later. Debug Session, Generation, and Trace History walks the full mapping.

When a surface is empty

An empty answer has four different meanings:

What you see	What it means
steps: [], status: running	The turn has not finished. Poll it.
steps: [], status: failed	The turn threw. A step is recorded when it finishes, and an aborted turn finishes none. The error on the generation is the whole story, and the trace carries the same payload.
steps: [] with content_redacted_at set	The content is gone, or was never written. content_redacted_by_principal_id is zero_retention when it was never stored, and the erasing principal's id when a purge or the retention sweep removed it.
An empty exceptions queue	Possibly the wrong surface — see below.

Two things that are not where you would look for them

A failed agent generation files no exception. Every kind names a specific producer: an orchestration run that died after exhausting retries, a guardrail tripwire, an approval that expired undecided, a chain that spent its budget, a self-feeding event trigger, a cost cap with no prices, or a manual filing. A plain POST /agents/{agent_id}/generate that failed is none of them; GET /generations?status=failed is the worklist for those.

An approval-gated tool call does not leave the generation pending. The intercepted call returns { "status": "pending_approval" } as the tool result and the turn completes normally. The real work happens later, in a continuation generation linked back through initiator_generation_id. "It said it queued the refund and nothing happened" is diagnosed on the chain, where expired means nobody decided in time.

What you can debug is a configuration choice

Zero-retention mode never writes the steps object, nor the generation's error (content by the same definition). A project on trace_content_mode: none keeps every skeleton field (ids, status, stop_reason, step_count, and the whole usage ledger), so metering, quotas and the audit log are untouched; it gives up the material this page tells you to read.

Setting	You keep	You give up
Default (full)	Everything on this page	Content lives until you purge it
Retention window	Full debugging inside the window	Anything older than the window, swept daily
Zero-retention	Status, stop reason, step count, cost	Steps, errors, and restart-recovery of a paused turn

Data Retention and Zero-Retention walks all three.

Next

• Debug Session, Generation, and Trace History — the id mapping as a runnable walkthrough.
• Replay a Bad Turn — freeze a bad turn as a fixture, fork the session, re-run the same context.