Skip to main content

Traces

Traces record the full execution history of agent generations, including every reasoning step and tool call.

Overview​

Every time an agent runs a generation, SOAT automatically records a trace: the sequence of steps the model took, the tools it invoked, the inputs and outputs at each step, and any errors encountered. Traces are stored as JSON files in the project's file storage and indexed in the database for fast retrieval. Traces support parent-child relationships, so the full execution tree of a multi-agent run can be reconstructed — see the Trace Ancestry Model.

See the Permissions Reference for the IAM action strings for this module.

Data Model​

FieldTypeDescription
idstringPublic identifier for the trace
project_idstringProject the trace belongs to
agent_idstringAgent that produced the trace
file_idstring | nullID of the file containing the serialized steps (JSON array)
step_countnumberNumber of reasoning steps recorded, across every generation grouped under the trace
parent_trace_idstring | nullID of the immediate parent trace; null when this trace is itself the root
root_trace_idstring | nullID of the root trace in a multi-agent call tree; null when this trace is itself the root
errorobject | nullStructured error payload recorded when a generation in this trace failed; null otherwise
content_redacted_atstring | nullWhen the trace's content was purged; null while content is intact
content_redacted_by_principal_typestring | nullPrincipal kind that purged the content (user or api_key)
content_redacted_by_principal_idstring | nullPublic ID of that principal — the key's own id for API-key auth
created_atstringISO 8601 creation timestamp

Key Concepts​

Generation Failures​

When a generation in a trace fails (e.g. the upstream AI provider returns an error), the structured error payload is recorded on the trace's error field and on the corresponding generation record. This makes failed runs distinguishable from runs that have not started yet (which also have step_count: 0).

Step Serialization and File Linkage​

Each trace stores the raw step objects produced by the Vercel AI SDK generateText call, as a file at /traces/{traceId}.json in the project's file storage; file_id points to it, so it can be downloaded via the Files API. Error instances are serialized to plain objects (message, name, enumerable properties) so tool failures are preserved faithfully.

Reading a Turn Back​

The steps object is the raw record, in the ai package's own shape. To read a turn without parsing it yourself, use the generation's transcript — an ordered projection of the same steps into a documented, stable schema:

soat get-generation-transcript --generation_id gen_abc

A transcript is scoped to one turn, which is why it is anchored on the generation rather than here: a trace can hold several generations, and status, stop_reason and agent_version are generation fields. A transcript reads back only the steps of its own generation's segment, so grouping several turns under one trace_id does not blur them together. See Generations → Transcript.

Grouping Generations Under One Trace​

POST /agents/{agent_id}/generate accepts a trace_id. Passing one that already exists groups the new generation with the earlier ones instead of starting a new tree — use it when several turns are one logical run and parent_trace_id / root_trace_id would misrepresent them as nested calls.

  • The steps object is the concatenation of every grouped generation's steps, in the order the generations first wrote. A second generation appends; it never replaces what the first one recorded.
  • step_count counts them all, so it stays the length of the object file_id points at.
  • A generation that writes twice rewrites only its own slice. This is what a run paused on a client tool does: the tool-outputs continuation re-sends the turn's earlier steps along with the new ones, and they replace — rather than duplicate — the ones already recorded.
  • Sub-agent calls are not grouping. An agent-to-agent call gets a trace of its own, linked through parent_trace_id / root_trace_id — see Trace Ancestry Model.

To read one grouped turn on its own rather than the whole object, use that generation's transcript.

Concurrent generations sharing one trace_id are serialized per server process. If several servers write to the same trace_id at the same moment, one turn's steps can still be lost; sequential turns — the ordinary grouping flow — are unaffected.

Debugging Joins (Trace, Generation, Session)​

Generation responses carry generation_id + trace_id; GET /generations?trace_id= returns all generations linked to a trace. Trace records do not include session_id — capture (session_id, generation_id, trace_id) from generation responses at your own boundary to correlate in both directions. See Debug Session, Generation, and Trace History - Step 5.

Content Purge​

DELETE /traces/{trace_id}/content deletes the trace's steps object from storage and clears its content columns. It requires the traces:PurgeTraceContent action.

  • The row survives as a skeleton. content_redacted_at, the ids, timestamps and step_count remain, so a purge is provable. Reads of a purged trace return the skeleton with the redaction marker set, never a 404.
  • The bytes are deleted, not orphaned. The purge commits the row changes, then deletes the storage objects; a failed object delete is logged for reconciliation rather than rolled back.
  • It cascades. Every descendant trace is purged too, along with all of their generations — a descendant holds its own steps object covering the same run.

The operation is idempotent: purging an already-purged trace succeeds and leaves the original content_redacted_at untouched. A purge does not touch the usage and audit ledger: each cascaded generation keeps action_id, trigger_id, orchestration_run_id, node_id, agent_version, routing, status and timestamps. See Generations for the per-generation operation.

Retention Policy​

Setting trace_content_retention_days on a project makes purging automatic: a daily sweep content-purges every trace in that project older than the window.

soat update-project --project_id proj_abc --trace_content_retention_days 90
  • Opt-in. null (the default) disables retention. Clear it with --trace_content_retention_days null.
  • Same purge path as DELETE /traces/{id}/content — same cascade, byte deletion, content_redacted_at semantics, audit entries and traces.content_purged events.
  • Scoped to the project, not the agent — every trace in a subtree shares one project, so a project-scoped window cannot conflict across a nested call the way a per-agent window would.
  • A run is purged as a unit. The sweep selects root traces; when a root crosses the window, its whole subtree goes with it.
  • Auditable. Sweep-driven purges are stamped content_redacted_by_principal_type: "system", content_redacted_by_principal_id: "retention_sweep".

Already-redacted traces are excluded from the due set, so a steady-state sweep costs work proportional to what is newly due.

Zero-Retention Mode​

Retention deletes content after the fact; zero-retention never writes it. Set trace_content_mode to none on a project (every agent in it) or on a single agent:

soat update-project --project_id proj_abc --trace_content_mode none   # whole project
soat patch-agent --agent_id agent_xyz --trace_content_mode none          # one agent

The project is a floor, the agent may only tighten: an agent can set none under a storing project, but setting full under a none project is refused with 400. An agent's null (the default) inherits the project.

What is not written is exactly the field set a content purge clears — the two features share one definition:

RecordNot written
Tracethe steps object (no File row, no bytes), error
Generationmetadata, error, extraction, pending_state

The skeleton is still written unchanged: ids, timestamps, status, stop_reason, step_count, and every usage-attribution column — metering, cost, quotas and audit behave identically. Rows written in this mode carry content_redacted_at with content_redacted_by_principal_id: "zero_retention", distinguishing never-stored from stored-then-erased.

Trade-off: pending_state (the message history of a generation paused on a client tool) is content, so it is not persisted. A paused generation resumes normally within a running server, but a generation paused when the server restarts cannot be recovered. If restart-recovery matters more than never-stored, use retention instead.

Configuration​

The retention sweep's schedule (not its per-project window, which is a project field):

Environment VariableRequiredDescription
CONTENT_RETENTION_SWEEP_INTERVAL_MSNoSweep interval in milliseconds (default 86400000, i.e. daily).
CONTENT_RETENTION_SWEEP_DISABLEDNoSet to true to disable the sweep entirely. Projects keep their trace_content_retention_days; nothing is purged while it is off.

Trace Ancestry Model​

This section is the canonical reference for how trace relationships work. All other SOAT documentation on traces points here.

Trace lineage is not a continuation chain. They are different axes, and a generation can sit on both. Trace lineage runs inward, through the calls one turn makes — agent A calls agent B via a tool, within a single request, bounded by max_call_depth. A continuation chain runs forward in time, through turns resumed after their request is gone — an approval decided three days later spawning a new generation, bounded by the chain budget.

They are also kept deliberately independent: a chain is identified by its root generation, not by trace lineage, because lineage is legitimately rewritten by operations that know nothing about chains (deleting an agent nulls the trace parentage of everything left beneath it). A chain keyed on lineage would have been re-rooted — and handed a fresh budget — by an unrelated cleanup.

Field Definitions​

FieldMeaning
parent_trace_idThe id of the trace that directly triggered this generation. Always the immediate parent — never a grandparent or higher node.
root_trace_idThe id of the top-level trace that started the entire tree. Every trace in a tree shares the same value.

Invariants​

  1. Root traces — parent_trace_id is null and root_trace_id is null. A trace is the root of its tree if and only if both fields are null.
  2. Child traces — parent_trace_id is always the immediate parent (never skipped levels). root_trace_id is always the top-level ancestor (never null for non-root traces).
  3. Sibling traces share the same parent_trace_id and root_trace_id.
  4. Depth-1 children of the root have parent_trace_id === root_trace_id.
  5. The GET /traces/{id}/tree endpoint accepts any id in the tree and always returns the same full tree rooted at the root trace.

Concrete Example​

A three-level tree — Agent A (top level) calls Agent B via a tool, and Agent B calls Agent C:

trace_A   (root)
└── trace_B   (child of A)
    └── trace_C   (child of B)
[
  {
    "id": "trace_A",
    "agent_id": "agent_orchestrator",
    "parent_trace_id": null,
    "root_trace_id": null
  },
  {
    "id": "trace_B",
    "agent_id": "agent_researcher",
    "parent_trace_id": "trace_A",
    "root_trace_id": "trace_A"
  },
  {
    "id": "trace_C",
    "agent_id": "agent_summarizer",
    "parent_trace_id": "trace_B",
    "root_trace_id": "trace_A"
  }
]

Note trace_C: parent_trace_id points to its immediate parent (trace_B), while root_trace_id still points to the top-level root (trace_A).

Reconstructing the Tree​

Recommended: supply any trace ID from the tree to the tree endpoint; the server resolves the root and returns the fully nested tree (root node with descendants under children) in one call:

GET /api/v1/traces/{any_trace_id}/tree

Alternatively, build it client-side from a flat list: the root is the trace with root_trace_id: null; group the rest by parent_trace_id and attach recursively. Steps in a parent trace that triggered a child generation also contain the child's trace_id in the create-agent-generation tool result, so the tree can be walked through step content.

Examples​

List traces​

soat list-traces --project-id proj_abc123

Get the full trace tree​

Includes nested sub-agent traces under children.

soat get-trace-tree --trace-id trace_abc123