Traces

Traces record the full execution history of agent generations, including every reasoning step and tool call.

Overview

Every agent generation records a trace: the model's steps, tool invocations, inputs, outputs and errors. Traces are stored as JSON files in the project's file storage and indexed in the database. Parent-child links reconstruct a multi-agent run's tree; see the Trace Ancestry Model.

See the Permissions Reference for the IAM action strings for this module.

Related Tutorials

• Debug Session, Generation, and Trace History - Step 5 (Inspect traces for each generation)
• Multi-Agent Sonnet with Nested Agent Calls - Step 12 (Inspect the trace tree)
• Deploy a Multi-Agent App with Agent Formation - Step 9 (Inspect the trace tree)
• Data Retention and Zero-Retention - Step 4 (Purge the trace on request)

Data Model

Field	Type	Description
id	string	Public identifier for the trace
project_id	string	Project the trace belongs to
agent_id	string	Agent that produced the trace
file_id	string | null	ID of the file containing the serialized steps (JSON array)
step_count	number	Number of reasoning steps recorded, across every generation grouped under the trace
parent_trace_id	string | null	ID of the immediate parent trace; null when this trace is itself the root
root_trace_id	string | null	ID of the root trace in a multi-agent call tree; null when this trace is itself the root
error	object | null	Why a generation in this trace failed, as a recorded failure { code, message, meta? }; null otherwise
content_redacted_at	string | null	When the trace's content was purged; null while content is intact
content_redacted_by_principal_type	string | null	Principal that purged the content: user, api_key, or system
content_redacted_by_principal_id	string | null	Public ID of that principal — the key's own id for API-key auth
created_at	string	ISO 8601 creation timestamp

Key Concepts

Generation Failures

When a generation fails (e.g. a provider error), the structured error is recorded on the trace's error field and on the generation record, distinguishing failed runs from not-yet-started ones. The steps the turn completed before failing are kept, so a turn whose final answer fails output_schema still shows the tool calls it made; a failure before the first step leaves step_count: 0.

Step Serialization and File Linkage

Raw step objects from the Vercel AI SDK generateText call are stored at /traces/{traceId}.json in the project's file storage; file_id points to it (downloadable via the Files API). Error instances are serialized to plain objects (message, name, enumerable properties).

Reading a Turn Back

The steps object is in the ai package's own shape. The generation's transcript is an ordered projection of the same steps into a stable schema:

CLI

soat get-generation-transcript --generation_id gen_abc

SDK

const { data, error } = await soat.generations.getGenerationTranscript({
  path: { generation_id: 'gen_abc' },
});
if (error) throw new Error(JSON.stringify(error));

curl

curl https://api.example.com/api/v1/generations/gen_abc/transcript \
  -H "Authorization: Bearer <token>"

A transcript is scoped to one turn (a trace can hold several generations; status, stop_reason and agent_version are generation fields) and reads back only its own generation's segment. See Generations → Transcript.

Grouping Generations Under One Trace

POST /agents/{agent_id}/generate accepts a trace_id; an existing one groups the new generation with the earlier ones. Use it when several turns are one logical run rather than nested calls.

• The steps object concatenates every grouped generation's steps, in first-write order; a second generation appends.
• step_count counts them all, matching the object file_id points at.
• A generation that writes twice rewrites only its own slice (a tool-outputs continuation re-sends the turn's earlier steps with the new ones; they replace, not duplicate).
• Sub-agent calls are not grouping; they get their own trace linked through parent_trace_id / root_trace_id (see Trace Ancestry Model).

Read one grouped turn via its transcript.

Concurrent generations on one trace_id are serialized per server process; simultaneous writes from several servers can lose one turn's steps. Sequential turns are unaffected.

Debugging Joins (Trace, Generation, Session)

Generation responses carry generation_id + trace_id; GET /generations?trace_id= lists a trace's generations. Traces do not include session_id; capture (session_id, generation_id, trace_id) from generation responses to correlate both ways. See Debug Session, Generation, and Trace History - Step 5.

Content Purge

DELETE /traces/{trace_id}/content deletes the steps object from storage and clears the content columns. Requires traces:PurgeTraceContent.

• The row survives as a skeleton: content_redacted_at, ids, timestamps and step_count remain. Reads return the skeleton, never a 404.
• The bytes are deleted. Row changes commit, then storage objects are deleted; a failed object delete is logged for reconciliation, not rolled back.
• It cascades to every descendant trace and their generations.

Idempotent: re-purging leaves the original content_redacted_at untouched. The usage and audit ledger is untouched: each cascaded generation keeps action_id, trigger_id, orchestration_run_id, node_id, agent_version, routing, status and timestamps. Per-generation operation: Generations.

Retention Policy

trace_content_retention_days on a project makes a daily sweep content-purge every trace older than the window. The sweep also runs once at server startup.

CLI

soat update-project --project_id proj_abc --trace_content_retention_days 90

SDK

const { data, error } = await soat.projects.updateProject({
  path: { project_id: 'proj_abc' },
  body: { trace_content_retention_days: 90 },
});
if (error) throw new Error(JSON.stringify(error));

curl

curl -X PATCH https://api.example.com/api/v1/projects/proj_abc \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{ "trace_content_retention_days": 90 }'

• Opt-in. null (default) disables retention; clear with --trace_content_retention_days null.
• Same purge path as DELETE /traces/{id}/content: cascade, byte deletion, content_redacted_at, audit entries and traces.content_purged events.
• Scoped to the project, not the agent, so a window cannot conflict across a nested call.
• A run is purged as a unit. The sweep selects root traces; a root crossing the window takes its subtree.
• Auditable. Stamped content_redacted_by_principal_type: "system", content_redacted_by_principal_id: "retention_sweep".

Already-redacted traces are excluded from the due set.

Zero-Retention Mode

Zero-retention never writes content. Set trace_content_mode to none on a project (every agent in it) or a single agent:

CLI

soat update-project --project_id proj_abc --trace_content_mode none   # whole project
soat patch-agent --agent_id agent_xyz --trace_content_mode none          # one agent

SDK

// whole project
const project = await soat.projects.updateProject({
  path: { project_id: 'proj_abc' },
  body: { trace_content_mode: 'none' },
});
if (project.error) throw new Error(JSON.stringify(project.error));

// one agent
const { data, error } = await soat.agents.patchAgent({
  path: { agent_id: 'agent_xyz' },
  body: { trace_content_mode: 'none' },
});
if (error) throw new Error(JSON.stringify(error));

curl

# whole project
curl -X PATCH https://api.example.com/api/v1/projects/proj_abc \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{ "trace_content_mode": "none" }'

# one agent
curl -X PATCH https://api.example.com/api/v1/agents/agent_xyz \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{ "trace_content_mode": "none" }'

The project is a floor; the agent may only tighten: full under a none project is refused with 400. An agent's null (default) inherits the project.

What is not written is exactly the field set a content purge clears:

Record	Not written
Trace	the steps object (no File row, no bytes), error
Generation	metadata, error, extraction, pending_state

The skeleton is still written: ids, timestamps, status, stop_reason, step_count, and every usage-attribution column, so metering, cost, quotas and audit are unchanged. Rows carry content_redacted_at with content_redacted_by_principal_id: "zero_retention".

Trade-off: pending_state (the message history of a generation paused on a client tool) is content and is not persisted, so a generation paused across a server restart cannot be recovered. If that matters, use retention instead.

Who may act on a trace

Every route that acts on one trace is authorized against that trace's SRN, srn:<project_id>:trace:<trace_id>, not against the project. A policy may therefore name the traces it covers:

{
  "statement": [
    {
      "effect": "Allow",
      "action": ["traces:GetTrace", "traces:GetTraceTree"],
      "resource": ["srn:proj_V1StGXR8Z5jdHi6B:trace:trace_V1StGXR8Z5jdHi6B"]
    }
  ]
}

Refusals keep the shapes IAM defines: a read the caller may not perform is 404 (a trace it may not see does not announce itself), a write is 403, and a credential scoped to another project is 403 API_KEY_PROJECT_SCOPE. A write on a resource in a project none of the caller's policies name is 404 too — the same answer their read would get, so a refusal never confirms existence across a tenant boundary.

Listing traces stays project-scoped: GET /api/v1/traces asks whether the caller may list traces in a project at all, so a policy that names individual traces grants no listing.

Configuration

The retention sweep's schedule (the per-project window is a project field):

Environment Variable	Required	Description
CONTENT_RETENTION_SWEEP_INTERVAL_MS	No	Sweep interval in milliseconds (default 86400000, i.e. daily).
CONTENT_RETENTION_SWEEP_DISABLED	No	Set to true to disable the sweep entirely. Projects keep their trace_content_retention_days; nothing is purged while it is off.

Trace Ancestry Model

Canonical reference for trace relationships.

Trace lineage is not a continuation chain. Lineage runs inward, through the calls one turn makes within a single request, bounded by max_call_depth. A chain runs forward in time, through turns resumed after their request is gone, bounded by the chain budget. A chain is keyed on its root generation, not on lineage, because lineage is rewritten by unrelated operations (deleting an agent nulls the trace parentage beneath it).

Field Definitions

Field	Meaning
parent_trace_id	The id of the trace that directly triggered this generation. Always the immediate parent — never a grandparent or higher node.
root_trace_id	The id of the top-level trace that started the entire tree. Every trace in a tree shares the same value.

Invariants

1. Root traces — parent_trace_id and root_trace_id are both null (iff).
2. Child traces — parent_trace_id is the immediate parent; root_trace_id is the top-level ancestor (never null for non-root traces).
3. Sibling traces share the same parent_trace_id and root_trace_id.
4. Depth-1 children of the root have parent_trace_id === root_trace_id.
5. The GET /traces/{id}/tree endpoint accepts any id in the tree and always returns the same full tree rooted at the root trace.

Concrete Example

Agent A calls Agent B via a tool; Agent B calls Agent C:

trace_A   (root)
└── trace_B   (child of A)
    └── trace_C   (child of B)

[
  {
    "id": "trace_A",
    "agent_id": "agent_orchestrator",
    "parent_trace_id": null,
    "root_trace_id": null
  },
  {
    "id": "trace_B",
    "agent_id": "agent_researcher",
    "parent_trace_id": "trace_A",
    "root_trace_id": "trace_A"
  },
  {
    "id": "trace_C",
    "agent_id": "agent_summarizer",
    "parent_trace_id": "trace_B",
    "root_trace_id": "trace_A"
  }
]

trace_C: parent_trace_id is trace_B, root_trace_id is trace_A.

Reconstructing the Tree

Recommended: pass any trace ID to the tree endpoint; it returns the nested tree (descendants under children) in one call:

GET /api/v1/traces/{any_trace_id}/tree

Tree nodes carry no generations; list a node's generations with GET /api/v1/generations?trace_id=.

Or walk it a level at a time: GET /api/v1/traces?parent_trace_id= pages the direct children of one trace; an id naming no trace the caller can read matches nothing. Or build it client-side: the root has root_trace_id: null; group the rest by parent_trace_id and attach recursively. A parent's create-agent-generation tool result also contains the child's trace_id.

Examples

List traces

CLI

soat list-traces --project-id proj_abc123

SDK

import { SoatClient } from '@soat/sdk';
const soat = new SoatClient({ baseUrl: 'https://api.example.com', token: 'sk_...' });

const { data, error } = await soat.traces.listTraces({
  query: { project_id: 'proj_abc123' },
});
if (error) throw new Error(JSON.stringify(error));

curl

curl "https://api.example.com/api/v1/traces?project_id=proj_abc123" \
  -H "Authorization: Bearer <token>"

List a trace's direct children

CLI

soat list-traces --parent-trace-id trace_abc123

SDK

const { data, error } = await soat.traces.listTraces({
  query: { parent_trace_id: 'trace_abc123' },
});
if (error) throw new Error(JSON.stringify(error));

curl

curl "https://api.example.com/api/v1/traces?parent_trace_id=trace_abc123" \
  -H "Authorization: Bearer <token>"

Get the full trace tree

Includes nested sub-agent traces under children.

CLI

soat get-trace-tree --trace-id trace_abc123

SDK

const { data, error } = await soat.traces.getTraceTree({
  path: { trace_id: 'trace_abc123' },
});
if (error) throw new Error(JSON.stringify(error));

curl

curl https://api.example.com/api/v1/traces/trace_abc123/tree \
  -H "Authorization: Bearer <token>"