Skip to main content

Data Retention and Zero-Retention

SOAT separates content (trace steps, a generation's metadata, error and extraction) from skeleton (ids, timestamps, status, stop_reason, token counts, cost, usage-attribution fields). Every mechanism here destroys or never writes the first and preserves the second, so an erasure never costs a billing record.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide to bring the stack up with Docker Compose.
  • Ollama running locally with qwen2.5:0.5b available. To connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.
  • New to SOAT? Read Key Concepts to understand projects, agents, and sessions first.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • For production hardening (secrets, env vars), see Configuration.
  • Server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Changing a project requires the admin role. See Users for authentication details.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Create a project, provider, and agent​

Retention is scoped to the project.

PROJECT_ID=$(soat create-project --name "Retention Demo" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Support Bot" \
  --instructions "You are a concise assistant. Answer in one short sentence." | jq -r '.id')

echo "PROJECT_ID: $PROJECT_ID"
echo "AGENT_ID: $AGENT_ID"

Step 3 — Run a turn, and confirm the content is stored​

Send a message with sensitive content, then read the trace back. file_id points to the stored steps object; content_redacted_at is null while the content is intact.

SESSION_ID=$(soat create-session --agent-id "$AGENT_ID" \
  --name "Retention session" --auto-generate false | jq -r '.id')

soat add-session-message --session-id "$SESSION_ID" \
  --message "My account number is 4455-9982. What are your support hours?" > /dev/null

TURN=$(soat generate-session-response --wait true --session-id "$SESSION_ID")
GENERATION_ID=$(printf '%s\n' "$TURN" | jq -r '.generation_id')
TRACE_ID=$(printf '%s\n' "$TURN" | jq -r '.trace_id')

soat get-trace --trace-id "$TRACE_ID" \
  | jq '{id, file_id, step_count, content_redacted_at}'

The steps object holds the whole exchange, account number included:

TRACE_FILE_ID=$(soat get-trace --trace-id "$TRACE_ID" | jq -r '.file_id')
soat download-file-base64 --file-id "$TRACE_FILE_ID" \
  | jq -r '.content' | base64 -d | jq 'length'

Expected output (ids will differ):

{
  "id": "trace_m9u9kHBRpiZ2BTSg",
  "file_id": "file_VsczNprUw29Hd8Nw",
  "step_count": 1,
  "content_redacted_at": null
}

Step 4 — Purge the trace on request​

purge-trace-content deletes the steps object from storage and clears the trace's content columns. It requires the traces:PurgeTraceContent action.

soat purge-trace-content --trace-id "$TRACE_ID" \
  | jq '{id, file_id, step_count, content_redacted_at, content_redacted_by_principal_type, content_redacted_by_principal_id}'

Expected output:

{
  "id": "trace_m9u9kHBRpiZ2BTSg",
  "file_id": null,
  "step_count": 1,
  "content_redacted_at": "2026-08-06T15:13:49.096Z",
  "content_redacted_by_principal_type": "user",
  "content_redacted_by_principal_id": "user_vAo7SYeHGV6DewYz"
}

Three things happened:

  • The row survived as a skeleton carrying content_redacted_at, so the erasure is provable to an auditor (a 404 would be indistinguishable from a resource that never existed).
  • The bytes are gone. The steps object and its File row were deleted from storage. Fetching it now fails:
# → expect-fail
soat download-file-base64 --file-id "$TRACE_FILE_ID"

Purging again is an idempotent no-op that keeps the original timestamp:

soat purge-trace-content --trace-id "$TRACE_ID" | jq '{content_redacted_at}'
  • It cascaded. Every descendant trace and its generations were purged too; a nested agent call writes its own steps object.

Step 5 — Purge a single generation​

purge-generation-content clears one generation's metadata, error, extraction and the internal recovery state of a paused run, without touching sibling generations. It requires generations:PurgeGenerationContent.

soat purge-generation-content --generation-id "$GENERATION_ID" \
  | jq '{id, status, stop_reason, metadata, content_redacted_at}'
warning

A generation purge does not delete the parent trace's steps object, which holds this generation's content alongside its siblings'. To erase a run completely, purge the trace.

Step 6 — Confirm the ledger survived​

The usage meter for the purged generation is intact.

soat list-usage-events --generation-id "$GENERATION_ID" \
  | jq '.data[0] | {generation_id, model, meter_type, components}'

soat get-generation --generation-id "$GENERATION_ID" \
  | jq '{id, status, agent_version, created_at, content_redacted_at, metadata}'

Ids, timestamps, status, stop_reason, token counts, cost, and every attribution field (action_id, trigger_id, orchestration_run_id, node_id, agent_version, routing) are preserved. cost_usd is null above only because this local model has no price rows (see Metering and Budgets); the purge does not touch it either way.

Step 7 — Automate it with a retention window​

trace_content_retention_days on the project makes erasure a policy: a daily sweep content-purges every trace in the project older than the window.

soat update-project --project-id "$PROJECT_ID" \
  --trace-content-retention-days 30 \
  | jq '{id, trace_content_retention_days}'

What the sweep guarantees:

  • Opt-in. null is the default and disables retention.
  • One purge implementation. The sweep uses the same code path as Step 4: same cascade, byte deletion, content_redacted_at semantics, audit entries and traces.content_purged events.
  • A run is purged as a unit. The sweep selects root traces; the subtree goes with the root.
  • Auditable. Sweep purges are stamped content_redacted_by_principal_type: "system" / content_redacted_by_principal_id: "retention_sweep".

The sweep schedule is server configuration; see Traces — Configuration for CONTENT_RETENTION_SWEEP_INTERVAL_MS and CONTENT_RETENTION_SWEEP_DISABLED.

Clear the window with null to keep content until purged on demand:

soat update-project --project-id "$PROJECT_ID" \
  --trace-content-retention-days null \
  | jq '{trace_content_retention_days}'

Step 8 — Zero-retention for one agent​

Retention deletes content after the fact; zero-retention never writes it. Create a second agent with trace_content_mode: none.

INTAKE_AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Regulated Intake" \
  --instructions "You are a concise intake assistant. Answer in one short sentence." | jq -r '.id')

soat patch-agent --agent-id "$INTAKE_AGENT_ID" --trace-content-mode none \
  | jq '{id, trace_content_mode}'

Run a turn through it. The reply still reaches the caller; only the durable record is a skeleton.

INTAKE_SESSION_ID=$(soat create-session --agent-id "$INTAKE_AGENT_ID" \
  --name "Intake session" --auto-generate false | jq -r '.id')

soat add-session-message --session-id "$INTAKE_SESSION_ID" \
  --message "My card ends in 4242. Is my payment late?" > /dev/null

INTAKE_TURN=$(soat generate-session-response --wait true --session-id "$INTAKE_SESSION_ID")
INTAKE_GENERATION_ID=$(printf '%s\n' "$INTAKE_TURN" | jq -r '.generation_id')
INTAKE_TRACE_ID=$(printf '%s\n' "$INTAKE_TURN" | jq -r '.trace_id')

printf '%s\n' "$INTAKE_TURN" | jq '{status}'

soat get-trace --trace-id "$INTAKE_TRACE_ID" \
  | jq '{id, file_id, content_redacted_at, content_redacted_by_principal_type, content_redacted_by_principal_id}'

soat get-generation --generation-id "$INTAKE_GENERATION_ID" \
  | jq '{id, status, metadata, content_redacted_by_principal_id}'

Expected output — the marker is set from the moment the row exists:

{
  "id": "trace_ZfPUnZBnUR3JAg1z",
  "file_id": null,
  "content_redacted_at": "2026-08-06T15:13:55.490Z",
  "content_redacted_by_principal_type": "system",
  "content_redacted_by_principal_id": "zero_retention"
}

The principal id distinguishes never stored (zero_retention) from stored-then-erased (a user, an API key, or retention_sweep). The skeleton is written unchanged, so metering is unaffected:

soat list-usage-events --generation-id "$INTAKE_GENERATION_ID" \
  | jq '.data[0] | {model, meter_type, components}'
Trade-off: no recovery after a restart

The state that resumes a generation paused on a client tool is content, so it is not persisted in this mode. A generation still pauses and resumes within a running server, but a generation paused when the server restarts cannot be recovered. If restart-recovery matters more, use a retention window.

Step 9 — Make it a project-wide mandate​

trace_content_mode: none on the project applies zero-retention to every agent in it, including agents created later.

REGULATED_PROJECT_ID=$(soat create-project --name "Regulated Tenant" | jq -r '.id')

soat update-project --project-id "$REGULATED_PROJECT_ID" \
  --trace-content-mode none | jq '{id, trace_content_mode}'

REGULATED_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$REGULATED_PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

REGULATED_AGENT_ID=$(soat create-agent \
  --project-id "$REGULATED_PROJECT_ID" \
  --ai-provider-id "$REGULATED_PROVIDER_ID" \
  --name "Tenant Agent" \
  --instructions "You are a concise assistant." | jq -r '.id')

The project is a floor: an agent may tighten to none but cannot loosen back to full.

# → expect-fail
soat patch-agent --agent-id "$REGULATED_AGENT_ID" --trace-content-mode full

An agent's null (the default) inherits the project; resolution fails closed: an unrecognised stored mode resolves to none.

Read next: Traces — Content Purge, Retention Policy, Zero-Retention, Generations — Content Purge, Projects, and Audit Log for the entries every purge writes.