Skip to main content

Replay a Bad Turn

An agent gave a bad answer at message 3 of a real conversation. You want to try a stricter prompt against that exact context — not a paraphrase of it typed from memory — and you want the bad answer to become a test case so it cannot come back quietly.

Three operations do that, and they compose into one loop:

OperationWhat it gives you
get-generation-transcriptthe turn read back step by step — what was asked, what it did
create-dataset-item-from-generationthat turn frozen as an eval fixture
fork-sessiona new session branched at any message, same context

You will produce a turn that goes wrong, read it back, capture it, branch the session at the customer's question, answer it with a different agent, and score the result against the captured fixture.

This tutorial assumes you have been through Evaluate an Agent.

Prerequisites​

  • SOAT running locally. Follow the Quick Start guide to bring the stack up with Docker Compose.
  • Ollama running locally with qwen2.5:0.5b available. This tutorial uses a local provider so it runs without external credentials — to connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.
  • New to SOAT? Read Key Concepts to understand projects, agents, and sessions first.
  • CLI installed and configured, or SDK set up. See CLI or SDK.
  • For production hardening (secrets, env vars), see Configuration.
  • Server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Admin is the built-in superuser role. See Users for authentication details.

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN

Step 2 — Produce the turn that goes wrong​

A support drafter with vague instructions, and a customer whose second message asks the question that matters. See Sessions for the session lifecycle.

PROJECT_ID=$(soat create-project --name "Replay Workshop" | jq -r '.id')

AI_PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Local Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Support Drafter" \
  --instructions "Reply to the customer in one short sentence." | jq -r '.id')

SESSION=$(soat create-session --agent-id "$AGENT_ID" --name "ticket-4471")
SESSION_ID=$(printf '%s' "$SESSION" | jq -r '.id')
CONVERSATION_ID=$(printf '%s' "$SESSION" | jq -r '.conversation_id')

soat add-session-message --session-id "$SESSION_ID" \
  --message "Hi, my order 4471 arrived with a cracked screen."
soat generate-session-response --session-id "$SESSION_ID" --wait true | jq '.status'

soat add-session-message --session-id "$SESSION_ID" \
  --message "So what do I do now? Do I get a refund or not?"
GENERATION_ID=$(soat generate-session-response --session-id "$SESSION_ID" --wait true | jq -r '.generation_id')

soat list-conversation-messages --conversation-id "$CONVERSATION_ID" \
  | jq '.data | map({position, role, document_id})'

The conversation is four messages, each backed by a Document — note the document_id values, they are the thing the fork will share:

[
  { "position": 0, "role": "user", "document_id": "doc_xhKfqZS0h81cSgaW" },
  { "position": 1, "role": "assistant", "document_id": "doc_opF0s5dnVb0iISJT" },
  { "position": 2, "role": "user", "document_id": "doc_YySNY6X8dUvUaoP0" },
  { "position": 3, "role": "assistant", "document_id": "doc_hKyysHRAWkdkmcyq" }
]

Position 3 is the answer that ducked the refund question. Everything below works from it.

Step 3 — Read the turn back​

A transcript is the turn reconstructed at read time: the messages it was asked, each model step in order with its tool calls and results, and the final answer. It is assembled from the generation record and the trace's steps object — never stored, so it dies with the content it projects rather than outliving a retention purge.

soat get-generation-transcript --generation-id "$GENERATION_ID" \
  | jq '{generation_id, agent_id, agent_version, status, step_count,
         asked: (.input | length),
         steps: (.steps | map({index, finish_reason, tool_calls: (.tool_calls | length)})),
         output}'

soat get-generation-transcript --generation-id "$GENERATION_ID" | jq -e '.status == "completed"'
{
  "generation_id": "gen_SlTCmreS0pKemdOU",
  "agent_id": "agent_aM5RVDxV9EJZpZzu",
  "agent_version": 1,
  "status": "completed",
  "step_count": 1,
  "asked": 4,
  "steps": [{ "index": 0, "finish_reason": "stop", "tool_calls": 0 }],
  "output": { "content": "…", "finish_reason": "stop" }
}

agent_version is the config that actually served the turn, step_count survives a purge even when steps is empty, and status disambiguates an empty transcript caused by a run still in flight from one caused by erased content. A turn that called tools shows them per step, which is how you see why an answer came out the way it did instead of guessing from the final text.

Step 4 — Freeze the turn as a fixture​

Hand-authored dataset items drift away from what production traffic looks like. Curating from a real generation copies the turn's input — and its output as expected_output when you do not supply one — into a dataset item that records where it came from.

Here the recorded answer is the bad one, so pass the answer you wanted:

DATASET_ID=$(soat create-dataset --project-id "$PROJECT_ID" --name "support-regressions" | jq -r '.id')

ITEM=$(soat create-dataset-item-from-generation \
  --dataset-id "$DATASET_ID" \
  --generation-id "$GENERATION_ID" \
  --expected-output "Apologize, confirm the refund, and give the timeline.")

printf '%s' "$ITEM" | jq '{id, source_generation_id, expected_output}'
printf '%s' "$ITEM" | jq -e --arg gen "$GENERATION_ID" '.source_generation_id == $gen'

The item is a copy, deliberately. source_generation_id records the provenance, but purging or deleting that generation neither deletes nor rewrites the item — a fixture that mutated under you would silently rewrite the baseline every regression report is compared against.

That is the opposite of what forking does next, and the difference is the whole design: a fixture must be frozen, a fork must stay bound to real history.

Step 5 — Fork the session and answer it differently​

Now the experiment. A second agent with instructions that actually commit to a decision, and the same conversation up to the customer's question.

STRICT_AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$AI_PROVIDER_ID" \
  --name "Support Drafter (strict)" \
  --instructions "Apologize once, state the refund or replacement decision explicitly, and give a timeline. Two sentences at most." | jq -r '.id')

FORK=$(soat fork-session \
  --session-id "$SESSION_ID" \
  --fork-at-position 2 \
  --agent-id "$STRICT_AGENT_ID" \
  --name "retry: strict drafter" \
  --tags '{"experiment":"strict-v1"}')

FORK_ID=$(printf '%s' "$FORK" | jq -r '.id')
FORK_CONVERSATION_ID=$(printf '%s' "$FORK" | jq -r '.conversation_id')

printf '%s' "$FORK" | jq '{id, agent_id, forked_from_session_id, forked_from_position, auto_generate, tags}'
{
  "id": "sess_FjoSC0ZDvvffv6kc",
  "agent_id": "agent_Ft2qBeIoZf7SLftN",
  "forked_from_session_id": "sess_O7PvO1CQ7Lq3h0wW",
  "forked_from_position": 2,
  "auto_generate": false,
  "tags": { "experiment": "strict-v1" }
}

fork_at_position branches after that position, so the fork holds messages 0–2 — up to and including the customer's question, and not the answer you are trying to replace. Omit it to branch at the tip.

The fork references the parent's documents​

The fork's conversation has its own message rows, but they point at the same document rows. Only the ordering was duplicated:

PARENT_DOC=$(soat list-conversation-messages --conversation-id "$CONVERSATION_ID" | jq -r '.data[0].document_id')
FORK_DOCS=$(soat list-conversation-messages --conversation-id "$FORK_CONVERSATION_ID")

printf '%s' "$FORK_DOCS" | jq '.data | map({position, role, document_id})'

# Same document row, not a copy of the text — and only up to the fork point.
printf '%s' "$FORK_DOCS" | jq -e --arg doc "$PARENT_DOC" '.data[0].document_id == $doc'
printf '%s' "$FORK_DOCS" | jq -e '(.data | length) == 3'

# The parent still has all four of its messages.
soat list-conversation-messages --conversation-id "$CONVERSATION_ID" | jq -e '(.data | length) == 4'

This is what keeps the branch honest. One stored copy of the content means a retention purge erases it from parent and fork together, storage stays proportional to the conversation rather than to the number of experiments, and the fork cannot drift from what actually happened.

Drive the branch​

The fork is created inert — auto_generate is false and no generation was triggered. Creating a branch and running it are separate acts:

soat generate-session-response --session-id "$FORK_ID" --wait true \
  | jq '{status, generation_id, content: .message.content}'

soat list-session-forks --session-id "$SESSION_ID" \
  | jq '{total, data: (.data | map({id, name, agent_id, forked_from_position}))}'

soat list-session-forks --session-id "$SESSION_ID" | jq -e --arg fork "$FORK_ID" \
  '[.data[] | select(.id == $fork)] | length == 1'

Two agents have now answered the same question from the same context, and both conversations are intact and addressable. GET /forks walks one level: forking a fork is allowed and unbounded, and each branch is listed under its own parent.

:::caution Forking replays tool results — it never re-invokes tools A forked turn re-encounters tool calls that already ran. Some are idempotent reads; send_email and charge_card are not. So the recorded results travel with the copied messages and are replayed as model input — exploring a "what if" cannot charge a card a second time.

The honest consequence: the fork sees the tool data as it was, not as it is now. That is what you want for a comparison (change one variable, not two) and wrong for "resume this session for real," which is not what forking is. :::

Two more things the fork deliberately does not inherit: it starts with no actor, because single session per actor allows one open session per (agent, actor) pair and inheriting would make forking impossible for exactly the agents that enforce it; and forked_from_session_id becomes null if you delete the parent, because a fork is a real session with its own history and must not vanish with it.

Step 6 — Prove the fix against the fixture​

The captured item is now an ordinary dataset item, so the eval machinery applies unchanged — this time with the strict agent under test:

EVAL_ID=$(soat create-eval \
  --project-id "$PROJECT_ID" \
  --name "refund-clarity" \
  --agent-id "$STRICT_AGENT_ID" \
  --dataset-id "$DATASET_ID" \
  --scorers '[{"type":"json_logic","expression":{"!=":[{"var":"output"},""]}}]' \
  --pass-threshold 0.5 | jq -r '.id')

RUN=$(soat start-eval-run --eval-id "$EVAL_ID" --wait true)
RUN_ID=$(printf '%s' "$RUN" | jq -r '.id')

printf '%s' "$RUN" | jq '{status, passed, completed_count, errored_count, aggregate_scores}'
printf '%s' "$RUN" | jq -e '.status == "completed"'

soat list-eval-results --eval-id "$EVAL_ID" --eval-run-id "$RUN_ID" \
  | jq '.data | map({dataset_item_id, scores})'

The run replays the item's recorded input — the exact message list that turn was asked — against the agent you name, so the only thing that changed between the production failure and this run is the agent. A json_logic scorer is the cheap structural floor used here; for output with no single correct string, bind an llm_judge scorer instead (Judge Open-Ended Answers).

What you built​

production turn  ──get-generation-transcript──▶  read it back, step by step
       │
       ├────create-dataset-item-from-generation──▶  frozen fixture  ──▶  eval run
       │                                            (copy: survives a purge)
       └────fork-session --fork-at-position 2────▶  branch  ──▶  new answer
                                                    (reference: dies with its parent's content)

The two halves pull in opposite directions on purpose. The fixture is a copy because a dataset must not be rewritten by what happens to production afterwards. The fork is a reference because a branch that drifted from the conversation it claims to continue would be an experiment about nothing.

What's next​