Skip to main content

Review Panel: Check a Draft's Fundamentals and Tone

This tutorial builds a real editorial workflow: a writer agent produces a draft, then hands it to a review panel — two reviewers that check the fundamentals (factual accuracy and logical structure) and one reviewer that checks the voice and tone. A final synthesis pass weighs all three reviews into a single, actionable verdict the writer can act on.

The panel is a Discussion — a reusable deliberation config whose participants are tool-less "voices". The writer invokes it mid-loop through a discussion-type tool, passing its own draft as the topic. This is how deep thinking works in SOAT after it moved off the agent's old reasoning config — see Migrating from agent reasoning.

You will:

  1. Log in as admin.
  2. Create a project and an Ollama AI provider.
  3. Create a review-panel discussion with three participants and a synthesis pass:
    • Fact Checker — verifies claims are accurate and supported.
    • Logic Reviewer — checks the argument is well-structured and free of gaps.
    • Voice & Tone Reviewer — checks the writing matches the intended voice.
  4. Run the panel directly on a sample draft to see the synthesized verdict.
  5. Wrap the panel in a discussion-type tool.
  6. Create a writer agent that drafts text and then calls the panel to review it.
  7. Run a generation and observe the writer consulting the panel.

By the end you will understand:

  • How to model a multi-reviewer review as participants plus a synthesis pass.
  • Why a single reviewer for fundamentals and a separate one for tone beats one do-everything prompt.
  • How an agent invokes a discussion as a tool and consumes only the synthesized outcome.

Prerequisites​

  • SOAT running locally with Ollama. Follow the Quick Start guide, and skim Key Concepts if SOAT's mental model is new to you. For production hardening (secrets, env vars), see Advanced Configuration.
  • An Ollama instance accessible at http://ollama:11434 with model qwen2.5:0.5b pulled (ollama pull qwen2.5:0.5b).
  • CLI, SDK, or curl available. The server is at http://localhost:5047.
export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in as admin​

Admin is the built-in superuser role. It bypasses policy evaluation entirely. See IAM — Authentication for details on JWT tokens and the admin role.

soat login-user --username admin --password Admin1234!
soat configure   # paste the token when prompted

Step 2 — Create a project​

Every resource in SOAT lives inside a project. Create one to hold the provider, the discussion, the tool, and the agent.

PROJECT_ID=$(soat create-project --name "Editorial Review" | jq -r '.id')
echo "Project: $PROJECT_ID"

Step 3 — Create an Ollama AI provider​

Set up a local AI provider backed by Ollama. Both the writer agent and every panel participant fall back to this provider. This tutorial uses a local Ollama provider so it can run without external credentials. To connect xAI, OpenAI, Anthropic, or Amazon Bedrock instead, see Connect Third-Party LLMs.

PROVIDER_ID=$(soat create-ai-provider \
  --project-id "$PROJECT_ID" \
  --name "Ollama" \
  --provider "ollama" \
  --default-model "qwen2.5:0.5b" | jq -r '.id')
echo "Provider: $PROVIDER_ID"

Step 4 — Create the review-panel discussion​

Create a discussion with three participants. Two of them cover the fundamentals — one checks factual accuracy, one checks logical structure — and the third checks voice and tone. Splitting fundamentals across two focused reviewers (rather than one "check everything" reviewer) keeps each turn narrow and its verdict crisp; see Deliberation and synthesis for how the turns run.

The synthesis object overrides the final pass that merges the three reviews. Its prompt uses the {topic} placeholder (the draft under review) and {steps.deliberation} (the transcript of all reviewer turns) — see Synthesis.

DISCUSSION_ID=$(soat create-discussion \
  --project-id "$PROJECT_ID" \
  --name "Draft review panel" \
  --description "Checks a draft for fundamentals (accuracy + logic) and voice/tone." \
  --ai-provider-id "$PROVIDER_ID" \
  --max-rounds 1 \
  --participants '[
    {
      "name": "Fact Checker",
      "prompt": "You verify fundamentals. List every factual claim in the draft and mark each Supported, Unsupported, or Wrong. Do not comment on style."
    },
    {
      "name": "Logic Reviewer",
      "prompt": "You review structure and reasoning. Point out gaps, non-sequiturs, and unstated assumptions. Do not comment on factual accuracy or style."
    },
    {
      "name": "Voice & Tone Reviewer",
      "prompt": "You review only voice and tone. The target voice is clear, confident, and concise. Flag hedging, jargon, and shifts in register. Do not comment on facts or logic."
    }
  ]' \
  --synthesis '{
    "prompt": "You are the editor. The draft under review is:\n\n{topic}\n\nThe panel gave these reviews:\n\n{steps.deliberation}\n\nProduce a single verdict: SHIP or REVISE, followed by the top 3 concrete fixes ordered by importance."
  }' | jq -r '.id')
echo "Discussion: $DISCUSSION_ID"

Step 5 — Run the panel directly on a sample draft​

Before wiring it to an agent, invoke the panel yourself to see the synthesized verdict. The topic is the draft under review. The run is LLM-dependent, so its outcome will vary — inspect the run but do not depend on exact wording.

# The outcome is free-form model text; print the raw run rather than piping it
# through jq (LLM text can contain bytes that break a full JSON parse).
soat create-discussion-run \
  --discussion-id "$DISCUSSION_ID" \
  --topic "The moon is the largest object in the solar system, so obviously we should build our data center there to cut latency for everyone on Earth."

The Fact Checker should flag the false claim, the Logic Reviewer the non-sequitur, and the Voice & Tone Reviewer the "obviously". The synthesis merges those into one REVISE verdict. The full transcript persists as a Conversation and the verdict as a Document on the run.

Step 6 — Wrap the panel in a tool​

To let an agent consult the panel mid-loop, expose it as a discussion-type tool. The tool references the discussion by ID; calling it with a topic runs the discussion synchronously and returns { outcome, run_id } — only the synthesized verdict, never the full transcript.

REVIEW_TOOL_ID=$(soat create-tool \
  --project-id "$PROJECT_ID" \
  --name "review-panel" \
  --type discussion \
  --description "Sends a draft to the editorial review panel and returns a SHIP/REVISE verdict with concrete fixes." \
  --discussion '{"discussion_id": "'"$DISCUSSION_ID"'"}' | jq -r '.id')
echo "Tool: $REVIEW_TOOL_ID"

Step 7 — Create the writer agent​

Create the writer agent and attach the review tool. Its instructions tell it to draft, then consult the panel, then revise if the verdict is REVISE. Because the tool result carries only the synthesized outcome, the writer's context stays small no matter how much the panel deliberated.

AGENT_ID=$(soat create-agent \
  --project-id "$PROJECT_ID" \
  --ai-provider-id "$PROVIDER_ID" \
  --name "Writer" \
  --instructions "You write short marketing blurbs. After drafting, call the review-panel tool with your draft as the topic. If the verdict is REVISE, apply the top fixes and return the improved blurb. If SHIP, return the draft as-is." \
  --tool-ids '["'"$REVIEW_TOOL_ID"'"]' | jq -r '.id')
echo "Agent: $AGENT_ID"

To force the writer to consult the panel before answering (rather than leaving it to the model), attach a step rule or set tool_choice: required for the first step.

Step 8 — Run a generation​

Ask the writer for a blurb. It drafts, calls review-panel with the draft, receives the verdict, and revises. The result is LLM-dependent — check status and inspect the output, but do not assert exact wording. Follow the trace to see the discussion run nested inside the generation.

# The result embeds the writer's draft and the panel's free-form verdict, so
# print the raw generation rather than piping it through jq.
soat create-agent-generation \
  --agent-id "$AGENT_ID" \
  --messages '[{"role":"user","content":"Write a one-sentence blurb for a note-taking app called Jotly."}]'

What happened​

  1. One discussion, three focused voices. The panel is a single Discussion with three participants. Each has a narrow persona — accuracy, logic, tone — so no single turn tries to do everything, and each verdict stays sharp.

  2. Synthesis merges the reviews. After the reviewers speak, the synthesis pass reads the whole transcript ({steps.deliberation}) plus the original draft ({topic}) and emits one SHIP/REVISE verdict with ranked fixes. This is the only text the caller sees.

  3. The agent consulted the panel as a tool. The writer attached a discussion-type tool. When it called the tool with its draft, the server ran the discussion synchronously and returned { outcome, run_id }. The full transcript persisted on the run — the writer's context received only the synthesized verdict.

  4. Deep thinking never breaks the flow. If a reviewer turn or the synthesis fails, the run degrades gracefully rather than erroring — see Deliberation and synthesis.

Next steps​

  • Add max_rounds: 2 so reviewers can rebut each other before synthesis — useful when fundamentals and tone pull in opposite directions.
  • Give the Voice & Tone Reviewer its own ai_provider_id or model override to grade tone with a stronger model than the fact check needs.
  • Force the review with a step rule so the writer can never skip the panel.
  • Read the Discussions module reference for the full participant and synthesis options, and Migrating from agent reasoning if you used the old reasoning config.