Skip to main content

Measuring Retrieval Quality

Knowledge search ships three ranking knobs, min_similarity, rrf_k and recency_half_life_days, and none has a default that is right for every corpus. This tutorial builds the instrument that settles the question for yours: a golden set of queries with known answers, scored with recall@5, recall@10 and MRR over the ranked output of POST /api/v1/knowledge/search.

The Evaluations module is not the tool for this: an Eval runs an agent per dataset item, so measuring a ranking through it costs one LLM generation per query and mixes model behaviour into the number. Retrieval is scored with a loop over the search endpoint and a few lines of arithmetic; no generation happens anywhere in this tutorial.

Prerequisites​

export SOAT_BASE_URL=http://localhost:5047

Step 1 — Log in and create a project​

Obtain an admin token (Users) and a project to hold the corpus (Projects).

ADMIN_TOKEN=$(soat login-user --username admin --password Admin1234! | jq -r '.token')
export SOAT_TOKEN=$ADMIN_TOKEN
PROJECT_ID=$(soat create-project --name "Retrieval Quality" | jq -r '.id')
echo "PROJECT_ID: $PROJECT_ID"

Step 2 — Seed the document corpus​

Ten short handbook pages, each on one topic, written with POST /api/v1/documents, which indexes a text document before it answers, so every page is ready before the first search (Documents). A useful golden set needs candidates that compete: pages on nearby topics (on-call, severity, escalation) are what make the ranks non-trivial.

Keep the ids. The golden set names its expected answers by document_id, which is what the search response returns.

DOC_ONCALL=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/on-call.txt" \
  --content "On-call rotation: engineers rotate weekly. The pager is held by the primary on-call; if they do not acknowledge within 15 minutes the secondary is paged. Production outages at night go to the pager, never to email or chat." | jq -r '.id')
DOC_SEVERITY=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/incident-severity.txt" \
  --content "Incident severity: SEV1 is a full outage or data loss affecting all customers. SEV2 is degraded service for a subset of customers. SEV3 is a cosmetic or single-user issue. Only SEV1 and SEV2 trigger an incident call." | jq -r '.id')
DOC_ESCALATION=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/escalation.txt" \
  --content "Escalation path: the incident commander escalates to the engineering manager after 30 minutes without a mitigation, and to the VP of Engineering for any SEV1 lasting more than two hours." | jq -r '.id')
DOC_FREEZE=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/deploy-freeze.txt" \
  --content "Deploy freeze: no production releases from December 20 to January 3 and during the last two business days of each quarter. Hotfixes for SEV1 incidents are exempt with manager approval." | jq -r '.id')
DOC_BACKUPS=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/database-backups.txt" \
  --content "Database backups: a full snapshot runs nightly at 02:00 UTC and is retained for 35 days. A restore drill into a scratch cluster runs on the first Tuesday of every month and its duration is recorded." | jq -r '.id')
DOC_REVIEW=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/code-review.txt" \
  --content "Code review: every change needs two approvals and a green CI run before merge. Reviewers respond within one business day. Force pushes to shared branches are forbidden." | jq -r '.id')
DOC_VPN=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/vpn-access.txt" \
  --content "VPN access: install the company VPN client, enrol a hardware security key for multi-factor authentication, and request the staging network group from IT. Access is reviewed quarterly." | jq -r '.id')
DOC_EXPENSES=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/expenses.txt" \
  --content "Expense policy: submit receipts within 30 days. Meals are reimbursed up to 60 USD per day while travelling. Flights over six hours may be booked in premium economy." | jq -r '.id')
DOC_LAPTOPS=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/laptops.txt" \
  --content "Laptop refresh: hardware is replaced every three years. Request a replacement through the IT portal; disk encryption and the device management agent are mandatory on every machine." | jq -r '.id')
DOC_TIMEOFF=$(soat create-document --project-id "$PROJECT_ID" --path "/handbook/time-off.txt" \
  --content "Time off: 25 days of paid leave per year, requested at least two weeks ahead. Unused days carry over up to five. Public holidays follow the office country." | jq -r '.id')
soat get-document-status --document-id "$DOC_ONCALL" | jq '{status, chunk_count}'

Expected output (each page is one chunk):

{ "status": "ready", "chunk_count": 1 }

Step 3 — Seed a memory​

A memory with five entries. Memory results share the result list with document chunks, so a golden set that has only document queries cannot see a change that only moves memory entries; the recency blend in Step 7 is exactly such a change.

MEMORY_ID=$(soat create-memory --project-id "$PROJECT_ID" --name "Team Facts" \
  --description "Facts about teams, owners and dates" | jq -r '.id')
ENTRY_PAYMENTS=$(soat create-memory-entry --memory-id "$MEMORY_ID" \
  --content "The payments service is owned by the Orion team; their on-call lead is Priya." | jq -r '.id')
ENTRY_CREDS=$(soat create-memory-entry --memory-id "$MEMORY_ID" \
  --content "Staging database credentials rotate on the first Monday of each month." | jq -r '.id')
ENTRY_OFFSITE=$(soat create-memory-entry --memory-id "$MEMORY_ID" \
  --content "The Q3 planning offsite is in Lisbon during the second week of October." | jq -r '.id')
ENTRY_ACME=$(soat create-memory-entry --memory-id "$MEMORY_ID" \
  --content "Customer Acme requires a four-hour response SLA on SEV1 incidents." | jq -r '.id')
ENTRY_SEARCH=$(soat create-memory-entry --memory-id "$MEMORY_ID" \
  --content "The search index is rebuilt every Sunday night by the Data Platform team." | jq -r '.id')
echo "MEMORY_ID: $MEMORY_ID"

Step 4 — Write the golden set​

A golden set is a list of {slice, query, expected} rows. expected is the id the top of the ranking should hold, a document_id or an entry_id. slice groups rows whose failure would mean different things; here it separates document questions from memory questions, the split that matters for Step 7.

Write queries as a user would ask them, not as the page is worded. A query that repeats the page's own sentence saturates the lexical channel of hybrid retrieval and measures nothing.

GOLDEN='[
  {"slice":"document","query":"who do I page when production is down at night","expected":"'"$DOC_ONCALL"'"},
  {"slice":"document","query":"how bad does an outage have to be to count as SEV1","expected":"'"$DOC_SEVERITY"'"},
  {"slice":"document","query":"can I ship a release the week before Christmas","expected":"'"$DOC_FREEZE"'"},
  {"slice":"document","query":"how often do we practise restoring the database","expected":"'"$DOC_BACKUPS"'"},
  {"slice":"document","query":"how many reviewers does a pull request need","expected":"'"$DOC_REVIEW"'"},
  {"slice":"document","query":"how do I get onto the staging network from home","expected":"'"$DOC_VPN"'"},
  {"slice":"memory","query":"which team owns payments","expected":"'"$ENTRY_PAYMENTS"'"},
  {"slice":"memory","query":"when do the staging DB passwords change","expected":"'"$ENTRY_CREDS"'"},
  {"slice":"memory","query":"where is the planning offsite","expected":"'"$ENTRY_OFFSITE"'"},
  {"slice":"memory","query":"what response time did Acme negotiate for outages","expected":"'"$ENTRY_ACME"'"}
]'
echo "$GOLDEN" | jq 'length'

Expected output:

10

Step 5 — Run the golden set through search​

One search per row at limit: 10, keeping only the ranked list of ids. Positions are raw result positions and score is never read (Retrieval Quality — Metrics).

RESULTS=$(echo "$GOLDEN" | jq -c '.[]' | while read -r item; do \
  soat search-knowledge --project-id "$PROJECT_ID" --limit 10 \
    --query "$(echo "$item" | jq -r '.query')" \
    | jq -c --argjson item "$item" '$item + {ranked: [.results[] | (.document_id // .entry_id)]}'; \
done | jq -s '.')
echo "$RESULTS" | jq -e 'all(.[]; (.ranked | length) > 0)' > /dev/null && echo "every query returned results"
echo "$RESULTS" | jq -e 'any(.[]; .expected as $e | .ranked[:10] | any(. == $e))' > /dev/null && echo "an expected key is in the top 10"

Expected output:

every query returned results
an expected key is in the top 10

Step 6 — Compute recall@k and MRR​

For each row, rank is the 1-based position of expected in ranked, 0 when it is missing.

Definitions: Retrieval Quality — Metrics.

node << EOF
const runs = $RESULTS;
const rankOf = (r) => r.ranked.indexOf(r.expected) + 1;
const metrics = (rows) => {
  const ranks = rows.map(rankOf);
  const recallAt = (k) => ranks.filter((p) => p > 0 && p <= k).length / ranks.length;
  const mrr = ranks.reduce((s, p) => s + (p > 0 ? 1 / p : 0), 0) / ranks.length;
  return { queries: rows.length, 'recall@5': recallAt(5).toFixed(4), 'recall@10': recallAt(10).toFixed(4), MRR: mrr.toFixed(4) };
};
const slices = [...new Set(runs.map((r) => r.slice))];
const table = { overall: metrics(runs) };
for (const s of slices) table[s] = metrics(runs.filter((r) => r.slice === s));
console.table(table);
runs.forEach((r) => console.log(String(rankOf(r)).padStart(2), r.slice.padEnd(8), r.query));
if (Number(table.overall['recall@10']) === 0) process.exit(1);
EOF

Expected output (your figures will differ; the shape is the point):

┌──────────┬─────────┬──────────┬───────────┬──────────┐
│ (index)  │ queries │ recall@5 │ recall@10 │ MRR      │
├──────────┼─────────┼──────────┼───────────┼──────────┤
│ overall  │ 10      │ '1.0000' │ '1.0000'  │ '0.9167' │
│ document │ 6       │ '1.0000' │ '1.0000'  │ '0.9167' │
│ memory   │ 4       │ '1.0000' │ '1.0000'  │ '0.9167' │
└──────────┴─────────┴──────────┴───────────┴──────────┘
 1 document who do I page when production is down at night
 2 document how bad does an outage have to be to count as SEV1
 ...

How to read it, and why both metrics are needed: Retrieval Quality — Reading the table.

Step 7 — Read a knob off the table​

The method for any knob: change one parameter, re-run Step 5 and Step 6, compare the two tables slice by slice. This step does it for rrf_k and shows what a score on fused output is worth (Relevance knobs).

Fused scores are compressed, so a multiplier applied after fusion is far stronger than it looks; the figures are in Retrieval Quality — What a multiplier costs. The block below reproduces them.

soat search-knowledge --project-id "$PROJECT_ID" --limit 3 --rrf-k 60 \
  --query "who do I page when production is down at night" \
  | jq '[.results[] | .score]'
soat search-knowledge --project-id "$PROJECT_ID" --limit 3 --rrf-k 5 \
  --query "who do I page when production is down at night" \
  | jq '[.results[] | .score]'
node << EOF
const ranksLost = (k, multiplier) => {
  const top = 1 / (k + 1);
  let rank = 1;
  while (1 / (k + rank + 1) > top * multiplier) rank++;
  return rank - 1;
};
for (const k of [60, 20, 5]) console.log('rrf_k', k, 'x0.87 costs', ranksLost(k, 0.87), 'rank(s)');
console.log('days of age for x0.87 at a 30-day half-life:', (-30 * Math.log2(0.87)).toFixed(1));
EOF

Expected output (the second list is wider spread than the first; absolute values vary):

[0.0328, 0.0161, 0.0159]
[0.3333, 0.1429, 0.125]
rrf_k 60 x0.87 costs 9 rank(s)
rrf_k 20 x0.87 costs 3 rank(s)
rrf_k 5 x0.87 costs 0 rank(s)
days of age for x0.87 at a 30-day half-life: 6.0

The recency blend is measured the same way, with one constraint this corpus cannot meet: every entry above was written seconds ago, so 2^(-age/half_life) is 1.0 for all of them and recency_half_life_days reorders nothing here. On a memory with real ages, run Step 5 with --recency-half-life-days 30, compute Step 6 again, and read the memory row against the run without it (the blend is never free).

What's next​

  • Score other knobs — min_similarity drops vector candidates below a cosine floor; a value that lifts MRR on one slice can zero recall on another (Relevance knobs).
  • Grow the golden set — add a row every time a user reports a miss, with the id they should have seen; a slice per question type keeps the table legible.
  • Scope what you rank — memory_ids and document_paths narrow the candidate set before fusion (Agent with Persistent Memory — Step 12).
  • Feed an agent — knowledge injection uses the same ranking; the table above is what the agent sees before it answers.