Retrieval Quality

Knowledge search exposes three ranking knobs, min_similarity, rrf_k and recency_half_life_days. None has a default that is right for every corpus, so a knob is set from a measurement, never from this page. This page defines the measurement; Measuring Retrieval Quality runs it against a corpus you own.

Metrics

A golden set is a list of rows {slice, query, expected}, where expected is the id (document_id or entry_id) that should top the ranking. Each row is searched once at a fixed limit; rank is the 1-based position of expected in the response, 0 when it is absent.

Metric	Per row	Reads as
recall@k	1 if 0 < rank <= k, else 0	Was the answer retrieved within the top k at all?
MRR	1 / rank, 0 when absent	How high did it land? Rank 1 scores 1.0, rank 2 0.5, rank 10 0.1

Both are averaged over the rows of a slice, and over every row for the overall figure.

Positions are raw result positions: a document occupying several slots costs the slots the caller sees, and a hit is the first slot any chunk of the expected document holds. The score field is never read. Its ordering is the contract, its value is not (Relevance scoring), so a metric built on it breaks at the next fusion change.

Reading the table

• recall@k cannot gate a change that only demotes. A relevant result pushed from rank 1 to rank 2 is still retrieved: recall@5 and recall@10 do not move. MRR falls from 1.0 to 0.5 on that row. A ranking change is judged on both; a gate that reads recall alone passes every demotion vacuously.
• The overall row hides a slice collapse. On the baseline, a 30-day recency half-life moved overall MRR from 0.8303 to 0.6667 while the entity slice's recall@10 fell from 1.0000 to 0.1667. Keep one slice per question type; the overall figure summarises them and never substitutes for them.
• Never pin the numbers. They depend on the embedding model. A check that runs on every deploy asserts structure (a metric was produced, a slice did not fall to zero) and a direction against the previous run, not a value.

What a multiplier costs on fused output

Reciprocal rank fusion scores are compressed. A result contributes 1 / (rrf_k + rank) per channel, so at rrf_k = 60 the top ten of one channel span 1/61 = 0.0164 to 1/70 = 0.0143: the whole top ten sits inside 13% of the top score. Any factor applied after fusion is therefore far stronger than it looks.

rrf_k	Ranks a ×0.87 multiplier costs a rank-1 result
60	9
20	3
5	0

The recency blend is such a multiplier, 2 ^ (-age_in_days / recency_half_life_days); 0.87 is six days of age at a 30-day half-life. A smaller rrf_k widens the gaps and buys more room; a larger one less.

The recency blend is never free

The decay applies to memory results and not to the document chunks they share a result list with, so ageing a fact costs it ground against every chunk as well as against fresher facts. On the baseline corpus, whose memory fixtures carry ages of 3 to 400 days, every half-life from 7 days to 20 years lifts the freshness slice's MRR to 1.0, and every one of them costs the entity slice, where the answer is a memory entry competing against undecayed document chunks:

KNOWLEDGE_RECENCY_HALF_LIFE_DAYS	freshness MRR	entity recall@10	entity MRR	overall MRR
0 (off)	0.6667	1.0000	0.9583	0.8303
30	1.0000	0.1667	0.1250	0.6667
90	1.0000	0.4167	0.3500	0.7158
365	1.0000	0.7500	0.5097	0.7506
1825	1.0000	1.0000	0.7736	0.8082
7300	1.0000	1.0000	0.9167	0.8394

Pick a half-life from a run against your own corpus, start long, and prefer scoping the search to memory_ids where freshness is what is actually being ranked.

Retrieval baseline

SOAT's own ranking changes are gated on a versioned golden set, not on judgement. This is a contributor harness: it needs a repository checkout and a test database, and it is not a way to measure a deployment. For that, follow the tutorial.

packages/server/tests/eval/knowledge/golden.json seeds a corpus (module-doc sections, synthetic documents carrying identifiers that occur exactly once, curated memory entries) and scores 55 labeled queries through searchKnowledge at limit: 10.

pnpm --filter @soat/server eval:knowledge                    # score and gate
pnpm --filter @soat/server eval:knowledge --update-baseline  # rewrite the baseline

Scope	recall@5	recall@10	MRR
Overall	0.8909	0.9273	0.8303
exact_token	1.0000	1.0000	1.0000
exact_name	1.0000	1.0000	0.9667
entity	1.0000	1.0000	0.9583
freshness	1.0000	1.0000	0.6667
semantic	0.6000	0.7333	0.5111

These figures are committed as baseline.json. The run exits non-zero when recall@10 or MRR drops below it, overall or for any single kind; recall@5 is reported, not gated. A ranking change lands with the diff of that file as its before/after table.

The freshness kind is the recency blend's own fixture: each query has one answer whose stale near-twin, seeded at a fixture age_days in the past, outranks it while the blend is off. Its 0.6667 is the blend-disabled figure this ships with, not a defect.

Two caveats on the absolute values:

• The embedder is a stand-in. CI has no embedding provider, so the eval substitutes a deterministic feature hasher that ranks by term overlap. Being itself lexical, it starts the exact_token row saturated: the gate can prove hybrid retrieval regresses nothing, but cannot show the lexical channel's win; that proof is a unit test over a chunk whose cosine sits below the floor. What the gate measures reliably is change.
• The corpus tracks these docs. Fixtures naming a source and a section are read from the module docs at seed time, so editing one of those sections moves the numbers. Re-run with --update-baseline and commit the diff.