999 B
Raw Permalink Blame History

RAG Evaluation

Maintain a versioned Golden Dataset and report Recall@1/3/5/10, MRR, NDCG@10, latency P50/P95.

Benchmark the full pipeline: Embedding × Retrieval Strategy × Fusion × Reranker.

Initial embedding candidates: bge-large-zh, Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, optional 8B.

Use synthetic/public/anonymized fixtures only. ARCH-006 expands the checked-in dataset to 32 fully synthetic queries across refund, technical support, lodging, and reimbursement-material sources. Offline Fake Provider regression must remain deterministic; target-provider acceptance uses the same records through make live-acceptance and reports aggregate metrics only.

Current live gates:

Recall@1  >= 0.90
Recall@5  = 1.00
MRR       >= 0.90
NDCG@5    >= 0.90
Latency P95 <= 10000 ms

Threshold overrides are allowed only as explicit test-run environment inputs and must be recorded with the run result. See docs/development/LIVE_ACCEPTANCE.md for safety and cleanup requirements.