999 B
RAG Evaluation
Maintain a versioned Golden Dataset and report Recall@1/3/5/10, MRR, NDCG@10, latency P50/P95.
Benchmark the full pipeline: Embedding × Retrieval Strategy × Fusion × Reranker.
Initial embedding candidates: bge-large-zh, Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, optional 8B.
Use synthetic/public/anonymized fixtures only. ARCH-006 expands the checked-in dataset to 32
fully synthetic queries across refund, technical support, lodging, and reimbursement-material
sources. Offline Fake Provider regression must remain deterministic; target-provider acceptance
uses the same records through make live-acceptance and reports aggregate metrics only.
Current live gates:
Recall@1 >= 0.90
Recall@5 = 1.00
MRR >= 0.90
NDCG@5 >= 0.90
Latency P95 <= 10000 ms
Threshold overrides are allowed only as explicit test-run environment inputs and must be recorded
with the run result. See docs/development/LIVE_ACCEPTANCE.md for safety and cleanup requirements.