knowledge-platform/docs/evaluation/RAG_EVALUATION.md

26 lines
999 B
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# RAG Evaluation
Maintain a versioned Golden Dataset and report Recall@1/3/5/10, MRR, NDCG@10, latency P50/P95.
Benchmark the full pipeline: Embedding × Retrieval Strategy × Fusion × Reranker.
Initial embedding candidates: bge-large-zh, Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, optional 8B.
Use synthetic/public/anonymized fixtures only. ARCH-006 expands the checked-in dataset to 32
fully synthetic queries across refund, technical support, lodging, and reimbursement-material
sources. Offline Fake Provider regression must remain deterministic; target-provider acceptance
uses the same records through `make live-acceptance` and reports aggregate metrics only.
Current live gates:
```text
Recall@1 >= 0.90
Recall@5 = 1.00
MRR >= 0.90
NDCG@5 >= 0.90
Latency P95 <= 10000 ms
```
Threshold overrides are allowed only as explicit test-run environment inputs and must be recorded
with the run result. See `docs/development/LIVE_ACCEPTANCE.md` for safety and cleanup requirements.