26 lines
999 B
Markdown
26 lines
999 B
Markdown
# RAG Evaluation
|
||
|
||
Maintain a versioned Golden Dataset and report Recall@1/3/5/10, MRR, NDCG@10, latency P50/P95.
|
||
|
||
Benchmark the full pipeline: Embedding × Retrieval Strategy × Fusion × Reranker.
|
||
|
||
Initial embedding candidates: bge-large-zh, Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, optional 8B.
|
||
|
||
Use synthetic/public/anonymized fixtures only. ARCH-006 expands the checked-in dataset to 32
|
||
fully synthetic queries across refund, technical support, lodging, and reimbursement-material
|
||
sources. Offline Fake Provider regression must remain deterministic; target-provider acceptance
|
||
uses the same records through `make live-acceptance` and reports aggregate metrics only.
|
||
|
||
Current live gates:
|
||
|
||
```text
|
||
Recall@1 >= 0.90
|
||
Recall@5 = 1.00
|
||
MRR >= 0.90
|
||
NDCG@5 >= 0.90
|
||
Latency P95 <= 10000 ms
|
||
```
|
||
|
||
Threshold overrides are allowed only as explicit test-run environment inputs and must be recorded
|
||
with the run result. See `docs/development/LIVE_ACCEPTANCE.md` for safety and cleanup requirements.
|