merge: ARCH-001 provider migration baseline
This commit is contained in:
commit
d5b1d4fe4e
12
README.md
12
README.md
@ -20,7 +20,8 @@ Enterprise internal knowledge infrastructure.
|
||||
- API First + Admin Console
|
||||
- Modular Monolith + async workers + MCP gateway
|
||||
- PostgreSQL + object storage as canonical source of truth
|
||||
- Mochow as the initial retrieval index, behind a replaceable adapter
|
||||
- OceanBase Vector as the target retrieval index, behind a replaceable adapter
|
||||
- Baidu BOS as target object storage; local storage remains for tests/development
|
||||
- Embedding / reranker / parser / LLM providers are replaceable
|
||||
- Raw source is immutable
|
||||
- Compiled Knowledge is derived and traceable
|
||||
@ -82,14 +83,19 @@ Documents remain `processing` until embedding/indexing completes; pipeline
|
||||
failures become `failed` without deleting the immutable original. DEMO-009
|
||||
exposes this pipeline through the versioned upload endpoint.
|
||||
|
||||
DEMO-006 adds a provider-neutral `EmbeddingProvider` and `VectorStore`, a
|
||||
validated Qianfan `bge-large-zh` adapter, and an isolated `pymochow` adapter.
|
||||
DEMO-006 added a provider-neutral `EmbeddingProvider` and `VectorStore`, a
|
||||
validated Qianfan `bge-large-zh` adapter, and the initial isolated `pymochow` adapter.
|
||||
Chunk vectors are upserted with source metadata under a versioned index profile;
|
||||
PostgreSQL records canonical index state while Mochow remains rebuildable. A
|
||||
document becomes `ready` only after every chunk is indexed. See
|
||||
`docs/development/MOCHOW_SETUP.md` for the required environment variables and
|
||||
pre-provisioned table schema.
|
||||
|
||||
The accepted provider migration is tracked by `ARCH-001` through `ARCH-006`:
|
||||
Baidu BOS, OceanBase Vector, and Qianfan `bce-reranker-base` replace the initial
|
||||
demo providers without changing PostgreSQL canonical ownership. Until the runtime
|
||||
cutover task is merged, the paragraphs above describe the current implementation.
|
||||
|
||||
DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The
|
||||
default is Top 20 with a configurable Mochow HNSW `ef`; the in-memory fake uses
|
||||
exact cosine search for deterministic offline regression. Returned objects are
|
||||
|
||||
@ -8,4 +8,4 @@ P0 business tools:
|
||||
- `get_source`
|
||||
- `list_knowledge_bases`
|
||||
|
||||
Every request maps to a user/service identity and uses the same authorization/application services as REST. Do not expose Mochow, Qwen, BGE, MinerU or other infrastructure choices.
|
||||
Every request maps to a user/service identity and uses the same authorization/application services as REST. Do not expose OceanBase Vector, BOS, model names, parsers or other infrastructure choices.
|
||||
|
||||
@ -3,8 +3,22 @@
|
||||
- ADR-001: API First + Admin Console — Accepted
|
||||
- ADR-002: Modular monolith + workers + MCP gateway for V1 — Accepted
|
||||
- ADR-003: PostgreSQL + object storage are canonical — Accepted
|
||||
- ADR-004: Mochow is initial retrieval backend but replaceable — Accepted
|
||||
- ADR-004: Mochow is initial retrieval backend but replaceable — Superseded by ADR-010
|
||||
- ADR-005: Provider abstraction for embedding/reranker/parser/LLM — Accepted
|
||||
- ADR-006: RAG + Knowledge Compilation — Accepted
|
||||
- ADR-007: ACL before/during retrieval — Accepted
|
||||
- ADR-008: MCP hides infrastructure details — Accepted
|
||||
- ADR-009: Baidu BOS is the primary object-storage provider; local storage remains for tests and local development — Accepted
|
||||
- ADR-010: OceanBase Vector is the primary rebuildable retrieval index behind `VectorStore`; PostgreSQL remains canonical — Accepted
|
||||
- ADR-011: Qianfan `bge-large-zh` plus `bce-reranker-base` is the initial Demo model baseline — Accepted
|
||||
|
||||
## Architecture migration boundary
|
||||
|
||||
ADR-009~011 adopt the verified internal platform provider baseline without copying its
|
||||
canonical-database design. ADR-003 remains binding: OceanBase Vector stores only rebuildable
|
||||
retrieval rows, while PostgreSQL and object storage retain canonical metadata and immutable
|
||||
source data.
|
||||
|
||||
The migration retains the Demo's Dense Top20 → Reranker Top5 behavior and current lightweight
|
||||
Parser scope. Top50 retrieval, distance thresholds, dual-query RRF, Marker/Pandoc and LLM-derived
|
||||
index content require separate product and evaluation decisions.
|
||||
|
||||
@ -64,8 +64,8 @@ Wiki / FAQ / SOP / Summary 等派生知识
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ PostgreSQL │ │ Object Store │ │ Mochow │
|
||||
│ Canonical DB │ │ S3/MinIO │ │ Search Index │
|
||||
│ PostgreSQL │ │ Object Store │ │ OceanBase │
|
||||
│ Canonical DB │ │ BOS │ │ Vector Index │
|
||||
└──────────────┘ └──────────────┘ └──────────────┘
|
||||
▲ ▲
|
||||
│ │
|
||||
@ -100,13 +100,13 @@ Wiki / FAQ / SOP / Summary 等派生知识
|
||||
|---|---|---|
|
||||
| Backend | Python + FastAPI | AI / 数据处理生态成熟,适合 MCP、异步任务和模型调用 |
|
||||
| Metadata DB | PostgreSQL | 保存用户、权限、文档、版本、任务、Compiled Knowledge 等 Canonical Data |
|
||||
| Object Storage | S3 Compatible / MinIO | 保存原始文件、解析产物、中间文件 |
|
||||
| Object Storage | Baidu BOS | 保存不可变原始文件、解析产物和中间文件;Local Adapter 仅用于测试/本地开发 |
|
||||
| Task Queue | Celery + RabbitMQ | 文件解析、Embedding、索引、Compilation 等异步任务 |
|
||||
| PDF / OCR Parser | MinerU 优先 | 中文复杂 PDF、扫描件、表格、版面解析能力较强 |
|
||||
| Office Parser | Docling 补充 | DOCX/PPTX/XLSX/HTML 等格式统一解析 |
|
||||
| Vector/Search DB | Mochow | 延续公司现有经验,支持向量、BM25、Hybrid、Filter |
|
||||
| Embedding | 多模型 Benchmark 决定 | 基线 bge-large-zh,对比 Qwen3 Embedding 等 |
|
||||
| Reranker | Qwen3 Reranker 等候选 | V1 正式检索链必选能力 |
|
||||
| Vector/Search DB | OceanBase Vector | 当前主检索索引,必须通过 `VectorStore` 隔离并可从 PostgreSQL 重建 |
|
||||
| Embedding | Qianfan bge-large-zh | 当前 1024 维基线,后续模型变更使用版本化 Shadow Index |
|
||||
| Reranker | Qianfan bce-reranker-base | 当前 Dense Top20 → Reranker Top5 基线 |
|
||||
| LLM | Provider 抽象,云端优先 | 用于 Knowledge Compilation、分类、摘要等 |
|
||||
| MCP | 独立 MCP Gateway | 面向 WorkBuddy 等客户端,隐藏内部技术栈 |
|
||||
| Frontend | Admin Console | 知识库、文档、权限、检索调试、Benchmark、任务管理 |
|
||||
@ -330,17 +330,19 @@ profile_v2
|
||||
|
||||
## 8. Vector Store 设计
|
||||
|
||||
### 当前默认:Mochow
|
||||
### 当前默认:OceanBase Vector
|
||||
|
||||
选择理由:
|
||||
|
||||
- 公司已有使用经验;
|
||||
- 已有百度云体系;
|
||||
- 公司现有知识平台主链已有使用经验;
|
||||
- 可复用现有 OceanBase 运维体系;
|
||||
- 支持 Dense Vector;
|
||||
- 支持 BM25 / Full Text;
|
||||
- 支持 Hybrid Search;
|
||||
- 支持 Metadata / Scalar Filter;
|
||||
- 满足 V1 ACL 下推需求。
|
||||
- 满足当前 Demo 的 Profile 过滤和后续 ACL 下推边界;
|
||||
- `pyobvector` 仅允许出现在 Provider Adapter 内。
|
||||
|
||||
BM25/Hybrid 仍是 V1 产品能力,但不在本轮 Provider 迁移中提前扩展;相关实现必须经
|
||||
独立 Task、契约测试和 RAG 回归后启用。
|
||||
|
||||
### 保留替换能力
|
||||
|
||||
@ -348,15 +350,17 @@ profile_v2
|
||||
|
||||
```text
|
||||
VectorStoreProvider
|
||||
├─ OceanBase Vector (Primary)
|
||||
├─ Mochow
|
||||
├─ Qdrant
|
||||
├─ Milvus
|
||||
└─ pgvector
|
||||
```
|
||||
|
||||
业务 Domain Model 不依赖 Mochow SDK 数据结构。
|
||||
业务 Domain Model 不依赖 OceanBase/`pyobvector` SDK 数据结构。
|
||||
|
||||
Mochow 中的数据全部视为 **可重建索引**,真实 Canonical Data 保存在 PostgreSQL / Object Storage。
|
||||
OceanBase Vector 中的数据全部视为 **可重建索引**。PostgreSQL 保存文档、Chunk、Profile
|
||||
和索引状态等 canonical 数据,BOS 保存不可变原始文件和派生产物。
|
||||
|
||||
---
|
||||
|
||||
@ -441,7 +445,7 @@ Manage
|
||||
User Identity
|
||||
→ Permission Resolver
|
||||
→ Allowed Scope / ACL Filter
|
||||
→ Mochow Hybrid Retrieval
|
||||
→ Provider-neutral Retrieval(OceanBase Vector 当前实现)
|
||||
```
|
||||
|
||||
MCP / REST / Admin Console 必须复用同一权限服务。
|
||||
@ -481,7 +485,8 @@ V1 建议工具:
|
||||
MCP Response 不出现:
|
||||
|
||||
```text
|
||||
Mochow
|
||||
OceanBase Vector
|
||||
BOS
|
||||
Qdrant
|
||||
bge-large-zh
|
||||
Qwen3
|
||||
@ -629,14 +634,14 @@ LLM V1 可继续使用云端 Provider,不要求部署本地大模型。
|
||||
- Markdown / JSON 中间产物;
|
||||
- 需要持久化的大对象。
|
||||
|
||||
### Mochow
|
||||
### OceanBase Vector
|
||||
|
||||
仅保存:
|
||||
|
||||
- Dense Vector;
|
||||
- Sparse/BM25 Index;
|
||||
- Retrieval metadata;
|
||||
- ACL 所需过滤字段。
|
||||
- Profile/ACL 所需过滤字段;
|
||||
- 经独立能力验证后启用的 Sparse/BM25 索引。
|
||||
|
||||
数据库损坏或更换时,可由 Canonical Data 全量重建。
|
||||
|
||||
@ -655,12 +660,12 @@ Reverse Proxy / Ingress
|
||||
|
||||
PostgreSQL
|
||||
RabbitMQ
|
||||
Object Storage
|
||||
Baidu BOS
|
||||
|
||||
CPU Worker
|
||||
GPU Worker
|
||||
|
||||
Mochow
|
||||
OceanBase Vector
|
||||
|
||||
Observability
|
||||
```
|
||||
@ -733,12 +738,12 @@ WorkBuddy 可以通过 MCP 完成知识搜索和 Source 获取。
|
||||
```text
|
||||
Backend Python + FastAPI
|
||||
Metadata PostgreSQL
|
||||
Object Storage S3 Compatible / MinIO
|
||||
Object Storage Baidu BOS(Local Adapter 仅测试/本地开发)
|
||||
Queue RabbitMQ + Celery
|
||||
Parser MinerU 主力 + Docling 补充
|
||||
Vector Store Mochow Primary,Adapter 可替换
|
||||
Embedding bge-large-zh 基线,多模型 Benchmark 后决定生产模型
|
||||
Reranker Qwen3 Reranker 等候选,V1 必选
|
||||
Vector Store OceanBase Vector Primary,Adapter 可替换,索引可重建
|
||||
Embedding Qianfan bge-large-zh(1024 维)
|
||||
Reranker Qianfan bce-reranker-base,Dense Top20 → Top5
|
||||
LLM Provider 抽象,允许云端
|
||||
Frontend Admin Console
|
||||
Protocol REST API + MCP
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
|
||||
该 Demo 的目标不是完成完整企业级知识平台,而是优先验证最核心的技术闭环:
|
||||
|
||||
> 文件上传 → 文档解析/清洗 → Chunk → Embedding → Mochow 入库 → 检索召回 → Reranker → 来源返回 → 简单管理页面
|
||||
> 文件上传 → 文档解析/清洗 → Chunk → Embedding → OceanBase Vector 入库 → 检索召回 → Reranker → 来源返回 → 简单管理页面
|
||||
|
||||
当前 Demo Scope 的优先级高于完整 V1 PRD。
|
||||
|
||||
@ -20,7 +20,7 @@
|
||||
|
||||
1. 中文企业文档是否能够稳定上传和解析;
|
||||
2. 清洗后的内容是否可以形成质量可接受的 Chunk;
|
||||
3. 使用现有 Embedding + Mochow 能否完成稳定召回;
|
||||
3. 使用现有 Embedding + OceanBase Vector 能否完成稳定召回;
|
||||
4. Reranker 是否能提升最终 TopK 结果质量;
|
||||
5. 检索结果是否能够准确返回文件名、页码、章节和原文;
|
||||
6. 非研发人员是否能够通过简单页面完成上传和检索验证。
|
||||
@ -49,13 +49,13 @@ FastAPI
|
||||
│ ↓
|
||||
│ Embedding
|
||||
│ ↓
|
||||
│ Mochow
|
||||
│ OceanBase Vector
|
||||
│
|
||||
└── Search
|
||||
↓
|
||||
Query Embedding
|
||||
↓
|
||||
Mochow TopN
|
||||
OceanBase Vector TopN
|
||||
↓
|
||||
Reranker
|
||||
↓
|
||||
@ -70,7 +70,7 @@ FastAPI
|
||||
SQLite / PostgreSQL
|
||||
└── document / chunk metadata
|
||||
|
||||
Local Storage / MinIO
|
||||
Local Storage / BOS
|
||||
└── original files
|
||||
```
|
||||
|
||||
@ -84,13 +84,13 @@ Demo 阶段优先简单实现,不引入不必要的基础设施。
|
||||
|---|---|---|
|
||||
| Backend | FastAPI | 简单、开发效率高 |
|
||||
| Metadata DB | SQLite 或 PostgreSQL | 单机 Demo 可 SQLite;已有环境可直接 PostgreSQL |
|
||||
| File Storage | 本地目录 | Demo 阶段优先简单;后续可替换 MinIO/S3 |
|
||||
| File Storage | 百度 BOS | Local Adapter 仅用于测试和本地开发,目标环境使用 BOS |
|
||||
| Parser | PyMuPDF + python-docx 或 MinerU | 优先快速跑通;复杂 PDF 后续增强 |
|
||||
| Cleaner | 自定义基础清洗 | 处理空行、空白、异常换行等 |
|
||||
| Chunker | 标题/段落优先 + Token 限制 | 暂不设计复杂 Chunk 策略 |
|
||||
| Embedding | bge-large-zh | 延续公司现有百度千帆方案 |
|
||||
| Vector DB | Mochow | 当前默认,不做多数据库实现 |
|
||||
| Reranker | Qwen3 Reranker 或现有可调用 Reranker | Demo 中保留 |
|
||||
| Vector DB | OceanBase Vector | 当前默认,通过 `VectorStore` 隔离 SDK,不做多数据库运行时切换 |
|
||||
| Reranker | `bce-reranker-base` | 延续当前可调用的千帆 Reranker,Demo 中保留 |
|
||||
| Frontend | 简单 React/Vue 或 FastAPI HTML | 以最小可用为目标 |
|
||||
| Deployment | 本地 / 单机 Docker | 暂不考虑 K8s |
|
||||
|
||||
@ -171,9 +171,9 @@ content
|
||||
|
||||
Demo 阶段不实现多 Embedding Benchmark 平台和动态模型切换,但代码层建议保留简单接口,避免业务逻辑直接写死调用细节。
|
||||
|
||||
### 5.6 Mochow 入库
|
||||
### 5.6 OceanBase Vector 入库
|
||||
|
||||
将 Chunk 写入 Mochow,至少保存:
|
||||
将 Chunk 写入 OceanBase Vector 专用索引表,至少保存:
|
||||
|
||||
```text
|
||||
chunk_id
|
||||
@ -187,7 +187,9 @@ embedding
|
||||
|
||||
支持:Upsert、按 document_id 删除相关 Chunk、Dense Search。
|
||||
|
||||
Demo 阶段不实现 Qdrant/Milvus/pgvector,也不实现完整 VectorStore Adapter 体系,但避免 Mochow SDK 调用散落在整个项目中。
|
||||
Demo 阶段不实现 Qdrant/Milvus/pgvector,也不实现多 Vector Store 动态切换,但避免
|
||||
OceanBase/pyobvector 调用散落在整个项目中。OceanBase Vector 行仍是可重建索引,不能替代
|
||||
PostgreSQL canonical Chunk。
|
||||
|
||||
### 5.7 Dense Retrieval
|
||||
|
||||
@ -196,7 +198,7 @@ Query
|
||||
↓
|
||||
bge-large-zh Embedding
|
||||
↓
|
||||
Mochow Dense Search
|
||||
OceanBase Vector Dense Search
|
||||
↓
|
||||
Top 20 Candidates
|
||||
```
|
||||
@ -246,7 +248,8 @@ DELETE /api/documents/{id}
|
||||
POST /api/search
|
||||
```
|
||||
|
||||
Demo 阶段删除可以直接执行:删除原始文件 + 删除 Metadata + 删除 Mochow 对应 Chunk,暂不要求 Soft Delete。
|
||||
Demo 阶段删除可以直接执行:删除 BOS/本地原始文件 + 删除 Metadata + 删除 OceanBase
|
||||
Vector 对应 Chunk,暂不要求 Soft Delete。
|
||||
|
||||
Search Request 示例:
|
||||
|
||||
@ -357,15 +360,16 @@ Reranker
|
||||
|
||||
```text
|
||||
DEMO-001 项目骨架 + FastAPI + 配置
|
||||
DEMO-002 Document 数据模型 + 文件存储
|
||||
DEMO-002 Document 数据模型 + 本地文件存储
|
||||
DEMO-003 PDF/DOCX/TXT/MD 上传
|
||||
DEMO-004 Parser + Cleaner
|
||||
DEMO-005 Chunk Pipeline
|
||||
DEMO-006 bge-large-zh Embedding + Mochow 入库
|
||||
DEMO-006 bge-large-zh Embedding + 初始 Mochow 入库(将由 ARCH-003 迁移)
|
||||
DEMO-007 Dense Retrieval
|
||||
DEMO-008 Reranker + Source Result
|
||||
DEMO-009 Document/Search REST API
|
||||
DEMO-010 简单 Admin Console
|
||||
ARCH-001~006 Provider 架构迁移与真实验收
|
||||
```
|
||||
|
||||
可选:
|
||||
@ -422,7 +426,7 @@ Upload
|
||||
→ Clean
|
||||
→ Chunk
|
||||
→ Embedding
|
||||
→ Mochow
|
||||
→ OceanBase Vector
|
||||
```
|
||||
|
||||
完整跑通。
|
||||
@ -432,7 +436,7 @@ Upload
|
||||
```text
|
||||
Query
|
||||
→ Embedding
|
||||
→ Mochow Dense Top20
|
||||
→ OceanBase Vector Dense Top20
|
||||
→ Reranker
|
||||
→ Top5
|
||||
```
|
||||
@ -464,7 +468,8 @@ Demo 阶段不设严格生产指标。
|
||||
|
||||
## 14. Codex 开发约束
|
||||
|
||||
Codex 当前只能实现 `DEMO-001 ~ DEMO-010`。
|
||||
Codex 当前只能实现 `DEMO-001 ~ DEMO-010` 和已批准的 `ARCH-001 ~ ARCH-006`。
|
||||
ARCH 迁移任务不授权实现下列被排除的完整 V1 能力。
|
||||
|
||||
除非明确要求,否则禁止提前实现权限、审计、Knowledge Compilation、多数据库、多模型平台、微服务、Kubernetes 和企业级 Job Queue。
|
||||
|
||||
@ -495,7 +500,8 @@ docs/demo/DEMO_SCOPE.md
|
||||
>
|
||||
完整 V1 PRD
|
||||
|
||||
只允许实现 DEMO-001 ~ DEMO-010。
|
||||
只允许实现 DEMO-001 ~ DEMO-010 以及本文件明确列出的 ARCH-001 ~ ARCH-006。
|
||||
Provider 架构迁移任务按顺序完成后,再继续 DEMO-010。
|
||||
|
||||
当前 Demo 的目标只有:
|
||||
|
||||
@ -503,7 +509,7 @@ Upload
|
||||
→ Parse/Clean
|
||||
→ Chunk
|
||||
→ bge-large-zh
|
||||
→ Mochow
|
||||
→ OceanBase Vector
|
||||
→ Dense Retrieval
|
||||
→ Reranker
|
||||
→ Source Result
|
||||
@ -529,7 +535,7 @@ Upload
|
||||
1. 当前仓库状态;
|
||||
2. 为完成 Demo 需要保留哪些现有结构;
|
||||
3. 哪些完整 V1 设计可以暂时不实现;
|
||||
4. DEMO-001 ~ DEMO-010 的建议执行顺序;
|
||||
4. DEMO-001 ~ DEMO-010 与 ARCH-001 ~ ARCH-006 的建议执行顺序;
|
||||
5. 下一步应该实施的第一个 Task。
|
||||
```
|
||||
|
||||
|
||||
82
docs/plans/ARCHITECTURE_MIGRATION_PLAN.md
Normal file
82
docs/plans/ARCHITECTURE_MIGRATION_PLAN.md
Normal file
@ -0,0 +1,82 @@
|
||||
# Provider 架构迁移计划
|
||||
|
||||
## 目标基线
|
||||
|
||||
本计划依据 2026-09-01 核对的内部平台配置现状调整 EKP Provider 选型,同时保留本仓库
|
||||
已经 Accepted 的 canonical-data 和 API-first 原则。
|
||||
|
||||
```text
|
||||
PostgreSQL canonical Metadata / Chunk / Index State
|
||||
Baidu BOS immutable source and derived artifacts
|
||||
OceanBase Vector rebuildable Dense retrieval index
|
||||
Qianfan bge-large-zh 1024-dimensional Embedding
|
||||
Qianfan BCE Reranker bce-reranker-base
|
||||
```
|
||||
|
||||
Provider、内部地址、数据库账户和凭据不得出现在 API/MCP Contract、日志、镜像或提交中。
|
||||
本地测试继续使用 LocalObjectStorage、FakeVectorStore、FakeEmbeddingProvider 和
|
||||
FakeRerankerProvider。
|
||||
|
||||
## 明确保留的行为
|
||||
|
||||
- PostgreSQL 保持 canonical database,OceanBase 只保存可重建索引;
|
||||
- 原始文件不可变;
|
||||
- Dense Top20 → Reranker Top5;
|
||||
- 当前 PDF/DOCX/TXT/Markdown Parser 和 `structure_v1` Chunk;
|
||||
- `/api/v1` 和 MCP 继续只暴露业务语义;
|
||||
- `index_profile_id` 过滤在召回阶段下推;
|
||||
- 新索引通过 shadow 表/Profile 构建和验证,不原地覆盖旧索引。
|
||||
|
||||
## 明确不随迁移引入
|
||||
|
||||
- OceanBase 取代 PostgreSQL canonical database;
|
||||
- Top50、距离阈值、标准答案捷径、双查询和 RRF;
|
||||
- Marker/Pandoc、LLM 摘要/问题/关键词索引;
|
||||
- RBAC/ACL、异步队列、MCP、Hybrid/BM25 或 Admin Console 扩展;
|
||||
- 旧系统 `.env`/镜像配置加载方式或任何现有敏感凭据。
|
||||
|
||||
## Task 与验收
|
||||
|
||||
### ARCH-001 架构和执行基线
|
||||
|
||||
更新 ADR、PRD、Architecture、Demo Scope、Tasks 和 Progress,消除目标 Provider 歧义。
|
||||
|
||||
### ARCH-002 BOS ObjectStorage
|
||||
|
||||
新增 provider-neutral BOS Adapter、配置、脱敏错误和离线单元测试。Local Adapter 保留为
|
||||
测试/开发实现。
|
||||
|
||||
依赖:ARCH-001。
|
||||
|
||||
### ARCH-003 OceanBase VectorStore
|
||||
|
||||
新增 OceanBase Vector Adapter 和专用索引 DDL,覆盖 Upsert、文档/Profile 删除、Dense
|
||||
TopN、Cosine Distance 到统一高分优先 `score` 的转换,以及 profile 过滤。
|
||||
|
||||
依赖:ARCH-002。
|
||||
|
||||
### ARCH-004 BCE Reranker
|
||||
|
||||
将运行基线改为 `bce-reranker-base`,按真实接口约束更新配置、Adapter、单元测试和 RAG
|
||||
Evaluation;保持 Top5 API 行为。
|
||||
|
||||
依赖:ARCH-003。
|
||||
|
||||
### ARCH-005 Runtime Cutover
|
||||
|
||||
将 API 运行时切换到 BOS、OceanBase Vector 和 BCE Reranker;提供从 PostgreSQL canonical
|
||||
Chunk 构建新 shadow index 的可重试命令,验证后激活新 Profile,再清理 Mochow 运行配置。
|
||||
|
||||
依赖:ARCH-004。
|
||||
|
||||
### ARCH-006 真实验收
|
||||
|
||||
在受控测试资源中执行 PostgreSQL → BOS → Qianfan Embedding → OceanBase Vector → BCE
|
||||
Reranker → canonical Source Result 全链路,并完成 30~50 条脱敏 Golden Query 验收。
|
||||
|
||||
依赖:ARCH-005、轮换后的运行凭据、已授权的测试数据库/表和 BOS 前缀。
|
||||
|
||||
## 安全边界
|
||||
|
||||
内部配置清单中的现有凭据按已暴露处理,只能使用轮换后的运行时 Secret。禁止读取凭据后
|
||||
生成 `.env`、测试快照或调试日志;真实集成测试必须使用专用测试资源并清理临时对象。
|
||||
@ -1,15 +1,15 @@
|
||||
# 项目进度
|
||||
|
||||
当前阶段:Knowledge Retrieval Demo v0.1
|
||||
当前阶段:Provider 架构迁移
|
||||
|
||||
已完成:DEMO-001、DEMO-002、DEMO-003、DEMO-004、DEMO-005、DEMO-006、DEMO-007、DEMO-008、DEMO-009
|
||||
已完成:DEMO-001~DEMO-009、ARCH-001
|
||||
进行中:无
|
||||
阻塞:无
|
||||
|
||||
下一任务:DEMO-010
|
||||
下一任务:ARCH-002
|
||||
|
||||
完整 V1 待办将在 Demo Scope 完成后继续执行。当前任务顺序以
|
||||
`docs/demo/DEMO_SCOPE.md` 为准。
|
||||
Provider 架构迁移按 `ARCH-001 → ARCH-002 → ARCH-003 → ARCH-004 → ARCH-005 → ARCH-006`
|
||||
执行;每个 Task 验证后独立推送并合并 `main`。全部迁移完成后再继续 DEMO-010。
|
||||
|
||||
## 任务记录
|
||||
|
||||
@ -132,3 +132,14 @@
|
||||
业务接口返回 `503`,上游异常使用收敛后的错误响应,避免泄露凭证或厂商正文。
|
||||
- 新增完全离线的 REST 全链路测试,覆盖上传、列表、详情、Dense/Reranker 检索、来源字段、
|
||||
直接删除、OpenAPI 路径、输入校验、404、Provider 延迟加载及删除失败保护。
|
||||
|
||||
### ARCH-001
|
||||
|
||||
- 接受 BOS、OceanBase Vector、千帆 `bge-large-zh` 和 `bce-reranker-base` 作为新的 V1
|
||||
Provider 基线;PostgreSQL 继续保存 canonical 数据,OceanBase 只保存可重建索引。
|
||||
- 新增 ADR-009~ADR-011 和独立迁移计划,将 ARCH-002~ARCH-006 的范围、依赖、验收顺序
|
||||
与安全边界写入任务清单。
|
||||
- 保留 Demo 的 Dense Top20 → Reranker Top5、现有 Parser/Chunk、API/MCP 业务契约;不随迁移
|
||||
引入 Top50、RRF、Marker/Pandoc、LLM 派生索引、ACL 或 Hybrid/BM25 扩展。
|
||||
- 本任务只调整架构和产品文档,不修改运行时代码;旧系统地址、账号和凭据均未写入仓库,
|
||||
真实验收只允许使用轮换后的 Secret 与受控测试资源。
|
||||
|
||||
@ -1,5 +1,28 @@
|
||||
# V1 Tasks
|
||||
|
||||
## Architecture provider migration
|
||||
|
||||
- [x] ARCH-001 Architecture decision, PRD, Scope and execution baseline
|
||||
- [ ] ARCH-002 BOS ObjectStorage adapter and runtime configuration
|
||||
- [ ] ARCH-003 OceanBase VectorStore adapter and index schema
|
||||
- [ ] ARCH-004 Qianfan `bce-reranker-base` alignment
|
||||
- [ ] ARCH-005 Runtime cutover, shadow rebuild workflow and legacy configuration cleanup
|
||||
- [ ] ARCH-006 Controlled live integration and RAG acceptance
|
||||
|
||||
Dependencies:
|
||||
|
||||
```text
|
||||
ARCH-001
|
||||
→ ARCH-002
|
||||
→ ARCH-003
|
||||
→ ARCH-004
|
||||
→ ARCH-005
|
||||
→ ARCH-006
|
||||
```
|
||||
|
||||
Each architecture task uses an independent branch and is merged into `main` only after its own
|
||||
validation. Provider credentials and internal endpoints are runtime inputs and never task artifacts.
|
||||
|
||||
## Phase 0
|
||||
- [ ] EKP-001 Repository bootstrap
|
||||
- [ ] EKP-002 Backend Python toolchain
|
||||
@ -18,7 +41,7 @@
|
||||
- [ ] EKP-015 Document + DocumentVersion model
|
||||
- [ ] EKP-016 ACL model
|
||||
- [ ] EKP-017 Job + audit models
|
||||
- [ ] EKP-020 ObjectStorage abstraction + S3 adapter
|
||||
- [ ] EKP-020 ObjectStorage abstraction + BOS adapter
|
||||
- [ ] EKP-021 Document upload API
|
||||
- [ ] EKP-022 Source hash/dedup/version activation
|
||||
- [ ] EKP-023 Async job framework
|
||||
@ -35,13 +58,13 @@
|
||||
- [ ] EKP-042 Qwen embedding adapter
|
||||
- [ ] EKP-043 IndexProfile model
|
||||
- [ ] EKP-044 VectorStore + fake
|
||||
- [ ] EKP-045 Mochow adapter
|
||||
- [ ] EKP-045 OceanBase Vector adapter
|
||||
- [ ] EKP-046 Index build + shadow index lifecycle
|
||||
- [ ] EKP-050 Permission resolver + ACL retrieval filters
|
||||
- [ ] EKP-051 Dense retrieval
|
||||
- [ ] EKP-052 BM25/sparse retrieval
|
||||
- [ ] EKP-053 Hybrid fusion
|
||||
- [ ] EKP-054 RerankerProvider + fake + Qwen adapter
|
||||
- [ ] EKP-054 RerankerProvider + fake + Qianfan BCE adapter
|
||||
- [ ] EKP-055 Source enrichment + Search service/API
|
||||
- [ ] EKP-056 Retrieval diagnostics / Search Lab backend
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
1. Repository / developer harness
|
||||
2. Canonical domain and ACL
|
||||
3. Document ingestion and parsing
|
||||
4. Embedding / Index Profile / Mochow adapter
|
||||
4. Embedding / Index Profile / OceanBase Vector adapter
|
||||
5. ACL-aware hybrid retrieval + reranker
|
||||
6. MCP gateway and WorkBuddy verification
|
||||
7. Admin Console
|
||||
|
||||
@ -112,7 +112,8 @@ Admin Console 是官方管理客户端,主要解决:
|
||||
|
||||
WorkBuddy、Cursor、Claude Code 或其他 Agent 不应该知道:
|
||||
|
||||
- Mochow;
|
||||
- OceanBase Vector;
|
||||
- BOS;
|
||||
- Qdrant;
|
||||
- Milvus;
|
||||
- bge-large-zh;
|
||||
@ -800,21 +801,24 @@ VectorStoreProvider
|
||||
|
||||
抽象。
|
||||
|
||||
默认首选:
|
||||
当前默认:
|
||||
|
||||
```text
|
||||
Mochow
|
||||
OceanBase Vector
|
||||
```
|
||||
|
||||
原因:
|
||||
|
||||
公司已有使用经验,同时 Mochow 当前官方能力已经支持向量检索、BM25、Hybrid Search 和标量过滤。
|
||||
公司现有知识平台主检索链已经使用 OceanBase Vector,并可与既有 OceanBase 运维体系复用。
|
||||
V1 通过标量条件下推满足 Profile 与 ACL 过滤;BM25/Hybrid 能力必须继续通过统一检索契约
|
||||
演进,不把特定数据库查询结构写入业务层。
|
||||
|
||||
但系统不得使用 Mochow-specific data model 污染业务 Domain Model。
|
||||
系统不得使用 OceanBase/`pyobvector`-specific data model 污染业务 Domain Model。
|
||||
|
||||
未来可以新增:
|
||||
|
||||
```text
|
||||
Mochow
|
||||
Qdrant
|
||||
Milvus
|
||||
pgvector
|
||||
@ -913,15 +917,13 @@ Provider:
|
||||
RerankerProvider
|
||||
```
|
||||
|
||||
初始候选:
|
||||
当前基线与后续候选:
|
||||
|
||||
- Qwen3-Reranker-0.6B;
|
||||
- Qwen3-Reranker-4B;
|
||||
- Qwen3-Reranker-8B;
|
||||
- bce-reranker-base;
|
||||
- bce-reranker-base(当前基线);
|
||||
- Qwen3-Reranker 系列(后续 Benchmark 候选);
|
||||
- Local Reranker。
|
||||
|
||||
千帆当前已经提供 Qwen3 Reranker 系列,最大输入支持 32K。
|
||||
模型与参数由版本化 Index/Profile 配置记录,业务 API 不暴露模型名。
|
||||
|
||||
---
|
||||
|
||||
@ -1871,15 +1873,15 @@ Embedding × Retrieval × Reranker
|
||||
|
||||
```text
|
||||
bge-large-zh
|
||||
+ Mochow Dense
|
||||
+ OceanBase Vector Dense
|
||||
+ No Reranker
|
||||
|
||||
bge-large-zh
|
||||
+ Mochow Hybrid
|
||||
+ Qwen3 Reranker
|
||||
+ OceanBase Vector Dense
|
||||
+ bce-reranker-base
|
||||
|
||||
Qwen3 Embedding
|
||||
+ Mochow Hybrid
|
||||
+ OceanBase Vector Hybrid(能力验证后)
|
||||
+ Qwen3 Reranker
|
||||
```
|
||||
|
||||
@ -2168,7 +2170,7 @@ MCP Error Rate
|
||||
10. Document Version;
|
||||
11. Hash 去重;
|
||||
12. Embedding Provider;
|
||||
13. Mochow Provider;
|
||||
13. OceanBase Vector Provider;
|
||||
14. Dense Search;
|
||||
15. BM25;
|
||||
16. Hybrid Retrieval;
|
||||
@ -2291,9 +2293,9 @@ Embedding 和 Reranker 可以通过 Provider 配置替换。
|
||||
|
||||
## Vector DB
|
||||
|
||||
Mochow 可以作为默认生产 Vector Store。
|
||||
OceanBase Vector 作为当前默认生产 Vector Store。
|
||||
|
||||
业务代码不存在必须依赖 Mochow SDK 数据结构的 API Contract。
|
||||
业务代码不存在必须依赖 OceanBase/`pyobvector` SDK 数据结构的 API Contract。
|
||||
|
||||
---
|
||||
|
||||
@ -2338,11 +2340,14 @@ get_source
|
||||
|
||||
# 73. 当前技术基线
|
||||
|
||||
PRD 阶段暂定但不最终锁定:
|
||||
当前 V1 架构基线:
|
||||
|
||||
```text
|
||||
Primary Vector Store:
|
||||
Mochow
|
||||
OceanBase Vector
|
||||
|
||||
Primary Object Storage:
|
||||
Baidu BOS
|
||||
|
||||
Embedding Baseline:
|
||||
bge-large-zh
|
||||
@ -2351,7 +2356,7 @@ Embedding Benchmark:
|
||||
Qwen3 Embedding
|
||||
|
||||
Reranker:
|
||||
Qwen3 Reranker / equivalent
|
||||
bce-reranker-base
|
||||
|
||||
Document Parsing:
|
||||
Benchmark MinerU / Docling
|
||||
@ -2387,7 +2392,8 @@ API First + Admin Console
|
||||
|
||||
**DEC-002**
|
||||
|
||||
Mochow 继续作为生产首选候选,同时必须保留 Vector Store 替换能力。
|
||||
OceanBase Vector 作为当前生产主检索索引,同时必须保留 Vector Store 替换能力。
|
||||
PostgreSQL 继续保存 canonical 数据,Vector Store 只保存可重建索引。
|
||||
|
||||
---
|
||||
|
||||
@ -2425,7 +2431,7 @@ RAG + Knowledge Compilation
|
||||
|
||||
**DEC-007**
|
||||
|
||||
MCP 只能暴露业务能力,禁止泄露 Mochow、Embedding Model 等底层实现。
|
||||
MCP 只能暴露业务能力,禁止泄露 OceanBase Vector、BOS、Embedding Model 等底层实现。
|
||||
|
||||
---
|
||||
|
||||
@ -2449,7 +2455,7 @@ PRD 评审通过后进入《总体技术架构设计》。
|
||||
6. MinerU vs Docling;
|
||||
7. OCR Pipeline;
|
||||
8. Chunk Strategy;
|
||||
9. Mochow Schema;
|
||||
9. OceanBase Vector Index Schema;
|
||||
10. VectorStore Adapter;
|
||||
11. Embedding Provider;
|
||||
12. Reranker Serving;
|
||||
@ -2473,7 +2479,9 @@ PRD 评审通过后进入《总体技术架构设计》。
|
||||
```text
|
||||
Qianfan
|
||||
+
|
||||
Mochow
|
||||
BOS
|
||||
+
|
||||
OceanBase Vector
|
||||
+
|
||||
Local Parsing
|
||||
```
|
||||
@ -2511,7 +2519,9 @@ Replaceable Embedding
|
||||
+
|
||||
Replaceable Reranker
|
||||
+
|
||||
Mochow Primary
|
||||
BOS Object Storage
|
||||
+
|
||||
OceanBase Vector Primary
|
||||
+
|
||||
Cloud LLM
|
||||
```
|
||||
@ -2524,9 +2534,12 @@ Cloud LLM
|
||||
|
||||
# 77. 调研依据备注
|
||||
|
||||
当前百度千帆模型列表表明 bge-large-zh 与 Qwen3 Embedding、Qwen3 Reranker 可以作为同平台 Benchmark 候选,因此早期无需为模型实验同时引入多个云厂商。
|
||||
内部最新配置基线确认现有知识平台主链使用千帆 `bge-large-zh`、`bce-reranker-base`、
|
||||
BOS 与 OceanBase Vector。该信息只用于确定 Provider 选型,不把连接地址、账号、密钥或
|
||||
历史环境加载方式写入产品文档和代码。
|
||||
|
||||
Mochow 当前官方 API 已支持向量检索与 BM25 全文检索组合的 Hybrid Search,同时可施加标量 Filter,与企业知识库的 Dense + Sparse + ACL Retrieval 需求相匹配。
|
||||
OceanBase Vector 的生产接入必须验证向量检索、标量 Filter、索引参数和版本兼容性;
|
||||
BM25/Hybrid 在相应任务验收前仍按产品能力要求保留,不在本轮 Provider 迁移中提前扩展。
|
||||
|
||||
WorkBuddy Enterprise 当前公开 MCP 文档支持自定义 MCP Server,并提供传输、权限、安全、远程 API Server 以及 iWiki 配置示例,因此“企业知识服务 → MCP → WorkBuddy”的产品路线具备正式产品支持基础。
|
||||
|
||||
@ -2534,4 +2547,4 @@ Karpathy 的 LLM Wiki 原始思路强调 immutable raw sources 与由 LLM 持续
|
||||
|
||||
---
|
||||
|
||||
**PRD v0.1 END**
|
||||
**PRD v0.1 END**
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user