docs: add reviewed product and architecture specifications

This commit is contained in:
d-robotics 2026-08-31 17:57:00 +08:00
commit 069c1964e5
49 changed files with 3783 additions and 0 deletions

15
.editorconfig Normal file
View File

@ -0,0 +1,15 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
[*.py]
indent_size = 4
[Makefile]
indent_style = tab

45
.env.example Normal file
View File

@ -0,0 +1,45 @@
APP_ENV=development
APP_NAME=enterprise-knowledge-platform
LOG_LEVEL=INFO
API_HOST=0.0.0.0
API_PORT=8000
DATABASE_URL=postgresql+psycopg://ekp:ekp@localhost:5432/ekp
REDIS_URL=redis://localhost:6379/0
RABBITMQ_URL=amqp://guest:guest@localhost:5672//
OBJECT_STORAGE_PROVIDER=s3
OBJECT_STORAGE_ENDPOINT=http://localhost:9000
OBJECT_STORAGE_ACCESS_KEY=
OBJECT_STORAGE_SECRET_KEY=
OBJECT_STORAGE_BUCKET=ekp
VECTOR_STORE_PROVIDER=mochow
MOCHOW_ENDPOINT=
MOCHOW_API_KEY=
MOCHOW_DATABASE=
MOCHOW_TABLE=
EMBEDDING_PROVIDER=qianfan
EMBEDDING_MODEL=bge-large-zh
EMBEDDING_ENDPOINT=
EMBEDDING_API_KEY=
RERANKER_PROVIDER=qianfan
RERANKER_MODEL=
RERANKER_ENDPOINT=
RERANKER_API_KEY=
LLM_PROVIDER=qianfan
LLM_MODEL=
LLM_ENDPOINT=
LLM_API_KEY=
PARSER_PROVIDER=mineru
OCR_PROVIDER=mineru
JWT_SECRET=
API_KEY_PEPPER=
ENABLE_KNOWLEDGE_COMPILATION=true
ENABLE_MCP=true

17
.gitattributes vendored Normal file
View File

@ -0,0 +1,17 @@
* text=auto eol=lf
*.sh text eol=lf
*.py text eol=lf
*.md text eol=lf
*.yaml text eol=lf
*.yml text eol=lf
*.json text eol=lf
*.toml text eol=lf
*.ts text eol=lf
*.tsx text eol=lf
*.png binary
*.jpg binary
*.jpeg binary
*.pdf binary
*.docx binary
*.pptx binary
*.xlsx binary

60
.gitignore vendored Normal file
View File

@ -0,0 +1,60 @@
# Environment / secrets
.env
.env.*
!.env.example
# Python
__pycache__/
*.py[cod]
.pytest_cache/
.mypy_cache/
.pyright/
.ruff_cache/
.coverage
htmlcov/
.venv/
venv/
# Node / frontend
node_modules/
dist/
build/
.next/
coverage/
*.tsbuildinfo
# IDE / OS
.idea/
.vscode/
*.swp
*.swo
.DS_Store
Thumbs.db
# Runtime / local data
logs/
tmp/
data/
storage/
uploads/
models/
artifacts/
reports/
# Secrets / credentials
*.pem
*.key
*.p12
credentials.json
secrets.json
# Model artifacts
*.bin
*.safetensors
*.onnx
*.pt
*.pth
# Local DB
*.sqlite
*.sqlite3

46
AGENTS.md Normal file
View File

@ -0,0 +1,46 @@
# Enterprise Knowledge Platform — Codex Rules
## Read first
Before modifying code, read the documents relevant to the task:
- `docs/product/PRD.md`
- `docs/architecture/ARCHITECTURE.md`
- `docs/architecture/ADR.md`
- `docs/api/API_CONTRACT.md`
- `docs/api/MCP_CONTRACT.md`
- `docs/plans/V1_EXECUTION_PLAN.md`
- `docs/plans/TASKS.md`
- `docs/development/DEFINITION_OF_DONE.md`
Do not redesign accepted architecture unless the task explicitly asks for an architecture change.
## Architecture invariants
1. API First.
2. V1 uses a modular monolith plus independent async/GPU workers and an MCP gateway.
3. PostgreSQL + object storage hold canonical data.
4. Vector databases are rebuildable indexes, not sources of truth.
5. Business services depend on `VectorStore` abstractions, never directly on Mochow.
6. Embedding, reranking, parsing, OCR, object storage, and LLMs use provider interfaces.
7. Original source documents are immutable.
8. Compiled Knowledge is derived and traceable to source document versions/chunks.
9. ACL is resolved before or during retrieval. Never retrieve global Top-K and filter afterward.
10. MCP exposes business capabilities, not infrastructure/provider details.
11. Index profiles are versioned; model/chunk changes use shadow indexes before activation.
12. Do not split V1 into unnecessary microservices.
13. Never hardcode credentials or production endpoints.
14. Core retrieval changes require tests and RAG regression evaluation.
## Task discipline
- Work on one Task ID at a time.
- Keep scope limited to the task.
- Do not opportunistically implement future tasks.
- Update relevant contracts/docs when behavior changes.
## Before finishing
Run relevant checks: format, lint, typecheck, unit tests, relevant integration tests, migration checks, API/MCP contract checks, and RAG evaluation for retrieval changes.
Report changed files, tests/results, unresolved risks, and follow-up tasks.

19
CODEX_START_HERE.md Normal file
View File

@ -0,0 +1,19 @@
# Codex Start Here
Do not ask Codex to build all of V1 in one task.
Recommended first prompt:
> You are taking over the Enterprise Knowledge Platform repository. Do not modify code yet. Read AGENTS.md and the product, architecture, API/MCP, execution-plan, task and Definition-of-Done documents it references. Inspect the repository. Then output: (1) current implementation status, (2) gap analysis against V1, (3) top five technical risks, (4) next 1–3 recommended Task IDs, and (5) blockers. Do not redesign accepted ADRs.
Preferred cadence:
1. Plan one task.
2. Review the plan.
3. Implement one task.
4. Run validation.
5. Self-review as a PR.
6. Commit / open PR.
7. Move to the next task.
One Task = one branch = one PR whenever possible.

35
Makefile Normal file
View File

@ -0,0 +1,35 @@
.PHONY: help install format lint typecheck test test-unit test-integration eval up down
help:
@echo "Enterprise Knowledge Platform"
@echo " install Install development dependencies"
@echo " format Format code"
@echo " lint Run linters"
@echo " typecheck Run static type checks"
@echo " test Run default tests"
@echo " test-unit Run unit tests"
@echo " test-integration Run integration tests"
@echo " eval Run RAG evaluation"
@echo " up Start local dependencies"
@echo " down Stop local dependencies"
install:
@echo "TODO: implement after toolchain selection"
format:
@echo "TODO: wire formatter"
lint:
@echo "TODO: wire linter"
typecheck:
@echo "TODO: wire type checker"
test:
@echo "TODO: wire test suite"
test-unit:
@echo "TODO: wire unit tests"
test-integration:
@echo "TODO: wire integration tests"
eval:
@echo "TODO: wire RAG benchmark"
up:
docker compose -f docker-compose.dev.yml up -d
down:
docker compose -f docker-compose.dev.yml down

36
README.md Normal file
View File

@ -0,0 +1,36 @@
# Enterprise Knowledge Platform
Enterprise internal knowledge infrastructure.
## Core capabilities
- Multi-format document ingestion
- OCR and document parsing
- Structure-aware chunking
- Multiple embedding providers
- Hybrid retrieval and reranking
- ACL-aware search
- Knowledge Compilation
- REST API
- MCP Server
- Admin Console
## Architecture baseline
- API First + Admin Console
- Modular Monolith + async workers + MCP gateway
- PostgreSQL + object storage as canonical source of truth
- Mochow as the initial retrieval index, behind a replaceable adapter
- Embedding / reranker / parser / LLM providers are replaceable
- Raw source is immutable
- Compiled Knowledge is derived and traceable
- ACL is enforced before or during retrieval, never after Top-K
## Start here
1. Read `AGENTS.md`
2. Read `CODEX_START_HERE.md`
3. Read `docs/product/PRD.md`
4. Read `docs/architecture/ARCHITECTURE.md`
5. Read `docs/plans/V1_EXECUTION_PLAN.md`
6. Pick exactly one task from `docs/plans/TASKS.md`

0
apps/admin-web/.gitkeep Normal file
View File

0
apps/api/.gitkeep Normal file
View File

0
apps/mcp-server/.gitkeep Normal file
View File

0
deploy/docker/.gitkeep Normal file
View File

30
docker-compose.dev.yml Normal file
View File

@ -0,0 +1,30 @@
services:
postgres:
image: postgres:17
environment:
POSTGRES_DB: ekp
POSTGRES_USER: ekp
POSTGRES_PASSWORD: ekp
ports: ["5432:5432"]
volumes: ["ekp_postgres:/var/lib/postgresql/data"]
redis:
image: redis:7
ports: ["6379:6379"]
rabbitmq:
image: rabbitmq:4-management
ports: ["5672:5672", "15672:15672"]
minio:
image: minio/minio
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
ports: ["9000:9000", "9001:9001"]
volumes: ["ekp_minio:/data"]
volumes:
ekp_postgres:
ekp_minio:

7
docs/api/API_CONTRACT.md Normal file
View File

@ -0,0 +1,7 @@
# REST API Contract
Version all public APIs under `/api/v1`.
Initial domains: auth, users/groups, knowledge-bases, documents, compiled-knowledge, search, evaluations, jobs, audit-logs, providers and MCP administration.
Use OpenAPI as executable source of truth once implementation starts. Provider-specific concepts must not leak into public contracts.

11
docs/api/MCP_CONTRACT.md Normal file
View File

@ -0,0 +1,11 @@
# MCP Contract
P0 business tools:
- `search_knowledge`
- `get_document`
- `get_document_section`
- `get_source`
- `list_knowledge_bases`
Every request maps to a user/service identity and uses the same authorization/application services as REST. Do not expose Mochow, Qwen, BGE, MinerU or other infrastructure choices.

10
docs/architecture/ADR.md Normal file
View File

@ -0,0 +1,10 @@
# Architecture Decision Records
- ADR-001: API First + Admin Console — Accepted
- ADR-002: Modular monolith + workers + MCP gateway for V1 — Accepted
- ADR-003: PostgreSQL + object storage are canonical — Accepted
- ADR-004: Mochow is initial retrieval backend but replaceable — Accepted
- ADR-005: Provider abstraction for embedding/reranker/parser/LLM — Accepted
- ADR-006: RAG + Knowledge Compilation — Accepted
- ADR-007: ACL before/during retrieval — Accepted
- ADR-008: MCP hides infrastructure details — Accepted

View File

@ -0,0 +1,749 @@
# 企业内部智能知识库 V1 技术方案(简版)
> 版本:v0.1
> 目标:完成企业内部知识库第一版本,支持多类型文件上传、自动解析清洗、混合检索、Reranker、权限控制、Knowledge Compilation、REST API、MCP 接入及 Admin Console。
---
## 1. V1 建设目标
V1 聚焦形成完整可用闭环:
```text
创建知识库
↓
上传文件
↓
解析 / OCR / 清洗
↓
结构化 Chunk
↓
Embedding + BM25 索引
↓
Hybrid Retrieval
↓
Reranker
↓
ACL 权限过滤
↓
REST API / MCP
↓
WorkBuddy 等客户端调用
```
同时支持第二条知识沉淀链路:
```text
原始知识 Source
↓
结构化知识 Structured Knowledge
↓
Knowledge Compilation
↓
Wiki / FAQ / SOP / Summary 等派生知识
↓
审核、发布、持续更新
```
---
## 2. V1 总体架构
```text
┌─────────────────────┐
│ Admin Console │
└──────────┬──────────┘
│ REST
▼
┌────────────┐ ┌─────────────────────┐
│ WorkBuddy │──── MCP ────▶│ Knowledge API │◀──── REST ─── Other Systems
└────────────┘ │ FastAPI / Python │
└──────────┬──────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ PostgreSQL │ │ Object Store │ │ Mochow │
│ Canonical DB │ │ S3/MinIO │ │ Search Index │
└──────────────┘ └──────────────┘ └──────────────┘
▲ ▲
│ │
┌───────┴────────────────────────┘
│
┌──────────────┐
│ Async Worker │
│ Celery │
└──────┬───────┘
│
┌──────▼───────┐
│ GPU Worker │
│ Parser/OCR │
│ Embedding │
│ Reranker │
└──────────────┘
```
### 架构原则
1. **API First**:后台、MCP、其他业务系统均调用统一 API/Service。
2. **模块化单体优先**:V1 不拆大量微服务,API 服务 + 异步 Worker + MCP Gateway 足够。
3. **Canonical Data 独立**:PostgreSQL + 对象存储保存事实数据,Vector DB 仅保存可重建索引。
4. **Provider 可替换**:Embedding、Reranker、Vector DB、Parser、LLM 均通过 Adapter 抽象。
5. **MCP 不暴露底层实现**:客户端只看到 `search_knowledge` 等业务工具。
---
## 3. 当前技术选型
| 模块 | V1 当前选型 | 说明 |
|---|---|---|
| Backend | Python + FastAPI | AI / 数据处理生态成熟,适合 MCP、异步任务和模型调用 |
| Metadata DB | PostgreSQL | 保存用户、权限、文档、版本、任务、Compiled Knowledge 等 Canonical Data |
| Object Storage | S3 Compatible / MinIO | 保存原始文件、解析产物、中间文件 |
| Task Queue | Celery + RabbitMQ | 文件解析、Embedding、索引、Compilation 等异步任务 |
| PDF / OCR Parser | MinerU 优先 | 中文复杂 PDF、扫描件、表格、版面解析能力较强 |
| Office Parser | Docling 补充 | DOCX/PPTX/XLSX/HTML 等格式统一解析 |
| Vector/Search DB | Mochow | 延续公司现有经验,支持向量、BM25、Hybrid、Filter |
| Embedding | 多模型 Benchmark 决定 | 基线 bge-large-zh,对比 Qwen3 Embedding 等 |
| Reranker | Qwen3 Reranker 等候选 | V1 正式检索链必选能力 |
| LLM | Provider 抽象,云端优先 | 用于 Knowledge Compilation、分类、摘要等 |
| MCP | 独立 MCP Gateway | 面向 WorkBuddy 等客户端,隐藏内部技术栈 |
| Frontend | Admin Console | 知识库、文档、权限、检索调试、Benchmark、任务管理 |
---
## 4. 核心模块
### 4.1 Knowledge Base 管理
支持:
- 创建 / 编辑 / 归档知识库;
- Folder 分类;
- Owner / Member;
- 知识库级和文档级权限;
- Retrieval Profile;
- Embedding / Reranker 配置;
- Knowledge Compilation 配置。
---
### 4.2 文件上传与自动入库
V1 支持:
- PDF
- DOCX
- PPTX
- XLSX
- CSV
- Markdown
- TXT
- HTML
- JPG / PNG
处理流程:
```text
Upload
→ Hash 去重
→ Parser Router
→ OCR(必要时)
→ Cleaning
→ Structure Normalize
→ Chunk
→ Embedding
→ BM25 / Vector Index
→ Completed
```
任务状态:
```text
Queued / Parsing / Cleaning / Chunking / Embedding / Indexing / Completed / Failed
```
文件必须支持:
- Hash 去重;
- Document Version;
- 失败重试;
- Re-parse / Re-embed / Re-index;
- 软删除;
- 来源页码和章节追踪。
---
### 4.3 文档结构与 Chunk
解析后形成统一结构:
```text
Document
├─ Heading
├─ Paragraph
├─ List
├─ Table
└─ Page
```
Chunk 采用 **Structure-aware + Token-aware** 策略,禁止仅按固定字符数粗暴截断。
Chunk 至少保存:
```text
chunk_id
knowledge_base_id
document_id
document_version
content
title
section_path
page_start
page_end
content_type
metadata
acl
index_profile_id
```
---
## 5. 检索方案
V1 正式检索链:
```text
Query
↓
Query Normalize
↓
ACL Resolve
↓
Dense Search + BM25
↓
Fusion
↓
Candidate Top N
↓
Reranker
↓
Top K
↓
Source Enrichment
```
### 必须支持
- Dense Semantic Search;
- BM25 / Keyword Search;
- Hybrid Search;
- Metadata Filter;
- ACL Filter;
- Reranker;
- Knowledge Base / Folder / Document Scope;
- 来源追踪。
禁止使用“全局检索后再删除无权限结果”的方式,权限过滤必须在召回阶段生效。
---
## 6. Embedding Benchmark
V1 不锁死单一 Embedding 模型。
### 首批候选
- bge-large-zh(现有基线);
- Qwen3-Embedding-0.6B;
- Qwen3-Embedding-4B;
- 视 GPU 与效果补充 8B 或其他中文模型。
### Benchmark 指标
```text
Recall@1 / 3 / 5 / 10
MRR
NDCG@10
P50 / P95 Latency
Index Throughput
GPU Memory
API Cost
Storage Cost
```
### 测试集
建立企业内部 Golden Dataset,V1 建议 200~500 条真实查询,覆盖:
- 中文自然语言;
- 精确术语 / 编号;
- 产品型号;
- 同义表达;
- 中英文混写;
- 长问题;
- 多条件问题;
- 跨文档问题。
最终比较的是完整 Pipeline:
```text
Embedding × Hybrid Retrieval × Reranker
```
而不是只比较 Embedding 排行榜。
---
## 7. Index Profile 与模型切换
Embedding、Chunk、Sparse、Reranker 配置统一形成 `IndexProfile`。
示例:
```text
profile_v1
- chunk_strategy: structure_v1
- embedding: bge-large-zh
- sparse: bm25
profile_v2
- chunk_strategy: structure_v2
- embedding: qwen3-embedding-4b
- sparse: bm25
```
新模型上线时:
```text
创建新 Profile
→ 构建 Shadow Index
→ Benchmark
→ 切换 Active Index
→ 保留旧索引回滚窗口
```
避免直接原地覆盖现有生产索引。
---
## 8. Vector Store 设计
### 当前默认:Mochow
选择理由:
- 公司已有使用经验;
- 已有百度云体系;
- 支持 Dense Vector;
- 支持 BM25 / Full Text;
- 支持 Hybrid Search;
- 支持 Metadata / Scalar Filter;
- 满足 V1 ACL 下推需求。
### 保留替换能力
统一:
```text
VectorStoreProvider
├─ Mochow
├─ Qdrant
├─ Milvus
└─ pgvector
```
业务 Domain Model 不依赖 Mochow SDK 数据结构。
Mochow 中的数据全部视为 **可重建索引**,真实 Canonical Data 保存在 PostgreSQL / Object Storage。
---
## 9. Knowledge Compilation
V1 支持 **RAG + Knowledge Compilation**。
三层知识模型:
```text
L0 Source
原始文件,不允许 AI 修改
L1 Structured
Parsed / Chunk / Index
L2 Compiled
Wiki / FAQ / SOP / Summary / Comparison
```
### Compilation Pipeline
```text
Source 新增或更新
→ Impact Analysis
→ 查找相关 Knowledge Page
→ LLM 生成 Draft
→ Source Verification
→ Conflict Detection
→ Review Required
→ Publish
```
### Compiled Knowledge 类型
- Wiki Page;
- FAQ;
- SOP;
- Summary;
- Concept;
- Entity;
- Comparison。
所有 L2 Knowledge 必须保存 underlying source,并可追溯至 L0。
Source 删除、更新或权限变化时,相关 Compiled Knowledge 自动标记 `Stale`。
V1 默认 AI 只生成 Draft,重要知识需人工审核后发布。
---
## 10. 权限模型
权限主体:
```text
User
Group
Role
```
资源层级:
```text
Organization
Knowledge Base
Folder
Document
```
权限类型:
```text
Read
Write
Manage
```
检索流程:
```text
User Identity
→ Permission Resolver
→ Allowed Scope / ACL Filter
→ Mochow Hybrid Retrieval
```
MCP / REST / Admin Console 必须复用同一权限服务。
---
## 11. MCP
对外提供统一:
```text
Enterprise Knowledge MCP Server
```
V1 建议工具:
### `search_knowledge`
搜索当前用户有权限访问的企业知识。
### `get_document`
读取文档 metadata。
### `get_document_section`
获取指定页 / Section / Chunk 的连续上下文。
### `get_source`
获取检索结果的来源信息。
### `list_knowledge_bases`
列出当前身份可访问的知识库。
MCP Response 不出现:
```text
Mochow
Qdrant
bge-large-zh
Qwen3
MinerU
```
等底层实现信息。
首要验证客户端:**WorkBuddy Enterprise**。
---
## 12. REST API
统一前缀:
```text
/api/v1/
```
主要 Domain:
```text
/auth
/users
/groups
/knowledge-bases
/documents
/compiled-knowledge
/search
/evaluations
/jobs
/audit-logs
/providers
/mcp
```
核心搜索接口:
```http
POST /api/v1/search
```
返回至少包含:
```text
request_id
results
source
score
rerank_score
document
page
section
chunk_id
source_type
```
---
## 13. Admin Console
V1 一级功能建议:
```text
Overview
Knowledge Bases
Documents
Compiled Knowledge
Search Lab
Evaluation
Models
MCP & API
Users & Permissions
Jobs
Audit Logs
System Settings
```
### Search Lab
Search Lab 是 V1 P0 工具,用于查看:
```text
Dense Recall
Sparse Recall
Fusion
Reranker
Final Result
```
帮助定位 Parser、Chunk、Embedding、BM25、Reranker 哪一层出现问题。
---
## 14. 异步任务与 GPU
异步 Job:
```text
ParseJob
OCRJob
ChunkJob
EmbeddingJob
IndexJob
CompilationJob
ReindexJob
EvaluationJob
```
GPU 节点优先承担:
- MinerU / OCR;
- Local Embedding;
- Local Reranker。
LLM V1 可继续使用云端 Provider,不要求部署本地大模型。
---
## 15. 数据存储原则
### PostgreSQL
保存:
- User / Group / Role;
- Knowledge Base;
- Document / Version;
- Chunk metadata;
- ACL;
- Job;
- Index Profile;
- Compiled Knowledge;
- Audit Log;
- Provider Config。
### Object Storage
保存:
- 原始文件;
- Parser 输出;
- OCR 结果;
- Markdown / JSON 中间产物;
- 需要持久化的大对象。
### Mochow
仅保存:
- Dense Vector;
- Sparse/BM25 Index;
- Retrieval metadata;
- ACL 所需过滤字段。
数据库损坏或更换时,可由 Canonical Data 全量重建。
---
## 16. 部署建议
V1 推荐按以下逻辑角色部署:
```text
Reverse Proxy / Ingress
│
├─ Admin Console
├─ Knowledge API
└─ MCP Gateway
PostgreSQL
RabbitMQ
Object Storage
CPU Worker
GPU Worker
Mochow
Observability
```
当前几十~百人内部用户规模,不建议一开始拆大量微服务。
优先采用:
```text
模块化单体 API
+ 独立 Async Worker
+ 独立 GPU Worker
+ 独立 MCP Gateway
```
未来根据负载再拆服务。
---
## 17. V1 验收重点
### 文件入库
P0 文件能够完成:
```text
Upload → Parse → Clean → Chunk → Embed → Index
```
### 检索
正式链路支持:
```text
Dense + BM25 + Hybrid + Reranker + ACL
```
### 来源
每条结果能够追踪:
```text
Source File → Version → Page → Section → Chunk
```
### 权限
Admin Console、REST、MCP 均不能访问无权限知识。
### Benchmark
能够使用同一个 Golden Dataset 对多套 Retrieval Pipeline 进行对比。
### Knowledge Compilation
能够完成:
```text
Source → Draft → Source Linking → Review → Publish → Stale Detection
```
### MCP
WorkBuddy 可以通过 MCP 完成知识搜索和 Source 获取。
---
## 18. V1 当前最终选型结论
```text
Backend Python + FastAPI
Metadata PostgreSQL
Object Storage S3 Compatible / MinIO
Queue RabbitMQ + Celery
Parser MinerU 主力 + Docling 补充
Vector Store Mochow Primary,Adapter 可替换
Embedding bge-large-zh 基线,多模型 Benchmark 后决定生产模型
Reranker Qwen3 Reranker 等候选,V1 必选
LLM Provider 抽象,允许云端
Frontend Admin Console
Protocol REST API + MCP
Architecture Modular Monolith + Workers + MCP Gateway
Knowledge RAG + Knowledge Compilation
```
核心原则:**先保证知识可正确解析、可靠检索、严格授权、完整追溯,再逐步增加更高级的知识编译和智能化能力。**

View File

@ -0,0 +1,8 @@
# Coding Rules
- Keep infrastructure SDKs behind provider interfaces.
- Do not hardcode credentials, tenant IDs, endpoints, model names or collection names.
- Prefer typed contracts and cohesive modules.
- Make DB migrations forward-only.
- Core tests must work without external SaaS, internet or GPU.
- Avoid premature microservices.

View File

@ -0,0 +1,3 @@
# Definition of Done
A task is done only when scope is implemented, architecture invariants hold, format/lint/typecheck pass, relevant tests pass, migrations/contracts/docs are updated as needed, no secrets are committed, retrieval changes include RAG regression results, and known limitations are documented.

View File

@ -0,0 +1,9 @@
# Security Baseline
- Enforce tenant + ACL scope before/during retrieval.
- Never log credentials or raw auth tokens.
- Protect source-document downloads.
- Audit sensitive reads and administrative changes.
- Support `cloud_allowed` and `local_only` data policies per Knowledge Base.
- Never send `local_only` content to cloud AI providers.
- Validate MCP identity before protected operations.

View File

@ -0,0 +1,16 @@
# Test Strategy
Unit tests must run without Mochow, Qianfan, internet, GPU or production credentials.
Provide deterministic fakes:
- FakeEmbeddingProvider
- FakeRerankerProvider
- FakeVectorStore
- FakeLLMProvider
- FakeObjectStorage
External integrations use separate integration suites/markers.
E2E covers upload -> parse -> chunk -> embed -> index -> retrieve -> source trace, and source -> compilation draft -> review -> publish -> retrieval.
Any parsing/chunking/embedding/fusion/retrieval/reranking/query-routing change must run the Golden Dataset evaluation.

View File

@ -0,0 +1,9 @@
# RAG Evaluation
Maintain a versioned Golden Dataset and report Recall@1/3/5/10, MRR, NDCG@10, latency P50/P95.
Benchmark the full pipeline: Embedding × Retrieval Strategy × Fusion × Reranker.
Initial embedding candidates: bge-large-zh, Qwen3-Embedding-0.6B, Qwen3-Embedding-4B, optional 8B.
Use synthetic/public/anonymized fixtures only.

7
docs/plans/PROGRESS.md Normal file
View File

@ -0,0 +1,7 @@
# Progress
Current phase: Phase 0 — Repository / developer harness
Completed: none
In progress: none
Blocked: none

61
docs/plans/TASKS.md Normal file
View File

@ -0,0 +1,61 @@
# V1 Tasks
## Phase 0
- [ ] EKP-001 Repository bootstrap
- [ ] EKP-002 Backend Python toolchain
- [ ] EKP-003 Admin Web toolchain
- [ ] EKP-004 Application configuration
- [ ] EKP-005 Structured logging
- [ ] EKP-006 CI baseline
- [ ] EKP-007 Fake provider framework
## Canonical domain / ingestion
- [ ] EKP-010 Core domain package
- [ ] EKP-011 PostgreSQL schema and migrations
- [ ] EKP-012 User/group/role model
- [ ] EKP-013 Knowledge Base model
- [ ] EKP-014 Folder model
- [ ] EKP-015 Document + DocumentVersion model
- [ ] EKP-016 ACL model
- [ ] EKP-017 Job + audit models
- [ ] EKP-020 ObjectStorage abstraction + S3 adapter
- [ ] EKP-021 Document upload API
- [ ] EKP-022 Source hash/dedup/version activation
- [ ] EKP-023 Async job framework
- [ ] EKP-024 Parser abstraction
- [ ] EKP-025 MinerU adapter
- [ ] EKP-026 Docling adapter
- [ ] EKP-027 OCR + cleaning pipeline
- [ ] EKP-028 Canonical parsed document model
- [ ] EKP-029 Structure-aware / spreadsheet chunking
## Indexing / retrieval
- [ ] EKP-040 EmbeddingProvider + fake
- [ ] EKP-041 Qianfan BGE adapter
- [ ] EKP-042 Qwen embedding adapter
- [ ] EKP-043 IndexProfile model
- [ ] EKP-044 VectorStore + fake
- [ ] EKP-045 Mochow adapter
- [ ] EKP-046 Index build + shadow index lifecycle
- [ ] EKP-050 Permission resolver + ACL retrieval filters
- [ ] EKP-051 Dense retrieval
- [ ] EKP-052 BM25/sparse retrieval
- [ ] EKP-053 Hybrid fusion
- [ ] EKP-054 RerankerProvider + fake + Qwen adapter
- [ ] EKP-055 Source enrichment + Search service/API
- [ ] EKP-056 Retrieval diagnostics / Search Lab backend
## MCP / Admin / Compilation / Evaluation
- [ ] EKP-070 MCP server + identity
- [ ] EKP-071 MCP P0 tools
- [ ] EKP-072 WorkBuddy compatibility test
- [ ] EKP-080 Admin Console core pages
- [ ] EKP-081 Permissions/jobs/chunk inspector/Search Lab UI
- [ ] EKP-090 CompiledKnowledge model/workflow
- [ ] EKP-091 Source linking/conflict/stale/review/publish
- [ ] EKP-092 Compiled retrieval integration
- [ ] EKP-100 Golden Dataset + metrics
- [ ] EKP-101 Multi-embedding benchmark
- [ ] EKP-102 Reranker benchmark
- [ ] EKP-103 Security/performance/rebuild tests
- [ ] EKP-104 V1 release hardening

View File

@ -0,0 +1,11 @@
# V1 Execution Plan
1. Repository / developer harness
2. Canonical domain and ACL
3. Document ingestion and parsing
4. Embedding / Index Profile / Mochow adapter
5. ACL-aware hybrid retrieval + reranker
6. MCP gateway and WorkBuddy verification
7. Admin Console
8. Knowledge Compilation
9. Evaluation, security and release hardening

2537
docs/product/PRD.md Normal file

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,3 @@
# Test Documents
Only commit synthetic, public, or properly anonymized fixtures. Never commit real internal enterprise documents, HR data, customer data, contracts, credentials, or confidential material.

View File

@ -0,0 +1,2 @@
{"id":"sample-001","query":"员工出差住宿标准是多少?","expected_documents":["sample-travel-policy.md"],"expected_sections":["住宿标准"],"tags":["制度","自然语言"]}
{"id":"sample-002","query":"产品A退款周期","expected_documents":["sample-product-a-policy.md"],"expected_sections":["退款"],"tags":["产品","短查询"]}

0
migrations/.gitkeep Normal file
View File

0
packages/core/.gitkeep Normal file
View File

0
packages/domain/.gitkeep Normal file
View File

View File

View File

View File

View File

View File

View File

0
packages/shared/.gitkeep Normal file
View File

37
prompts/CODEX_PROMPTS.md Normal file
View File

@ -0,0 +1,37 @@
# Codex Prompt Library
## P00 — Repository takeover
Read AGENTS.md and all referenced product/architecture/API/MCP/plan/DoD docs. Do not modify code. Inspect the repository and report current status, V1 gaps, top 5 risks, next 1–3 Task IDs, and blockers. Do not redesign accepted ADRs.
## P01 — Plan one task
Plan `<TASK_ID>` only. Read relevant docs/code. Output goal, touched modules, steps, tests, migration/API impact, risks and out-of-scope. Do not modify files.
## P02 — Implement one task
Implement `<TASK_ID>` only. Preserve AGENTS.md invariants. Do not implement future tasks. Run format/lint/typecheck/tests and RAG eval if retrieval behavior changes. Report summary, changed files, validation, risks and follow-ups.
## P03 — Self-review as PR
Review the current branch for correctness, architecture violations, ACL leakage, secrets, provider coupling, migration safety, idempotency, test coverage, API/MCP compatibility and source traceability. Fix Critical/High findings within scope and rerun validation.
## P04 — Bug fix
Reproduce `<BUG>`, identify root cause, add failing regression test, implement smallest safe fix, run relevant checks, and report root cause + validation.
## P05 — Retrieval change
For `<TASK_ID>`, identify changed retrieval stage. Preserve ACL-before/during-retrieval and source metadata. Run Golden Dataset before/after and report quality + latency deltas.
## P06 — Embedding benchmark
Compare `<MODELS>` using the same Golden Dataset, corpus, chunk profile and retrieval config. Report Recall@K, MRR, NDCG@10, P50/P95 latency, throughput, dimensions, GPU/API needs, storage/cost and recommendation.
## P07 — Parser benchmark
Compare MinerU and Docling on anonymized Chinese enterprise docs: native/scanned PDF, tables, multi-column, DOCX, PPTX, XLSX where applicable. Evaluate completeness, order, structure, tables, traceability, OCR, latency and GPU. Recommend routing rules.
## P08 — MCP task
Implement/review `<TASK_ID>`. Expose business semantics only; bind identity; reuse REST authorization/application services; preserve traceability; do not expose underlying provider choices.
## P09 — ACL audit
Verify Knowledge Base, folder inheritance, document overrides, REST, MCP, source download, retrieval, compiled knowledge and jobs. Fail review if unauthorized content can enter candidate Top-K before filtering.
## P10 — Knowledge Compilation
Implement `<TASK_ID>` while preserving immutable source, source links, permission inheritance/intersection, stale detection and conflict safety. Never silently merge conflicting claims.
## P11 — Release readiness
Assess V1 against PRD/DoD/tasks/ADRs: functionality, migrations, ACL/security, secrets, observability, backup/rebuild, performance, RAG quality, MCP, Admin Console and Compilation. Return blockers, high risks, known limitations and go/no-go.

0
scripts/.gitkeep Normal file
View File

0
tests/e2e/.gitkeep Normal file
View File

View File

View File

0
tests/unit/.gitkeep Normal file
View File

View File

View File

View File

0
workers/parser/.gitkeep Normal file
View File