hy4/_kp 改为普通目录(去除嵌套git引用)

This commit is contained in:
D-robotics 2026-11-12 11:14:02 +08:00
parent a74a276767
commit 9658cee5e6
364 changed files with 67456 additions and 1 deletions

@ -1 +0,0 @@
Subproject commit 853050f38e268134a31d3d9b785b4f165382bc98

19
hy4/_kp/.dockerignore Normal file
View File

@ -0,0 +1,19 @@
# 排除生产镜像不需要的本地状态和生成文件。
.env*
!.env.dev
.git
.venv
.*_cache
__pycache__
*.py[cod]
*.sqlite*
*.egg-info
node_modules
celerybeat-schedule*
data
dist
docker-compose.dev.yml
storage
tmp
tests
docs

5
hy4/_kp/.drone.yml Normal file
View File

@ -0,0 +1,5 @@
kind: template
load: knowledge-platform.jsonnet
data:
App: "knowledge-platform,knowledge-platform-worker,knowledge-platform-mcp"
DisableDockerfile: "true"

15
hy4/_kp/.editorconfig Normal file
View File

@ -0,0 +1,15 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
[*.py]
indent_size = 4
[Makefile]
indent_style = tab

17
hy4/_kp/.gitattributes vendored Normal file
View File

@ -0,0 +1,17 @@
* text=auto eol=lf
*.sh text eol=lf
*.py text eol=lf
*.md text eol=lf
*.yaml text eol=lf
*.yml text eol=lf
*.json text eol=lf
*.toml text eol=lf
*.ts text eol=lf
*.tsx text eol=lf
*.png binary
*.jpg binary
*.jpeg binary
*.pdf binary
*.docx binary
*.pptx binary
*.xlsx binary

58
hy4/_kp/.gitignore vendored Normal file
View File

@ -0,0 +1,58 @@
# 本地环境;环境变体默认不得进入版本库
.env
.env.*
# Python 生成文件
__pycache__/
*.py[cod]
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
htmlcov/
.venv/
venv/
*.egg-info/
dist/
build/
# 管理端 Web 工具链生成文件
node_modules/
npm-debug.log*
# 编辑器与操作系统
.idea/
.vscode/
*.swp
*.swo
.DS_Store
Thumbs.db
# 运行状态与本地数据
logs/
tmp/
data/
storage/
uploads/
celerybeat-schedule*
models/
artifacts/
reports/
# 凭据
*.pem
*.key
*.p12
credentials.json
secrets.json
# 模型产物
*.bin
*.safetensors
*.onnx
*.pt
*.pth
# 本地数据库
*.sqlite
*.sqlite3

1
hy4/_kp/.node-version Normal file
View File

@ -0,0 +1 @@
20.18.0

1
hy4/_kp/.python-version Normal file
View File

@ -0,0 +1 @@
3.12

63
hy4/_kp/AGENTS.md Normal file
View File

@ -0,0 +1,63 @@
# 企业知识平台 — Codex 规则
## 开始前必读
修改代码前,先阅读与任务相关的以下文档:
- `docs/README.md`
- `docs/product/PRD.md`
- `docs/architecture/ARCHITECTURE.md`
- `docs/architecture/ADR.md`
- `docs/api/API_CONTRACT.md`
- `docs/api/MCP_CONTRACT.md`
- `docs/plans/V1_EXECUTION_PLAN.md`
- `docs/plans/TASKS.md`
- `docs/development/DEFINITION_OF_DONE.md`
除非任务明确要求修改架构,否则不得重新设计已接受的架构。
## 架构不变量
1. API First。
2. V1 采用模块化单体,并配备独立的异步/GPU Worker 和 MCP Gateway。
3. OceanBase 关系表与对象存储共同保存规范数据(canonical data)。
4. 即使与规范关系表部署在同一数据库中,OceanBase 向量表也只是可重建索引,而不是事实源。
5. 业务服务依赖 `VectorStore` 抽象,绝不直接依赖 Mochow。
6. Embedding、Reranker、解析、OCR、对象存储和 LLM 均使用 Provider 接口。
7. 原始来源文档不可变。
8. Compiled Knowledge 是派生数据,且必须可追溯到来源文档版本和 Chunk。
9. ACL 必须在检索前或检索过程中解析。禁止先执行全局 Top-K 检索再做过滤。
10. MCP 只暴露业务能力,不暴露基础设施或 Provider 细节。
11. Index Profile 必须版本化;模型或 Chunk 变更应先构建 Shadow Index,再激活。
12. 不得把 V1 拆分成不必要的微服务。
13. 只有在用户明确授权具体文件和具体环境时,才允许提交凭据、密钥、密码和环境 Endpoint;
所有未经批准的 Secret 都必须排除在 Git 之外。
14. 核心检索变更必须补充测试并执行 RAG 回归评估。
## 任务纪律
- 每次只处理一个 Task ID。
- 默认只使用当前仓库根目录作为工作区,在同一目录内按任务顺序切换分支;除非用户明确要求并行
隔离或授权额外 worktree,不得为每个 Task 新建同级项目目录。切换分支前必须先用提交或命名
stash 保存并核验未完成修改,禁止覆盖、丢弃或 reset 用户工作。文档、任务分支、里程碑 Tag、
stash 与 worktree 的盘点、归档、删除和恢复流程遵循
`docs/development/REPOSITORY_GOVERNANCE.md`。
- 严格限定在当前任务范围内。
- 不得顺手实现未来任务。
- 完成 V1 开发功能是第一优先级。文档治理、Benchmark、部署/现场验收等非开发工作使用独立任务
类型和执行队列,同步记录状态,但不得成为后续开发 Task 的依赖或阻塞项。
- 行为发生变化时,更新相关契约和文档。
- 添加有价值的代码注释或文档字符串,并在相关文档或进度日志中记录文件或行为变化。
- 提交经明确授权的环境凭据时,必须记录授权范围。
- 验证完成后,先推送任务分支并合并到 `main`,再开始下一个 Task ID。
## 完成前检查
运行相关检查:格式检查、Lint、类型检查、单元测试、相关集成测试、Migration 检查、API/MCP 契约检查;如果涉及检索变更,还必须运行 RAG 评估。
每次新增或改变 API、前端页面、按钮或用户入口,都必须在真实启动的应用上完成端到端测试:通过
真实 HTTP 请求或浏览器操作,从入口贯穿对应应用服务和持久化/对象存储边界,并记录非敏感结果。
仅有单元测试、`TestClient`、静态文本断言、Mock 或截图不能替代该端到端测试;外部 Provider 与
目标集群现场验收仍作为独立非开发任务记录,不阻塞已由本地真实应用链路证明的功能开发。
报告变更文件、测试及结果、尚未解决的风险和后续任务。

View File

@ -0,0 +1,22 @@
# Codex 入门指南
不要要求 Codex 在一个任务中完成整个 V1。
先阅读 `AGENTS.md` 和 `docs/README.md`。后者定义文档权威性、生命周期和当前活动计划,避免从历史计划中误判当前范围。
推荐的第一个提示词:
> 你正在接手 Enterprise Knowledge Platform 仓库。暂时不要修改代码。阅读 `AGENTS.md` 及其引用的产品、架构、API/MCP、执行计划、任务和完成定义文档,并检查仓库。然后输出:(1) 当前实现状态,(2) 与 V1 的差距分析,(3) 五项主要技术风险,(4) 接下来推荐的 1~3 个 Task ID,以及 (5) 阻塞项。不要重新设计已接受的 ADR。
推荐节奏:
1. 规划一个任务。
2. 审查计划。
3. 实现一个任务。
4. 运行验证。
5. 按 PR 标准进行自审。
6. 提交或创建 PR。
7. 进入下一个任务。
在条件允许时,遵循一个 Task 对应一个分支和一个 PR。任务切换、文档精简、分支归档和里程碑
Tag 规则见 `docs/development/REPOSITORY_GOVERNANCE.md`。

84
hy4/_kp/Dockerfile Normal file
View File

@ -0,0 +1,84 @@
# syntax=docker/dockerfile:1.7
# 一个锁定的运行时构建三个可独立部署的进程镜像。
ARG PYTHON_IMAGE=python:3.12.14-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.12.5@sha256:e85be844203885286c60ffad8a858d48afb6c5a5c237ca0e67f12e74b8f174b1
FROM ${UV_IMAGE} AS uv-bin
FROM ${PYTHON_IMAGE} AS runtime
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
UV_COMPILE_BYTECODE=1 \
UV_LINK_MODE=copy \
VIRTUAL_ENV=/app/.venv \
PATH=/app/.venv/bin:${PATH} \
HOME=/tmp/ekp \
XDG_CACHE_HOME=/tmp/ekp/cache \
APP_ENV=production \
API_HOST=0.0.0.0 \
MCP_HOST=0.0.0.0 \
ADMIN_WEB_ROOT=/app/apps/admin-web
WORKDIR /app
COPY --from=uv-bin /uv /uvx /usr/local/bin/
COPY pyproject.toml uv.lock README.md alembic.ini ./
COPY apps ./apps
COPY packages ./packages
COPY migrations ./migrations
# 固定依赖锁文件,不使用可编辑安装,也不把开发依赖带入运行镜像。
RUN uv sync --frozen --no-dev --no-editable \
&& test -f /app/apps/admin-web/index.html \
&& groupadd --gid 10001 ekp \
&& useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin ekp \
&& mkdir -p /tmp/ekp/cache \
&& chown -R 10001:10001 /tmp/ekp \
&& rm -rf /root/.cache/uv
USER 10001:10001
# MCP 服务
FROM runtime AS mcp
EXPOSE 8766
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD ["python", "-c", "import os; os.kill(1, 0)"]
CMD ["ekp-mcp"]
# 异步 Worker
FROM runtime AS worker
# 本地镜像入口会同时启动 Celery Beat;可写工作目录避免把调度状态写入只读应用目录。
WORKDIR /tmp
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD ["python", "-c", "import os; os.kill(1, 0)"]
CMD ["ekp-worker"]
# API 服务;放在最后,因此不指定 target 时默认构建 API
FROM runtime AS api
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"]
CMD ["uvicorn", "ekp_api.main:app", "--host", "0.0.0.0", "--port", "8000", "--no-access-log"]
# 中央 dev 流水线构建默认最终 target;仅该 target 携带已授权的 dev 配置。
# 显式 api/mcp/worker target 继续保持无环境文件的生产镜像边界。
FROM api AS dev
ENV APP_ENV=development \
EKP_SETTINGS_PROFILE=dev-image
COPY --chown=10001:10001 --chmod=0400 .env.dev /app/.env.dev
# 中央单副本 dev 流水线没有 Migration Job;仅 dev target 在监听流量前升级规范库。
CMD ["python", "-m", "ekp_api.dev_startup"]

112
hy4/_kp/Makefile Normal file
View File

@ -0,0 +1,112 @@
.PHONY: help lock install toolchain-check run run-mcp run-worker images images-print images-push deploy deploy-lint deploy-render cluster-acceptance mcp-remote-acceptance format format-check lint typecheck test test-unit test-integration migration-check eval live-acceptance up down
.PHONY: web-check ci test-e2e
UV_CACHE_DIR ?= /tmp/ekp-uv-cache
export UV_CACHE_DIR
UV ?= uv
UV_RUN := $(UV) run --locked
help:
@echo "Enterprise Knowledge Platform"
@echo " lock Intentionally refresh the Python dependency lock"
@echo " install Install development dependencies"
@echo " toolchain-check Validate the locked Python toolchain contract"
@echo " web-check Validate the zero-build Admin Web assets"
@echo " ci Run the complete provider-free CI baseline"
@echo " run Start the development API"
@echo " run-mcp Start the authenticated MCP gateway"
@echo " run-worker Start ingestion/compilation workers and recovery scheduler"
@echo " images Build production images for the current builder"
@echo " images-print Render the multi-architecture image build plan"
@echo " images-push Build and push amd64/arm64 production images"
@echo " deploy Install or upgrade Kubernetes with VALUES=<file>"
@echo " deploy-lint Validate Helm values and templates"
@echo " deploy-render Render non-secret Kubernetes manifests"
@echo " cluster-acceptance Validate the explicitly selected target cluster"
@echo " mcp-remote-acceptance Verify five-tool MCP client compatibility"
@echo " format Format code"
@echo " format-check Check formatting without modifying files"
@echo " lint Run linters"
@echo " typecheck Run static type checks"
@echo " test Run default tests"
@echo " test-unit Run unit tests"
@echo " test-integration Run integration tests"
@echo " test-e2e Run real-process loopback API/MCP tests"
@echo " migration-check Validate the forward migration path"
@echo " eval Run RAG evaluation"
@echo " live-acceptance Run controlled target-provider acceptance"
@echo " up Start local dependencies"
@echo " down Stop local dependencies"
lock:
$(UV) lock
install:
$(UV) sync --locked --all-groups
toolchain-check:
$(UV) lock --check
$(UV) sync --locked --all-groups --dry-run
$(UV_RUN) pytest -q tests/unit/test_python_toolchain.py
web-check:
npm --prefix apps/admin-web ci --ignore-scripts --no-audit --no-fund
npm --prefix apps/admin-web run check
ci:
$(MAKE) toolchain-check
$(MAKE) web-check
$(MAKE) format-check
$(MAKE) lint
$(MAKE) typecheck
$(MAKE) test
$(MAKE) migration-check
$(MAKE) eval
$(MAKE) test-e2e
run:
$(UV_RUN) uvicorn ekp_api.main:app --host "$${API_HOST:-0.0.0.0}" --port "$${API_PORT:-8000}"
run-mcp:
$(UV_RUN) ekp-mcp
run-worker:
$(UV_RUN) ekp-worker
images:
docker buildx bake -f deploy/docker/docker-bake.hcl
images-print:
docker buildx bake -f deploy/docker/docker-bake.hcl --print
images-push:
docker buildx bake -f deploy/docker/docker-bake.hcl --push
deploy:
./scripts/deploy.sh "$(VALUES)"
deploy-lint:
@test -n "$(VALUES)" || (echo "VALUES must point to an environment values file" >&2; exit 2)
helm lint deploy/helm/knowledge-platform --values "$(VALUES)"
deploy-render:
@test -n "$(VALUES)" || (echo "VALUES must point to an environment values file" >&2; exit 2)
helm template "$${RELEASE:-knowledge-platform}" deploy/helm/knowledge-platform --namespace "$${NAMESPACE:-knowledge-platform}" --values "$(VALUES)"
cluster-acceptance:
./scripts/cluster_acceptance.sh "$(VALUES)"
mcp-remote-acceptance:
$(UV_RUN) python scripts/mcp_remote_acceptance.py
format:
$(UV_RUN) ruff format .
format-check:
$(UV_RUN) ruff format --check .
lint:
$(UV_RUN) ruff check .
typecheck:
$(UV_RUN) mypy apps/api/src apps/mcp-server/src packages tests
test:
$(UV_RUN) pytest tests/unit tests/integration
test-unit:
$(UV_RUN) pytest tests/unit
test-integration:
$(UV_RUN) pytest tests/integration
test-e2e:
$(UV_RUN) pytest -q tests/e2e
migration-check:
$(UV_RUN) python scripts/migration_contract.py
eval:
$(UV_RUN) pytest tests/evaluation
$(UV_RUN) ekp-evaluate
live-acceptance:
$(UV_RUN) ekp-live-acceptance
up:
docker compose -f docker-compose.dev.yml up -d
down:
docker compose -f docker-compose.dev.yml down

127
hy4/_kp/README.md Normal file
View File

@ -0,0 +1,127 @@
# 企业知识平台
面向企业内部的知识基础设施,提供文档入库、可追溯检索、REST API、MCP Gateway 和管理控制台。
## 当前状态
当前开发基线已完成至 `EKP-103`,支持企业受控网络内的可信 User/Service 身份、多 Knowledge
Base 与检索前 ACL。已经实现:
- PDF、DOCX、PPTX、XLSX、CSV、HTML、TXT、Markdown 上传、解析、清洗和结构感知 Chunk;
XLSX/CSV 保留 Sheet/Header/Cell 与行列范围,显式启用 MinerU 后支持 JPG/JPEG/PNG 与扫描
PDF OCR;
- SHA-256 Source Hash 去重、同名文档递增版本和索引成功后激活;
- 持久化异步 Job、批量提交、有界重试和重试前派生数据清理;
- 千帆 `bge-large-zh` Embedding、OceanBase Dense/Sparse、加权 RRF、`bce-reranker-base` TopK;
- 来源补全、Index Profile、Shadow Index 重建与切换;
- REST API、包含五个 P0 Tool 的 MCP Gateway、知识运营与权限管理控制台;
- 可追溯 Compiled Knowledge 草稿、冲突/Stale、审核发布、Published-only 联合检索,以及持久化
Compilation Job/Attempt、后台重试和租约恢复;
- 生产容器镜像、Kubernetes/Helm、结构化日志和基础指标。
尚未完成的工作独立跟踪:工程基础任务 `EKP-001~007` 正在逐项正式关单;`EKP-101~102`
等待真实 Qwen Endpoint/GPU 环境;远程 MCP/WorkBuddy、目标集群现场验收与 `EKP-104` 发布验收
不阻塞开发主队列。完整审计查询管理面和 Append-only 数据库权限仍是已记录的 V1 差距。
`EKP-025~027` 已提供自托管 MinerU `/file_parse` HTTP Adapter、显式 OCR 路由、页级质量证据
和增强清洗。默认仍为轻量 Parser;启用方式与现场限制见
[`docs/development/MINERU_SETUP.md`](docs/development/MINERU_SETUP.md)。
`EKP-026` 已实现本地 Docling Adapter,并以 DOCX、PPTX、XLSX、CSV、HTML 真实格式样例验证
上游结构读取。`EKP-028~029` 已建立 schema v4 规范解析模型和 `structure_v2` Chunk,接入
PPTX/XLSX/CSV/HTML 上传与 Worker 路由,并在检索来源中返回 Spreadsheet 范围。边界见
[`docs/development/DOCLING_SETUP.md`](docs/development/DOCLING_SETUP.md) 和
[`docs/development/STRUCTURED_CHUNKING.md`](docs/development/STRUCTURED_CHUNKING.md)。
`DEPLOY-003` 已按开发侧范围闭环:`dev` 流水线与 GitOps 更新成功,公网 Admin/API/Worker/Search
已完成烟测。集群资源、Secret、滚动/回滚、监控和备份由运维人员管控;远程 MCP/WorkBuddy
当前搁置。`BUILD-001` 已完成应用镜像构建来源收敛并通过 Drone #30;`EKP-015` 已完成
DocumentVersion 模型与行为闭环;`EKP-052~053` 已建立 BM25/Sparse 与 Hybrid 检索;
`EKP-025~027` 已完成可选 MinerU OCR 与清洗链路;`EKP-026~029` 已完成 Docling Adapter、
规范解析模型、结构化 Spreadsheet Chunk 和 P0 文件路由;`EKP-100` 已建立 40 条合成 Query 的
版本化 Golden Dataset 及 Dense/Sparse/Hybrid/Reranked 聚合指标基线;`EKP-042` 已提供未激活的
Qwen3 0.6B/4B/8B OpenAI-compatible Embedding Adapter,当前默认模型保持不变。
状态与边界参见:
- [`docs/plans/TASKS.md`](docs/plans/TASKS.md)
- [`docs/plans/PROGRESS.md`](docs/plans/PROGRESS.md)
- [`docs/development/CONTAINER_IMAGES.md`](docs/development/CONTAINER_IMAGES.md)
## 架构基线
```text
Admin Console ─ REST ─┐
├─ Knowledge Service ─ OceanBase MySQL / BOS / OceanBase Vector
WorkBuddy ─ MCP ──────┘ │
Celery / RabbitMQ Worker
```
核心原则:
- API First;
- 模块化单体 + 独立异步 Worker + 独立 MCP Gateway;
- OceanBase 关系表与 BOS 保存规范数据,向量表只是可重建索引;
- Provider 通过接口隔离,业务层不依赖基础设施 SDK;
- 原始 Source 不可变,派生知识必须可追溯;
- ACL 必须在检索前或检索过程中执行,禁止 Top-K 后过滤。
完整架构参见 [`docs/architecture/ARCHITECTURE.md`](docs/architecture/ARCHITECTURE.md),已接受决策参见 [`docs/architecture/ADR.md`](docs/architecture/ADR.md)。
## 本地启动
前置条件:Python 3.12、`uv`;修改管理端静态资源时还需 Node.js 20.18/npm 10.8;需要本地依赖时
再安装 Docker。
```bash
cp .env.example .env
make install
make run-worker
make run
```
`make install` 和所有 Python 运行/检查入口使用冻结的 `uv.lock`。有意修改 `pyproject.toml` 依赖时,
先执行 `make lock` 并审查锁文件差异;可用 `make toolchain-check` 验证解释器、静态工具目标和冻结
安装计划一致。
管理端保持零构建且无第三方 npm 依赖;修改 `apps/admin-web` 后运行 `make web-check`,检查锁文件、
JavaScript 模块语法和本地版本化资源引用。
MCP Gateway 正式身份从运行时 `MCP_IDENTITY_CREDENTIALS` 读取;仅在本地公共兼容模式下使用长度
至少为 32 个字符的 `MCP_AUTH_TOKEN`:
```bash
export MCP_AUTH_TOKEN="<runtime-secret-with-at-least-32-characters>"
make run-mcp
```
默认入口:
| 能力 | 地址 |
|---|---|
| 管理控制台 | `http://localhost:8000/admin` |
| 健康检查 | `http://localhost:8000/healthz` |
| 指标 | `http://localhost:8000/metrics` |
| OpenAPI | `http://localhost:8000/api/v1/openapi.json` |
| MCP | `http://127.0.0.1:8766/mcp` |
运行时配置、身份边界和部署要求分别参见 `docs/development/` 下的对应手册。Secret 只能通过受控运行环境注入;不得把未获授权的凭据写入 Git、镜像或日志。
## 开发门禁
```bash
make ci
```
该入口包含锁文件/工具链、管理端、静态检查、Unit/Integration、Migration、离线 RAG Evaluation 和
真实进程 API/MCP E2E;范围与安全边界见
[`docs/development/CI_BASELINE.md`](docs/development/CI_BASELINE.md)。
真实 Provider 验收默认禁用。只有满足 [`docs/development/LIVE_ACCEPTANCE.md`](docs/development/LIVE_ACCEPTANCE.md) 的安全确认和清理要求后,才能运行:
```bash
make live-acceptance
```
## 文档入口
文档职责、权威性、生命周期和完整导航统一记录在 [`docs/README.md`](docs/README.md)。开始开发前还必须阅读 [`AGENTS.md`](AGENTS.md) 和 [`CODEX_START_HERE.md`](CODEX_START_HERE.md)。

38
hy4/_kp/alembic.ini Normal file
View File

@ -0,0 +1,38 @@
[alembic]
script_location = %(here)s/migrations
prepend_sys_path = .
path_separator = os
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

View File

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,200 @@
/* Low-risk readability and state affordances layered over the admin console. */
.stage-final [class*="candidate"],
.stage-final [class*="result"] {
grid-template-columns: minmax(0, 1fr) !important;
}
.ux-clear-query {
min-height: 42px;
padding: 0 18px;
border: 1px solid var(--border-color, #cbd5df);
border-radius: 10px;
background: #fff;
color: var(--text-color, #24313d);
font: inherit;
font-weight: 600;
cursor: pointer;
}
.ux-clear-query:hover {
border-color: #8395a7;
background: #f6f8fa;
}
.ux-readable-dialog label,
.ux-readable-form label,
.ux-readable-dialog .field-label,
.ux-readable-form .field-label,
.ux-readable-dialog [class*="label"],
.ux-readable-form [class*="label"] {
font-size: 15px;
line-height: 1.5;
}
.ux-readable-dialog select,
.ux-readable-form select,
.ux-readable-dialog option,
.ux-readable-form option,
.ux-readable-dialog input,
.ux-readable-form input,
.ux-readable-dialog textarea {
min-height: 44px;
font-size: 15px;
line-height: 1.5;
}
.ux-readable-dialog select,
.ux-readable-form select,
.ux-readable-dialog input,
.ux-readable-form input,
.ux-readable-dialog textarea,
.ux-readable-form textarea {
padding: 10px 12px;
}
.ux-management-page table {
font-size: 15px;
}
.ux-management-page th {
padding-block: 12px;
color: #526272;
font-size: 13px;
font-weight: 700;
letter-spacing: 0.02em;
}
.ux-management-page td {
padding-block: 15px;
line-height: 1.55;
vertical-align: middle;
}
.ux-management-page :is(.card, [class*="card"], [class*="item"], [class*="row"]) strong {
color: #1d2a36;
font-size: 16px;
line-height: 1.45;
}
.ux-management-page :is(.card, [class*="card"], [class*="item"], [class*="row"]) :is(small, .muted, [class*="meta"], [class*="description"]) {
font-size: 13px;
line-height: 1.55;
}
.ux-management-page :is(.card, [class*="card"], [class*="item"]) {
gap: 10px;
}
@media (max-width: 640px) {
.ux-management-page table {
font-size: 14px;
}
}
#search-page .search-panel label,
#search-page .top-k-control label {
font-size: 14px;
font-weight: 600;
line-height: 1.5;
}
#search-page .top-k-control {
width: 88px;
min-width: 88px;
}
#search-top-k {
width: 88px;
min-height: 38px;
padding: 6px 28px 6px 10px;
font-size: 14px;
line-height: 1.4;
}
.page select,
.page select option,
dialog select,
dialog select option {
color: #344454;
font-size: 14px;
font-weight: 400;
letter-spacing: 0;
}
.page select:disabled,
dialog select:disabled,
.page select option:disabled,
dialog select option:disabled {
color: #7c8995;
}
.page select:focus,
dialog select:focus {
border-color: #75889a;
box-shadow: 0 0 0 3px rgba(58, 78, 96, 0.1);
}
#search-page .ux-clear-query {
width: auto !important;
min-width: 0;
min-height: 32px;
padding: 0 10px;
border-radius: 7px;
font-size: 12px;
font-weight: 600;
flex: 0 0 auto;
}
#search-page .query-heading {
display: flex;
align-items: center;
justify-content: space-between;
gap: 16px;
}
#search-page .query-heading label {
margin: 0;
}
.brand-mark {
width: 60px;
height: 45px;
flex: 0 0 auto;
}
.brand {
gap: 14px;
}
.brand strong {
font-size: 18px;
line-height: 1.25;
}
.brand small {
font-size: 11px;
line-height: 1.35;
}
.workspace-label strong {
font-size: 19px;
line-height: 1.3;
}
.sidebar-foot {
font-size: 13px;
}
@media (max-width: 540px) {
#search-page .top-k-control,
#search-top-k {
width: 80px;
min-width: 80px;
}
.brand-mark {
width: 54px;
height: 41px;
}
}

View File

@ -0,0 +1,670 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta
name="description"
content="Enterprise Knowledge Platform 管理控制台"
/>
<title>地瓜知识平台</title>
<link rel="stylesheet" href="/admin/assets/styles.css?v=knowledge-tree-14" />
<link rel="stylesheet" href="/admin/assets/ux-enhancements.css?v=admin-ux-refresh-8" />
<script type="module" src="/admin/assets/app.js?v=knowledge-tree-14"></script>
</head>
<body>
<div class="app-shell">
<aside class="sidebar" aria-label="管理控制台导航">
<a class="brand" href="/admin/overview" aria-label="地瓜机器人首页">
<svg class="brand-mark" viewBox="0 0 48 36" aria-hidden="true">
<path d="M15 3h17a15 15 0 0 1 0 30H15v-9h17a6 6 0 0 0 0-12H15z" />
<circle cx="11" cy="18" r="8" />
</svg>
<span>
<strong>地瓜机器人</strong>
<small>Knowledge Platform</small>
</span>
</a>
<div class="workspace-label">
<span>CONTROL DESK</span>
<strong>地瓜知识平台</strong>
</div>
<nav class="primary-nav">
<p class="nav-group-label">知识运营</p>
<a data-nav="overview" href="/admin/overview">
知识总览
</a>
<a data-nav="upload" data-required-scope="knowledge:write" href="/admin/upload">
文档导入
</a>
<a data-nav="documents" href="/admin/documents">
文档管理
</a>
<a data-nav="compiled-knowledge" href="/admin/compiled-knowledge">
编纂知识
</a>
<a data-nav="search" href="/admin/search">
知识检索
</a>
<p class="nav-group-label">系统管理</p>
<a data-nav="knowledge-bases" data-required-scope="knowledge:manage" href="/admin/knowledge-bases">
知识库管理
</a>
<a data-nav="permissions" data-required-scope="knowledge:manage" href="/admin/permissions">
权限管理
</a>
</nav>
<div class="sidebar-foot">
<span class="environment-dot" aria-hidden="true"></span>
地瓜知识平台 · V1
</div>
</aside>
<main class="workspace">
<header class="topbar">
<div>
<p class="eyebrow">KNOWLEDGE OPERATIONS</p>
<h1 id="page-title">知识总览</h1>
</div>
<div class="topbar-actions">
<button id="identity-button" class="identity-button" type="button">
<span id="identity-name">连接访问身份</span>
<small id="identity-organization">访问令牌</small>
</button>
<span class="system-state"><i aria-hidden="true"></i>API FIRST · V1</span>
<a class="api-link" href="/api/v1/docs" target="_blank" rel="noreferrer">
API 文档 <span aria-hidden="true">↗</span>
</a>
</div>
</header>
<div class="signal-line" aria-hidden="true"><span></span><span></span><span></span></div>
<div id="notice" class="notice" role="status" aria-live="polite" hidden></div>
<section id="overview-page" class="page" data-page="overview" aria-labelledby="page-title">
<div class="page-heading overview-heading">
<div>
<p class="section-label">OVERVIEW / LIVE INVENTORY</p>
<h2>查看知识处理与检索状态</h2>
<p>快速了解文档总量、处理进度、可检索比例和需要关注的异常。</p>
</div>
<button id="refresh-overview" class="secondary-button" type="button">刷新总览</button>
</div>
<section class="knowledge-flow" aria-labelledby="knowledge-flow-title">
<div class="flow-intro">
<p class="section-label">KNOWLEDGE FLOW</p>
<h3 id="knowledge-flow-title">文档处理概览</h3>
<span id="overview-updated-at">尚未加载</span>
</div>
<div class="flow-rail" aria-live="polite">
<article class="flow-node flow-source">
<span class="flow-code">SOURCE</span>
<strong id="overview-total">—</strong>
<small>全部文档</small>
</article>
<span class="flow-connector" aria-hidden="true"></span>
<article class="flow-node flow-processing">
<span class="flow-code">PROCESS</span>
<strong id="overview-processing">—</strong>
<small>等待或处理中</small>
</article>
<span class="flow-connector" aria-hidden="true"></span>
<article class="flow-node flow-ready">
<span class="flow-code">READY</span>
<strong id="overview-ready">—</strong>
<small>可检索</small>
</article>
<span class="flow-connector" aria-hidden="true"></span>
<article class="flow-node flow-retrieval">
<span class="flow-code">RETRIEVAL</span>
<strong id="overview-readiness">—</strong>
<small>入库就绪率</small>
</article>
</div>
<div id="overview-alert" class="flow-alert" hidden></div>
</section>
<div class="overview-grid">
<section class="overview-ledger" aria-labelledby="recent-documents-title">
<div class="overview-section-heading">
<div>
<p class="section-label">RECENT SOURCES</p>
<h3 id="recent-documents-title">最近文档</h3>
</div>
<a class="text-link" href="/admin/documents">查看全部 <span>→</span></a>
</div>
<div id="overview-recent-documents" class="overview-document-list" aria-live="polite">
<div class="content-state">正在读取文档库存…</div>
</div>
</section>
<aside class="overview-actions" aria-labelledby="next-actions-title">
<div class="overview-section-heading">
<div>
<p class="section-label">NEXT ACTION</p>
<h3 id="next-actions-title">继续操作</h3>
</div>
</div>
<a class="operation-link" href="/admin/upload">
<span>导入文档</span>
<strong>上传并处理知识文件</strong>
<i aria-hidden="true">↗</i>
</a>
<a class="operation-link" href="/admin/search">
<span>测试检索</span>
<strong>检索并核对来源</strong>
<i aria-hidden="true">↗</i>
</a>
<div class="scope-note">
<span class="scope-indicator" aria-hidden="true"></span>
<p><strong>当前可访问范围</strong>这里只显示当前身份有权查看和检索的知识。</p>
</div>
</aside>
</div>
</section>
<section id="upload-page" class="page" data-page="upload" aria-labelledby="page-title" hidden>
<div class="page-heading">
<div>
<p class="section-label">DOCUMENT INGESTION</p>
<h2>上传并处理知识文件</h2>
<p>选择所属知识库和文件夹后上传文件,系统会逐个完成解析并建立检索索引。</p>
</div>
</div>
<form id="upload-form" class="upload-panel">
<div class="scope-selectors">
<label for="upload-knowledge-base">目标知识库
<select id="upload-knowledge-base" name="knowledge_base_id" required>
<option value="">请先连接访问身份并选择知识库</option>
</select>
</label>
<label for="upload-folder">目标文件夹(可选)
<select id="upload-folder" name="folder_id" disabled>
<option value="">知识库根目录</option>
</select>
</label>
</div>
<div id="drop-zone" class="drop-zone">
<span class="upload-icon" aria-hidden="true">↑</span>
<strong>选择文件、整个文件夹或 ZIP 压缩包</strong>
<span>支持 PDF、DOCX、PPTX、XLSX、CSV、HTML、TXT 和 Markdown;图片与扫描文档识别能力取决于当前解析配置</span>
<div class="upload-source-actions" aria-label="选择上传来源">
<label class="upload-source-button" for="file-input">
选择文件
<input
id="file-input"
name="files"
type="file"
multiple
accept=".pdf,.docx,.pptx,.xlsx,.csv,.html,.htm,.txt,.md,.jpg,.jpeg,.png,application/pdf,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.presentationml.presentation,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,text/csv,text/html,text/plain,text/markdown,image/jpeg,image/png"
/>
</label>
<label class="upload-source-button directory" for="folder-input">
选择文件夹
<input
id="folder-input"
name="directory_files"
type="file"
multiple
webkitdirectory
directory
accept=".pdf,.docx,.pptx,.xlsx,.csv,.html,.htm,.txt,.md,.jpg,.jpeg,.png,application/pdf,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.presentationml.presentation,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,text/csv,text/html,text/plain,text/markdown,image/jpeg,image/png"
/>
</label>
<label class="upload-source-button archive" for="archive-input">
选择 ZIP 压缩包
<input
id="archive-input"
name="archive"
type="file"
accept=".zip,application/zip,application/x-zip-compressed"
/>
</label>
</div>
<span class="upload-limit-hint">文件夹和 ZIP 会保留多层目录;普通批次最多 20 个文件,ZIP 最多 1000 个文件</span>
<span id="selected-file" class="selected-file">尚未选择文件、文件夹或 ZIP</span>
</div>
<button id="upload-button" class="primary-button" type="submit">上传并建立目录</button>
</form>
<article id="upload-result" class="result-card" hidden>
<div>
<p class="section-label">LATEST BATCH</p>
<h3 id="upload-batch-title"></h3>
</div>
<span id="upload-status" class="status-pill"></span>
<dl class="metadata-grid">
<div><dt>总文件</dt><dd id="upload-total"></dd></div>
<div><dt>已完成</dt><dd id="upload-completed"></dd></div>
<div><dt>接收 / 拒绝</dt><dd id="upload-accepted"></dd></div>
</dl>
<div id="batch-items" class="batch-items" aria-live="polite"></div>
<a class="text-link" href="/admin/search">前往知识检索 <span>→</span></a>
</article>
</section>
<section id="documents-page" class="page" data-page="documents" aria-labelledby="page-title" hidden>
<div class="page-heading heading-row">
<div>
<p class="section-label">DOCUMENT MANAGEMENT</p>
<h2>查看文档与处理结果</h2>
<p>查看文档状态、原始文件、解析内容和处理记录,并核对检索使用的来源。</p>
</div>
<button id="refresh-documents" class="secondary-button" type="button">刷新列表</button>
</div>
<section class="document-scope-filter" aria-labelledby="document-filter-title">
<div class="document-filter-heading">
<div>
<p class="section-label">DOCUMENT SCOPE</p>
<h3 id="document-filter-title">缩小文档范围</h3>
</div>
<button id="reset-document-filters" class="table-action" type="button">
查看全部
</button>
</div>
<div class="document-filter-fields">
<label for="document-filter-knowledge-base">
<span><b>01</b> 知识库</span>
<select id="document-filter-knowledge-base">
<option value="">全部可访问知识库</option>
</select>
</label>
<label for="document-filter-folder">
<span><b>02</b> 文件夹范围</span>
<select id="document-filter-folder" disabled>
<option value="all">先选择知识库</option>
</select>
</label>
</div>
<p id="document-filter-summary" class="document-filter-summary" aria-live="polite">
当前显示全部可访问文档
</p>
</section>
<div class="table-card">
<div class="table-meta">
<span id="document-count">0 个文档</span>
<span id="document-refresh-time">尚未加载</span>
</div>
<div class="table-scroll">
<table>
<thead>
<tr>
<th>文件名</th>
<th>类型</th>
<th>上传时间</th>
<th>状态</th>
<th aria-label="操作"></th>
</tr>
</thead>
<tbody id="document-rows"></tbody>
</table>
</div>
<div id="document-empty" class="empty-state" hidden>
<strong id="document-empty-title">还没有文档</strong>
<span id="document-empty-detail">上传第一个文件后,它会出现在这里。</span>
<button id="document-empty-reset" class="secondary-button compact" type="button" hidden>
清除筛选
</button>
<a id="document-empty-upload" class="primary-button compact" href="/admin/upload">
导入文档
</a>
</div>
</div>
</section>
<section id="search-page" class="page" data-page="search" aria-labelledby="page-title" hidden>
<div class="page-heading">
<div>
<p class="section-label">KNOWLEDGE RETRIEVAL</p>
<h2>检索知识并核对来源</h2>
<p>输入问题或关键词,查看匹配的知识来源及检索过程;这里不会生成回答。</p>
</div>
</div>
<form id="search-form" class="search-panel">
<div class="query-heading">
<label for="search-query">问题或关键词</label>
<button id="clear-search-query" class="ux-clear-query" type="button">清空查询</button>
</div>
<div class="search-controls">
<textarea
id="search-query"
name="query"
rows="3"
maxlength="1600"
placeholder="例如:地瓜干的作用是什么?"
required
></textarea>
<label class="top-k-control" for="search-top-k">
结果数量
<select id="search-top-k" name="top_k">
<option value="1">1</option>
<option value="3">3</option>
<option value="5" selected>5</option>
</select>
</label>
<button id="search-button" class="primary-button" type="submit">开始检索</button>
</div>
</form>
<section id="search-diagnostics" class="diagnostic-board" aria-labelledby="diagnostic-title" hidden>
<div class="diagnostic-heading">
<div>
<p class="section-label">RETRIEVAL TRACE</p>
<h3 id="diagnostic-title">检索过程</h3>
</div>
<span id="diagnostic-summary"></span>
</div>
<dl id="diagnostic-context" class="diagnostic-context"></dl>
<div id="diagnostic-stages" class="diagnostic-stages"></div>
</section>
<div class="results-heading">
<div>
<p class="section-label">RESULTS</p>
<h3>来源结果</h3>
</div>
<span id="search-summary">等待检索</span>
</div>
<div id="search-results" class="search-results" aria-live="polite">
<div class="empty-state search-empty">
<strong>检索结果会显示在这里</strong>
<span>结果包含可读上下文、命中片段、来源位置和原文入口。</span>
</div>
</div>
</section>
<section id="compiled-knowledge-page" class="page" data-page="compiled-knowledge" aria-labelledby="page-title" hidden>
<div class="page-heading heading-row">
<div>
<p class="section-label">COMPILED KNOWLEDGE / REVIEW</p>
<h2>让每条知识带着来源走完审核</h2>
<p>从真实片段形成草稿,核验版本和权限后提交审核;冲突必须留下双来源证据并解决后才能发布。</p>
</div>
<button id="refresh-compiled-knowledge" class="secondary-button" type="button">刷新草稿</button>
</div>
<div class="compilation-layout">
<form id="compiled-draft-form" class="management-list compilation-editor" data-required-scope="knowledge:write">
<div class="management-heading">
<div><p class="section-label">DRAFT EDITOR</p><h3 id="compiled-editor-title">新建草稿</h3></div>
<button id="cancel-compiled-edit" class="table-action" type="button" hidden>取消编辑</button>
</div>
<label>知识库<select id="compiled-knowledge-base" required></select></label>
<label>知识类型
<select id="compiled-knowledge-type" required>
<option value="summary">Summary</option>
<option value="entity">Entity</option>
<option value="concept">Concept</option>
<option value="faq">FAQ</option>
<option value="sop">SOP</option>
<option value="comparison">Comparison</option>
<option value="overview">Overview</option>
<option value="custom_wiki_page">Custom Wiki Page</option>
</select>
</label>
<label>标题<input id="compiled-title" maxlength="255" required /></label>
<label>草稿正文<textarea id="compiled-content" rows="10" maxlength="100000" required></textarea></label>
<fieldset id="compiled-source-fieldset" class="compilation-sources">
<legend>可追溯来源</legend>
<label>来源文档<select id="compiled-source-document" required></select></label>
<div id="compiled-source-chunks" class="source-choice-list"><div class="content-state">请选择知识库和来源文档。</div></div>
</fieldset>
<p id="compiled-edit-source-note" class="field-note" hidden>普通编辑保留原来源;修订失效知识时必须重新选择当前有效片段。</p>
<div class="management-actions compilation-actions">
<button id="save-compiled-draft" class="primary-button compact" type="submit">保存手工草稿</button>
<button id="generate-compiled-draft" class="secondary-button" type="button">AI 生成草稿</button>
</div>
</form>
<section class="management-list" aria-labelledby="compiled-list-title">
<div class="management-heading">
<div><p class="section-label">KNOWLEDGE LIFECYCLE</p><h3 id="compiled-list-title">可见知识与审核状态</h3></div>
<span id="compiled-count">0 条</span>
</div>
<div id="compiled-list" class="resource-list"><div class="content-state">连接访问身份后加载草稿…</div></div>
</section>
</div>
</section>
<section id="knowledge-bases-page" class="page" data-page="knowledge-bases" aria-labelledby="page-title" hidden>
<div class="page-heading heading-row">
<div>
<p class="section-label">KNOWLEDGE BASE MANAGEMENT</p>
<h2>管理知识库与文件夹</h2>
<p>展开知识库和文件夹层级,快速看清内容组织;需要新增内容时,从右侧按钮进入独立操作窗口。</p>
</div>
<button id="open-knowledge-base-dialog" class="primary-button compact" type="button">新建知识库</button>
</div>
<section class="management-list" aria-labelledby="knowledge-base-list-title">
<div class="management-heading"><div><p class="section-label">KNOWLEDGE SPACES</p><h3 id="knowledge-base-list-title">知识库与文件夹</h3></div><button id="refresh-knowledge-bases" class="secondary-button" type="button">刷新</button></div>
<div id="knowledge-base-list" class="resource-list"><div class="content-state">连接访问身份后加载知识库…</div></div>
</section>
</section>
<section id="permissions-page" class="page" data-page="permissions" aria-labelledby="page-title" hidden>
<div class="page-heading">
<p class="section-label">IDENTITY & ACCESS</p>
<h2>管理成员与知识库权限</h2>
<p>集中查看用户、用户组、角色和授权关系;新增与分配操作通过独立窗口完成。</p>
</div>
<div class="management-actions" aria-label="权限管理新增操作">
<button id="open-user-dialog" class="primary-button compact" type="button">新建用户</button>
<button id="open-group-dialog" class="secondary-button" type="button">新建用户组</button>
<button id="open-role-dialog" class="secondary-button" type="button">新建角色</button>
<button id="open-group-membership-dialog" class="secondary-button" type="button">添加用户组成员</button>
<button id="open-user-role-dialog" class="secondary-button" type="button">分配用户角色</button>
<button id="open-group-role-dialog" class="secondary-button" type="button">分配用户组角色</button>
<button id="open-acl-dialog" class="secondary-button" type="button">新增授权规则</button>
</div>
<section class="management-list">
<div class="management-heading"><div><p class="section-label">DIRECTORY</p><h3>用户与权限关系</h3></div><button id="refresh-permissions" class="secondary-button" type="button">刷新</button></div>
<h4 class="directory-heading">用户</h4>
<div id="user-list" class="resource-list"><div class="content-state">连接访问身份后加载用户…</div></div>
<h4 class="directory-heading">用户组</h4>
<div id="group-list" class="resource-list"><div class="content-state">连接访问身份后加载用户组…</div></div>
<h4 class="directory-heading">角色</h4>
<div id="role-list" class="resource-list"><div class="content-state">连接访问身份后加载角色…</div></div>
<div class="acl-view-heading">
<h4 class="directory-heading">知识库授权</h4>
<label>查看知识库<select id="acl-view-knowledge-base"></select></label>
</div>
<div id="acl-list" class="resource-list compact-list"></div>
</section>
</section>
</main>
</div>
<dialog id="compiled-action-dialog" class="management-dialog wide-management-dialog" aria-labelledby="compiled-action-title">
<form id="compiled-action-form">
<div class="management-dialog-heading">
<div><p class="section-label">SOURCE REVIEW</p><h2 id="compiled-action-title">记录来源冲突</h2></div>
<button class="icon-button dialog-close" type="button" aria-label="关闭">×</button>
</div>
<p id="compiled-action-context" class="dialog-context"></p>
<input id="compiled-action-mode" type="hidden" />
<input id="compiled-action-id" type="hidden" />
<input id="compiled-action-revision" type="hidden" />
<input id="compiled-action-conflict-id" type="hidden" />
<div id="compiled-conflict-fields">
<label>冲突主张<textarea id="compiled-conflict-claim" rows="3" maxlength="2000"></textarea></label>
<div class="conflict-source-selectors">
<label>来源 A<select id="compiled-conflict-left"></select></label>
<label>来源 B<select id="compiled-conflict-right"></select></label>
</div>
</div>
<label id="compiled-action-note-field" hidden><span id="compiled-action-note-label">处理说明</span><textarea id="compiled-action-note" rows="4" maxlength="2000"></textarea></label>
<div class="dialog-actions">
<button class="secondary-button dialog-cancel" type="button">取消</button>
<button id="compiled-action-submit" class="primary-button" type="submit">记录冲突</button>
</div>
</form>
</dialog>
<dialog id="knowledge-base-dialog" class="management-dialog" aria-labelledby="knowledge-base-dialog-title">
<form id="knowledge-base-form">
<div class="management-dialog-heading"><div><p class="section-label">NEW KNOWLEDGE BASE</p><h2 id="knowledge-base-dialog-title">新建知识库</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>名称<input id="knowledge-base-name" maxlength="255" required /></label>
<label>描述<textarea id="knowledge-base-description" rows="3"></textarea></label>
<label>可见性<select id="knowledge-base-visibility"><option value="private">私有</option><option value="organization">组织内可见</option></select></label>
<label class="check-field"><input id="knowledge-base-mcp" type="checkbox" />允许 MCP 使用</label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建知识库</button></div>
</form>
</dialog>
<dialog id="folder-dialog" class="management-dialog" aria-labelledby="folder-dialog-title">
<form id="folder-form">
<div class="management-dialog-heading"><div><p class="section-label">NEW FOLDER</p><h2 id="folder-dialog-title">新建文件夹</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<p id="folder-knowledge-base-context" class="dialog-context"></p>
<input id="folder-knowledge-base-id" type="hidden" />
<label>文件夹名称<input id="folder-name" maxlength="255" required /></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建文件夹</button></div>
</form>
</dialog>
<dialog id="user-dialog" class="management-dialog" aria-labelledby="user-dialog-title">
<form id="user-form">
<div class="management-dialog-heading"><div><p class="section-label">NEW USER</p><h2 id="user-dialog-title">新建用户</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>显示名称<input id="user-display-name" maxlength="255" required /></label>
<label>身份签发方(Issuer)<input id="user-issuer" maxlength="255" value="local" required /></label>
<label>身份唯一标识(Subject)<input id="user-subject" maxlength="255" required /></label>
<label>邮箱(可选)<input id="user-email" type="email" maxlength="255" /></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建用户</button></div>
</form>
</dialog>
<dialog id="group-dialog" class="management-dialog" aria-labelledby="group-dialog-title">
<form id="group-form">
<div class="management-dialog-heading"><div><p class="section-label">NEW GROUP</p><h2 id="group-dialog-title">新建用户组</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>用户组名称<input id="group-name" maxlength="255" required /></label>
<label>描述(可选)<textarea id="group-description" rows="2"></textarea></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建用户组</button></div>
</form>
</dialog>
<dialog id="role-dialog" class="management-dialog" aria-labelledby="role-dialog-title">
<form id="role-form">
<div class="management-dialog-heading"><div><p class="section-label">NEW ROLE</p><h2 id="role-dialog-title">新建角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>角色名称<input id="role-name" maxlength="255" required /></label>
<label>描述(可选)<textarea id="role-description" rows="2"></textarea></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建角色</button></div>
</form>
</dialog>
<dialog id="group-membership-dialog" class="management-dialog" aria-labelledby="group-membership-dialog-title">
<form id="group-membership-form">
<div class="management-dialog-heading"><div><p class="section-label">GROUP MEMBERSHIP</p><h2 id="group-membership-dialog-title">添加用户组成员</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>用户<select id="membership-user" required></select></label>
<label>用户组<select id="membership-group" required></select></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">添加成员</button></div>
</form>
</dialog>
<dialog id="user-role-dialog" class="management-dialog" aria-labelledby="user-role-dialog-title">
<form id="user-role-form">
<div class="management-dialog-heading"><div><p class="section-label">USER ROLE</p><h2 id="user-role-dialog-title">分配用户角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>用户<select id="user-role-user" required></select></label>
<label>角色<select id="user-role-role" required></select></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">分配角色</button></div>
</form>
</dialog>
<dialog id="group-role-dialog" class="management-dialog" aria-labelledby="group-role-dialog-title">
<form id="group-role-form">
<div class="management-dialog-heading"><div><p class="section-label">GROUP ROLE</p><h2 id="group-role-dialog-title">分配用户组角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>用户组<select id="group-role-group" required></select></label>
<label>角色<select id="group-role-role" required></select></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">分配角色</button></div>
</form>
</dialog>
<dialog id="acl-dialog" class="management-dialog wide-management-dialog" aria-labelledby="acl-dialog-title">
<form id="acl-form">
<div class="management-dialog-heading"><div><p class="section-label">ACCESS RULE</p><h2 id="acl-dialog-title">新增授权规则</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
<label>授权对象类型<select id="acl-subject-type"><option value="user">用户</option><option value="group">用户组</option><option value="role">角色</option></select></label>
<label>授权对象<select id="acl-subject" required></select></label>
<label>知识库<select id="acl-knowledge-base" required></select></label>
<label>权限<select id="acl-permission"><option value="read">读取</option><option value="write">写入</option><option value="manage">管理</option></select></label>
<label>规则<select id="acl-effect"><option value="allow">允许</option><option value="deny">拒绝</option></select></label>
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">保存授权规则</button></div>
</form>
</dialog>
<dialog id="document-dialog" class="document-dialog" aria-labelledby="dialog-filename">
<form method="dialog"><button class="icon-button" aria-label="关闭文档">×</button></form>
<div class="dialog-heading">
<div>
<p class="section-label">DOCUMENT INSPECTOR</p>
<h2 id="dialog-filename"></h2>
</div>
<button id="dialog-source-link" class="secondary-button source-link" type="button">
新窗口打开原始文件 <span aria-hidden="true">↗</span>
</button>
</div>
<details class="inspector-metadata">
<summary>
<span>文档信息</span>
<small id="dialog-metadata-summary"></small>
</summary>
<dl id="dialog-metadata" class="dialog-metadata"></dl>
</details>
<details id="processing-inspector" class="processing-inspector">
<summary>
<span>处理流水线</span>
<small id="processing-summary"></small>
</summary>
<div id="processing-content" class="processing-content"></div>
</details>
<div class="inspector-view-switcher" role="tablist" aria-label="文档查看方式">
<button
id="source-preview-tab"
class="inspector-tab active"
type="button"
role="tab"
aria-controls="source-preview-panel"
aria-selected="true"
>原始文件预览</button>
<button
id="parsed-content-tab"
class="inspector-tab"
type="button"
role="tab"
aria-controls="parsed-content-panel"
aria-selected="false"
>解析与溯源</button>
<span id="inspector-view-status">正在读取…</span>
</div>
<div id="document-quality" class="quality-banner" role="status" hidden></div>
<section
id="source-preview-panel"
class="inspector-panel source-preview-panel"
role="tabpanel"
aria-labelledby="source-preview-tab"
>
<div id="source-preview" class="source-preview" aria-live="polite"></div>
</section>
<section
id="parsed-content-panel"
class="inspector-panel document-content"
role="tabpanel"
aria-labelledby="parsed-content-tab"
hidden
>
<div class="document-content-heading">
<div>
<p class="section-label">SOURCE TRACK</p>
<h3 id="document-content-title">解析内容(按 Chunk 顺序)</h3>
</div>
<span id="dialog-content-summary">正在读取…</span>
</div>
<div id="dialog-content" class="source-track" aria-live="polite"></div>
</section>
</dialog>
<dialog id="identity-dialog" class="identity-dialog" aria-labelledby="identity-dialog-title">
<form id="identity-form">
<p class="section-label">ACCESS IDENTITY</p>
<h2 id="identity-dialog-title">使用访问令牌进入</h2>
<p>访问令牌仅保存在当前浏览器标签页中,关闭标签页后会自动清除。</p>
<label for="identity-token">访问令牌</label>
<input id="identity-token" type="password" minlength="32" autocomplete="off" required />
<div class="dialog-actions"><button id="identity-cancel" class="secondary-button" type="button">取消</button><button class="primary-button" type="submit">验证并进入</button></div>
</form>
</dialog>
</body>
</html>

15
hy4/_kp/apps/admin-web/package-lock.json generated Normal file
View File

@ -0,0 +1,15 @@
{
"name": "@ekp/admin-web",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@ekp/admin-web",
"version": "0.1.0",
"engines": {
"node": ">=20.18.0 <21"
}
}
}
}

View File

@ -0,0 +1,16 @@
{
"name": "@ekp/admin-web",
"version": "0.1.0",
"private": true,
"description": "Zero-build Enterprise Knowledge Platform administration console",
"type": "module",
"packageManager": "npm@10.8.2",
"engines": {
"node": ">=20.18.0 <21"
},
"scripts": {
"check": "npm run check:javascript && npm run check:assets",
"check:javascript": "node --check assets/app.js",
"check:assets": "node scripts/check-assets.mjs"
}
}

View File

@ -0,0 +1,33 @@
/** Validate that the zero-build console references only present, local runtime assets. */
import { accessSync, readFileSync } from "node:fs";
import { join } from "node:path";
const webRoot = process.cwd();
const html = readFileSync(join(webRoot, "index.html"), "utf8");
const runtimeReferences = [
...html.matchAll(/<(?:img|link|script)\b[^>]*?(?:href|src)="([^"]+)"/gu),
].map((match) => match[1]);
const remoteReferences = runtimeReferences.filter((reference) =>
/^(?:https?:)?\/\//u.test(reference),
);
if (remoteReferences.length > 0) {
throw new Error(`Admin Web runtime assets must be local: ${remoteReferences.join(", ")}`);
}
const adminAssets = runtimeReferences.filter((reference) =>
reference.startsWith("/admin/assets/"),
);
if (adminAssets.length === 0) {
throw new Error("Admin Web must reference at least one versioned runtime asset");
}
for (const reference of adminAssets) {
const [pathname, cacheVersion] = reference.split("?", 2);
if (!cacheVersion) {
throw new Error(`Admin Web asset is missing a cache version: ${reference}`);
}
accessSync(join(webRoot, pathname.replace("/admin/", "")));
}
console.log(`Admin Web asset contract ok: ${adminAssets.length} local versioned assets`);

View File

View File

@ -0,0 +1 @@
"""Enterprise Knowledge Platform API application."""

View File

@ -0,0 +1,411 @@
"""Authenticated REST administration for canonical V1 identities and resources."""
from typing import Annotated
from uuid import UUID
from fastapi import APIRouter, Depends, HTTPException, Response, status
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
from ekp_api.providers import ApiServices
from ekp_api.routes import get_api_services
from ekp_api.schemas import (
AccessControlEntryCreateRequest,
AccessControlEntryResponse,
AdministrationSessionResponse,
FolderCreateRequest,
FolderResponse,
GroupResponse,
IdentityCollectionCreateRequest,
KnowledgeBaseCreateRequest,
KnowledgeBaseResponse,
RoleResponse,
UserCreateRequest,
UserResponse,
)
from ekp_core.administration import (
AdministrationConflictError,
AdministrationIdentityError,
AdministrationNotFoundError,
AdministrationService,
)
from ekp_domain import AclResourceType
router = APIRouter(prefix="/administration", tags=["administration"])
Services = Annotated[ApiServices, Depends(get_api_services)]
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
@router.get("/session", response_model=AdministrationSessionResponse)
def get_administration_session(
services: Services, identity: Identity
) -> AdministrationSessionResponse:
"""Validate the secret-to-canonical-User mapping used by the console."""
organization_id, user_id = require_canonical_identity(identity)
if services.administration is None:
raise HTTPException(status_code=503, detail="administration service is unavailable")
try:
value = services.administration.get_session(organization_id, user_id)
except AdministrationIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
return AdministrationSessionResponse(
organization_id=value.organization.id,
organization_name=value.organization.name,
user_id=value.user.id,
display_name=value.user.display_name,
scopes=tuple(sorted(identity.scopes)),
credential_type=identity.credential_type,
)
@router.get("/users", response_model=list[UserResponse])
def list_users(services: Services, identity: Identity) -> list[UserResponse]:
organization_id, user_id, administration = _management_context(services, identity)
return [
UserResponse.from_user(value)
for value in administration.list_users(organization_id, user_id)
]
@router.post("/users", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
def create_user(payload: UserCreateRequest, services: Services, identity: Identity) -> UserResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_user(
organization_id,
user_id,
issuer=payload.issuer,
subject=payload.subject,
display_name=payload.display_name,
email=payload.email,
)
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="user already exists") from error
return UserResponse.from_user(value)
@router.get("/groups", response_model=list[GroupResponse])
def list_groups(services: Services, identity: Identity) -> list[GroupResponse]:
organization_id, user_id, administration = _management_context(services, identity)
return [
GroupResponse.from_managed_group(value)
for value in administration.list_groups(organization_id, user_id)
]
@router.post("/groups", response_model=GroupResponse, status_code=status.HTTP_201_CREATED)
def create_group(
payload: IdentityCollectionCreateRequest, services: Services, identity: Identity
) -> GroupResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_group(
organization_id,
user_id,
name=payload.name,
description=payload.description,
)
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="group already exists") from error
return GroupResponse.from_managed_group(value)
@router.put("/groups/{group_id}/members/{member_user_id}", status_code=204)
def add_group_member(
group_id: UUID,
member_user_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_group_membership(group_id, member_user_id, services, identity, present=True)
@router.delete("/groups/{group_id}/members/{member_user_id}", status_code=204)
def remove_group_member(
group_id: UUID,
member_user_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_group_membership(group_id, member_user_id, services, identity, present=False)
@router.get("/roles", response_model=list[RoleResponse])
def list_roles(services: Services, identity: Identity) -> list[RoleResponse]:
organization_id, user_id, administration = _management_context(services, identity)
return [
RoleResponse.from_managed_role(value)
for value in administration.list_roles(organization_id, user_id)
]
@router.post("/roles", response_model=RoleResponse, status_code=status.HTTP_201_CREATED)
def create_role(
payload: IdentityCollectionCreateRequest, services: Services, identity: Identity
) -> RoleResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_role(
organization_id,
user_id,
name=payload.name,
description=payload.description,
)
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="role already exists") from error
return RoleResponse.from_managed_role(value)
@router.put("/roles/{role_id}/users/{assigned_user_id}", status_code=204)
def assign_role_to_user(
role_id: UUID,
assigned_user_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_user_role(role_id, assigned_user_id, services, identity, present=True)
@router.delete("/roles/{role_id}/users/{assigned_user_id}", status_code=204)
def revoke_role_from_user(
role_id: UUID,
assigned_user_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_user_role(role_id, assigned_user_id, services, identity, present=False)
@router.put("/roles/{role_id}/groups/{group_id}", status_code=204)
def assign_role_to_group(
role_id: UUID,
group_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_group_role(role_id, group_id, services, identity, present=True)
@router.delete("/roles/{role_id}/groups/{group_id}", status_code=204)
def revoke_role_from_group(
role_id: UUID,
group_id: UUID,
services: Services,
identity: Identity,
) -> Response:
return _set_group_role(role_id, group_id, services, identity, present=False)
@router.get("/knowledge-bases", response_model=list[KnowledgeBaseResponse])
def list_knowledge_bases(services: Services, identity: Identity) -> list[KnowledgeBaseResponse]:
organization_id, user_id, administration = _management_context(services, identity)
return [
KnowledgeBaseResponse.from_knowledge_base(value)
for value in administration.list_knowledge_bases(organization_id, user_id)
]
@router.post(
"/knowledge-bases",
response_model=KnowledgeBaseResponse,
status_code=status.HTTP_201_CREATED,
)
def create_knowledge_base(
payload: KnowledgeBaseCreateRequest, services: Services, identity: Identity
) -> KnowledgeBaseResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_knowledge_base(
organization_id,
user_id,
name=payload.name,
description=payload.description,
visibility=payload.visibility,
data_processing_policy=payload.data_processing_policy,
mcp_enabled=payload.mcp_enabled,
)
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="knowledge base already exists") from error
return KnowledgeBaseResponse.from_knowledge_base(value)
@router.get(
"/knowledge-bases/{knowledge_base_id}/folders",
response_model=list[FolderResponse],
)
def list_folders(
knowledge_base_id: UUID, services: Services, identity: Identity
) -> list[FolderResponse]:
organization_id, user_id, administration = _management_context(services, identity)
try:
values = administration.list_folders(organization_id, user_id, knowledge_base_id)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="knowledge base not found") from error
return [FolderResponse.from_folder(value) for value in values]
@router.post(
"/knowledge-bases/{knowledge_base_id}/folders",
response_model=FolderResponse,
status_code=status.HTTP_201_CREATED,
)
def create_folder(
knowledge_base_id: UUID,
payload: FolderCreateRequest,
services: Services,
identity: Identity,
) -> FolderResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_folder(
organization_id,
user_id,
knowledge_base_id,
name=payload.name,
description=payload.description,
parent_folder_id=payload.parent_folder_id,
)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="knowledge base or parent not found") from error
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="folder already exists") from error
return FolderResponse.from_folder(value)
@router.get(
"/knowledge-bases/{knowledge_base_id}/access-control",
response_model=list[AccessControlEntryResponse],
)
def list_access_control(
knowledge_base_id: UUID, services: Services, identity: Identity
) -> list[AccessControlEntryResponse]:
organization_id, user_id, administration = _management_context(services, identity)
try:
values = administration.list_access_control_entries(
organization_id, user_id, knowledge_base_id
)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="knowledge base not found") from error
return [AccessControlEntryResponse.from_entry(value) for value in values]
@router.post(
"/knowledge-bases/{knowledge_base_id}/access-control",
response_model=AccessControlEntryResponse,
status_code=status.HTTP_201_CREATED,
)
def create_access_control(
knowledge_base_id: UUID,
payload: AccessControlEntryCreateRequest,
services: Services,
identity: Identity,
) -> AccessControlEntryResponse:
organization_id, user_id, administration = _management_context(services, identity)
try:
value = administration.create_access_control_entry(
organization_id,
user_id,
knowledge_base_id,
subject_type=payload.subject_type,
subject_id=payload.subject_id,
resource_type=AclResourceType(payload.resource_type),
resource_id=payload.resource_id,
permission=payload.permission,
effect=payload.effect,
applies_to_descendants=payload.applies_to_descendants,
)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="ACL subject or resource not found") from error
except AdministrationConflictError as error:
raise HTTPException(status_code=409, detail="access rule already exists") from error
return AccessControlEntryResponse.from_entry(value)
@router.delete("/access-control/{entry_id}", status_code=status.HTTP_204_NO_CONTENT)
def delete_access_control(entry_id: UUID, services: Services, identity: Identity) -> Response:
organization_id, user_id, administration = _management_context(services, identity)
try:
administration.delete_access_control_entry(organization_id, user_id, entry_id)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="access rule not found") from error
return Response(status_code=status.HTTP_204_NO_CONTENT)
def _management_context(
services: ApiServices, identity: ApiIdentity
) -> tuple[UUID, UUID, AdministrationService]:
"""Enforce the management scope before exposing canonical directory data."""
require_scope(identity, "knowledge:manage")
organization_id, user_id = require_canonical_identity(identity)
if services.administration is None:
raise HTTPException(status_code=503, detail="administration service is unavailable")
try:
services.administration.get_session(organization_id, user_id)
except AdministrationIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
return organization_id, user_id, services.administration
def _set_group_membership(
group_id: UUID,
member_user_id: UUID,
services: ApiServices,
identity: ApiIdentity,
*,
present: bool,
) -> Response:
organization_id, actor_user_id, administration = _management_context(services, identity)
try:
administration.set_group_membership(
organization_id,
actor_user_id,
group_id,
member_user_id,
present=present,
)
except (AdministrationIdentityError, AdministrationNotFoundError) as error:
raise HTTPException(status_code=404, detail="group or user not found") from error
return Response(status_code=204)
def _set_user_role(
role_id: UUID,
assigned_user_id: UUID,
services: ApiServices,
identity: ApiIdentity,
*,
present: bool,
) -> Response:
organization_id, actor_user_id, administration = _management_context(services, identity)
try:
administration.set_user_role_assignment(
organization_id,
actor_user_id,
role_id,
assigned_user_id,
present=present,
)
except (AdministrationIdentityError, AdministrationNotFoundError) as error:
raise HTTPException(status_code=404, detail="role or user not found") from error
return Response(status_code=204)
def _set_group_role(
role_id: UUID,
group_id: UUID,
services: ApiServices,
identity: ApiIdentity,
*,
present: bool,
) -> Response:
organization_id, actor_user_id, administration = _management_context(services, identity)
try:
administration.set_group_role_assignment(
organization_id,
actor_user_id,
role_id,
group_id,
present=present,
)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="role or group not found") from error
return Response(status_code=204)

View File

@ -0,0 +1,120 @@
"""Explicit first-identity provisioning for runtime REST bearer mappings."""
import argparse
from dataclasses import dataclass
from sqlalchemy import select
from ekp_core.config import ApiIdentityCredential, Settings, get_settings
from ekp_core.database import create_database_engine, create_session_factory
from ekp_domain import IdentityStatus, Organization, User
class IdentityBootstrapError(RuntimeError):
"""Raised when requested canonical IDs conflict with existing facts."""
@dataclass(frozen=True, slots=True)
class IdentityBootstrapResult:
"""Non-sensitive result suitable for operator output and tests."""
organization_created: bool
user_created: bool
organization_id: str
user_id: str
def provision_api_identity(
settings: Settings,
*,
client_id: str,
organization_slug: str,
organization_name: str,
display_name: str,
) -> IdentityBootstrapResult:
"""Idempotently create the exact Organization/User named by one secret mapping."""
credential = _credential_by_client_id(settings, client_id)
session_factory = create_session_factory(create_database_engine(settings))
organization_created = False
user_created = False
with session_factory.begin() as session:
organization = session.get(Organization, credential.organization_id)
if organization is None:
conflicting = session.scalar(
select(Organization.id).where(Organization.slug == organization_slug.strip())
)
if conflicting is not None:
raise IdentityBootstrapError("organization slug belongs to another identity")
session.add(
Organization(
id=credential.organization_id,
slug=organization_slug.strip(),
name=organization_name.strip(),
)
)
session.flush()
organization_created = True
user = session.get(User, credential.user_id)
if user is None:
session.add(
User(
id=credential.user_id,
organization_id=credential.organization_id,
issuer=f"ekp-api:{credential.credential_type}",
subject=credential.subject,
display_name=display_name.strip(),
status=IdentityStatus.ACTIVE,
)
)
user_created = True
elif (
user.organization_id != credential.organization_id
or user.subject != credential.subject
or user.status is not IdentityStatus.ACTIVE
):
raise IdentityBootstrapError("configured User mapping conflicts with canonical data")
return IdentityBootstrapResult(
organization_created=organization_created,
user_created=user_created,
organization_id=str(credential.organization_id),
user_id=str(credential.user_id),
)
def _credential_by_client_id(settings: Settings, client_id: str) -> ApiIdentityCredential:
matches = [
credential
for credential in settings.api_identity_credentials
if credential.client_id == client_id
]
if len(matches) != 1:
raise IdentityBootstrapError("client_id must select exactly one API credential")
return matches[0]
def main() -> None:
"""Provision one configured identity without accepting a secret on the command line."""
parser = argparse.ArgumentParser(description="Provision one configured EKP API identity")
parser.add_argument("--client-id", required=True)
parser.add_argument("--organization-slug", required=True)
parser.add_argument("--organization-name", required=True)
parser.add_argument("--display-name", required=True)
arguments = parser.parse_args()
result = provision_api_identity(
get_settings(),
client_id=arguments.client_id,
organization_slug=arguments.organization_slug,
organization_name=arguments.organization_name,
display_name=arguments.display_name,
)
print(
"API identity provisioned "
f"organization_id={result.organization_id} user_id={result.user_id} "
f"organization_created={result.organization_created} user_created={result.user_created}"
)
if __name__ == "__main__":
main()

View File

@ -0,0 +1,104 @@
"""Request-scoped REST identity resolved from runtime-secret bearer credentials."""
from dataclasses import dataclass
from secrets import compare_digest
from typing import Literal, Never
from uuid import UUID
from fastapi import HTTPException, Request, status
from ekp_core.config import ApiIdentityCredential
ApiCredentialType = Literal["legacy_public", "user_token", "service_api_key"]
@dataclass(frozen=True, slots=True)
class ApiIdentity:
"""Authenticated REST principal with its canonical authorization subject."""
client_id: str
subject: str
scopes: frozenset[str]
credential_type: ApiCredentialType
organization_id: UUID | None = None
user_id: UUID | None = None
@property
def is_canonical(self) -> bool:
return self.organization_id is not None and self.user_id is not None
class ConfiguredApiIdentityVerifier:
"""Compare every configured secret and return only canonical identity metadata."""
def __init__(self, credentials: tuple[ApiIdentityCredential, ...]) -> None:
if not credentials:
raise ValueError("at least one API identity credential is required")
tokens = [credential.token.get_secret_value() for credential in credentials]
if len(tokens) != len(set(tokens)):
raise ValueError("API identity credential tokens must be unique")
self._credentials = credentials
def verify(self, token: str) -> ApiIdentity | None:
"""Use constant-time comparisons without reflecting or logging bearer values."""
selected: ApiIdentityCredential | None = None
for credential in self._credentials:
if compare_digest(token, credential.token.get_secret_value()):
selected = credential
if selected is None:
return None
return ApiIdentity(
client_id=selected.client_id,
subject=selected.subject,
scopes=frozenset(selected.scopes),
credential_type=selected.credential_type,
organization_id=selected.organization_id,
user_id=selected.user_id,
)
def get_api_identity(request: Request) -> ApiIdentity:
"""Authenticate one REST request or enter explicitly enabled public compatibility."""
settings = request.app.state.settings
credentials = tuple(settings.api_identity_credentials)
authorization = request.headers.get("authorization", "")
test_compatibility = settings.app_env == "test"
if (
not credentials
and (settings.api_allow_legacy_public or test_compatibility)
and not authorization
):
return ApiIdentity(
client_id="legacy-admin-web",
subject="public-knowledge",
scopes=frozenset({"knowledge:read", "knowledge:write", "knowledge:search"}),
credential_type="legacy_public",
)
scheme, separator, token = authorization.partition(" ")
if separator != " " or scheme.lower() != "bearer" or not token:
_raise_unauthorized()
identity = ConfiguredApiIdentityVerifier(credentials).verify(token) if credentials else None
if identity is None:
_raise_unauthorized()
return identity
def require_scope(identity: ApiIdentity, scope: str) -> None:
"""Reject a valid credential that lacks one explicit operation scope."""
if scope not in identity.scopes:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="operation is forbidden")
def require_canonical_identity(identity: ApiIdentity) -> tuple[UUID, UUID]:
"""Return canonical IDs or reject legacy compatibility from protected operations."""
if identity.organization_id is None or identity.user_id is None:
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="operation is forbidden")
return identity.organization_id, identity.user_id
def _raise_unauthorized() -> Never:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="authentication required",
headers={"WWW-Authenticate": "Bearer"},
)

View File

@ -0,0 +1,81 @@
"""Celery/RabbitMQ adapter behind the queue-neutral ingestion dispatcher."""
from uuid import UUID
from celery import Celery # type: ignore[import-untyped]
from ekp_core.config import Settings
INGESTION_TASK_NAME = "ekp.ingestion.run"
RECOVERY_TASK_NAME = "ekp.ingestion.recover"
COMPILATION_TASK_NAME = "ekp.compilation.run"
COMPILATION_RECOVERY_TASK_NAME = "ekp.compilation.recover"
def create_celery_app(settings: Settings) -> Celery:
"""Create a secret-safe broker client without opening a connection eagerly."""
if settings.rabbitmq_url is None or not settings.rabbitmq_url.get_secret_value().strip():
raise ValueError("RABBITMQ_URL is required for asynchronous ingestion")
app = Celery(
"enterprise-knowledge-platform",
broker=settings.rabbitmq_url.get_secret_value(),
)
app.conf.update(
task_ignore_result=True,
task_acks_late=True,
task_reject_on_worker_lost=True,
worker_prefetch_multiplier=1,
broker_connection_retry_on_startup=True,
# Preserve the shared JSON handler instead of letting Celery replace it
# with process-specific plain-text formatting.
worker_hijack_root_logger=False,
timezone="UTC",
enable_utc=True,
task_routes={
INGESTION_TASK_NAME: {"queue": "ingestion"},
RECOVERY_TASK_NAME: {"queue": "ingestion"},
COMPILATION_TASK_NAME: {"queue": "compilation"},
COMPILATION_RECOVERY_TASK_NAME: {"queue": "compilation"},
},
beat_schedule={
"recover-durable-ingestion-jobs": {
"task": RECOVERY_TASK_NAME,
"schedule": float(settings.ingestion_recovery_interval_seconds),
},
"recover-durable-compilation-jobs": {
"task": COMPILATION_RECOVERY_TASK_NAME,
"schedule": float(settings.compilation_recovery_interval_seconds),
},
},
)
return app
class CeleryIngestionDispatcher:
"""Publish only a durable Job id; source bytes and state stay canonical."""
def __init__(self, app: Celery) -> None:
self._app = app
def enqueue(self, job_id: UUID) -> None:
"""Deliver one idempotent execution hint to the ingestion queue."""
self._app.send_task(
INGESTION_TASK_NAME,
args=[str(job_id)],
queue="ingestion",
)
class CeleryCompilationDispatcher:
"""Publish only a durable Compilation Job ID to the dedicated queue."""
def __init__(self, app: Celery) -> None:
self._app = app
def enqueue(self, job_id: UUID) -> None:
"""Deliver one idempotent generation hint without source content."""
self._app.send_task(
COMPILATION_TASK_NAME,
args=[str(job_id)],
queue="compilation",
)

View File

@ -0,0 +1,492 @@
"""Authenticated REST routes for source-backed Compiled Knowledge drafts."""
from typing import Annotated
from uuid import UUID
from fastapi import APIRouter, Depends, HTTPException, Query, status
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
from ekp_api.providers import ApiServices
from ekp_api.routes import (
_resolved_document_ids,
_resolved_knowledge_base_ids,
get_api_services,
)
from ekp_api.schemas import (
CompilationJobResponse,
CompiledKnowledgeConflictCreateRequest,
CompiledKnowledgeConflictResolveRequest,
CompiledKnowledgeDraftCreateRequest,
CompiledKnowledgeDraftUpdateRequest,
CompiledKnowledgeGenerateRequest,
CompiledKnowledgeRejectRequest,
CompiledKnowledgeResponse,
CompiledKnowledgeTransitionRequest,
)
from ekp_core.compilation_jobs import (
AsyncCompilationSubmissionService,
CompilationJobNotFoundError,
CompilationJobService,
)
from ekp_core.compiled_knowledge import (
CompilationDataPolicyError,
CompilationProviderUnavailableError,
CompilationSourceError,
CompiledKnowledgeConflictError,
CompiledKnowledgeDraftStateError,
CompiledKnowledgeNotFoundError,
CompiledKnowledgeRevisionConflictError,
CompiledKnowledgeService,
CompiledKnowledgeSourceStaleError,
)
from ekp_domain import AclPermission
router = APIRouter(prefix="/compiled-knowledge", tags=["compiled-knowledge"])
Services = Annotated[ApiServices, Depends(get_api_services)]
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
@router.get("", response_model=list[CompiledKnowledgeResponse])
def list_compiled_knowledge(
services: Services,
identity: Identity,
knowledge_base_id: Annotated[UUID | None, Query()] = None,
) -> list[CompiledKnowledgeResponse]:
"""List only drafts whose KB and every source Document remain readable."""
service, organization_id, readable_kbs, readable_documents = _read_context(services, identity)
values = service.list_items(
organization_id,
readable_kbs,
readable_documents,
knowledge_base_id=knowledge_base_id,
)
return [CompiledKnowledgeResponse.from_view(value) for value in values]
@router.get("/jobs/{job_id}", response_model=CompilationJobResponse)
def get_compilation_job(
job_id: UUID,
services: Services,
identity: Identity,
) -> CompilationJobResponse:
"""Poll a Job only while its KB and every source Document remain readable."""
require_scope(identity, "knowledge:read")
organization_id, _ = require_canonical_identity(identity)
jobs = _job_service(services)
try:
value = jobs.get_authorized(
job_id=job_id,
organization_id=organization_id,
allowed_knowledge_base_ids=_resolved_knowledge_base_ids(
identity, services, AclPermission.READ
),
allowed_source_document_ids=_resolved_document_ids(
identity,
services,
AclPermission.READ,
include_deleted=True,
)
or (),
)
except CompilationJobNotFoundError as error:
raise HTTPException(status_code=404, detail="compilation job not found") from error
return CompilationJobResponse.from_view(value)
@router.get("/{compiled_id}", response_model=CompiledKnowledgeResponse)
def get_compiled_knowledge(
compiled_id: UUID,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Read one item through the same source-intersection authorization rule."""
service, organization_id, readable_kbs, readable_documents = _read_context(services, identity)
try:
value = service.get_item(
organization_id,
readable_kbs,
readable_documents,
compiled_id,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
return CompiledKnowledgeResponse.from_view(value)
@router.post(
"/drafts",
response_model=CompiledKnowledgeResponse,
status_code=status.HTTP_201_CREATED,
)
def create_compiled_knowledge_draft(
payload: CompiledKnowledgeDraftCreateRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Create a manual draft in one writable KB from readable source Chunks."""
service, organization_id, actor_user_id, readable_documents = _write_context(
services, identity, payload.knowledge_base_id
)
try:
value = service.create_manual_draft(
organization_id=organization_id,
knowledge_base_id=payload.knowledge_base_id,
actor_user_id=actor_user_id,
title=payload.title,
knowledge_type=payload.knowledge_type,
content=payload.content,
source_chunk_ids=payload.source_chunk_ids,
allowed_source_document_ids=readable_documents,
)
except (CompilationSourceError, ValueError) as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post(
"/drafts:generate",
response_model=CompilationJobResponse,
status_code=status.HTTP_202_ACCEPTED,
)
def generate_compiled_knowledge_draft(
payload: CompiledKnowledgeGenerateRequest,
services: Services,
identity: Identity,
) -> CompilationJobResponse:
"""Validate and enqueue durable AI generation without blocking the API."""
_, organization_id, actor_user_id, readable_documents = _write_context(
services, identity, payload.knowledge_base_id
)
submission = _submission_service(services)
try:
value = submission.submit(
organization_id=organization_id,
knowledge_base_id=payload.knowledge_base_id,
actor_user_id=actor_user_id,
knowledge_type=payload.knowledge_type,
source_chunk_ids=payload.source_chunk_ids,
allowed_source_document_ids=readable_documents,
title_hint=payload.title_hint,
instruction=payload.instruction,
)
except CompilationProviderUnavailableError as error:
raise HTTPException(
status_code=503, detail="compilation provider is unavailable"
) from error
except CompilationDataPolicyError as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except (CompilationSourceError, ValueError) as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompilationJobResponse.from_view(value)
@router.put("/{compiled_id}/draft", response_model=CompiledKnowledgeResponse)
def update_compiled_knowledge_draft(
compiled_id: UUID,
payload: CompiledKnowledgeDraftUpdateRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Edit only a draft visible through the current source permission intersection."""
require_scope(identity, "knowledge:write")
organization_id, actor_user_id = require_canonical_identity(identity)
service = _service(services)
writable_kbs = _resolved_knowledge_base_ids(identity, services, AclPermission.WRITE)
readable_documents = (
_resolved_document_ids(
identity,
services,
AclPermission.READ,
include_deleted=True,
)
or ()
)
try:
value = service.update_draft(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=writable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
expected_revision=payload.expected_revision,
title=payload.title,
knowledge_type=payload.knowledge_type,
content=payload.content,
source_chunk_ids=payload.source_chunk_ids,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except ValueError as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post("/{compiled_id}:submit-review", response_model=CompiledKnowledgeResponse)
def submit_compiled_knowledge_review(
compiled_id: UUID,
payload: CompiledKnowledgeTransitionRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Verify exact source freshness before entering human review."""
service, organization_id, actor_user_id, writable_kbs, readable_documents = _lifecycle_context(
services, identity, AclPermission.WRITE, "knowledge:write"
)
try:
value = service.submit_for_review(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=writable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
expected_revision=payload.expected_revision,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
CompiledKnowledgeSourceStaleError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post("/{compiled_id}/conflicts", response_model=CompiledKnowledgeResponse)
def report_compiled_knowledge_conflict(
compiled_id: UUID,
payload: CompiledKnowledgeConflictCreateRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Record a reviewer-confirmed contradiction with exact source evidence."""
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
)
try:
value = service.report_conflict(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=manageable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
expected_revision=payload.expected_revision,
claim=payload.claim,
left_source_id=payload.left_source_id,
right_source_id=payload.right_source_id,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeConflictError,
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except ValueError as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post(
"/{compiled_id}/conflicts/{conflict_id}:resolve",
response_model=CompiledKnowledgeResponse,
)
def resolve_compiled_knowledge_conflict(
compiled_id: UUID,
conflict_id: UUID,
payload: CompiledKnowledgeConflictResolveRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Close one conflict only with an explicit reviewer explanation."""
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
)
try:
value = service.resolve_conflict(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=manageable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
conflict_id=conflict_id,
expected_revision=payload.expected_revision,
resolution_note=payload.resolution_note,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeConflictError,
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except ValueError as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post("/{compiled_id}:publish", response_model=CompiledKnowledgeResponse)
def publish_compiled_knowledge(
compiled_id: UUID,
payload: CompiledKnowledgeTransitionRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Publish only a verified review revision with no open conflict."""
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
)
try:
value = service.publish(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=manageable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
expected_revision=payload.expected_revision,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeConflictError,
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
CompiledKnowledgeSourceStaleError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
@router.post("/{compiled_id}:reject", response_model=CompiledKnowledgeResponse)
def reject_compiled_knowledge(
compiled_id: UUID,
payload: CompiledKnowledgeRejectRequest,
services: Services,
identity: Identity,
) -> CompiledKnowledgeResponse:
"""Reject one review revision with an editor-facing reason."""
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
)
try:
value = service.reject(
organization_id=organization_id,
actor_user_id=actor_user_id,
allowed_knowledge_base_ids=manageable_kbs,
allowed_source_document_ids=readable_documents,
compiled_id=compiled_id,
expected_revision=payload.expected_revision,
reason=payload.reason,
)
except CompiledKnowledgeNotFoundError as error:
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
except (
CompiledKnowledgeDraftStateError,
CompiledKnowledgeRevisionConflictError,
) as error:
raise HTTPException(status_code=409, detail=str(error)) from error
except ValueError as error:
raise HTTPException(status_code=422, detail=str(error)) from error
return CompiledKnowledgeResponse.from_view(value)
def _service(services: ApiServices) -> CompiledKnowledgeService:
"""Fail closed if a custom test/runtime graph omitted compilation."""
if services.compiled_knowledge is None:
raise HTTPException(status_code=503, detail="compiled knowledge service is unavailable")
return services.compiled_knowledge
def _job_service(services: ApiServices) -> CompilationJobService:
"""Fail closed when a custom runtime omitted durable generation Jobs."""
if services.compilation_jobs is None:
raise HTTPException(status_code=503, detail="compilation job service is unavailable")
return services.compilation_jobs
def _submission_service(services: ApiServices) -> AsyncCompilationSubmissionService:
"""Fail closed when a custom runtime omitted async generation submission."""
if services.compilation_submissions is None:
raise HTTPException(status_code=503, detail="compilation submission is unavailable")
return services.compilation_submissions
def _read_context(
services: ApiServices, identity: ApiIdentity
) -> tuple[CompiledKnowledgeService, UUID, tuple[UUID, ...], tuple[UUID, ...]]:
require_scope(identity, "knowledge:read")
organization_id, _ = require_canonical_identity(identity)
return (
_service(services),
organization_id,
_resolved_knowledge_base_ids(identity, services, AclPermission.READ),
_resolved_document_ids(
identity,
services,
AclPermission.READ,
include_deleted=True,
)
or (),
)
def _write_context(
services: ApiServices,
identity: ApiIdentity,
knowledge_base_id: UUID,
) -> tuple[CompiledKnowledgeService, UUID, UUID, tuple[UUID, ...]]:
require_scope(identity, "knowledge:write")
organization_id, actor_user_id = require_canonical_identity(identity)
if knowledge_base_id not in _resolved_knowledge_base_ids(
identity, services, AclPermission.WRITE
):
raise HTTPException(status_code=404, detail="knowledge base not found")
return (
_service(services),
organization_id,
actor_user_id,
_resolved_document_ids(
identity,
services,
AclPermission.READ,
include_deleted=True,
)
or (),
)
def _lifecycle_context(
services: ApiServices,
identity: ApiIdentity,
permission: AclPermission,
scope: str,
) -> tuple[
CompiledKnowledgeService,
UUID,
UUID,
tuple[UUID, ...],
tuple[UUID, ...],
]:
"""Resolve one lifecycle action before reading or mutating L2 content."""
require_scope(identity, scope)
organization_id, actor_user_id = require_canonical_identity(identity)
return (
_service(services),
organization_id,
actor_user_id,
_resolved_knowledge_base_ids(identity, services, permission),
_resolved_document_ids(
identity,
services,
AclPermission.READ,
include_deleted=True,
)
or (),
)

View File

@ -0,0 +1,113 @@
"""Dev-only API startup with canonical database migration."""
import logging
import os
from collections.abc import Callable
from pathlib import Path
import uvicorn
from alembic import command
from alembic.config import Config
from sqlalchemy import inspect, text
from ekp_core.config import Settings, get_settings, get_settings_profile
from ekp_core.database import create_database_engine
from ekp_core.observability import configure_json_logging
_DEV_PROFILE = "dev-image"
logger = logging.getLogger(__name__)
class DevMigrationError(RuntimeError):
"""Report a safe dev migration failure without connection or SQL details."""
def __init__(self, *, error_type: str) -> None:
super().__init__("dev canonical migration failed")
self.error_type = error_type
def _run_alembic(database_url: str) -> None:
"""Upgrade the canonical schema using an explicit, non-rendered URL."""
# The non-editable image installs this module into site-packages while the
# Docker WORKDIR remains /app, where Alembic config and migrations live.
config = Config(str(Path.cwd() / "alembic.ini"))
config.attributes["database_url"] = database_url
command.upgrade(config, "head")
def migrate_dev_database(
settings: Settings,
*,
migration_runner: Callable[[str], None] = _run_alembic,
) -> None:
"""Upgrade dev canonical tables before serving traffic.
The central dev pipeline can only update one default image and provides no
Migration Job. Production targets retain the Helm hook and never call this
entrypoint. The central dev deployment is constrained to one API replica;
multi-replica environments must use the independent Migration Job instead.
"""
if settings.app_env != "development" or get_settings_profile() != _DEV_PROFILE:
raise DevMigrationError(error_type="invalid_profile")
database_url = settings.database_url.get_secret_value()
try:
migration_runner(database_url)
except Exception as error:
raise DevMigrationError(error_type=type(error).__name__) from error
def _read_canonical_revision(settings: Settings) -> str:
"""Return only a safe revision identifier for dev startup diagnostics."""
engine = create_database_engine(settings)
try:
with engine.connect() as connection:
if not inspect(connection).has_table("alembic_version"):
return "unversioned"
revision = connection.scalar(text("SELECT version_num FROM alembic_version"))
return revision if isinstance(revision, str) and revision else "unversioned"
except Exception:
return "unavailable"
finally:
engine.dispose()
def main() -> int:
"""Migrate the authorized dev database, then start the API process."""
settings = get_settings()
configure_json_logging(settings.log_level)
try:
migrate_dev_database(settings)
except DevMigrationError as error:
os.environ["EKP_DEV_MIGRATION_STATUS"] = "failed"
os.environ["EKP_DEV_MIGRATION_REVISION"] = _read_canonical_revision(settings)
os.environ["EKP_DEV_MIGRATION_ERROR_TYPE"] = error.error_type
logger.error(
"Dev canonical migration failed",
extra={
"event": "dev_canonical_migration_failed",
"dependency": "canonical_database",
},
)
else:
os.environ["EKP_DEV_MIGRATION_STATUS"] = "completed"
os.environ["EKP_DEV_MIGRATION_REVISION"] = _read_canonical_revision(settings)
os.environ.pop("EKP_DEV_MIGRATION_ERROR_TYPE", None)
logger.info(
"Dev canonical migration completed",
extra={
"event": "dev_canonical_migration_completed",
"dependency": "canonical_database",
},
)
uvicorn.run(
"ekp_api.main:app",
host=settings.api_host,
port=settings.api_port,
access_log=False,
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,85 @@
"""Controlled CLI for the ARCH-005 shadow index cutover workflow."""
import argparse
import json
import sys
from collections.abc import Sequence
from dataclasses import asdict
from ekp_api.providers import (
create_embedding_provider,
create_index_profile,
create_object_storage,
create_vector_store,
)
from ekp_core.chunking import StructureAwareChunker
from ekp_core.config import Settings
from ekp_core.database import create_database_engine, create_session_factory
from ekp_core.document_chunking import CanonicalChunkRebuildService
from ekp_core.index_rebuild import IndexBuildSummary, IndexRebuildService
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Build, verify, activate, or roll back a versioned vector index."
)
parser.add_argument(
"command",
choices=("build", "verify", "activate", "rollback"),
help="Lifecycle step to execute for INDEX_PROFILE_ID.",
)
return parser
def _create_service(settings: Settings) -> IndexRebuildService:
"""Compose the command from the same provider settings as the API runtime."""
engine = create_database_engine(settings)
session_factory = create_session_factory(engine)
target_profile = create_index_profile(settings)
return IndexRebuildService(
session_factory=session_factory,
embedding_provider=create_embedding_provider(settings),
vector_store=create_vector_store(settings, target_profile),
target_profile=target_profile,
chunk_rebuilder=CanonicalChunkRebuildService(
session_factory,
create_object_storage(settings),
StructureAwareChunker(
max_tokens=target_profile.chunk_max_tokens,
overlap_tokens=target_profile.chunk_overlap_tokens,
max_characters=target_profile.chunk_max_characters,
strategy_version=target_profile.chunk_strategy_version,
),
),
)
def _execute(service: IndexRebuildService, command: str) -> IndexBuildSummary:
"""Run exactly one explicit lifecycle step."""
if command == "build":
return service.rebuild()
if command == "verify":
return service.verify()
if command == "activate":
return service.activate()
if command == "rollback":
return service.rollback()
raise ValueError("unsupported index rebuild command")
def main(argv: Sequence[str] | None = None) -> int:
"""Return a shell-safe status without printing endpoints, SQL, or credentials."""
args = _parser().parse_args(argv)
try:
summary = _execute(_create_service(Settings()), args.command)
except Exception:
print("index profile command failed; inspect sanitized service logs", file=sys.stderr)
return 1
print(json.dumps(asdict(summary), sort_keys=True, separators=(",", ":")))
# Shell automation must not interpret an incomplete build/verification as
# a successful cutover prerequisite merely because safe JSON was emitted.
return 0 if summary.complete else 2
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,275 @@
"""Controlled end-to-end acceptance for the target provider chain."""
import json
import os
import sys
from collections.abc import Sequence
from dataclasses import asdict, dataclass, replace
from io import BytesIO
from pathlib import Path
from time import perf_counter_ns
from uuid import uuid4
from ekp_api.providers import ApiServices, create_api_services
from ekp_core.config import Settings
from ekp_core.evaluation import (
EvaluationCaseResult,
GoldenDataset,
GoldenDatasetError,
GoldenSource,
RankedSource,
RetrievalMetrics,
evaluate_rankings,
load_golden_dataset,
)
from ekp_core.search import SearchItem, SourceResult
MIN_GOLDEN_QUERIES = 30
MAX_GOLDEN_QUERIES = 500
DEFAULT_MIN_RECALL_AT_1 = 0.90
DEFAULT_MIN_RECALL_AT_5 = 1.0
DEFAULT_MIN_MRR = 0.90
DEFAULT_MIN_NDCG_AT_10 = 0.90
DEFAULT_MAX_P95_MS = 10_000.0
class LiveAcceptanceError(RuntimeError):
"""Raised for a safe precondition or acceptance failure."""
@dataclass(frozen=True, slots=True)
class AcceptanceReport:
"""Provider-neutral quality and latency metrics without retrieved content."""
query_count: int
recall_at_1: float
recall_at_3: float
recall_at_5: float
recall_at_10: float
mrr: float
ndcg_at_10: float
latency_p50_ms: float
latency_p95_ms: float
dataset_id: str
dataset_version: str
index_profile_id: str
passed: bool
@dataclass(frozen=True, slots=True)
class AcceptanceThresholds:
"""Explicit gates for controlled live acceptance."""
min_recall_at_1: float = DEFAULT_MIN_RECALL_AT_1
min_recall_at_5: float = DEFAULT_MIN_RECALL_AT_5
min_mrr: float = DEFAULT_MIN_MRR
min_ndcg_at_10: float = DEFAULT_MIN_NDCG_AT_10
max_p95_ms: float = DEFAULT_MAX_P95_MS
def __post_init__(self) -> None:
quality_values = (
self.min_recall_at_1,
self.min_recall_at_5,
self.min_mrr,
self.min_ndcg_at_10,
)
if any(not 0 <= value <= 1 for value in quality_values):
raise ValueError("live quality thresholds must be between zero and one")
if self.max_p95_ms <= 0:
raise ValueError("live latency threshold must be positive")
def _project_root() -> Path:
return Path(__file__).resolve().parents[4]
def _load_live_dataset(path: Path) -> GoldenDataset:
"""Load the shared manifest so live and offline evaluation cannot drift."""
try:
dataset = load_golden_dataset(path)
except GoldenDatasetError as error:
raise LiveAcceptanceError("live Golden Dataset is invalid") from error
if not MIN_GOLDEN_QUERIES <= len(dataset.cases) <= MAX_GOLDEN_QUERIES:
raise LiveAcceptanceError("live Golden Dataset must contain 30 to 500 queries")
return dataset
def _ranked_sources(results: Sequence[SearchItem]) -> tuple[RankedSource, ...]:
"""Project live canonical results onto the provider-neutral scoring schema."""
if any(not isinstance(result, SourceResult) for result in results):
raise LiveAcceptanceError("Raw provider acceptance received a Compiled result")
return tuple(
RankedSource(
filename=result.filename,
section=result.section,
sheet_name=result.sheet_name,
row_range=result.row_range,
column_range=result.column_range,
)
for result in results
if isinstance(result, SourceResult)
)
def _live_expected_source(source: GoldenSource, live_filename: str) -> GoldenSource:
"""Apply the random run prefix to filename-derived CSV sheet identities."""
sheet_name = source.sheet_name
original_path = Path(source.filename)
if original_path.suffix.casefold() == ".csv" and sheet_name == original_path.stem:
sheet_name = Path(live_filename).stem
return replace(source, filename=live_filename, sheet_name=sheet_name)
def _build_report(
results: Sequence[EvaluationCaseResult],
thresholds: AcceptanceThresholds,
*,
dataset: GoldenDataset,
index_profile_id: str,
) -> AcceptanceReport:
"""Calculate the shared metrics and apply explicit live acceptance gates."""
metrics = evaluate_rankings(tuple(results))
passed = (
metrics.recall_at_1 >= thresholds.min_recall_at_1
and metrics.recall_at_5 >= thresholds.min_recall_at_5
and metrics.mrr >= thresholds.min_mrr
and metrics.ndcg_at_10 >= thresholds.min_ndcg_at_10
and metrics.latency_p95_ms <= thresholds.max_p95_ms
)
return _acceptance_report(metrics, dataset, index_profile_id, passed)
def _acceptance_report(
metrics: RetrievalMetrics,
dataset: GoldenDataset,
index_profile_id: str,
passed: bool,
) -> AcceptanceReport:
"""Attach immutable evaluation identities to the shared metric values."""
return AcceptanceReport(
query_count=metrics.query_count,
recall_at_1=metrics.recall_at_1,
recall_at_3=metrics.recall_at_3,
recall_at_5=metrics.recall_at_5,
recall_at_10=metrics.recall_at_10,
mrr=metrics.mrr,
ndcg_at_10=metrics.ndcg_at_10,
latency_p50_ms=metrics.latency_p50_ms,
latency_p95_ms=metrics.latency_p95_ms,
dataset_id=dataset.dataset_id,
dataset_version=dataset.version,
index_profile_id=index_profile_id,
passed=passed,
)
def _thresholds_from_environment() -> AcceptanceThresholds:
return AcceptanceThresholds(
min_recall_at_1=float(os.getenv("EKP_LIVE_MIN_RECALL_AT_1", str(DEFAULT_MIN_RECALL_AT_1))),
min_recall_at_5=float(os.getenv("EKP_LIVE_MIN_RECALL_AT_5", str(DEFAULT_MIN_RECALL_AT_5))),
min_mrr=float(os.getenv("EKP_LIVE_MIN_MRR", str(DEFAULT_MIN_MRR))),
min_ndcg_at_10=float(os.getenv("EKP_LIVE_MIN_NDCG_AT_10", str(DEFAULT_MIN_NDCG_AT_10))),
max_p95_ms=float(os.getenv("EKP_LIVE_MAX_P95_MS", str(DEFAULT_MAX_P95_MS))),
)
def _require_safe_live_settings(settings: Settings) -> None:
"""Refuse external writes unless rotation and resource authorization are explicit."""
required_flags = (
"EKP_RUN_LIVE_TESTS",
"EKP_CREDENTIALS_AUTHORIZED",
"EKP_LIVE_RESOURCE_ACK",
)
if any(os.getenv(name) != "1" for name in required_flags):
raise LiveAcceptanceError("live acceptance safety flags are not enabled")
if settings.app_env != "test":
raise LiveAcceptanceError("live acceptance requires APP_ENV=test")
if settings.object_storage_provider != "bos" or settings.vector_store_provider != "oceanbase":
raise LiveAcceptanceError("live acceptance requires the target provider baseline")
if not settings.bos_key_prefix.startswith("live-acceptance/"):
raise LiveAcceptanceError("BOS_KEY_PREFIX must use a dedicated live-acceptance prefix")
def _cleanup_run_documents(services: ApiServices, filename_prefix: str) -> None:
"""Delete only documents created under this random acceptance-run prefix."""
documents = tuple(
document
for document in services.documents.list_documents()
if document.filename.startswith(filename_prefix)
)
for document in documents:
services.documents.delete_document(document.id)
def _run_live(settings: Settings) -> AcceptanceReport:
"""Upload synthetic sources, execute every query, and clean run-owned data."""
_require_safe_live_settings(settings)
dataset = _load_live_dataset(_project_root() / "fixtures/evaluation/manifest.json")
services = create_api_services(settings)
filename_prefix = f"live-{uuid4().hex[:12]}-"
uploaded_names: dict[str, str] = {}
cleanup_error: Exception | None = None
try:
for corpus_file in dataset.corpus:
live_name = f"{filename_prefix}{corpus_file.filename}"
services.ingestion.ingest(
live_name,
corpus_file.media_type,
BytesIO(corpus_file.path.read_bytes()),
)
uploaded_names[corpus_file.filename] = live_name
case_results: list[EvaluationCaseResult] = []
index_profile_id: str | None = None
for case in dataset.cases:
started = perf_counter_ns()
result = services.search.search(case.query, top_k=10)
latency_ms = (perf_counter_ns() - started) / 1_000_000
if index_profile_id is None:
index_profile_id = result.index_profile_id
elif result.index_profile_id != index_profile_id:
raise LiveAcceptanceError("live search crossed the selected index profile")
live_case = replace(
case,
expected_sources=tuple(
_live_expected_source(source, uploaded_names[source.filename])
for source in case.expected_sources
),
)
case_results.append(
EvaluationCaseResult(
case=live_case,
ranked_sources=_ranked_sources(result.results),
latency_ms=latency_ms,
)
)
if index_profile_id is None:
raise LiveAcceptanceError("live evaluation returned no index profile")
return _build_report(
case_results,
_thresholds_from_environment(),
dataset=dataset,
index_profile_id=index_profile_id,
)
finally:
try:
_cleanup_run_documents(services, filename_prefix)
except Exception as error:
cleanup_error = error
if cleanup_error is not None:
raise LiveAcceptanceError("live acceptance cleanup failed") from cleanup_error
def main() -> int:
"""Print only aggregate metrics and return non-zero for unsafe or failed runs."""
try:
report = _run_live(Settings())
except Exception:
print("live acceptance failed; no provider details were emitted", file=sys.stderr)
return 1
print(json.dumps(asdict(report), sort_keys=True, separators=(",", ":")))
return 0 if report.passed else 2
if __name__ == "__main__":
raise SystemExit(main())

View File

@ -0,0 +1,156 @@
"""FastAPI application entry point."""
import logging
import os
import re
from collections.abc import Awaitable, Callable
from pathlib import Path
from time import monotonic
from uuid import uuid4
from fastapi import FastAPI, Request, Response
from fastapi.responses import FileResponse, PlainTextResponse, RedirectResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from ekp_api.admin_routes import router as administration_router
from ekp_api.compiled_routes import router as compiled_knowledge_router
from ekp_api.metrics import HttpMetrics
from ekp_api.providers import ApiServices
from ekp_api.routes import router
from ekp_core.config import Settings, get_settings
from ekp_core.observability import bind_request_id, configure_json_logging, reset_request_id
ADMIN_WEB_ROOT = Path(__file__).resolve().parents[3] / "admin-web"
REQUEST_ID_PATTERN = re.compile(r"^[A-Za-z0-9._:-]{1,64}$")
logger = logging.getLogger(__name__)
class HealthResponse(BaseModel):
"""Health endpoint response."""
status: str
application: str
environment: str
migration_status: str | None = None
migration_revision: str | None = None
migration_error_type: str | None = None
def create_app(
settings: Settings | None = None,
api_services: ApiServices | None = None,
) -> FastAPI:
"""Create the API application with explicit, testable settings."""
resolved_settings = settings or get_settings()
configure_json_logging(resolved_settings.log_level)
application = FastAPI(
title=resolved_settings.app_name,
version="0.1.0",
docs_url=f"{resolved_settings.api_prefix}/docs",
openapi_url=f"{resolved_settings.api_prefix}/openapi.json",
)
application.state.settings = resolved_settings
application.state.api_services = api_services
application.state.http_metrics = HttpMetrics()
application.include_router(router, prefix=resolved_settings.api_prefix)
application.include_router(administration_router, prefix=resolved_settings.api_prefix)
application.include_router(compiled_knowledge_router, prefix=resolved_settings.api_prefix)
admin_web_root = (resolved_settings.admin_web_root or ADMIN_WEB_ROOT).resolve()
# The Demo console is a deliberately thin, same-origin client. It contains
# no business logic and calls only the versioned public REST API.
application.mount(
"/admin/assets",
StaticFiles(directory=admin_web_root / "assets"),
name="admin-assets",
)
@application.get("/", include_in_schema=False)
async def admin_console_entry() -> RedirectResponse:
"""Send the public web entry to the canonical Admin Console route."""
return RedirectResponse(url="/admin", status_code=307)
@application.middleware("http")
async def observe_http(
request: Request,
call_next: Callable[[Request], Awaitable[Response]],
) -> Response:
"""Correlate requests and emit bounded metrics without body/query data."""
incoming_request_id = request.headers.get("x-request-id", "")
request_id = (
incoming_request_id
if REQUEST_ID_PATTERN.fullmatch(incoming_request_id)
else uuid4().hex
)
token = bind_request_id(request_id)
started_at = monotonic()
status_code = 500
try:
response = await call_next(request)
status_code = response.status_code
response.headers["x-request-id"] = request_id
return response
finally:
duration_seconds = monotonic() - started_at
route = request.scope.get("route")
route_path = getattr(route, "path", "unmatched")
application.state.http_metrics.observe(
request.method,
route_path,
status_code,
duration_seconds,
)
logger.info(
"HTTP request completed",
extra={
"event": "http_request_completed",
"method": request.method,
"path": route_path,
"status_code": status_code,
"duration_ms": round(duration_seconds * 1000, 3),
},
)
reset_request_id(token)
@application.get("/admin", include_in_schema=False)
@application.get("/admin/overview", include_in_schema=False)
@application.get("/admin/upload", include_in_schema=False)
@application.get("/admin/documents", include_in_schema=False)
@application.get("/admin/compiled-knowledge", include_in_schema=False)
@application.get("/admin/search", include_in_schema=False)
@application.get("/admin/knowledge-bases", include_in_schema=False)
@application.get("/admin/permissions", include_in_schema=False)
async def admin_console() -> FileResponse:
"""Serve the API-first Admin Console shell for every core client route."""
return FileResponse(admin_web_root / "index.html")
@application.get(
"/healthz",
response_model=HealthResponse,
response_model_exclude_none=True,
tags=["system"],
)
async def health() -> HealthResponse:
migration_status = os.getenv("EKP_DEV_MIGRATION_STATUS")
return HealthResponse(
status="degraded" if migration_status == "failed" else "ok",
application=resolved_settings.app_name,
environment=resolved_settings.app_env,
migration_status=migration_status,
migration_revision=os.getenv("EKP_DEV_MIGRATION_REVISION"),
migration_error_type=os.getenv("EKP_DEV_MIGRATION_ERROR_TYPE"),
)
@application.get("/metrics", include_in_schema=False)
async def metrics() -> PlainTextResponse:
"""Expose aggregate process/HTTP metrics without provider or user data."""
return PlainTextResponse(
application.state.http_metrics.render(),
media_type="text/plain; version=0.0.4",
)
return application
app = create_app()

View File

@ -0,0 +1,59 @@
"""Small dependency-free Prometheus endpoint for cluster smoke monitoring."""
from collections import defaultdict
from threading import Lock
from time import monotonic
class HttpMetrics:
"""Keep bounded per-process HTTP counters using route templates as labels."""
def __init__(self) -> None:
self._started_at = monotonic()
self._lock = Lock()
self._counts: defaultdict[tuple[str, str, int], int] = defaultdict(int)
self._durations: defaultdict[tuple[str, str], tuple[int, float]] = defaultdict(
lambda: (0, 0.0)
)
def observe(self, method: str, path: str, status_code: int, duration_seconds: float) -> None:
"""Record one request without retaining query strings or user content."""
with self._lock:
self._counts[(method, path, status_code)] += 1
count, total = self._durations[(method, path)]
self._durations[(method, path)] = (count + 1, total + duration_seconds)
def render(self) -> str:
"""Return Prometheus text exposition with stable low-cardinality labels."""
with self._lock:
counts = tuple(sorted(self._counts.items()))
durations = tuple(sorted(self._durations.items()))
lines = [
"# HELP ekp_process_uptime_seconds Process uptime in seconds.",
"# TYPE ekp_process_uptime_seconds gauge",
f"ekp_process_uptime_seconds {monotonic() - self._started_at:.6f}",
"# HELP ekp_http_requests_total HTTP requests handled by this API process.",
"# TYPE ekp_http_requests_total counter",
]
for (method, path, status_code), value in counts:
labels = _labels(method=method, path=path, status=str(status_code))
lines.append(f"ekp_http_requests_total{{{labels}}} {value}")
lines.extend(
(
"# HELP ekp_http_request_duration_seconds HTTP request duration.",
"# TYPE ekp_http_request_duration_seconds summary",
)
)
for (method, path), (count, total) in durations:
labels = _labels(method=method, path=path)
lines.append(f"ekp_http_request_duration_seconds_count{{{labels}}} {count}")
lines.append(f"ekp_http_request_duration_seconds_sum{{{labels}}} {total:.6f}")
return "\n".join(lines) + "\n"
def _labels(**values: str) -> str:
"""Escape the limited label set according to Prometheus text rules."""
return ",".join(
f'{name}="{value.replace(chr(92), chr(92) * 2).replace(chr(10), chr(92) + "n").replace(chr(34), chr(92) + chr(34))}"'
for name, value in values.items()
)

View File

@ -0,0 +1,250 @@
"""Deterministic, provider-free RAG evaluation for the versioned Golden Dataset."""
import argparse
import json
from collections.abc import Sequence
from dataclasses import asdict, dataclass
from io import BytesIO
from pathlib import Path
from tempfile import TemporaryDirectory
from time import perf_counter_ns
from uuid import UUID
from sqlalchemy import create_engine, select
from ekp_core.chunking import StructureAwareChunker
from ekp_core.database import create_session_factory
from ekp_core.dense_retrieval import DenseRetrievalService
from ekp_core.document_chunking import DocumentChunkingService, DocumentPipelineService
from ekp_core.document_indexing import DocumentIndexingPipelineService, DocumentIndexingService
from ekp_core.document_parsing import DocumentParsingService
from ekp_core.document_upload import (
DOCLING_DOCUMENT_EXTENSIONS,
STRUCTURED_DOCUMENT_EXTENSIONS,
DocumentUploadService,
)
from ekp_core.evaluation import (
EvaluationCaseResult,
GoldenDataset,
GoldenKeywordEmbeddingProvider,
RankedSource,
RetrievalMetrics,
evaluate_rankings,
load_golden_dataset,
)
from ekp_core.hybrid_retrieval import HybridRetrievalService
from ekp_core.search import SearchService, SourceResult
from ekp_core.sparse_retrieval import SparseRetrievalService
from ekp_domain import Base, Chunk, IndexProfile
from ekp_object_storage import LocalObjectStorage
from ekp_parser import DoclingParser, ParserRouter
from ekp_reranker import FakeRerankerProvider
from ekp_vector_store import FakeVectorStore
DEFAULT_MANIFEST_PATH = Path("fixtures/evaluation/manifest.json")
OFFLINE_PROFILE_ID = "golden-structure-v2-offline-v1"
@dataclass(frozen=True, slots=True)
class EvaluationStages:
"""Required metrics for every decision point in the retrieval pipeline."""
dense: RetrievalMetrics
sparse: RetrievalMetrics
hybrid: RetrievalMetrics
reranked: RetrievalMetrics
@dataclass(frozen=True, slots=True)
class OfflineEvaluationReport:
"""Aggregate-only report safe to persist in CI logs."""
dataset_id: str
dataset_version: str
index_profile_id: str
query_count: int
stages: EvaluationStages
def run_offline_evaluation(manifest_path: Path = DEFAULT_MANIFEST_PATH) -> OfflineEvaluationReport:
"""Ingest the hashed fixture corpus and evaluate every retrieval stage."""
dataset = load_golden_dataset(manifest_path)
with TemporaryDirectory(prefix="ekp-evaluation-") as temporary_directory:
return _run_with_storage(dataset, Path(temporary_directory))
def _run_with_storage(dataset: GoldenDataset, storage_root: Path) -> OfflineEvaluationReport:
"""Build one isolated canonical/index state for a complete evaluation run."""
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
session_factory = create_session_factory(engine)
storage = LocalObjectStorage(storage_root)
docling = DoclingParser()
parser = ParserRouter({extension: docling for extension in DOCLING_DOCUMENT_EXTENSIONS})
upload = DocumentUploadService(
session_factory,
storage,
max_upload_bytes=50 * 1024 * 1024,
allowed_extensions=STRUCTURED_DOCUMENT_EXTENSIONS,
)
embedding = GoldenKeywordEmbeddingProvider()
vector_store = FakeVectorStore()
profile = IndexProfile(
OFFLINE_PROFILE_ID,
"structure_v2",
embedding.model_name,
embedding.dimensions,
hybrid_top_n=10,
reranker_model="golden-reranker-v1",
final_top_k=10,
)
pipeline = DocumentIndexingPipelineService(
DocumentPipelineService(
upload,
DocumentParsingService(session_factory, storage, parser),
DocumentChunkingService(session_factory, storage, StructureAwareChunker()),
),
DocumentIndexingService(session_factory, embedding, vector_store, profile),
)
for corpus_file in dataset.corpus:
pipeline.ingest(
corpus_file.filename,
corpus_file.media_type,
BytesIO(corpus_file.path.read_bytes()),
)
dense = DenseRetrievalService(embedding, vector_store, profile)
sparse = SparseRetrievalService(vector_store, profile)
hybrid = HybridRetrievalService(
dense,
sparse,
index_profile_id=profile.profile_id,
default_top_n=10,
)
search = SearchService(
session_factory,
hybrid,
FakeRerankerProvider(model_name="golden-reranker-v1"),
profile,
)
with session_factory() as session:
chunks = {chunk.chunk_id: chunk for chunk in session.scalars(select(Chunk)).all()}
dense_results: list[EvaluationCaseResult] = []
sparse_results: list[EvaluationCaseResult] = []
hybrid_results: list[EvaluationCaseResult] = []
reranked_results: list[EvaluationCaseResult] = []
for case in dataset.cases:
started = perf_counter_ns()
dense_result = dense.search(case.query, top_n=10)
dense_results.append(
EvaluationCaseResult(
case,
_ranked_sources(
tuple(candidate.chunk_id for candidate in dense_result.candidates),
chunks,
),
_elapsed_ms(started),
)
)
started = perf_counter_ns()
sparse_result = sparse.search(case.query, top_n=10)
sparse_results.append(
EvaluationCaseResult(
case,
_ranked_sources(
tuple(candidate.chunk_id for candidate in sparse_result.candidates),
chunks,
),
_elapsed_ms(started),
)
)
started = perf_counter_ns()
hybrid_result = hybrid.search(case.query, top_n=10)
hybrid_results.append(
EvaluationCaseResult(
case,
_ranked_sources(
tuple(candidate.chunk_id for candidate in hybrid_result.candidates),
chunks,
),
_elapsed_ms(started),
)
)
started = perf_counter_ns()
search_result = search.search(case.query, top_k=10)
if any(not isinstance(result, SourceResult) for result in search_result.results):
raise RuntimeError("offline Raw evaluation received a Compiled result")
reranked_results.append(
EvaluationCaseResult(
case,
tuple(
RankedSource(
filename=result.filename,
section=result.section,
sheet_name=result.sheet_name,
row_range=result.row_range,
column_range=result.column_range,
)
for result in search_result.results
if isinstance(result, SourceResult)
),
_elapsed_ms(started),
)
)
return OfflineEvaluationReport(
dataset_id=dataset.dataset_id,
dataset_version=dataset.version,
index_profile_id=profile.profile_id,
query_count=len(dataset.cases),
stages=EvaluationStages(
dense=evaluate_rankings(tuple(dense_results)),
sparse=evaluate_rankings(tuple(sparse_results)),
hybrid=evaluate_rankings(tuple(hybrid_results)),
reranked=evaluate_rankings(tuple(reranked_results)),
),
)
def _ranked_sources(
chunk_ids: tuple[UUID, ...],
chunks: dict[UUID, Chunk],
) -> tuple[RankedSource, ...]:
"""Enrich rebuildable index candidates with canonical location metadata."""
return tuple(
RankedSource(
filename=chunks[chunk_id].filename,
section=chunks[chunk_id].section,
sheet_name=chunks[chunk_id].sheet_name,
row_range=chunks[chunk_id].row_range,
column_range=chunks[chunk_id].column_range,
)
for chunk_id in chunk_ids
)
def _elapsed_ms(started_ns: int) -> float:
"""Convert monotonic elapsed time to the report's millisecond unit."""
return (perf_counter_ns() - started_ns) / 1_000_000
def main(argv: Sequence[str] | None = None) -> None:
"""Write a stable aggregate JSON report for local and CI evaluation runs."""
parser = argparse.ArgumentParser(description="Run the offline EKP Golden Dataset")
parser.add_argument(
"--manifest",
type=Path,
default=DEFAULT_MANIFEST_PATH,
help="path to the versioned Golden Dataset manifest",
)
arguments = parser.parse_args(argv)
report = run_offline_evaluation(arguments.manifest)
print(json.dumps(asdict(report), ensure_ascii=False, indent=2, sort_keys=True))
if __name__ == "__main__":
main()

View File

@ -0,0 +1,624 @@
"""Runtime provider and application-service composition from safe settings."""
from dataclasses import dataclass
from sqlalchemy.orm import Session, sessionmaker
from ekp_api.celery_queue import (
CeleryCompilationDispatcher,
CeleryIngestionDispatcher,
create_celery_app,
)
from ekp_core.administration import AdministrationService
from ekp_core.archive_ingestion import ZipIngestionService
from ekp_core.chunking import DEFAULT_CHUNK_STRATEGY_VERSION, StructureAwareChunker
from ekp_core.compilation_jobs import (
AsyncCompilationSubmissionService,
CompilationJobRunner,
CompilationJobService,
)
from ekp_core.compiled_knowledge import CompiledKnowledgeService
from ekp_core.config import Settings
from ekp_core.database import create_database_engine, create_session_factory
from ekp_core.dense_retrieval import DenseRetrievalService
from ekp_core.document_chunking import DocumentChunkingService, DocumentPipelineService
from ekp_core.document_indexing import (
DocumentIndexingPipelineService,
DocumentIndexingService,
)
from ekp_core.document_management import DocumentManagementService
from ekp_core.document_parsing import DocumentParsingService
from ekp_core.document_upload import (
DOCLING_DOCUMENT_EXTENSIONS,
MINERU_DOCUMENT_EXTENSIONS,
DocumentUploadService,
)
from ekp_core.hybrid_retrieval import HybridRetrievalService
from ekp_core.index_rebuild import resolve_active_index_profile
from ekp_core.ingestion_batches import IngestionBatchService
from ekp_core.ingestion_jobs import (
AsyncIngestionSubmissionService,
DerivedDocumentCleanupService,
IngestionJobRunner,
IngestionJobService,
)
from ekp_core.knowledge_access import AuthorizedKnowledgeAccessService
from ekp_core.permission_resolver import PermissionResolver
from ekp_core.search import AuthorizedSearchService, SearchService
from ekp_core.sparse_retrieval import SparseRetrievalService
from ekp_domain import IndexProfile
from ekp_domain.index_profile import DEFAULT_SPARSE_STRATEGY
from ekp_embedding import (
BGE_LARGE_ZH_DIMENSIONS,
EmbeddingProvider,
QianfanEmbeddingProvider,
)
from ekp_llm import LLMProvider, OpenAICompatibleLLMProvider
from ekp_object_storage import ObjectStorage, create_bos_object_storage
from ekp_object_storage.local import LocalObjectStorage
from ekp_parser import MINERU_EXTENSIONS, DoclingParser, MinerUParser, Parser, ParserRouter
from ekp_reranker import QianfanRerankerProvider, RerankerProvider
from ekp_vector_store import (
RetrievalStore,
SparseStore,
VectorStore,
create_oceanbase_vector_store,
validate_retrieval_store,
)
DEMO_CHUNK_STRATEGY_VERSION = DEFAULT_CHUNK_STRATEGY_VERSION
class ProviderConfigurationError(ValueError):
"""Raised when a runtime provider lacks required injected configuration."""
@dataclass(frozen=True, slots=True)
class ApiServices:
"""Business services injected into REST routes without provider leakage."""
ingestion: DocumentIndexingPipelineService
submissions: AsyncIngestionSubmissionService
batches: IngestionBatchService
jobs: IngestionJobService
documents: DocumentManagementService
search: SearchService
authorized_search: AuthorizedSearchService | None = None
knowledge_access: AuthorizedKnowledgeAccessService | None = None
permission_resolver: PermissionResolver | None = None
administration: AdministrationService | None = None
compiled_knowledge: CompiledKnowledgeService | None = None
compilation_jobs: CompilationJobService | None = None
compilation_submissions: AsyncCompilationSubmissionService | None = None
archives: ZipIngestionService | None = None
@dataclass(frozen=True, slots=True)
class RuntimeSearchServices:
"""Shared public and ACL-aware search graph for independent transports."""
search: SearchService
authorized_search: AuthorizedSearchService
knowledge_access: AuthorizedKnowledgeAccessService
@dataclass(frozen=True, slots=True)
class WorkerServices:
"""Provider graph required by Celery without REST search-only services."""
runner: IngestionJobRunner
jobs: IngestionJobService
dispatcher: CeleryIngestionDispatcher
compilation_runner: CompilationJobRunner
compilation_jobs: CompilationJobService
compilation_dispatcher: CeleryCompilationDispatcher
@dataclass(frozen=True, slots=True)
class _IngestionComponents:
"""Shared composition result used by API and independent worker processes."""
session_factory: sessionmaker[Session]
storage: ObjectStorage
vector_store: RetrievalStore
embedding_provider: EmbeddingProvider
index_profile: IndexProfile
upload: DocumentUploadService
parsing: DocumentParsingService
chunking: DocumentChunkingService
indexing: DocumentIndexingService
jobs: IngestionJobService
dispatcher: CeleryIngestionDispatcher
compilation_dispatcher: CeleryCompilationDispatcher
def create_index_profile(
settings: Settings,
session_factory: sessionmaker[Session] | None = None,
) -> IndexProfile:
"""Resolve the canonical active profile or the fresh-deployment baseline."""
configured = IndexProfile(
profile_id=settings.index_profile_id,
chunk_strategy_version=DEMO_CHUNK_STRATEGY_VERSION,
embedding_model=settings.embedding_model,
vector_dimension=BGE_LARGE_ZH_DIMENSIONS,
chunk_max_tokens=settings.chunk_max_tokens,
chunk_overlap_tokens=settings.chunk_overlap_tokens,
chunk_max_characters=settings.chunk_max_characters,
dense_top_n=settings.dense_top_n,
sparse_top_n=settings.sparse_top_n,
hybrid_top_n=settings.hybrid_top_n,
hybrid_rrf_k=settings.hybrid_rrf_k,
hybrid_dense_weight=settings.hybrid_dense_weight,
hybrid_sparse_weight=settings.hybrid_sparse_weight,
reranker_model=settings.reranker_model,
final_top_k=settings.search_top_k,
)
if session_factory is None:
return configured
return resolve_active_index_profile(session_factory, configured)
def create_embedding_provider(settings: Settings) -> QianfanEmbeddingProvider:
"""Create Qianfan only when its API key is supplied by the runtime."""
if settings.baidu_qianfan_api_key is None:
raise ProviderConfigurationError("BAIDU_QIANFAN_API_KEY is required")
return QianfanEmbeddingProvider(
api_key=settings.baidu_qianfan_api_key.get_secret_value(),
endpoint=settings.embedding_endpoint,
timeout_seconds=settings.embedding_timeout_seconds,
)
def create_object_storage(settings: Settings) -> ObjectStorage:
"""Create local or BOS storage while keeping credentials out of core services."""
if settings.object_storage_provider == "local":
return LocalObjectStorage(settings.object_storage_path)
if settings.bos_endpoint is None:
raise ProviderConfigurationError("BOS_ENDPOINT is required")
if settings.bos_access_key_id is None:
raise ProviderConfigurationError("BOS_ACCESS_KEY_ID is required")
if settings.bos_secret_access_key is None:
raise ProviderConfigurationError("BOS_SECRET_ACCESS_KEY is required")
if settings.bos_bucket is None:
raise ProviderConfigurationError("BOS_BUCKET is required")
return create_bos_object_storage(
endpoint=settings.bos_endpoint,
access_key_id=settings.bos_access_key_id.get_secret_value(),
secret_access_key=settings.bos_secret_access_key.get_secret_value(),
bucket_name=settings.bos_bucket,
key_prefix=settings.bos_key_prefix,
timeout_seconds=settings.bos_timeout_seconds,
)
def create_mineru_parser(settings: Settings) -> MinerUParser:
"""Build the optional MinerU adapter from validated runtime settings."""
if settings.mineru_endpoint is None:
raise ProviderConfigurationError("MINERU_ENDPOINT is required")
api_token = (
settings.mineru_api_token.get_secret_value()
if settings.mineru_api_token is not None
else None
)
return MinerUParser(
endpoint=str(settings.mineru_endpoint),
backend=settings.mineru_backend,
parse_method=settings.mineru_parse_method,
language=settings.mineru_language,
effort=settings.mineru_effort,
api_token=api_token,
timeout_seconds=settings.mineru_timeout_seconds,
max_source_bytes=settings.max_upload_bytes,
low_ocr_confidence_threshold=settings.mineru_low_ocr_confidence_threshold,
)
def create_docling_parser(settings: Settings) -> DoclingParser:
"""Build the bounded local Docling adapter used by structured v2 ingestion."""
return DoclingParser(max_source_bytes=settings.max_upload_bytes)
def create_parser_router(
settings: Settings,
mineru_parser: Parser | None = None,
docling_parser: Parser | None = None,
*,
enable_structured: bool = True,
) -> ParserRouter:
"""Route OCR and structured formats while retaining lightweight text parsing."""
routes: dict[str, Parser] = {}
if enable_structured:
selected_docling = docling_parser or create_docling_parser(settings)
routes.update({extension: selected_docling for extension in DOCLING_DOCUMENT_EXTENSIONS})
if settings.parser_provider == "mineru":
selected_mineru = mineru_parser or create_mineru_parser(settings)
routes.update(
{
extension: selected_mineru
for extension in MINERU_EXTENSIONS
if extension in MINERU_DOCUMENT_EXTENSIONS
}
)
return ParserRouter(routes)
def create_vector_store(
settings: Settings,
index_profile: IndexProfile | None = None,
) -> RetrievalStore:
"""Create the OceanBase adapter from explicit, secret-safe settings."""
if settings.oceanbase_vector_uri is None:
raise ProviderConfigurationError("OCEANBASE_VECTOR_URI is required")
if settings.oceanbase_vector_user is None:
raise ProviderConfigurationError("OCEANBASE_VECTOR_USER is required")
if settings.oceanbase_vector_password is None:
raise ProviderConfigurationError("OCEANBASE_VECTOR_PASSWORD is required")
if settings.oceanbase_vector_database is None:
raise ProviderConfigurationError("OCEANBASE_VECTOR_DATABASE is required")
return validate_retrieval_store(
create_oceanbase_vector_store(
uri=settings.oceanbase_vector_uri,
user=settings.oceanbase_vector_user,
password=settings.oceanbase_vector_password.get_secret_value(),
database_name=settings.oceanbase_vector_database,
table_name=settings.oceanbase_vector_table,
index_name=settings.oceanbase_vector_index,
vector_dimension=(
BGE_LARGE_ZH_DIMENSIONS if index_profile is None else index_profile.vector_dimension
),
batch_size=settings.oceanbase_vector_batch_size,
timeout_seconds=settings.oceanbase_vector_timeout_seconds,
)
)
def create_dense_retrieval_service(
embedding_provider: EmbeddingProvider,
vector_store: VectorStore,
index_profile: IndexProfile,
) -> DenseRetrievalService:
"""Compose the profile-scoped dense branch used by hybrid retrieval."""
return DenseRetrievalService(
embedding_provider=embedding_provider,
vector_store=vector_store,
index_profile=index_profile,
default_top_n=index_profile.dense_top_n,
)
def create_sparse_retrieval_service(
sparse_store: SparseStore,
index_profile: IndexProfile,
) -> SparseRetrievalService:
"""Compose the profile-scoped sparse branch used by hybrid retrieval."""
if index_profile.sparse_strategy != DEFAULT_SPARSE_STRATEGY:
raise ProviderConfigurationError("active index profile uses an unsupported sparse strategy")
return SparseRetrievalService(
sparse_store=sparse_store,
index_profile=index_profile,
default_top_n=index_profile.sparse_top_n,
)
def create_hybrid_retrieval_service(
dense_retriever: DenseRetrievalService,
sparse_retriever: SparseRetrievalService,
index_profile: IndexProfile,
) -> HybridRetrievalService:
"""Compose weighted rank fusion without leaking provider-native scores."""
return HybridRetrievalService(
dense_retriever,
sparse_retriever,
index_profile_id=index_profile.profile_id,
default_top_n=index_profile.hybrid_top_n,
rrf_k=index_profile.hybrid_rrf_k,
dense_weight=index_profile.hybrid_dense_weight,
sparse_weight=index_profile.hybrid_sparse_weight,
)
def create_reranker_provider(
settings: Settings,
index_profile: IndexProfile,
) -> QianfanRerankerProvider:
"""Create the Demo BCE reranker using an explicit or shared Qianfan key."""
secret = settings.reranker_api_key or settings.baidu_qianfan_api_key
if secret is None:
raise ProviderConfigurationError("RERANKER_API_KEY or BAIDU_QIANFAN_API_KEY is required")
return QianfanRerankerProvider(
api_key=secret.get_secret_value(),
endpoint=settings.reranker_endpoint,
model_name=index_profile.reranker_model,
timeout_seconds=settings.reranker_timeout_seconds,
)
def create_llm_provider(settings: Settings) -> LLMProvider | None:
"""Create optional knowledge-compilation generation without blocking manual drafts."""
if settings.llm_provider == "disabled":
return None
if settings.llm_endpoint is None or settings.llm_api_key is None or settings.llm_model is None:
raise ProviderConfigurationError("LLM provider configuration is incomplete")
try:
return OpenAICompatibleLLMProvider(
endpoint=settings.llm_endpoint,
api_key=settings.llm_api_key.get_secret_value(),
model_name=settings.llm_model,
processing_location=settings.llm_processing_location,
timeout_seconds=settings.llm_timeout_seconds,
)
except ValueError as error:
raise ProviderConfigurationError("LLM provider configuration is invalid") from error
def create_search_service(
session_factory: sessionmaker[Session],
hybrid_retriever: HybridRetrievalService,
reranker_provider: RerankerProvider,
index_profile: IndexProfile,
) -> SearchService:
"""Compose reranked hybrid search independently from REST schemas."""
return SearchService(
session_factory=session_factory,
hybrid_retriever=hybrid_retriever,
reranker_provider=reranker_provider,
index_profile=index_profile,
)
def create_runtime_search_services(settings: Settings) -> RuntimeSearchServices:
"""Compose one read graph with both compatibility and ACL-aware boundaries."""
engine = create_database_engine(settings)
session_factory = create_session_factory(engine)
index_profile = create_index_profile(settings, session_factory)
embedding_provider = create_embedding_provider(settings)
vector_store = create_vector_store(settings, index_profile)
dense_retrieval = create_dense_retrieval_service(
embedding_provider,
vector_store,
index_profile,
)
sparse_retrieval = create_sparse_retrieval_service(
vector_store,
index_profile,
)
hybrid_retrieval = create_hybrid_retrieval_service(
dense_retrieval,
sparse_retrieval,
index_profile,
)
search = create_search_service(
session_factory,
hybrid_retrieval,
create_reranker_provider(settings, index_profile),
index_profile,
)
permission_resolver = PermissionResolver(session_factory)
return RuntimeSearchServices(
search=search,
authorized_search=AuthorizedSearchService(
permission_resolver,
search,
),
knowledge_access=AuthorizedKnowledgeAccessService(
session_factory,
permission_resolver,
active_index_profile_id=index_profile.profile_id,
),
)
def create_runtime_search_service(settings: Settings) -> SearchService:
"""Preserve the public SearchService factory for existing callers."""
return create_runtime_search_services(settings).search
def _create_ingestion_components(settings: Settings) -> _IngestionComponents:
"""Compose provider-neutral ingestion stages and the Celery delivery adapter."""
if settings.rabbitmq_url is None or not settings.rabbitmq_url.get_secret_value().strip():
raise ProviderConfigurationError("RABBITMQ_URL is required")
engine = create_database_engine(settings)
session_factory = create_session_factory(engine)
storage = create_object_storage(settings)
index_profile = create_index_profile(settings, session_factory)
embedding_provider = create_embedding_provider(settings)
vector_store = create_vector_store(settings, index_profile)
structured_chunks_enabled = (
index_profile.chunk_strategy_version == DEFAULT_CHUNK_STRATEGY_VERSION
)
parser_router = create_parser_router(
settings,
enable_structured=structured_chunks_enabled,
)
upload_service = DocumentUploadService(
session_factory,
storage,
max_upload_bytes=settings.max_upload_bytes,
# Derive admission from the exact runtime router so upload and Worker
# capabilities cannot drift as providers are added or replaced.
allowed_extensions=parser_router.supported_extensions,
)
parsing_service = DocumentParsingService(
session_factory,
storage,
parser_router,
)
chunking_service = DocumentChunkingService(
session_factory,
storage,
StructureAwareChunker(
max_tokens=index_profile.chunk_max_tokens,
overlap_tokens=index_profile.chunk_overlap_tokens,
max_characters=index_profile.chunk_max_characters,
strategy_version=index_profile.chunk_strategy_version,
),
)
indexing_service = DocumentIndexingService(
session_factory,
embedding_provider,
vector_store,
index_profile,
)
jobs = IngestionJobService(session_factory, settings.ingestion_max_attempts)
celery_app = create_celery_app(settings)
dispatcher = CeleryIngestionDispatcher(celery_app)
compilation_dispatcher = CeleryCompilationDispatcher(celery_app)
return _IngestionComponents(
session_factory=session_factory,
storage=storage,
vector_store=vector_store,
embedding_provider=embedding_provider,
index_profile=index_profile,
upload=upload_service,
parsing=parsing_service,
chunking=chunking_service,
indexing=indexing_service,
jobs=jobs,
dispatcher=dispatcher,
compilation_dispatcher=compilation_dispatcher,
)
def _create_runner(settings: Settings, components: _IngestionComponents) -> IngestionJobRunner:
"""Compose retry cleanup and execution policy from non-sensitive settings."""
cleanup = DerivedDocumentCleanupService(
components.session_factory,
components.storage,
components.vector_store,
components.index_profile.profile_id,
)
return IngestionJobRunner(
components.session_factory,
components.parsing,
components.chunking,
components.indexing,
cleanup,
retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
retry_backoff_max_seconds=settings.ingestion_retry_backoff_max_seconds,
lease_seconds=settings.ingestion_worker_lease_seconds,
)
def create_worker_services(settings: Settings) -> WorkerServices:
"""Build the independent worker runtime from the same application boundaries."""
components = _create_ingestion_components(settings)
permission_resolver = PermissionResolver(components.session_factory)
compiler = CompiledKnowledgeService(
components.session_factory,
create_llm_provider(settings),
)
compilation_jobs = CompilationJobService(
components.session_factory,
settings.compilation_max_attempts,
)
return WorkerServices(
runner=_create_runner(settings, components),
jobs=components.jobs,
dispatcher=components.dispatcher,
compilation_runner=CompilationJobRunner(
components.session_factory,
compiler,
permission_resolver,
retry_backoff_seconds=settings.compilation_retry_backoff_seconds,
retry_backoff_max_seconds=settings.compilation_retry_backoff_max_seconds,
lease_seconds=settings.compilation_worker_lease_seconds,
),
compilation_jobs=compilation_jobs,
compilation_dispatcher=components.compilation_dispatcher,
)
def create_api_services(settings: Settings) -> ApiServices:
"""Compose REST services while ingestion execution remains in workers."""
components = _create_ingestion_components(settings)
document_pipeline = DocumentPipelineService(
components.upload,
components.parsing,
components.chunking,
)
ingestion = DocumentIndexingPipelineService(document_pipeline, components.indexing)
submissions = AsyncIngestionSubmissionService(
components.upload,
components.jobs,
components.dispatcher,
)
batches = IngestionBatchService(
components.session_factory,
submissions,
max_files=settings.batch_max_files,
max_total_bytes=settings.batch_max_total_bytes,
max_archive_files=settings.zip_max_files,
)
archives = ZipIngestionService(
batches,
max_archive_bytes=settings.max_upload_bytes,
max_files=settings.zip_max_files,
max_entry_bytes=settings.max_upload_bytes,
max_total_bytes=settings.batch_max_total_bytes,
max_compression_ratio=settings.zip_max_compression_ratio,
)
dense_retrieval = create_dense_retrieval_service(
components.embedding_provider,
components.vector_store,
components.index_profile,
)
sparse_retrieval = create_sparse_retrieval_service(
components.vector_store,
components.index_profile,
)
hybrid_retrieval = create_hybrid_retrieval_service(
dense_retrieval,
sparse_retrieval,
components.index_profile,
)
reranker_provider = create_reranker_provider(settings, components.index_profile)
search = create_search_service(
components.session_factory,
hybrid_retrieval,
reranker_provider,
components.index_profile,
)
documents = DocumentManagementService(
components.session_factory,
components.storage,
components.vector_store,
active_index_profile_id=components.index_profile.profile_id,
)
permission_resolver = PermissionResolver(components.session_factory)
compiled_knowledge = CompiledKnowledgeService(
components.session_factory,
create_llm_provider(settings),
)
compilation_jobs = CompilationJobService(
components.session_factory,
settings.compilation_max_attempts,
)
return ApiServices(
ingestion=ingestion,
submissions=submissions,
batches=batches,
jobs=components.jobs,
documents=documents,
search=search,
authorized_search=AuthorizedSearchService(
permission_resolver,
search,
),
knowledge_access=AuthorizedKnowledgeAccessService(
components.session_factory,
permission_resolver,
active_index_profile_id=components.index_profile.profile_id,
),
permission_resolver=permission_resolver,
administration=AdministrationService(components.session_factory),
compiled_knowledge=compiled_knowledge,
compilation_jobs=compilation_jobs,
compilation_submissions=AsyncCompilationSubmissionService(
compiled_knowledge,
compilation_jobs,
components.compilation_dispatcher,
),
archives=archives,
)

View File

@ -0,0 +1,838 @@
"""Versioned document and search routes for the retrieval Demo."""
import logging
from collections.abc import Iterator
from typing import Annotated, BinaryIO
from urllib.parse import quote
from uuid import UUID
from fastapi import (
APIRouter,
Depends,
File,
Form,
HTTPException,
Query,
Request,
Response,
UploadFile,
status,
)
from fastapi.responses import StreamingResponse
from sqlalchemy.exc import SQLAlchemyError
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
from ekp_api.providers import ApiServices, ProviderConfigurationError, create_api_services
from ekp_api.schemas import (
DocumentContentResponse,
DocumentResponse,
FolderResponse,
IngestionAcceptedResponse,
IngestionBatchResponse,
IngestionJobResponse,
KnowledgeBaseResponse,
SearchDiagnosticsResponse,
SearchRequest,
SearchResponse,
)
from ekp_core.administration import AdministrationNotFoundError
from ekp_core.archive_ingestion import (
EmptyZipArchiveError,
InvalidZipArchiveError,
UnsafeZipArchiveError,
UnsupportedZipArchiveError,
ZipArchiveLimitError,
)
from ekp_core.dense_retrieval import InvalidRetrievalQueryError
from ekp_core.document_management import (
DocumentArtifactCleanupError,
DocumentFolderScope,
DocumentIngestionActiveError,
DocumentNotFoundError,
DocumentScopeNotFoundError,
DocumentSourceUnavailableError,
DocumentVersionNotFoundError,
)
from ekp_core.document_upload import (
DocumentTooLargeError,
DocumentUploadError,
DuplicateDocumentSourceError,
InvalidDocumentScopeError,
MediaTypeMismatchError,
UnsupportedDocumentTypeError,
)
from ekp_core.ingestion_batches import (
BatchDirectoryScopeError,
BatchFileCountLimitError,
BatchFolderScopeError,
BatchNotFoundError,
BatchSource,
)
from ekp_core.ingestion_jobs import IngestionJobNotFoundError
from ekp_core.permission_resolver import PermissionIdentityError
from ekp_core.search import SearchConfigurationError
from ekp_domain import AclPermission
from ekp_vector_store import OceanBaseConfigurationError, OceanBaseVectorStoreError
router = APIRouter()
logger = logging.getLogger(__name__)
def get_api_services(request: Request) -> ApiServices:
"""Resolve provider composition lazily so health checks need no cloud credentials."""
configured = getattr(request.app.state, "api_services", None)
if isinstance(configured, ApiServices):
return configured
try:
services = create_api_services(request.app.state.settings)
except ProviderConfigurationError as error:
logger.error(
"API provider composition failed",
extra={"event": "api_provider_configuration_failed"},
)
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="API providers are not fully configured",
) from error
except SQLAlchemyError as error:
# Return only the stable dependency boundary. SQLAlchemy messages may
# contain SQL or endpoints and therefore remain out of HTTP responses.
logger.error(
"API dependency initialization failed",
extra={
"event": "api_dependency_initialization_failed",
"dependency": "canonical_database",
},
)
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="API canonical database is unavailable",
headers={"X-EKP-Dependency": "canonical-database"},
) from error
except (OceanBaseConfigurationError, OceanBaseVectorStoreError) as error:
logger.error(
"API dependency initialization failed",
extra={
"event": "api_dependency_initialization_failed",
"dependency": "vector_index",
},
)
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="API vector index is unavailable",
headers={"X-EKP-Dependency": "vector-index"},
) from error
request.app.state.api_services = services
return services
Services = Annotated[ApiServices, Depends(get_api_services)]
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
@router.get(
"/knowledge-bases",
response_model=list[KnowledgeBaseResponse],
tags=["knowledge-bases"],
)
def list_accessible_knowledge_bases(
services: Services, identity: Identity
) -> list[KnowledgeBaseResponse]:
"""List only active Knowledge Bases readable by the current canonical User."""
require_scope(identity, "knowledge:read")
if not identity.is_canonical:
return []
allowed = frozenset(_resolved_knowledge_base_ids(identity, services, AclPermission.READ))
if services.administration is None:
raise HTTPException(status_code=503, detail="administration service is unavailable")
organization_id, user_id = require_canonical_identity(identity)
return [
KnowledgeBaseResponse.from_knowledge_base(value)
for value in services.administration.list_knowledge_bases(organization_id, user_id)
if value.id in allowed
]
@router.get(
"/knowledge-bases/{knowledge_base_id}/folders",
response_model=list[FolderResponse],
tags=["knowledge-bases"],
)
def list_accessible_folders(
knowledge_base_id: UUID, services: Services, identity: Identity
) -> list[FolderResponse]:
"""List folder metadata only after resolving READ access to its Knowledge Base."""
require_scope(identity, "knowledge:read")
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
if knowledge_base_id not in allowed:
raise HTTPException(status_code=404, detail="knowledge base not found")
if services.administration is None:
raise HTTPException(status_code=503, detail="administration service is unavailable")
organization_id, user_id = require_canonical_identity(identity)
try:
values = services.administration.list_folders(organization_id, user_id, knowledge_base_id)
except AdministrationNotFoundError as error:
raise HTTPException(status_code=404, detail="knowledge base not found") from error
return [FolderResponse.from_folder(value) for value in values]
@router.post(
"/documents",
response_model=IngestionAcceptedResponse,
status_code=status.HTTP_202_ACCEPTED,
tags=["documents"],
)
def upload_document(
services: Services,
identity: Identity,
file: Annotated[
UploadFile,
File(description="One configured P0 document, spreadsheet, HTML, or image file"),
],
knowledge_base_id: Annotated[UUID | None, Form()] = None,
folder_id: Annotated[UUID | None, Form()] = None,
) -> IngestionAcceptedResponse:
"""Persist one immutable source and durable Job, then return immediately."""
try:
require_scope(identity, "knowledge:write")
organization_id = _authorize_upload_scope(
identity, services, knowledge_base_id, AclPermission.WRITE
)
submission = services.submissions.submit(
file.filename or "",
file.content_type,
file.file,
organization_id=organization_id,
knowledge_base_id=knowledge_base_id,
folder_id=folder_id,
)
except DuplicateDocumentSourceError as error:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail={
"code": "document_source_already_exists",
"document_id": str(error.document_id),
"document_version_id": str(error.document_version_id),
},
) from error
except DocumentTooLargeError as error:
raise HTTPException(
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
detail="document exceeds the configured upload limit",
) from error
except (UnsupportedDocumentTypeError, MediaTypeMismatchError) as error:
raise HTTPException(
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
detail="unsupported document type or media type",
) from error
except InvalidDocumentScopeError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="knowledge base or folder not found",
) from error
except DocumentUploadError as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="document could not be accepted",
) from error
except HTTPException:
raise
except Exception as error:
# Provider and persistence exceptions are intentionally not reflected to clients.
logger.error(
"Document acceptance failed",
extra={"event": "document_acceptance_failed"},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="document acceptance failed",
) from error
logger.info(
"Document accepted for ingestion",
extra={
"event": "document_accepted",
"document_id": str(submission.document.id),
"document_version_id": str(submission.version.id),
"job_id": str(submission.job.id),
},
)
return IngestionAcceptedResponse.from_submission(submission)
@router.get("/jobs/{job_id}", response_model=IngestionJobResponse, tags=["jobs"])
def get_ingestion_job(job_id: UUID, services: Services, identity: Identity) -> IngestionJobResponse:
"""Return durable stage, attempt and safe failure state for one Job."""
try:
job = services.jobs.get(job_id)
except IngestionJobNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="job not found"
) from error
require_scope(identity, "knowledge:read")
_ensure_document_access(identity, services, job.document_id, AclPermission.READ)
return IngestionJobResponse.from_job(job)
@router.post(
"/ingestion-batches",
response_model=IngestionBatchResponse,
status_code=status.HTTP_202_ACCEPTED,
tags=["jobs"],
)
def upload_ingestion_batch(
services: Services,
identity: Identity,
files: Annotated[
list[UploadFile],
File(description="One or more configured P0 documents, spreadsheets, HTML, or images"),
],
knowledge_base_id: Annotated[UUID | None, Form()] = None,
folder_id: Annotated[UUID | None, Form()] = None,
relative_paths: Annotated[
list[str] | None,
Form(description="Ordered browser-relative paths for a directory upload"),
] = None,
) -> IngestionBatchResponse:
"""Accept a bounded file or directory batch with per-file isolation."""
try:
require_scope(identity, "knowledge:write")
if relative_paths is not None and len(relative_paths) != len(files):
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="relative paths must match the ordered file list",
)
organization_id = _authorize_upload_scope(
identity, services, knowledge_base_id, AclPermission.WRITE
)
view = services.batches.submit(
tuple(
BatchSource(
filename=file.filename or "",
media_type=file.content_type,
source=file.file,
relative_path=(relative_paths[index] if relative_paths is not None else None),
)
for index, file in enumerate(files)
),
organization_id=organization_id,
knowledge_base_id=knowledge_base_id,
folder_id=folder_id,
)
except BatchFileCountLimitError as error:
raise HTTPException(
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
detail="batch file count exceeds the configured limit",
) from error
except BatchDirectoryScopeError as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="directory upload requires a knowledge base",
) from error
except BatchFolderScopeError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="folder not found",
) from error
except HTTPException:
raise
except Exception as error:
logger.error(
"Batch acceptance failed",
extra={"event": "ingestion_batch_acceptance_failed"},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="batch could not be accepted",
) from error
logger.info(
"Ingestion batch accepted",
extra={"event": "ingestion_batch_accepted", "batch_id": str(view.batch.id)},
)
return IngestionBatchResponse.from_view(view)
@router.post(
"/ingestion-archives",
response_model=IngestionBatchResponse,
status_code=status.HTTP_202_ACCEPTED,
tags=["jobs"],
)
def upload_ingestion_archive(
services: Services,
identity: Identity,
archive: Annotated[
UploadFile,
File(description="One ZIP archive containing a bounded document tree"),
],
knowledge_base_id: Annotated[UUID | None, Form()] = None,
folder_id: Annotated[UUID | None, Form()] = None,
) -> IngestionBatchResponse:
"""Safely expand one ZIP into independent canonical document Jobs."""
try:
require_scope(identity, "knowledge:write")
organization_id = _authorize_upload_scope(
identity, services, knowledge_base_id, AclPermission.WRITE
)
if organization_id is None or knowledge_base_id is None:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="ZIP upload requires a knowledge base",
)
if services.archives is None:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="ZIP ingestion is unavailable",
)
view = services.archives.submit(
archive.filename or "",
archive.content_type,
archive.file,
organization_id=organization_id,
knowledge_base_id=knowledge_base_id,
folder_id=folder_id,
)
except UnsupportedZipArchiveError as error:
raise HTTPException(
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
detail="only ZIP archives are supported",
) from error
except (ZipArchiveLimitError, BatchFileCountLimitError) as error:
raise HTTPException(
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
detail="ZIP archive exceeds a configured safety limit",
) from error
except (UnsafeZipArchiveError, InvalidZipArchiveError, EmptyZipArchiveError) as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="ZIP archive is invalid or unsafe",
) from error
except BatchFolderScopeError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="folder not found",
) from error
except HTTPException:
raise
except Exception as error:
logger.error(
"ZIP ingestion failed",
extra={"event": "zip_ingestion_failed"},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="ZIP archive could not be accepted",
) from error
logger.info(
"ZIP ingestion accepted",
extra={"event": "zip_ingestion_accepted", "batch_id": str(view.batch.id)},
)
return IngestionBatchResponse.from_view(view)
@router.get(
"/ingestion-batches/{batch_id}",
response_model=IngestionBatchResponse,
tags=["jobs"],
)
def get_ingestion_batch(
batch_id: UUID, services: Services, identity: Identity
) -> IngestionBatchResponse:
"""Return aggregate and per-file status for Admin Console polling."""
try:
view = services.batches.get(batch_id)
except BatchNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="ingestion batch not found",
) from error
require_scope(identity, "knowledge:read")
_ensure_batch_access(
identity,
services,
view.batch.organization_id,
view.batch.knowledge_base_id,
)
return IngestionBatchResponse.from_view(view)
@router.get("/documents", response_model=list[DocumentResponse], tags=["documents"])
def list_documents(
services: Services,
identity: Identity,
knowledge_base_id: Annotated[
UUID | None,
Query(description="Optional readable Knowledge Base boundary"),
] = None,
folder_scope: Annotated[
DocumentFolderScope,
Query(description="all, root, or one selected folder subtree"),
] = DocumentFolderScope.ALL,
folder_id: Annotated[
UUID | None,
Query(description="Folder root required only when folder_scope=subtree"),
] = None,
) -> list[DocumentResponse]:
"""List canonical documents after intersecting ACL and optional folder scope."""
require_scope(identity, "knowledge:read")
document_ids = _resolved_document_ids(identity, services, AclPermission.READ)
if knowledge_base_id is None:
if folder_scope is not DocumentFolderScope.ALL or folder_id is not None:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="folder scope requires knowledge_base_id",
)
organization_id = None
else:
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
if knowledge_base_id not in allowed:
raise HTTPException(status_code=404, detail="knowledge base not found")
organization_id, _ = require_canonical_identity(identity)
if folder_scope is DocumentFolderScope.SUBTREE and folder_id is None:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="folder_id is required for subtree scope",
)
if folder_scope is not DocumentFolderScope.SUBTREE and folder_id is not None:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="folder_id is only valid for subtree scope",
)
try:
documents = services.documents.list_documents(
document_ids,
organization_id=organization_id,
knowledge_base_id=knowledge_base_id,
folder_scope=folder_scope,
folder_id=folder_id,
)
except DocumentScopeNotFoundError as error:
raise HTTPException(status_code=404, detail="folder not found") from error
return [DocumentResponse.from_document(document) for document in documents]
@router.get("/documents/{document_id}", response_model=DocumentResponse, tags=["documents"])
def get_document(document_id: UUID, services: Services, identity: Identity) -> DocumentResponse:
"""Return one document's status and safe processing diagnostic."""
try:
require_scope(identity, "knowledge:read")
_ensure_document_access(identity, services, document_id, AclPermission.READ)
document = services.documents.get_document(document_id)
except DocumentNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
) from error
return DocumentResponse.from_document(document)
@router.get(
"/documents/{document_id}/content",
response_model=DocumentContentResponse,
tags=["documents"],
)
def get_document_content(
document_id: UUID,
services: Services,
identity: Identity,
document_version_id: UUID | None = None,
) -> DocumentContentResponse:
"""Return canonical indexed content for the selected immutable source version."""
try:
require_scope(identity, "knowledge:read")
_ensure_document_access(identity, services, document_id, AclPermission.READ)
content = services.documents.get_document_content(document_id, document_version_id)
except DocumentNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
) from error
except DocumentVersionNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document version not found"
) from error
return DocumentContentResponse.from_view(content)
@router.get("/documents/{document_id}/source", tags=["documents"])
def get_document_source(
document_id: UUID,
services: Services,
identity: Identity,
document_version_id: UUID | None = None,
) -> StreamingResponse:
"""Stream immutable source bytes without exposing storage keys or Provider details."""
try:
require_scope(identity, "knowledge:read")
_ensure_document_access(identity, services, document_id, AclPermission.READ)
source = services.documents.open_document_source(document_id, document_version_id)
except DocumentNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
) from error
except DocumentVersionNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document version not found"
) from error
except DocumentSourceUnavailableError as error:
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="document source is unavailable",
) from error
inline_media_types = {
"application/pdf",
"image/jpeg",
"image/png",
"text/markdown",
"text/plain",
}
disposition = "inline" if source.media_type in inline_media_types else "attachment"
response_media_type = (
source.media_type if source.media_type in inline_media_types else "application/octet-stream"
)
encoded_filename = quote(source.filename, safe="")
return StreamingResponse(
_read_source_chunks(source.stream),
media_type=response_media_type,
headers={
"Content-Disposition": (
f"{disposition}; filename=\"source\"; filename*=UTF-8''{encoded_filename}"
),
"Content-Length": str(source.size_bytes),
"X-Content-Type-Options": "nosniff",
},
)
def _read_source_chunks(stream: BinaryIO, chunk_size: int = 64 * 1024) -> Iterator[bytes]:
"""Close Provider streams deterministically after bounded response reads."""
try:
while True:
chunk = stream.read(chunk_size)
if not chunk:
return
yield chunk
finally:
stream.close()
@router.delete(
"/documents/{document_id}",
status_code=status.HTTP_204_NO_CONTENT,
tags=["documents"],
)
def delete_document(document_id: UUID, services: Services, identity: Identity) -> Response:
"""Delete an unreferenced document or tombstone an L2-referenced source."""
try:
require_scope(identity, "knowledge:write")
_ensure_document_access(identity, services, document_id, AclPermission.WRITE)
services.documents.delete_document(document_id)
except DocumentNotFoundError as error:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
) from error
except DocumentIngestionActiveError as error:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="document ingestion is still active",
) from error
except DocumentArtifactCleanupError as error:
logger.error(
"Document artifact cleanup failed",
extra={
"event": "document_artifact_cleanup_failed",
"document_id": str(document_id),
},
)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="document metadata was deleted but artifact cleanup failed",
) from error
except Exception as error:
logger.error(
"Document deletion failed",
extra={"event": "document_deletion_failed", "document_id": str(document_id)},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="document deletion failed",
) from error
return Response(status_code=status.HTTP_204_NO_CONTENT)
@router.post("/search", response_model=SearchResponse, tags=["search"])
def search_knowledge(
payload: SearchRequest, services: Services, identity: Identity
) -> SearchResponse:
"""Return reranked Top-K sources without leaking provider or index details."""
try:
require_scope(identity, "knowledge:search")
if identity.is_canonical:
organization_id, user_id = require_canonical_identity(identity)
if services.authorized_search is None:
raise HTTPException(status_code=503, detail="authorized search is unavailable")
result = services.authorized_search.search(
organization_id=organization_id,
user_id=user_id,
query=payload.query,
top_k=payload.top_k,
)
else:
result = services.search.search(payload.query, payload.top_k)
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="search request is invalid for the active index profile",
) from error
except PermissionIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
except HTTPException:
raise
except Exception as error:
logger.error(
"Knowledge search provider chain failed",
extra={"event": "knowledge_search_failed"},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="search providers failed",
) from error
return SearchResponse.from_result(result)
@router.post(
"/search/diagnostics",
response_model=SearchDiagnosticsResponse,
tags=["search"],
)
def diagnose_knowledge_search(
payload: SearchRequest,
services: Services,
identity: Identity,
) -> SearchDiagnosticsResponse:
"""Expose safe stage evidence for the same pipeline used by public Search."""
try:
require_scope(identity, "knowledge:search")
if identity.is_canonical:
organization_id, user_id = require_canonical_identity(identity)
if services.authorized_search is None:
raise HTTPException(status_code=503, detail="authorized search is unavailable")
diagnostics = services.authorized_search.diagnose(
organization_id=organization_id,
user_id=user_id,
query=payload.query,
top_k=payload.top_k,
)
else:
diagnostics = services.search.diagnose(payload.query, payload.top_k)
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
detail="search request is invalid for the active index profile",
) from error
except PermissionIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
except HTTPException:
raise
except Exception as error:
logger.error(
"Knowledge search diagnostics failed",
extra={"event": "knowledge_search_diagnostics_failed"},
)
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail="search diagnostics failed",
) from error
return SearchDiagnosticsResponse.from_result(diagnostics)
def _resolved_document_ids(
identity: ApiIdentity,
services: ApiServices,
permission: AclPermission,
*,
include_deleted: bool = False,
) -> tuple[UUID, ...] | None:
"""Resolve canonical ACL scope while retaining explicit legacy compatibility."""
if not identity.is_canonical:
return None
organization_id, user_id = require_canonical_identity(identity)
if services.permission_resolver is None:
raise HTTPException(status_code=503, detail="permission resolver is unavailable")
try:
return services.permission_resolver.resolve_documents(
organization_id=organization_id,
user_id=user_id,
permission=permission,
include_deleted=include_deleted,
).document_ids
except PermissionIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
def _ensure_document_access(
identity: ApiIdentity,
services: ApiServices,
document_id: UUID,
permission: AclPermission,
) -> None:
"""Use a uniform not-found response for resources outside canonical scope."""
allowed = _resolved_document_ids(identity, services, permission)
if allowed is not None and document_id not in allowed:
raise HTTPException(status_code=404, detail="document not found")
def _authorize_upload_scope(
identity: ApiIdentity,
services: ApiServices,
knowledge_base_id: UUID | None,
permission: AclPermission,
) -> UUID | None:
"""Require canonical uploads to select one writable Knowledge Base."""
if not identity.is_canonical:
return None
organization_id, _ = require_canonical_identity(identity)
if knowledge_base_id is None:
raise HTTPException(status_code=422, detail="knowledge_base_id is required")
allowed = _resolved_knowledge_base_ids(identity, services, permission)
if knowledge_base_id not in allowed:
raise HTTPException(status_code=404, detail="knowledge base not found")
return organization_id
def _resolved_knowledge_base_ids(
identity: ApiIdentity,
services: ApiServices,
permission: AclPermission,
) -> tuple[UUID, ...]:
"""Resolve exact Knowledge Base scope for one canonical REST identity."""
organization_id, user_id = require_canonical_identity(identity)
if services.permission_resolver is None:
raise HTTPException(status_code=503, detail="permission resolver is unavailable")
try:
return services.permission_resolver.resolve_knowledge_bases(
organization_id=organization_id,
user_id=user_id,
permission=permission,
).knowledge_base_ids
except PermissionIdentityError as error:
raise HTTPException(status_code=403, detail="operation is forbidden") from error
def _ensure_batch_access(
identity: ApiIdentity,
services: ApiServices,
batch_organization_id: UUID | None,
batch_knowledge_base_id: UUID | None,
) -> None:
"""Authorize even all-rejected batches from their persisted submission scope."""
if not identity.is_canonical:
if batch_organization_id is not None or batch_knowledge_base_id is not None:
raise HTTPException(status_code=404, detail="ingestion batch not found")
return
organization_id, _ = require_canonical_identity(identity)
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
if (
batch_organization_id != organization_id
or batch_knowledge_base_id is None
or batch_knowledge_base_id not in allowed
):
raise HTTPException(status_code=404, detail="ingestion batch not found")

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,124 @@
"""Celery worker entry point for durable ingestion Jobs."""
import logging
from functools import lru_cache
from typing import Any
from uuid import UUID
from ekp_api.celery_queue import (
COMPILATION_RECOVERY_TASK_NAME,
COMPILATION_TASK_NAME,
INGESTION_TASK_NAME,
RECOVERY_TASK_NAME,
create_celery_app,
)
from ekp_api.providers import WorkerServices, create_worker_services
from ekp_core.compilation_jobs import CompilationRunAction, dispatch_pending_compilation_jobs
from ekp_core.config import get_settings
from ekp_core.ingestion_jobs import JobRunAction, dispatch_pending_jobs
from ekp_core.observability import configure_json_logging
_settings = get_settings()
configure_json_logging(_settings.log_level)
celery_app = create_celery_app(_settings)
logger = logging.getLogger(__name__)
@lru_cache
def _worker_services() -> WorkerServices:
"""Build one provider/service graph per worker process."""
return create_worker_services(_settings)
@celery_app.task( # type: ignore[untyped-decorator]
bind=True,
name=INGESTION_TASK_NAME,
max_retries=None,
)
def run_ingestion(self: Any, job_id: str) -> None:
"""Execute one delivery and ask Celery to redeliver only when required.
Celery delivery retries are unbounded because valid-lease deferrals are not
business Attempts; the durable runner independently enforces max_attempts.
"""
logger.info(
"Ingestion delivery received",
extra={"event": "ingestion_delivery_received", "job_id": job_id},
)
result = _worker_services().runner.run(UUID(job_id))
logger.info(
"Ingestion delivery completed",
extra={
"event": "ingestion_delivery_completed",
"job_id": job_id,
"job_status": result.status.value,
"retry_delay_seconds": result.retry_delay_seconds,
},
)
if result.action == JobRunAction.RETRY:
raise self.retry(countdown=result.retry_delay_seconds or 1, max_retries=None)
@celery_app.task(name=RECOVERY_TASK_NAME) # type: ignore[untyped-decorator]
def recover_ingestion_jobs() -> int:
"""Redeliver due durable Jobs after API/broker or worker interruptions."""
services = _worker_services()
dispatch_count = dispatch_pending_jobs(services.jobs, services.dispatcher)
logger.info(
"Durable ingestion recovery scan completed",
extra={
"event": "ingestion_recovery_scan_completed",
"dispatch_count": dispatch_count,
},
)
return dispatch_count
@celery_app.task( # type: ignore[untyped-decorator]
bind=True,
name=COMPILATION_TASK_NAME,
max_retries=None,
)
def run_compilation(self: Any, job_id: str) -> None:
"""Execute one durable AI generation delivery outside the API process."""
logger.info(
"Compilation delivery received",
extra={"event": "compilation_delivery_received", "job_id": job_id},
)
result = _worker_services().compilation_runner.run(UUID(job_id))
logger.info(
"Compilation delivery completed",
extra={
"event": "compilation_delivery_completed",
"job_id": job_id,
"job_status": result.status.value,
"retry_delay_seconds": result.retry_delay_seconds,
},
)
if result.action == CompilationRunAction.RETRY:
raise self.retry(countdown=result.retry_delay_seconds or 1, max_retries=None)
@celery_app.task(name=COMPILATION_RECOVERY_TASK_NAME) # type: ignore[untyped-decorator]
def recover_compilation_jobs() -> int:
"""Redeliver due durable Compilation Jobs after process interruptions."""
services = _worker_services()
dispatch_count = dispatch_pending_compilation_jobs(
services.compilation_jobs,
services.compilation_dispatcher,
)
logger.info(
"Durable compilation recovery scan completed",
extra={
"event": "compilation_recovery_scan_completed",
"dispatch_count": dispatch_count,
},
)
return dispatch_count
def main() -> None:
"""Run one worker with the embedded Demo recovery scheduler."""
celery_app.worker_main(
["worker", "--loglevel=INFO", "--queues=ingestion,compilation", "--beat"]
)

View File

View File

@ -0,0 +1,5 @@
"""Enterprise Knowledge MCP gateway."""
from ekp_mcp.server import create_mcp_http_app, create_mcp_server
__all__ = ["create_mcp_http_app", "create_mcp_server"]

View File

@ -0,0 +1,157 @@
"""Request-scoped MCP service identity and bearer-token verification."""
from dataclasses import dataclass
from secrets import compare_digest
from typing import Literal, cast
from uuid import UUID
from mcp.server.auth.middleware.auth_context import get_access_token
from mcp.server.auth.provider import AccessToken
from mcp.server.mcpserver.exceptions import ToolError
from pydantic import SecretStr
from ekp_core.config import McpIdentityCredential
McpCredentialType = Literal["legacy_public", "user_token", "service_api_key"]
@dataclass(frozen=True, slots=True)
class McpIdentity:
"""Authenticated principal and its canonical ACL subject for one request."""
client_id: str
subject: str
scopes: frozenset[str]
credential_type: McpCredentialType = "legacy_public"
organization_id: UUID | None = None
acl_user_id: UUID | None = None
class StaticServiceTokenVerifier:
"""Validate one runtime-injected, least-privilege Demo service credential.
The verifier is intentionally small for DEMO-012. It establishes the MCP
identity boundary without pretending to be the V1 OAuth or ACL solution.
"""
__slots__ = (
"_client_id",
"_expected_token",
"_issuer",
"_scope",
"_subject",
)
def __init__(
self,
expected_token: SecretStr,
*,
client_id: str,
subject: str,
scope: str,
issuer: str,
) -> None:
self._expected_token = expected_token
self._client_id = client_id
self._subject = subject
self._scope = scope
self._issuer = issuer
async def verify_token(self, token: str) -> AccessToken | None:
"""Return scoped identity metadata only for a constant-time token match."""
if not compare_digest(token, self._expected_token.get_secret_value()):
return None
return AccessToken(
token=token,
client_id=self._client_id,
subject=self._subject,
scopes=[self._scope],
claims={"iss": self._issuer},
)
class ConfiguredIdentityTokenVerifier:
"""Verify runtime-secret user/service credentials without logging their values."""
__slots__ = ("_credentials", "_issuer")
def __init__(
self,
credentials: tuple[McpIdentityCredential, ...],
*,
issuer: str,
) -> None:
if not credentials:
raise ValueError("at least one MCP identity credential is required")
tokens = [credential.token.get_secret_value() for credential in credentials]
if len(set(tokens)) != len(tokens):
raise ValueError("MCP identity credential tokens must be unique")
self._credentials = credentials
self._issuer = issuer
async def verify_token(self, token: str) -> AccessToken | None:
"""Map a matching bearer secret to its configured canonical identity."""
selected: McpIdentityCredential | None = None
# Compare every configured secret so match position does not become a
# useful timing signal when multiple principals share one gateway.
for credential in self._credentials:
if compare_digest(token, credential.token.get_secret_value()):
selected = credential
if selected is None:
return None
return AccessToken(
token=token,
client_id=selected.client_id,
subject=selected.subject,
scopes=list(selected.scopes),
claims={
"iss": self._issuer,
"ekp_credential_type": selected.credential_type,
"ekp_organization_id": str(selected.organization_id),
"ekp_acl_user_id": str(selected.acl_user_id),
},
)
def current_mcp_identity() -> McpIdentity:
"""Resolve the SDK-authenticated principal for the active tool request."""
access_token = get_access_token()
if access_token is None:
# HTTP middleware normally rejects this earlier; retaining the guard
# prevents accidental unauthenticated in-process tool invocation.
raise ToolError("authentication required")
subject = access_token.subject or access_token.client_id
claims = access_token.claims or {}
raw_credential_type = claims.get("ekp_credential_type", "legacy_public")
credential_type = cast(McpCredentialType, raw_credential_type)
if credential_type not in {"legacy_public", "user_token", "service_api_key"}:
raise ToolError("authentication identity is invalid")
organization_id = _optional_uuid_claim(claims, "ekp_organization_id")
acl_user_id = _optional_uuid_claim(claims, "ekp_acl_user_id")
if (organization_id is None) != (acl_user_id is None):
raise ToolError("authentication identity is invalid")
if credential_type != "legacy_public" and organization_id is None:
raise ToolError("authentication identity is invalid")
if credential_type == "legacy_public" and organization_id is not None:
raise ToolError("authentication identity is invalid")
return McpIdentity(
client_id=access_token.client_id,
subject=subject,
scopes=frozenset(access_token.scopes),
credential_type=credential_type,
organization_id=organization_id,
acl_user_id=acl_user_id,
)
def _optional_uuid_claim(claims: dict[str, object], name: str) -> UUID | None:
"""Parse an optional UUID claim without reflecting attacker-controlled text."""
value = claims.get(name)
if value is None:
return None
if not isinstance(value, str):
raise ToolError("authentication identity is invalid")
try:
return UUID(value)
except ValueError as error:
raise ToolError("authentication identity is invalid") from error

View File

@ -0,0 +1,308 @@
"""Identity-aware MCP read boundary for documents, sections, sources, and spaces."""
from dataclasses import dataclass
from typing import Literal, Protocol
from uuid import UUID
from pydantic import BaseModel
from ekp_core.knowledge_access import (
KnowledgeBaseReadView,
KnowledgeDocumentReadView,
KnowledgeSectionReadView,
KnowledgeSourceReadView,
)
from ekp_domain import DocumentStatus, KnowledgeBaseVisibility
from ekp_mcp.auth import McpIdentity
class KnowledgeReader(Protocol):
"""Subset of the shared authorized read service consumed by MCP."""
def list_knowledge_bases(
self,
*,
organization_id: UUID,
user_id: UUID,
) -> tuple[KnowledgeBaseReadView, ...]: ...
def get_document(
self,
*,
organization_id: UUID,
user_id: UUID,
document_id: UUID,
document_version_id: UUID | None = None,
) -> KnowledgeDocumentReadView: ...
def get_source(
self,
*,
organization_id: UUID,
user_id: UUID,
document_id: UUID,
chunk_id: UUID,
document_version_id: UUID | None = None,
) -> KnowledgeSourceReadView: ...
def get_document_section(
self,
*,
organization_id: UUID,
user_id: UUID,
document_id: UUID,
anchor_chunk_id: UUID,
document_version_id: UUID | None = None,
offset: int = 0,
max_chunks: int = 12,
) -> KnowledgeSectionReadView: ...
class McpKnowledgeBaseItem(BaseModel):
"""One authorized MCP-enabled Knowledge Base."""
knowledge_base_id: UUID
name: str
description: str | None
visibility: KnowledgeBaseVisibility
readable_document_count: int
@classmethod
def from_view(cls, view: KnowledgeBaseReadView) -> "McpKnowledgeBaseItem":
return cls(
knowledge_base_id=view.id,
name=view.name,
description=view.description,
visibility=view.visibility,
readable_document_count=view.readable_document_count,
)
class McpKnowledgeBaseListResponse(BaseModel):
"""Stable structured response for `list_knowledge_bases`."""
knowledge_bases: list[McpKnowledgeBaseItem]
class McpDocumentResponse(BaseModel):
"""Document metadata and bounded outline without bulk body transfer."""
document_id: UUID
knowledge_base_id: UUID
knowledge_base_name: str
document_version_id: UUID
document_version: int
filename: str
media_type: str
size_bytes: int
status: DocumentStatus
is_active: bool
chunk_count: int
section_paths: list[tuple[str, ...]]
outline_truncated: bool
@classmethod
def from_view(cls, view: KnowledgeDocumentReadView) -> "McpDocumentResponse":
return cls(
document_id=view.document_id,
knowledge_base_id=view.knowledge_base_id,
knowledge_base_name=view.knowledge_base_name,
document_version_id=view.document_version_id,
document_version=view.document_version,
filename=view.filename,
media_type=view.media_type,
size_bytes=view.size_bytes,
status=view.status,
is_active=view.is_active,
chunk_count=view.chunk_count,
section_paths=list(view.section_paths),
outline_truncated=view.outline_truncated,
)
class McpKnowledgeChunk(BaseModel):
"""Exact canonical Chunk and its source coordinates."""
chunk_id: UUID
chunk_index: int
page: int | None
page_end: int | None
section: str | None
section_path: tuple[str, ...]
sheet_name: str | None
row_range: tuple[int, int] | None
column_range: tuple[int, int] | None
content: str
source_type: Literal["raw"]
class McpSourceResponse(BaseModel):
"""One exact source passage addressable from a Search result."""
document_id: UUID
knowledge_base_id: UUID
document_version_id: UUID
document_version: int
filename: str
chunk: McpKnowledgeChunk
@classmethod
def from_view(cls, view: KnowledgeSourceReadView) -> "McpSourceResponse":
return cls(
document_id=view.document_id,
knowledge_base_id=view.knowledge_base_id,
document_version_id=view.document_version_id,
document_version=view.document_version,
filename=view.filename,
chunk=McpKnowledgeChunk.model_validate(view.chunk, from_attributes=True),
)
class McpDocumentSectionResponse(BaseModel):
"""Bounded source-order page of one canonical section."""
document_id: UUID
knowledge_base_id: UUID
document_version_id: UUID
document_version: int
filename: str
anchor_chunk_id: UUID
section_path: tuple[str, ...]
offset: int
total_chunk_count: int
chunks: list[McpKnowledgeChunk]
next_offset: int | None
@classmethod
def from_view(cls, view: KnowledgeSectionReadView) -> "McpDocumentSectionResponse":
return cls(
document_id=view.document_id,
knowledge_base_id=view.knowledge_base_id,
document_version_id=view.document_version_id,
document_version=view.document_version,
filename=view.filename,
anchor_chunk_id=view.anchor_chunk_id,
section_path=view.section_path,
offset=view.offset,
total_chunk_count=view.total_chunk_count,
chunks=[
McpKnowledgeChunk.model_validate(chunk, from_attributes=True)
for chunk in view.chunks
],
next_offset=view.next_offset,
)
class McpKnowledgeReader(Protocol):
"""Transport-facing application contract registered by the MCP server."""
def list_knowledge_bases(self, identity: McpIdentity) -> McpKnowledgeBaseListResponse: ...
def get_document(
self,
identity: McpIdentity,
document_id: UUID,
document_version_id: UUID | None,
) -> McpDocumentResponse: ...
def get_source(
self,
identity: McpIdentity,
document_id: UUID,
chunk_id: UUID,
document_version_id: UUID | None,
) -> McpSourceResponse: ...
def get_document_section(
self,
identity: McpIdentity,
document_id: UUID,
anchor_chunk_id: UUID,
document_version_id: UUID | None,
offset: int,
max_chunks: int,
) -> McpDocumentSectionResponse: ...
@dataclass(frozen=True, slots=True)
class McpKnowledgeApplication:
"""Require a scoped canonical identity for every non-search knowledge read."""
reader: KnowledgeReader
required_scope: str
def list_knowledge_bases(self, identity: McpIdentity) -> McpKnowledgeBaseListResponse:
organization_id, user_id = self._authorize(identity)
return McpKnowledgeBaseListResponse(
knowledge_bases=[
McpKnowledgeBaseItem.from_view(view)
for view in self.reader.list_knowledge_bases(
organization_id=organization_id,
user_id=user_id,
)
]
)
def get_document(
self,
identity: McpIdentity,
document_id: UUID,
document_version_id: UUID | None,
) -> McpDocumentResponse:
organization_id, user_id = self._authorize(identity)
return McpDocumentResponse.from_view(
self.reader.get_document(
organization_id=organization_id,
user_id=user_id,
document_id=document_id,
document_version_id=document_version_id,
)
)
def get_source(
self,
identity: McpIdentity,
document_id: UUID,
chunk_id: UUID,
document_version_id: UUID | None,
) -> McpSourceResponse:
organization_id, user_id = self._authorize(identity)
return McpSourceResponse.from_view(
self.reader.get_source(
organization_id=organization_id,
user_id=user_id,
document_id=document_id,
chunk_id=chunk_id,
document_version_id=document_version_id,
)
)
def get_document_section(
self,
identity: McpIdentity,
document_id: UUID,
anchor_chunk_id: UUID,
document_version_id: UUID | None,
offset: int,
max_chunks: int,
) -> McpDocumentSectionResponse:
organization_id, user_id = self._authorize(identity)
return McpDocumentSectionResponse.from_view(
self.reader.get_document_section(
organization_id=organization_id,
user_id=user_id,
document_id=document_id,
anchor_chunk_id=anchor_chunk_id,
document_version_id=document_version_id,
offset=offset,
max_chunks=max_chunks,
)
)
def _authorize(self, identity: McpIdentity) -> tuple[UUID, UUID]:
"""Fail closed for legacy or partially mapped identities."""
if self.required_scope not in identity.scopes:
raise PermissionError("MCP identity lacks the knowledge read scope")
if identity.organization_id is None or identity.acl_user_id is None:
raise PermissionError("MCP identity is not bound to a canonical ACL subject")
return identity.organization_id, identity.acl_user_id

View File

@ -0,0 +1,32 @@
"""Command-line entry point for the independent MCP gateway process."""
from mcp.server.transport_security import TransportSecuritySettings
from ekp_core.config import get_settings
from ekp_core.observability import configure_json_logging
from ekp_mcp.server import create_mcp_server
def main() -> None:
"""Run the authenticated Streamable HTTP server from environment settings."""
settings = get_settings()
configure_json_logging(settings.log_level)
server = create_mcp_server(settings)
server.run(
transport="streamable-http",
host=settings.mcp_host,
port=settings.mcp_port,
streamable_http_path=settings.mcp_path,
json_response=True,
stateless_http=True,
max_request_body_size=settings.mcp_max_request_body_bytes,
transport_security=TransportSecuritySettings(
enable_dns_rebinding_protection=True,
allowed_hosts=list(settings.mcp_allowed_hosts),
allowed_origins=list(settings.mcp_allowed_origins),
),
)
if __name__ == "__main__": # pragma: no cover - exercised through the console script.
main()

View File

@ -0,0 +1,168 @@
"""MCP-specific application boundary over the provider-neutral SearchService."""
from dataclasses import dataclass
from typing import Annotated, Literal, Protocol
from uuid import UUID
from pydantic import BaseModel, ConfigDict, Field
from ekp_core.search import CompiledSourceResult, SearchResult
from ekp_domain import CompiledKnowledgeType
from ekp_mcp.auth import McpIdentity
class KnowledgeSearcher(Protocol):
"""Subset of SearchService required by the MCP gateway."""
def search(self, query: str, top_k: int | None = None) -> SearchResult:
"""Return reranked canonical sources for a normalized query."""
class AuthorizedKnowledgeSearcher(Protocol):
"""Subset of AuthorizedSearchService required by the MCP gateway."""
def search(
self,
*,
organization_id: UUID,
user_id: UUID,
query: str,
top_k: int | None = None,
mcp_enabled_only: bool = False,
) -> SearchResult:
"""Resolve ACL scope before either retrieval Top-N branch."""
class McpSourceResponse(BaseModel):
"""Provider-neutral source shape exposed by `search_knowledge`."""
document_id: UUID
document_version_id: UUID
document_version: int
filename: str
page: int | None
page_end: int | None = None
section: str | None
section_path: tuple[str, ...]
sheet_name: str | None = None
row_range: tuple[int, int] | None = None
column_range: tuple[int, int] | None = None
chunk_id: UUID
content: str
context: str
score: float
rerank_score: float
source_type: Literal["raw"]
class McpCompiledUnderlyingSourceResponse(BaseModel):
"""Exact source evidence beneath a published Compiled result."""
model_config = ConfigDict(from_attributes=True)
document_id: UUID
document_version_id: UUID
document_version: int
filename: str
page: int | None
page_end: int | None
section: str | None
section_path: tuple[str, ...]
sheet_name: str | None
row_range: tuple[int, int] | None
column_range: tuple[int, int] | None
chunk_id: UUID
chunk_index: int
content: str
class McpCompiledSearchResponse(BaseModel):
"""Published L2 result aligned exactly with the REST response shape."""
model_config = ConfigDict(from_attributes=True)
compiled_knowledge_id: UUID
knowledge_base_id: UUID
title: str
knowledge_type: CompiledKnowledgeType
content: str
context: str
underlying_sources: tuple[McpCompiledUnderlyingSourceResponse, ...]
score: float
rerank_score: float
source_type: Literal["compiled"]
McpSearchSourceResponse = Annotated[
McpSourceResponse | McpCompiledSearchResponse,
Field(discriminator="source_type"),
]
class McpSearchResponse(BaseModel):
"""Structured MCP result aligned with the REST search source fields."""
results: list[McpSearchSourceResponse]
class McpKnowledgeSearch(Protocol):
"""Identity-aware search boundary consumed by the protocol server."""
def search(self, identity: McpIdentity, query: str, top_k: int) -> McpSearchResponse:
"""Return authorized, structured knowledge sources."""
@dataclass(frozen=True, slots=True)
class McpSearchApplication:
"""Enforce MCP scope and canonical Document ACL before shared retrieval."""
searcher: KnowledgeSearcher
required_scope: str
authorized_searcher: AuthorizedKnowledgeSearcher | None = None
allow_legacy_public: bool = False
def search(self, identity: McpIdentity, query: str, top_k: int) -> McpSearchResponse:
"""Authorize the principal, fail closed, and serialize canonical results."""
if self.required_scope not in identity.scopes:
raise PermissionError("MCP identity lacks the knowledge search scope")
if identity.organization_id is not None and identity.acl_user_id is not None:
if self.authorized_searcher is None:
raise PermissionError("MCP ACL search service is unavailable")
result = self.authorized_searcher.search(
organization_id=identity.organization_id,
user_id=identity.acl_user_id,
query=query,
top_k=top_k,
mcp_enabled_only=True,
)
elif identity.credential_type == "legacy_public" and self.allow_legacy_public:
result = self.searcher.search(query, top_k)
else:
raise PermissionError("MCP identity is not bound to a canonical ACL subject")
responses: list[McpSourceResponse | McpCompiledSearchResponse] = []
for source in result.results:
if isinstance(source, CompiledSourceResult):
responses.append(McpCompiledSearchResponse.model_validate(source))
else:
responses.append(
McpSourceResponse(
document_id=source.document_id,
document_version_id=source.document_version_id,
document_version=source.document_version,
filename=source.filename,
page=source.page,
page_end=source.page_end,
section=source.section,
section_path=source.section_path,
sheet_name=source.sheet_name,
row_range=source.row_range,
column_range=source.column_range,
chunk_id=source.chunk_id,
content=source.content,
context=source.context,
score=source.score,
rerank_score=source.rerank_score,
source_type=source.source_type,
)
)
return McpSearchResponse(results=responses)

View File

@ -0,0 +1,323 @@
"""Authenticated Streamable HTTP MCP server exposing Demo business tools."""
import logging
from functools import partial
from typing import Annotated
from uuid import UUID
import anyio
from mcp.server import MCPServer
from mcp.server.auth.provider import TokenVerifier
from mcp.server.auth.settings import AuthSettings
from mcp.server.mcpserver.exceptions import ToolError
from mcp.server.transport_security import TransportSecuritySettings
from pydantic import Field
from starlette.applications import Starlette
from ekp_api.providers import ApiServices, create_runtime_search_services
from ekp_core.config import Settings, get_settings
from ekp_core.dense_retrieval import InvalidRetrievalQueryError
from ekp_core.knowledge_access import (
KnowledgeAccessNotFoundError,
KnowledgeSectionOffsetError,
)
from ekp_core.permission_resolver import PermissionResolutionError
from ekp_core.search import SearchConfigurationError
from ekp_mcp.auth import (
ConfiguredIdentityTokenVerifier,
StaticServiceTokenVerifier,
current_mcp_identity,
)
from ekp_mcp.knowledge import (
McpDocumentResponse,
McpDocumentSectionResponse,
McpKnowledgeApplication,
McpKnowledgeBaseListResponse,
McpKnowledgeReader,
McpSourceResponse,
)
from ekp_mcp.search import McpKnowledgeSearch, McpSearchApplication, McpSearchResponse
logger = logging.getLogger(__name__)
class McpConfigurationError(ValueError):
"""Raised when the gateway lacks its dedicated runtime credential."""
def _create_applications(
settings: Settings,
api_services: ApiServices | None,
) -> tuple[McpSearchApplication, McpKnowledgeApplication]:
if api_services is not None:
searcher = api_services.search
authorized_searcher = api_services.authorized_search
knowledge_access = api_services.knowledge_access
else:
runtime = create_runtime_search_services(settings)
searcher = runtime.search
authorized_searcher = runtime.authorized_search
knowledge_access = runtime.knowledge_access
if knowledge_access is None:
raise McpConfigurationError("MCP knowledge access service is required")
return (
McpSearchApplication(
searcher,
settings.mcp_required_scope,
authorized_searcher=authorized_searcher,
allow_legacy_public=(
settings.mcp_auth_token is not None and not settings.mcp_identity_credentials
),
),
McpKnowledgeApplication(knowledge_access, settings.mcp_required_scope),
)
def _create_token_verifier(settings: Settings) -> TokenVerifier:
"""Build the local secret verifier while keeping external IdP injection open."""
if settings.mcp_identity_credentials:
return ConfiguredIdentityTokenVerifier(
tuple(settings.mcp_identity_credentials),
issuer=str(settings.mcp_issuer_url),
)
legacy_token = settings.mcp_auth_token
if legacy_token is None or len(legacy_token.get_secret_value()) < 32:
raise McpConfigurationError(
"MCP_IDENTITY_CREDENTIALS or a legacy MCP_AUTH_TOKEN of at least "
"32 characters is required"
)
return StaticServiceTokenVerifier(
legacy_token,
client_id=settings.mcp_client_id,
subject=settings.mcp_service_identity,
scope=settings.mcp_required_scope,
issuer=str(settings.mcp_issuer_url),
)
def create_mcp_server(
settings: Settings | None = None,
*,
api_services: ApiServices | None = None,
search_application: McpKnowledgeSearch | None = None,
knowledge_application: McpKnowledgeReader | None = None,
token_verifier: TokenVerifier | None = None,
) -> MCPServer[dict[str, object]]:
"""Create the authenticated MCP gateway with all five V1 P0 read tools."""
resolved_settings = settings or get_settings()
resolved_token_verifier = token_verifier or _create_token_verifier(resolved_settings)
if search_application is None or knowledge_application is None:
default_search, default_knowledge = _create_applications(
resolved_settings,
api_services,
)
application = search_application or default_search
knowledge = knowledge_application or default_knowledge
else:
application = search_application
knowledge = knowledge_application
server: MCPServer[dict[str, object]] = MCPServer(
name="enterprise-knowledge",
title="Enterprise Knowledge MCP Server",
description="Authenticated enterprise knowledge search backed by canonical sources.",
version="0.1.0",
token_verifier=resolved_token_verifier,
auth=AuthSettings(
issuer_url=resolved_settings.mcp_issuer_url,
resource_server_url=resolved_settings.mcp_public_url,
required_scopes=[resolved_settings.mcp_required_scope],
),
log_level=resolved_settings.log_level,
)
@server.tool(
name="search_knowledge",
description=(
"Search authorized enterprise knowledge and return traceable source passages. "
"Use for factual questions that should be grounded in indexed company documents."
),
structured_output=True,
)
async def search_knowledge(
query: Annotated[str, Field(min_length=1, max_length=1600)],
top_k: Annotated[int, Field(ge=1, le=64, strict=True)] = 5,
) -> McpSearchResponse:
"""Run the shared synchronous search pipeline outside the ASGI event loop."""
identity = current_mcp_identity()
try:
return await anyio.to_thread.run_sync(
partial(application.search, identity, query, top_k)
)
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
raise ToolError("search request is invalid for the active index profile") from error
except (PermissionError, PermissionResolutionError) as error:
raise ToolError("knowledge search is not authorized") from error
except Exception:
# Do not reflect provider, database, endpoint, or credential details
# into MCP content or logs. Operations can correlate this generic
# event with upstream sanitized service telemetry.
logger.error(
"MCP knowledge search failed",
extra={"event": "mcp_knowledge_search_failed"},
)
raise ToolError("knowledge search failed") from None
@server.tool(
name="list_knowledge_bases",
description=(
"List Knowledge Bases the current identity may read through MCP, including "
"the count of readable documents in each space."
),
structured_output=True,
)
async def list_knowledge_bases() -> McpKnowledgeBaseListResponse:
"""List authorized spaces without exposing ACL rows or infrastructure."""
identity = current_mcp_identity()
try:
return await anyio.to_thread.run_sync(partial(knowledge.list_knowledge_bases, identity))
except (PermissionError, PermissionResolutionError) as error:
raise ToolError("knowledge access is not authorized") from error
except Exception:
_log_knowledge_failure("mcp_list_knowledge_bases_failed")
raise ToolError("knowledge access failed") from None
@server.tool(
name="get_document",
description=(
"Get authorized document/version metadata and a bounded section outline. "
"Use get_document_section for readable body text."
),
structured_output=True,
)
async def get_document(
document_id: UUID,
document_version_id: UUID | None = None,
) -> McpDocumentResponse:
"""Return metadata for one authorized document without bulk source transfer."""
identity = current_mcp_identity()
try:
return await anyio.to_thread.run_sync(
partial(
knowledge.get_document,
identity,
document_id,
document_version_id,
)
)
except KnowledgeAccessNotFoundError as error:
raise ToolError("knowledge resource is unavailable") from error
except (PermissionError, PermissionResolutionError) as error:
raise ToolError("knowledge access is not authorized") from error
except Exception:
_log_knowledge_failure("mcp_get_document_failed")
raise ToolError("knowledge access failed") from None
@server.tool(
name="get_document_section",
description=(
"Read one authorized canonical section in source order. Pass an anchor chunk "
"from search/get_source and use next_offset until it is null."
),
structured_output=True,
)
async def get_document_section(
document_id: UUID,
anchor_chunk_id: UUID,
document_version_id: UUID | None = None,
offset: Annotated[int, Field(ge=0, le=1_000_000, strict=True)] = 0,
max_chunks: Annotated[int, Field(ge=1, le=32, strict=True)] = 12,
) -> McpDocumentSectionResponse:
"""Return a bounded page of the anchor Chunk's exact section."""
identity = current_mcp_identity()
try:
return await anyio.to_thread.run_sync(
partial(
knowledge.get_document_section,
identity,
document_id,
anchor_chunk_id,
document_version_id,
offset,
max_chunks,
)
)
except KnowledgeAccessNotFoundError as error:
raise ToolError("knowledge resource is unavailable") from error
except KnowledgeSectionOffsetError as error:
raise ToolError("document section request is invalid") from error
except (PermissionError, PermissionResolutionError) as error:
raise ToolError("knowledge access is not authorized") from error
except Exception:
_log_knowledge_failure("mcp_get_document_section_failed")
raise ToolError("knowledge access failed") from None
@server.tool(
name="get_source",
description=(
"Resolve one authorized search chunk to its exact canonical text and source "
"coordinates. This tool returns source evidence, not raw file bytes."
),
structured_output=True,
)
async def get_source(
document_id: UUID,
chunk_id: UUID,
document_version_id: UUID | None = None,
) -> McpSourceResponse:
"""Return exact canonical source evidence for one authorized Chunk."""
identity = current_mcp_identity()
try:
return await anyio.to_thread.run_sync(
partial(
knowledge.get_source,
identity,
document_id,
chunk_id,
document_version_id,
)
)
except KnowledgeAccessNotFoundError as error:
raise ToolError("knowledge resource is unavailable") from error
except (PermissionError, PermissionResolutionError) as error:
raise ToolError("knowledge access is not authorized") from error
except Exception:
_log_knowledge_failure("mcp_get_source_failed")
raise ToolError("knowledge access failed") from None
return server
def create_mcp_http_app(
settings: Settings | None = None,
*,
api_services: ApiServices | None = None,
search_application: McpKnowledgeSearch | None = None,
knowledge_application: McpKnowledgeReader | None = None,
token_verifier: TokenVerifier | None = None,
) -> Starlette:
"""Build the stateless Streamable HTTP app used by WorkBuddy and tests."""
resolved_settings = settings or get_settings()
server = create_mcp_server(
resolved_settings,
api_services=api_services,
search_application=search_application,
knowledge_application=knowledge_application,
token_verifier=token_verifier,
)
return server.streamable_http_app(
streamable_http_path=resolved_settings.mcp_path,
json_response=True,
stateless_http=True,
max_request_body_size=resolved_settings.mcp_max_request_body_bytes,
host=resolved_settings.mcp_host,
transport_security=TransportSecuritySettings(
enable_dns_rebinding_protection=True,
allowed_hosts=list(resolved_settings.mcp_allowed_hosts),
allowed_origins=list(resolved_settings.mcp_allowed_origins),
),
)
def _log_knowledge_failure(event: str) -> None:
"""Emit only an allowlisted event name for non-search MCP failures."""
logger.error("MCP knowledge access failed", extra={"event": event})

View File

@ -0,0 +1,36 @@
# Reproducible target names consumed by local and CI buildx invocations.
variable "IMAGE_REGISTRY" {
default = "knowledge-platform"
}
variable "IMAGE_TAG" {
default = "dev"
}
group "default" {
targets = ["api", "mcp", "worker"]
}
target "common" {
context = "."
dockerfile = "Dockerfile"
platforms = ["linux/amd64", "linux/arm64"]
}
target "api" {
inherits = ["common"]
target = "api"
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-api:${IMAGE_TAG}"]
}
target "mcp" {
inherits = ["common"]
target = "mcp"
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-mcp:${IMAGE_TAG}"]
}
target "worker" {
inherits = ["common"]
target = "worker"
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-worker:${IMAGE_TAG}"]
}

View File

@ -0,0 +1,73 @@
# Copy this file outside the repository for one environment. It contains only
# non-sensitive settings and Kubernetes Secret references; never add values for
# DATABASE_URL, RABBITMQ_URL, provider keys, passwords or MCP_AUTH_TOKEN here.
existingSecret: knowledge-platform-runtime
images:
api:
repository: registry.example.internal/knowledge/enterprise-knowledge-api
tag: "0.1.0"
digest: ""
mcp:
repository: registry.example.internal/knowledge/enterprise-knowledge-mcp
tag: "0.1.0"
digest: ""
worker:
repository: registry.example.internal/knowledge/enterprise-knowledge-worker
tag: "0.1.0"
digest: ""
edge:
repository: registry.example.internal/mirror/nginx
tag: "1.29.1-alpine"
digest: ""
podDisruptionBudget:
enabled: true
apiMinAvailable: 1
mcpMinAvailable: 1
edgeMinAvailable: 1
observability:
metrics:
enabled: true
path: /metrics
ingress:
enabled: true
className: nginx
api:
host: knowledge.example.internal
tlsSecretName: knowledge-platform-api-tls
mcp:
host: mcp.example.internal
tlsSecretName: knowledge-platform-mcp-tls
# Use this mode instead of Ingress on clusters whose external gateway already
# targets one NodePort. Stage with an unused port before taking over the live one.
edge:
enabled: false
host: knowledge.example.internal
service:
nodePort: 0
config:
objectStorage:
endpoint: https://s3.example.internal
bucket: knowledge-platform
vectorStore:
uri: oceanbase.example.internal:2881
user: knowledge-platform
database: knowledge_platform
mcp:
publicUrl: https://mcp.example.internal/mcp
issuerUrl: https://mcp.example.internal
allowedHosts:
- mcp.example.internal
allowedOrigins:
- https://mcp.example.internal
# External RabbitMQ is the production default. If this local dependency is
# explicitly enabled for a demo cluster, create its Secret before deployment.
rabbitmq:
enabled: false
existingSecret: knowledge-platform-rabbitmq

View File

@ -0,0 +1,15 @@
apiVersion: v2
name: knowledge-platform
description: Enterprise Knowledge Platform public-knowledge deployment
type: application
version: 0.1.0
appVersion: "0.1.0"
# This local dependency is disabled by default. Production deployments should
# use an externally managed RabbitMQ service with an independently owned data
# lifecycle; the subchart exists only for explicit development/demo use.
dependencies:
- name: rabbitmq
version: 0.1.0
repository: file://charts/rabbitmq
condition: rabbitmq.enabled

View File

@ -0,0 +1,6 @@
apiVersion: v2
name: rabbitmq
description: Minimal single-node RabbitMQ dependency for local demo clusters only
type: application
version: 0.1.0
appVersion: "4.1.4"

View File

@ -0,0 +1,11 @@
{{- define "rabbitmq.fullname" -}}
{{- printf "%s-rabbitmq" .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "rabbitmq.image" -}}
{{- if .Values.image.digest -}}
{{- printf "%s@%s" .Values.image.repository .Values.image.digest -}}
{{- else -}}
{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}}
{{- end -}}
{{- end -}}

View File

@ -0,0 +1,16 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "rabbitmq.fullname" . }}
labels:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
clusterIP: None
ports:
- name: amqp
port: {{ .Values.service.port }}
targetPort: amqp
selector:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}

View File

@ -0,0 +1,8 @@
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "rabbitmq.fullname" . }}
labels:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}
automountServiceAccountToken: false

View File

@ -0,0 +1,109 @@
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "rabbitmq.fullname" . }}
labels:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
serviceName: {{ include "rabbitmq.fullname" . }}
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}
template:
metadata:
labels:
app.kubernetes.io/name: rabbitmq
app.kubernetes.io/instance: {{ .Release.Name }}
spec:
serviceAccountName: {{ include "rabbitmq.fullname" . }}
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 999
runAsGroup: 999
fsGroup: 999
seccompProfile:
type: RuntimeDefault
containers:
- name: rabbitmq
image: {{ include "rabbitmq.image" . | quote }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
ports:
- name: amqp
containerPort: 5672
env:
- name: RABBITMQ_DEFAULT_USER
valueFrom:
secretKeyRef:
name: {{ .Values.existingSecret }}
key: {{ .Values.secretKeys.username }}
- name: RABBITMQ_DEFAULT_PASS
valueFrom:
secretKeyRef:
name: {{ .Values.existingSecret }}
key: {{ .Values.secretKeys.password }}
- name: RABBITMQ_ERLANG_COOKIE
valueFrom:
secretKeyRef:
name: {{ .Values.existingSecret }}
key: {{ .Values.secretKeys.erlangCookie }}
startupProbe:
exec:
command: ["rabbitmq-diagnostics", "-q", "ping"]
periodSeconds: 5
failureThreshold: 24
readinessProbe:
exec:
command: ["rabbitmq-diagnostics", "-q", "check_running"]
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
exec:
command: ["rabbitmq-diagnostics", "-q", "ping"]
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 3
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumeMounts:
- name: data
mountPath: /var/lib/rabbitmq
- name: config
mountPath: /etc/rabbitmq/conf.d
- name: log
mountPath: /var/log/rabbitmq
- name: tmp
mountPath: /tmp
volumes:
- name: config
emptyDir: {}
- name: log
emptyDir: {}
- name: tmp
emptyDir: {}
{{- if not .Values.persistence.enabled }}
- name: data
emptyDir: {}
{{- end }}
{{- if .Values.persistence.enabled }}
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [{{ .Values.persistence.accessMode | quote }}]
{{- if .Values.persistence.storageClass }}
storageClassName: {{ .Values.persistence.storageClass | quote }}
{{- end }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- end }}

View File

@ -0,0 +1,69 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"additionalProperties": false,
"required": ["enabled", "existingSecret", "secretKeys", "image", "service", "persistence", "resources"],
"properties": {
"global": {"type": "object"},
"enabled": {"type": "boolean"},
"existingSecret": {"type": "string", "minLength": 1},
"secretKeys": {
"type": "object",
"additionalProperties": false,
"required": ["username", "password", "erlangCookie"],
"properties": {
"username": {"type": "string", "minLength": 1},
"password": {"type": "string", "minLength": 1},
"erlangCookie": {"type": "string", "minLength": 1}
}
},
"image": {
"type": "object",
"additionalProperties": false,
"required": ["repository", "tag", "digest", "pullPolicy"],
"properties": {
"repository": {"type": "string", "minLength": 1},
"tag": {"type": "string", "minLength": 1, "not": {"enum": ["latest"]}},
"digest": {"type": "string", "pattern": "^$|^sha256:[a-f0-9]{64}$"},
"pullPolicy": {"enum": ["Always", "IfNotPresent", "Never"]}
}
},
"service": {
"type": "object",
"additionalProperties": false,
"required": ["port"],
"properties": {"port": {"type": "integer", "minimum": 1, "maximum": 65535}}
},
"persistence": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "storageClass", "accessMode", "size"],
"properties": {
"enabled": {"type": "boolean"},
"storageClass": {"type": "string"},
"accessMode": {"enum": ["ReadWriteOnce", "ReadWriteMany"]},
"size": {"type": "string", "minLength": 2}
}
},
"resources": {
"type": "object",
"additionalProperties": false,
"required": ["requests", "limits"],
"properties": {
"requests": {"$ref": "#/definitions/resourcePair"},
"limits": {"$ref": "#/definitions/resourcePair"}
}
}
},
"definitions": {
"resourcePair": {
"type": "object",
"additionalProperties": false,
"required": ["cpu", "memory"],
"properties": {
"cpu": {"type": ["string", "number"]},
"memory": {"type": "string", "minLength": 2}
}
}
}
}

View File

@ -0,0 +1,25 @@
enabled: false
existingSecret: knowledge-platform-rabbitmq
secretKeys:
username: username
password: password
erlangCookie: erlang-cookie
image:
repository: rabbitmq
tag: 4.1.4-alpine
digest: sha256:b736d649308e1b3e1a116c3f36986b605ee3d03e88f10166be2900083d2e63f2
pullPolicy: IfNotPresent
service:
port: 5672
persistence:
enabled: true
storageClass: ""
accessMode: ReadWriteOnce
size: 8Gi
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi

View File

@ -0,0 +1,17 @@
Enterprise Knowledge Platform {{ .Chart.AppVersion }} is managed by Helm release {{ .Release.Name }}.
Runtime credentials are read only from Secret {{ .Values.existingSecret }}.
{{- if .Values.ingress.enabled }}
Admin/API: https://{{ .Values.ingress.api.host }}{{ .Values.ingress.api.path }}
MCP: https://{{ .Values.ingress.mcp.host }}{{ .Values.ingress.mcp.path }}
{{- else if .Values.edge.enabled }}
Host: https://{{ .Values.edge.host }}
Edge Service: {{ include "knowledge-platform.fullname" . }}-edge
NodePort: {{ if gt (int .Values.edge.service.nodePort) 0 }}{{ .Values.edge.service.nodePort }}{{ else }}allocated by Kubernetes{{ end }}
Routes: /mcp -> MCP; all other paths -> Admin/API
{{- else }}
Ingress and edge are disabled. Enable one entry mode before remote use.
{{- end }}
Inspect rollout state without printing environment variables:
kubectl -n {{ .Release.Namespace }} get deploy,job,pod

View File

@ -0,0 +1,126 @@
{{/* Stable resource name helpers keep every workload under one release identity. */}}
{{- define "knowledge-platform.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "knowledge-platform.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := include "knowledge-platform.name" . -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "knowledge-platform.labels" -}}
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | quote }}
app.kubernetes.io/name: {{ include "knowledge-platform.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{- define "knowledge-platform.selectorLabels" -}}
app.kubernetes.io/name: {{ include "knowledge-platform.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{- define "knowledge-platform.serviceAccountName" -}}
{{- if .Values.serviceAccount.create -}}
{{- default (include "knowledge-platform.fullname" .) .Values.serviceAccount.name -}}
{{- else -}}
{{- required "serviceAccount.name is required when serviceAccount.create=false" .Values.serviceAccount.name -}}
{{- end -}}
{{- end -}}
{{/* A supplied digest takes precedence over the immutable release tag. */}}
{{- define "knowledge-platform.image" -}}
{{- if .digest -}}
{{- printf "%s@%s" .repository .digest -}}
{{- else -}}
{{- printf "%s:%s" .repository .tag -}}
{{- end -}}
{{- end -}}
{{- define "knowledge-platform.configMapName" -}}
{{- printf "%s-runtime" (include "knowledge-platform.fullname" .) -}}
{{- end -}}
{{/* Cross-field rules complement values.schema.json's structural checks. */}}
{{- define "knowledge-platform.validateValues" -}}
{{- if lt (float64 .Values.config.ingestion.retryBackoffMaxSeconds) (float64 .Values.config.ingestion.retryBackoffSeconds) -}}
{{- fail "config.ingestion.retryBackoffMaxSeconds must be >= retryBackoffSeconds" -}}
{{- end -}}
{{- if lt (float64 .Values.config.ingestion.batchMaxTotalBytes) (float64 .Values.config.ingestion.maxUploadBytes) -}}
{{- fail "config.ingestion.batchMaxTotalBytes must be >= maxUploadBytes" -}}
{{- end -}}
{{- if ge (float64 .Values.config.chunking.overlapTokens) (float64 .Values.config.chunking.maxTokens) -}}
{{- fail "config.chunking.overlapTokens must be < maxTokens" -}}
{{- end -}}
{{- if gt (float64 .Values.config.retrieval.searchTopK) (float64 .Values.config.retrieval.hybridTopN) -}}
{{- fail "config.retrieval.searchTopK must be <= hybridTopN" -}}
{{- end -}}
{{- if eq .Values.config.parser.provider "mineru" -}}
{{- $mineruEndpoint := required "config.parser.mineruEndpoint is required when provider=mineru" .Values.config.parser.mineruEndpoint -}}
{{- if le (float64 .Values.config.ingestion.workerLeaseSeconds) (float64 .Values.config.parser.mineruTimeoutSeconds) -}}
{{- fail "config.ingestion.workerLeaseSeconds must exceed config.parser.mineruTimeoutSeconds" -}}
{{- end -}}
{{- end -}}
{{- if and .Values.ingress.enabled .Values.edge.enabled -}}
{{- fail "ingress.enabled and edge.enabled cannot both be true" -}}
{{- end -}}
{{- if .Values.podDisruptionBudget.enabled -}}
{{- if ge (int .Values.podDisruptionBudget.apiMinAvailable) (int .Values.api.replicaCount) -}}
{{- fail "podDisruptionBudget.apiMinAvailable must be < api.replicaCount" -}}
{{- end -}}
{{- if ge (int .Values.podDisruptionBudget.mcpMinAvailable) (int .Values.mcp.replicaCount) -}}
{{- fail "podDisruptionBudget.mcpMinAvailable must be < mcp.replicaCount" -}}
{{- end -}}
{{- if and .Values.edge.enabled (ge (int .Values.podDisruptionBudget.edgeMinAvailable) (int .Values.edge.replicaCount)) -}}
{{- fail "podDisruptionBudget.edgeMinAvailable must be < edge.replicaCount" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- define "knowledge-platform.podSecurityContext" -}}
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
{{- end -}}
{{- define "knowledge-platform.containerSecurityContext" -}}
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
capabilities:
drop:
- ALL
{{- end -}}
{{- define "knowledge-platform.envFrom" -}}
- configMapRef:
name: {{ include "knowledge-platform.configMapName" . }}
- secretRef:
name: {{ .Values.existingSecret }}
{{- end -}}
{{- define "knowledge-platform.tmpVolume" -}}
- name: tmp
emptyDir:
sizeLimit: 1Gi
{{- end -}}
{{- define "knowledge-platform.tmpMount" -}}
- name: tmp
mountPath: /tmp
{{- end -}}

View File

@ -0,0 +1,80 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "knowledge-platform.fullname" . }}-api
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: api
spec:
replicas: {{ .Values.api.replicaCount }}
minReadySeconds: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: api
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: api
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
{{- if .Values.observability.metrics.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: "8000"
prometheus.io/path: {{ .Values.observability.metrics.path | quote }}
{{- end }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
automountServiceAccountToken: false
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: api
image: {{ include "knowledge-platform.image" .Values.images.api | quote }}
imagePullPolicy: {{ .Values.images.api.pullPolicy }}
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
ports:
- name: http
containerPort: 8000
protocol: TCP
envFrom:
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
startupProbe:
httpGet:
path: /healthz
port: http
{{- toYaml .Values.api.probes.startup | nindent 12 }}
readinessProbe:
httpGet:
path: /healthz
port: http
{{- toYaml .Values.api.probes.readiness | nindent 12 }}
livenessProbe:
httpGet:
path: /healthz
port: http
{{- toYaml .Values.api.probes.liveness | nindent 12 }}
resources:
{{- toYaml .Values.api.resources | nindent 12 }}
volumeMounts:
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}

View File

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "knowledge-platform.fullname" . }}-api
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: api
spec:
type: {{ .Values.api.service.type }}
ports:
- name: http
port: {{ .Values.api.service.port }}
targetPort: http
protocol: TCP
selector:
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: api

View File

@ -0,0 +1,84 @@
{{- include "knowledge-platform.validateValues" . }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "knowledge-platform.configMapName" . }}
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
data:
# Only the explicitly enumerated non-sensitive settings are rendered here.
APP_ENV: "production"
APP_NAME: {{ .Values.config.appName | quote }}
LOG_LEVEL: {{ .Values.config.logLevel | quote }}
API_HOST: "0.0.0.0"
API_PORT: "8000"
API_PREFIX: "/api/v1"
DATABASE_POOL_SIZE: {{ .Values.config.database.poolSize | quote }}
DATABASE_MAX_OVERFLOW: {{ .Values.config.database.maxOverflow | quote }}
DATABASE_POOL_RECYCLE_SECONDS: {{ .Values.config.database.poolRecycleSeconds | quote }}
DATABASE_POOL_TIMEOUT_SECONDS: {{ .Values.config.database.poolTimeoutSeconds | quote }}
DATABASE_CONNECT_TIMEOUT_SECONDS: {{ .Values.config.database.connectTimeoutSeconds | quote }}
INGESTION_MAX_ATTEMPTS: {{ .Values.config.ingestion.maxAttempts | quote }}
INGESTION_RETRY_BACKOFF_SECONDS: {{ .Values.config.ingestion.retryBackoffSeconds | quote }}
INGESTION_RETRY_BACKOFF_MAX_SECONDS: {{ .Values.config.ingestion.retryBackoffMaxSeconds | quote }}
INGESTION_WORKER_LEASE_SECONDS: {{ .Values.config.ingestion.workerLeaseSeconds | quote }}
INGESTION_RECOVERY_INTERVAL_SECONDS: {{ .Values.config.ingestion.recoveryIntervalSeconds | quote }}
COMPILATION_MAX_ATTEMPTS: {{ .Values.config.compilation.maxAttempts | quote }}
COMPILATION_RETRY_BACKOFF_SECONDS: {{ .Values.config.compilation.retryBackoffSeconds | quote }}
COMPILATION_RETRY_BACKOFF_MAX_SECONDS: {{ .Values.config.compilation.retryBackoffMaxSeconds | quote }}
COMPILATION_WORKER_LEASE_SECONDS: {{ .Values.config.compilation.workerLeaseSeconds | quote }}
COMPILATION_RECOVERY_INTERVAL_SECONDS: {{ .Values.config.compilation.recoveryIntervalSeconds | quote }}
BATCH_MAX_FILES: {{ .Values.config.ingestion.batchMaxFiles | quote }}
BATCH_MAX_TOTAL_BYTES: {{ printf "%.0f" .Values.config.ingestion.batchMaxTotalBytes | quote }}
MAX_UPLOAD_BYTES: {{ printf "%.0f" .Values.config.ingestion.maxUploadBytes | quote }}
OBJECT_STORAGE_PROVIDER: {{ .Values.config.objectStorage.provider | quote }}
BOS_ENDPOINT: {{ .Values.config.objectStorage.endpoint | quote }}
BOS_BUCKET: {{ .Values.config.objectStorage.bucket | quote }}
BOS_KEY_PREFIX: {{ .Values.config.objectStorage.keyPrefix | quote }}
BOS_TIMEOUT_SECONDS: {{ .Values.config.objectStorage.timeoutSeconds | quote }}
PARSER_PROVIDER: {{ .Values.config.parser.provider | quote }}
MINERU_ENDPOINT: {{ .Values.config.parser.mineruEndpoint | quote }}
MINERU_BACKEND: {{ .Values.config.parser.mineruBackend | quote }}
MINERU_PARSE_METHOD: {{ .Values.config.parser.mineruParseMethod | quote }}
MINERU_LANGUAGE: {{ .Values.config.parser.mineruLanguage | quote }}
MINERU_EFFORT: {{ .Values.config.parser.mineruEffort | quote }}
MINERU_TIMEOUT_SECONDS: {{ .Values.config.parser.mineruTimeoutSeconds | quote }}
MINERU_LOW_OCR_CONFIDENCE_THRESHOLD: {{ .Values.config.parser.lowOcrConfidenceThreshold | quote }}
CHUNK_MAX_TOKENS: {{ .Values.config.chunking.maxTokens | quote }}
CHUNK_OVERLAP_TOKENS: {{ .Values.config.chunking.overlapTokens | quote }}
CHUNK_MAX_CHARACTERS: {{ .Values.config.chunking.maxCharacters | quote }}
EMBEDDING_PROVIDER: {{ .Values.config.embedding.provider | quote }}
EMBEDDING_MODEL: {{ .Values.config.embedding.model | quote }}
EMBEDDING_ENDPOINT: {{ .Values.config.embedding.endpoint | quote }}
EMBEDDING_TIMEOUT_SECONDS: {{ .Values.config.embedding.timeoutSeconds | quote }}
VECTOR_STORE_PROVIDER: {{ .Values.config.vectorStore.provider | quote }}
OCEANBASE_VECTOR_URI: {{ .Values.config.vectorStore.uri | quote }}
OCEANBASE_VECTOR_USER: {{ .Values.config.vectorStore.user | quote }}
OCEANBASE_VECTOR_DATABASE: {{ .Values.config.vectorStore.database | quote }}
OCEANBASE_VECTOR_TABLE: {{ .Values.config.vectorStore.table | quote }}
OCEANBASE_VECTOR_INDEX: {{ .Values.config.vectorStore.index | quote }}
OCEANBASE_VECTOR_BATCH_SIZE: {{ .Values.config.vectorStore.batchSize | quote }}
OCEANBASE_VECTOR_TIMEOUT_SECONDS: {{ .Values.config.vectorStore.timeoutSeconds | quote }}
INDEX_PROFILE_ID: {{ .Values.config.retrieval.indexProfileId | quote }}
DENSE_TOP_N: {{ .Values.config.retrieval.denseTopN | quote }}
SPARSE_TOP_N: {{ .Values.config.retrieval.sparseTopN | quote }}
HYBRID_TOP_N: {{ .Values.config.retrieval.hybridTopN | quote }}
HYBRID_RRF_K: {{ .Values.config.retrieval.hybridRrfK | quote }}
HYBRID_DENSE_WEIGHT: {{ .Values.config.retrieval.hybridDenseWeight | quote }}
HYBRID_SPARSE_WEIGHT: {{ .Values.config.retrieval.hybridSparseWeight | quote }}
SEARCH_TOP_K: {{ .Values.config.retrieval.searchTopK | quote }}
RERANKER_PROVIDER: {{ .Values.config.reranker.provider | quote }}
RERANKER_MODEL: {{ .Values.config.reranker.model | quote }}
RERANKER_ENDPOINT: {{ .Values.config.reranker.endpoint | quote }}
RERANKER_TIMEOUT_SECONDS: {{ .Values.config.reranker.timeoutSeconds | quote }}
MCP_HOST: "0.0.0.0"
MCP_PORT: "8766"
MCP_PATH: "/mcp"
MCP_PUBLIC_URL: {{ .Values.config.mcp.publicUrl | quote }}
MCP_ISSUER_URL: {{ .Values.config.mcp.issuerUrl | quote }}
MCP_CLIENT_ID: {{ .Values.config.mcp.clientId | quote }}
MCP_SERVICE_IDENTITY: {{ .Values.config.mcp.serviceIdentity | quote }}
MCP_REQUIRED_SCOPE: {{ .Values.config.mcp.requiredScope | quote }}
MCP_ALLOWED_HOSTS: {{ .Values.config.mcp.allowedHosts | toJson | quote }}
MCP_ALLOWED_ORIGINS: {{ .Values.config.mcp.allowedOrigins | toJson | quote }}
MCP_MAX_REQUEST_BODY_BYTES: {{ printf "%.0f" .Values.config.mcp.maxRequestBodyBytes | quote }}

View File

@ -0,0 +1,117 @@
{{- if .Values.edge.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "knowledge-platform.fullname" . }}-edge
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: edge
data:
# The edge is intentionally limited to path routing. Authentication and all
# business behavior remain in the API and MCP applications behind it.
nginx.conf: |
pid /tmp/nginx.pid;
error_log /dev/stderr info;
events {
worker_connections 1024;
}
http {
client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;
server_tokens off;
client_max_body_size {{ .Values.edge.clientMaxBodySize }};
map $http_x_request_id $correlation_id {
default $http_x_request_id;
"" $request_id;
}
map $http_x_forwarded_proto $forwarded_proto {
default $http_x_forwarded_proto;
"" $scheme;
}
log_format edge_json escape=json
'{"timestamp":"$time_iso8601","event":"edge_access",'
'"request_id":"$correlation_id","method":"$request_method",'
'"path":"$uri","status":$status,"duration_seconds":$request_time, '
'"upstream_status":"$upstream_status"}';
access_log /dev/stdout edge_json;
upstream api_backend {
server {{ include "knowledge-platform.fullname" . }}-api:{{ .Values.api.service.port }};
keepalive 32;
}
upstream mcp_backend {
server {{ include "knowledge-platform.fullname" . }}-mcp:{{ .Values.mcp.service.port }};
keepalive 32;
}
server {
listen 8080;
location = /edge-healthz {
access_log off;
default_type text/plain;
return 200 "ok\n";
}
# The public health endpoint follows API readiness so an external load
# balancer cannot direct traffic to an edge with no usable API backend.
location = /healthz {
proxy_pass http://api_backend;
proxy_set_header Host $host;
proxy_set_header X-Request-ID $correlation_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $forwarded_proto;
}
# Streamable HTTP may keep a request open, so buffering is disabled only
# for the MCP route while the gateway keeps ownership of authentication.
location = /mcp {
proxy_pass http://mcp_backend;
proxy_http_version 1.1;
proxy_buffering off;
proxy_request_buffering off;
proxy_read_timeout {{ .Values.edge.proxyReadTimeoutSeconds }}s;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Request-ID $correlation_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $forwarded_proto;
}
location ^~ /mcp/ {
proxy_pass http://mcp_backend;
proxy_http_version 1.1;
proxy_buffering off;
proxy_request_buffering off;
proxy_read_timeout {{ .Values.edge.proxyReadTimeoutSeconds }}s;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Request-ID $correlation_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $forwarded_proto;
}
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Request-ID $correlation_id;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $forwarded_proto;
}
}
}
{{- end }}

View File

@ -0,0 +1,81 @@
{{- if .Values.edge.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "knowledge-platform.fullname" . }}-edge
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: edge
spec:
replicas: {{ .Values.edge.replicaCount }}
minReadySeconds: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: edge
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: edge
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
checksum/edge-config: {{ include (print $.Template.BasePath "/edge-configmap.yaml") . | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
automountServiceAccountToken: false
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: edge
image: {{ include "knowledge-platform.image" .Values.images.edge | quote }}
imagePullPolicy: {{ .Values.images.edge.pullPolicy }}
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
ports:
- name: http
containerPort: 8080
protocol: TCP
startupProbe:
httpGet:
path: /edge-healthz
port: http
{{- toYaml .Values.edge.probes.startup | nindent 12 }}
readinessProbe:
httpGet:
path: /healthz
port: http
{{- toYaml .Values.edge.probes.readiness | nindent 12 }}
livenessProbe:
httpGet:
path: /edge-healthz
port: http
{{- toYaml .Values.edge.probes.liveness | nindent 12 }}
resources:
{{- toYaml .Values.edge.resources | nindent 12 }}
volumeMounts:
- name: edge-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
readOnly: true
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
- name: edge-config
configMap:
name: {{ include "knowledge-platform.fullname" . }}-edge
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
{{- end }}

View File

@ -0,0 +1,22 @@
{{- if .Values.edge.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "knowledge-platform.fullname" . }}-edge
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: edge
spec:
type: {{ .Values.edge.service.type }}
ports:
- name: http
port: {{ .Values.edge.service.port }}
targetPort: http
protocol: TCP
{{- if gt (int .Values.edge.service.nodePort) 0 }}
nodePort: {{ .Values.edge.service.nodePort }}
{{- end }}
selector:
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: edge
{{- end }}

View File

@ -0,0 +1,42 @@
{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "knowledge-platform.fullname" . }}
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
ingressClassName: {{ .Values.ingress.className }}
tls:
- hosts:
- {{ .Values.ingress.api.host | quote }}
secretName: {{ .Values.ingress.api.tlsSecretName }}
- hosts:
- {{ .Values.ingress.mcp.host | quote }}
secretName: {{ .Values.ingress.mcp.tlsSecretName }}
rules:
- host: {{ .Values.ingress.api.host | quote }}
http:
paths:
- path: {{ .Values.ingress.api.path }}
pathType: Prefix
backend:
service:
name: {{ include "knowledge-platform.fullname" . }}-api
port:
number: {{ .Values.api.service.port }}
- host: {{ .Values.ingress.mcp.host | quote }}
http:
paths:
- path: {{ .Values.ingress.mcp.path }}
pathType: Prefix
backend:
service:
name: {{ include "knowledge-platform.fullname" . }}-mcp
port:
number: {{ .Values.mcp.service.port }}
{{- end }}

View File

@ -0,0 +1,72 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "knowledge-platform.fullname" . }}-mcp
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: mcp
spec:
replicas: {{ .Values.mcp.replicaCount }}
minReadySeconds: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: mcp
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: mcp
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
automountServiceAccountToken: false
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: mcp
image: {{ include "knowledge-platform.image" .Values.images.mcp | quote }}
imagePullPolicy: {{ .Values.images.mcp.pullPolicy }}
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
ports:
- name: mcp
containerPort: 8766
protocol: TCP
envFrom:
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
startupProbe:
tcpSocket:
port: mcp
{{- toYaml .Values.mcp.probes.startup | nindent 12 }}
readinessProbe:
tcpSocket:
port: mcp
{{- toYaml .Values.mcp.probes.readiness | nindent 12 }}
livenessProbe:
tcpSocket:
port: mcp
{{- toYaml .Values.mcp.probes.liveness | nindent 12 }}
resources:
{{- toYaml .Values.mcp.resources | nindent 12 }}
volumeMounts:
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}

View File

@ -0,0 +1,17 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "knowledge-platform.fullname" . }}-mcp
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: mcp
spec:
type: {{ .Values.mcp.service.type }}
ports:
- name: mcp
port: {{ .Values.mcp.service.port }}
targetPort: mcp
protocol: TCP
selector:
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
app.kubernetes.io/component: mcp

View File

@ -0,0 +1,59 @@
{{- if .Values.migration.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "knowledge-platform.fullname" . }}-migration
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: migration
annotations:
# Helm waits for pre-install/pre-upgrade hooks. A failed migration blocks
# the application rollout, and the previous Job remains until the next run.
helm.sh/hook: pre-install,pre-upgrade
helm.sh/hook-weight: "-10"
helm.sh/hook-delete-policy: before-hook-creation
spec:
backoffLimit: {{ .Values.migration.backoffLimit }}
activeDeadlineSeconds: {{ .Values.migration.activeDeadlineSeconds }}
ttlSecondsAfterFinished: {{ .Values.migration.ttlSecondsAfterFinished }}
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: migration
spec:
restartPolicy: Never
# Pre-install hooks run before the release ServiceAccount exists. The
# namespace default account is used with token mounting disabled; the
# migration needs no Kubernetes API permissions.
serviceAccountName: default
automountServiceAccountToken: false
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: migration
image: {{ include "knowledge-platform.image" .Values.images.api | quote }}
imagePullPolicy: {{ .Values.images.api.pullPolicy }}
command: ["alembic"]
args: ["upgrade", "head"]
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
env:
- name: APP_ENV
value: production
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ .Values.existingSecret }}
key: DATABASE_URL
resources:
{{- toYaml .Values.migration.resources | nindent 12 }}
volumeMounts:
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
{{- end }}

View File

@ -0,0 +1,49 @@
{{- if .Values.podDisruptionBudget.enabled }}
# DEPLOY-003 protects both user-facing paths during voluntary disruptions. PDBs
# do not cover involuntary failures, which remain handled by replica recovery.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include "knowledge-platform.fullname" . }}-api
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: api
spec:
minAvailable: {{ .Values.podDisruptionBudget.apiMinAvailable }}
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: api
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include "knowledge-platform.fullname" . }}-mcp
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: mcp
spec:
minAvailable: {{ .Values.podDisruptionBudget.mcpMinAvailable }}
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: mcp
{{- if .Values.edge.enabled }}
---
# The edge proxy is the single external path on clusters without Ingress, so it
# receives the same voluntary-disruption protection as API and MCP.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: {{ include "knowledge-platform.fullname" . }}-edge
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: edge
spec:
minAvailable: {{ .Values.podDisruptionBudget.edgeMinAvailable }}
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: edge
{{- end }}
{{- end }}

View File

@ -0,0 +1,71 @@
{{- if .Values.scheduler.enabled }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "knowledge-platform.fullname" . }}-scheduler
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: scheduler
spec:
# A single Celery Beat owns periodic durable-Job recovery. Recreate prevents
# two schedulers from overlapping during upgrades.
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: scheduler
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: scheduler
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
automountServiceAccountToken: false
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: scheduler
image: {{ include "knowledge-platform.image" .Values.images.worker | quote }}
imagePullPolicy: {{ .Values.images.worker.pullPolicy }}
workingDir: /tmp
command: ["celery"]
args:
- --app=ekp_api.worker:celery_app
- beat
- --loglevel=INFO
- --schedule=/tmp/celerybeat-schedule
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
envFrom:
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
startupProbe:
exec:
command: ["python", "-c", "import os; os.kill(1, 0)"]
{{- toYaml .Values.scheduler.probes.startup | nindent 12 }}
livenessProbe:
exec:
command: ["python", "-c", "import os; os.kill(1, 0)"]
{{- toYaml .Values.scheduler.probes.liveness | nindent 12 }}
resources:
{{- toYaml .Values.scheduler.resources | nindent 12 }}
volumeMounts:
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
{{- end }}

View File

@ -0,0 +1,13 @@
{{- if .Values.serviceAccount.create }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "knowledge-platform.serviceAccountName" . }}
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
automountServiceAccountToken: false
{{- end }}

View File

@ -0,0 +1,29 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "knowledge-platform.fullname" . }}-preflight-references
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
annotations:
# The Make preflight renders this test hook to discover only resource
# names. It never renders or reads Secret data.
helm.sh/hook: test
data:
runtimeSecret: {{ .Values.existingSecret | quote }}
ingressEnabled: {{ .Values.ingress.enabled | quote }}
edgeEnabled: {{ .Values.edge.enabled | quote }}
edgeHost: {{ .Values.edge.host | quote }}
edgeService: {{ printf "%s-edge" (include "knowledge-platform.fullname" .) | quote }}
edgeDeployment: {{ printf "%s-edge" (include "knowledge-platform.fullname" .) | quote }}
edgeNodePort: {{ .Values.edge.service.nodePort | quote }}
apiHost: {{ .Values.ingress.api.host | quote }}
mcpHost: {{ .Values.ingress.mcp.host | quote }}
apiDeployment: {{ printf "%s-api" (include "knowledge-platform.fullname" .) | quote }}
mcpDeployment: {{ printf "%s-mcp" (include "knowledge-platform.fullname" .) | quote }}
workerDeployment: {{ printf "%s-worker" (include "knowledge-platform.fullname" .) | quote }}
schedulerDeployment: {{ printf "%s-scheduler" (include "knowledge-platform.fullname" .) | quote }}
metricsPath: {{ .Values.observability.metrics.path | quote }}
apiTlsSecret: {{ .Values.ingress.api.tlsSecretName | quote }}
mcpTlsSecret: {{ .Values.ingress.mcp.tlsSecretName | quote }}
rabbitmqEnabled: {{ .Values.rabbitmq.enabled | quote }}
rabbitmqSecret: {{ .Values.rabbitmq.existingSecret | quote }}

View File

@ -0,0 +1,71 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "knowledge-platform.fullname" . }}-worker
labels:
{{- include "knowledge-platform.labels" . | nindent 4 }}
app.kubernetes.io/component: worker
spec:
replicas: {{ .Values.worker.replicaCount }}
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: worker
template:
metadata:
labels:
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: worker
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
annotations:
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
{{- with .Values.podAnnotations }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
automountServiceAccountToken: false
terminationGracePeriodSeconds: {{ .Values.worker.terminationGracePeriodSeconds }}
securityContext:
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: worker
image: {{ include "knowledge-platform.image" .Values.images.worker | quote }}
imagePullPolicy: {{ .Values.images.worker.pullPolicy }}
workingDir: /tmp
command: ["celery"]
args:
- --app=ekp_api.worker:celery_app
- worker
- --loglevel=INFO
- --queues=ingestion,compilation
securityContext:
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
envFrom:
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
startupProbe:
exec:
command: ["python", "-c", "import os; os.kill(1, 0)"]
{{- toYaml .Values.worker.probes.startup | nindent 12 }}
livenessProbe:
exec:
command: ["python", "-c", "import os; os.kill(1, 0)"]
{{- toYaml .Values.worker.probes.liveness | nindent 12 }}
resources:
{{- toYaml .Values.worker.resources | nindent 12 }}
volumeMounts:
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
volumes:
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}

View File

@ -0,0 +1,450 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"title": "Enterprise Knowledge Platform Helm values",
"type": "object",
"additionalProperties": false,
"required": [
"existingSecret",
"images",
"serviceAccount",
"podDisruptionBudget",
"observability",
"api",
"mcp",
"edge",
"worker",
"scheduler",
"migration",
"ingress",
"config",
"rabbitmq"
],
"properties": {
"nameOverride": {"type": "string"},
"fullnameOverride": {"type": "string"},
"existingSecret": {"$ref": "#/definitions/kubernetesName"},
"imagePullSecrets": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name"],
"properties": {"name": {"$ref": "#/definitions/kubernetesName"}}
}
},
"images": {
"type": "object",
"additionalProperties": false,
"required": ["api", "mcp", "worker", "edge"],
"properties": {
"api": {"$ref": "#/definitions/image"},
"mcp": {"$ref": "#/definitions/image"},
"worker": {"$ref": "#/definitions/image"},
"edge": {"$ref": "#/definitions/image"}
}
},
"serviceAccount": {
"type": "object",
"additionalProperties": false,
"required": ["create", "name", "annotations"],
"properties": {
"create": {"type": "boolean"},
"name": {"type": "string"},
"annotations": {"$ref": "#/definitions/stringMap"}
}
},
"podAnnotations": {"$ref": "#/definitions/stringMap"},
"podLabels": {"$ref": "#/definitions/stringMap"},
"podDisruptionBudget": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "apiMinAvailable", "mcpMinAvailable", "edgeMinAvailable"],
"properties": {
"enabled": {"type": "boolean"},
"apiMinAvailable": {"type": "integer", "minimum": 1},
"mcpMinAvailable": {"type": "integer", "minimum": 1},
"edgeMinAvailable": {"type": "integer", "minimum": 1}
}
},
"observability": {
"type": "object",
"additionalProperties": false,
"required": ["metrics"],
"properties": {
"metrics": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "path"],
"properties": {
"enabled": {"type": "boolean"},
"path": {"const": "/metrics"}
}
}
}
},
"api": {"$ref": "#/definitions/httpWorkload"},
"mcp": {"$ref": "#/definitions/httpWorkload"},
"edge": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "host", "replicaCount", "service", "clientMaxBodySize", "proxyReadTimeoutSeconds", "resources", "probes"],
"properties": {
"enabled": {"type": "boolean"},
"host": {"type": "string", "minLength": 1},
"replicaCount": {"type": "integer", "minimum": 2, "maximum": 100},
"service": {"$ref": "#/definitions/edgeService"},
"clientMaxBodySize": {"type": "string", "pattern": "^[1-9][0-9]*[kKmMgG]$"},
"proxyReadTimeoutSeconds": {"type": "integer", "minimum": 1, "maximum": 86400},
"resources": {"$ref": "#/definitions/resources"},
"probes": {"$ref": "#/definitions/httpProbes"}
}
},
"worker": {
"type": "object",
"additionalProperties": false,
"required": ["replicaCount", "terminationGracePeriodSeconds", "resources", "probes"],
"properties": {
"replicaCount": {"type": "integer", "minimum": 1, "maximum": 100},
"terminationGracePeriodSeconds": {"type": "integer", "minimum": 30, "maximum": 3600},
"resources": {"$ref": "#/definitions/resources"},
"probes": {"$ref": "#/definitions/processProbes"}
}
},
"scheduler": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "resources", "probes"],
"properties": {
"enabled": {"type": "boolean"},
"resources": {"$ref": "#/definitions/resources"},
"probes": {"$ref": "#/definitions/processProbes"}
}
},
"migration": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "activeDeadlineSeconds", "backoffLimit", "ttlSecondsAfterFinished", "resources"],
"properties": {
"enabled": {"type": "boolean"},
"activeDeadlineSeconds": {"type": "integer", "minimum": 60, "maximum": 3600},
"backoffLimit": {"type": "integer", "minimum": 0, "maximum": 3},
"ttlSecondsAfterFinished": {"type": "integer", "minimum": 0},
"resources": {"$ref": "#/definitions/resources"}
}
},
"ingress": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "className", "annotations", "api", "mcp"],
"properties": {
"enabled": {"type": "boolean"},
"className": {"type": "string", "minLength": 1},
"annotations": {"$ref": "#/definitions/stringMap"},
"api": {"$ref": "#/definitions/ingressRoute"},
"mcp": {"$ref": "#/definitions/ingressRoute"}
}
},
"config": {
"type": "object",
"additionalProperties": false,
"required": ["appName", "logLevel", "database", "ingestion", "compilation", "objectStorage", "parser", "chunking", "embedding", "vectorStore", "retrieval", "reranker", "mcp"],
"properties": {
"appName": {"type": "string", "minLength": 1},
"logLevel": {"enum": ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]},
"database": {"$ref": "#/definitions/databaseConfig"},
"ingestion": {"$ref": "#/definitions/ingestionConfig"},
"compilation": {"$ref": "#/definitions/compilationConfig"},
"objectStorage": {"$ref": "#/definitions/objectStorageConfig"},
"parser": {"$ref": "#/definitions/parserConfig"},
"chunking": {"$ref": "#/definitions/chunkingConfig"},
"embedding": {"$ref": "#/definitions/embeddingConfig"},
"vectorStore": {"$ref": "#/definitions/vectorStoreConfig"},
"retrieval": {"$ref": "#/definitions/retrievalConfig"},
"reranker": {"$ref": "#/definitions/rerankerConfig"},
"mcp": {"$ref": "#/definitions/mcpConfig"}
}
},
"rabbitmq": {
"type": "object",
"required": ["enabled", "existingSecret"],
"properties": {
"enabled": {"type": "boolean"},
"existingSecret": {"$ref": "#/definitions/kubernetesName"}
}
}
},
"definitions": {
"kubernetesName": {
"type": "string",
"minLength": 1,
"maxLength": 253,
"pattern": "^[a-z0-9]([-a-z0-9.]*[a-z0-9])?$"
},
"stringMap": {
"type": "object",
"additionalProperties": {"type": "string"}
},
"image": {
"type": "object",
"additionalProperties": false,
"required": ["repository", "tag", "digest", "pullPolicy"],
"properties": {
"repository": {"type": "string", "minLength": 1},
"tag": {"type": "string", "minLength": 1, "not": {"enum": ["latest"]}},
"digest": {"type": "string", "pattern": "^$|^sha256:[a-f0-9]{64}$"},
"pullPolicy": {"enum": ["Always", "IfNotPresent", "Never"]}
}
},
"probeTiming": {
"type": "object",
"additionalProperties": false,
"required": ["failureThreshold", "periodSeconds"],
"properties": {
"failureThreshold": {"type": "integer", "minimum": 1},
"periodSeconds": {"type": "integer", "minimum": 1},
"timeoutSeconds": {"type": "integer", "minimum": 1}
}
},
"httpProbes": {
"type": "object",
"additionalProperties": false,
"required": ["startup", "readiness", "liveness"],
"properties": {
"startup": {"$ref": "#/definitions/probeTiming"},
"readiness": {"$ref": "#/definitions/probeTiming"},
"liveness": {"$ref": "#/definitions/probeTiming"}
}
},
"processProbes": {
"type": "object",
"additionalProperties": false,
"required": ["startup", "liveness"],
"properties": {
"startup": {"$ref": "#/definitions/probeTiming"},
"liveness": {"$ref": "#/definitions/probeTiming"}
}
},
"resources": {
"type": "object",
"additionalProperties": false,
"required": ["requests", "limits"],
"properties": {
"requests": {"$ref": "#/definitions/resourcePair"},
"limits": {"$ref": "#/definitions/resourcePair"}
}
},
"resourcePair": {
"type": "object",
"additionalProperties": false,
"required": ["cpu", "memory"],
"properties": {
"cpu": {"type": ["string", "number"]},
"memory": {"type": "string", "minLength": 2}
}
},
"httpWorkload": {
"type": "object",
"additionalProperties": false,
"required": ["replicaCount", "service", "resources", "probes"],
"properties": {
"replicaCount": {"type": "integer", "minimum": 2, "maximum": 100},
"service": {
"type": "object",
"additionalProperties": false,
"required": ["type", "port"],
"properties": {
"type": {"enum": ["ClusterIP"]},
"port": {"type": "integer", "minimum": 1, "maximum": 65535}
}
},
"resources": {"$ref": "#/definitions/resources"},
"probes": {"$ref": "#/definitions/httpProbes"}
}
},
"edgeService": {
"type": "object",
"additionalProperties": false,
"required": ["type", "port", "nodePort"],
"properties": {
"type": {"enum": ["NodePort"]},
"port": {"type": "integer", "minimum": 1, "maximum": 65535},
"nodePort": {
"oneOf": [
{"const": 0},
{"type": "integer", "minimum": 30000, "maximum": 32767}
]
}
}
},
"ingressRoute": {
"type": "object",
"additionalProperties": false,
"required": ["host", "path", "tlsSecretName"],
"properties": {
"host": {"type": "string", "minLength": 1},
"path": {"type": "string", "pattern": "^/"},
"tlsSecretName": {"$ref": "#/definitions/kubernetesName"}
}
},
"databaseConfig": {
"type": "object",
"additionalProperties": false,
"required": ["poolSize", "maxOverflow", "poolRecycleSeconds", "poolTimeoutSeconds", "connectTimeoutSeconds"],
"properties": {
"poolSize": {"type": "integer", "minimum": 1, "maximum": 100},
"maxOverflow": {"type": "integer", "minimum": 0, "maximum": 100},
"poolRecycleSeconds": {"type": "integer", "minimum": 1},
"poolTimeoutSeconds": {"type": "number", "exclusiveMinimum": 0},
"connectTimeoutSeconds": {"type": "integer", "minimum": 1}
}
},
"ingestionConfig": {
"type": "object",
"additionalProperties": false,
"required": ["maxAttempts", "retryBackoffSeconds", "retryBackoffMaxSeconds", "workerLeaseSeconds", "recoveryIntervalSeconds", "batchMaxFiles", "batchMaxTotalBytes", "maxUploadBytes"],
"properties": {
"maxAttempts": {"type": "integer", "minimum": 1, "maximum": 10},
"retryBackoffSeconds": {"type": "integer", "minimum": 1},
"retryBackoffMaxSeconds": {"type": "integer", "minimum": 1},
"workerLeaseSeconds": {"type": "integer", "minimum": 30},
"recoveryIntervalSeconds": {"type": "integer", "minimum": 5},
"batchMaxFiles": {"type": "integer", "minimum": 1, "maximum": 100},
"batchMaxTotalBytes": {"type": "integer", "minimum": 1},
"maxUploadBytes": {"type": "integer", "minimum": 1}
}
},
"compilationConfig": {
"type": "object",
"additionalProperties": false,
"required": ["maxAttempts", "retryBackoffSeconds", "retryBackoffMaxSeconds", "workerLeaseSeconds", "recoveryIntervalSeconds"],
"properties": {
"maxAttempts": {"type": "integer", "minimum": 1, "maximum": 10},
"retryBackoffSeconds": {"type": "integer", "minimum": 1},
"retryBackoffMaxSeconds": {"type": "integer", "minimum": 1},
"workerLeaseSeconds": {"type": "integer", "minimum": 30},
"recoveryIntervalSeconds": {"type": "integer", "minimum": 5}
}
},
"objectStorageConfig": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "endpoint", "bucket", "keyPrefix", "timeoutSeconds"],
"properties": {
"provider": {"const": "bos"},
"endpoint": {"type": "string", "minLength": 1},
"bucket": {"type": "string", "minLength": 1},
"keyPrefix": {"type": "string", "minLength": 1},
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
}
},
"parserConfig": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "mineruEndpoint", "mineruBackend", "mineruParseMethod", "mineruLanguage", "mineruEffort", "mineruTimeoutSeconds", "lowOcrConfidenceThreshold"],
"properties": {
"provider": {"enum": ["lightweight", "mineru"]},
"mineruEndpoint": {"type": "string"},
"mineruBackend": {"enum": ["pipeline", "vlm-engine", "hybrid-engine"]},
"mineruParseMethod": {"enum": ["auto", "txt", "ocr"]},
"mineruLanguage": {"type": "string", "minLength": 1, "maxLength": 32, "pattern": "^[A-Za-z0-9_-]+$"},
"mineruEffort": {"enum": ["medium", "high"]},
"mineruTimeoutSeconds": {"type": "number", "exclusiveMinimum": 0, "maximum": 3600},
"lowOcrConfidenceThreshold": {"type": "number", "minimum": 0, "maximum": 1}
},
"allOf": [
{
"if": {"properties": {"provider": {"const": "mineru"}}},
"then": {
"properties": {
"mineruEndpoint": {"type": "string", "pattern": "^https?://[^?#]+/file_parse$"}
}
}
}
]
},
"chunkingConfig": {
"type": "object",
"additionalProperties": false,
"required": ["maxTokens", "overlapTokens", "maxCharacters"],
"properties": {
"maxTokens": {"type": "integer", "minimum": 1},
"overlapTokens": {"type": "integer", "minimum": 0},
"maxCharacters": {"type": "integer", "minimum": 4}
}
},
"embeddingConfig": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "model", "endpoint", "timeoutSeconds"],
"properties": {
"provider": {"const": "qianfan"},
"model": {"const": "bge-large-zh"},
"endpoint": {"type": "string", "minLength": 1},
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
}
},
"vectorStoreConfig": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "uri", "user", "database", "table", "index", "batchSize", "timeoutSeconds"],
"properties": {
"provider": {"const": "oceanbase"},
"uri": {"type": "string", "minLength": 1},
"user": {"type": "string", "minLength": 1},
"database": {"type": "string", "minLength": 1},
"table": {"type": "string", "minLength": 1},
"index": {"type": "string", "minLength": 1},
"batchSize": {"type": "integer", "minimum": 1, "maximum": 1000},
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
}
},
"retrievalConfig": {
"type": "object",
"additionalProperties": false,
"not": {
"required": ["hybridDenseWeight", "hybridSparseWeight"],
"properties": {
"hybridDenseWeight": {"const": 0},
"hybridSparseWeight": {"const": 0}
}
},
"required": ["indexProfileId", "denseTopN", "sparseTopN", "hybridTopN", "hybridRrfK", "hybridDenseWeight", "hybridSparseWeight", "searchTopK"],
"properties": {
"indexProfileId": {"type": "string", "minLength": 1},
"denseTopN": {"type": "integer", "minimum": 1, "maximum": 64},
"sparseTopN": {"type": "integer", "minimum": 1, "maximum": 64},
"hybridTopN": {"type": "integer", "minimum": 1, "maximum": 64},
"hybridRrfK": {"type": "integer", "minimum": 1, "maximum": 1000},
"hybridDenseWeight": {"type": "number", "minimum": 0, "maximum": 10},
"hybridSparseWeight": {"type": "number", "minimum": 0, "maximum": 10},
"searchTopK": {"type": "integer", "minimum": 1, "maximum": 64}
}
},
"rerankerConfig": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "model", "endpoint", "timeoutSeconds"],
"properties": {
"provider": {"const": "qianfan"},
"model": {"const": "bce-reranker-base"},
"endpoint": {"type": "string", "minLength": 1},
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
}
},
"mcpConfig": {
"type": "object",
"additionalProperties": false,
"required": ["publicUrl", "issuerUrl", "clientId", "serviceIdentity", "requiredScope", "allowedHosts", "allowedOrigins", "maxRequestBodyBytes"],
"properties": {
"publicUrl": {"type": "string", "pattern": "^https://"},
"issuerUrl": {"type": "string", "pattern": "^https://"},
"clientId": {"type": "string", "minLength": 1},
"serviceIdentity": {"type": "string", "minLength": 1},
"requiredScope": {"const": "knowledge:search"},
"allowedHosts": {"type": "array", "minItems": 1, "items": {"type": "string", "minLength": 1}},
"allowedOrigins": {"type": "array", "minItems": 1, "items": {"type": "string", "pattern": "^https://"}},
"maxRequestBodyBytes": {"type": "integer", "minimum": 1024, "maximum": 4194304}
}
}
}
}

View File

@ -0,0 +1,294 @@
# DEPLOY-002 safe defaults. Real endpoints, immutable image coordinates and
# Secret names are supplied in an environment-specific values file.
nameOverride: ""
fullnameOverride: ""
existingSecret: knowledge-platform-runtime
imagePullSecrets: []
images:
api:
repository: knowledge-platform/enterprise-knowledge-api
tag: "0.1.0"
digest: ""
pullPolicy: IfNotPresent
mcp:
repository: knowledge-platform/enterprise-knowledge-mcp
tag: "0.1.0"
digest: ""
pullPolicy: IfNotPresent
worker:
repository: knowledge-platform/enterprise-knowledge-worker
tag: "0.1.0"
digest: ""
pullPolicy: IfNotPresent
# The optional edge proxy uses a standard upstream image. Production
# environments should mirror it into their approved Registry and pin a digest.
edge:
repository: nginx
tag: "1.29.1-alpine"
digest: "sha256:42a516af16b852e33b7682d5ef8acbd5d13fe08fecadc7ed98605ba5e3b26ab8"
pullPolicy: IfNotPresent
serviceAccount:
create: true
name: ""
annotations: {}
podAnnotations: {}
podLabels: {}
# DEPLOY-003 keeps one API and MCP replica available during voluntary node
# maintenance. This complements, rather than replaces, RollingUpdate settings.
podDisruptionBudget:
enabled: true
apiMinAvailable: 1
mcpMinAvailable: 1
edgeMinAvailable: 1
observability:
metrics:
enabled: true
path: /metrics
api:
replicaCount: 2
service:
type: ClusterIP
port: 80
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
probes:
startup:
failureThreshold: 30
periodSeconds: 2
readiness:
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
liveness:
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3
mcp:
replicaCount: 2
service:
type: ClusterIP
port: 80
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
probes:
startup:
failureThreshold: 30
periodSeconds: 2
readiness:
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
liveness:
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3
# For clusters without an Ingress Controller, this infrastructure-only proxy
# preserves one external endpoint while API and MCP remain separate workloads.
# Use a temporary NodePort for staging, then change it to the existing public
# NodePort only during the controlled cutover described in the runbook.
edge:
enabled: false
host: knowledge.example.invalid
replicaCount: 2
service:
type: NodePort
port: 80
nodePort: 0
clientMaxBodySize: 210m
proxyReadTimeoutSeconds: 3600
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi
probes:
startup:
failureThreshold: 30
periodSeconds: 2
readiness:
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
liveness:
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3
worker:
replicaCount: 2
terminationGracePeriodSeconds: 60
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 2Gi
probes:
startup:
failureThreshold: 30
periodSeconds: 2
liveness:
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3
# Celery Beat is separated from scalable workers so only one recovery
# scheduler publishes due durable Jobs at a time.
scheduler:
enabled: true
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
probes:
startup:
failureThreshold: 30
periodSeconds: 2
liveness:
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 3
migration:
enabled: true
activeDeadlineSeconds: 600
backoffLimit: 0
ttlSecondsAfterFinished: 86400
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
ingress:
enabled: false
className: nginx
annotations: {}
api:
host: knowledge.example.invalid
path: /
tlsSecretName: knowledge-platform-api-tls
mcp:
host: mcp.example.invalid
path: /mcp
tlsSecretName: knowledge-platform-mcp-tls
# Only non-sensitive runtime settings belong here. The templates enumerate
# keys explicitly so credentials cannot be added to this ConfigMap by mistake.
config:
appName: enterprise-knowledge-platform
logLevel: INFO
database:
poolSize: 10
maxOverflow: 10
poolRecycleSeconds: 600
poolTimeoutSeconds: 30
connectTimeoutSeconds: 30
ingestion:
maxAttempts: 3
retryBackoffSeconds: 2
retryBackoffMaxSeconds: 60
workerLeaseSeconds: 300
recoveryIntervalSeconds: 30
batchMaxFiles: 20
batchMaxTotalBytes: 209715200
maxUploadBytes: 52428800
compilation:
maxAttempts: 3
retryBackoffSeconds: 2
retryBackoffMaxSeconds: 60
workerLeaseSeconds: 300
recoveryIntervalSeconds: 30
objectStorage:
provider: bos
endpoint: https://s3.example.invalid
bucket: knowledge-platform
keyPrefix: knowledge-platform
timeoutSeconds: 30
parser:
# Enabling MinerU also requires a reachable endpoint and a worker lease
# strictly greater than mineruTimeoutSeconds.
provider: lightweight
mineruEndpoint: ""
mineruBackend: hybrid-engine
mineruParseMethod: auto
mineruLanguage: ch
mineruEffort: medium
mineruTimeoutSeconds: 600
lowOcrConfidenceThreshold: 0.70
chunking:
maxTokens: 450
overlapTokens: 75
maxCharacters: 2000
embedding:
provider: qianfan
model: bge-large-zh
endpoint: https://qianfan.baidubce.com/v2/embeddings
timeoutSeconds: 30
vectorStore:
provider: oceanbase
uri: oceanbase.example.invalid:2881
user: knowledge-platform
database: knowledge_platform
table: knowledge_chunks_v1
index: vidx_knowledge_chunks_embedding
batchSize: 100
timeoutSeconds: 30
retrieval:
indexProfileId: demo-bge-large-zh-structure-v2-oceanbase-v1
denseTopN: 20
sparseTopN: 20
hybridTopN: 20
hybridRrfK: 60
hybridDenseWeight: 1.0
hybridSparseWeight: 1.0
searchTopK: 5
reranker:
provider: qianfan
model: bce-reranker-base
endpoint: https://qianfan.baidubce.com/v2/rerank
timeoutSeconds: 30
mcp:
publicUrl: https://mcp.example.invalid/mcp
issuerUrl: https://mcp.example.invalid
clientId: workbuddy-enterprise
serviceIdentity: workbuddy-public-knowledge
requiredScope: knowledge:search
allowedHosts:
- mcp.example.invalid
allowedOrigins:
- https://mcp.example.invalid
maxRequestBodyBytes: 1048576
# Optional local dependency. Keep disabled for production and provide
# RABBITMQ_URL through existingSecret for an externally managed broker.
rabbitmq:
enabled: false
existingSecret: knowledge-platform-rabbitmq

View File

@ -0,0 +1,40 @@
-- ARCH-003/EKP-052 rebuildable OceanBase dense and sparse chunk index.
-- Canonical metadata is co-located in OceanBase relational tables; this vector
-- table remains a derived index and may be rebuilt from canonical chunks.
-- Requires OceanBase 4.3.5.4+; ARCH-006 verifies PRE-FILTER in the target test cluster.
CREATE TABLE IF NOT EXISTS knowledge_chunks_v1 (
index_record_id VARCHAR(36) NOT NULL,
chunk_id VARCHAR(36) NOT NULL,
document_id VARCHAR(36) NOT NULL,
document_version_id VARCHAR(36) NOT NULL,
filename VARCHAR(255) NOT NULL,
page INT NULL,
page_end INT NULL,
section VARCHAR(1024) NULL,
chunk_index INT NOT NULL,
content MEDIUMTEXT NOT NULL,
embedding VECTOR(1024) NOT NULL,
embedding_model VARCHAR(128) NOT NULL,
index_profile_id VARCHAR(128) NOT NULL,
-- Applied by PRE-FILTER before ANN Top-K; historical rows remain rebuildable.
is_active TINYINT(1) NOT NULL DEFAULT 0,
PRIMARY KEY (index_record_id),
KEY idx_knowledge_chunks_document_profile (document_id, index_profile_id),
KEY idx_knowledge_chunks_version_profile (document_version_id, index_profile_id),
KEY idx_knowledge_chunks_active_profile (index_profile_id, is_active),
KEY idx_knowledge_chunks_profile (index_profile_id),
-- NGRAM size 2 gives deterministic Chinese and mixed-term keyword recall.
FULLTEXT INDEX ftidx_knowledge_chunks_content (content)
WITH PARSER NGRAM
PARSER_PROPERTIES=(ngram_token_size=2),
VECTOR KEY vidx_knowledge_chunks_embedding (embedding)
WITH (
DISTANCE = COSINE,
TYPE = HNSW,
LIB = VSAG,
M = 16,
EF_CONSTRUCTION = 256,
EF_SEARCH = 200
) BLOCK_SIZE 16384
) ORGANIZATION = HEAP;

View File

@ -0,0 +1,19 @@
-- EKP-022 forward migration for an existing knowledge_chunks_v1 vector table.
-- Apply once before deploying application code that emits version-scoped rows.
-- Legacy version 1 deliberately reused document_id, so this backfill is exact.
ALTER TABLE knowledge_chunks_v1
ADD COLUMN document_version_id VARCHAR(36) NULL AFTER document_id,
ADD COLUMN is_active TINYINT(1) NOT NULL DEFAULT 1 AFTER index_profile_id;
UPDATE knowledge_chunks_v1
SET document_version_id = document_id
WHERE document_version_id IS NULL;
ALTER TABLE knowledge_chunks_v1
MODIFY COLUMN document_version_id VARCHAR(36) NOT NULL,
ADD KEY idx_knowledge_chunks_version_profile (
document_version_id,
index_profile_id
),
ADD KEY idx_knowledge_chunks_active_profile (index_profile_id, is_active);

View File

@ -0,0 +1,8 @@
-- EKP-052 forward upgrade for an existing knowledge_chunks_v1 table.
-- Apply once before enabling sparse retrieval. The index is derived entirely
-- from existing content rows and does not change canonical Chunk ownership.
CREATE FULLTEXT INDEX ftidx_knowledge_chunks_content
ON knowledge_chunks_v1(content)
WITH PARSER NGRAM
PARSER_PROPERTIES=(ngram_token_size=2);

View File

@ -0,0 +1,12 @@
{
"mcpServers": {
"enterprise-knowledge": {
"type": "http",
"url": "${EKP_MCP_URL:-http://127.0.0.1:8766/mcp}",
"headers": {
"Authorization": "Bearer ${EKP_MCP_TOKEN}"
},
"description": "企业知识检索、来源定位与章节阅读"
}
}
}

View File

@ -0,0 +1,4 @@
services:
rabbitmq:
image: rabbitmq:4-management
ports: ["5672:5672", "15672:15672"]

154
hy4/_kp/docs/README.md Normal file
View File

@ -0,0 +1,154 @@
# 文档中心
本文档是 Enterprise Knowledge Platform 的文档导航与治理入口。阅读或修改文档时,先确认文档的职责与优先级,避免在多个文件中重复维护同一结论。
## 权威性顺序
发生冲突时,按以下顺序判断:
1. `AGENTS.md`:仓库协作规则与不可违反的工程约束。
2. `product/PRD.md`:产品范围、需求和 V1 验收标准。
3. `architecture/ADR.md`:已接受、已取代的架构决策。
4. `architecture/ARCHITECTURE.md`:当前架构及模块边界。
5. `api/API_CONTRACT.md`、`api/MCP_CONTRACT.md`:公开接口契约。
6. `plans/TASKS.md` 与当前活动计划:任务状态、依赖和当前执行顺序。
7. `development/`:实现、部署和运维操作说明,不得重新定义产品或架构。
8. 历史计划与进度记录:只用于追溯,不得覆盖当前基线。
如果 PRD 与 ADR 冲突,应先明确是产品范围调整还是架构调整,不得在操作手册中自行消解冲突。
## 核心文档
| 文档 | 职责 | 生命周期 |
|---|---|---|
| [`../README.md`](../README.md) | 项目简介、启动方式和最短阅读路径 | 活跃 |
| [`product/PRD.md`](product/PRD.md) | 产品需求与 V1 验收基线 | 权威 |
| [`architecture/ADR.md`](architecture/ADR.md) | 架构决策状态 | 权威 |
| [`architecture/ARCHITECTURE.md`](architecture/ARCHITECTURE.md) | 当前技术架构 | 权威 |
| [`api/API_CONTRACT.md`](api/API_CONTRACT.md) | REST API 公开契约 | 权威 |
| [`api/MCP_CONTRACT.md`](api/MCP_CONTRACT.md) | MCP 公开契约 | 权威 |
| [`plans/TASKS.md`](plans/TASKS.md) | Task ID、状态和依赖 | 权威 |
| [`plans/PROGRESS.md`](plans/PROGRESS.md) | 当前阶段摘要和最近进展 | 活跃 |
| [`development/DEFINITION_OF_DONE.md`](development/DEFINITION_OF_DONE.md) | 任务完成标准 | 权威 |
| [`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md) | 文档、分支、Tag、stash 与 worktree 治理规范 | 权威 |
## 当前活动文档
`DEPLOY-003` 已按开发侧范围闭环。`BUILD-001` 已完成根 Dockerfile、Bake、静态测试和 `dev`
分支差异收敛;`EKP-015~022` 已完成 DocumentVersion 模型、Source Hash 去重和版本激活,
`EKP-052~053` 已完成 BM25/Sparse 候选检索与 Hybrid Fusion;`EKP-025~027` 已完成可选
MinerU OCR 路由、质量证据与清洗流水线;`EKP-026` 已完成未接线的 Docling Adapter。当前执行
链中的 `EKP-028~029` 已完成规范解析模型、结构化表格 Chunk 和 P0 文件路由;`EKP-100` 已完成
版本化 Golden Dataset 与统一指标基线,`EKP-042` 已补齐 Qwen3 候选 Adapter。由于尚无真实
Qwen Endpoint/GPU 环境,`EKP-101` 保持未完成并暂停;`EKP-081` 已补齐高可读性 Source/解析
双视图、页范围、页级质量和 Job/Attempt Inspector,用户确认权限视图合并到后续 User/KB/ACL
任务。`EKP-010~017` 已完成领域包、OceanBase Migration 门禁、Organization/User/Group/Role、
KnowledgeBase、Folder、ACL、持久化 Ingestion Job/Attempt 及 AuditLog 规范模型;`EKP-020~021`
已补齐 ObjectStorage 与 Document 上传 API,`EKP-022~024` 已完成版本语义、异步 Job 和 Parser
抽象,`EKP-040~046` 已闭环 EmbeddingProvider、千帆/Qwen Adapter、完整 Index Profile、
VectorStore/Fake、OceanBase Vector Adapter 与完整 Shadow Index 生命周期,`EKP-050~056` 已完成
内部 Permission Resolver、Document Folder 归属约束、Dense/Sparse 检索前 ACL 过滤、完整
Dense/Sparse 候选、Hybrid Fusion、Reranker Provider、规范 Source 补全和 Search Service/API,
以及五阶段检索诊断/Search Lab;`EKP-070` 已完成 MCP User Token/Service API Key、规范身份映射、
ACL 与 `mcp_enabled` 检索前过滤;`EKP-071` 已补齐文档、大纲、章节、精确 Source 与 Knowledge
Base 列表五个 P0 Tool,`EKP-072` 已完成官方 MCP Client 五 Tool 来源链兼容验收。`EKP-080` 已
增加实时知识流运营总览并闭环 Overview、上传、文档、Search Lab 四个核心操作面;`EKP-082` 已
完成可信 REST User/Service 身份、首身份 Bootstrap、User/Group/Role 关系、KB/Folder/ACL 管理、
KB-aware Document 归属和权限操作面,不创建占位权限数据。`EKP-090` 已完成可追溯 Compiled
Knowledge 模型、手工/AI 草稿工作流、REST 和管理端操作面;`EKP-091` 已完成 Source 变化传播、
冲突/Stale、审核与发布;`EKP-092` 已完成 Published-only Compiled 检索、Raw/Compiled 联合精排、
REST/MCP 来源链及 Search Lab 六阶段;`EKP-103` 已完成持久化 Compilation Job、后台重试、租约
恢复以及安全/性能/重建门禁。`EKP-001~004` 已完成交付;`EKP-005` 结构化日志已完成实现与验证:
字段白名单覆盖现有业务事件,无效/非结构化依赖消息失败关闭,请求关联 ID、标量边界和幂等配置由
测试保护,并已合并至 `main@010c0a2`;`EKP-006` CI 基线已完成实现与验证:Provider-free
`make ci` 串联锁文件、Web、静态检查、Unit/Integration、Migration、离线 RAG 与真实进程 E2E,
且不改变中央发布模板,已合并至 `main@4327ede`;`EKP-007` Fake Provider 框架已完成实现与验证:
四类公共 Fake 复用 Provider-neutral 契约、禁止外部 I/O 和生产回退,并明确不形成 Benchmark 结论;
实现与首轮追溯文档提交为 `c312364`,任务分支已推送并合并至 `main@5957a5f`。原 V1 开发主队列
已完成;2026-09-15 根据新增产品需求重新打开补充开发队列,依次处理 `EKP-083` 文档范围筛选、
`EKP-084` 文件夹上传、`EKP-085` 安全 ZIP 导入和 `EKP-086` Knowledge Base/Folder 层级树均已
完成;新增管理体验补充开发队列已闭环。
文档治理、Benchmark、部署与现场验收继续独立排队,不阻塞该开发队列。
- [`plans/TASKS.md`](plans/TASKS.md):当前 Task 状态、开发/运维边界和后续顺序。
- [`plans/PROGRESS.md`](plans/PROGRESS.md):最近验证、风险和下一步。
- [`plans/DEPLOY_003_REPLAN.md`](plans/DEPLOY_003_REPLAN.md):历史部署门禁与运维交接依据。
- [`development/DEPLOY_003_ACCEPTANCE.md`](development/DEPLOY_003_ACCEPTANCE.md):开发侧公网验收与运维待确认项。
- [`development/CLUSTER_OPERATIONS.md`](development/CLUSTER_OPERATIONS.md):部署、故障恢复和运维操作。
完整 V1 MCP 业务能力保留在开发主队列;远程 MCP/WorkBuddy 与集群现场验收独立排队。任务状态
以 `plans/TASKS.md` 为准,详细现场证据以验收记录为准,`PROGRESS.md` 不重复保存完整流水账。
## 开发与运行手册
| 主题 | 文档 |
|---|---|
| 本地配置 | [`development/LOCAL_PERSISTENT_CONFIG.md`](development/LOCAL_PERSISTENT_CONFIG.md) |
| CI 基线 | [`development/CI_BASELINE.md`](development/CI_BASELINE.md) |
| 仓库治理 | [`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md) |
| Fake Provider | [`development/FAKE_PROVIDERS.md`](development/FAKE_PROVIDERS.md) |
| REST API | [`development/REST_API_SETUP.md`](development/REST_API_SETUP.md) |
| MCP Gateway | [`development/MCP_SETUP.md`](development/MCP_SETUP.md) |
| 管理控制台 | [`development/ADMIN_CONSOLE.md`](development/ADMIN_CONSOLE.md) |
| Compiled Knowledge | [`development/COMPILED_KNOWLEDGE.md`](development/COMPILED_KNOWLEDGE.md) |
| 异步入库 | [`development/ASYNC_INGESTION_SETUP.md`](development/ASYNC_INGESTION_SETUP.md) |
| BOS | [`development/BOS_SETUP.md`](development/BOS_SETUP.md) |
| OceanBase Vector | [`development/OCEANBASE_VECTOR_SETUP.md`](development/OCEANBASE_VECTOR_SETUP.md) |
| Sparse 检索 | [`development/SPARSE_RETRIEVAL.md`](development/SPARSE_RETRIEVAL.md) |
| Dense 检索 | [`development/DENSE_RETRIEVAL.md`](development/DENSE_RETRIEVAL.md) |
| Hybrid Fusion | [`development/HYBRID_RETRIEVAL.md`](development/HYBRID_RETRIEVAL.md) |
| 规范来源补全 | [`development/SOURCE_ENRICHMENT.md`](development/SOURCE_ENRICHMENT.md) |
| Search 诊断 | [`development/SEARCH_DIAGNOSTICS.md`](development/SEARCH_DIAGNOSTICS.md) |
| Embedding Provider | [`development/EMBEDDING_PROVIDER.md`](development/EMBEDDING_PROVIDER.md) |
| 千帆 BGE Embedding | [`development/QIANFAN_EMBEDDING_SETUP.md`](development/QIANFAN_EMBEDDING_SETUP.md) |
| Index Profile | [`development/INDEX_PROFILE.md`](development/INDEX_PROFILE.md) |
| VectorStore | [`development/VECTOR_STORE.md`](development/VECTOR_STORE.md) |
| Parser 抽象 | [`development/PARSER_ABSTRACTION.md`](development/PARSER_ABSTRACTION.md) |
| MinerU Parser | [`development/MINERU_SETUP.md`](development/MINERU_SETUP.md) |
| Docling Parser | [`development/DOCLING_SETUP.md`](development/DOCLING_SETUP.md) |
| Parsed Artifact | [`development/PARSED_DOCUMENT_SCHEMA.md`](development/PARSED_DOCUMENT_SCHEMA.md) |
| 结构化 Chunk | [`development/STRUCTURED_CHUNKING.md`](development/STRUCTURED_CHUNKING.md) |
| Reranker | [`development/RERANKER_SETUP.md`](development/RERANKER_SETUP.md) |
| Shadow Index | [`development/INDEX_REBUILD.md`](development/INDEX_REBUILD.md) |
| 权限解析与检索前过滤 | [`development/PERMISSION_RESOLUTION.md`](development/PERMISSION_RESOLUTION.md) |
| Qwen3 Embedding | [`development/QWEN_EMBEDDING_SETUP.md`](development/QWEN_EMBEDDING_SETUP.md) |
| 容器镜像 | [`development/CONTAINER_IMAGES.md`](development/CONTAINER_IMAGES.md) |
| Kubernetes | [`development/KUBERNETES_DEPLOYMENT.md`](development/KUBERNETES_DEPLOYMENT.md) |
| 集群运维 | [`development/CLUSTER_OPERATIONS.md`](development/CLUSTER_OPERATIONS.md) |
| 安全 | [`development/SECURITY.md`](development/SECURITY.md) |
| 测试 | [`development/TEST_STRATEGY.md`](development/TEST_STRATEGY.md) |
| RAG 评估 | [`evaluation/RAG_EVALUATION.md`](evaluation/RAG_EVALUATION.md) |
## 历史与冻结文档
以下文档记录已经完成的阶段或过去的技术迁移,只保留追溯价值:
- `demo/DEMO_SCOPE.md`
- `plans/ARCHITECTURE_MIGRATION_PLAN.md`
- `plans/ASYNC_INGESTION_MCP_PLAN.md`
- `plans/PUBLIC_KNOWLEDGE_CLUSTER_PLAN.md`
冻结文档不再同步当前实现细节。若其中内容与核心文档冲突,以核心文档和当前活动计划为准。旧版本的逐任务实现细节通过 Git 历史查询,不继续堆积到活跃文档。
## 文档维护规则
1. 每个概念只设一个事实源;其他文档使用链接引用,不复制整段内容。
2. PRD 只描述产品需求,Architecture 只描述技术结构,Contract 只描述外部行为,Runbook 只描述操作步骤。
3. `TASKS.md` 只记录任务状态和依赖;`PROGRESS.md` 只保留当前阶段、最近结果、风险和下一步。
4. 已完成的计划标记为“历史/冻结”,不继续加入新状态。
5. 行为变化必须同步对应契约;仅实现细节变化不应改写 PRD。
6. 不在文档中保存未获授权的密码、Token、Key、内部 Endpoint 或真实业务正文。
7. 命令、路径、配置键、API 字段、状态值和 Task ID 保持原样,避免翻译导致不可执行。
8. 删除或合并文档前,先更新所有内部引用并确认权威信息已有归属。
9. 文档精简、任务分支归档、里程碑 Tag 和 worktree 清理统一遵循
[`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md)。
## 变更检查
文档变更至少执行:
```bash
git diff --check
```
涉及代码行为、Migration、REST/MCP 契约或检索结果的变更,仍须按 `development/DEFINITION_OF_DONE.md` 运行相应工程门禁。

View File

@ -0,0 +1,431 @@
# REST API 契约
所有公开 API 均在 `/api/v1` 下进行版本化。
初始领域包括:认证、用户/组、知识库、文档、Compiled Knowledge、检索、评估、Job、审计日志、Provider 和 MCP 管理。
实现开始后,以 OpenAPI 作为可执行事实源。Provider 特有概念不得泄漏到公开契约中。
## Demo v0.1 P0 范围
DEMO-009 公开以下最小接口。`docs/demo/DEMO_SCOPE.md` 中的 `/api/*` 逻辑路径按本契约统一
版本化为 `/api/v1/*`。
`DEMO-015` 已将单文件上传切换为持久化异步 Job;`DEMO-016` 已增加批量提交和聚合查询。
| 方法 | 路径 | 行为 |
|---|---|---|
| `POST` | `/api/v1/documents` | multipart 单文件接收;持久化 Source、Document、Job 后返回 `202` |
| `POST` | `/api/v1/ingestion-batches` | multipart `files` 多文件接收;可选同序 `relative_paths` 保留文件夹层级;返回批次与逐文件状态 |
| `POST` | `/api/v1/ingestion-archives` | multipart `archive` 安全展开单个 ZIP;保留多层目录并返回批次与逐文件状态 |
| `GET` | `/api/v1/ingestion-batches/{batch_id}` | 返回批次汇总和逐文件最新 Job 状态 |
| `GET` | `/api/v1/documents` | 按创建时间倒序返回 ACL 交集内的文档列表,可选 KB/Folder 范围 |
| `GET` | `/api/v1/documents/{document_id}` | 返回单个文档状态与安全错误信息 |
| `GET` | `/api/v1/documents/{document_id}/content` | 返回指定或当前版本的规范 Chunk 正文与来源位置 |
| `GET` | `/api/v1/documents/{document_id}/source` | 流式返回指定或当前版本的不可变原始文件 |
| `DELETE` | `/api/v1/documents/{document_id}` | 无活动入库 Job 时删除;有 Compiled/Compilation Job 引用则保留不可变证据并逻辑删除,否则清理全部产物 |
| `GET` | `/api/v1/jobs/{job_id}` | 返回阶段、状态、attempt、最大尝试次数和安全错误信息 |
| `GET` | `/api/v1/knowledge-bases` | 按当前 User 的有效 Read ACL 列出可读 Knowledge Base |
| `GET` | `/api/v1/knowledge-bases/{knowledge_base_id}/folders` | 在可读 KB 内列出 Folder Metadata |
| `GET` | `/api/v1/compiled-knowledge` | 列出当前身份可读来源交集内的 Compiled Knowledge |
| `GET` | `/api/v1/compiled-knowledge/{compiled_id}` | 返回一条可追溯 Compiled Knowledge |
| `POST` | `/api/v1/compiled-knowledge/drafts` | 从真实 Chunk 创建手工草稿 |
| `POST` | `/api/v1/compiled-knowledge/drafts:generate` | 校验来源并持久化 Compilation Job,返回 `202` |
| `GET` | `/api/v1/compiled-knowledge/jobs/{job_id}` | 按 KB 与全部来源权限交集返回后台生成状态与 Attempt |
| `PUT` | `/api/v1/compiled-knowledge/{compiled_id}/draft` | 使用期望修订号替换草稿编辑字段 |
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:submit-review` | 核验来源后提交人工审核 |
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}/conflicts` | 记录绑定两个确切来源的冲突主张 |
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}/conflicts/{conflict_id}:resolve` | 以说明解决一个冲突 |
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:publish` | 发布已核验且无未解决冲突的审核版本 |
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:reject` | 携带原因退回审核版本 |
| `POST` | `/api/v1/search` | Dense/Sparse Top20、Weighted RRF Top20、Reranker、TopK、来源补全 |
| `POST` | `/api/v1/search/diagnostics` | 执行同一检索链一次并返回 Scope/Profile、各阶段候选、耗时和最终来源 |
除 `/healthz`、OpenAPI 和同源 Admin 静态资源外,REST 业务接口必须携带
`Authorization: Bearer <token>`。运行时 `API_IDENTITY_CREDENTIALS` 将 User Token 或 Service API
Key 映射到规范 Organization/User,并显式声明 `knowledge:read`、`knowledge:write`、
`knowledge:search`、`knowledge:manage` Scope;Token 不进入 URL、响应或日志。缺失/无效凭据返回
`401`,有效凭据缺少操作 Scope 或映射到失效 User 返回 `403`,无权读取的具体资源与不存在统一返回
`404`。`API_ALLOW_LEGACY_PUBLIC=true` 只在未配置任何正式凭据时保留旧公共知识兼容,不具备管理权限。
`EKP-053` 不改变普通 Search 请求或响应字段。响应中的 `score` 表示融合分数,`rerank_score` 表示
精排分数;Dense/Sparse 原始分数、名次、权重和 Index Profile 只通过 `EKP-056` 的管理端诊断
接口提供,普通 Search 与 MCP 不返回这些字段。Provider Endpoint、Credential 和厂商响应在任何
接口中都不可见。
`EKP-051` 只补强内部 Dense Query/Top-N、Profile Query Instruction、空 ACL Scope 和候选失败关闭
边界,不改变上述公开路径、请求、响应或状态码。
`EKP-054` 同样只补强内部 Reranker 协议、输入/输出门禁、Active Profile 模型/候选规模对齐及
千帆 BCE 安全故障边界;不新增或改变公开字段、路径、状态码或客户端入口。
`EKP-055` 将最终结果严格绑定 Active Index Profile 与可验证的规范 Chunk/Index State/Ready Active
Version,并补充 `section_path` 和固定为 `raw` 的 `source_type`。客户端请求的 `top_k` 不得大于
Active Profile 的 `final_top_k`,禁止一次请求绕过已评测配置扩大最终结果数。
上传字段名为 `file`;正式身份还必须提交 `knowledge_base_id`,可选 `folder_id` 必须属于同一知识库。
上传前按当前 User 的有效 `write` 权限验证知识库,随后把 Organization/KB/Folder 原样写入 Document
与 DocumentVersion。`structure_v2` Active 时支持 PDF、DOCX、PPTX、XLSX、CSV、HTML/HTM、
TXT 和 Markdown;PPTX/XLSX/CSV/HTML 由本地 Docling 解析。`PARSER_PROVIDER=mineru` 时额外
接受 JPG/JPEG/PNG,并将 PDF、DOCX 和图片交给 MinerU `auto` 解析,其他格式保持各自路由。
数据库仍为 `structure_v1` Active 时新增 Docling 格式返回 `415`,直至完成 Shadow Profile 切换。
上传准入由实际 Worker Parser Router 的不可变能力集合生成,不维护第二份运行时白名单;该内部
加固不改变支持格式、请求/响应字段或状态码。
成功返回 `202`、Document 范围字段 `organization_id`、`knowledge_base_id`、`folder_id` 及
`document_version_id`、`document_version`、`source_hash`、`job_id`、`job_status=queued`、
`max_attempts`;响应不等待 Parse、Clean、Chunk、Embedding 或 Index。Document 响应不暴露
`storage_key`、Provider 配置或索引实现。schema v4 Page/Sheet/Table 结构及页级 OCR
Confidence/警告保存在内部 Parsed Artifact,不属于 Demo Document 响应;检索来源只公开已命中
Chunk 的 Sheet 与行列范围,完整 Inspector API/UI 由 `EKP-081` 定义。
上传响应中的 `id` 是稳定逻辑 Document ID;`filename`、`media_type`、`size_bytes`、`status`、
`error_message`、`created_at` 和 `updated_at` 描述本次接收的不可变 DocumentVersion。上传同名新版本
时,旧 Active Version 继续作为 `GET /documents/{id}` 的当前投影,`POST` 的 `202` 响应不得混入
旧版大小或 `ready` 状态。Job、Document 与 Version 标识不一致或 Source Hash 缺失时失败关闭。
服务流式计算 SHA-256 Source Hash。同一 Knowledge Base 内 Hash 相同返回 `409` 和稳定错误码
`document_source_already_exists`,并给出既有 `document_id`、`document_version_id`;同名但 Hash
不同且位于同一 Folder 时复用逻辑 Document ID 并创建递增的 inactive Version;不同 Folder 中的
同名文件是独立逻辑 Document。不同 Knowledge Base 可接收相同文件名和 Hash。旧公共兼容数据保留
在独立 `public/public` 唯一范围。Job、Chunk 和派生产物绑定该 Version,
索引成功后才激活。失败候选不会改变列表/详情所投影的旧 Active Version。
首次并发接收同名不同内容时,数据库唯一约束决定逻辑 Document 胜者,失败请求在确认胜者存在后
执行有界 Metadata 重试并创建后续 Version;不会把正常创建竞争直接映射为 `502`。
`GET /api/v1/documents` 可省略筛选参数以列出全部可读文档,也可提交 `knowledge_base_id` 缩小到一个
可读知识库。知识库范围内的 `folder_scope` 为 `all`(默认,包含根目录与全部 Folder)、`root`
(只含 `folder_id=null` 的知识库根目录文档)或 `subtree`(指定 Folder 及全部后代);`subtree`
必须同时提交 `folder_id`,其他范围禁止提交 `folder_id`。ACL Document ID 集合始终先解析,KB/
Folder 条件只能与该集合取交集,不能扩大权限。未授权 Knowledge Base、未知 Folder 或跨 KB Folder
统一返回 `404`;缺少 KB、缺少 Subtree Folder 或参数组合冲突返回 `422`。旧公共兼容身份只能使用
无筛选列表,不得借筛选参数进入规范 Knowledge Base。
## Compiled Knowledge 生命周期与检索契约(EKP-090~092)
V1 类型为 `summary | entity | concept | faq | sop | comparison | overview | custom_wiki_page`。
EKP-090 创建 `draft`;EKP-091 实现 `draft → review_required → published|rejected`,以及由 Source
变化产生的 `stale`;EKP-092 把仍可核验的 `published` 内容接入正式 Search。所有写操作携带
`expected_revision`,过期修订返回 `409`;AI 始终只创建 Draft,不存在自动发布路径。
手工草稿请求包含 `knowledge_base_id`、`title`、`knowledge_type`、`content` 和 1~50 个去重的
`source_chunk_ids`。所有 Chunk 必须真实存在、属于同一 Organization/Knowledge Base,且调用者对
其 Document 具有 Read 权限、对目标 KB 具有 Write 权限;不接受客户端上传的来源正文、Document
Version 或 Provider 信息。AI 请求以同一来源集合加可选 `title_hint`/`instruction` 先完成 Provider、
来源权限与数据策略校验,再持久化 Compilation Job 并返回 `202 Accepted`;API 请求不调用 LLM。
未配置 Provider 返回 `503`;`local_only` KB 在接收 Job 前拒绝 cloud Provider 并返回 `409`。
Compilation Job 状态为 `queued | running | retrying | succeeded | failed | cancelled`。查询响应包含
稳定 `job_id`、知识库/类型、有序来源 Chunk ID、Attempt 次数/上限、安全错误、下次执行时间、
Attempt 历史和成功后的 `compiled_knowledge_id`,不返回 instruction、来源正文、Provider 响应或租约
Token。Worker 每次执行前重新解析提交者的 KB Write 与全部 Document Read 权限,再由编纂服务重新
核验 Chunk 所属 Active Version 和数据处理策略。瞬时 LLM/数据库故障有界退避重试;权限、来源、
契约或策略变化永久失败关闭。Broker 只携带 Job ID,定时恢复扫描补投 queued、到期 retrying 和
租约过期 running Job;`generation_job_id` 唯一约束确保重复投递只产生一条 Draft。
响应至少包含 `compiled_id`、标题/类型/正文、`source_document_ids`、`source_chunk_ids`、明确的
`source_document_versions` Document/version 对、有序 `sources`、`created_by`、创建/更新时间、
状态、`origin=manual|ai`、可空 `provider_model`、`revision`、`last_verified_at` 和
`permissions={mode: source_intersection, resolved: true}`。每个 Source 还返回稳定 `source_id`、文件名、
Chunk 顺序/章节、Source Version 创建时间和 `state=current|outdated|deleted`;生命周期返回
submitter/reviewer、提交/
审核/发布时间、退回原因和稳定 Stale 原因。读取时必须同时满足 KB Read 权限和全部
来源 Document Read 权限,任一来源无权时整条结果隐藏;禁止先返回 Compiled 内容再过滤来源。
更新使用完整 title/type/content 与 `expected_revision`;`draft` 可直接更新,`rejected|stale` 可修订
回新 Draft。修订 Stale 时可用 1~50 个 `source_chunk_ids` 替换整组来源;旧冲突配对随旧来源集合
失效,生命周期 AuditLog 仍保留动作证据。
提交审核与发布都会核验每个精确 Source 仍属于未删除 Document 的 Ready Active Version;失败会
原子标记 `stale` 并返回 `409`。新 DocumentVersion 激活、引用文档逻辑删除、ACL 新增或删除会在
同一规范事务内自动使受影响知识进入 `stale`,原因分别为 `source_version_changed`、
`source_deleted`、`source_permissions_changed`。引用文档的 DELETE 返回 `204`,但只移出检索并
保留 Source/Version/Chunk 作为不可变证据;普通 Document API 隐藏该逻辑删除项。无 Compiled
引用的 Document 暂时沿用既有直接清理路径。
冲突由审核人明确确认:请求提供有界 Claim 和同一知识项内两个不同 `source_id`;响应保留 Claim、
两侧 Source/Version/创建时间、报告人与解决状态。未解决冲突阻断 Publish,解决必须提供有界说明。
系统不会让 LLM 自动折中冲突,也不会在日志中保存 Claim、正文或标题。提交、冲突、解决、发布、
退回、修订和自动 Stale 均写入脱敏 AuditLog。
Search 请求:
```json
{
"query": "产品A退款周期是多少?",
"top_k": 5
}
```
Search 响应:
```json
{
"results": [
{
"document_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
"document_version_id": "a23c91d7-6a8c-4a66-b4ca-6220d8e3fb09",
"document_version": 2,
"filename": "产品A说明书.pdf",
"page": 12,
"page_end": 13,
"section": "退款政策",
"section_path": ["产品A", "售后服务", "退款政策"],
"sheet_name": null,
"row_range": null,
"column_range": null,
"chunk_id": "76ba1c30-bff7-44e5-9989-fb8372f54f46",
"content": "……",
"context": "命中 Chunk 前后各一个规范 Chunk 组成的有界可读上下文……",
"score": 0.82,
"rerank_score": 0.94,
"source_type": "raw"
}
]
}
```
`page`、`page_end` 是命中 Chunk 覆盖的一基起止页;单页命中两者相同,非分页来源均为 `null`。
跨页命中只能把原始 PDF 从 `page` 打开,客户端必须展示完整页范围,不得把起始页描述成精确页。
Spreadsheet/Table 命中时,`sheet_name` 为 Sheet 名或 `null`,`row_range`、`column_range` 为
一基闭区间二元数组,例如 `[2, 8]`;非表格 Chunk 三个字段均为 `null`。这些来源字段来自
OceanBase 规范 Chunk,不信任向量索引中的冗余 Metadata。
`content` 始终是参与召回和 Reranker 的精确命中 Chunk;`context` 从同一 DocumentVersion、同一
Chunk 策略的前后各一个规范 Chunk 拼接,最多 2400 字符,用于避免标题或极短结构节点脱离正文。
上下文不改变候选排序和分数,也不从向量 Metadata 恢复内容。
`section_path` 是从顶层到命中节点的完整规范章节路径;`section` 保留为兼容摘要字段。
`source_type=raw` 明示结果来自不可变 Source 的派生 Chunk,后续 Compiled 检索不得复用该值伪装
成 Raw 结果。所有来源字段以关系库中的规范记录为准;缺少当前 Profile 的 `ChunkIndexState`、
模型/维度或 Chunk 策略不一致、Version 非 Ready Active、文件名或正文漂移时整次检索失败关闭。
正式身份 Search 还可返回 `source_type=compiled`。Compiled 候选只从已经通过检索前 ACL Filter 的
Raw Source 候选扩展,其所属 Knowledge Base 和全部底层 Document 都必须位于调用者已解析 Read
Scope;旧公共兼容入口
不返回 Compiled 内容。只有 `status=published`、`published_at`/`last_verified_at` 完整且每个精确
Source 仍为未删除 Ready Active Version 的知识可以进入联合 Reranker。Draft、ReviewRequired、
Rejected、Stale 和 Archived 全部排除。
Compiled Result 不伪造 Raw Document/Chunk 字段,返回 `compiled_knowledge_id`、
`knowledge_base_id`、`title`、`knowledge_type`、已发布 `content`、有界 `context`、排序分数和
`underlying_sources[]`。每个底层来源包含 Document/Version、文件名、页/章节/Sheet/行列范围、
Chunk ID/顺序与精确正文,供客户端进入同一 Document Inspector 或不可变 Source。`score` 是触发
该知识的最强授权 Raw 证据的 Fusion Score;标题与正文随后和 Raw Chunk 一起进入同一次 Reranker,
联合候选总数仍不超过 Active Profile 的 `hybrid_top_n`,最终结果不超过请求/Profile TopK。
`GET /api/v1/documents/{document_id}/content` 可通过 `document_version_id` 查询参数选择精确不可变
版本;省略时优先读取 Active Version,尚未激活时读取最新候选。响应包含 Version、状态、Active
标记、Index Profile 及按 `chunk_index` 排序的 Chunk 正文、Page/Section/Sheet/行列位置。只返回
当前 Active Index Profile 已成功索引的规范 Chunk,处理中尚无完整索引时 `chunks` 为空。
`quality_status` 区分 `available`、`not_recorded` 和 `unavailable`;`page_quality` 只返回稳定的页码、
OCR Confidence 和警告,不暴露 Provider 原始响应。`processing_job` 返回该 Version 的持久化 Job
及按次数排序的 Attempt 历史,不暴露 Broker 或 Provider 细节。质量 Artifact 缺失或损坏时仍
返回规范 Chunk,并以 `unavailable` 明示降级。该接口是解析/检索溯源视图,不保留 PDF 或 Office
排版,也不应被客户端描述为完整原文。
`GET /api/v1/documents/{document_id}/source` 使用相同版本选择规则,流式读取 ObjectStorage 中的
不可变 Source,不回显 `storage_key`。PDF、纯文本、Markdown 和图片使用安全 inline 响应;HTML、
Office 与其他格式强制下载并使用 `X-Content-Type-Options: nosniff`,避免同源主动内容执行。
公开错误语义:请求校验失败返回 `422`,不支持的上传类型返回 `415`,文件过大返回 `413`,
文档、Version、Job 或批次不存在返回 `404`;重复 Source 或活动 Job 返回 `409`,前者的
`detail.code` 为 `document_source_already_exists`;运行时 Provider 未配置返回 `503`。上游 Provider
或对象存储异常只返回经过收敛的 `5xx` 信息,不回显凭证、存储 Key、请求正文或厂商响应。MinerU 瞬时网络、
限流和服务端错误进入既有有界 Job 重试;内容或结构错误按永久失败处理。
业务服务首次装配时,规范数据库异常返回 `503 API canonical database is unavailable` 并附带
`X-EKP-Dependency: canonical-database`;向量索引初始化、配置兼容或 Schema 检查异常返回
`503 API vector index is unavailable` 并附带 `X-EKP-Dependency: vector-index`。这些稳定分类仅用于
部署诊断,不返回 SQL、连接地址、表名、Provider 响应或底层异常文本。
Search Query 非法或 `top_k` 超过 Active Profile 的 `final_top_k` 时返回安全、稳定的 `422`,响应
不披露 Profile ID、模型名或内部候选规模。
### Search 诊断响应
`POST /api/v1/search/diagnostics` 复用相同 `SearchRequest`,只供受控网络内的管理端 Search Lab。
它必须只执行一次完整链路,不能为收集阶段信息再次发送 Query 或产生与普通 Search 不同的排序。
响应包含:
- 规范化 Query;正式身份返回 `document_filter` Scope、`acl_filter_applied=true` 及已解析 Document
数量;显式旧公共兼容入口才返回 `public_knowledge`/`false`,不得混淆两种安全边界;
- Active Profile ID、Chunk/Embedding/Sparse/Reranker 标识、Dense/Sparse/Hybrid/Final 数量、RRF
常量和权重;不包含 Endpoint、Key 或 Provider 原始配置;
- `dense_candidates`、`sparse_candidates`、`fusion_candidates`、`compiled_candidates`、
`reranked_candidates` 与最终
`results`。阶段候选提供规范身份、最多 1200 字符的显式截断正文、名次和适用的分支/Fusion/
Reranker 分数;最终 `results.content` 仍是完整精确命中 Chunk;
- Dense、Sparse、Fusion、Compiled Expansion、Reranker、Source Enrichment 和总耗时,单位为
毫秒且使用单调时钟。
阶段候选仍受检索前 Scope 约束。诊断接口不允许客户端修改 Profile、权重、模型或候选规模,也不
提供独立 Provider 调用能力;非法 Query/Profile TopK 返回与普通 Search 一致的 `422`,其他错误
收敛为不含内部异常的 `502`。
## Demo 管理端客户端路由
DEMO-010 提供 `/admin/upload`、`/admin/documents` 和 `/admin/search` 三个客户端路由;`EKP-080`
新增 `/admin/overview` 运营总览,以同一个 `GET /documents` 响应聚合 Source、Processing、Ready
库存、就绪率和最近文档,不引入第二套统计事实源。`EKP-081`
在既有路由内增加 Document Inspector、规范 Chunk 来源轨道、原始文件入口,以及 Search Lab 的
可读上下文、精确命中片段和命中来源定位。Document Inspector 提供“原始文件预览”和“解析与
溯源”双视图:PDF 从命中起始页打开,Markdown、纯文本和图片安全在线预览,Office/HTML 等主动
或浏览器不稳定格式保持下载回退。跨页 Chunk 展示完整起止页范围,不把起始页误述为精确页;
Inspector 同时展示持久化处理流水线、Attempt 历史和低置信度 OCR 页警告。`EKP-056` 将 Search
Lab 接到 `/search/diagnostics`,显示 Query/Scope/ACL、Profile、Dense、Sparse、Fusion、Reranker
和 Final 信号轨;最终来源操作仍复用同一 Inspector 与不可变 Source 入口。
`EKP-082` 新增 `/admin/knowledge-bases` 和 `/admin/permissions`,前者管理 Knowledge Base/Folder,
后者管理 User/Group/Role、成员/角色关系和显式 ACL。浏览器 Token 仅保存在当前标签页的
`sessionStorage`;原文预览与新窗口打开先发送带认证 Header 的请求,再使用短生命周期 Blob URL,
不会把 Token 写入链接。
`EKP-083` 在 `/admin/documents` 增加两级范围筛选:先选择一个可读 Knowledge Base,再选择全部
文件夹、知识库根目录或显示完整祖先路径的 Folder 子树;列表同步显示结果数量、当前范围和筛选空
状态。页面直接调用上述 Document 列表查询参数,不在浏览器下载全部文档后自行模拟权限筛选。
`EKP-090~091` 的 `/admin/compiled-knowledge` 从可读 KB、Ready Document 和真实 Chunk 选择来源,
创建手工/AI 草稿、按 revision 编辑,展示当前/历史/删除来源轨道,并提供核验提交、双来源冲突记录/
解决、发布、退回和 Stale 修订入口;来源跳转仍进入 Document Inspector,不把顺序 Chunk 描述为原文。
`/admin` 默认展示运营总览;服务根路径 `/` 以 `307 Temporary Redirect` 同源跳转到 `/admin`,
使用户直接打开部署域名即可进入服务台。它们是同源静态页面,不是新增业务 API,也不改变
上述公开 REST 契约;所有业务操作仍调用 `/api/v1`。
`EKP-092` 在 `/admin/search` 的诊断轨中增加“已发布知识”阶段。Final 中 Raw 结果继续显示上下文、
精确命中和原始文件入口;Compiled 结果显示“已审核知识”、已发布正文与底层来源证据轨道,每个
来源分别提供“定位底层来源”和“查看或下载原始文件”,不把 L2 正文描述成原文。
## V1 身份与管理契约(EKP-082)
`knowledge:manage` 是 Organization 管理员 Scope;除 Session 自省外,所有管理路径还必须具备该
Scope 并绑定一个当前 Organization 内的 Active 规范 User。Session 向任意有效规范身份返回自身
`scopes` 和 `credential_type`,供客户端收敛入口。管理接口只返回身份/资源 Metadata,不创建或
返回 Token:
| 方法 | 路径 | 行为 |
|---|---|---|
| `GET` | `/api/v1/administration/session` | 验证当前 Organization/User 映射 |
| `GET/POST` | `/api/v1/administration/users` | 列出或创建无凭据 User |
| `GET/POST` | `/api/v1/administration/groups` | 列出或创建 Group,列表包含成员与 Role ID |
| `PUT/DELETE` | `/api/v1/administration/groups/{group_id}/members/{user_id}` | 幂等添加/移除成员 |
| `GET/POST` | `/api/v1/administration/roles` | 列出或创建 Role,列表包含直接 User/Group 分配 |
| `PUT/DELETE` | `/api/v1/administration/roles/{role_id}/users/{user_id}` | 幂等分配/撤销 User Role |
| `PUT/DELETE` | `/api/v1/administration/roles/{role_id}/groups/{group_id}` | 幂等分配/撤销 Group Role |
| `GET/POST` | `/api/v1/administration/knowledge-bases` | 列出或创建 Knowledge Base |
| `GET/POST` | `/api/v1/administration/knowledge-bases/{knowledge_base_id}/folders` | 列出或创建同 KB Folder |
| `GET/POST` | `/api/v1/administration/knowledge-bases/{knowledge_base_id}/access-control` | 列出或创建 User/Group/Role ACL |
| `DELETE` | `/api/v1/administration/access-control/{entry_id}` | 删除一个显式 ACL |
Permission Resolver 在 Document 列表、详情、内容、原文、Job、Batch、删除、Search 和 Search
Diagnostics 前解析同一规范 User/Group/Role/ACL 数据。Document 无权与不存在统一隐藏,空 Scope
不调用检索 Provider。旧未分配 Document 仍属于兼容公共范围,不会自动迁入任意知识库或 ACL。
面向上传选择器的非管理 `GET /knowledge-bases` 与 `GET /knowledge-bases/{id}/folders` 只要求
`knowledge:read`,且只投影当前 User 可读的 Active KB/Folder;无权 KB 返回 `404`。
首次配置可通过 `ekp-api-identity-bootstrap` 按已存在于运行时 Secret 的映射显式创建精确
Organization/User;命令行不接受 Token 参数。
`EKP-017` 复用既有持久化 Ingestion Job/Attempt 并新增内部 AuditLog 模型;本任务不新增公开
审计接口,也未把登录、文档、检索、权限或 MCP 操作接入 AuditLog。
集群重启、扩缩容和滚动部署不得改变本契约或要求浏览器重新注入 Provider 配置。数据库、BOS、
模型与队列凭据由服务端 Secret 提供,任何 API 响应都不得返回这些值。
DEPLOY-003 增加两个不进入 `/api/v1` OpenAPI 的运维端点:`GET /healthz` 只表示 API 进程可用;
`GET /metrics` 使用 Prometheus text format 返回进程存活时间、按 HTTP method/路由模板/status
聚合的请求量和按 method/路由模板聚合的耗时。两者不得包含 Query、文件名、正文、身份、
Provider Endpoint 或凭据。响应附带安全 `X-Request-ID`,客户端提供的值只有匹配
`[A-Za-z0-9._:-]{1,64}` 时才会沿用。
默认 dev 镜像执行规范库迁移时,`/healthz` 额外返回 `migration_status`、安全的
`migration_revision` 和 `migration_error_type`;迁移失败时 `status=degraded`,但仍返回 `200` 供
中央部署链诊断,所有业务入口继续失败关闭。显式生产 target 不返回这些 dev 专用字段。
## 异步入库契约(DEMO-015~016 已实现)
| 方法 | 路径 | 行为 |
|---|---|---|
| `POST` | `/api/v1/documents` | 已实现:单文件接收并返回 Document/Job 标识 |
| `POST` | `/api/v1/ingestion-batches` | multipart 多文件/文件夹接收,返回批次及逐文件 Document/Job 标识 |
| `POST` | `/api/v1/ingestion-archives` | multipart 单 ZIP 安全展开,返回批次及逐文件 Document/Job 标识 |
| `GET` | `/api/v1/ingestion-batches/{batch_id}` | 返回批次汇总和逐文件最新状态 |
| `GET` | `/api/v1/jobs/{job_id}` | 已实现:返回持久化 Job 状态 |
单文件响应示意:
```json
{
"id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
"organization_id": "11111111-1111-4111-8111-111111111111",
"knowledge_base_id": "22222222-2222-4222-8222-222222222222",
"folder_id": null,
"filename": "退款政策.md",
"media_type": "text/markdown",
"size_bytes": 1024,
"status": "uploaded",
"error_message": null,
"created_at": "2026-09-02T12:00:00Z",
"updated_at": "2026-09-02T12:00:00Z",
"job_id": "7adc4278-1157-4c3d-95a4-6926ad77a04e",
"document_version_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
"document_version": 1,
"source_hash": "6ca13d52ca70c883e0f0bb101e425a89e8624de51db2d2392593af6a84118090",
"job_status": "queued",
"max_attempts": 3
}
```
普通批量字段名为 `files`。文件夹上传必须再按相同顺序重复提交 `relative_paths`,每个值使用浏览器
提供的 POSIX 相对路径(例如 `产品手册/区域/退款.md`);两组数量不一致返回 `422`。目录模式必须
指定正式 `knowledge_base_id`,可选 `folder_id` 是目录层级的基准父节点,跨 Knowledge Base 或
不存在的基准 Folder 返回 `404`,且不会创建批次、Folder 或对象。成功返回 `202`。响应示意:
```json
{
"batch_id": "d21f8012-4c4e-4bb4-89af-e11c0f9f470e",
"organization_id": "11111111-1111-4111-8111-111111111111",
"knowledge_base_id": "22222222-2222-4222-8222-222222222222",
"status": "processing",
"total_files": 2,
"accepted_files": 1,
"rejected_files": 1,
"completed_files": 1,
"created_at": "2026-09-02T12:00:00Z",
"items": [
{
"item_index": 0,
"filename": "退款政策.md",
"relative_path": "产品手册/华东/退款政策.md",
"acceptance_status": "accepted",
"document_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
"document_version_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
"job_id": "7adc4278-1157-4c3d-95a4-6926ad77a04e",
"job_status": "queued",
"stage": null,
"attempt": 0,
"max_attempts": 3,
"error_message": null
}
]
}
```
Job 公开状态为 `queued | running | retrying | succeeded | failed | cancelled`,并通过 `stage`
表示 `parsing | cleaning | chunking | embedding | indexing`。错误字段只返回有界、安全摘要。
正式身份创建的批次保存并返回 `organization_id`/`knowledge_base_id`,即使所有文件均在准入阶段
被拒绝,轮询仍按该 KB 的有效 Read Scope 鉴权;旧公共兼容批次两字段为 `null`。批次状态为
`processing | completed | partial_failure | failed`;逐文件接收状态为
`pending | accepted | rejected`。批次允许逐文件部分接收/失败,不以一个文件的处理结果回滚
其他文件。文件数量、单文件大小和请求总大小分别由 `BATCH_MAX_FILES`、`MAX_UPLOAD_BYTES`
和 `BATCH_MAX_TOTAL_BYTES` 限制。数量超限返回 `413`;类型、大小或内容不合规作为该文件的
`rejected` 安全结果返回,其他合规文件继续接收。
服务端只持久化通过校验的 `relative_path`,普通文件上传和非法路径均返回 `null`。合法路径最长
1024 字符、目录最多 32 层、每段最多 255 字符;绝对路径、反斜线、空段、`.`、`..`、控制字符、
首尾空白以及路径末尾与 multipart 文件名不一致均只拒绝对应文件并返回固定安全错误。合法目录段
在已授权基准 Folder 下逐级创建或复用;空目录不会产生 Folder 或 Document。文件类型、大小、共享
预算、同 KB Source Hash 去重和同名版本语义与普通批次一致。
ZIP 字段名为 `archive`,必须同时指定正式 `knowledge_base_id`,可选 `folder_id` 作为包内目录的基准
父节点。服务在创建 Batch、Folder、Document 或对象前验证完整中央目录、CRC 和压缩/展开预算;拒绝
绝对路径、父级跳转、反斜线、重复路径、符号链接/特殊文件、加密条目、嵌套 ZIP 和损坏包,忽略
`__MACOSX`、`.DS_Store` 与 AppleDouble 元数据。结构错误整包返回 `422`,格式错误返回 `415`,
预算超限返回 `413`;结构安全后,不支持的普通文件类型作为逐项 `rejected`,其余文件继续入库。
根层文件放入 ZIP 文件名对应的 Folder,包内既有多根目录保持原路径;空目录不落库,ZIP 本身不保存
为 Source。压缩包、条目、展开总量、文件数和压缩比分别受 `MAX_UPLOAD_BYTES`、
`MAX_UPLOAD_BYTES`、`BATCH_MAX_TOTAL_BYTES`、`ZIP_MAX_FILES` 和
`ZIP_MAX_COMPRESSION_RATIO` 约束,目录仍遵守 32 层/1024 字符安全路径上限。
Job 查询响应包含 `document_version_id`。自动重试是服务端 Job 行为,不要求客户端再次上传,也
不新增公开手动重试接口。每次重试前只清理该 Job/Version 的派生数据,保留原始文件、稳定的
Document ID、历史 Version 和旧 Active 检索;达到次数上限或清理失败后进入 `failed`。内部
Lease Token 只用于隔离过期 Worker,不进入 Job 响应,也不改变上述公开状态和字段。

View File

@ -0,0 +1,124 @@
# MCP 契约
> 当前状态:`EKP-070~072` 已完成 MCP 身份、ACL-aware Search、五个 P0 Tool 与客户端兼容
> 验收器的本地开发闭环。远程 MCP Gateway 与 WorkBuddy 现场验收仍作为独立非开发
> 任务搁置,不得据此声称目标环境 MCP 已验收。
P0 业务 Tool:
- `search_knowledge`
- `get_document`
- `get_document_section`
- `get_source`
- `list_knowledge_bases`
每个请求都必须映射到用户或服务身份,并使用与 REST 相同的授权和应用服务。不得暴露 OceanBase Vector、BOS、模型名、Parser 或其他基础设施选型。
## P0 Tool 契约
所有 Tool 都要求 `knowledge:search` scope 和规范 Organization/ACL User;除历史
`search_knowledge` 公共 Demo 兼容外,旧静态 Token 调用其他 Tool 必须失败关闭。Document、Version
或 Chunk 不存在、跨归属或无权访问统一返回 `knowledge resource is unavailable`,不得帮助调用者
枚举越权资源。
`list_knowledge_bases` 无参数,只返回当前身份可读且 `mcp_enabled=true` 的 Active Knowledge Base:
`knowledge_base_id`、`name`、`description`、`visibility` 和 `readable_document_count`。即使空间暂无
Document,只要调用者对 Knowledge Base 本身有 READ 权限也可见;仅通过 Folder/Document ACL
获得下级内容时,对应容器也可见。响应不返回 Owner、ACL 行或数据处理内部配置。
`get_document` 输入 `document_id` 和可选 `document_version_id`。省略 Version 时优先选择 Active
Version,否则选择最新候选;响应返回 Knowledge Base、Version、文件名、媒体类型、大小、状态、
Active 标记、当前 Profile 已索引的 Chunk 数,以及最多 256 条去重 `section_paths`。
`outline_truncated=true` 表示仍有未返回的大纲,不得把该响应描述为正文或原文件。
`get_source` 输入 `document_id`、`chunk_id` 和可选 `document_version_id`,用于把 Search 命中解析为
一个精确规范 Chunk。响应返回 Document/Knowledge Base/Version/文件名,以及 Chunk 的
`chunk_id`、`chunk_index`、Page/Section/Sheet/行列位置、`content` 和 `source_type=raw`。
该 Tool 返回可引用的文本证据,不返回原文件字节;原始 PDF/Office/图片等下载继续使用受控 REST
`GET /api/v1/documents/{document_id}/source`。
`get_document_section` 输入 `document_id`、`anchor_chunk_id`、可选 `document_version_id`、
`offset`(默认 0)与 `max_chunks`(默认 12,范围 1~32)。它按 `chunk_index` 返回与 Anchor 具有
相同完整 `section_path` 的规范 Chunk,并提供 `total_chunk_count` 和 `next_offset`;调用方应持续
使用同一 Anchor 和 `next_offset`,直至其为 `null`。空章节路径只返回 Anchor 本身,避免把整份
无结构文档误当成一个无界响应。
以上读取只接受当前 Active Index Profile 有 `ChunkIndexState` 证据的规范 Chunk;不从向量
Metadata 恢复正文,不返回 Index Profile、Provider 或存储 Key。
## Demo 优先纵向切片(DEMO-012,已实现)
`DEMO-012` 已提升为当前最高实现优先级。首个可运行 Gateway 只注册
`search_knowledge`,其输入与 `/api/v1/search` 对齐:
```json
{
"query": "产品A退款周期是多少?",
"top_k": 5
}
```
输出保留业务来源字段:`document_id`、`document_version_id`、`document_version`、`filename`、
`page`、`page_end`、`section`、`section_path`、`sheet_name`、`row_range`、`column_range`、
`chunk_id`、`content`、`context`、排序分数和 `source_type`。`content` 是精确命中 Chunk;
`context` 是与 REST 一致的有界相邻段落,只改善阅读与
来源理解,不改变候选排序或分数。
`EKP-092` 后正式 User/Service 身份还可收到 `source_type=compiled` 的已发布知识;其响应与 REST
一致,包含 `compiled_knowledge_id`、Knowledge Base、标题/类型/正文、排序分数和完整
`underlying_sources[]`。每个底层来源保留精确 DocumentVersion/Chunk 与位置/正文,可继续调用
`get_source`、`get_document_section` 或 REST Source 入口。Gateway 不自行复制扩展、排序或权限
逻辑;Draft/ReviewRequired/Rejected/Stale/Archived、所属 Knowledge Base 或任一来源不在调用者
READ Scope,以及未启用 MCP 的 Knowledge Base 都不会进入候选。旧公共 Demo 兼容身份继续只返回
Raw 结果。
分页范围与 REST 一样使用一基起止页,单页时两者相同;Spreadsheet 范围使用一基闭区间二元数组,
非对应来源字段为 `null`。Gateway 必须复用
现有 Search Service 或正式 REST API,不得复制检索、来源补全或排序逻辑;默认只返回 Active
Version。
`EKP-053` 后共享 Search Service 使用 Dense/Sparse Weighted RRF + Reranker;输出 `score` 为融合
分数,`rerank_score` 为精排分数。该行为复用不新增 MCP Tool,也不恢复已搁置的远程验收任务。
Demo 验证也必须建立 user/service identity 映射边界,禁止使用共享超级管理员 Token。由于
`EKP-070` 在保留 Demo 兼容层的同时新增运行时 User Token 与 Service API Key 映射。每个正式
凭据都必须绑定 `client_id`、`subject`、`organization_id`、规范 `acl_user_id` 与最小 scopes;
Gateway 调用 `AuthorizedSearchService`,在 Dense/Sparse Top-N 前应用同一精确 Document Filter,
并只纳入 `mcp_enabled=true` 的 Knowledge Base。身份缺失、User 不存在/禁用/跨 Organization、
ACL 服务未装配或规范绑定不完整时全部失败关闭。Service API Key 的 `acl_user_id` 必须是为该
服务专门配置的规范授权主体,不得映射到共享超级管理员。
旧 `MCP_AUTH_TOKEN` 只允许单一公共知识 Demo 兼容路径,不能与正式身份列表同时配置,也不能
用于非公共知识、ACL 完成声明或远程生产接入。`DEMO-012` 只能证明身份传递、认证拒绝和业务
服务复用,不能替代 `EKP-070` 的规范身份与 ACL 证据。错误响应和日志不得包含 Provider
Endpoint、凭据、数据库表名或模型名。
开发验收覆盖:五个 P0 Tool 发现、参数校验、成功/空结果、User/Service 身份、ACL 越权隐藏、
章节分页、认证失败和安全错误收敛。`EKP-072` 还提供可执行的官方 MCP Client 验收器,在同一
会话中验证 Search → Source/Document/Section/Knowledge Base 的来源链;远程只接受 HTTPS,明文
HTTP 仅允许本机回环地址。WorkBuddy Enterprise 的目标环境现场验收属于独立非开发验收,不由
本地确定性 Provider 结果替代。
实现使用独立 Streamable HTTP Gateway,默认端点为 `http://127.0.0.1:8766/mcp`。请求必须
携带 `Authorization: Bearer <token>`;正式凭据列表只从运行时 Secret 的
`MCP_IDENTITY_CREDENTIALS` 读取,支持 `user_token` 和 `service_api_key`,并映射为显式
Client、主体、Organization、ACL User 和 scopes。配置是本地受控凭据适配层,为 OIDC/OAuth2/
企业 SSO Token Verifier 保留替换边界;不伪装成已经接入外部 IdP。
公开错误边界:缺失/无效 Token 为 HTTP `401`,scope 不足为 `403`;Tool 参数不合法以及
`top_k` 超过 Active Profile 最终上限时返回安全的 MCP 请求错误;其他检索运行失败只返回
`knowledge search failed`。Host/Origin allowlist 默认只允许
本机,远程部署必须显式注入公开域名。
## 历史公共知识库部署边界(PLAN-002)
PLAN-002 历史集群切片只暴露只读 `search_knowledge`,并为 WorkBuddy 配置稳定的 HTTPS MCP 地址。
Credential 由 Kubernetes Secret 在服务端运行时注入,客户端只保存管理员分发的最小
`knowledge:search` Token;Pod 重启或滚动升级不得要求重新生成配置。公共知识域不等于匿名
访问或超级管理员身份,未来引入非公共知识前必须升级为用户委托身份与 ACL-aware Search。
当前代码升级到 EKP-071 后应发现本契约列出的五个 P0 Tool;若目标环境仍只发现
`search_knowledge`,表示其仍运行历史镜像,不构成当前版本兼容验收。恢复远程 MCP 任务后,验收
必须从最终 HTTPS 地址发现且仅发现五个批准 Tool,并完成一次真实、有来源的完整读取链。Gateway
失败日志只记录固定事件名,不写请求 Query、异常正文、Provider
Endpoint 或 Token;具体请求量与 Pod 可用性由集群入口/Kubernetes 指标补充,当前不新增 MCP
业务 Tool。

View File

@ -0,0 +1,40 @@
# 架构决策记录
- ADR-001:API First + 管理控制台 — 已接受
- ADR-002:V1 采用模块化单体 + Worker + MCP Gateway — 已接受
- ADR-003:PostgreSQL + 对象存储保存规范数据 — 已由 ADR-012 取代
- ADR-004:Mochow 是初始检索后端,但必须可替换 — 已由 ADR-010 取代
- ADR-005:为 Embedding、Reranker、Parser 和 LLM 建立 Provider 抽象 — 已接受
- ADR-006:RAG + Knowledge Compilation — 已接受
- ADR-007:在检索前或检索过程中执行 ACL — 已接受
- ADR-008:MCP 隐藏基础设施细节 — 已接受
- ADR-009:百度 BOS 是主要对象存储 Provider;本地存储继续用于测试和本地开发 — 已接受
- ADR-010:OceanBase Vector 是 `VectorStore` 背后的主要可重建检索索引;PostgreSQL 仍保存规范数据 — 已由 ADR-012 取代
- ADR-011:千帆 `bge-large-zh` + `bce-reranker-base` 是初始 Demo 模型基线 — 已接受
- ADR-012:OceanBase MySQL 关系表 + BOS 保存规范数据;OceanBase 向量表仍为可重建索引 — 已接受
## 架构迁移边界
ARCH-006 的只读环境核验确认,目标平台没有独立 PostgreSQL,OceanBase MySQL 模式同时承载
关系数据和向量数据。因此 ADR-012 取代 ADR-003/010 中的 PostgreSQL 部署假设:OceanBase
关系表保存 canonical metadata,BOS 保存不可变源文件,OceanBase 专用向量表仍只保存可由
canonical Chunk 重建的检索行。共用数据库实例不改变两类数据的所有权边界,也不允许业务层
绕过 `VectorStore` 直接依赖 `pyobvector`。
目标 test 集群为 OceanBase 4.3.5.4 MySQL 模式。ARCH-006 已用只读 `EXPLAIN` 验证
`VECTOR_INDEX(... pre_filter)` 产生 `VECTOR INDEX ADAPTIVE SCAN (PRE-FILTER)`;版本下限与
运行检查据此调整为 4.3.5.4。生产升级和重新验收仍由运维流程控制。
## 公共知识交付边界
PLAN-002 将当前交付阶段限定为企业受控网络内的单一公共知识可见域,并加入 Kubernetes/Helm
集群交付任务。这是部署范围和实施顺序调整,不是新的架构选型:继续采用 ADR-002 的模块化
单体 API、独立 Worker 和 MCP Gateway,以及架构既定的 Celery/RabbitMQ;OceanBase/BOS
边界继续遵循 ADR-012。
ADR-007 仍为已接受状态。内部 Permission Resolver 和检索前 ACL 下推已实现;开始接收任何非公共
知识前,还必须完成可信请求身份、KB-aware Document 归属及 REST/MCP/Source 授权接线。公共知识库也不等于匿名
公网开放,Admin 写操作只允许经受控网络或统一接入网关访问,MCP 使用最小只读服务身份。
本次迁移保留 Demo 的 Dense Top20 → Reranker Top5 行为和当前轻量 Parser 范围。Top50 检索、
距离阈值、双 Query RRF、Marker/Pandoc 和 LLM 派生索引内容,需要分别作出产品与评估决策。

View File

@ -0,0 +1,408 @@
# 企业内部智能知识库 V1 技术架构
> 生命周期:权威、活跃。产品范围以 [`../product/PRD.md`](../product/PRD.md) 为准,架构决策状态以 [`ADR.md`](ADR.md) 为准。本文只描述当前技术结构和模块边界,不记录任务流水账或操作步骤。
## 1. 建设目标
平台提供两条相互关联的知识链路:
```text
Source → Parse/OCR/Clean → Chunk → Index → Retrieval → REST/MCP
Source/L1 → Knowledge Compilation → Review/Publish → Compiled Retrieval
```
V1 最终目标包括多格式入库、Hybrid Retrieval、Reranker、ACL、来源追踪、Knowledge Compilation、REST、MCP 和管理控制台。当前已从公共兼容域演进到可信 REST 身份与 Knowledge Base/ACL 运行边界;Knowledge Compilation 等差距见 [`../plans/TASKS.md`](../plans/TASKS.md)。
## 2. 架构不变量
1. API First:管理端、MCP 和其他系统复用同一业务服务。
2. V1 采用模块化单体,不拆分无必要的微服务。
3. OceanBase MySQL 关系表与 BOS 保存规范数据;OceanBase 向量表仅为可重建索引。
4. Embedding、Reranker、VectorStore、Parser、OCR、ObjectStorage 和 LLM 均通过 Provider 接口隔离。
5. 原始 Source 不可变;Parsed、Chunk、Index 和 Compiled Knowledge 均为可追溯派生数据。
6. ACL 必须在检索前或检索过程中解析并下推,禁止先做全局 Top-K 再过滤。
7. MCP 只暴露业务语义,不暴露数据库、存储、模型或 Parser 实现。
8. Index Profile 必须版本化;模型或 Chunk 变更先构建 Shadow Index,验证后再激活。
这些约束由 [`ADR.md`](ADR.md) 和根目录 [`../../AGENTS.md`](../../AGENTS.md) 共同约束。
## 3. 总体拓扑
```text
┌──────────────────┐
│ Admin Console │
└────────┬─────────┘
│ REST
▼
WorkBuddy ─ MCP ─▶ MCP Gateway ─▶ Knowledge Application Services ◀─ REST ─ Other Systems
│ │
┌────────────┘ └────────────┐
▼ ▼
OceanBase MySQL + Vector BOS
规范关系表 / 可重建索引 Source / 派生产物
▲
│
Celery / RabbitMQ
│
Async Worker
│
Parser / Embedding / Index
```
逻辑进程边界:
- API/Admin:FastAPI 提供 REST、静态管理端、健康检查与指标。
- MCP Gateway:独立进程,负责协议、认证与业务 Tool 映射。
- Async Worker:独立 Celery Worker,执行持久化 Ingestion 与 Compilation Job。
- Recovery Scheduler:单实例调度过期租约和待重投递 Job。
- Provider:BOS、OceanBase Vector、千帆等外部能力的 Adapter。
业务核心仍属于一个模块化单体,各进程复用同一 Domain 和 Application Service。
## 4. 代码模块
```text
apps/
├── api/ FastAPI、REST、Provider 装配、Celery 入口
├── mcp-server/ MCP Gateway、认证、Tool 映射
└── admin-web/ 同源静态管理端
packages/
├── domain/ SQLAlchemy 领域模型与值对象
├── core/ 应用服务和业务编排
└── providers/ ObjectStorage、Parser、Embedding、Reranker、VectorStore Adapter
migrations/ OceanBase MySQL 规范关系表 Migration
deploy/ 容器、Helm 与集群交付物
tests/ 单元、集成和 RAG 回归
```
`workers/`、`packages/shared/` 和 `packages/retrieval/` 当前是未来模块占位目录;现有 Worker 实现在 `apps/api` 中,不应把占位目录误认为独立服务。
`ekp_domain.base` 是全部规范模型唯一的 SQLAlchemy Metadata 根,统一约束命名和 OceanBase/SQLite
可移植字符串 Enum。领域包只依赖 Python/SQLAlchemy 和自身模块,不得反向依赖 `apps`、`ekp_core`
或 Provider;该边界由结构测试保护。新增领域模型及表结构仍分别进入对应模型与 Migration Task。
规范 Schema 使用单 Base、单 Head 的线性 Alembic 历史向前演进。已发布 Revision 由 SHA-256 清单
显式锁定;新增表或字段必须追加 Revision,不得重写既有历史。离线门禁使用 MySQL Dialect 验证
关系表 DDL,不把独立管理的 OceanBase Vector 表纳入 Alembic 所有权。
## 5. 数据所有权
| 存储 | 保存内容 | 是否事实源 |
|---|---|---|
| OceanBase MySQL 关系表 | Organization、User、Group、Role、成员/角色关系、KnowledgeBase、Folder、ACL、Document、DocumentVersion、Chunk、Job、Attempt、Batch、AuditLog、Index Profile、ChunkIndexState | 是 |
| BOS | 不可变原始文件、解析 JSON 和其他大对象产物 | Source 是;派生产物可重建 |
| OceanBase Vector 表 | Dense Vector 与检索过滤 Metadata | 否,可由规范 Chunk 重建 |
| RabbitMQ | Job 投递提示 | 否,Job 状态以数据库为准 |
物理共库不改变所有权边界。业务服务不得从向量行反向恢复规范业务状态,也不得绕过 `VectorStore` 直接依赖 `pyobvector`。
## 6. 文档入库
### 6.1 接收边界
API 完成以下操作后返回 `202 Accepted`:
```text
验证输入并流式计算 Source Hash
→ 保存不可变 Source
→ 持久化 DocumentVersion/Job
→ 投递 Job
```
当前支持 PDF、DOCX、PPTX、XLSX、CSV、HTML/HTM、TXT 和 Markdown;显式启用 MinerU 后额外
支持 JPG/JPEG/PNG 与扫描件 OCR。文件大小、批量数量和批量总大小分别由运行时配置限制。批次
只负责提交和聚合状态,每个文件拥有独立 Document 与 Job,不构成跨文件事务。浏览器文件夹上传
额外携带与文件严格同序的安全相对路径;服务端在已授权 Knowledge Base/基准 Folder 内逐段创建或
复用规范 Folder,只持久化校验通过的路径。路径逃逸只拒绝对应文件,跨 Knowledge Base 基准 Folder
则在任何批次、目录和对象写入前失败关闭;空目录不创建占位资源。
ZIP 导入在该 Batch 边界前增加完整结构与资源预算预检;包结构不安全时不产生任何规范写入,安全
条目以延迟流分别进入既有上传服务,ZIP 自身与平台元数据不成为 Source。不同 Folder 的同名文件
使用独立 Document 身份,同一 Folder 内继续使用不可变 DocumentVersion 演进。
`Document` 是稳定逻辑身份,`DocumentVersion` 是不可变、带序号的 Source 身份;`documents`
只保留 Active Version 投影以兼容列表与详情 API。正式入库在同一 Knowledge Base/Folder 内以文件名归并
逻辑 Document,并以 SHA-256 Source Hash 去重:Hash 相同返回已存在提示,同名而 Hash 不同创建 inactive 的下一
版本。数据库生成列唯一索引保证每个 Document 最多一个 Active Version。Chunk、Parsed Artifact、
Job 和向量行均绑定具体 Version;只有索引完整成功后才切换关系与向量 Active Marker,失败候选不
影响旧 Active Version。不同 Knowledge Base 使用独立唯一范围;旧未分配数据保留在隔离的
`public/public` 兼容范围,不自动迁入任意 KB。
同名候选的 `202` 响应描述新接收的 Version,同时保留稳定 Document ID;旧 Active Document 投影
在新索引成功前不变。多 API 副本首次并发创建同名 Document 时,数据库唯一约束选出胜者,失败方
只在确认同名胜者存在后有界重试 Version Metadata,Source Hash 相同则仍收敛为重复 Source。
### 6.2 Worker 流水线
```text
Parsing → Cleaning → Chunking → Embedding → Indexing
```
- 默认 PDF 使用 PyMuPDF,DOCX 使用 python-docx,TXT/Markdown 使用轻量 Parser。显式选择
MinerU Provider 后,PDF、DOCX 和 JPG/JPEG/PNG 进入自托管 `/file_parse` 的 `auto` 路由,
TXT/Markdown 保持轻量实现;不存在静默 Provider 回退。
- Docling 本地 Adapter 负责 PPTX、XLSX、CSV 和 HTML/HTM;它保留 reading order、章节、Page、
Sheet、Table Cell 坐标和 Image Reference。DOCX 继续由轻量 Parser 或显式 MinerU 处理,避免
同一格式的隐式 Provider 切换。
- `Parser` 是可运行时检查的结构协议;`ParserRouter` 持有不可变扩展名能力集合,上传准入直接
使用该集合。Router 同时校验 Provider 输出的 Document ID 与文件名,拒绝错误 Source 绑定。
- Parsed Artifact 使用 schema v4 版本化 JSON;每个结构节点显式绑定 Source Document 和数字
Version,并保留 Page、Section、类型、内容、页级 OCR Confidence 和低置信度警告;旧 schema
v1~v3 仍可读取且不伪造缺失结构。
- Cleaning 处理 Unicode/控制字符、空白和异常换行、Latin 连字符断词、标点噪声与精确重复块;
MinerU 辅助页眉页脚在映射阶段移除,Original Source 不被修改。
- `structure_v2` 保留 Title/Heading/Section 硬边界和约 450 Token、75 Token Overlap;Table 不与
叙述文本或其他 Sheet 混合,窗口重复列标题,并把一基闭区间 Sheet/Row/Column 来源写入规范
Chunk。`structure_v1` 只用于既有 Active Profile 的兼容读取与安全切换。
- 文档只有在全部 Chunk 完成索引后才进入 `ready`。
- 流水线失败不会删除不可变原始文件。
### 6.3 Job、重试与恢复
OceanBase 中的 `IngestionJob`/`IngestionAttempt` 与 `CompilationJob`/`CompilationAttempt` 分别是
入库和编纂执行事实源。Broker 重投递、Worker 重启和租约恢复必须收敛到同一状态。每次 Worker
Claim 都写入新的 Lease Fencing Token;只有仍持有当前 Token 且 Attempt Number 匹配的执行者可以
结束 Job,过期 Worker 恢复运行时只能安全退出。Compilation Job 仅持久化规范化意图和有序 Chunk
标识;执行时重新解析提交者的 KB Write、全部来源 Document Read、Active Version 与数据策略。
生成 Draft 通过唯一 `generation_job_id` 与 Job 一对一关联,从数据库层收敛重复投递。
瞬时错误执行有界重试。下一次 Attempt 开始前按以下顺序清理派生数据:
```text
Vector/Sparse rows
→ ChunkIndexState
→ Chunk
→ Parsed Artifact / parsed_storage_key
→ 重置 Document 状态
```
清理只作用于 Job 绑定的 DocumentVersion,保留全部不可变 Source、Document、历史 Version、Job
和 Attempt。清理失败时必须终止,禁止在未知脏状态继续执行。详细运行方式见
[`../development/ASYNC_INGESTION_SETUP.md`](../development/ASYNC_INGESTION_SETUP.md)。
AuditLog 规范模型保存 User/Service/System/Anonymous Actor、REST/MCP/Worker/Scheduler/System
来源、Action、Outcome、稳定资源引用和脱敏详情。资源引用不设删除级联,以保留历史证据;
Compiled 生命周期及 Compilation Job 接收、成功和失败已接入业务写点,完整查询 API 与
Append-only 数据库权限仍属于后续治理范围。
## 7. 索引与检索
### 7.1 当前实现
```text
Query
→ Normalize
→ Active Profile + Active DocumentVersion 预过滤范围内 Dense Top20 + Sparse Top20
→ Weighted Reciprocal Rank Fusion Top20
→ Reranker
→ TopK
→ 从规范 Chunk 补全来源
```
默认基线:
| 能力 | 当前实现 |
|---|---|
| Embedding | 千帆 `bge-large-zh`,1024 维 |
| VectorStore | OceanBase Vector |
| SparseStore | OceanBase `MATCH ... AGAINST` + NGRAM Full-Text Index |
| Fusion | Weighted RRF,`k=60`,Dense/Sparse 默认等权,输出 Top20 |
| Reranker | 千帆 `bce-reranker-base` |
| 来源 | Document/Version、文件名、页码、完整章节路径、Sheet/行列范围、精确 Chunk、有界上下文、Raw 类型、Fusion/Reranker 分数 |
`index_profile_id + is_active` 必须在 Dense ANN 或 Sparse 排序的 TopN 截断前过滤;候选的 Version
非 Active,或索引内容与规范 Chunk 不一致时失败关闭,不能返回历史版本或无法验证的来源。
Dense 与 Sparse 复用 `knowledge_chunks_v1` 可重建行及其激活、删除和 Shadow Profile 生命周期;
关系 Chunk 仍是规范数据。测试 Fake 使用确定性 BM25,OceanBase 使用原生 Full-Text 相关度,
两者分数只在各自候选列表内排序,不作为跨 Provider 的绝对值契约。Fusion 使用名次而不是原始
分数,并按 Chunk 去重;跨路来源标识或正文冲突时失败关闭。
Provider-neutral VectorStore 对 Row、Query、Profile/Version 范围和运行时实现形状执行统一校验;
确定性 Fake 在 Profile + Active Version 预过滤后执行稳定精确 Cosine,并拒绝维度/模型漂移、
Source 身份改绑和同 Document/Profile 的多个 Active Version。厂商 Adapter 不得降低这些边界。
Dense 应用服务还会在任何 Provider 调用前校验 Query/Top-N,空授权范围不发送 Query,并要求
Query Instruction 与 Active Profile 一致;返回后再次验证来源身份、唯一 Chunk、Cosine 范围、
降序、Profile 和 ACL Scope。
OceanBase Adapter 在启动时只读验证最低版本与预建 Dense/Sparse Schema,版本切换先锁定并确认
目标行再于同一事务更新 Active Marker;ANN 查询显式要求 `pre_filter`,所有值均使用绑定参数,
厂商输出在进入业务服务前校验来源、分数、排序和唯一身份。
最终 Search Service 固定使用 Active Index Profile 的 Hybrid TopN、Reranker Model 与 Final TopK;
单次请求只能缩小、不能扩大 Profile 的最终结果数。选中候选必须在规范关系库中同时找到匹配的
Chunk、该 Profile 的 ChunkIndexState 和 Ready Active DocumentVersion,并复验模型、维度、Chunk
策略、来源身份、文件名与正文。任一证据缺失或漂移即失败关闭。`context` 只读取同 Version、同
策略相邻 Chunk,最多 2400 字符且保留命中正文;它改善可读性,但不参与排序或替代不可变原文。
离线质量门禁读取哈希绑定的版本化 Golden Dataset,通过与生产相同的 Parser、Chunk、Dense、
Sparse、Fusion、Reranker 和规范来源补全边界生成聚合报告;基础设施由 SQLite、Local Object
Storage 和确定性 Fake Provider 替代。评测模块不依赖具体生产模型,报告用 Dataset 版本和 Index
Profile 标识输入,避免不同语料、Chunk 或模型结果被错误横向比较。
### 7.2 V1 目标
完整检索链仍需实现:
```text
Query Normalize
→ Permission Resolver
→ Dense + BM25/Sparse
→ Fusion
→ Candidate TopN
→ Reranker
→ TopK
→ Source Enrichment
```
Dense、Sparse、Hybrid Fusion 及 Knowledge Base/Folder/Document ACL Scope 已完成;MCP 与 REST
正式身份均接入授权 Search,REST 上传显式接收 KB/Folder Scope,管理入口维护规范身份与 ACL。
### 7.3 Index Profile
Index Profile 是完整且不可变的检索配置快照,绑定 Chunk 窗口、Embedding 模型/输入上限/Query
Instruction/向量维度与 Float32 存储、Sparse 策略、Dense/Sparse/Hybrid 候选规模、Weighted RRF、
Reranker 模型和最终 TopK。运行时从数据库 Active Profile 读取这些值,避免环境变量与已建索引
语义漂移;Credential、Endpoint、业务 Query 和正文不进入 Profile。生命周期为:
```text
shadow → active → retired
```
新配置必须创建新 Profile,使用规范 Chunk 构建 Shadow Index,核对全部 `ChunkIndexState` 后原子激活。任一时刻最多一个 Active Profile。操作和回滚边界见 [`../development/INDEX_REBUILD.md`](../development/INDEX_REBUILD.md)。
字段和数据库约束见 [`../development/INDEX_PROFILE.md`](../development/INDEX_PROFILE.md)。
`structure_v2` 切换时,受控 Build 会先从 Active Version 的 Parsed Artifact 幂等生成目标策略
Chunk,再构建 Shadow Index;Verify 必须把缺少目标 Chunk 的 Active Version 计为不完整。运行时
始终使用数据库登记的 Active Profile,v1 尚未激活到 v2 时不得接收新增 Docling 格式。
## 8. Provider 边界
| 接口 | 当前 Provider | 后续能力 |
|---|---|---|
| `ObjectStorage` | 百度 BOS;本地 Fake/Adapter 用于测试 | 其他对象存储 |
| `Parser` | 运行时可检查协议与不可变 Router;PyMuPDF、python-docx、文本/Markdown、本地 Docling Office/Spreadsheet/HTML;可选 MinerU HTTP OCR | Parser Benchmark 与性能加固 |
| `EmbeddingProvider` | 运行时可检查协议、统一能力/向量门禁与确定性 Fake;千帆 BGE 的安全 HTTPS/错误分类 Adapter;未激活的 Qwen3 OpenAI-compatible Adapter | 本地模型与 Provider Benchmark |
| `RerankerProvider` | 运行时可检查协议、统一请求/结果门禁与确定性 Fake;千帆 BCE 的安全 HTTPS/错误分类 Adapter | Qwen 或本地 Reranker Benchmark |
| `VectorStore` | OceanBase Vector | 其他向量数据库 Adapter |
| `LLMProvider` | Provider-neutral 协议、OpenAI-compatible Adapter、确定性 Fake;运行时可选 | 后续 Provider 扩展与评测 |
配置、Credential 和厂商响应只能停留在运行时装配与 Provider Adapter 中,不得进入 Domain、REST 或 MCP 契约。
## 9. 权限与身份
V1 权限主体包括 User、Group、Role,资源层级包括 Organization、Knowledge Base、Folder 和 Document。标准链路为:
```text
Identity → Permission Resolver → Allowed Scope → Provider-neutral Retrieval Filter
```
REST、MCP 和管理端必须复用同一权限服务。
当前已建立 Organization、User、Group、Role 及同 Organization 成员/角色关系的规范模型,并以
类型化复合外键保存 Organization/KB/Folder/Document ACL。内部 `PermissionResolver` 已展开 Active
User/Group/Role、解析同 KB Folder 父链和 Deny/Allow,形成精确 Document Scope;Dense/Sparse
Adapter 在各自 Top-N 前消费同一个 Provider-neutral Filter,`AuthorizedSearchService` 固化授权编排。
`EKP-082` 提供 REST User Token/Service API Key 映射、显式首身份 Bootstrap、User/Group/Role 与
成员/角色关系、KB/Folder/ACL 管理 API,并在 Document/Batch/Job/Source/Search 前复用 Permission
Resolver。`EKP-070` 的 MCP 身份继续使用同一规范授权数据并限制 `mcp_enabled` Knowledge Base;
外部 IdP 仍是可替换的 Token Verifier 扩展点,当前运行配置不负责签发 Token。
KnowledgeBase 已保存同 Organization Owner、状态、可见性意图、数据处理策略、MCP Availability
与 Metadata Schema。Folder 已以 Organization/KB 复合范围保存树结构,并拒绝跨 KB 父子关系。
Document 提供 Organization/KB/Folder 复合范围,使 ACL 无法引用未分配的旧公共 Document;正式
上传必须选择有 Write 权限的 KB,并把同一范围复制到不可变 DocumentVersion。未分配范围不是授权
默认值,旧公共数据不会自动归入任意 Organization,也不会创建占位授权数据。
## 10. 对外接口
### REST
公开 API 统一位于 `/api/v1`。当前提供 Document 上传/查询/删除、规范 Chunk 正文读取、不可变
Source 访问、Batch、Job 和 Search 接口;错误响应不得回显 Provider、数据库、存储或 Credential。
完整行为以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
### MCP
独立 Streamable HTTP Gateway 已注册 `search_knowledge`、`get_document`、
`get_document_section`、`get_source` 与 `list_knowledge_bases` 五个 P0 Tool,复用与 REST 相同的
Search、Permission 和 Knowledge Access 应用服务。正式运行时 Bearer Token 支持 User Token/
Service API Key,并绑定规范 Organization/ACL User;旧静态凭据只保留公共 Search Demo 兼容。
`get_source` 返回精确规范 Chunk 证据而非原文件字节,原文件仍由受控 REST Source 接口流式读取。
外部 IdP 和远程现场验收尚未完成。完整契约见 [`../api/MCP_CONTRACT.md`](../api/MCP_CONTRACT.md)。
### 管理控制台
当前管理端是同源静态客户端,提供运营总览、批量上传、文档列表、Document/Chunk Inspector、
Search Lab、Knowledge Base/Folder 及 User/Group/Role/ACL 管理。运营总览复用公开 Document 列表
按状态投影知识流库存和最近来源,不保存独立聚合状态;
Inspector 以不可变 Source 预览和规范 Chunk 溯源组成双视图:PDF 保持原版式并从命中起始页打开,
Markdown/纯文本连续阅读,图片在线预览,Office/HTML 安全下载;跨页命中同时显示起止页和打开页。
Search Lab 区分可读上下文与精确命中 Chunk,并通过一次实际 Search 执行展示 Query、Scope/ACL、
Active Profile、Dense、Sparse、Fusion、Reranker、Final 与各阶段耗时。它不承载或复制检索业务
逻辑,所有操作均调用 `/api/v1`;诊断响应不含 Endpoint、Credential 或厂商原始数据。
页级质量来自 Version 绑定的 Parsed Artifact,Job/Attempt 来自规范关系表;质量派生数据不可读
不会隐藏规范 Chunk。管理端 Token 只保存于当前标签页,并以认证 Fetch/Blob 打开不可变 Source。
Compiled Knowledge 页面允许从可读规范 Chunk 创建/生成草稿、编辑和回到 Document Inspector 核对
来源;EKP-091 已接入来源当前性、冲突证据、审核、退回、发布与 Stale 修订。EKP-092 已把仍可
核验的 Published 内容接入正式 REST/MCP Search,并在六阶段 Search Lab 中保留全部底层来源。
Benchmark 作为独立非开发队列记录。
## 11. Knowledge Compilation
长期知识模型:
```text
L0 Source:不可变原始事实
L1 Structured:Parsed / Chunk / Index
L2 Compiled:Wiki / FAQ / SOP / Summary / Comparison
```
EKP-090 已建立 Organization/Knowledge Base 范围的 `CompiledKnowledge` 和有序
`CompiledKnowledgeSource`,每个草稿至少绑定一个确切 Document、DocumentVersion 与 Chunk。读路径
先解析 KB 和全部来源 Document 的权限交集;写路径要求目标 KB Write 加来源 Read。手工草稿与可选
LLMProvider 生成均只进入 Draft,AI 结果不会覆盖既有内容。`local_only` KB 不得调用声明为 cloud 的
Provider;OpenAI-compatible Adapter 仅接受安全 Endpoint、Bearer Secret 与严格 JSON title/content,
厂商字段不进入 Domain/API。
EKP-091 在同一规范事务内把 Source 新版本激活、被引用 Document 删除及 ACL 增删传播为稳定的
Stale 状态;被引用 Document 使用 tombstone 逻辑删除,退出检索和普通 Document API,但保留不可变
Source/Version/Chunk 证据。审核人把冲突主张绑定到两个确切 Source,未解决冲突阻断发布;AI 不会
自动折中或发布。Compiled 读取继续动态执行 KB 与全部来源 Document 的权限交集。EKP-092 只从
检索前已授权的 Raw Source 候选扩展 Published L2,按全部来源权限交集过滤后与 Raw 共同精排;
最终结果再次核验规范 Source,并返回精确 `underlying_sources[]`。当前不建立未版本化的独立 L2
向量索引,Compiled Recall 继承底层 Raw Source 召回。
## 12. 部署与可观测性
生产逻辑角色:API/Admin、MCP Gateway、Async Worker、Recovery Scheduler,以及可选 Edge Proxy。外部依赖包括 OceanBase、BOS、RabbitMQ 和 AI Provider。
当前开发交付已完成 API/Admin、Worker 和检索公网烟测;远程 MCP/WorkBuddy 现场验收进入非开发
独立队列。这不删除 MCP Gateway 进程边界,也不改变 ADR-002。实际 GitOps Values、Secret、
Namespace、工作负载副本、滚动/回滚、监控和备份由运维人员管控,开发仓库继续提供对应 Helm
能力、运行契约和验收工具。
Helm 基线要求:
- API/MCP 默认多副本,Worker 可独立扩缩;
- 串行 Migration Job;
- 生产 target 只引用预建 Secret,不在 Values 或镜像中保存凭据;当前中央 dev 流水线的默认
`dev` target 是经明确授权、限制在 dev Registry/环境的例外;
- Readiness/Liveness、资源限制、非 root、只读根文件系统和 PDB;
- Ingress/TLS,或在无 Ingress 场景使用仅负责路径代理的 Edge;
- 字段白名单 JSON 日志、请求关联 ID 和低基数 `/metrics`。
部署命令和运维步骤分别见 [`../development/KUBERNETES_DEPLOYMENT.md`](../development/KUBERNETES_DEPLOYMENT.md) 与 [`../development/CLUSTER_OPERATIONS.md`](../development/CLUSTER_OPERATIONS.md)。当前远程验收状态见 [`../development/DEPLOY_003_ACCEPTANCE.md`](../development/DEPLOY_003_ACCEPTANCE.md)。
## 13. 验收边界
完整 V1 只有同时满足以下条件才可完成:
- P0 文件完成 Upload → Parse/OCR → Clean → Chunk → Embed → Dense/BM25 Index;
- Hybrid + Reranker + ACL 检索通过质量与性能门禁;
- 每条结果可追踪 Source、Version、Page、Section 和 Chunk;
- REST、MCP、管理端和 Source 下载执行同一授权边界;
- Knowledge Compilation 完成 Source Link、冲突/Stale、审核和发布;
- Golden Dataset、Provider Benchmark、安全、可观测性、备份和重建通过验收。
当前 Demo/Deploy 纵向切片不能自动完成对应 `EKP-*` 任务。完成标准以 [`../development/DEFINITION_OF_DONE.md`](../development/DEFINITION_OF_DONE.md) 和 [`../plans/TASKS.md`](../plans/TASKS.md) 为准。

View File

@ -0,0 +1,165 @@
# EKP 知识检索 Demo v0.1
> 生命周期:历史基线、冻结。本文只定义 Demo 验收边界,不再同步当前部署状态。当前任务和实现状态分别以 [`../plans/TASKS.md`](../plans/TASKS.md) 与 [`../plans/PROGRESS.md`](../plans/PROGRESS.md) 为准。
## 1. 目标
Demo 用于验证最小知识入库与检索闭环:
```text
文件上传
→ 文档解析与基础清洗
→ 结构感知 Chunk
→ Embedding
→ OceanBase Vector
→ Dense Top20
→ Reranker Top5
→ 来源结果
→ 简单管理页面
```
成功标准是核心链路可用并具备演进基础,不是完整企业级知识平台完成。
## 2. 架构
```text
Admin Web ─ REST ─ FastAPI
├─ Upload → Parser → Cleaner → Chunker → Embedding → VectorStore
└─ Search → Query Embedding → Dense TopN → Reranker → Source
OceanBase MySQL:规范 Document、Chunk、Job 与索引状态
BOS / Local:不可变原始文件和解析产物
OceanBase Vector:可重建 Dense 检索索引
```
Provider 迁移后,目标运行时为 OceanBase MySQL + BOS + OceanBase Vector + 千帆模型。业务层必须通过 Provider 接口访问基础设施。
## 3. P0 范围
### 3.1 文件与解析
支持:
- 文本型 PDF:提取正文并尽量保留页码;
- DOCX:提取标题、段落和基础章节结构;
- UTF-8 TXT/Markdown:直接读取并保留 Markdown 标题层级;
- 单文件与有界批量上传;
- 原始文件不可覆盖。
暂不支持 PPTX、XLSX、CSV、HTML、图片、扫描 PDF、OCR 和复杂版面恢复。
### 3.2 清洗与 Chunk
基础清洗处理多余空行、连续空格、异常换行和首尾空白,不修改原始 Source。
Chunk 优先按标题、章节和段落组合,再按 Token 限制切分;默认约 400~500 Token,并保留 50~100 Token Overlap。每个 Chunk 至少包含:
```text
chunk_id
document_id
filename
page / page_end
section / section_path
chunk_index
content
strategy_version
```
### 3.3 Embedding、索引与检索
- `EmbeddingProvider` 当前使用千帆 `bge-large-zh`;
- `VectorStore` 当前使用 OceanBase Vector;
- Dense Retrieval 默认召回 Top20;
- `RerankerProvider` 当前使用千帆 `bce-reranker-base`;
- 最终返回 TopK,并保留 Dense 与 Reranker 分数;
- 来源必须从规范关系表中的 Chunk 补全,陈旧或不一致索引失败关闭。
结果至少包含:
```text
document_id
filename
page
section
chunk_id
content
score
rerank_score
```
### 3.4 异步 Job 与批量接收
Source、Document 和 Job 持久化后,API 返回 `202`,不等待解析和索引。Celery/RabbitMQ Worker 执行流水线,OceanBase Job/Attempt 是执行事实源。
瞬时失败执行有界重试;重试前清理 Parsed Artifact、Chunk、ChunkIndexState 和 Vector Row,同时保留原始 Source、Document、Job 与 Attempt 历史。批次按文件隔离,单个文件失败不得回滚其他文件。
## 4. REST API
当前 Demo 接口统一位于 `/api/v1`:
| 方法 | 路径 | 行为 |
|---|---|---|
| `POST` | `/documents` | 单文件异步接收 |
| `POST` | `/ingestion-batches` | 多文件异步接收 |
| `GET` | `/ingestion-batches/{batch_id}` | 批次与逐文件状态 |
| `GET` | `/jobs/{job_id}` | Job、Stage、Attempt 和安全错误信息 |
| `GET` | `/documents` | 文档列表 |
| `GET` | `/documents/{document_id}` | 文档详情 |
| `DELETE` | `/documents/{document_id}` | Demo 直接删除 |
| `POST` | `/search` | Dense → Reranker → 来源结果 |
公开契约以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
## 5. 管理控制台与 MCP
管理控制台只提供:
1. 批量上传与处理状态;
2. 文档列表、详情和删除;
3. Search Lab,展示来源、Chunk、Dense 分数和 Reranker 分数。
MCP Gateway 只暴露 `search_knowledge`,复用 REST 相同的 `SearchService`,并使用最小服务身份。该能力不代表完整 MCP P0 Tool 或用户委托 ACL 已完成。MCP 契约见 [`../api/MCP_CONTRACT.md`](../api/MCP_CONTRACT.md)。
## 6. 明确后置
Demo 不完成以下 V1 能力:
- 多租户、User/Group/Role、Knowledge Base、Folder 和 Document ACL;
- LDAP、OIDC、SSO、审计日志和 Source 下载授权;
- DocumentVersion、Source Hash 去重、软删除;
- BM25、Hybrid Fusion 和查询路由;
- OCR、MinerU、Docling 和复杂表格/版面;
- Knowledge Compilation、审核、冲突和 Stale;
- 多 Embedding/Reranker Benchmark 平台;
- MCP 其余 P0 Tool;
- 完整 Observability、跨区域灾备和企业级生产加固。
公共知识域不等于匿名公网。开始接收任何非公共知识前,必须先完成 Permission Resolver 和检索前/检索中 ACL 下推。
## 7. Task 与完成状态
Demo 纵向切片包括:
```text
DEMO-001~010
DEMO-012
DEMO-014~016
ARCH-001~006
PLAN-001~002
DEPLOY-001~003
```
截至本基线冻结时,Demo、Provider 迁移、MCP、异步批量入库和 DEPLOY-001~002 已完成;DEPLOY-003 仍进行中。最新状态只在 [`../plans/TASKS.md`](../plans/TASKS.md) 更新。
## 8. Demo 完成定义
- PDF、DOCX、TXT、Markdown 可以成功接收;
- Upload → Parse → Clean → Chunk → Embed → Index 全链路完成;
- Query → Dense Top20 → Reranker Top5 完成;
- 每条结果包含文件名、页码、章节、Chunk 内容和两阶段分数;
- 批量接收具备逐文件隔离、持久化状态和失败恢复;
- 非研发用户可以通过管理端完成上传、等待处理、检索和查看来源;
- 默认自动化测试不依赖真实云服务、互联网或 GPU;
- 检索变更运行 Golden Dataset 回归。
完整任务完成标准以 [`../development/DEFINITION_OF_DONE.md`](../development/DEFINITION_OF_DONE.md) 为准。

View File

@ -0,0 +1,207 @@
# Admin Console 使用说明
管理端在 FastAPI 服务中提供一个轻量、同源的管理客户端。页面只负责展示和交互,知识总览、上传、
文档管理与检索仍全部通过 `/api/v1` 公开业务接口完成,不复制服务端业务逻辑。
## 启动与访问
完成运行时配置后启动 Worker 与 API:
```bash
make run-worker
make run
```
页面地址:
| 页面 | 地址 | 对应业务接口 |
|---|---|---|
| 知识总览 | `/admin`、`/admin/overview` | `GET /api/v1/documents` |
| 文档导入 | `/admin/upload` | `POST/GET /api/v1/ingestion-batches*` |
| 文档管理 | `/admin/documents` | `GET/DELETE /api/v1/documents*` |
| 知识检索 | `/admin/search` | `POST /api/v1/search/diagnostics` |
| 知识库管理 | `/admin/knowledge-bases` | `GET/POST /api/v1/administration/knowledge-bases*` |
| 权限管理 | `/admin/permissions` | `GET/POST/PUT/DELETE /api/v1/administration/*` |
`/admin` 默认展示知识总览。OpenAPI 文档仍位于 `/api/v1/docs`。
## 品牌视觉基线
管理台延续地瓜机器人公开官网的设计语言,使用品牌橙 `#FF3C00`、黑白高对比、中文
无衬线标题、紧凑数据字体、低圆角容器和宽松留白。贯穿页面标题的“计算信号轨”是唯一的
强调性图形。`EKP-080` 用 Source → Processing → Ready → Retrieval 运行轨道编码真实知识处理顺序,
一级导航按“知识运营”和“系统管理”分组,不把并列功能伪装成固定步骤。
品牌图形使用仓库内的代码原生 SVG/CSS 绘制,不复制或热链官网图片、字体和脚本;页面
运行时不依赖品牌官网。前端设计方法参考 GitHub `anthropics/skills` 的 Apache-2.0
`frontend-design` Skill,并以当前产品信息架构、可访问性和业务正确性为优先约束。
## 核心操作闭环
1. 从知识总览查看全部、处理中、可检索文档和就绪率;处理失败时直接打开最近文档的证据链,
或进入文档管理页;
2. 首次进入时输入运行环境分配的 REST Token;Token 只保存在当前标签页 `sessionStorage`,经
`/administration/session` 验证后,页面显示规范 User 和 Organization,并按返回的 Scope 隐藏
上传、删除、知识库和权限管理入口;
3. 在知识库管理页展开“知识库根目录”和有子项的 Folder,沿缩进与分支轨道查看真实父子关系;
目录树保留知识库可见性、数据处理策略、MCP 状态与文件夹计数。“新建知识库”或知识库卡片中的
“新建文件夹”继续打开独立弹窗;在权限管理页通过对应操作按钮打开用户、用户组、角色、成员
关系、角色关系和授权规则弹窗。页面主体只用于查看资源和关系,不与新增表单混排;
4. 在上传页选择目标 Knowledge Base/Folder,再使用独立的“选择文件”或“选择文件夹”入口提交;
文件夹模式保留所选根目录及其子目录层级,每个文件仍独立进入 Job。支持 PDF、DOCX、PPTX、
XLSX、CSV、HTML/HTM、TXT 或 Markdown;服务端显式启用 `PARSER_PROVIDER=mineru` 后还可选择
JPG/JPEG/PNG,扫描 PDF 由 MinerU `auto` 判定;
5. 接口返回 `202` 后,页面自动轮询批次,逐文件显示接收状态、处理阶段、尝试次数和安全错误;
6. 批次进入 `Completed`、`Partial failure` 或 `Failed` 后停止轮询;在文档管理页可先按可读
Knowledge Base 联动选择全部文件夹、知识库根目录或某个 Folder 子树,核对当前范围与结果数量,
再点击“查看文档”。Inspector 默认展示安全的原始文件预览,也可切换到按来源顺序排列的已索引
解析 Chunk;
7. 在知识检索页输入问题并查看 Query、Scope/ACL 实际状态、Active Profile、Dense、Sparse、
Fusion、Reranker、Final 候选和阶段耗时;最终 Top-K 继续展示文件名、页码、章节、Sheet/行列
范围、相邻 Chunk 上下文和精确命中;“定位命中来源”进入双视图 Inspector,
“查看或下载原始文件”在新窗口读取可预览的不可变 Source,不支持浏览器预览的二进制格式保留
原始文件名下载;
8. 如需删除文档,在文档管理页确认后直接删除。
## 范围与安全边界
- 页面静态文件位于 `apps/admin-web`,由同一个 FastAPI 进程提供,避免额外前端运行时;
- 页面保持零构建、无第三方 npm 依赖;Node/npm 只用于开发期 JavaScript 模块语法、锁文件和本地
版本化资源引用检查,不进入 API 或生产浏览器运行时;
- API 返回内容仅通过 DOM `textContent` 渲染,不作为 HTML 注入;
- 所有业务请求发送 Bearer Header;浏览器不持久保存 Token,原文预览/打开使用认证 Fetch 与短生命
周期 Blob URL,不把 Token 放入 URL;关闭标签页即清除身份;
- 低权限用户的上传/管理导航和文档删除按钮使用 Session Scope 真实收敛;CSS 对语义 `hidden`
使用强制隐藏规则,静态资源带任务版本标识,避免旧缓存重新暴露已隐藏入口;
- 文档范围筛选只使用 `/documents` 的 ACL 交集查询,不在客户端缓存全量资源后过滤;Folder 选项
来自已授权 KB 的 Metadata 接口,选项显示祖先路径并明确“含子文件夹”。切换身份会重新解析可读
Knowledge Base,未知、跨 KB 或越权范围不会降级为无约束列表;
- 知识库管理把现有 Folder API 的 `parent_folder_id` 平面投影在浏览器中派生为只读目录树,不复制
服务端层级事实。知识库根节点和含子项 Folder 使用原生 `details/summary`,支持鼠标与键盘展开;
深层和长名称保持单行省略并通过完整路径提示保留语义,移动端隐藏次要计数而不丢失层级;
- 页面不保存或展示 Provider 凭据、内部 Endpoint 或 storage key;知识检索页只展示诊断所需的
Active Index Profile 与模型标识,不返回 Provider 原始配置;
- Console 上传页只轮询当前批次的聚合接口;Document Inspector 展示所选 Version 的持久化 Job、
Attempt 与安全错误,但不读取 Provider/队列细节,也不提供人工重试按钮;
- Inspector 从内部 Parsed Artifact 投影页级 OCR Confidence 和稳定警告,不返回完整 Artifact 或
Provider 原始响应;证据不可读取时明确显示降级状态,不隐藏仍有效的规范 Chunk;
- “查看文档”和“定位命中来源”打开 Source 预览/解析溯源双视图。PDF 使用浏览器原版式阅读器并
从命中起始页打开;Markdown、纯文本和图片安全在线预览;Office/HTML 强制下载。跨页 Chunk
显示完整页范围和实际打开的起始页,不把起始页误述为精确页;
- 检索过程使用候选数量标签而非无说明序号;Dense、Sparse、Fusion、Published Compiled Expansion、
Reranker 和 Final 形成六阶段证据轨。Final 中 Raw 继续展示精确命中与原文入口,Compiled 使用
“已审核知识”标识和底层来源轨道。文本 Source 的 Blob 显式声明 UTF-8,二进制附件保持下载语义,
避免浏览器把原始字节渲染为乱码;
- 文件夹上传依赖浏览器目录选择 API,只传输文件及其相对路径,空目录不会创建占位 Folder;服务端
校验路径并在已授权目标下创建/复用目录。ZIP 入口在完整安全预检后保留包内多级目录并逐文件
入队;当前仍不包含 RAR/7z/tar、断点续传或跨批次去重;
- 当前页面已接入知识总览、Document Version、Hybrid/BM25、检索诊断、可信 REST 身份、User/Group/
Role 关系、Knowledge Base/Folder、ACL 管理,以及 Compiled Knowledge 草稿、来源当前性、冲突证据、
审核、退回、发布和 Published 检索;审计查询等后续能力不创建空白占位入口;
- 删除无引用文档时清理 canonical Metadata、派生产物、源文件与向量行。被 Compiled Knowledge 引用
时改为 tombstone 逻辑删除:从检索和普通文档入口隐藏、移除向量并标记关联 L2 为 Stale,同时保留
Source/Version/Chunk 追溯证据。
`/admin/compiled-knowledge` 把真实来源片段整理成可审核知识:编辑器从可读知识库、Ready Document
和规范 Chunk 中选择来源,手工保存或调用可选 LLMProvider 生成 Draft;库存通过来源证据轨道展示每个
Document Version 的当前性,并可回到 Inspector 核对。Draft 可提交审核;Manage 身份可记录/解决
双来源冲突、发布或填写原因退回。Stale 项修订时重新选择来源,所有写操作携带当前 revision。
`/admin/search` 对 Compiled Final 展示完整已发布正文,并把每个 `underlying_sources[]` 渲染为可读
证据轨道:文件名、Version、Chunk 顺序、章节/页、精确正文摘要,以及“定位底层来源”和“查看或
下载原始文件”。这些入口继续使用认证 Fetch/Blob 和现有 Inspector,不把 L2 正文或 Chunk 拼接
描述成原始文件。
## 验证
开发期先运行 `make web-check`。该命令通过锁定且无第三方依赖的 `package-lock.json` 建立干净
npm 环境,再检查 `app.js` 模块语法、`index.html` 是否只引用存在的本地版本化运行资源。Node/npm
版本分别由根 `.node-version` 与 `package.json#packageManager`/`engines` 约束。
静态路由和范围契约由 `tests/integration/test_admin_console.py` 覆盖;REST 全链路测试覆盖上传、
列表、解析内容、原始文件、搜索与删除。新增或改变 API/前端入口还必须启动真实服务并通过浏览器
执行入口、内容读取、命中定位和 Source 访问,不能只依赖 `TestClient` 或静态断言。浏览器验收可
使用确定性本地 Provider 和合成文档,不读取真实企业文档;外部 Provider/目标集群验收独立记录。
2026-09-15 的 `EKP-083` 验收使用真实 Uvicorn/TCP、SQLite、Local ObjectStorage、合成规范身份及
两级 Folder 数据。桌面浏览器选择 `Browser Filter KB` 后显示 3 份文档;选择 `Runbooks` 子树只
显示父级和 `Runbooks / East` 中的 2 份文档,选择知识库根目录只显示 1 份,选择空 Folder 后显示
“当前范围没有文档”及清除筛选入口。Folder 选项按父级在前排列并展示完整祖先路径。390×844
复核为单列筛选、页面无横向溢出,浏览器 Console warning/error 为 0。真实 TCP 自动测试同时覆盖
ACL 交集、根目录、后代展开、跨 KB Folder `404` 和非法参数 `422`;Fake Provider 只替代外部模型,
不替代浏览器、网络、应用服务、规范数据库或对象存储边界。
2026-09-15 的 `EKP-084` 自动 E2E 在真实 Uvicorn/TCP 进程中通过管理 API 创建 Knowledge Base 与
基准 Folder,经 multipart `relative_paths` 上传两级目录,实际 `IngestionJobRunner` 完成解析、
Chunk 与索引后批次直接进入 `completed`;随后读取规范 Folder 层级、不可变 Source 字节和 Content
Chunk,证明 API、Worker、SQLite 与 Local ObjectStorage 边界贯通。可见 Chromium 进一步读取用户
授权的 `/Users/d-robotics/Downloads/知识库`:首次验收发现目录 File 默认把完整相对路径作为
multipart `filename`,修复为显式 basename 后,同一批次 6 个 Markdown/DOCX 全部完成,`.DS_Store`
单项安全拒绝;页面显示 `6 / 1` 和每项完整相对路径,文档管理页可见规范“知识库” Folder 与 6 份
Ready 文档,Console Error 为 0。两级嵌套目录另由真实进程自动 E2E 覆盖。
同日 `EKP-086` 验收启动真实 Uvicorn/TCP、SQLite 和规范管理身份:先从页面“新建文件夹”创建根
Folder,再经真实管理 API 写入 4 层目录、并行根目录、兄弟叶子及长名称,刷新后目录树显示 6 个
Folder 的准确父子关系。浏览器以鼠标展开深层节点,并聚焦相同原生 Summary 后使用 Enter 完成
收起/再次展开;空树提示、中文状态标签、完整路径辅助信息均可见,Console warning/error 为 0。
真实 Safari 窄窗口验收触发 820px 单列导航后,再以浏览器放大触发 540px 断点:卡片标题与状态
纵向排列、隐藏叶子次要计数,三级长名称保持在树容器内并保留层级轨道,不改变目录顺序或可展开
操作。浏览器窗口、缩放和测试标签页在验收后均已恢复。
2026-09-09 的 `EKP-080` 验收启动进程隔离 Uvicorn,并通过真实 TCP 读取 `/admin/overview`、静态
资源和 `GET /api/v1/documents`;合成 Markdown 完成实际入库后,总览显示 `1` 份 Source、`0`
份处理中、`1` 份 Ready 和 `100%` 就绪率。可见浏览器分别以 1440×900 与 390×844 复核桌面/
移动布局,并从最近文档点击“查看证据链”打开真实 Source 预览;浏览器控制台无错误。
2026-09-09 的 `EKP-081` 运行态验收使用本机真实 FastAPI 进程、SQLite 持久化、Local
ObjectStorage 和确定性 Provider,加载合成 Markdown 与三页 PDF Source。浏览器中完成了:文档
列表“查看文档”默认连续渲染 Markdown 并可切换查看 4 个顺序 Chunk;知识检索页经真实 HTTP
返回带上下文和精确命中片段的结果;三页 PDF 命中显示 `Page 1–3`,“定位命中来源”打开真实 PDF
阅读器,明确提示“命中范围第 1–3 页、从第 1 页打开”,Source URL 带 `#page=1`。同一运行态还
使用真实持久化 Job/Attempt 和 schema v4 页质量 Artifact 验证:展开“处理流水线”可见五阶段及
成功 Attempt;第 2 页 OCR 42% 显示为需复核警告。该记录证明本地用户路径与持久化/存储接线,
不代表外部 Provider、GPU 或目标集群现场验收。
2026-09-14 的 `EKP-091` 运行态验收使用隔离 Uvicorn、SQLite、Local ObjectStorage 和合成身份/
Markdown,在管理端完成双来源草稿创建、来源核验与提交审核、双来源冲突记录、未解决时隐藏发布、
填写解决说明后发布,以及第二条知识的退回原因和修订为新草稿。来源轨道和冲突配对均显示真实
文件名、Version、Chunk 顺序与章节,并能进入 Document Inspector 定位精确 Chunk。桌面操作后以
390×844 复核完整列表和 Inspector 弹窗,控制台 warning/error 为空。首轮浏览器操作发现序号来源
难辨认及同文档同版本来源歧义,已在本任务内修正后重新完成全链路验收。
同日 `EKP-092` 运行态验收使用隔离 Uvicorn、真实 TCP、SQLite、Local ObjectStorage 和确定性
Provider。可见浏览器登录后检索“审核发布的产品 A 退款统一结论”,页面显示 Dense、Sparse、
Fusion、Published Compiled Expansion、Reranker、Final 六阶段;Final 首位展示已发布正文和
“已审核知识”标识。展开底层来源可见 `refund-policy.md · V1 · Chunk 2`、章节位置和精确 Source
摘要;点击“定位底层来源”后,Document Inspector 打开确切 Chunk 2。桌面路径完成后以 390×844
复核诊断轨、Compiled 卡片和来源轨道,浏览器控制台 warning/error 为空。该证据只证明本地真实
应用/网络/存储接线和确定性 Provider 契约,不替代外部模型 Benchmark 或目标集群验收。
同日 `EKP-056` 验收启动更新后的真实 Uvicorn 进程并使用可见浏览器打开 `/admin/search`:提交
“产品 A 的退款周期是多少?”后,页面显示 `public_knowledge / ACL 未接线`、`e2e-profile-v1`、
Dense/Sparse/Fusion/Reranker/Final 五阶段真实候选数量与单调时钟耗时;展开 Dense 可见候选名次、
分数和安全正文摘要,Final 结果可继续打开 Inspector 并定位精确 Chunk。窄屏信号轨支持横向滚动,
折叠阶段不被展开列强制等高。该过程实际调用 `/search/diagnostics`、Document 内容和 Source API。
2026-09-10 的 `EKP-082` 验收启动真实 Uvicorn 进程,通过 TCP 完成严格 `401`、规范 Session、
Knowledge Base/Folder、User/Group/Role、成员与角色关系、Role ACL、KB-scoped multipart 上传和
ACL-aware Search/Document 读取。可见浏览器在真实 `/admin/permissions`、`/admin/knowledge-bases`、
`/admin/upload` 与 `/admin/documents` 页面完成身份连接、资源创建、授权、范围选择和文档阅读;
“新窗口打开原始文件”产生认证读取后的 Blob 页面,地址不含 Token。Fake Provider 只替代外部
Embedding/Vector/Reranker,不替代浏览器、网络、应用服务、SQLite 规范数据或 Local ObjectStorage。
最终低权限复验使用只含 Read/Search Scope 的 Alice:侧栏只保留总览、文档和知识检索,删除
按钮不可见,列表只显示 Role ACL 授权的一个文档;点击“查看文档”连续呈现 Markdown 标题和正文。
2026-09-10 的 `EKP-080` 体验补充验收继续使用真实 Uvicorn、TCP、SQLite、Local ObjectStorage
和确定性 Provider。可见浏览器完成身份登录、知识检索、页面刷新后的查询/结构化诊断响应恢复,
并从恢复结果成功打开“定位命中来源”;“清空查询”只清除输入,不删除当前结果。知识库新增弹窗和
390×844 权限授权弹窗均可读,浏览器 Console 无 warning/error。会话证据在切换访问身份时清除,
避免跨身份保留。自动门禁为:管理台静态集成 `3 passed`、真实 TCP E2E `4 passed`、完整
Unit/Integration `709 passed`、Evaluation `8 passed`、Mypy 166 个源文件以及 Migration head
`20260909_0020` 全部通过。
- 品牌命名边界:左上品牌主体使用“地瓜机器人”,工作台标题和左下平台标识使用“地瓜知识平台”;左上品牌组合按比例放大图标、主名称、英文副标题和间距。
- 文档导入区提供文件、多层文件夹和单 ZIP 三个明确入口;普通批次提示 `BATCH_MAX_FILES=20`,ZIP
提示 `ZIP_MAX_FILES=1000`。两种目录来源均显示保留后的相对路径,结构安全但格式不支持的条目
单项拒绝。
- 查询留存、清空按钮和诊断恢复已收敛到 `app.js` 与 `index.html`,不再依赖已删除的 `ux-enhancements.js`。

View File

@ -0,0 +1,157 @@
# 异步入库、编纂与失败恢复说明
## 范围
DEMO-015 将单文件上传从 API 请求线程迁移到独立 Celery Worker,DEMO-016 在其上增加逐文件
隔离的批量接收和状态聚合。OceanBase 中的
`ingestion_jobs` 和 `ingestion_attempts` 是执行状态事实源;每个 Job 固定绑定一个
`document_version_id`,RabbitMQ 消息只携带 Job ID,
不携带文件内容、Provider 凭据或业务状态。
批次只聚合接收与 Job 状态,不改变每个文件的独立执行和重试边界。
`EKP-103` 复用同一数据库事实源与消息提示原则,为 AI 草稿新增独立
`compilation_jobs`、`compilation_job_sources`、`compilation_attempts` 和 `compilation` queue。
Compilation 消息同样只携带 Job ID,不携带 instruction、来源正文或 Provider 响应。
## 运行时配置
以下值通过环境变量、部署 Secret 或用户明确授权提交的具体环境文件注入:
```text
RABBITMQ_URL=amqp://<user>:<password>@<host>:5672/<vhost>
INGESTION_MAX_ATTEMPTS=3
INGESTION_RETRY_BACKOFF_SECONDS=2
INGESTION_RETRY_BACKOFF_MAX_SECONDS=60
INGESTION_WORKER_LEASE_SECONDS=300
INGESTION_RECOVERY_INTERVAL_SECONDS=30
COMPILATION_MAX_ATTEMPTS=3
COMPILATION_RETRY_BACKOFF_SECONDS=2
COMPILATION_RETRY_BACKOFF_MAX_SECONDS=60
COMPILATION_WORKER_LEASE_SECONDS=300
COMPILATION_RECOVERY_INTERVAL_SECONDS=30
BATCH_MAX_FILES=20
BATCH_MAX_TOTAL_BYTES=209715200
ZIP_MAX_FILES=1000
ZIP_MAX_COMPRESSION_RATIO=100
```
Worker 与 API 还需要相同的 OceanBase、BOS、Embedding、VectorStore 和 Index Profile 配置。
RabbitMQ 用户应只具备指定 vhost/queue 的最小权限。
启用 `PARSER_PROVIDER=mineru` 时,Worker 还必须获得相同的 `MINERU_*` 配置,并令
`INGESTION_WORKER_LEASE_SECONDS` 严格大于 `MINERU_TIMEOUT_SECONDS` 和其他 Provider Timeout。
默认 MinerU Timeout 为 600 秒,因此租约至少设置为 601 秒;Settings 与 Helm 会拒绝不安全值。
启用 `LLM_PROVIDER=openai_compatible` 时,`COMPILATION_WORKER_LEASE_SECONDS` 必须严格大于
`LLM_TIMEOUT_SECONDS`。
## 启动顺序
```bash
uv run alembic upgrade head
make run-worker
make run
```
`make run-worker` 在当前 Demo 中同时启动 Worker 和恢复调度器。恢复调度器周期扫描:
- 尚未成功投递的 `queued` Job;
- 已到重试时间的 `retrying` Job;
- Worker 丢失后租约已过期的 `running` Job。
任务投递使用 late acknowledgement、worker-lost reject 和单任务 prefetch。重复消息会先锁定
Job 行;有效租约内不会创建第二个 Attempt。租约过期后,旧 Attempt 以安全错误关闭,再按重试
规则恢复。每次 Claim 同时写入不可公开的 Lease Fencing Token;Stage 续租、Attempt 结束和清理
失败收敛都必须携带当前 Token。若旧 Worker 在新 Claim 后恢复,它会被识别为已替换执行并安全
确认该旧消息,不能覆盖新 Attempt 的状态。
Compilation Worker 在每次 Attempt 执行时重新解析提交 User 当前的 Knowledge Base Write 和全部
来源 Document Read 权限,并重新核验 Active Version 与 Data Processing Policy。只有当前租约
持有者能写入通过唯一 `generation_job_id` 绑定的 Draft;重复投递或旧 Worker 恢复最多复用既有
结果,不能创建第二条 Draft 或覆盖新 Attempt。
任一历史 Compilation Job 引用的 Chunk 会随 Document 逻辑删除保留,避免删除未生成 Draft 的失败
Job 时破坏执行证据;向量/稀疏索引仍会移除,已生成的 Compiled Knowledge 同时进入 Stale。
## 重试与清理
确定性的内容错误(无法解析、清洗后为空、无 Chunk、派生数据身份不一致)直接进入 `failed`。
其余运行时/Provider 错误按指数退避重试,默认最多三次总 Attempt。
第二次及后续 Attempt 创建前严格执行:
```text
该 DocumentVersion 的 OceanBase Vector rows
→ ChunkIndexState
→ Chunk
→ parsed JSON object
→ reset DocumentVersion processing state
```
不可变源文件、Document、历史 Version、Job 和已结束 Attempt 始终保留。旧 Active Version 的
派生数据和检索状态不受候选 Version 重试影响。任一清理步骤失败时不创建新 Attempt,Job 直接以
`retry cleanup failed` 终止,禁止在未知脏状态继续执行。
## API 状态
`POST /api/v1/documents` 成功返回 `202`、`document_version_id` 和 `job_id`。查询:
```text
GET /api/v1/jobs/{job_id}
```
Job 状态为 `queued | running | retrying | succeeded | failed | cancelled`;阶段为
`parsing | cleaning | chunking | embedding | indexing`。公开错误只返回固定安全摘要,不保存或
返回 Broker/Provider 异常正文。活动 Job 的 Document 暂不允许删除,API 返回 `409`。
批量提交与查询:
```text
POST /api/v1/ingestion-batches # multipart 字段 files;目录模式再提供同序 relative_paths
POST /api/v1/ingestion-archives # multipart 字段 archive;单个安全 ZIP
GET /api/v1/ingestion-batches/{batch_id}
```
批次创建后逐文件返回 `accepted` 或 `rejected`;已接收文件继续通过上述 Job 状态异步推进。
批次聚合状态为 `processing | completed | partial_failure | failed`。请求文件数超过
`BATCH_MAX_FILES` 时整体返回 `413`;单文件或共享总字节预算超限只拒绝当前文件,不删除此前
已接收文件。默认总预算 200 MiB,且配置不得小于单文件 `MAX_UPLOAD_BYTES`。
文件夹上传由浏览器为每个 `files` 提供 `webkitRelativePath`,并以重复 `relative_paths` 字段按同序
提交。服务端拒绝绝对路径、父级跳转、反斜线、空段、控制字符和文件名错位,只把验证后的相对路径
写入 `ingestion_batch_items.relative_path`。目录层级在已授权 Knowledge Base/目标 Folder 下创建或
复用;基准 Folder 跨 KB 时在对象存储前返回 `404`。文件数量、单文件/共享字节预算、去重、Job
重试和逐文件部分失败语义不因目录模式改变。
ZIP 入口使用独立文件数上限,并在任何规范写入前完成 CRC、路径、链接/特殊文件、加密、嵌套包、
压缩包大小、单文件/展开总量和压缩比预检。平台元数据和空目录不入库;不支持的普通文件类型仍按
批次逐项拒绝。不同 Folder 的重复文件名是独立 Document,同一 Folder 内保持版本演进。
AI 编纂提交与查询:
```text
POST /api/v1/compiled-knowledge/drafts:generate # 返回 202 + Job
GET /api/v1/compiled-knowledge/jobs/{job_id} # 权限交集内轮询
```
成功响应通过 `compiled_knowledge_id` 指向新 Draft。查询不返回 instruction、来源内容、租约 Token
或原始错误;管理端的轮询最多等待 120 秒,超时不会取消仍由 Worker 持续推进的规范 Job。
## 故障检查
1. Job 长期为 `queued`:检查 RabbitMQ 连通性、恢复调度器以及 `ingestion` queue;
2. Job 长期为 `running`:检查 Worker,租约到期后恢复调度器会重新投递;
3. Job 为 `retrying`:等待 `next_run_at`,客户端不应重新上传同一文件;
4. Job 为 `failed` 且错误为 `retry cleanup failed`:停止人工重试,先核对向量、关系数据与 BOS
派生对象的一致性;
5. 不得通过日志打印 `RABBITMQ_URL`、数据库 URL、BOS/模型凭据或原始 Provider 响应。
6. Compilation Job 长期等待时同时检查 `compilation` queue、LLM 配置、Worker 与恢复调度器;失败
后只按公开安全错误定位,原始 Provider 正文不得写入 Job、Attempt、Audit 或日志。
`EKP-023` 的本地集成回归覆盖 Job/Attempt 持久化、Broker 失败恢复、有效租约重复投递、过期
租约恢复、Fencing Token 拒绝旧 Worker、有界瞬时重试、永久错误、清理前置、清理失败关闭及
Source 保留。该任务未新增或改变 API、页面、按钮或用户入口,因此不触发新的运行态 E2E;
`EKP-021` 已记录的真实 HTTP 上传链路继续证明 Job 创建入口。
`EKP-103` 回归覆盖 Compilation Job/Attempt 持久化、瞬时故障有界重试、执行时权限重验、有效与
过期租约 Fencing、重复投递单 Draft、固定查询数、Shadow Index 重建保留 L2/L1 规范数据,以及
真实 TCP 和浏览器 `202 → Job → Draft` 链路。

View File

@ -0,0 +1,73 @@
# ObjectStorage 与 BOS 配置
`ARCH-002` 新增百度 BOS Adapter,`EKP-020` 补齐并关闭完整 `ObjectStorage` Provider 范围。
业务服务仍只使用逻辑 key,不知道 Bucket、Prefix、SDK 类型或凭据。`ARCH-005` 已将 BOS 设为
正式运行默认值;Local Adapter 继续用于单元测试和显式配置的本地开发。
## 运行配置
配置由部署环境、Secret Manager、本机 `.env` 或用户明确授权提交的环境文件注入;真实值不得
进入镜像、日志或测试快照,提交范围必须记录并限定到获授权的环境文件。
```text
OBJECT_STORAGE_PROVIDER=bos
BOS_ENDPOINT=<已授权的 BOS Host 或 HTTP(S) Origin>
BOS_ACCESS_KEY_ID=<运行时注入>
BOS_SECRET_ACCESS_KEY=<运行时注入>
BOS_BUCKET=<预创建的 Bucket>
BOS_KEY_PREFIX=knowledge-platform
BOS_TIMEOUT_SECONDS=30
```
- Endpoint 必须是无用户名、密码、Path、Query 和 Fragment 的 Host 或 HTTP(S) Origin;旧配置
的纯 Host 会在 Provider 边界强制规范化为 HTTPS,禁止沿用 SDK 的默认 HTTP;
- Bucket 名必须符合 BOS 的小写字母、数字和连字符规则;
- Prefix 可为空;非空时使用安全的 POSIX 相对 key,不允许空段、`.`、`..` 或反斜杠;
- 应用不会创建、删除或修改 Bucket 配置。
## 对象与不可变约束
canonical `Document.storage_key` 保存与 Provider 无关的逻辑 key,例如:
```text
documents/<document_id>/<filename>
```
BOS Adapter 发起请求时才映射为:
```text
<BOS_KEY_PREFIX>/documents/<document_id>/<filename>
```
上传先写入可回卷的临时流,以满足官方 SDK 对长度和 CRC 计算的要求;随后使用
`If-None-Match: *` 执行单对象 Put,服务端返回 412 时统一映射为
`ObjectAlreadyExistsError`。这样不会用“先 HEAD、再 PUT”的竞态检查破坏不可变来源约束。
下载返回 SDK 的流式 Response Body,由调用方负责关闭;存在性检查只发 HEAD,不下载正文。
404、412 和其他 SDK 异常映射为稳定的 Provider-neutral 错误,外层错误不包含 Endpoint、
SDK Response Body 或凭据。
Local 与 BOS 必须通过同一份 Provider 契约测试:安全逻辑 Key、Put/Open/Exists/Delete 生命周期、
缺失对象、不可覆盖和写入期间不可见。Local Put 先在目标目录写入临时文件并落盘,再使用不覆盖的
原子硬链接发布;因此读取方不会看到部分 Source,并发写入也不能替换已经成功的对象。输入流或
Provider I/O 失败统一收敛为不包含本地路径、Endpoint 或上游响应的 `ObjectStorageError`。
Worker 租约只计算当前启用的 Provider Timeout。显式选择 Local 时,不再被未使用的
`BOS_TIMEOUT_SECONDS` 限制;生产默认 BOS 仍要求租约严格长于 BOS、Embedding、VectorStore 及
已启用 Parser 的最长 Timeout。
## 权限与验收边界
运行身份只授予目标 Bucket/Prefix 所需的 Put、Get/Head 和 Delete 权限,不授予 Bucket 管理或
全局 List 权限。若生产策略要求来源不可物理删除,应在后续 Document Version/Retention
设计中单独处理;当前 Demo 的直接删除行为保持不变。
默认测试使用 Local Adapter 和 Fake BOS Transport,不连接 BOS。真实连接、权限和临时对象清理
已在 `ARCH-006` 的受控 test 资源中验证;该现场证据不把凭据写入仓库、日志或测试快照。后续目标
环境复验属于独立部署验收,不阻塞 `EKP-020` 的 Provider 开发闭环。
官方参考:
- [BOS Python SDK 上传文件](https://cloud.baidu.com/doc/BOS/s/yl5jfscvi)
- [BOS 获取 Object](https://cloud.baidu.com/doc/BOS/s/Mk4xtzv8v)
- [BOS 访问控制](https://cloud.baidu.com/doc/BOS/s/Tjwvysda9)

View File

@ -0,0 +1,30 @@
# CI 基线
`EKP-006` 提供与具体托管平台无关的单一入口:
```bash
make ci
```
该命令按顺序、失败即停止地执行:Python 锁文件/工具链检查、管理端依赖锁定与静态资源检查、
Ruff Format/Lint、Mypy、Unit/Integration、Migration 契约、离线 RAG Evaluation,以及通过本机
回环网络启动真实 API/MCP 进程的 E2E。所有 Python 命令使用 `uv run --locked`,Web 检查使用
锁文件驱动的 `npm ci`。
## 安全边界
默认 CI 不读取真实 Provider 凭据,不连接目标数据库、对象存储、模型服务或 GPU,也不执行
`live-acceptance`、Embedding/Reranker Benchmark、镜像推送、集群部署、远程 MCP/WorkBuddy 验收或
依赖锁重写。上述操作分别属于显式验收、Benchmark 或部署/运维任务,必须单独授权和记录。
仓库当前的中央 dev 发布由受管 Drone 模板和 `dev` 分支负责。`make ci` 是可由本地或任意托管
Runner 调用的开发质量契约,不复制、不猜测中央模板,也不会因为目标环境暂不可用而阻塞开发主
队列。托管 Runner 的接线与运行证据属于独立部署/运维队列。
## 维护规则
- 新增通用开发门禁时,同步加入 `ci` 目标及其契约测试。
- CI 只能调用只读、确定性、无真实 Provider 依赖的目标;发布和现场验收不得混入默认入口。
- 新增或改变 API、页面、按钮或用户入口时,必须扩展 `tests/e2e` 的真实运行路径;`make ci` 会自动
执行该套件。
- 本地复现必须直接运行 `make ci`,不得用一次成功的目标环境部署替代工程门禁。

View File

@ -0,0 +1,179 @@
# 公共知识库集群运行与验收手册
## 1. 范围与安全边界
本手册覆盖 DEPLOY-003 的目标集群安装、幂等升级、滚动恢复、Worker 故障恢复、最小日志/指标、
备份责任和远程 MCP/WorkBuddy 验收。该 DEPLOY-003 现场基线仍按企业受控网络内的单一公共知识库
验收;应用代码已在 `EKP-082` 支持可信 REST 身份、多 Knowledge Base 与 Document ACL,目标环境
启用这些能力前必须配置规范身份/授权 Secret 并完成独立迁移验收。本手册不提供完整 Observability
平台和跨区域灾备。
所有敏感值必须由 Secret 管理系统写入 Kubernetes Secret。Values、Git、镜像、命令行参数、
验收记录和日志中不得出现数据库 URL、RabbitMQ URL、云密钥、Provider Key 或 MCP Token。
## 2. 交付责任边界
| 资源 | 平台仓库负责 | 环境/运维负责 |
|---|---|---|
| 应用镜像 | 可重复构建的 API、MCP、Worker target | 目标 Registry、不可变 tag/digest、镜像扫描和保留策略 |
| Kubernetes | Helm Chart、Migration、探针、PDB、资源和 Pod 安全默认值 | 集群、Namespace、Ingress Controller、配额和网络策略 |
| 敏感配置 | Secret 引用和必需键名预检 | Secret 创建、轮换、审计和访问控制 |
| HTTPS | Ingress Host/TLS Secret 引用 | DNS、证书签发/续期与 TLS Secret |
| 数据 | canonical/可重建边界和恢复顺序 | OceanBase、BOS、RabbitMQ 的备份、容量、HA 和恢复演练 |
| 监控 | JSON 事件、API `/metrics`、验收检查 | 日志/指标采集、留存、查询、告警和 On-call |
## 3. 部署前检查
操作者需要准备仓库外的环境 Values、Namespace、运行时 Secret、TLS Secret、ImagePullSecret,
并确认集群可访问 OceanBase、BOS、RabbitMQ、Embedding 和 Reranker。先显式选择并核对 context:
```bash
kubectl config current-context
kubectl auth can-i get deployments -n knowledge-platform
kubectl auth can-i patch deployments -n knowledge-platform
```
镜像必须使用 Registry 中已存在的不可变 tag 或 digest,禁止 `latest`。部署前执行:
```bash
make deploy-lint VALUES=/secure/path/environment-values.yaml
make deploy-render VALUES=/secure/path/environment-values.yaml >/tmp/ekp-rendered.yaml
```
渲染文件只能包含非敏感 ConfigMap 和 Secret 引用;完成检查后应删除临时文件。
## 4. 安装、幂等升级与只读验收
首次安装或普通升级使用同一入口:
```bash
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
make deploy VALUES=/secure/path/environment-values.yaml
```
只读验收不会触发安装、重启或故障注入,但强制 context 与操作者声明完全一致:
```bash
ACCEPTANCE_KUBE_CONTEXT="$(kubectl config current-context)" \
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
make cluster-acceptance VALUES=/secure/path/environment-values.yaml
```
## 5. 完整可用性验收
完整模式会连续执行两次相同 Helm 入口、滚动重启 API/MCP,并在一个合成入库 Job 运行时强制
删除一个明确选中的 Worker Pod。只能在已批准的验收 Namespace 执行:
```bash
context="$(kubectl config current-context)"
ACCEPTANCE_MODE=full \
ACCEPTANCE_KUBE_CONTEXT="${context}" \
ACCEPTANCE_CONFIRM="DEPLOY-003:${context}:knowledge-platform:knowledge-platform" \
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
make cluster-acceptance VALUES=/secure/path/environment-values.yaml
```
脚本验证以下内容:
- 相同 Values 的安装/升级可以重复成功,Migration Hook 先于工作负载更新完成;
- API/MCP 以及启用的 Edge 至少两个副本全部 Ready/Available,PDB 存在;
- Admin、健康检查、指标和 MCP 均通过稳定 HTTPS Host 到达;
- API/MCP/Edge 滚动期间连续健康探测无中断;
- Worker 被非优雅中断后,同一 canonical Job 经租约到期和恢复调度器完成,不要求重新上传;
- 合成文档验收完成后调用公开删除 API 清理,脚本不访问 Secret 值。
若合成文档处理过快,脚本会失败而不是虚报 Worker 中断已验证。可在受控验收环境调整 Provider
延迟或使用批准的非敏感验收文件重跑,不得加入测试后门。
### 5.1 复用现有域名和 NodePort 的替换流程
当集群没有 Ingress、外部 HTTPS 网关已固定转发现网 NodePort 时,启用 Edge 模式并按以下顺序
切换。以下旧入口切换步骤仅保留为历史回滚参考;DEPLOY-003 当前使用独立开发域名
`knowledge-dev.d-robotics.cc` 和 `dev` 流水线发布,不执行 NodePort 替换。命令中的资源名必须先
从目标集群核对,不能
直接套用到其他环境。
1. 在与现网入口相同的 Namespace 内安装新 Release,Edge 使用自动分配或确认空闲的临时
NodePort;旧 Deployment/Service 保持不变。
2. 通过集群内请求或受控端口转发验证 `/healthz`、Admin/API、带认证 `/mcp`、批量异步入库和
Worker 故障恢复;确认 API/MCP/Edge PDB 与滚动升级均通过。
3. 将旧 Deployment、Service、ConfigMap 引用和公开的镜像坐标保存到权限受控的仓库外目录。
不导出 Secret data,不把快照提交 Git。
4. 先把旧入口 Service 从 NodePort 改为 ClusterIP 以释放 `30081`,随后立即把新 Edge Service
更新为 `30081`。只切换 Service,旧 Deployment 暂时保留。
5. 从正式域名复验 `/healthz`、`/admin/upload`、REST、MCP 和真实检索,并观察错误率与 Pod
状态。失败时先释放新 Edge 的 `30081`,再按快照恢复旧 Service。
6. 稳定观察窗口结束并确认 WorkBuddy 远程验收后,才清理旧 Deployment/Service;数据库
Migration 为 forward-only,回切应用前必须确认版本兼容。
整个切换窗口禁止同时创建两个占用 `30081` 的 Service,也禁止通过删除旧 Deployment 来释放
NodePort。外部 HTTPS 网关和证书保持不变,因此用户端无需重新配置域名。
## 6. 远程 MCP 与 WorkBuddy
先用官方 MCP Python Client 验证远程协议、严格五个 P0 Tool 和一次完整来源读取链。Token 通过静默输入进入
当前进程环境,不写入命令参数或配置文件:
```bash
export EKP_MCP_URL=https://mcp.example.internal/mcp
export EKP_MCP_ACCEPTANCE_QUERY='已批准的公共知识验收问题'
read -rsp 'MCP token: ' EKP_MCP_TOKEN && export EKP_MCP_TOKEN && echo
make mcp-remote-acceptance
unset EKP_MCP_TOKEN
```
随后在与最终用户相同网络路径的 WorkBuddy 客户端加载
[`workbuddy-mcp.example.json`](../../deploy/workbuddy-mcp.example.json),只通过客户端安全环境注入
`EKP_MCP_URL` 和 `EKP_MCP_TOKEN`。验收必须确认:连接状态正常、只发现五个批准的 P0 Tool、
真实问题返回至少一个带来源字段的结果,并可继续读取 Source、Document、Section 和所属 Knowledge
Base。记录客户端版本、时间、网络区域、
Tool 名和成功/失败即可,禁止复制 Token、正文或内部 Provider 信息。
## 7. 日志、指标与最小告警
应用事件为单行 JSON,固定基础字段为 `timestamp`、`level`、`logger`、`message` 和 `event`;可选
白名单字段包括 `request_id`、HTTP 方法/路由模板/状态/耗时、Job/Document/Version/Batch ID、
`attempt_number`、`stage`、`dependency`、`job_status`、`retry_delay_seconds`、`retryable` 和
`dispatch_count`。事件名必须是最长 64 字符的小写 `snake_case`;请求上下文中的已校验
`request_id` 优先于日志调用附加值。应用日志消息必须是代码字面量并显式声明 `event`,异常正文、
插值参数、嵌套对象和非白名单字段不会被格式化器写出;没有有效结构化 `event` 的依赖日志只输出
`unstructured_log_suppressed` 安全标记。常用检查:
```bash
kubectl -n knowledge-platform logs deployment/knowledge-platform-api --since=15m
kubectl -n knowledge-platform logs deployment/knowledge-platform-worker --since=15m
curl --fail https://knowledge.example.internal/metrics
kubectl -n knowledge-platform get deployments,pods,poddisruptionbudgets
```
环境监控至少采集并告警:API/MCP Ready 副本不足、Pod 重启持续增加、HTTP 5xx、P95 请求耗时、
Worker 无 Ready 副本、`ingestion_attempt_failed`/`ingestion_retry_cleanup_failed`、恢复扫描持续
失败、Migration Job 失败,以及 OceanBase/BOS/RabbitMQ/模型服务在应用日志中的安全失败事件。
阈值由真实流量基线确定,不在本任务中固化完整监控平台。
## 8. 故障处置
API/MCP 不可用时先检查 Deployment/Endpoint/Ingress/TLS,再按单个 Deployment 滚动重启;禁止
直接打印 Pod 环境变量。Job 卡住时使用 Admin/API 的 Job ID 查询状态,再按相同 Job ID 搜索
Worker JSON 事件。处于有效租约的 Job 不应手工重复提交;租约到期后恢复调度器会重投递。
更新同名 Secret 不会自动改变已运行进程。由运维完成 Secret 轮换后,依次滚动重启 API、MCP、
Worker 和 Scheduler,并复跑只读验收。稳定 Host 和客户端 Token 未轮换时,Admin/WorkBuddy
无需重新配置;MCP Token 轮换时必须通过客户端 Secret 分发流程更新。
## 9. 备份与恢复顺序
OceanBase 关系表和 BOS 不可变源文件共同构成事实数据,必须由资源所有者执行一致性备份策略。
OceanBase 向量表是可重建索引,不得代替 canonical 备份。RabbitMQ 消息只提供执行提示,
canonical Job/Attempt 才是恢复依据,但生产 Broker 仍应按其 SLA 做持久化和 HA。
恢复顺序为:恢复/验证 OceanBase canonical 表 → 恢复并核对 BOS 对象 → 部署相同或兼容应用版本
并执行 forward-only Migration → 恢复 RabbitMQ 连通 → 启动 Worker/Scheduler → 按 active Index
Profile 重建缺失向量 → 验证来源追踪和真实检索。Helm 回滚只能回退兼容的应用清单,不能反向
撤销数据库 Migration。完整灾备演练仍是后置任务。
## 10. 验收记录
每次目标集群验收按 [`DEPLOY_003_ACCEPTANCE.md`](DEPLOY_003_ACCEPTANCE.md) 记录非敏感结果。
只有完整自动化验收、真实 MCP 协议调用和远程 WorkBuddy 三部分全部通过,DEPLOY-003 才能
标记完成并合并到 `main`。

View File

@ -0,0 +1,33 @@
# 编码规则
- 基础设施 SDK 必须封装在 Provider 接口之后。
- 不得硬编码凭据、租户 ID、Endpoint、模型名或 Collection 名。
- 优先使用带类型的契约和高内聚模块。
- 数据库 Migration 只允许向前演进。
- 核心测试必须能在没有外部 SaaS、互联网或 GPU 的情况下运行。
- 避免过早拆分微服务。
## 仓库协作
- 一个 Task ID 对应一个临时任务分支;默认只使用仓库根目录这一处 worktree。
- 切换任务前必须提交修改或保存并核验命名 stash,禁止覆盖、reset 或删除用户工作。
- 完成分支先推送、验证并合并 `main`,再按里程碑归档;`dev` 只承担开发环境发布。
- 文档事实源、生命周期、分支/Tag 清理和恢复步骤统一遵循
[`REPOSITORY_GOVERNANCE.md`](REPOSITORY_GOVERNANCE.md)。
## Python 依赖与工具链
- Python 基线由 `.python-version`、`pyproject.toml` 的 `requires-python`、Ruff/Mypy 目标版本、
`tool.uv.required-version` 和 `uv.lock` 共同约束;调整任一版本时必须同步更新其他契约并运行
`make toolchain-check`。
- 日常安装、运行和质量门禁统一使用冻结的 `uv.lock`,不得由普通 `make` 入口隐式改写依赖解析。
- 只有在有意修改 `pyproject.toml` 依赖后才运行 `make lock`,审查 `uv.lock` 差异并再次执行完整
工程门禁。
## 持续集成
- 本地和托管 Runner 统一执行 `make ci`;具体门禁与外部操作排除项见
[`CI_BASELINE.md`](CI_BASELINE.md)。
- 默认 CI 必须保持确定性和 Provider-free,不读取真实凭据,不执行发布、部署、Benchmark 或目标
环境验收。
- API/MCP 真实进程回环 E2E 属于默认 CI;新增或改变用户入口时必须同步扩展该套件。

View File

@ -0,0 +1,91 @@
# Compiled Knowledge 审核与发布
EKP-090 提供 L2 草稿纵向切片;EKP-091 在同一模型上补齐 Source 当前性、冲突证据、人工审核和发布;
EKP-092 将仍可核验的 Published 内容接入正式检索。
公开字段与状态码以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
## 数据与权限边界
- 每条草稿属于一个 Organization/Knowledge Base,并至少引用一个真实 Chunk;来源行同时保存
Document、DocumentVersion 与 Chunk ID,顺序稳定;
- 创建要求目标 KB Write 和每个来源 Document Read;读取要求 KB Read 与全部来源 Document Read
的交集,任一来源不可读时整条知识隐藏;
- `permissions.mode=source_intersection` 描述请求时动态解析结果,不保存或伪造一份独立 ACL;
- `draft` 可提交审核,`review_required` 只能由 Manage 身份发布或退回;`rejected|stale` 修订后回到
新 Draft。Stale 修订可替换完整来源集合,所有变化使用 expected revision 防止覆盖并发编辑;
- Source 新版本激活、被引用 Document 删除及 ACL 增删会自动标记 Stale。引用文档删除保留不可变
Source/Version/Chunk 作为证据,但退出原始检索和普通 Document API;
- 冲突是审核人明确记录的 Claim 与两个确切 Source 的配对,未解决冲突阻断发布。系统不自动推断、
折中或让 AI 发布;只有 Published 内容允许进入检索。
## Published 检索
正式身份的 Raw Dense/Sparse/Weighted RRF 已先应用精确 Document Filter。Search Service 只从这些
授权 Raw 候选引用到的 Compiled Knowledge 扩展 L2 候选,并以“所属 Knowledge Base 与全部来源
Document 都在调用者 Read Scope”作为 SQL 前置条件;MCP 还继承 `mcp_enabled` Scope。没有规范身份 Filter 的历史公共入口
不扩展 L2,避免把 Organization 内容带入兼容域。
扩展只接受 `published` 且具有发布时间、最后核验时间和非空来源的条目。触发分数采用最强命中
Source 的 Fusion Score;L2 标题和正文与 Raw Chunk 一起进入同一次 Reranker,联合池按 Active
Profile `hybrid_top_n` 截断,分数完全相同时 Raw 优先。最终回查再次核验每个 Source 的 Organization/
KB/Document/Version/Chunk 归属、未删除、Ready Active 状态、文件名和调用者 Scope,异常时整次
Search 失败关闭。
本实现没有把 L2 伪装成 Raw Chunk,也没有建立未版本化的第二套向量索引。Compiled Recall 继承其
底层 Source 的 Raw 召回:只有至少一个底层 Source 进入 Hybrid 候选时才扩展 L2;后续如引入独立
Compiled Embedding/Index,必须纳入版本化 Index Profile、Shadow Index 与 RAG 回归,不能绕过既有
索引边界。
Compiled Search Result 返回完整已发布正文和底层来源数组;每个来源包含精确 Chunk 正文与位置,
支持 Search Lab、REST 和 MCP 走同一来源链核对。非 Published 状态全部排除。
## LLMProvider
`LLM_PROVIDER=disabled` 是默认值,手工草稿不依赖外部模型。启用 OpenAI-compatible Chat
Completions 时配置:
```dotenv
LLM_PROVIDER=openai_compatible
LLM_ENDPOINT=https://llm.example.test/v1/chat/completions
LLM_API_KEY=<runtime-secret>
LLM_MODEL=<model-id>
LLM_PROCESSING_LOCATION=cloud
LLM_TIMEOUT_SECONDS=60
```
Secret 只能进入授权环境文件或运行时 Secret。Endpoint 必须为 HTTPS,本机回环模型可使用 HTTP。
`LLM_PROCESSING_LOCATION` 是部署者对处理位置的明确声明;`local_only` KB 会在发送正文前拒绝 cloud
Provider。Adapter 使用 Bearer Header、禁止重定向转发、限制响应大小,并只接受一个 JSON
`title`/`content` 对象。确定性 Fake 只用于测试,不能作为模型质量或目标环境验收结果。
`FakeLLMProvider` 在构造时执行同一能力校验,并在直接返回前执行公共 Compilation Result 校验,
因此测试替身不能绕过生产服务消费的非空和长度边界。
## 操作路径
管理端打开 `/admin/compiled-knowledge`:选择知识库 → 选择 Ready 文档 → 勾选一个或多个规范 Chunk
→ 填写类型、标题和正文后“保存手工草稿”,或使用“AI 生成草稿”。库存用来源证据轨道逐项显示
文件名、Version、Chunk 顺序/章节、创建时间与 `current|outdated|deleted`;可从轨道进入 Inspector 核对仍可读取的
来源。草稿提交审核前由服务端重新核验全部精确来源。
审核态允许记录两个确切来源之间的冲突、填写解决说明、发布或携带原因退回。未解决冲突时管理端
不展示可执行的发布按钮,服务端也以 `409` 失败关闭。退回或 Stale 项可修订;Stale 修订要求重新
选择来源,避免无意继续引用已删除或过期 Chunk。
AI 入口返回 `503 compilation provider is unavailable` 时,表示运行时保持默认 disabled 或配置未
进入 API 进程;这不影响手工草稿。数据策略冲突返回 `409`。请求通过后返回 `202` 和持久化 Job,
管理端轮询 `/compiled-knowledge/jobs/{job_id}`;成功后以 `compiled_knowledge_id` 读取 Draft,失败仅
展示稳定脱敏错误。
`EKP-103` 已接入独立 `compilation_jobs`、有序 `compilation_job_sources` 和
`compilation_attempts`。RabbitMQ 消息不携带 Prompt 或来源正文;Worker 重新解析提交者当前权限并
复核来源与数据策略,使用租约 Fencing、有界指数退避和恢复扫描收敛重复/中断执行。Job 对生成结果
的一对一唯一引用保证重复投递不创建重复草稿。真实外部 LLM 的质量、成本与目标环境可用性仍必须
另行验收,不能用本地 Fake 测试替代。
## 验证
自动测试覆盖模型/迁移、Job/Attempt、重复投递、瞬时失败重试、租约恢复、执行时权限重验、来源
范围、来源权限交集、revision 冲突、local-only 阻断、Adapter 输入输出、
审核/冲突/发布/退回、Source 自动失效及真实 Uvicorn/TCP REST。每次改变页面或按钮还必须用真实浏览器
完成身份进入、来源选择、创建/编辑、提交审核、冲突记录与解决、发布/退回和 Inspector 来源跳转,
并确认控制台无错误。

View File

@ -0,0 +1,143 @@
# 生产容器镜像说明
## 范围
DEPLOY-001 使用同一个锁定运行时构建三个独立镜像目标:
| Target | 默认镜像名 | 入口 | 端口 | 镜像健康检查 |
|---|---|---|---|---|
| `api` | `enterprise-knowledge-api` | Uvicorn + API/Admin | `8000` | `GET /healthz` |
| `mcp` | `enterprise-knowledge-mcp` | `ekp-mcp` | `8766` | PID 1 存活 |
| `worker` | `enterprise-knowledge-worker` | `ekp-worker` | 无 | PID 1 存活 |
Dockerfile 另有默认最终 `dev` target:它继承 `api`,只用于当前中央 dev 流水线,并自动加载已获
明确授权的 `.env.dev`。显式构建上述三个 target 时不会包含该文件。
镜像没有内置数据库迁移、Helm、Ingress 或 Secret 对象;DEPLOY-002 已在独立 Chart 中提供这些
编排能力。API 镜像包含 Alembic 和 migrations,由一次性 Migration Job 覆盖命令后复用。详见
[`KUBERNETES_DEPLOYMENT.md`](KUBERNETES_DEPLOYMENT.md)。
`BUILD-001` 已将应用镜像定义收敛到仓库根 `Dockerfile`。Bake 与静态契约测试均引用该文件,
`deploy/docker/` 只保存多架构 Bake 计划,不再维护第二份 Dockerfile。
## 可重复构建基线
- Python 基础镜像固定为 `3.12.14-slim-bookworm`,uv 固定为 `0.12.5`,两者同时锁定 manifest
digest;
- Python 依赖只使用提交的 `uv.lock`,以 `uv sync --frozen --no-dev --no-editable` 安装;
- 三个目标共享相同依赖层,不复制不同版本的业务运行时;
- 进程均以固定 UID/GID `10001:10001` 运行,不具备 root 权限;
- `.dockerignore` 排除 `.env*`、Git、虚拟环境、缓存、本地存储、测试和开发产物,只对白名单
`.env.dev` 例外供默认 `dev` target 显式复制;
- Bake 计划同时声明 `linux/amd64` 与 `linux/arm64`,不下载或打包平台专属模型。
基础镜像 tag/digest 与应用依赖均已固定;镜像发布系统仍应保存本项目镜像的构建 digest,并让
Helm 按 digest 部署。本项目镜像 digest 由实际 Registry 发布产生,不把某个环境的 Registry
地址写入仓库。
## 构建命令
先确认 Docker Buildx builder 支持两个目标平台:
```bash
make images-print
```
构建全部目标:
```bash
IMAGE_REGISTRY=registry.example.internal/knowledge \
IMAGE_TAG=<immutable-release-tag> \
make images
```
只有发布流水线才执行推送:
```bash
IMAGE_REGISTRY=registry.example.internal/knowledge \
IMAGE_TAG=<immutable-release-tag> \
make images-push
```
不要把 Registry 密码放入变量或命令行;构建节点应提前通过其凭据助手登录。禁止使用
`latest` 作为集群部署版本。
## 运行时配置
显式 `api`、`mcp`、`worker` target 只内置非敏感进程默认值:`APP_ENV=production`、容器监听
地址、静态页面绝对路径和临时缓存目录;OceanBase、BOS、RabbitMQ、Embedding、Reranker 与
MCP Token 必须在启动时注入。默认 `dev` target 是受控例外:它设置 `APP_ENV=development` 和
固定 Profile,并以 UID `10001`、权限 `0400` 保存已授权 `.env.dev`;不得将该 target 发布到
非 dev 环境,也不得把其中值写入构建日志、标签或镜像检查输出。
API 的 `/healthz` 只证明进程存活,不读取云端凭据或探测依赖。Kubernetes readiness、资源限制、
只读根文件系统所需临时卷及依赖就绪策略由 DEPLOY-002 Chart 配置。MCP endpoint 需要认证,Worker
没有 HTTP server,因此这两个镜像的 Docker 健康检查只验证主进程;业务可恢复性仍以持久化
Job/Attempt 和 DEPLOY-003 故障验收为准。
## 本地单目标检查
需要加载到本地 Docker 时,可直接从根 Dockerfile 构建当前平台的一个 target:
```bash
docker build --target api -f Dockerfile -t ekp-api:test .
docker build --target mcp -f Dockerfile -t ekp-mcp:test .
docker build --target worker -f Dockerfile -t ekp-worker:test .
```
Worker target 保持非 root,并使用 `/tmp` 作为工作目录,使本地 `ekp-worker` 入口携带的 Celery
Beat 可以写入调度状态。Helm 仍将 Worker 与单例 Scheduler 作为独立工作负载运行。
## `dev` 分支差异审查
`BUILD-001` 对 `decef23`、`a314dff`、`3cbe42b` 的历史结论如下:
- 不吸收基于 `ENV` 隐式选择并打包 `.env.<环境>` 的实现;环境配置继续由运行时环境或 Secret
注入,镜像只复制明确的代码与构建输入;
- 不吸收 API/Worker `combined` target;两者继续独立部署、扩缩容和恢复;
- 不吸收 root 运行;所有应用 target 继续使用 UID/GID `10001:10001`。
启动时使用 `--env-file` 或编排系统的 Secret 引用,不在命令中展开敏感值。API/Admin smoke test
至少检查 `/healthz`、`/admin/upload` 和 `/api/v1/openapi.json`;MCP 与 Worker 的真实依赖启动
验收在具备 RabbitMQ 和目标 Provider 的受控环境执行。
`BUILD-002` 根据当前 dev 流水线实际约束增加受限的默认 `dev` target,不恢复 `ENV` 到任意
`.env.<环境>` 的路径映射,也不改变显式生产 target、非 root 运行或 API/Worker 独立进程边界。
`main@f03c48e` 合入 `dev@5c72218` 后,公网 `/healthz` 返回 `environment=development`,已证明
中央流水线发布的是该默认 target 且固定 Profile 生效。随后业务 API 的 500 属于依赖 Schema
验收问题,不是 dotenv 选择回退为失败。
中央 dev 模板只更新默认镜像和 GitOps tag,未提供独立 Migration 步骤。默认 `dev` target 因此
使用 `python -m ekp_api.dev_startup`:严格要求 `APP_ENV=development` 和
`EKP_SETTINGS_PROFILE=dev-image`,先执行 `alembic upgrade head`,成功后才启动 Uvicorn;失败则
安全记录 `canonical_database` 分类,并以 `degraded` 健康响应暴露当前 Revision 和异常类型,
使无目标集群日志权限时仍可诊断,业务入口继续失败关闭。该兼容路径只适用于当前中央 dev
单副本部署;显式生产 `api` target 仍直接启动 Uvicorn,并继续由 Helm Migration Job 串行迁移。
若 dev 扩为多副本,必须先接入独立 Migration Job,禁止并发依赖应用启动迁移。
## BUILD-001 CI 实证
Drone #27 已从 `codex/build-001@416a1c0` 成功检出任务提交,但任务分支只执行仓库名校验和通知,
没有执行镜像构建,因此该记录不产生发布证据。完成合并和 `dev` 差异治理后,Drone #30 从
`dev@974c276c` 执行 `image-build-push-dev` 和 `gitops-dev`,并确认:
- 构建入口为根 `Dockerfile`,基础镜像按 digest 拉取;
- Admin 静态资源断言通过,最终用户为 `10001:10001`;
- Kaniko 解析并构建 `mcp`、`worker`、`api` 阶段,Worker 使用 `/tmp` 工作目录;
- 发布 tag 为 `974c276c-1788852979`,OCI digest 为
`sha256:5d18409db5ef2fd310fc00c11b7e6bb7dd68ea16408c186f9f623b818b8a56f2`;
- GitOps 更新成功,开发环境 `/healthz` 返回正常。
中央模板当前只推送根 Dockerfile 的默认最终 target,即 API 镜像;Bake 仍是三个独立 target 的
权威多架构构建入口。若发布环境要求分别推送 MCP/Worker 镜像,应在后续流水线治理中显式增加
target,而不能把单个默认镜像误写成三个独立镜像产物。
## DEPLOY-001 构建验收
2026-09-02 已使用 Buildx 分别构建并加载三个 `linux/arm64` target 和三个 `linux/amd64`
target。API/Admin 以非 root 用户完成健康、静态页面和 OpenAPI smoke test;MCP/Worker 完成非
root 安装入口检查。构建历史与容器文件系统检查未发现 `.env`、Git、测试文件或敏感配置。
这次验收只生成本地 `deploy001-*` 测试 tag,没有登录或推送 Registry,因此不产生可供集群
引用的项目镜像 digest。DEPLOY-002 必须使用操作者指定的不可变 Registry tag/digest,不得把
本地测试 tag 当作发布版本。

View File

@ -0,0 +1,38 @@
# 完成定义
只有同时满足以下条件,任务才算完成:任务范围已经实现;架构不变量得到遵守;格式检查、Lint 和类型检查通过;相关测试通过;按需更新 Migration、契约和文档;未提交任何未经批准的 Secret;已记录所有经明确授权的凭据文件及其授权范围;检索变更包含 RAG 回归结果;已记录已知限制。
Task 的进度说明必须与仓库文档保持同一状态:每次向用户报告开发中、阻塞、恢复或完成等变化前,
先同步 `PROGRESS.md`、`TASKS.md` 以及受影响的契约/开发文档。会话上下文接近需要切换时,先写明
当前 Task、分支/工作树、已完成验证、未完成步骤和阻塞,再明确提醒用户切换会话。
日常开发默认只使用仓库根目录这一处工作区,在同一目录内按 Task 顺序切换任务分支。除非用户
明确要求并行隔离或授权额外 worktree,不得为每个 Task 创建同级项目目录。存在未提交修改时,
必须先提交或保存为带来源说明的命名 stash,并核对其覆盖的文件后才能切换分支;禁止使用 reset、
覆盖文件或删除工作区来换取干净状态。
新增或改变任何 API、前端页面、按钮或用户入口时,必须启动真实应用进程,通过网络 HTTP 或真实
浏览器完成用户路径测试,并贯穿对应应用服务及数据库/对象存储边界。必须记录入口、操作、结果和
非敏感证据。单元测试、进程内 `TestClient`、静态资源字符串断言、Mock/Fake 结果或截图只能作为
补充,不能单独满足端到端完成条件;确定性本地 Provider 可以用于可重复测试,但不得伪装成外部
Provider 或目标环境验收。
对于异步入库或重试任务,相关测试还必须证明:Job/Attempt 状态可持久化、有界重试分类正确、
重复投递具备幂等性、重试前执行清理、不可变 Source 得到保留,以及清理失败时能够正确终止。
批量入库必须覆盖逐文件部分失败和配置限制。
对于 MCP 任务,契约测试必须覆盖 Tool 发现、身份传播、授权/错误边界、REST 与应用服务的一致性、
基础设施信息脱敏,以及明确指定的 MCP Client 兼容性检查。
对于开发团队负责的集群交付任务,完成条件还包括:可重复构建的生产镜像;明确的环境配置契约;
串行 Migration、Readiness/Liveness Probe、资源与 Pod 安全默认值等交付清单;开发侧测试通过;
目标分支 CI/CD 成功;已批准的公网 Admin/API、异步 Worker 与检索烟测通过。实际 GitOps Values、
Secret、Namespace、镜像拉取、集群副本/PDB、幂等安装、滚动重启、回滚、监控、告警和备份由运维
负责时,应形成明确交接,但不作为开发 Task 的完成阻塞项。
文档治理、Benchmark、部署/运维和目标环境验收属于独立非开发任务类型。它们必须在 Task 与进度
文档中同步状态、结论和风险,但除非开发功能本身依赖其产出,不得阻塞 V1 开发主队列。
只有明确包含 MCP 的任务才要求稳定 MCP Endpoint 与远程 WorkBuddy 验收;搁置状态不得写成完成。
必须在各自责任范围内检查 Manifest、日志和镜像是否泄漏未经批准的凭据。公共知识交付不会完成或
豁免 V1 用户/ACL 任务。

View File

@ -0,0 +1,50 @@
# EKP-051 Dense 检索
`EKP-051` 提供独立、Provider-neutral 的 Dense Top-N 候选服务。它复用 `EKP-040~046` 的
Embedding、Index Profile、VectorStore 与 Active/Shadow 生命周期,并在 `EKP-050` 的精确
Document Scope 内检索;本任务不新增或改变 REST/MCP 字段、页面、按钮或用户入口。
## 执行顺序
```text
规范化并校验 Query / Top-N
→ 空授权范围直接返回
→ Query Embedding
→ 向量数量、维度、有限值和正范数校验
→ Profile + Active Version + Document Scope 预过滤
→ Cosine Top-N
→ 候选身份、来源、分数、顺序和范围复验
```
Query 去除首尾空白后必须非空、不含 NUL 且不超过 1600 字符;Top-N 必须是正整数且不得超过
Active Profile 固化的 `dense_top_n`,布尔值不作为整数接受。诊断可以缩小候选集,不能在请求时
扩大版本化 Pipeline。验证在调用外部 Embedding 前完成。授权 Document Scope 为空时不会调用
Embedding 或 VectorStore,避免无结果请求仍向外部 Provider 披露 Query。
Embedding Provider 的模型和维度必须与 Index Profile 一致。Profile 配置了
`embedding_query_instruction` 时,Query Provider 必须公开完全一致的 `query_instruction`;缺失、
非法或漂移均在装配时失败关闭。文档 Embedding 不添加 Instruction,Qwen Query Provider 在自身
Adapter 内按已固化 Instruction 格式化输入,公开检索服务不感知厂商格式。
Dense 候选保留 `index_record_id`、Document/Version/Chunk、文件名、Page Range、Section、正文、
Chunk Index、Profile 和 Cosine Score。业务服务再次拒绝非 Tuple、超额、重复 Chunk/Index Row、
无效来源字段、跨 Profile/ACL、非有限或超出 `[-1, 1]` 的分数,以及非降序结果,不能依赖某个
Provider 恰好正确。
## 配置与验证
默认候选数来自 Active Index Profile 的 `dense_top_n`;当前运行配置 `DENSE_TOP_N` 和 Helm
`config.retrieval.denseTopN` 收敛为 `1~64`。Profile 或模型/维度/Instruction 变化必须创建并构建
新的 Shadow Profile,禁止原地改变 Active Index。
离线验证:
```bash
uv run pytest tests/unit/test_dense_retrieval.py -q
uv run pytest tests/integration/test_dense_retrieval_integration.py -q
uv run pytest tests/evaluation/test_dense_retrieval_regression.py -q
```
集成测试覆盖真实 SQLite 规范关系表、Local ObjectStorage 与 Upload→Parse→Chunk→Embed→Index→
Dense 链路;Golden Dataset 覆盖 40 条合成 Query 的 Recall/MRR 门禁。Fake 的精确 Cosine 不代表
目标 OceanBase HNSW 的近似召回、延迟或容量,真实查询计划和性能由 `EKP-103`/现场验收独立记录。

View File

@ -0,0 +1,255 @@
# DEPLOY-003 目标集群验收记录
**状态:** dev 500 已修复;API 身份凭据与集群资源级事项由独立环境队列接管;远程 MCP/WorkBuddy 搁置
**Task:** DEPLOY-003
**记录规则:** 只记录非敏感标识和聚合结果,不记录 Secret、业务正文或内部 Provider 响应。
## 2026-09-10 受控发现性部署
- 远程 `dev` 从 `974c276` 以普通 merge 推进到 `8141f23`,合并父提交为 `dev@974c276` 与
`main@83f55fb`;推送前确认最终文件树与 `main` 完全一致,未强推或改写 `dev` 历史;
- 推送前 Ruff Format/Lint、Mypy、27 个部署与 Migration 相关测试、Migration Contract、Helm
Lint 和 Admin JavaScript 语法检查通过;Migration Contract 的唯一 Head 为 `20260909_0020`;
- 推送后公网 `/healthz` 与 `/api/v1/openapi.json` 返回 `200`,Admin HTML 出现 `ekp-082`
资源标记,浏览器实际加载了含知识库、身份与权限入口的新运营台,证明 API 默认镜像和 Admin
静态资源已发生滚动更新;
- 业务接口未通过运行验收:`/api/v1/administration/session` 与 `/api/v1/documents` 均返回
`503`,公开错误为 `API providers are not fully configured`。这表明当前 GitOps Runtime
Config/Secret 尚未满足新版本 Provider 装配要求,不能将本次部署记为业务可用;
- 健康响应仍报告 `environment=production`,与 dev 域名语义不一致;HTTPS `/admin/` 还会生成
指向 HTTP 的 `307`,均作为环境配置问题移交;
- `/mcp` 继续返回 `404`。中央 Drone 模板不在本仓库内,且当前仍缺少 MCP/Worker 独立目标镜像
的发布证据,因此本次只确认 API/Admin 更新,不确认 Migration Job、MCP 或 Worker 已更新;
- 本次发现性部署只形成可复现的环境问题清单,不撤销下方历史闭环结论,也不阻塞 V1 开发主队列。
## 2026-09-10 BUILD-002 配置加载复验
- 用户明确授权仓库 `.env.dev` 的现有配置进入默认 dev 镜像及对应 Registry,授权范围仅为 dev;
显式 `api`、`mcp`、`worker` 生产 target 继续不包含 dotenv;
- `BUILD-002` 以 `main@f03c48e` 合入 `dev@5c72218` 并触发发布。新实例 `/healthz` 稳定返回
`200` 和 `environment=development`,证明默认 dev target 已固定选择镜像内 `.env.dev`;
- 原先两个业务入口的 `503 API providers are not fully configured` 已消失,说明 Provider 必填
配置门禁已通过;`/api/v1/documents` 与 `/api/v1/administration/session` 当前改为返回 `500`;
- 一次无敏感信息的聚合指标快照记录到 85 次 `/healthz` 200、7 次 `/documents` 500 和 1 次
`/administration/session` 500;业务请求约在 `0.08~0.16s` 内失败,不符合外部 Provider
连接超时特征;
- 当前代码在旧可用 dev 基线之后新增规范库 Migration `0009~0020`、完整 Index Profile 字段、
向量表版本字段和 Sparse 全文索引,而中央流水线仍无 Migration Job 运行证据。因此旧 Schema
是当前最强嫌疑,但没有 Pod 异常日志和 dev 数据库只读检查结果,不能写成已证实根因;
- 后续由独立部署队列先核对 Pod 的结构化错误事件、`alembic_version`、`index_profiles` 列集合
以及向量表 DDL。任何真实 Migration、向量 DDL 或数据修复均属于外部写操作,必须在备份与
明确授权后执行。该 500 不否定 BUILD-002 的配置选择验收,也不阻塞 V1 开发主队列。
## 2026-09-10 规范库故障定位与修复
- Drone #126 从 `dev@ed69886` 成功构建并更新 GitOps;新业务响应由原始 500 收敛为
`503 API canonical database is unavailable`,且响应头为
`X-EKP-Dependency: canonical-database`。真实异常发生在规范库初始化,向量 Schema 尚未成为
当前首个故障点;
- Drone #121/#126 的可见阶段均只有镜像构建和 GitOps tag 更新,没有执行 Alembic 或独立
Migration Job。结合旧版本曾使用同一数据库连接、当前代码要求 Head `20260909_0020` 且失败
快速发生,规范库未随代码升级是当前具备完整证据链的修复方向;实际 Head 仍需迁移启动器在
目标网络内读取并由运行结果最终确认;
- 中央 GitOps 仓库和实际运行集群不在当前授权 kubeconfig 范围内,不能直接创建 Job。为恢复
dev,默认 dev 镜像增加受限启动器:只允许 `development + dev-image`,在 Uvicorn 监听前执行
`alembic upgrade head`;失败时健康响应标记 `degraded` 并仅公开当前 Revision/异常类型,业务
服务继续失败关闭。生产 target 和 Helm Migration Job 保持不变;
- OceanBase 实测不支持 MySQL `GET_LOCK/RELEASE_LOCK`,因此未引入未经支持的命名锁。兼容启动器
仅适用于中央模板当前的单 API 副本;扩容前必须把迁移恢复为独立、串行的 Migration Job。
- Drone #132 发布安全诊断后,`/healthz` 报告 `migration_revision=20260902_0008`、
`migration_error_type=CommandError`,直接证实规范库落后 12 个 Revision。该错误发生在 Alembic
Config 定位阶段:非 editable 镜像中的模块位于 site-packages,不能从 `__file__` 反推 `/app`;
Revision 保持 `0008`,未发生部分 DDL。启动器改为使用 Docker 明确保证的 `/app` 工作目录后
再执行真实迁移。
- Drone #135 从 `dev@8ec40b7` 发布路径修复后,Alembic 已真实执行,健康诊断推进到
`migration_revision=20260908_0009`、`migration_error_type=OperationalError`。只读结构检查确认
OceanBase 的非事务 DDL 已完成 `0010` 的 Version、Chunk 和 Job 回填,但旧 Job 唯一索引仍被
外键使用;2 条 Job 均已正确回填且没有孤儿。先补普通外键索引,再按 `0010` 目标完成非空、
外键和唯一约束,经结构校验后将 Revision 推进到 `0010`;
- `0011~0013` 正常升级。OceanBase `4.3.5.4` 在 `0014` 拒绝复合自引用 Folder 外键,即使补齐
子表索引或使用会话级 `foreign_key_checks=0` 仍返回 `1215`。目标表为空,因此先创建完整列、
Check、Unique 和 Organization/KB 外键;自引用兼容为 `parent_folder_id → id ON DELETE CASCADE`。
现有 Folder 创建服务仍按 Organization + KB 查询父目录,跨域父目录会在应用写入前失败;数据库
复合同域自引用约束须在 OceanBase 升级并实测支持后恢复。结构校验后 `0015~0020` 全部成功;
- 向量派生表共有 73 条既有记录,仍缺少仓库已定义的 `002` 版本/激活列及 `003` NGRAM 全文索引。
先验证 `document_id` 全部有效,再按既有 DDL 精确回填并建索引。真实 `SHOW CREATE TABLE` 使用
`INT(11) DEFAULT NULL` 表示与 `INT NULL` 等价的列,Adapter 原校验未归一化该厂商渲染,且
`OceanBaseConfigurationError` 未进入安全 503 分类;两项均已补充回归测试并修复;
- Drone #138 从 `dev@f4be521` 成功构建并更新 GitOps。滚动后 `/healthz` 返回 `200`、
`migration_status=completed`、`migration_revision=20260909_0020`;本地使用当前 `main` 和授权配置
对真实 dev 依赖执行只读服务组合成功,并列出 2 条文档;
- 公网 `/api/v1/documents` 与 `/api/v1/administration/session` 已由原始 `500` 收敛为稳定的
`401 authentication required`。该轮仓库 `.env`、`.env.dev` 尚未配置
`API_IDENTITY_CREDENTIALS`,因此没有伪造认证成功验收;该身份配置缺口与本次
canonical/vector 500 根因相互独立,后续处置见下节。
## 2026-09-10 dev 首个 API 身份
- 用户明确授权为 dev 生成 API Token、写入仓库 `.env.dev` 并创建首个管理员;授权仅覆盖默认
dev 镜像及对应 Registry,不扩大到生产环境、MCP 或其他文件;
- 本地生成 64 位十六进制随机 Token 和独立 Organization/User UUID,不在命令、日志、文档或
回复中回显。Credential 使用 `user_token`、`client_id=admin-console`、
`subject=dev-platform-admin`,并声明 `knowledge:read/write/search/manage`;
- Bootstrap 已在真实 dev 规范库创建 `D-Robotics` Organization 和 Active
`Platform Administrator` User。随后通过真实 Uvicorn/TCP、真实 canonical/vector 依赖和生成
Token 验证 `/api/v1/administration/session` 与 `/api/v1/documents` 均为 `200`;当前管理员尚无
Knowledge Base/ACL,文档列表按权限失败关闭为空,不把两条旧公共文档自动归属给新身份。
- Drone #143 从 `dev@a172524` 成功构建并更新 GitOps。公网携带生成 Token 后,`/healthz`、
`/api/v1/administration/session`、`/api/v1/documents` 和 `/api/v1/knowledge-bases` 均返回 `200`,
Revision 为 `20260909_0020`;Document/Knowledge Base 数量均为 0。相同 Document 请求不携带
Token 时仍返回 `401`,证明认证成功路径和匿名失败关闭路径均已真实验证。
## 2026-09-08 开发侧闭环结论
- `main` 与 `dev` 已推进到开发交付基线;Drone #19 的镜像构建、推送和 `gitops-dev` 均成功;
- 公网 `knowledge-dev.d-robotics.cc` 的 `/`、Admin 页面与静态资源、`/healthz`、`/metrics`、
Document API 和 Search API 均正常;
- 远程两个文档均为 `ready`,真实搜索返回 Top5,证明 API、OceanBase、BOS、Embedding、Vector、
Reranker 和 Worker 主链路可用;
- `/mcp` 当前返回 `404`,远程 MCP/WorkBuddy 按用户决定搁置,不计入本次开发闭环,也不得标记
为完整 V1 MCP 已完成;
- 实际 GitOps Values/Secret、Namespace、Migration 运行记录、镜像拉取、工作负载副本/PDB、
滚动/回滚、监控、告警和备份转由运维确认。下方未勾选项目作为运维交接记录保留,不再阻塞
开发 Task;
- 双 Dockerfile 构建来源问题转为独立开发维护任务 `BUILD-001`。
闭环后核对发现远程 `dev` 又新增 `decef23`、`a314dff`、`3cbe42b` 三个未进入 `main` 的部署
提交。它们支撑当前公网运行,但包含 API/Worker 同容器监管和 root 运行,不作为已接受架构或
开发基线,也不在本记录中追认。差异审查和安全收敛由 `BUILD-001` 在下一次开发代码发布前完成;
当前运维人员不得假定后续 `main → dev` 推送会自动保留这些分支独有改动。
## 环境信息
- Kubernetes context:`2c1284bae51b4bf3a2e623557844c6d6@kubernetes`
- 集群版本/节点:Kubernetes `v1.34.2`,两个 `linux/amd64` 节点均 Ready
- Namespace / Release:等待 `dev` 流水线/Argo CD 实证,不沿用旧 `d-wiki-service` 假设
- API HTTPS Host:`knowledge-dev.d-robotics.cc`
- MCP HTTPS Host:`knowledge-dev.d-robotics.cc/mcp`
- 镜像仓库:`gitea.d-robotics.cc/d-cloud/knowledge-platform`
- 代码仓库:`d-cloud/knowledge-platform` 的 DEPLOY-003 分支已持续推送
- 首次 `dev` 触发提交:`3776bbf`;最新 dev 500 修复基线:`f4be521`
- 验收时间与操作者:2026-09-03,Codex 只读预检
目标 Namespace 已存在 `d-wiki-http-service` 与 `d-wiki-service` 两个 Deployment,以及同名
ClusterIP/NodePort Service,运行的镜像为既有 `d-robotics/d-wiki-service:1.0.13`,不是本项目
镜像。集群没有列出 IngressClass 或 Ingress;正式域名看起来由集群外入口转发至现有 NodePort。
最初提供的 kubeconfig 身份只有 `get/list/watch`。后续 `agent.conf` 已确认可创建 Namespace,
并对 Deployment、Service、ConfigMap、Secret、ServiceAccount、Job、PodDisruptionBudget 和
Ingress 具有 create/patch/delete 权限。为复用现有 ImagePullSecret 并进行最小中断回切,新
Release 计划部署到已有 `d-wiki-service` Namespace;入口和运行时依赖未就绪前仍未执行 Helm
安装或故障注入。
2026-09-03 曾确认复用 `d-wiki.d-robotics.cc` 并直接替换现有服务;随后目标更新为独立开发域名
`knowledge-dev.d-robotics.cc`,远程 `dev` 分支推送会触发流水线部署。后者是当前有效方案,原
NodePort 直接替换方案停止执行,旧 `d-wiki` Service/Deployment 不得修改。正式路径仍保持
`/mcp` 进入独立 MCP Gateway,其余路径进入 API/Admin,不合并两个应用进程。
仓库已为无 Ingress 环境增加可选双副本 Edge Proxy 和 NodePort Service。Edge 与 Ingress 互斥,
包含零不可用滚动策略、PDB、非 root/只读根文件系统、关联 ID、安全 JSON 访问日志和 MCP 流式
代理配置。`nginx:1.29.1-alpine` 的 manifest digest 已固定;实际部署时仍需镜像到批准的目标
Registry。Edge 模式通过 Helm 单测、16 个资源的 strict Kubernetes Schema 校验,以及 UID/GID
`10001` + 只读根文件系统的容器启动烟测。
最终 Edge 清单使用临时 NodePort `30083` 完成目标 Kubernetes API server-side dry-run,16 个资源
全部通过且未创建对象;`30082` 已由 `agent-hub/data-dashboard-web` 占用,已明确排除。集群内没有
监听 `5671/5672/15672` 的 Service,也没有 RabbitMQ Deployment/StatefulSet,不能复用现存 Broker。
用户提供的 `d-cloud/knowledge-platform.git` 已确认是 Git 代码仓库,不是可直接交给 Helm 的镜像
坐标;同域 `/v2/` 端点要求鉴权并支持 OCI。初次核对时本机 Docker 未登录,Git Keychain 凭据
没有在未授权状态下复用于容器 Registry。获得授权后已用实际 push 验证单仓库镜像坐标
`gitea.d-robotics.cc/d-cloud/knowledge-platform` 及写权限,API/MCP/Worker/Edge 使用独立不可变 tag。
获得用户明确授权后,使用本机 Gitea Keychain 凭据临时登录同域 OCI Registry,以下
`linux/amd64` 镜像均已成功推送,临时 Docker 配置随后已删除:
- `api-c3529ce`:`sha256:133678975e232add5c0e84aa3f67a0d05a02608161c1c9ec9691b738598c7bd8`
- `mcp-c3529ce`:`sha256:6bd6e142832063238561264d73b06f8ab2ad7d4ce1372933ad8d7be992449ec7`
- `worker-c3529ce`:`sha256:e2036b9c8c0bf119417a27e5615721a36d7bd71e9dd2e3a2acd6a8d505d03e18`
- `edge-nginx-1.29.1`:`sha256:77d740efa8f9c4753f2a7212d8422b8c77411681971f400ea03d07fe38476cac`
远程 `main` 随后新增 `.drone.yml` 与根 `Dockerfile`,并由用户指定为最新配置依据;本任务分支
已合入这两个提交。`.drone.yml` 加载仓库外的中央模板 `argocd-python.jsonnet`,且设置字符串
`DisableDockerfile: "true"`;仓库内无法静态确认该参数的真实语义。同时,现有 Bake 与镜像静态
测试仍指向 `deploy/docker/Dockerfile`,而上述三个应用镜像正是由该旧路径构建。因此三个应用
digest 降级为候选记录,在完成 Drone 模板实证、根 Dockerfile 单一来源和 CI 重新构建前,不得
用于正式 Values 或现场切流。Edge digest 不依赖应用 Dockerfile,但仍须完成集群拉取验证。
重新规划后的严格门禁记录在
[`DEPLOY_003_REPLAN.md`](../plans/DEPLOY_003_REPLAN.md):D3-G0 Drone 模板实证、D3-G1
Dockerfile 收敛、D3-G2 CI 新镜像、D3-G3 集群拉取、D3-G4 外部依赖、D3-G5 开发域名部署、D3-G6
可用性、D3-G7 发布回滚、D3-G8 远程验收。D3-G5 已调整为开发域名流水线部署而非临时入口;
下面的现场复选项只有在前四个门禁完成后才开始执行。
2026-09-03 已在确认远程 `dev` 仍位于 `ceca075` 且为当前提交祖先后,将 `3776bbf` 以快进方式
推送至 `dev`,远程接受该引用更新。推送后立即及约 30 秒后,从最终用户路径探测
`knowledge-dev.d-robotics.cc` 的 `/healthz`、`/admin/`、`/mcp`,两次均在 DNS 阶段失败,HTTP
状态为 `000`;因此尚不能验证 TLS 或应用路由。使用现有 `agent.conf` 只读检查 Deployment、Pod、
Service 和 Ingress,也没有发现本项目新资源,旧 `d-wiki` 工作负载保持原状。该观察可能表示
流水线尚未完成、流水线失败、DNS 尚未配置,或流水线部署到另一个集群;在取得实际 Drone 构建
状态、日志与 Argo CD 目标前不能区分,D3-G0 保持未完成。
2026-09-07 用户确认该开发域名只在公网环境解析。因此本机或受限执行环境 DNS 失败不再作为
流水线失败或服务不可用的结论;HTTPS、根路径、Admin、API 与 MCP 必须由公网探针或最终用户
网络路径验收。当前分支相对首次触发的 `dev@3776bbf` 增加 7 个提交,包含持久开发配置、根路径
重定向以及真实 OceanBase 单文件上传响应修复;须合并 `main` 后重新推进 `dev`,以新流水线产物
作为后续门禁依据。
运行字段复核确认受控提交的 `.env.dev` 已覆盖 OceanBase、BOS、Embedding、Reranker、Vector
Store、共享外部 RabbitMQ 和 Admin/MCP 地址。共享 RabbitMQ 已在本地完成认证及真实异步入库,
不再属于“Broker 未提供”阻塞项;远程 Pod 到该 Broker 的网络与认证仍待验证。`MCP_AUTH_TOKEN`
仍为占位值,必须在远程启用前替换为固定 token,并与 WorkBuddy 客户端一致。BUILD-002 后,
`.dockerignore` 排除 `.env*` 并仅对白名单 `.env.dev` 例外;默认最终 `dev` target 固定设置
`EKP_SETTINGS_PROFILE=dev-image` 并以 `0400` 权限复制该文件。真实发布后的
`environment=development` 已证明默认 dev 镜像自动加载生效。显式生产 target 仍必须从实际
GitOps `values.yaml` 或渲染 Deployment 证明 Runtime Secret/ConfigMap 注入,不得复用此例外。
创建 Namespace 级 `knowledge-platform-registry` ImagePullSecret 属于凭据长期存储,尚未获得单独
明确授权,因此未创建,集群拉取烟测也尚未执行。
目标集群没有 StorageClass,现有 PersistentVolume 全部 Bound,因此不在该集群补建 RabbitMQ。
项目已采用用户提供的共享外部 RabbitMQ;本地认证和异步任务执行成功,目标 Pod 的网络连通、
凭据注入、队列隔离与恢复行为仍须在 D3-G4~D3-G6 验证。
## 运维接管的自动化验收
以下项目由运维人员确认,保留未勾选状态表示开发侧没有替运维作出结论;它们不再阻塞
`DEPLOY-003` 开发侧闭环。
- [ ] 同一 Values 连续两次 `upgrade --install` 成功
- [ ] Migration Hook 成功且失败保护已核对
- [ ] API/MCP/Edge 各至少两个 Ready/Available 副本
- [ ] API/MCP/Edge PodDisruptionBudget 生效
- [ ] Admin、健康检查、`/metrics` 和 MCP 使用有效 HTTPS
- [ ] API/MCP/Edge 滚动重启期间连续健康探测无失败
- [ ] Worker 非优雅中断后同一 Job 自动恢复成功
- [ ] 验收合成文档已清理
- [ ] Manifest、镜像与验收日志凭据扫描为 0
仓库与清单预检:Ruff Format/Lint、Mypy、Bash 语法、三种 Helm lint、Alembic Head/离线 SQL、
Unit + Integration `202 passed`、RAG Evaluation `5 passed`;目标 API Server dry-run 16 个资源通过。
真实安装、升级与故障注入的执行结果仍待现场依赖就绪后记录。
## 远程 MCP / WorkBuddy(搁置)
- [ ] 官方 MCP Client 只发现当前五个批准的 P0 Tool
- [ ] 官方 MCP Client 完成一次真实、有来源的检索与 Source/Document/Section/Knowledge Base 读取链
- [ ] 最终用户网络路径上的 WorkBuddy 连接成功
- [ ] WorkBuddy 只发现当前五个批准 Tool 并完成真实检索与来源读取链
客户端版本、网络区域和非敏感结果:按用户决定搁置,恢复任务后再执行。
## 运维接管的运行保障
- [ ] 日志采集可按 `request_id` / `job_id` 定位
- [ ] 最小指标已被环境监控系统抓取
- [ ] Ready 副本、5xx、耗时、Pod 重启和 Job 失败告警责任人已确认
- [ ] OceanBase/BOS/RabbitMQ 备份责任人、周期和恢复顺序已确认
## 结论
开发侧实现、工程门禁、`dev` 流水线和公网 Admin/API/Worker/Search 验证均已完成,`DEPLOY-003`
按调整后的开发责任范围闭环。未勾选的集群资源、滚动/回滚、监控和备份项目由运维接管,不表示
开发任务失败,也不得伪装为开发已验证。远程 MCP/WorkBuddy 继续搁置;完整 V1 MCP 目标不变。
构建来源收敛转入 `BUILD-001`,不再回写本验收记录的历史门禁。

View File

@ -0,0 +1,58 @@
# Docling Parser Adapter
本文记录 `EKP-026` 的本地 Docling Provider 边界。Adapter 已通过真实格式样例验证,`EKP-028`
已建立规范解析模型,`EKP-029` 已在 `structure_v2` 中消费表格结构并接入 P0 上传/Worker 路由。
## 版本与依赖
- 依赖:`docling-slim[convert-core,format-html,format-office,format-pdf]>=2.126,<3`;
- 当前锁定版本:`docling-slim 2.126.0`;
- `format-pdf` 是 Docling `DocumentConverter` 当前导入链所需的兼容依赖,不表示本 Adapter 接管
PDF;PDF 仍由轻量 Parser 或显式 MinerU 路由处理;
- 未安装 `models-local`,因此不会把 Torch、Transformers 或 Docling 本地 OCR 模型栈带入应用镜像。
上游依据:
- [Docling 支持格式](https://docling-project.github.io/docling/usage/supported_formats/)
- [DocumentConverter 与 DocumentStream](https://github.com/docling-project/docling/blob/main/docs/v2.md)
- [DoclingDocument 结构和遍历](https://docling-project.github.io/docling/concepts/docling_document/)
- [docling-slim 可选依赖](https://github.com/docling-project/docling/blob/main/packages/docling-slim/README.md)
## 当前 Adapter 边界
`DoclingParser` 通过现有 `Parser` 协议接收内存流,支持:
- DOCX;
- PPTX;
- XLSX;
- CSV;
- HTML/HTM。
输入受 `MAX_UPLOAD_BYTES` 约束,不写临时文件。HTML 的远程资源、本地文件和图片抓取均显式
关闭,避免文档内容触发外部或宿主文件访问。Docling 实现异常会统一转换为不包含底层详情的
`DoclingConversionError`;该本地格式错误属于永久解析失败,不进入瞬时重试。
Adapter 按 Docling body reading order 映射 Title、Heading、Paragraph、List、Table 和
Image Reference,并保留一基 Page。XLSX/CSV 进一步保留 Sheet、Header、Cell 与一基闭区间行列
坐标;可读 Markdown Table 只是兼容内容,结构事实以 schema v4 Cell Grid 为准。
## 已完成启用顺序
1. `EKP-026`(已完成):稳定 Docling Adapter 与上游结构读取边界;
2. `EKP-028`(已完成):扩展版本化规范解析文档模型,覆盖 PRD 要求的节点及来源位置;
3. `EKP-029`(已完成):让结构感知与 Spreadsheet Chunk 消费规范结构;
4. PPTX、XLSX、CSV 和 HTML/HTM 已加入 `structure_v2` 上传白名单并路由到 Docling;DOCX
保持轻量 Parser 或显式 MinerU 路由。
数据库仍为 `structure_v1` Active Profile 时,新格式保持关闭;完成 v2 Shadow Build/Verify/Activate
并重启运行进程后才启用,详见 [`STRUCTURED_CHUNKING.md`](STRUCTURED_CHUNKING.md)。不得绕过
schema v4 Cell Grid,将 Markdown 表格文本直接作为电子表格 Chunk 完成证据。
## 验证
离线测试使用内存生成的 DOCX、PPTX、XLSX,以及 CSV、HTML 样例调用真实 Docling 转换器,同时
覆盖读取顺序、章节路径、表格可读中间文本、HTML 抓取禁用、大小边界和安全错误信息。运行:
```bash
uv run pytest tests/unit/test_docling_parser.py tests/unit/test_provider_wiring.py -q
```

View File

@ -0,0 +1,47 @@
# Embedding Provider 抽象
本文记录 `EKP-040` 的 Provider-neutral Embedding 契约。千帆和 Qwen3 的具体运行配置分别见
[`QIANFAN_EMBEDDING_SETUP.md`](QIANFAN_EMBEDDING_SETUP.md) 与
[`QWEN_EMBEDDING_SETUP.md`](QWEN_EMBEDDING_SETUP.md),不进入 REST/MCP 契约。
## 运行时能力
`EmbeddingProvider` 是可运行时检查的结构协议,固定公开 `model_name`、`dimensions`、
`max_batch_size` 和按输入顺序返回向量的 `embed()`。装配到索引或查询服务前必须满足:
- 模型名为无首尾空白的非空稳定字符串;
- 向量维度和单次最大批量均为正整数,布尔值不作为整数接受;
- Provider 模型名、维度与所绑定的版本化 Index Profile 严格一致。
Index Profile 同时记录模型最大输入 Token、可选 Query Instruction、Float32 数据类型及由维度推导
的原始存储需求;Dense 装配要求 Query Provider 的可选 `query_instruction` 与 Profile 精确一致,
缺失或漂移时失败关闭。完整配置边界见 [`INDEX_PROFILE.md`](INDEX_PROFILE.md)。
Credential、Endpoint、厂商请求/响应和 SDK 类型只存在于具体 Adapter;Domain、REST、MCP 与向量
存储只依赖该中立协议。
## 向量输出门禁
Provider 输出在进入可重建索引或 Dense Query 前统一校验:外层与单向量均为不可变 Tuple,数量与
输入一致,维度与 Profile 一致,全部值为有限数值,且每个向量的余弦范数为正。任一条件不满足均
失败关闭,不写入部分索引,也不向 VectorStore 发起查询。
空批量返回空 Tuple;非空输入必须保持顺序。具体 Provider 可以附加输入长度、批量、Query
Instruction 和安全 Endpoint 限制,但不能弱化上述公共边界。
## 确定性 Fake
`FakeEmbeddingProvider` 仅用于单元测试、集成测试和离线开发验证。它将模型身份与文本共同写入
分块 SHA-256 输入,扩展为指定维度并执行 L2 归一化,因此具有稳定、非零、非循环和输入顺序可
验证的输出;它不联网、不读取 Credential,也不模拟真实模型语义质量。
Fake 结果不得用于 Embedding 选型、生产质量、延迟、吞吐、成本或 GPU 结论。`EKP-101` 仍必须在
获得真实 Qwen Endpoint/GPU 环境后使用同一 Golden Dataset 执行真实候选 Benchmark。
Fake 的非法批量、空文本和输入类型统一返回 `EmbeddingInputError`,与生产 Provider 的永久请求
错误分类一致,避免异步 Job 把确定性测试输入错误误判为暂时故障。
## 验证边界
单元与集成测试覆盖运行时协议、非法能力、稳定性、顺序、跨摘要块维度、单位范数、批量/空文本
错误,以及索引和 Dense Query 对错误输出的失败关闭。本任务不新增或改变 API、前端页面、按钮或
用户入口,因此不触发新的真实运行态 E2E;检索核心变更仍执行完整 RAG 回归评估。

View File

@ -0,0 +1,30 @@
# Fake Provider 框架
`EKP-007` 的 Fake 仅用于确定性测试和离线开发,不是运行时降级方案。当前公共替身包括:
| Provider 边界 | 确定性实现 | 共享契约 |
|---|---|---|
| Embedding | `FakeEmbeddingProvider` | 能力、请求错误分类、向量数量/维度/有限非零 Tuple |
| VectorStore | `FakeVectorStore` | Row/Query、Profile/Version/ACL 预过滤、原子写入与稳定排序 |
| Reranker | `FakeRerankerProvider` | Query/候选/Top-N、索引映射和有限降序分数 |
| LLM | `FakeLLMProvider` | 模型/处理位置能力和有界非空 Compilation Result |
Parser 使用真实的轻量级/Docling 本地解析实现,对象存储使用真实的 `LocalObjectStorage` 文件语义,
canonical 数据测试使用 SQLite 执行同一 SQLAlchemy 模型;这些组件不伪装成外部 Provider。
## 强制边界
- 公共 Fake 必须实现对应运行时 Protocol,并复用 Provider-neutral 输入/输出验证器和错误分类。
- Fake 实现不得读取环境配置或凭据,不得导入网络客户端,也不得连接 SaaS、数据库、GPU 或模型。
- 同一输入和显式模型身份必须得到稳定结果;默认模型名必须明确以 `fake-` 标识。
- 生产 `apps/api/src/ekp_api/providers.py` 不得导入或回退到任何 `Fake*`。真实配置缺失时继续失败关闭。
- 测试专用 HTTP Client、故障注入类可以留在具体测试文件中,不作为公共 Provider 实现导出。
上述边界由 `tests/unit/test_fake_provider_framework.py` 跨包检查;各 Provider 自身的详细契约仍由对应
单元/集成测试负责。
## 结果使用限制
Fake 可以证明协议、业务编排、ACL、来源追溯、幂等性和错误处理的可重复行为;不得用于模型选型、
生产质量、真实 Recall、延迟、吞吐、容量、成本或 GPU 结论。`EKP-101~102` 仍须等待真实候选环境,
任何确定性测试数字都不能替代 Benchmark 报告。

Some files were not shown because too many files have changed in this diff Show More