hy4/_kp 改为普通目录(去除嵌套git引用)
This commit is contained in:
parent
a74a276767
commit
9658cee5e6
1
hy4/_kp
1
hy4/_kp
@ -1 +0,0 @@
|
||||
Subproject commit 853050f38e268134a31d3d9b785b4f165382bc98
|
||||
19
hy4/_kp/.dockerignore
Normal file
19
hy4/_kp/.dockerignore
Normal file
@ -0,0 +1,19 @@
|
||||
# 排除生产镜像不需要的本地状态和生成文件。
|
||||
.env*
|
||||
!.env.dev
|
||||
.git
|
||||
.venv
|
||||
.*_cache
|
||||
__pycache__
|
||||
*.py[cod]
|
||||
*.sqlite*
|
||||
*.egg-info
|
||||
node_modules
|
||||
celerybeat-schedule*
|
||||
data
|
||||
dist
|
||||
docker-compose.dev.yml
|
||||
storage
|
||||
tmp
|
||||
tests
|
||||
docs
|
||||
5
hy4/_kp/.drone.yml
Normal file
5
hy4/_kp/.drone.yml
Normal file
@ -0,0 +1,5 @@
|
||||
kind: template
|
||||
load: knowledge-platform.jsonnet
|
||||
data:
|
||||
App: "knowledge-platform,knowledge-platform-worker,knowledge-platform-mcp"
|
||||
DisableDockerfile: "true"
|
||||
15
hy4/_kp/.editorconfig
Normal file
15
hy4/_kp/.editorconfig
Normal file
@ -0,0 +1,15 @@
|
||||
root = true
|
||||
|
||||
[*]
|
||||
charset = utf-8
|
||||
end_of_line = lf
|
||||
insert_final_newline = true
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
trim_trailing_whitespace = true
|
||||
|
||||
[*.py]
|
||||
indent_size = 4
|
||||
|
||||
[Makefile]
|
||||
indent_style = tab
|
||||
17
hy4/_kp/.gitattributes
vendored
Normal file
17
hy4/_kp/.gitattributes
vendored
Normal file
@ -0,0 +1,17 @@
|
||||
* text=auto eol=lf
|
||||
*.sh text eol=lf
|
||||
*.py text eol=lf
|
||||
*.md text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.yml text eol=lf
|
||||
*.json text eol=lf
|
||||
*.toml text eol=lf
|
||||
*.ts text eol=lf
|
||||
*.tsx text eol=lf
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
*.jpeg binary
|
||||
*.pdf binary
|
||||
*.docx binary
|
||||
*.pptx binary
|
||||
*.xlsx binary
|
||||
58
hy4/_kp/.gitignore
vendored
Normal file
58
hy4/_kp/.gitignore
vendored
Normal file
@ -0,0 +1,58 @@
|
||||
# 本地环境;环境变体默认不得进入版本库
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Python 生成文件
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.venv/
|
||||
venv/
|
||||
*.egg-info/
|
||||
dist/
|
||||
build/
|
||||
|
||||
# 管理端 Web 工具链生成文件
|
||||
node_modules/
|
||||
npm-debug.log*
|
||||
|
||||
# 编辑器与操作系统
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# 运行状态与本地数据
|
||||
logs/
|
||||
tmp/
|
||||
data/
|
||||
storage/
|
||||
uploads/
|
||||
celerybeat-schedule*
|
||||
models/
|
||||
artifacts/
|
||||
reports/
|
||||
|
||||
# 凭据
|
||||
*.pem
|
||||
*.key
|
||||
*.p12
|
||||
credentials.json
|
||||
secrets.json
|
||||
|
||||
# 模型产物
|
||||
*.bin
|
||||
*.safetensors
|
||||
*.onnx
|
||||
*.pt
|
||||
*.pth
|
||||
|
||||
# 本地数据库
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
1
hy4/_kp/.node-version
Normal file
1
hy4/_kp/.node-version
Normal file
@ -0,0 +1 @@
|
||||
20.18.0
|
||||
1
hy4/_kp/.python-version
Normal file
1
hy4/_kp/.python-version
Normal file
@ -0,0 +1 @@
|
||||
3.12
|
||||
63
hy4/_kp/AGENTS.md
Normal file
63
hy4/_kp/AGENTS.md
Normal file
@ -0,0 +1,63 @@
|
||||
# 企业知识平台 — Codex 规则
|
||||
|
||||
## 开始前必读
|
||||
|
||||
修改代码前,先阅读与任务相关的以下文档:
|
||||
|
||||
- `docs/README.md`
|
||||
- `docs/product/PRD.md`
|
||||
- `docs/architecture/ARCHITECTURE.md`
|
||||
- `docs/architecture/ADR.md`
|
||||
- `docs/api/API_CONTRACT.md`
|
||||
- `docs/api/MCP_CONTRACT.md`
|
||||
- `docs/plans/V1_EXECUTION_PLAN.md`
|
||||
- `docs/plans/TASKS.md`
|
||||
- `docs/development/DEFINITION_OF_DONE.md`
|
||||
|
||||
除非任务明确要求修改架构,否则不得重新设计已接受的架构。
|
||||
|
||||
## 架构不变量
|
||||
|
||||
1. API First。
|
||||
2. V1 采用模块化单体,并配备独立的异步/GPU Worker 和 MCP Gateway。
|
||||
3. OceanBase 关系表与对象存储共同保存规范数据(canonical data)。
|
||||
4. 即使与规范关系表部署在同一数据库中,OceanBase 向量表也只是可重建索引,而不是事实源。
|
||||
5. 业务服务依赖 `VectorStore` 抽象,绝不直接依赖 Mochow。
|
||||
6. Embedding、Reranker、解析、OCR、对象存储和 LLM 均使用 Provider 接口。
|
||||
7. 原始来源文档不可变。
|
||||
8. Compiled Knowledge 是派生数据,且必须可追溯到来源文档版本和 Chunk。
|
||||
9. ACL 必须在检索前或检索过程中解析。禁止先执行全局 Top-K 检索再做过滤。
|
||||
10. MCP 只暴露业务能力,不暴露基础设施或 Provider 细节。
|
||||
11. Index Profile 必须版本化;模型或 Chunk 变更应先构建 Shadow Index,再激活。
|
||||
12. 不得把 V1 拆分成不必要的微服务。
|
||||
13. 只有在用户明确授权具体文件和具体环境时,才允许提交凭据、密钥、密码和环境 Endpoint;
|
||||
所有未经批准的 Secret 都必须排除在 Git 之外。
|
||||
14. 核心检索变更必须补充测试并执行 RAG 回归评估。
|
||||
|
||||
## 任务纪律
|
||||
|
||||
- 每次只处理一个 Task ID。
|
||||
- 默认只使用当前仓库根目录作为工作区,在同一目录内按任务顺序切换分支;除非用户明确要求并行
|
||||
隔离或授权额外 worktree,不得为每个 Task 新建同级项目目录。切换分支前必须先用提交或命名
|
||||
stash 保存并核验未完成修改,禁止覆盖、丢弃或 reset 用户工作。文档、任务分支、里程碑 Tag、
|
||||
stash 与 worktree 的盘点、归档、删除和恢复流程遵循
|
||||
`docs/development/REPOSITORY_GOVERNANCE.md`。
|
||||
- 严格限定在当前任务范围内。
|
||||
- 不得顺手实现未来任务。
|
||||
- 完成 V1 开发功能是第一优先级。文档治理、Benchmark、部署/现场验收等非开发工作使用独立任务
|
||||
类型和执行队列,同步记录状态,但不得成为后续开发 Task 的依赖或阻塞项。
|
||||
- 行为发生变化时,更新相关契约和文档。
|
||||
- 添加有价值的代码注释或文档字符串,并在相关文档或进度日志中记录文件或行为变化。
|
||||
- 提交经明确授权的环境凭据时,必须记录授权范围。
|
||||
- 验证完成后,先推送任务分支并合并到 `main`,再开始下一个 Task ID。
|
||||
|
||||
## 完成前检查
|
||||
|
||||
运行相关检查:格式检查、Lint、类型检查、单元测试、相关集成测试、Migration 检查、API/MCP 契约检查;如果涉及检索变更,还必须运行 RAG 评估。
|
||||
|
||||
每次新增或改变 API、前端页面、按钮或用户入口,都必须在真实启动的应用上完成端到端测试:通过
|
||||
真实 HTTP 请求或浏览器操作,从入口贯穿对应应用服务和持久化/对象存储边界,并记录非敏感结果。
|
||||
仅有单元测试、`TestClient`、静态文本断言、Mock 或截图不能替代该端到端测试;外部 Provider 与
|
||||
目标集群现场验收仍作为独立非开发任务记录,不阻塞已由本地真实应用链路证明的功能开发。
|
||||
|
||||
报告变更文件、测试及结果、尚未解决的风险和后续任务。
|
||||
22
hy4/_kp/CODEX_START_HERE.md
Normal file
22
hy4/_kp/CODEX_START_HERE.md
Normal file
@ -0,0 +1,22 @@
|
||||
# Codex 入门指南
|
||||
|
||||
不要要求 Codex 在一个任务中完成整个 V1。
|
||||
|
||||
先阅读 `AGENTS.md` 和 `docs/README.md`。后者定义文档权威性、生命周期和当前活动计划,避免从历史计划中误判当前范围。
|
||||
|
||||
推荐的第一个提示词:
|
||||
|
||||
> 你正在接手 Enterprise Knowledge Platform 仓库。暂时不要修改代码。阅读 `AGENTS.md` 及其引用的产品、架构、API/MCP、执行计划、任务和完成定义文档,并检查仓库。然后输出:(1) 当前实现状态,(2) 与 V1 的差距分析,(3) 五项主要技术风险,(4) 接下来推荐的 1~3 个 Task ID,以及 (5) 阻塞项。不要重新设计已接受的 ADR。
|
||||
|
||||
推荐节奏:
|
||||
|
||||
1. 规划一个任务。
|
||||
2. 审查计划。
|
||||
3. 实现一个任务。
|
||||
4. 运行验证。
|
||||
5. 按 PR 标准进行自审。
|
||||
6. 提交或创建 PR。
|
||||
7. 进入下一个任务。
|
||||
|
||||
在条件允许时,遵循一个 Task 对应一个分支和一个 PR。任务切换、文档精简、分支归档和里程碑
|
||||
Tag 规则见 `docs/development/REPOSITORY_GOVERNANCE.md`。
|
||||
84
hy4/_kp/Dockerfile
Normal file
84
hy4/_kp/Dockerfile
Normal file
@ -0,0 +1,84 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
# 一个锁定的运行时构建三个可独立部署的进程镜像。
|
||||
|
||||
ARG PYTHON_IMAGE=python:3.12.14-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
|
||||
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.12.5@sha256:e85be844203885286c60ffad8a858d48afb6c5a5c237ca0e67f12e74b8f174b1
|
||||
|
||||
FROM ${UV_IMAGE} AS uv-bin
|
||||
|
||||
FROM ${PYTHON_IMAGE} AS runtime
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
UV_COMPILE_BYTECODE=1 \
|
||||
UV_LINK_MODE=copy \
|
||||
VIRTUAL_ENV=/app/.venv \
|
||||
PATH=/app/.venv/bin:${PATH} \
|
||||
HOME=/tmp/ekp \
|
||||
XDG_CACHE_HOME=/tmp/ekp/cache \
|
||||
APP_ENV=production \
|
||||
API_HOST=0.0.0.0 \
|
||||
MCP_HOST=0.0.0.0 \
|
||||
ADMIN_WEB_ROOT=/app/apps/admin-web
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=uv-bin /uv /uvx /usr/local/bin/
|
||||
COPY pyproject.toml uv.lock README.md alembic.ini ./
|
||||
COPY apps ./apps
|
||||
COPY packages ./packages
|
||||
COPY migrations ./migrations
|
||||
|
||||
# 固定依赖锁文件,不使用可编辑安装,也不把开发依赖带入运行镜像。
|
||||
RUN uv sync --frozen --no-dev --no-editable \
|
||||
&& test -f /app/apps/admin-web/index.html \
|
||||
&& groupadd --gid 10001 ekp \
|
||||
&& useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin ekp \
|
||||
&& mkdir -p /tmp/ekp/cache \
|
||||
&& chown -R 10001:10001 /tmp/ekp \
|
||||
&& rm -rf /root/.cache/uv
|
||||
|
||||
USER 10001:10001
|
||||
|
||||
# MCP 服务
|
||||
FROM runtime AS mcp
|
||||
|
||||
EXPOSE 8766
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
|
||||
CMD ["ekp-mcp"]
|
||||
|
||||
# 异步 Worker
|
||||
FROM runtime AS worker
|
||||
|
||||
# 本地镜像入口会同时启动 Celery Beat;可写工作目录避免把调度状态写入只读应用目录。
|
||||
WORKDIR /tmp
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
|
||||
CMD ["ekp-worker"]
|
||||
|
||||
# API 服务;放在最后,因此不指定 target 时默认构建 API
|
||||
FROM runtime AS api
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=2)"]
|
||||
|
||||
CMD ["uvicorn", "ekp_api.main:app", "--host", "0.0.0.0", "--port", "8000", "--no-access-log"]
|
||||
|
||||
# 中央 dev 流水线构建默认最终 target;仅该 target 携带已授权的 dev 配置。
|
||||
# 显式 api/mcp/worker target 继续保持无环境文件的生产镜像边界。
|
||||
FROM api AS dev
|
||||
|
||||
ENV APP_ENV=development \
|
||||
EKP_SETTINGS_PROFILE=dev-image
|
||||
|
||||
COPY --chown=10001:10001 --chmod=0400 .env.dev /app/.env.dev
|
||||
|
||||
# 中央单副本 dev 流水线没有 Migration Job;仅 dev target 在监听流量前升级规范库。
|
||||
CMD ["python", "-m", "ekp_api.dev_startup"]
|
||||
112
hy4/_kp/Makefile
Normal file
112
hy4/_kp/Makefile
Normal file
@ -0,0 +1,112 @@
|
||||
.PHONY: help lock install toolchain-check run run-mcp run-worker images images-print images-push deploy deploy-lint deploy-render cluster-acceptance mcp-remote-acceptance format format-check lint typecheck test test-unit test-integration migration-check eval live-acceptance up down
|
||||
.PHONY: web-check ci test-e2e
|
||||
|
||||
UV_CACHE_DIR ?= /tmp/ekp-uv-cache
|
||||
export UV_CACHE_DIR
|
||||
UV ?= uv
|
||||
UV_RUN := $(UV) run --locked
|
||||
|
||||
help:
|
||||
@echo "Enterprise Knowledge Platform"
|
||||
@echo " lock Intentionally refresh the Python dependency lock"
|
||||
@echo " install Install development dependencies"
|
||||
@echo " toolchain-check Validate the locked Python toolchain contract"
|
||||
@echo " web-check Validate the zero-build Admin Web assets"
|
||||
@echo " ci Run the complete provider-free CI baseline"
|
||||
@echo " run Start the development API"
|
||||
@echo " run-mcp Start the authenticated MCP gateway"
|
||||
@echo " run-worker Start ingestion/compilation workers and recovery scheduler"
|
||||
@echo " images Build production images for the current builder"
|
||||
@echo " images-print Render the multi-architecture image build plan"
|
||||
@echo " images-push Build and push amd64/arm64 production images"
|
||||
@echo " deploy Install or upgrade Kubernetes with VALUES=<file>"
|
||||
@echo " deploy-lint Validate Helm values and templates"
|
||||
@echo " deploy-render Render non-secret Kubernetes manifests"
|
||||
@echo " cluster-acceptance Validate the explicitly selected target cluster"
|
||||
@echo " mcp-remote-acceptance Verify five-tool MCP client compatibility"
|
||||
@echo " format Format code"
|
||||
@echo " format-check Check formatting without modifying files"
|
||||
@echo " lint Run linters"
|
||||
@echo " typecheck Run static type checks"
|
||||
@echo " test Run default tests"
|
||||
@echo " test-unit Run unit tests"
|
||||
@echo " test-integration Run integration tests"
|
||||
@echo " test-e2e Run real-process loopback API/MCP tests"
|
||||
@echo " migration-check Validate the forward migration path"
|
||||
@echo " eval Run RAG evaluation"
|
||||
@echo " live-acceptance Run controlled target-provider acceptance"
|
||||
@echo " up Start local dependencies"
|
||||
@echo " down Stop local dependencies"
|
||||
|
||||
lock:
|
||||
$(UV) lock
|
||||
install:
|
||||
$(UV) sync --locked --all-groups
|
||||
toolchain-check:
|
||||
$(UV) lock --check
|
||||
$(UV) sync --locked --all-groups --dry-run
|
||||
$(UV_RUN) pytest -q tests/unit/test_python_toolchain.py
|
||||
web-check:
|
||||
npm --prefix apps/admin-web ci --ignore-scripts --no-audit --no-fund
|
||||
npm --prefix apps/admin-web run check
|
||||
ci:
|
||||
$(MAKE) toolchain-check
|
||||
$(MAKE) web-check
|
||||
$(MAKE) format-check
|
||||
$(MAKE) lint
|
||||
$(MAKE) typecheck
|
||||
$(MAKE) test
|
||||
$(MAKE) migration-check
|
||||
$(MAKE) eval
|
||||
$(MAKE) test-e2e
|
||||
run:
|
||||
$(UV_RUN) uvicorn ekp_api.main:app --host "$${API_HOST:-0.0.0.0}" --port "$${API_PORT:-8000}"
|
||||
run-mcp:
|
||||
$(UV_RUN) ekp-mcp
|
||||
run-worker:
|
||||
$(UV_RUN) ekp-worker
|
||||
images:
|
||||
docker buildx bake -f deploy/docker/docker-bake.hcl
|
||||
images-print:
|
||||
docker buildx bake -f deploy/docker/docker-bake.hcl --print
|
||||
images-push:
|
||||
docker buildx bake -f deploy/docker/docker-bake.hcl --push
|
||||
deploy:
|
||||
./scripts/deploy.sh "$(VALUES)"
|
||||
deploy-lint:
|
||||
@test -n "$(VALUES)" || (echo "VALUES must point to an environment values file" >&2; exit 2)
|
||||
helm lint deploy/helm/knowledge-platform --values "$(VALUES)"
|
||||
deploy-render:
|
||||
@test -n "$(VALUES)" || (echo "VALUES must point to an environment values file" >&2; exit 2)
|
||||
helm template "$${RELEASE:-knowledge-platform}" deploy/helm/knowledge-platform --namespace "$${NAMESPACE:-knowledge-platform}" --values "$(VALUES)"
|
||||
cluster-acceptance:
|
||||
./scripts/cluster_acceptance.sh "$(VALUES)"
|
||||
mcp-remote-acceptance:
|
||||
$(UV_RUN) python scripts/mcp_remote_acceptance.py
|
||||
format:
|
||||
$(UV_RUN) ruff format .
|
||||
format-check:
|
||||
$(UV_RUN) ruff format --check .
|
||||
lint:
|
||||
$(UV_RUN) ruff check .
|
||||
typecheck:
|
||||
$(UV_RUN) mypy apps/api/src apps/mcp-server/src packages tests
|
||||
test:
|
||||
$(UV_RUN) pytest tests/unit tests/integration
|
||||
test-unit:
|
||||
$(UV_RUN) pytest tests/unit
|
||||
test-integration:
|
||||
$(UV_RUN) pytest tests/integration
|
||||
test-e2e:
|
||||
$(UV_RUN) pytest -q tests/e2e
|
||||
migration-check:
|
||||
$(UV_RUN) python scripts/migration_contract.py
|
||||
eval:
|
||||
$(UV_RUN) pytest tests/evaluation
|
||||
$(UV_RUN) ekp-evaluate
|
||||
live-acceptance:
|
||||
$(UV_RUN) ekp-live-acceptance
|
||||
up:
|
||||
docker compose -f docker-compose.dev.yml up -d
|
||||
down:
|
||||
docker compose -f docker-compose.dev.yml down
|
||||
127
hy4/_kp/README.md
Normal file
127
hy4/_kp/README.md
Normal file
@ -0,0 +1,127 @@
|
||||
# 企业知识平台
|
||||
|
||||
面向企业内部的知识基础设施,提供文档入库、可追溯检索、REST API、MCP Gateway 和管理控制台。
|
||||
|
||||
## 当前状态
|
||||
|
||||
当前开发基线已完成至 `EKP-103`,支持企业受控网络内的可信 User/Service 身份、多 Knowledge
|
||||
Base 与检索前 ACL。已经实现:
|
||||
|
||||
- PDF、DOCX、PPTX、XLSX、CSV、HTML、TXT、Markdown 上传、解析、清洗和结构感知 Chunk;
|
||||
XLSX/CSV 保留 Sheet/Header/Cell 与行列范围,显式启用 MinerU 后支持 JPG/JPEG/PNG 与扫描
|
||||
PDF OCR;
|
||||
- SHA-256 Source Hash 去重、同名文档递增版本和索引成功后激活;
|
||||
- 持久化异步 Job、批量提交、有界重试和重试前派生数据清理;
|
||||
- 千帆 `bge-large-zh` Embedding、OceanBase Dense/Sparse、加权 RRF、`bce-reranker-base` TopK;
|
||||
- 来源补全、Index Profile、Shadow Index 重建与切换;
|
||||
- REST API、包含五个 P0 Tool 的 MCP Gateway、知识运营与权限管理控制台;
|
||||
- 可追溯 Compiled Knowledge 草稿、冲突/Stale、审核发布、Published-only 联合检索,以及持久化
|
||||
Compilation Job/Attempt、后台重试和租约恢复;
|
||||
- 生产容器镜像、Kubernetes/Helm、结构化日志和基础指标。
|
||||
|
||||
尚未完成的工作独立跟踪:工程基础任务 `EKP-001~007` 正在逐项正式关单;`EKP-101~102`
|
||||
等待真实 Qwen Endpoint/GPU 环境;远程 MCP/WorkBuddy、目标集群现场验收与 `EKP-104` 发布验收
|
||||
不阻塞开发主队列。完整审计查询管理面和 Append-only 数据库权限仍是已记录的 V1 差距。
|
||||
|
||||
`EKP-025~027` 已提供自托管 MinerU `/file_parse` HTTP Adapter、显式 OCR 路由、页级质量证据
|
||||
和增强清洗。默认仍为轻量 Parser;启用方式与现场限制见
|
||||
[`docs/development/MINERU_SETUP.md`](docs/development/MINERU_SETUP.md)。
|
||||
|
||||
`EKP-026` 已实现本地 Docling Adapter,并以 DOCX、PPTX、XLSX、CSV、HTML 真实格式样例验证
|
||||
上游结构读取。`EKP-028~029` 已建立 schema v4 规范解析模型和 `structure_v2` Chunk,接入
|
||||
PPTX/XLSX/CSV/HTML 上传与 Worker 路由,并在检索来源中返回 Spreadsheet 范围。边界见
|
||||
[`docs/development/DOCLING_SETUP.md`](docs/development/DOCLING_SETUP.md) 和
|
||||
[`docs/development/STRUCTURED_CHUNKING.md`](docs/development/STRUCTURED_CHUNKING.md)。
|
||||
|
||||
`DEPLOY-003` 已按开发侧范围闭环:`dev` 流水线与 GitOps 更新成功,公网 Admin/API/Worker/Search
|
||||
已完成烟测。集群资源、Secret、滚动/回滚、监控和备份由运维人员管控;远程 MCP/WorkBuddy
|
||||
当前搁置。`BUILD-001` 已完成应用镜像构建来源收敛并通过 Drone #30;`EKP-015` 已完成
|
||||
DocumentVersion 模型与行为闭环;`EKP-052~053` 已建立 BM25/Sparse 与 Hybrid 检索;
|
||||
`EKP-025~027` 已完成可选 MinerU OCR 与清洗链路;`EKP-026~029` 已完成 Docling Adapter、
|
||||
规范解析模型、结构化 Spreadsheet Chunk 和 P0 文件路由;`EKP-100` 已建立 40 条合成 Query 的
|
||||
版本化 Golden Dataset 及 Dense/Sparse/Hybrid/Reranked 聚合指标基线;`EKP-042` 已提供未激活的
|
||||
Qwen3 0.6B/4B/8B OpenAI-compatible Embedding Adapter,当前默认模型保持不变。
|
||||
状态与边界参见:
|
||||
|
||||
- [`docs/plans/TASKS.md`](docs/plans/TASKS.md)
|
||||
- [`docs/plans/PROGRESS.md`](docs/plans/PROGRESS.md)
|
||||
- [`docs/development/CONTAINER_IMAGES.md`](docs/development/CONTAINER_IMAGES.md)
|
||||
|
||||
## 架构基线
|
||||
|
||||
```text
|
||||
Admin Console ─ REST ─┐
|
||||
├─ Knowledge Service ─ OceanBase MySQL / BOS / OceanBase Vector
|
||||
WorkBuddy ─ MCP ──────┘ │
|
||||
Celery / RabbitMQ Worker
|
||||
```
|
||||
|
||||
核心原则:
|
||||
|
||||
- API First;
|
||||
- 模块化单体 + 独立异步 Worker + 独立 MCP Gateway;
|
||||
- OceanBase 关系表与 BOS 保存规范数据,向量表只是可重建索引;
|
||||
- Provider 通过接口隔离,业务层不依赖基础设施 SDK;
|
||||
- 原始 Source 不可变,派生知识必须可追溯;
|
||||
- ACL 必须在检索前或检索过程中执行,禁止 Top-K 后过滤。
|
||||
|
||||
完整架构参见 [`docs/architecture/ARCHITECTURE.md`](docs/architecture/ARCHITECTURE.md),已接受决策参见 [`docs/architecture/ADR.md`](docs/architecture/ADR.md)。
|
||||
|
||||
## 本地启动
|
||||
|
||||
前置条件:Python 3.12、`uv`;修改管理端静态资源时还需 Node.js 20.18/npm 10.8;需要本地依赖时
|
||||
再安装 Docker。
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
make install
|
||||
make run-worker
|
||||
make run
|
||||
```
|
||||
|
||||
`make install` 和所有 Python 运行/检查入口使用冻结的 `uv.lock`。有意修改 `pyproject.toml` 依赖时,
|
||||
先执行 `make lock` 并审查锁文件差异;可用 `make toolchain-check` 验证解释器、静态工具目标和冻结
|
||||
安装计划一致。
|
||||
|
||||
管理端保持零构建且无第三方 npm 依赖;修改 `apps/admin-web` 后运行 `make web-check`,检查锁文件、
|
||||
JavaScript 模块语法和本地版本化资源引用。
|
||||
|
||||
MCP Gateway 正式身份从运行时 `MCP_IDENTITY_CREDENTIALS` 读取;仅在本地公共兼容模式下使用长度
|
||||
至少为 32 个字符的 `MCP_AUTH_TOKEN`:
|
||||
|
||||
```bash
|
||||
export MCP_AUTH_TOKEN="<runtime-secret-with-at-least-32-characters>"
|
||||
make run-mcp
|
||||
```
|
||||
|
||||
默认入口:
|
||||
|
||||
| 能力 | 地址 |
|
||||
|---|---|
|
||||
| 管理控制台 | `http://localhost:8000/admin` |
|
||||
| 健康检查 | `http://localhost:8000/healthz` |
|
||||
| 指标 | `http://localhost:8000/metrics` |
|
||||
| OpenAPI | `http://localhost:8000/api/v1/openapi.json` |
|
||||
| MCP | `http://127.0.0.1:8766/mcp` |
|
||||
|
||||
运行时配置、身份边界和部署要求分别参见 `docs/development/` 下的对应手册。Secret 只能通过受控运行环境注入;不得把未获授权的凭据写入 Git、镜像或日志。
|
||||
|
||||
## 开发门禁
|
||||
|
||||
```bash
|
||||
make ci
|
||||
```
|
||||
|
||||
该入口包含锁文件/工具链、管理端、静态检查、Unit/Integration、Migration、离线 RAG Evaluation 和
|
||||
真实进程 API/MCP E2E;范围与安全边界见
|
||||
[`docs/development/CI_BASELINE.md`](docs/development/CI_BASELINE.md)。
|
||||
|
||||
真实 Provider 验收默认禁用。只有满足 [`docs/development/LIVE_ACCEPTANCE.md`](docs/development/LIVE_ACCEPTANCE.md) 的安全确认和清理要求后,才能运行:
|
||||
|
||||
```bash
|
||||
make live-acceptance
|
||||
```
|
||||
|
||||
## 文档入口
|
||||
|
||||
文档职责、权威性、生命周期和完整导航统一记录在 [`docs/README.md`](docs/README.md)。开始开发前还必须阅读 [`AGENTS.md`](AGENTS.md) 和 [`CODEX_START_HERE.md`](CODEX_START_HERE.md)。
|
||||
38
hy4/_kp/alembic.ini
Normal file
38
hy4/_kp/alembic.ini
Normal file
@ -0,0 +1,38 @@
|
||||
[alembic]
|
||||
script_location = %(here)s/migrations
|
||||
prepend_sys_path = .
|
||||
path_separator = os
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
0
hy4/_kp/apps/admin-web/.gitkeep
Normal file
0
hy4/_kp/apps/admin-web/.gitkeep
Normal file
2737
hy4/_kp/apps/admin-web/assets/app.js
Normal file
2737
hy4/_kp/apps/admin-web/assets/app.js
Normal file
File diff suppressed because it is too large
Load Diff
2648
hy4/_kp/apps/admin-web/assets/styles.css
Normal file
2648
hy4/_kp/apps/admin-web/assets/styles.css
Normal file
File diff suppressed because it is too large
Load Diff
200
hy4/_kp/apps/admin-web/assets/ux-enhancements.css
Normal file
200
hy4/_kp/apps/admin-web/assets/ux-enhancements.css
Normal file
@ -0,0 +1,200 @@
|
||||
/* Low-risk readability and state affordances layered over the admin console. */
|
||||
|
||||
.stage-final [class*="candidate"],
|
||||
.stage-final [class*="result"] {
|
||||
grid-template-columns: minmax(0, 1fr) !important;
|
||||
}
|
||||
|
||||
.ux-clear-query {
|
||||
min-height: 42px;
|
||||
padding: 0 18px;
|
||||
border: 1px solid var(--border-color, #cbd5df);
|
||||
border-radius: 10px;
|
||||
background: #fff;
|
||||
color: var(--text-color, #24313d);
|
||||
font: inherit;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.ux-clear-query:hover {
|
||||
border-color: #8395a7;
|
||||
background: #f6f8fa;
|
||||
}
|
||||
|
||||
.ux-readable-dialog label,
|
||||
.ux-readable-form label,
|
||||
.ux-readable-dialog .field-label,
|
||||
.ux-readable-form .field-label,
|
||||
.ux-readable-dialog [class*="label"],
|
||||
.ux-readable-form [class*="label"] {
|
||||
font-size: 15px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.ux-readable-dialog select,
|
||||
.ux-readable-form select,
|
||||
.ux-readable-dialog option,
|
||||
.ux-readable-form option,
|
||||
.ux-readable-dialog input,
|
||||
.ux-readable-form input,
|
||||
.ux-readable-dialog textarea {
|
||||
min-height: 44px;
|
||||
font-size: 15px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.ux-readable-dialog select,
|
||||
.ux-readable-form select,
|
||||
.ux-readable-dialog input,
|
||||
.ux-readable-form input,
|
||||
.ux-readable-dialog textarea,
|
||||
.ux-readable-form textarea {
|
||||
padding: 10px 12px;
|
||||
}
|
||||
|
||||
.ux-management-page table {
|
||||
font-size: 15px;
|
||||
}
|
||||
|
||||
.ux-management-page th {
|
||||
padding-block: 12px;
|
||||
color: #526272;
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.02em;
|
||||
}
|
||||
|
||||
.ux-management-page td {
|
||||
padding-block: 15px;
|
||||
line-height: 1.55;
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.ux-management-page :is(.card, [class*="card"], [class*="item"], [class*="row"]) strong {
|
||||
color: #1d2a36;
|
||||
font-size: 16px;
|
||||
line-height: 1.45;
|
||||
}
|
||||
|
||||
.ux-management-page :is(.card, [class*="card"], [class*="item"], [class*="row"]) :is(small, .muted, [class*="meta"], [class*="description"]) {
|
||||
font-size: 13px;
|
||||
line-height: 1.55;
|
||||
}
|
||||
|
||||
.ux-management-page :is(.card, [class*="card"], [class*="item"]) {
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
@media (max-width: 640px) {
|
||||
.ux-management-page table {
|
||||
font-size: 14px;
|
||||
}
|
||||
}
|
||||
|
||||
#search-page .search-panel label,
|
||||
#search-page .top-k-control label {
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
#search-page .top-k-control {
|
||||
width: 88px;
|
||||
min-width: 88px;
|
||||
}
|
||||
|
||||
#search-top-k {
|
||||
width: 88px;
|
||||
min-height: 38px;
|
||||
padding: 6px 28px 6px 10px;
|
||||
font-size: 14px;
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
.page select,
|
||||
.page select option,
|
||||
dialog select,
|
||||
dialog select option {
|
||||
color: #344454;
|
||||
font-size: 14px;
|
||||
font-weight: 400;
|
||||
letter-spacing: 0;
|
||||
}
|
||||
|
||||
.page select:disabled,
|
||||
dialog select:disabled,
|
||||
.page select option:disabled,
|
||||
dialog select option:disabled {
|
||||
color: #7c8995;
|
||||
}
|
||||
|
||||
.page select:focus,
|
||||
dialog select:focus {
|
||||
border-color: #75889a;
|
||||
box-shadow: 0 0 0 3px rgba(58, 78, 96, 0.1);
|
||||
}
|
||||
|
||||
#search-page .ux-clear-query {
|
||||
width: auto !important;
|
||||
min-width: 0;
|
||||
min-height: 32px;
|
||||
padding: 0 10px;
|
||||
border-radius: 7px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
#search-page .query-heading {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
#search-page .query-heading label {
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.brand-mark {
|
||||
width: 60px;
|
||||
height: 45px;
|
||||
flex: 0 0 auto;
|
||||
}
|
||||
|
||||
.brand {
|
||||
gap: 14px;
|
||||
}
|
||||
|
||||
.brand strong {
|
||||
font-size: 18px;
|
||||
line-height: 1.25;
|
||||
}
|
||||
|
||||
.brand small {
|
||||
font-size: 11px;
|
||||
line-height: 1.35;
|
||||
}
|
||||
|
||||
.workspace-label strong {
|
||||
font-size: 19px;
|
||||
line-height: 1.3;
|
||||
}
|
||||
|
||||
.sidebar-foot {
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
@media (max-width: 540px) {
|
||||
#search-page .top-k-control,
|
||||
#search-top-k {
|
||||
width: 80px;
|
||||
min-width: 80px;
|
||||
}
|
||||
|
||||
.brand-mark {
|
||||
width: 54px;
|
||||
height: 41px;
|
||||
}
|
||||
}
|
||||
670
hy4/_kp/apps/admin-web/index.html
Normal file
670
hy4/_kp/apps/admin-web/index.html
Normal file
@ -0,0 +1,670 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<meta
|
||||
name="description"
|
||||
content="Enterprise Knowledge Platform 管理控制台"
|
||||
/>
|
||||
<title>地瓜知识平台</title>
|
||||
<link rel="stylesheet" href="/admin/assets/styles.css?v=knowledge-tree-14" />
|
||||
<link rel="stylesheet" href="/admin/assets/ux-enhancements.css?v=admin-ux-refresh-8" />
|
||||
<script type="module" src="/admin/assets/app.js?v=knowledge-tree-14"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div class="app-shell">
|
||||
<aside class="sidebar" aria-label="管理控制台导航">
|
||||
<a class="brand" href="/admin/overview" aria-label="地瓜机器人首页">
|
||||
<svg class="brand-mark" viewBox="0 0 48 36" aria-hidden="true">
|
||||
<path d="M15 3h17a15 15 0 0 1 0 30H15v-9h17a6 6 0 0 0 0-12H15z" />
|
||||
<circle cx="11" cy="18" r="8" />
|
||||
</svg>
|
||||
<span>
|
||||
<strong>地瓜机器人</strong>
|
||||
<small>Knowledge Platform</small>
|
||||
</span>
|
||||
</a>
|
||||
<div class="workspace-label">
|
||||
<span>CONTROL DESK</span>
|
||||
<strong>地瓜知识平台</strong>
|
||||
</div>
|
||||
<nav class="primary-nav">
|
||||
<p class="nav-group-label">知识运营</p>
|
||||
<a data-nav="overview" href="/admin/overview">
|
||||
知识总览
|
||||
</a>
|
||||
<a data-nav="upload" data-required-scope="knowledge:write" href="/admin/upload">
|
||||
文档导入
|
||||
</a>
|
||||
<a data-nav="documents" href="/admin/documents">
|
||||
文档管理
|
||||
</a>
|
||||
<a data-nav="compiled-knowledge" href="/admin/compiled-knowledge">
|
||||
编纂知识
|
||||
</a>
|
||||
<a data-nav="search" href="/admin/search">
|
||||
知识检索
|
||||
</a>
|
||||
<p class="nav-group-label">系统管理</p>
|
||||
<a data-nav="knowledge-bases" data-required-scope="knowledge:manage" href="/admin/knowledge-bases">
|
||||
知识库管理
|
||||
</a>
|
||||
<a data-nav="permissions" data-required-scope="knowledge:manage" href="/admin/permissions">
|
||||
权限管理
|
||||
</a>
|
||||
</nav>
|
||||
<div class="sidebar-foot">
|
||||
<span class="environment-dot" aria-hidden="true"></span>
|
||||
地瓜知识平台 · V1
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<main class="workspace">
|
||||
<header class="topbar">
|
||||
<div>
|
||||
<p class="eyebrow">KNOWLEDGE OPERATIONS</p>
|
||||
<h1 id="page-title">知识总览</h1>
|
||||
</div>
|
||||
<div class="topbar-actions">
|
||||
<button id="identity-button" class="identity-button" type="button">
|
||||
<span id="identity-name">连接访问身份</span>
|
||||
<small id="identity-organization">访问令牌</small>
|
||||
</button>
|
||||
<span class="system-state"><i aria-hidden="true"></i>API FIRST · V1</span>
|
||||
<a class="api-link" href="/api/v1/docs" target="_blank" rel="noreferrer">
|
||||
API 文档 <span aria-hidden="true">↗</span>
|
||||
</a>
|
||||
</div>
|
||||
</header>
|
||||
<div class="signal-line" aria-hidden="true"><span></span><span></span><span></span></div>
|
||||
|
||||
<div id="notice" class="notice" role="status" aria-live="polite" hidden></div>
|
||||
|
||||
<section id="overview-page" class="page" data-page="overview" aria-labelledby="page-title">
|
||||
<div class="page-heading overview-heading">
|
||||
<div>
|
||||
<p class="section-label">OVERVIEW / LIVE INVENTORY</p>
|
||||
<h2>查看知识处理与检索状态</h2>
|
||||
<p>快速了解文档总量、处理进度、可检索比例和需要关注的异常。</p>
|
||||
</div>
|
||||
<button id="refresh-overview" class="secondary-button" type="button">刷新总览</button>
|
||||
</div>
|
||||
|
||||
<section class="knowledge-flow" aria-labelledby="knowledge-flow-title">
|
||||
<div class="flow-intro">
|
||||
<p class="section-label">KNOWLEDGE FLOW</p>
|
||||
<h3 id="knowledge-flow-title">文档处理概览</h3>
|
||||
<span id="overview-updated-at">尚未加载</span>
|
||||
</div>
|
||||
<div class="flow-rail" aria-live="polite">
|
||||
<article class="flow-node flow-source">
|
||||
<span class="flow-code">SOURCE</span>
|
||||
<strong id="overview-total">—</strong>
|
||||
<small>全部文档</small>
|
||||
</article>
|
||||
<span class="flow-connector" aria-hidden="true"></span>
|
||||
<article class="flow-node flow-processing">
|
||||
<span class="flow-code">PROCESS</span>
|
||||
<strong id="overview-processing">—</strong>
|
||||
<small>等待或处理中</small>
|
||||
</article>
|
||||
<span class="flow-connector" aria-hidden="true"></span>
|
||||
<article class="flow-node flow-ready">
|
||||
<span class="flow-code">READY</span>
|
||||
<strong id="overview-ready">—</strong>
|
||||
<small>可检索</small>
|
||||
</article>
|
||||
<span class="flow-connector" aria-hidden="true"></span>
|
||||
<article class="flow-node flow-retrieval">
|
||||
<span class="flow-code">RETRIEVAL</span>
|
||||
<strong id="overview-readiness">—</strong>
|
||||
<small>入库就绪率</small>
|
||||
</article>
|
||||
</div>
|
||||
<div id="overview-alert" class="flow-alert" hidden></div>
|
||||
</section>
|
||||
|
||||
<div class="overview-grid">
|
||||
<section class="overview-ledger" aria-labelledby="recent-documents-title">
|
||||
<div class="overview-section-heading">
|
||||
<div>
|
||||
<p class="section-label">RECENT SOURCES</p>
|
||||
<h3 id="recent-documents-title">最近文档</h3>
|
||||
</div>
|
||||
<a class="text-link" href="/admin/documents">查看全部 <span>→</span></a>
|
||||
</div>
|
||||
<div id="overview-recent-documents" class="overview-document-list" aria-live="polite">
|
||||
<div class="content-state">正在读取文档库存…</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<aside class="overview-actions" aria-labelledby="next-actions-title">
|
||||
<div class="overview-section-heading">
|
||||
<div>
|
||||
<p class="section-label">NEXT ACTION</p>
|
||||
<h3 id="next-actions-title">继续操作</h3>
|
||||
</div>
|
||||
</div>
|
||||
<a class="operation-link" href="/admin/upload">
|
||||
<span>导入文档</span>
|
||||
<strong>上传并处理知识文件</strong>
|
||||
<i aria-hidden="true">↗</i>
|
||||
</a>
|
||||
<a class="operation-link" href="/admin/search">
|
||||
<span>测试检索</span>
|
||||
<strong>检索并核对来源</strong>
|
||||
<i aria-hidden="true">↗</i>
|
||||
</a>
|
||||
<div class="scope-note">
|
||||
<span class="scope-indicator" aria-hidden="true"></span>
|
||||
<p><strong>当前可访问范围</strong>这里只显示当前身份有权查看和检索的知识。</p>
|
||||
</div>
|
||||
</aside>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="upload-page" class="page" data-page="upload" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading">
|
||||
<div>
|
||||
<p class="section-label">DOCUMENT INGESTION</p>
|
||||
<h2>上传并处理知识文件</h2>
|
||||
<p>选择所属知识库和文件夹后上传文件,系统会逐个完成解析并建立检索索引。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form id="upload-form" class="upload-panel">
|
||||
<div class="scope-selectors">
|
||||
<label for="upload-knowledge-base">目标知识库
|
||||
<select id="upload-knowledge-base" name="knowledge_base_id" required>
|
||||
<option value="">请先连接访问身份并选择知识库</option>
|
||||
</select>
|
||||
</label>
|
||||
<label for="upload-folder">目标文件夹(可选)
|
||||
<select id="upload-folder" name="folder_id" disabled>
|
||||
<option value="">知识库根目录</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
<div id="drop-zone" class="drop-zone">
|
||||
<span class="upload-icon" aria-hidden="true">↑</span>
|
||||
<strong>选择文件、整个文件夹或 ZIP 压缩包</strong>
|
||||
<span>支持 PDF、DOCX、PPTX、XLSX、CSV、HTML、TXT 和 Markdown;图片与扫描文档识别能力取决于当前解析配置</span>
|
||||
<div class="upload-source-actions" aria-label="选择上传来源">
|
||||
<label class="upload-source-button" for="file-input">
|
||||
选择文件
|
||||
<input
|
||||
id="file-input"
|
||||
name="files"
|
||||
type="file"
|
||||
multiple
|
||||
accept=".pdf,.docx,.pptx,.xlsx,.csv,.html,.htm,.txt,.md,.jpg,.jpeg,.png,application/pdf,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.presentationml.presentation,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,text/csv,text/html,text/plain,text/markdown,image/jpeg,image/png"
|
||||
/>
|
||||
</label>
|
||||
<label class="upload-source-button directory" for="folder-input">
|
||||
选择文件夹
|
||||
<input
|
||||
id="folder-input"
|
||||
name="directory_files"
|
||||
type="file"
|
||||
multiple
|
||||
webkitdirectory
|
||||
directory
|
||||
accept=".pdf,.docx,.pptx,.xlsx,.csv,.html,.htm,.txt,.md,.jpg,.jpeg,.png,application/pdf,application/vnd.openxmlformats-officedocument.wordprocessingml.document,application/vnd.openxmlformats-officedocument.presentationml.presentation,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,text/csv,text/html,text/plain,text/markdown,image/jpeg,image/png"
|
||||
/>
|
||||
</label>
|
||||
<label class="upload-source-button archive" for="archive-input">
|
||||
选择 ZIP 压缩包
|
||||
<input
|
||||
id="archive-input"
|
||||
name="archive"
|
||||
type="file"
|
||||
accept=".zip,application/zip,application/x-zip-compressed"
|
||||
/>
|
||||
</label>
|
||||
</div>
|
||||
<span class="upload-limit-hint">文件夹和 ZIP 会保留多层目录;普通批次最多 20 个文件,ZIP 最多 1000 个文件</span>
|
||||
<span id="selected-file" class="selected-file">尚未选择文件、文件夹或 ZIP</span>
|
||||
</div>
|
||||
<button id="upload-button" class="primary-button" type="submit">上传并建立目录</button>
|
||||
</form>
|
||||
|
||||
<article id="upload-result" class="result-card" hidden>
|
||||
<div>
|
||||
<p class="section-label">LATEST BATCH</p>
|
||||
<h3 id="upload-batch-title"></h3>
|
||||
</div>
|
||||
<span id="upload-status" class="status-pill"></span>
|
||||
<dl class="metadata-grid">
|
||||
<div><dt>总文件</dt><dd id="upload-total"></dd></div>
|
||||
<div><dt>已完成</dt><dd id="upload-completed"></dd></div>
|
||||
<div><dt>接收 / 拒绝</dt><dd id="upload-accepted"></dd></div>
|
||||
</dl>
|
||||
<div id="batch-items" class="batch-items" aria-live="polite"></div>
|
||||
<a class="text-link" href="/admin/search">前往知识检索 <span>→</span></a>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section id="documents-page" class="page" data-page="documents" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading heading-row">
|
||||
<div>
|
||||
<p class="section-label">DOCUMENT MANAGEMENT</p>
|
||||
<h2>查看文档与处理结果</h2>
|
||||
<p>查看文档状态、原始文件、解析内容和处理记录,并核对检索使用的来源。</p>
|
||||
</div>
|
||||
<button id="refresh-documents" class="secondary-button" type="button">刷新列表</button>
|
||||
</div>
|
||||
|
||||
<section class="document-scope-filter" aria-labelledby="document-filter-title">
|
||||
<div class="document-filter-heading">
|
||||
<div>
|
||||
<p class="section-label">DOCUMENT SCOPE</p>
|
||||
<h3 id="document-filter-title">缩小文档范围</h3>
|
||||
</div>
|
||||
<button id="reset-document-filters" class="table-action" type="button">
|
||||
查看全部
|
||||
</button>
|
||||
</div>
|
||||
<div class="document-filter-fields">
|
||||
<label for="document-filter-knowledge-base">
|
||||
<span><b>01</b> 知识库</span>
|
||||
<select id="document-filter-knowledge-base">
|
||||
<option value="">全部可访问知识库</option>
|
||||
</select>
|
||||
</label>
|
||||
<label for="document-filter-folder">
|
||||
<span><b>02</b> 文件夹范围</span>
|
||||
<select id="document-filter-folder" disabled>
|
||||
<option value="all">先选择知识库</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
<p id="document-filter-summary" class="document-filter-summary" aria-live="polite">
|
||||
当前显示全部可访问文档
|
||||
</p>
|
||||
</section>
|
||||
|
||||
<div class="table-card">
|
||||
<div class="table-meta">
|
||||
<span id="document-count">0 个文档</span>
|
||||
<span id="document-refresh-time">尚未加载</span>
|
||||
</div>
|
||||
<div class="table-scroll">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>文件名</th>
|
||||
<th>类型</th>
|
||||
<th>上传时间</th>
|
||||
<th>状态</th>
|
||||
<th aria-label="操作"></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="document-rows"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
<div id="document-empty" class="empty-state" hidden>
|
||||
<strong id="document-empty-title">还没有文档</strong>
|
||||
<span id="document-empty-detail">上传第一个文件后,它会出现在这里。</span>
|
||||
<button id="document-empty-reset" class="secondary-button compact" type="button" hidden>
|
||||
清除筛选
|
||||
</button>
|
||||
<a id="document-empty-upload" class="primary-button compact" href="/admin/upload">
|
||||
导入文档
|
||||
</a>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="search-page" class="page" data-page="search" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading">
|
||||
<div>
|
||||
<p class="section-label">KNOWLEDGE RETRIEVAL</p>
|
||||
<h2>检索知识并核对来源</h2>
|
||||
<p>输入问题或关键词,查看匹配的知识来源及检索过程;这里不会生成回答。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form id="search-form" class="search-panel">
|
||||
<div class="query-heading">
|
||||
<label for="search-query">问题或关键词</label>
|
||||
<button id="clear-search-query" class="ux-clear-query" type="button">清空查询</button>
|
||||
</div>
|
||||
<div class="search-controls">
|
||||
<textarea
|
||||
id="search-query"
|
||||
name="query"
|
||||
rows="3"
|
||||
maxlength="1600"
|
||||
placeholder="例如:地瓜干的作用是什么?"
|
||||
required
|
||||
></textarea>
|
||||
<label class="top-k-control" for="search-top-k">
|
||||
结果数量
|
||||
<select id="search-top-k" name="top_k">
|
||||
<option value="1">1</option>
|
||||
<option value="3">3</option>
|
||||
<option value="5" selected>5</option>
|
||||
</select>
|
||||
</label>
|
||||
<button id="search-button" class="primary-button" type="submit">开始检索</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<section id="search-diagnostics" class="diagnostic-board" aria-labelledby="diagnostic-title" hidden>
|
||||
<div class="diagnostic-heading">
|
||||
<div>
|
||||
<p class="section-label">RETRIEVAL TRACE</p>
|
||||
<h3 id="diagnostic-title">检索过程</h3>
|
||||
</div>
|
||||
<span id="diagnostic-summary"></span>
|
||||
</div>
|
||||
<dl id="diagnostic-context" class="diagnostic-context"></dl>
|
||||
<div id="diagnostic-stages" class="diagnostic-stages"></div>
|
||||
</section>
|
||||
|
||||
<div class="results-heading">
|
||||
<div>
|
||||
<p class="section-label">RESULTS</p>
|
||||
<h3>来源结果</h3>
|
||||
</div>
|
||||
<span id="search-summary">等待检索</span>
|
||||
</div>
|
||||
<div id="search-results" class="search-results" aria-live="polite">
|
||||
<div class="empty-state search-empty">
|
||||
<strong>检索结果会显示在这里</strong>
|
||||
<span>结果包含可读上下文、命中片段、来源位置和原文入口。</span>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="compiled-knowledge-page" class="page" data-page="compiled-knowledge" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading heading-row">
|
||||
<div>
|
||||
<p class="section-label">COMPILED KNOWLEDGE / REVIEW</p>
|
||||
<h2>让每条知识带着来源走完审核</h2>
|
||||
<p>从真实片段形成草稿,核验版本和权限后提交审核;冲突必须留下双来源证据并解决后才能发布。</p>
|
||||
</div>
|
||||
<button id="refresh-compiled-knowledge" class="secondary-button" type="button">刷新草稿</button>
|
||||
</div>
|
||||
|
||||
<div class="compilation-layout">
|
||||
<form id="compiled-draft-form" class="management-list compilation-editor" data-required-scope="knowledge:write">
|
||||
<div class="management-heading">
|
||||
<div><p class="section-label">DRAFT EDITOR</p><h3 id="compiled-editor-title">新建草稿</h3></div>
|
||||
<button id="cancel-compiled-edit" class="table-action" type="button" hidden>取消编辑</button>
|
||||
</div>
|
||||
<label>知识库<select id="compiled-knowledge-base" required></select></label>
|
||||
<label>知识类型
|
||||
<select id="compiled-knowledge-type" required>
|
||||
<option value="summary">Summary</option>
|
||||
<option value="entity">Entity</option>
|
||||
<option value="concept">Concept</option>
|
||||
<option value="faq">FAQ</option>
|
||||
<option value="sop">SOP</option>
|
||||
<option value="comparison">Comparison</option>
|
||||
<option value="overview">Overview</option>
|
||||
<option value="custom_wiki_page">Custom Wiki Page</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>标题<input id="compiled-title" maxlength="255" required /></label>
|
||||
<label>草稿正文<textarea id="compiled-content" rows="10" maxlength="100000" required></textarea></label>
|
||||
<fieldset id="compiled-source-fieldset" class="compilation-sources">
|
||||
<legend>可追溯来源</legend>
|
||||
<label>来源文档<select id="compiled-source-document" required></select></label>
|
||||
<div id="compiled-source-chunks" class="source-choice-list"><div class="content-state">请选择知识库和来源文档。</div></div>
|
||||
</fieldset>
|
||||
<p id="compiled-edit-source-note" class="field-note" hidden>普通编辑保留原来源;修订失效知识时必须重新选择当前有效片段。</p>
|
||||
<div class="management-actions compilation-actions">
|
||||
<button id="save-compiled-draft" class="primary-button compact" type="submit">保存手工草稿</button>
|
||||
<button id="generate-compiled-draft" class="secondary-button" type="button">AI 生成草稿</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<section class="management-list" aria-labelledby="compiled-list-title">
|
||||
<div class="management-heading">
|
||||
<div><p class="section-label">KNOWLEDGE LIFECYCLE</p><h3 id="compiled-list-title">可见知识与审核状态</h3></div>
|
||||
<span id="compiled-count">0 条</span>
|
||||
</div>
|
||||
<div id="compiled-list" class="resource-list"><div class="content-state">连接访问身份后加载草稿…</div></div>
|
||||
</section>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section id="knowledge-bases-page" class="page" data-page="knowledge-bases" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading heading-row">
|
||||
<div>
|
||||
<p class="section-label">KNOWLEDGE BASE MANAGEMENT</p>
|
||||
<h2>管理知识库与文件夹</h2>
|
||||
<p>展开知识库和文件夹层级,快速看清内容组织;需要新增内容时,从右侧按钮进入独立操作窗口。</p>
|
||||
</div>
|
||||
<button id="open-knowledge-base-dialog" class="primary-button compact" type="button">新建知识库</button>
|
||||
</div>
|
||||
<section class="management-list" aria-labelledby="knowledge-base-list-title">
|
||||
<div class="management-heading"><div><p class="section-label">KNOWLEDGE SPACES</p><h3 id="knowledge-base-list-title">知识库与文件夹</h3></div><button id="refresh-knowledge-bases" class="secondary-button" type="button">刷新</button></div>
|
||||
<div id="knowledge-base-list" class="resource-list"><div class="content-state">连接访问身份后加载知识库…</div></div>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section id="permissions-page" class="page" data-page="permissions" aria-labelledby="page-title" hidden>
|
||||
<div class="page-heading">
|
||||
<p class="section-label">IDENTITY & ACCESS</p>
|
||||
<h2>管理成员与知识库权限</h2>
|
||||
<p>集中查看用户、用户组、角色和授权关系;新增与分配操作通过独立窗口完成。</p>
|
||||
</div>
|
||||
<div class="management-actions" aria-label="权限管理新增操作">
|
||||
<button id="open-user-dialog" class="primary-button compact" type="button">新建用户</button>
|
||||
<button id="open-group-dialog" class="secondary-button" type="button">新建用户组</button>
|
||||
<button id="open-role-dialog" class="secondary-button" type="button">新建角色</button>
|
||||
<button id="open-group-membership-dialog" class="secondary-button" type="button">添加用户组成员</button>
|
||||
<button id="open-user-role-dialog" class="secondary-button" type="button">分配用户角色</button>
|
||||
<button id="open-group-role-dialog" class="secondary-button" type="button">分配用户组角色</button>
|
||||
<button id="open-acl-dialog" class="secondary-button" type="button">新增授权规则</button>
|
||||
</div>
|
||||
<section class="management-list">
|
||||
<div class="management-heading"><div><p class="section-label">DIRECTORY</p><h3>用户与权限关系</h3></div><button id="refresh-permissions" class="secondary-button" type="button">刷新</button></div>
|
||||
<h4 class="directory-heading">用户</h4>
|
||||
<div id="user-list" class="resource-list"><div class="content-state">连接访问身份后加载用户…</div></div>
|
||||
<h4 class="directory-heading">用户组</h4>
|
||||
<div id="group-list" class="resource-list"><div class="content-state">连接访问身份后加载用户组…</div></div>
|
||||
<h4 class="directory-heading">角色</h4>
|
||||
<div id="role-list" class="resource-list"><div class="content-state">连接访问身份后加载角色…</div></div>
|
||||
<div class="acl-view-heading">
|
||||
<h4 class="directory-heading">知识库授权</h4>
|
||||
<label>查看知识库<select id="acl-view-knowledge-base"></select></label>
|
||||
</div>
|
||||
<div id="acl-list" class="resource-list compact-list"></div>
|
||||
</section>
|
||||
</section>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<dialog id="compiled-action-dialog" class="management-dialog wide-management-dialog" aria-labelledby="compiled-action-title">
|
||||
<form id="compiled-action-form">
|
||||
<div class="management-dialog-heading">
|
||||
<div><p class="section-label">SOURCE REVIEW</p><h2 id="compiled-action-title">记录来源冲突</h2></div>
|
||||
<button class="icon-button dialog-close" type="button" aria-label="关闭">×</button>
|
||||
</div>
|
||||
<p id="compiled-action-context" class="dialog-context"></p>
|
||||
<input id="compiled-action-mode" type="hidden" />
|
||||
<input id="compiled-action-id" type="hidden" />
|
||||
<input id="compiled-action-revision" type="hidden" />
|
||||
<input id="compiled-action-conflict-id" type="hidden" />
|
||||
<div id="compiled-conflict-fields">
|
||||
<label>冲突主张<textarea id="compiled-conflict-claim" rows="3" maxlength="2000"></textarea></label>
|
||||
<div class="conflict-source-selectors">
|
||||
<label>来源 A<select id="compiled-conflict-left"></select></label>
|
||||
<label>来源 B<select id="compiled-conflict-right"></select></label>
|
||||
</div>
|
||||
</div>
|
||||
<label id="compiled-action-note-field" hidden><span id="compiled-action-note-label">处理说明</span><textarea id="compiled-action-note" rows="4" maxlength="2000"></textarea></label>
|
||||
<div class="dialog-actions">
|
||||
<button class="secondary-button dialog-cancel" type="button">取消</button>
|
||||
<button id="compiled-action-submit" class="primary-button" type="submit">记录冲突</button>
|
||||
</div>
|
||||
</form>
|
||||
</dialog>
|
||||
|
||||
<dialog id="knowledge-base-dialog" class="management-dialog" aria-labelledby="knowledge-base-dialog-title">
|
||||
<form id="knowledge-base-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">NEW KNOWLEDGE BASE</p><h2 id="knowledge-base-dialog-title">新建知识库</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>名称<input id="knowledge-base-name" maxlength="255" required /></label>
|
||||
<label>描述<textarea id="knowledge-base-description" rows="3"></textarea></label>
|
||||
<label>可见性<select id="knowledge-base-visibility"><option value="private">私有</option><option value="organization">组织内可见</option></select></label>
|
||||
<label class="check-field"><input id="knowledge-base-mcp" type="checkbox" />允许 MCP 使用</label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建知识库</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="folder-dialog" class="management-dialog" aria-labelledby="folder-dialog-title">
|
||||
<form id="folder-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">NEW FOLDER</p><h2 id="folder-dialog-title">新建文件夹</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<p id="folder-knowledge-base-context" class="dialog-context"></p>
|
||||
<input id="folder-knowledge-base-id" type="hidden" />
|
||||
<label>文件夹名称<input id="folder-name" maxlength="255" required /></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建文件夹</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="user-dialog" class="management-dialog" aria-labelledby="user-dialog-title">
|
||||
<form id="user-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">NEW USER</p><h2 id="user-dialog-title">新建用户</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>显示名称<input id="user-display-name" maxlength="255" required /></label>
|
||||
<label>身份签发方(Issuer)<input id="user-issuer" maxlength="255" value="local" required /></label>
|
||||
<label>身份唯一标识(Subject)<input id="user-subject" maxlength="255" required /></label>
|
||||
<label>邮箱(可选)<input id="user-email" type="email" maxlength="255" /></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建用户</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="group-dialog" class="management-dialog" aria-labelledby="group-dialog-title">
|
||||
<form id="group-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">NEW GROUP</p><h2 id="group-dialog-title">新建用户组</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>用户组名称<input id="group-name" maxlength="255" required /></label>
|
||||
<label>描述(可选)<textarea id="group-description" rows="2"></textarea></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建用户组</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="role-dialog" class="management-dialog" aria-labelledby="role-dialog-title">
|
||||
<form id="role-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">NEW ROLE</p><h2 id="role-dialog-title">新建角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>角色名称<input id="role-name" maxlength="255" required /></label>
|
||||
<label>描述(可选)<textarea id="role-description" rows="2"></textarea></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">创建角色</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="group-membership-dialog" class="management-dialog" aria-labelledby="group-membership-dialog-title">
|
||||
<form id="group-membership-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">GROUP MEMBERSHIP</p><h2 id="group-membership-dialog-title">添加用户组成员</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>用户<select id="membership-user" required></select></label>
|
||||
<label>用户组<select id="membership-group" required></select></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">添加成员</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="user-role-dialog" class="management-dialog" aria-labelledby="user-role-dialog-title">
|
||||
<form id="user-role-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">USER ROLE</p><h2 id="user-role-dialog-title">分配用户角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>用户<select id="user-role-user" required></select></label>
|
||||
<label>角色<select id="user-role-role" required></select></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">分配角色</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="group-role-dialog" class="management-dialog" aria-labelledby="group-role-dialog-title">
|
||||
<form id="group-role-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">GROUP ROLE</p><h2 id="group-role-dialog-title">分配用户组角色</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>用户组<select id="group-role-group" required></select></label>
|
||||
<label>角色<select id="group-role-role" required></select></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">分配角色</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
<dialog id="acl-dialog" class="management-dialog wide-management-dialog" aria-labelledby="acl-dialog-title">
|
||||
<form id="acl-form">
|
||||
<div class="management-dialog-heading"><div><p class="section-label">ACCESS RULE</p><h2 id="acl-dialog-title">新增授权规则</h2></div><button class="icon-button dialog-close" type="button" aria-label="关闭">×</button></div>
|
||||
<label>授权对象类型<select id="acl-subject-type"><option value="user">用户</option><option value="group">用户组</option><option value="role">角色</option></select></label>
|
||||
<label>授权对象<select id="acl-subject" required></select></label>
|
||||
<label>知识库<select id="acl-knowledge-base" required></select></label>
|
||||
<label>权限<select id="acl-permission"><option value="read">读取</option><option value="write">写入</option><option value="manage">管理</option></select></label>
|
||||
<label>规则<select id="acl-effect"><option value="allow">允许</option><option value="deny">拒绝</option></select></label>
|
||||
<div class="dialog-actions"><button class="secondary-button dialog-cancel" type="button">取消</button><button class="primary-button" type="submit">保存授权规则</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
|
||||
<dialog id="document-dialog" class="document-dialog" aria-labelledby="dialog-filename">
|
||||
<form method="dialog"><button class="icon-button" aria-label="关闭文档">×</button></form>
|
||||
<div class="dialog-heading">
|
||||
<div>
|
||||
<p class="section-label">DOCUMENT INSPECTOR</p>
|
||||
<h2 id="dialog-filename"></h2>
|
||||
</div>
|
||||
<button id="dialog-source-link" class="secondary-button source-link" type="button">
|
||||
新窗口打开原始文件 <span aria-hidden="true">↗</span>
|
||||
</button>
|
||||
</div>
|
||||
<details class="inspector-metadata">
|
||||
<summary>
|
||||
<span>文档信息</span>
|
||||
<small id="dialog-metadata-summary"></small>
|
||||
</summary>
|
||||
<dl id="dialog-metadata" class="dialog-metadata"></dl>
|
||||
</details>
|
||||
<details id="processing-inspector" class="processing-inspector">
|
||||
<summary>
|
||||
<span>处理流水线</span>
|
||||
<small id="processing-summary"></small>
|
||||
</summary>
|
||||
<div id="processing-content" class="processing-content"></div>
|
||||
</details>
|
||||
<div class="inspector-view-switcher" role="tablist" aria-label="文档查看方式">
|
||||
<button
|
||||
id="source-preview-tab"
|
||||
class="inspector-tab active"
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-controls="source-preview-panel"
|
||||
aria-selected="true"
|
||||
>原始文件预览</button>
|
||||
<button
|
||||
id="parsed-content-tab"
|
||||
class="inspector-tab"
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-controls="parsed-content-panel"
|
||||
aria-selected="false"
|
||||
>解析与溯源</button>
|
||||
<span id="inspector-view-status">正在读取…</span>
|
||||
</div>
|
||||
<div id="document-quality" class="quality-banner" role="status" hidden></div>
|
||||
<section
|
||||
id="source-preview-panel"
|
||||
class="inspector-panel source-preview-panel"
|
||||
role="tabpanel"
|
||||
aria-labelledby="source-preview-tab"
|
||||
>
|
||||
<div id="source-preview" class="source-preview" aria-live="polite"></div>
|
||||
</section>
|
||||
<section
|
||||
id="parsed-content-panel"
|
||||
class="inspector-panel document-content"
|
||||
role="tabpanel"
|
||||
aria-labelledby="parsed-content-tab"
|
||||
hidden
|
||||
>
|
||||
<div class="document-content-heading">
|
||||
<div>
|
||||
<p class="section-label">SOURCE TRACK</p>
|
||||
<h3 id="document-content-title">解析内容(按 Chunk 顺序)</h3>
|
||||
</div>
|
||||
<span id="dialog-content-summary">正在读取…</span>
|
||||
</div>
|
||||
<div id="dialog-content" class="source-track" aria-live="polite"></div>
|
||||
</section>
|
||||
</dialog>
|
||||
<dialog id="identity-dialog" class="identity-dialog" aria-labelledby="identity-dialog-title">
|
||||
<form id="identity-form">
|
||||
<p class="section-label">ACCESS IDENTITY</p>
|
||||
<h2 id="identity-dialog-title">使用访问令牌进入</h2>
|
||||
<p>访问令牌仅保存在当前浏览器标签页中,关闭标签页后会自动清除。</p>
|
||||
<label for="identity-token">访问令牌</label>
|
||||
<input id="identity-token" type="password" minlength="32" autocomplete="off" required />
|
||||
<div class="dialog-actions"><button id="identity-cancel" class="secondary-button" type="button">取消</button><button class="primary-button" type="submit">验证并进入</button></div>
|
||||
</form>
|
||||
</dialog>
|
||||
</body>
|
||||
</html>
|
||||
15
hy4/_kp/apps/admin-web/package-lock.json
generated
Normal file
15
hy4/_kp/apps/admin-web/package-lock.json
generated
Normal file
@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "@ekp/admin-web",
|
||||
"version": "0.1.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@ekp/admin-web",
|
||||
"version": "0.1.0",
|
||||
"engines": {
|
||||
"node": ">=20.18.0 <21"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
16
hy4/_kp/apps/admin-web/package.json
Normal file
16
hy4/_kp/apps/admin-web/package.json
Normal file
@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "@ekp/admin-web",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Zero-build Enterprise Knowledge Platform administration console",
|
||||
"type": "module",
|
||||
"packageManager": "npm@10.8.2",
|
||||
"engines": {
|
||||
"node": ">=20.18.0 <21"
|
||||
},
|
||||
"scripts": {
|
||||
"check": "npm run check:javascript && npm run check:assets",
|
||||
"check:javascript": "node --check assets/app.js",
|
||||
"check:assets": "node scripts/check-assets.mjs"
|
||||
}
|
||||
}
|
||||
33
hy4/_kp/apps/admin-web/scripts/check-assets.mjs
Normal file
33
hy4/_kp/apps/admin-web/scripts/check-assets.mjs
Normal file
@ -0,0 +1,33 @@
|
||||
/** Validate that the zero-build console references only present, local runtime assets. */
|
||||
import { accessSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
const webRoot = process.cwd();
|
||||
const html = readFileSync(join(webRoot, "index.html"), "utf8");
|
||||
const runtimeReferences = [
|
||||
...html.matchAll(/<(?:img|link|script)\b[^>]*?(?:href|src)="([^"]+)"/gu),
|
||||
].map((match) => match[1]);
|
||||
const remoteReferences = runtimeReferences.filter((reference) =>
|
||||
/^(?:https?:)?\/\//u.test(reference),
|
||||
);
|
||||
|
||||
if (remoteReferences.length > 0) {
|
||||
throw new Error(`Admin Web runtime assets must be local: ${remoteReferences.join(", ")}`);
|
||||
}
|
||||
|
||||
const adminAssets = runtimeReferences.filter((reference) =>
|
||||
reference.startsWith("/admin/assets/"),
|
||||
);
|
||||
if (adminAssets.length === 0) {
|
||||
throw new Error("Admin Web must reference at least one versioned runtime asset");
|
||||
}
|
||||
|
||||
for (const reference of adminAssets) {
|
||||
const [pathname, cacheVersion] = reference.split("?", 2);
|
||||
if (!cacheVersion) {
|
||||
throw new Error(`Admin Web asset is missing a cache version: ${reference}`);
|
||||
}
|
||||
accessSync(join(webRoot, pathname.replace("/admin/", "")));
|
||||
}
|
||||
|
||||
console.log(`Admin Web asset contract ok: ${adminAssets.length} local versioned assets`);
|
||||
0
hy4/_kp/apps/api/.gitkeep
Normal file
0
hy4/_kp/apps/api/.gitkeep
Normal file
1
hy4/_kp/apps/api/src/ekp_api/__init__.py
Normal file
1
hy4/_kp/apps/api/src/ekp_api/__init__.py
Normal file
@ -0,0 +1 @@
|
||||
"""Enterprise Knowledge Platform API application."""
|
||||
411
hy4/_kp/apps/api/src/ekp_api/admin_routes.py
Normal file
411
hy4/_kp/apps/api/src/ekp_api/admin_routes.py
Normal file
@ -0,0 +1,411 @@
|
||||
"""Authenticated REST administration for canonical V1 identities and resources."""
|
||||
|
||||
from typing import Annotated
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response, status
|
||||
|
||||
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
|
||||
from ekp_api.providers import ApiServices
|
||||
from ekp_api.routes import get_api_services
|
||||
from ekp_api.schemas import (
|
||||
AccessControlEntryCreateRequest,
|
||||
AccessControlEntryResponse,
|
||||
AdministrationSessionResponse,
|
||||
FolderCreateRequest,
|
||||
FolderResponse,
|
||||
GroupResponse,
|
||||
IdentityCollectionCreateRequest,
|
||||
KnowledgeBaseCreateRequest,
|
||||
KnowledgeBaseResponse,
|
||||
RoleResponse,
|
||||
UserCreateRequest,
|
||||
UserResponse,
|
||||
)
|
||||
from ekp_core.administration import (
|
||||
AdministrationConflictError,
|
||||
AdministrationIdentityError,
|
||||
AdministrationNotFoundError,
|
||||
AdministrationService,
|
||||
)
|
||||
from ekp_domain import AclResourceType
|
||||
|
||||
router = APIRouter(prefix="/administration", tags=["administration"])
|
||||
Services = Annotated[ApiServices, Depends(get_api_services)]
|
||||
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
|
||||
|
||||
|
||||
@router.get("/session", response_model=AdministrationSessionResponse)
|
||||
def get_administration_session(
|
||||
services: Services, identity: Identity
|
||||
) -> AdministrationSessionResponse:
|
||||
"""Validate the secret-to-canonical-User mapping used by the console."""
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.administration is None:
|
||||
raise HTTPException(status_code=503, detail="administration service is unavailable")
|
||||
try:
|
||||
value = services.administration.get_session(organization_id, user_id)
|
||||
except AdministrationIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
return AdministrationSessionResponse(
|
||||
organization_id=value.organization.id,
|
||||
organization_name=value.organization.name,
|
||||
user_id=value.user.id,
|
||||
display_name=value.user.display_name,
|
||||
scopes=tuple(sorted(identity.scopes)),
|
||||
credential_type=identity.credential_type,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/users", response_model=list[UserResponse])
|
||||
def list_users(services: Services, identity: Identity) -> list[UserResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
return [
|
||||
UserResponse.from_user(value)
|
||||
for value in administration.list_users(organization_id, user_id)
|
||||
]
|
||||
|
||||
|
||||
@router.post("/users", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create_user(payload: UserCreateRequest, services: Services, identity: Identity) -> UserResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_user(
|
||||
organization_id,
|
||||
user_id,
|
||||
issuer=payload.issuer,
|
||||
subject=payload.subject,
|
||||
display_name=payload.display_name,
|
||||
email=payload.email,
|
||||
)
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="user already exists") from error
|
||||
return UserResponse.from_user(value)
|
||||
|
||||
|
||||
@router.get("/groups", response_model=list[GroupResponse])
|
||||
def list_groups(services: Services, identity: Identity) -> list[GroupResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
return [
|
||||
GroupResponse.from_managed_group(value)
|
||||
for value in administration.list_groups(organization_id, user_id)
|
||||
]
|
||||
|
||||
|
||||
@router.post("/groups", response_model=GroupResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create_group(
|
||||
payload: IdentityCollectionCreateRequest, services: Services, identity: Identity
|
||||
) -> GroupResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_group(
|
||||
organization_id,
|
||||
user_id,
|
||||
name=payload.name,
|
||||
description=payload.description,
|
||||
)
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="group already exists") from error
|
||||
return GroupResponse.from_managed_group(value)
|
||||
|
||||
|
||||
@router.put("/groups/{group_id}/members/{member_user_id}", status_code=204)
|
||||
def add_group_member(
|
||||
group_id: UUID,
|
||||
member_user_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_group_membership(group_id, member_user_id, services, identity, present=True)
|
||||
|
||||
|
||||
@router.delete("/groups/{group_id}/members/{member_user_id}", status_code=204)
|
||||
def remove_group_member(
|
||||
group_id: UUID,
|
||||
member_user_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_group_membership(group_id, member_user_id, services, identity, present=False)
|
||||
|
||||
|
||||
@router.get("/roles", response_model=list[RoleResponse])
|
||||
def list_roles(services: Services, identity: Identity) -> list[RoleResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
return [
|
||||
RoleResponse.from_managed_role(value)
|
||||
for value in administration.list_roles(organization_id, user_id)
|
||||
]
|
||||
|
||||
|
||||
@router.post("/roles", response_model=RoleResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create_role(
|
||||
payload: IdentityCollectionCreateRequest, services: Services, identity: Identity
|
||||
) -> RoleResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_role(
|
||||
organization_id,
|
||||
user_id,
|
||||
name=payload.name,
|
||||
description=payload.description,
|
||||
)
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="role already exists") from error
|
||||
return RoleResponse.from_managed_role(value)
|
||||
|
||||
|
||||
@router.put("/roles/{role_id}/users/{assigned_user_id}", status_code=204)
|
||||
def assign_role_to_user(
|
||||
role_id: UUID,
|
||||
assigned_user_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_user_role(role_id, assigned_user_id, services, identity, present=True)
|
||||
|
||||
|
||||
@router.delete("/roles/{role_id}/users/{assigned_user_id}", status_code=204)
|
||||
def revoke_role_from_user(
|
||||
role_id: UUID,
|
||||
assigned_user_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_user_role(role_id, assigned_user_id, services, identity, present=False)
|
||||
|
||||
|
||||
@router.put("/roles/{role_id}/groups/{group_id}", status_code=204)
|
||||
def assign_role_to_group(
|
||||
role_id: UUID,
|
||||
group_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_group_role(role_id, group_id, services, identity, present=True)
|
||||
|
||||
|
||||
@router.delete("/roles/{role_id}/groups/{group_id}", status_code=204)
|
||||
def revoke_role_from_group(
|
||||
role_id: UUID,
|
||||
group_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> Response:
|
||||
return _set_group_role(role_id, group_id, services, identity, present=False)
|
||||
|
||||
|
||||
@router.get("/knowledge-bases", response_model=list[KnowledgeBaseResponse])
|
||||
def list_knowledge_bases(services: Services, identity: Identity) -> list[KnowledgeBaseResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
return [
|
||||
KnowledgeBaseResponse.from_knowledge_base(value)
|
||||
for value in administration.list_knowledge_bases(organization_id, user_id)
|
||||
]
|
||||
|
||||
|
||||
@router.post(
|
||||
"/knowledge-bases",
|
||||
response_model=KnowledgeBaseResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_knowledge_base(
|
||||
payload: KnowledgeBaseCreateRequest, services: Services, identity: Identity
|
||||
) -> KnowledgeBaseResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_knowledge_base(
|
||||
organization_id,
|
||||
user_id,
|
||||
name=payload.name,
|
||||
description=payload.description,
|
||||
visibility=payload.visibility,
|
||||
data_processing_policy=payload.data_processing_policy,
|
||||
mcp_enabled=payload.mcp_enabled,
|
||||
)
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="knowledge base already exists") from error
|
||||
return KnowledgeBaseResponse.from_knowledge_base(value)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/knowledge-bases/{knowledge_base_id}/folders",
|
||||
response_model=list[FolderResponse],
|
||||
)
|
||||
def list_folders(
|
||||
knowledge_base_id: UUID, services: Services, identity: Identity
|
||||
) -> list[FolderResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
values = administration.list_folders(organization_id, user_id, knowledge_base_id)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found") from error
|
||||
return [FolderResponse.from_folder(value) for value in values]
|
||||
|
||||
|
||||
@router.post(
|
||||
"/knowledge-bases/{knowledge_base_id}/folders",
|
||||
response_model=FolderResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_folder(
|
||||
knowledge_base_id: UUID,
|
||||
payload: FolderCreateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> FolderResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_folder(
|
||||
organization_id,
|
||||
user_id,
|
||||
knowledge_base_id,
|
||||
name=payload.name,
|
||||
description=payload.description,
|
||||
parent_folder_id=payload.parent_folder_id,
|
||||
)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="knowledge base or parent not found") from error
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="folder already exists") from error
|
||||
return FolderResponse.from_folder(value)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/knowledge-bases/{knowledge_base_id}/access-control",
|
||||
response_model=list[AccessControlEntryResponse],
|
||||
)
|
||||
def list_access_control(
|
||||
knowledge_base_id: UUID, services: Services, identity: Identity
|
||||
) -> list[AccessControlEntryResponse]:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
values = administration.list_access_control_entries(
|
||||
organization_id, user_id, knowledge_base_id
|
||||
)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found") from error
|
||||
return [AccessControlEntryResponse.from_entry(value) for value in values]
|
||||
|
||||
|
||||
@router.post(
|
||||
"/knowledge-bases/{knowledge_base_id}/access-control",
|
||||
response_model=AccessControlEntryResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_access_control(
|
||||
knowledge_base_id: UUID,
|
||||
payload: AccessControlEntryCreateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> AccessControlEntryResponse:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
value = administration.create_access_control_entry(
|
||||
organization_id,
|
||||
user_id,
|
||||
knowledge_base_id,
|
||||
subject_type=payload.subject_type,
|
||||
subject_id=payload.subject_id,
|
||||
resource_type=AclResourceType(payload.resource_type),
|
||||
resource_id=payload.resource_id,
|
||||
permission=payload.permission,
|
||||
effect=payload.effect,
|
||||
applies_to_descendants=payload.applies_to_descendants,
|
||||
)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="ACL subject or resource not found") from error
|
||||
except AdministrationConflictError as error:
|
||||
raise HTTPException(status_code=409, detail="access rule already exists") from error
|
||||
return AccessControlEntryResponse.from_entry(value)
|
||||
|
||||
|
||||
@router.delete("/access-control/{entry_id}", status_code=status.HTTP_204_NO_CONTENT)
|
||||
def delete_access_control(entry_id: UUID, services: Services, identity: Identity) -> Response:
|
||||
organization_id, user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
administration.delete_access_control_entry(organization_id, user_id, entry_id)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="access rule not found") from error
|
||||
return Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
|
||||
|
||||
def _management_context(
|
||||
services: ApiServices, identity: ApiIdentity
|
||||
) -> tuple[UUID, UUID, AdministrationService]:
|
||||
"""Enforce the management scope before exposing canonical directory data."""
|
||||
require_scope(identity, "knowledge:manage")
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.administration is None:
|
||||
raise HTTPException(status_code=503, detail="administration service is unavailable")
|
||||
try:
|
||||
services.administration.get_session(organization_id, user_id)
|
||||
except AdministrationIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
return organization_id, user_id, services.administration
|
||||
|
||||
|
||||
def _set_group_membership(
|
||||
group_id: UUID,
|
||||
member_user_id: UUID,
|
||||
services: ApiServices,
|
||||
identity: ApiIdentity,
|
||||
*,
|
||||
present: bool,
|
||||
) -> Response:
|
||||
organization_id, actor_user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
administration.set_group_membership(
|
||||
organization_id,
|
||||
actor_user_id,
|
||||
group_id,
|
||||
member_user_id,
|
||||
present=present,
|
||||
)
|
||||
except (AdministrationIdentityError, AdministrationNotFoundError) as error:
|
||||
raise HTTPException(status_code=404, detail="group or user not found") from error
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
def _set_user_role(
|
||||
role_id: UUID,
|
||||
assigned_user_id: UUID,
|
||||
services: ApiServices,
|
||||
identity: ApiIdentity,
|
||||
*,
|
||||
present: bool,
|
||||
) -> Response:
|
||||
organization_id, actor_user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
administration.set_user_role_assignment(
|
||||
organization_id,
|
||||
actor_user_id,
|
||||
role_id,
|
||||
assigned_user_id,
|
||||
present=present,
|
||||
)
|
||||
except (AdministrationIdentityError, AdministrationNotFoundError) as error:
|
||||
raise HTTPException(status_code=404, detail="role or user not found") from error
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
def _set_group_role(
|
||||
role_id: UUID,
|
||||
group_id: UUID,
|
||||
services: ApiServices,
|
||||
identity: ApiIdentity,
|
||||
*,
|
||||
present: bool,
|
||||
) -> Response:
|
||||
organization_id, actor_user_id, administration = _management_context(services, identity)
|
||||
try:
|
||||
administration.set_group_role_assignment(
|
||||
organization_id,
|
||||
actor_user_id,
|
||||
role_id,
|
||||
group_id,
|
||||
present=present,
|
||||
)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="role or group not found") from error
|
||||
return Response(status_code=204)
|
||||
120
hy4/_kp/apps/api/src/ekp_api/api_identity_bootstrap.py
Normal file
120
hy4/_kp/apps/api/src/ekp_api/api_identity_bootstrap.py
Normal file
@ -0,0 +1,120 @@
|
||||
"""Explicit first-identity provisioning for runtime REST bearer mappings."""
|
||||
|
||||
import argparse
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from ekp_core.config import ApiIdentityCredential, Settings, get_settings
|
||||
from ekp_core.database import create_database_engine, create_session_factory
|
||||
from ekp_domain import IdentityStatus, Organization, User
|
||||
|
||||
|
||||
class IdentityBootstrapError(RuntimeError):
|
||||
"""Raised when requested canonical IDs conflict with existing facts."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class IdentityBootstrapResult:
|
||||
"""Non-sensitive result suitable for operator output and tests."""
|
||||
|
||||
organization_created: bool
|
||||
user_created: bool
|
||||
organization_id: str
|
||||
user_id: str
|
||||
|
||||
|
||||
def provision_api_identity(
|
||||
settings: Settings,
|
||||
*,
|
||||
client_id: str,
|
||||
organization_slug: str,
|
||||
organization_name: str,
|
||||
display_name: str,
|
||||
) -> IdentityBootstrapResult:
|
||||
"""Idempotently create the exact Organization/User named by one secret mapping."""
|
||||
credential = _credential_by_client_id(settings, client_id)
|
||||
session_factory = create_session_factory(create_database_engine(settings))
|
||||
organization_created = False
|
||||
user_created = False
|
||||
with session_factory.begin() as session:
|
||||
organization = session.get(Organization, credential.organization_id)
|
||||
if organization is None:
|
||||
conflicting = session.scalar(
|
||||
select(Organization.id).where(Organization.slug == organization_slug.strip())
|
||||
)
|
||||
if conflicting is not None:
|
||||
raise IdentityBootstrapError("organization slug belongs to another identity")
|
||||
session.add(
|
||||
Organization(
|
||||
id=credential.organization_id,
|
||||
slug=organization_slug.strip(),
|
||||
name=organization_name.strip(),
|
||||
)
|
||||
)
|
||||
session.flush()
|
||||
organization_created = True
|
||||
|
||||
user = session.get(User, credential.user_id)
|
||||
if user is None:
|
||||
session.add(
|
||||
User(
|
||||
id=credential.user_id,
|
||||
organization_id=credential.organization_id,
|
||||
issuer=f"ekp-api:{credential.credential_type}",
|
||||
subject=credential.subject,
|
||||
display_name=display_name.strip(),
|
||||
status=IdentityStatus.ACTIVE,
|
||||
)
|
||||
)
|
||||
user_created = True
|
||||
elif (
|
||||
user.organization_id != credential.organization_id
|
||||
or user.subject != credential.subject
|
||||
or user.status is not IdentityStatus.ACTIVE
|
||||
):
|
||||
raise IdentityBootstrapError("configured User mapping conflicts with canonical data")
|
||||
|
||||
return IdentityBootstrapResult(
|
||||
organization_created=organization_created,
|
||||
user_created=user_created,
|
||||
organization_id=str(credential.organization_id),
|
||||
user_id=str(credential.user_id),
|
||||
)
|
||||
|
||||
|
||||
def _credential_by_client_id(settings: Settings, client_id: str) -> ApiIdentityCredential:
|
||||
matches = [
|
||||
credential
|
||||
for credential in settings.api_identity_credentials
|
||||
if credential.client_id == client_id
|
||||
]
|
||||
if len(matches) != 1:
|
||||
raise IdentityBootstrapError("client_id must select exactly one API credential")
|
||||
return matches[0]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Provision one configured identity without accepting a secret on the command line."""
|
||||
parser = argparse.ArgumentParser(description="Provision one configured EKP API identity")
|
||||
parser.add_argument("--client-id", required=True)
|
||||
parser.add_argument("--organization-slug", required=True)
|
||||
parser.add_argument("--organization-name", required=True)
|
||||
parser.add_argument("--display-name", required=True)
|
||||
arguments = parser.parse_args()
|
||||
result = provision_api_identity(
|
||||
get_settings(),
|
||||
client_id=arguments.client_id,
|
||||
organization_slug=arguments.organization_slug,
|
||||
organization_name=arguments.organization_name,
|
||||
display_name=arguments.display_name,
|
||||
)
|
||||
print(
|
||||
"API identity provisioned "
|
||||
f"organization_id={result.organization_id} user_id={result.user_id} "
|
||||
f"organization_created={result.organization_created} user_created={result.user_created}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
104
hy4/_kp/apps/api/src/ekp_api/auth.py
Normal file
104
hy4/_kp/apps/api/src/ekp_api/auth.py
Normal file
@ -0,0 +1,104 @@
|
||||
"""Request-scoped REST identity resolved from runtime-secret bearer credentials."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from secrets import compare_digest
|
||||
from typing import Literal, Never
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import HTTPException, Request, status
|
||||
|
||||
from ekp_core.config import ApiIdentityCredential
|
||||
|
||||
ApiCredentialType = Literal["legacy_public", "user_token", "service_api_key"]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApiIdentity:
|
||||
"""Authenticated REST principal with its canonical authorization subject."""
|
||||
|
||||
client_id: str
|
||||
subject: str
|
||||
scopes: frozenset[str]
|
||||
credential_type: ApiCredentialType
|
||||
organization_id: UUID | None = None
|
||||
user_id: UUID | None = None
|
||||
|
||||
@property
|
||||
def is_canonical(self) -> bool:
|
||||
return self.organization_id is not None and self.user_id is not None
|
||||
|
||||
|
||||
class ConfiguredApiIdentityVerifier:
|
||||
"""Compare every configured secret and return only canonical identity metadata."""
|
||||
|
||||
def __init__(self, credentials: tuple[ApiIdentityCredential, ...]) -> None:
|
||||
if not credentials:
|
||||
raise ValueError("at least one API identity credential is required")
|
||||
tokens = [credential.token.get_secret_value() for credential in credentials]
|
||||
if len(tokens) != len(set(tokens)):
|
||||
raise ValueError("API identity credential tokens must be unique")
|
||||
self._credentials = credentials
|
||||
|
||||
def verify(self, token: str) -> ApiIdentity | None:
|
||||
"""Use constant-time comparisons without reflecting or logging bearer values."""
|
||||
selected: ApiIdentityCredential | None = None
|
||||
for credential in self._credentials:
|
||||
if compare_digest(token, credential.token.get_secret_value()):
|
||||
selected = credential
|
||||
if selected is None:
|
||||
return None
|
||||
return ApiIdentity(
|
||||
client_id=selected.client_id,
|
||||
subject=selected.subject,
|
||||
scopes=frozenset(selected.scopes),
|
||||
credential_type=selected.credential_type,
|
||||
organization_id=selected.organization_id,
|
||||
user_id=selected.user_id,
|
||||
)
|
||||
|
||||
|
||||
def get_api_identity(request: Request) -> ApiIdentity:
|
||||
"""Authenticate one REST request or enter explicitly enabled public compatibility."""
|
||||
settings = request.app.state.settings
|
||||
credentials = tuple(settings.api_identity_credentials)
|
||||
authorization = request.headers.get("authorization", "")
|
||||
test_compatibility = settings.app_env == "test"
|
||||
if (
|
||||
not credentials
|
||||
and (settings.api_allow_legacy_public or test_compatibility)
|
||||
and not authorization
|
||||
):
|
||||
return ApiIdentity(
|
||||
client_id="legacy-admin-web",
|
||||
subject="public-knowledge",
|
||||
scopes=frozenset({"knowledge:read", "knowledge:write", "knowledge:search"}),
|
||||
credential_type="legacy_public",
|
||||
)
|
||||
scheme, separator, token = authorization.partition(" ")
|
||||
if separator != " " or scheme.lower() != "bearer" or not token:
|
||||
_raise_unauthorized()
|
||||
identity = ConfiguredApiIdentityVerifier(credentials).verify(token) if credentials else None
|
||||
if identity is None:
|
||||
_raise_unauthorized()
|
||||
return identity
|
||||
|
||||
|
||||
def require_scope(identity: ApiIdentity, scope: str) -> None:
|
||||
"""Reject a valid credential that lacks one explicit operation scope."""
|
||||
if scope not in identity.scopes:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="operation is forbidden")
|
||||
|
||||
|
||||
def require_canonical_identity(identity: ApiIdentity) -> tuple[UUID, UUID]:
|
||||
"""Return canonical IDs or reject legacy compatibility from protected operations."""
|
||||
if identity.organization_id is None or identity.user_id is None:
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="operation is forbidden")
|
||||
return identity.organization_id, identity.user_id
|
||||
|
||||
|
||||
def _raise_unauthorized() -> Never:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_401_UNAUTHORIZED,
|
||||
detail="authentication required",
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
81
hy4/_kp/apps/api/src/ekp_api/celery_queue.py
Normal file
81
hy4/_kp/apps/api/src/ekp_api/celery_queue.py
Normal file
@ -0,0 +1,81 @@
|
||||
"""Celery/RabbitMQ adapter behind the queue-neutral ingestion dispatcher."""
|
||||
|
||||
from uuid import UUID
|
||||
|
||||
from celery import Celery # type: ignore[import-untyped]
|
||||
|
||||
from ekp_core.config import Settings
|
||||
|
||||
INGESTION_TASK_NAME = "ekp.ingestion.run"
|
||||
RECOVERY_TASK_NAME = "ekp.ingestion.recover"
|
||||
COMPILATION_TASK_NAME = "ekp.compilation.run"
|
||||
COMPILATION_RECOVERY_TASK_NAME = "ekp.compilation.recover"
|
||||
|
||||
|
||||
def create_celery_app(settings: Settings) -> Celery:
|
||||
"""Create a secret-safe broker client without opening a connection eagerly."""
|
||||
if settings.rabbitmq_url is None or not settings.rabbitmq_url.get_secret_value().strip():
|
||||
raise ValueError("RABBITMQ_URL is required for asynchronous ingestion")
|
||||
app = Celery(
|
||||
"enterprise-knowledge-platform",
|
||||
broker=settings.rabbitmq_url.get_secret_value(),
|
||||
)
|
||||
app.conf.update(
|
||||
task_ignore_result=True,
|
||||
task_acks_late=True,
|
||||
task_reject_on_worker_lost=True,
|
||||
worker_prefetch_multiplier=1,
|
||||
broker_connection_retry_on_startup=True,
|
||||
# Preserve the shared JSON handler instead of letting Celery replace it
|
||||
# with process-specific plain-text formatting.
|
||||
worker_hijack_root_logger=False,
|
||||
timezone="UTC",
|
||||
enable_utc=True,
|
||||
task_routes={
|
||||
INGESTION_TASK_NAME: {"queue": "ingestion"},
|
||||
RECOVERY_TASK_NAME: {"queue": "ingestion"},
|
||||
COMPILATION_TASK_NAME: {"queue": "compilation"},
|
||||
COMPILATION_RECOVERY_TASK_NAME: {"queue": "compilation"},
|
||||
},
|
||||
beat_schedule={
|
||||
"recover-durable-ingestion-jobs": {
|
||||
"task": RECOVERY_TASK_NAME,
|
||||
"schedule": float(settings.ingestion_recovery_interval_seconds),
|
||||
},
|
||||
"recover-durable-compilation-jobs": {
|
||||
"task": COMPILATION_RECOVERY_TASK_NAME,
|
||||
"schedule": float(settings.compilation_recovery_interval_seconds),
|
||||
},
|
||||
},
|
||||
)
|
||||
return app
|
||||
|
||||
|
||||
class CeleryIngestionDispatcher:
|
||||
"""Publish only a durable Job id; source bytes and state stay canonical."""
|
||||
|
||||
def __init__(self, app: Celery) -> None:
|
||||
self._app = app
|
||||
|
||||
def enqueue(self, job_id: UUID) -> None:
|
||||
"""Deliver one idempotent execution hint to the ingestion queue."""
|
||||
self._app.send_task(
|
||||
INGESTION_TASK_NAME,
|
||||
args=[str(job_id)],
|
||||
queue="ingestion",
|
||||
)
|
||||
|
||||
|
||||
class CeleryCompilationDispatcher:
|
||||
"""Publish only a durable Compilation Job ID to the dedicated queue."""
|
||||
|
||||
def __init__(self, app: Celery) -> None:
|
||||
self._app = app
|
||||
|
||||
def enqueue(self, job_id: UUID) -> None:
|
||||
"""Deliver one idempotent generation hint without source content."""
|
||||
self._app.send_task(
|
||||
COMPILATION_TASK_NAME,
|
||||
args=[str(job_id)],
|
||||
queue="compilation",
|
||||
)
|
||||
492
hy4/_kp/apps/api/src/ekp_api/compiled_routes.py
Normal file
492
hy4/_kp/apps/api/src/ekp_api/compiled_routes.py
Normal file
@ -0,0 +1,492 @@
|
||||
"""Authenticated REST routes for source-backed Compiled Knowledge drafts."""
|
||||
|
||||
from typing import Annotated
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, status
|
||||
|
||||
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
|
||||
from ekp_api.providers import ApiServices
|
||||
from ekp_api.routes import (
|
||||
_resolved_document_ids,
|
||||
_resolved_knowledge_base_ids,
|
||||
get_api_services,
|
||||
)
|
||||
from ekp_api.schemas import (
|
||||
CompilationJobResponse,
|
||||
CompiledKnowledgeConflictCreateRequest,
|
||||
CompiledKnowledgeConflictResolveRequest,
|
||||
CompiledKnowledgeDraftCreateRequest,
|
||||
CompiledKnowledgeDraftUpdateRequest,
|
||||
CompiledKnowledgeGenerateRequest,
|
||||
CompiledKnowledgeRejectRequest,
|
||||
CompiledKnowledgeResponse,
|
||||
CompiledKnowledgeTransitionRequest,
|
||||
)
|
||||
from ekp_core.compilation_jobs import (
|
||||
AsyncCompilationSubmissionService,
|
||||
CompilationJobNotFoundError,
|
||||
CompilationJobService,
|
||||
)
|
||||
from ekp_core.compiled_knowledge import (
|
||||
CompilationDataPolicyError,
|
||||
CompilationProviderUnavailableError,
|
||||
CompilationSourceError,
|
||||
CompiledKnowledgeConflictError,
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeNotFoundError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
CompiledKnowledgeService,
|
||||
CompiledKnowledgeSourceStaleError,
|
||||
)
|
||||
from ekp_domain import AclPermission
|
||||
|
||||
router = APIRouter(prefix="/compiled-knowledge", tags=["compiled-knowledge"])
|
||||
Services = Annotated[ApiServices, Depends(get_api_services)]
|
||||
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
|
||||
|
||||
|
||||
@router.get("", response_model=list[CompiledKnowledgeResponse])
|
||||
def list_compiled_knowledge(
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
knowledge_base_id: Annotated[UUID | None, Query()] = None,
|
||||
) -> list[CompiledKnowledgeResponse]:
|
||||
"""List only drafts whose KB and every source Document remain readable."""
|
||||
service, organization_id, readable_kbs, readable_documents = _read_context(services, identity)
|
||||
values = service.list_items(
|
||||
organization_id,
|
||||
readable_kbs,
|
||||
readable_documents,
|
||||
knowledge_base_id=knowledge_base_id,
|
||||
)
|
||||
return [CompiledKnowledgeResponse.from_view(value) for value in values]
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=CompilationJobResponse)
|
||||
def get_compilation_job(
|
||||
job_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompilationJobResponse:
|
||||
"""Poll a Job only while its KB and every source Document remain readable."""
|
||||
require_scope(identity, "knowledge:read")
|
||||
organization_id, _ = require_canonical_identity(identity)
|
||||
jobs = _job_service(services)
|
||||
try:
|
||||
value = jobs.get_authorized(
|
||||
job_id=job_id,
|
||||
organization_id=organization_id,
|
||||
allowed_knowledge_base_ids=_resolved_knowledge_base_ids(
|
||||
identity, services, AclPermission.READ
|
||||
),
|
||||
allowed_source_document_ids=_resolved_document_ids(
|
||||
identity,
|
||||
services,
|
||||
AclPermission.READ,
|
||||
include_deleted=True,
|
||||
)
|
||||
or (),
|
||||
)
|
||||
except CompilationJobNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compilation job not found") from error
|
||||
return CompilationJobResponse.from_view(value)
|
||||
|
||||
|
||||
@router.get("/{compiled_id}", response_model=CompiledKnowledgeResponse)
|
||||
def get_compiled_knowledge(
|
||||
compiled_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Read one item through the same source-intersection authorization rule."""
|
||||
service, organization_id, readable_kbs, readable_documents = _read_context(services, identity)
|
||||
try:
|
||||
value = service.get_item(
|
||||
organization_id,
|
||||
readable_kbs,
|
||||
readable_documents,
|
||||
compiled_id,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/drafts",
|
||||
response_model=CompiledKnowledgeResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def create_compiled_knowledge_draft(
|
||||
payload: CompiledKnowledgeDraftCreateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Create a manual draft in one writable KB from readable source Chunks."""
|
||||
service, organization_id, actor_user_id, readable_documents = _write_context(
|
||||
services, identity, payload.knowledge_base_id
|
||||
)
|
||||
try:
|
||||
value = service.create_manual_draft(
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=payload.knowledge_base_id,
|
||||
actor_user_id=actor_user_id,
|
||||
title=payload.title,
|
||||
knowledge_type=payload.knowledge_type,
|
||||
content=payload.content,
|
||||
source_chunk_ids=payload.source_chunk_ids,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
)
|
||||
except (CompilationSourceError, ValueError) as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/drafts:generate",
|
||||
response_model=CompilationJobResponse,
|
||||
status_code=status.HTTP_202_ACCEPTED,
|
||||
)
|
||||
def generate_compiled_knowledge_draft(
|
||||
payload: CompiledKnowledgeGenerateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompilationJobResponse:
|
||||
"""Validate and enqueue durable AI generation without blocking the API."""
|
||||
_, organization_id, actor_user_id, readable_documents = _write_context(
|
||||
services, identity, payload.knowledge_base_id
|
||||
)
|
||||
submission = _submission_service(services)
|
||||
try:
|
||||
value = submission.submit(
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=payload.knowledge_base_id,
|
||||
actor_user_id=actor_user_id,
|
||||
knowledge_type=payload.knowledge_type,
|
||||
source_chunk_ids=payload.source_chunk_ids,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
title_hint=payload.title_hint,
|
||||
instruction=payload.instruction,
|
||||
)
|
||||
except CompilationProviderUnavailableError as error:
|
||||
raise HTTPException(
|
||||
status_code=503, detail="compilation provider is unavailable"
|
||||
) from error
|
||||
except CompilationDataPolicyError as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except (CompilationSourceError, ValueError) as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompilationJobResponse.from_view(value)
|
||||
|
||||
|
||||
@router.put("/{compiled_id}/draft", response_model=CompiledKnowledgeResponse)
|
||||
def update_compiled_knowledge_draft(
|
||||
compiled_id: UUID,
|
||||
payload: CompiledKnowledgeDraftUpdateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Edit only a draft visible through the current source permission intersection."""
|
||||
require_scope(identity, "knowledge:write")
|
||||
organization_id, actor_user_id = require_canonical_identity(identity)
|
||||
service = _service(services)
|
||||
writable_kbs = _resolved_knowledge_base_ids(identity, services, AclPermission.WRITE)
|
||||
readable_documents = (
|
||||
_resolved_document_ids(
|
||||
identity,
|
||||
services,
|
||||
AclPermission.READ,
|
||||
include_deleted=True,
|
||||
)
|
||||
or ()
|
||||
)
|
||||
try:
|
||||
value = service.update_draft(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=writable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
title=payload.title,
|
||||
knowledge_type=payload.knowledge_type,
|
||||
content=payload.content,
|
||||
source_chunk_ids=payload.source_chunk_ids,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post("/{compiled_id}:submit-review", response_model=CompiledKnowledgeResponse)
|
||||
def submit_compiled_knowledge_review(
|
||||
compiled_id: UUID,
|
||||
payload: CompiledKnowledgeTransitionRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Verify exact source freshness before entering human review."""
|
||||
service, organization_id, actor_user_id, writable_kbs, readable_documents = _lifecycle_context(
|
||||
services, identity, AclPermission.WRITE, "knowledge:write"
|
||||
)
|
||||
try:
|
||||
value = service.submit_for_review(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=writable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
CompiledKnowledgeSourceStaleError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post("/{compiled_id}/conflicts", response_model=CompiledKnowledgeResponse)
|
||||
def report_compiled_knowledge_conflict(
|
||||
compiled_id: UUID,
|
||||
payload: CompiledKnowledgeConflictCreateRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Record a reviewer-confirmed contradiction with exact source evidence."""
|
||||
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
|
||||
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
|
||||
)
|
||||
try:
|
||||
value = service.report_conflict(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=manageable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
claim=payload.claim,
|
||||
left_source_id=payload.left_source_id,
|
||||
right_source_id=payload.right_source_id,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeConflictError,
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/{compiled_id}/conflicts/{conflict_id}:resolve",
|
||||
response_model=CompiledKnowledgeResponse,
|
||||
)
|
||||
def resolve_compiled_knowledge_conflict(
|
||||
compiled_id: UUID,
|
||||
conflict_id: UUID,
|
||||
payload: CompiledKnowledgeConflictResolveRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Close one conflict only with an explicit reviewer explanation."""
|
||||
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
|
||||
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
|
||||
)
|
||||
try:
|
||||
value = service.resolve_conflict(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=manageable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
conflict_id=conflict_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
resolution_note=payload.resolution_note,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeConflictError,
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post("/{compiled_id}:publish", response_model=CompiledKnowledgeResponse)
|
||||
def publish_compiled_knowledge(
|
||||
compiled_id: UUID,
|
||||
payload: CompiledKnowledgeTransitionRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Publish only a verified review revision with no open conflict."""
|
||||
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
|
||||
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
|
||||
)
|
||||
try:
|
||||
value = service.publish(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=manageable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeConflictError,
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
CompiledKnowledgeSourceStaleError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
@router.post("/{compiled_id}:reject", response_model=CompiledKnowledgeResponse)
|
||||
def reject_compiled_knowledge(
|
||||
compiled_id: UUID,
|
||||
payload: CompiledKnowledgeRejectRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> CompiledKnowledgeResponse:
|
||||
"""Reject one review revision with an editor-facing reason."""
|
||||
service, organization_id, actor_user_id, manageable_kbs, readable_documents = (
|
||||
_lifecycle_context(services, identity, AclPermission.MANAGE, "knowledge:manage")
|
||||
)
|
||||
try:
|
||||
value = service.reject(
|
||||
organization_id=organization_id,
|
||||
actor_user_id=actor_user_id,
|
||||
allowed_knowledge_base_ids=manageable_kbs,
|
||||
allowed_source_document_ids=readable_documents,
|
||||
compiled_id=compiled_id,
|
||||
expected_revision=payload.expected_revision,
|
||||
reason=payload.reason,
|
||||
)
|
||||
except CompiledKnowledgeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="compiled knowledge not found") from error
|
||||
except (
|
||||
CompiledKnowledgeDraftStateError,
|
||||
CompiledKnowledgeRevisionConflictError,
|
||||
) as error:
|
||||
raise HTTPException(status_code=409, detail=str(error)) from error
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=422, detail=str(error)) from error
|
||||
return CompiledKnowledgeResponse.from_view(value)
|
||||
|
||||
|
||||
def _service(services: ApiServices) -> CompiledKnowledgeService:
|
||||
"""Fail closed if a custom test/runtime graph omitted compilation."""
|
||||
if services.compiled_knowledge is None:
|
||||
raise HTTPException(status_code=503, detail="compiled knowledge service is unavailable")
|
||||
return services.compiled_knowledge
|
||||
|
||||
|
||||
def _job_service(services: ApiServices) -> CompilationJobService:
|
||||
"""Fail closed when a custom runtime omitted durable generation Jobs."""
|
||||
if services.compilation_jobs is None:
|
||||
raise HTTPException(status_code=503, detail="compilation job service is unavailable")
|
||||
return services.compilation_jobs
|
||||
|
||||
|
||||
def _submission_service(services: ApiServices) -> AsyncCompilationSubmissionService:
|
||||
"""Fail closed when a custom runtime omitted async generation submission."""
|
||||
if services.compilation_submissions is None:
|
||||
raise HTTPException(status_code=503, detail="compilation submission is unavailable")
|
||||
return services.compilation_submissions
|
||||
|
||||
|
||||
def _read_context(
|
||||
services: ApiServices, identity: ApiIdentity
|
||||
) -> tuple[CompiledKnowledgeService, UUID, tuple[UUID, ...], tuple[UUID, ...]]:
|
||||
require_scope(identity, "knowledge:read")
|
||||
organization_id, _ = require_canonical_identity(identity)
|
||||
return (
|
||||
_service(services),
|
||||
organization_id,
|
||||
_resolved_knowledge_base_ids(identity, services, AclPermission.READ),
|
||||
_resolved_document_ids(
|
||||
identity,
|
||||
services,
|
||||
AclPermission.READ,
|
||||
include_deleted=True,
|
||||
)
|
||||
or (),
|
||||
)
|
||||
|
||||
|
||||
def _write_context(
|
||||
services: ApiServices,
|
||||
identity: ApiIdentity,
|
||||
knowledge_base_id: UUID,
|
||||
) -> tuple[CompiledKnowledgeService, UUID, UUID, tuple[UUID, ...]]:
|
||||
require_scope(identity, "knowledge:write")
|
||||
organization_id, actor_user_id = require_canonical_identity(identity)
|
||||
if knowledge_base_id not in _resolved_knowledge_base_ids(
|
||||
identity, services, AclPermission.WRITE
|
||||
):
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found")
|
||||
return (
|
||||
_service(services),
|
||||
organization_id,
|
||||
actor_user_id,
|
||||
_resolved_document_ids(
|
||||
identity,
|
||||
services,
|
||||
AclPermission.READ,
|
||||
include_deleted=True,
|
||||
)
|
||||
or (),
|
||||
)
|
||||
|
||||
|
||||
def _lifecycle_context(
|
||||
services: ApiServices,
|
||||
identity: ApiIdentity,
|
||||
permission: AclPermission,
|
||||
scope: str,
|
||||
) -> tuple[
|
||||
CompiledKnowledgeService,
|
||||
UUID,
|
||||
UUID,
|
||||
tuple[UUID, ...],
|
||||
tuple[UUID, ...],
|
||||
]:
|
||||
"""Resolve one lifecycle action before reading or mutating L2 content."""
|
||||
require_scope(identity, scope)
|
||||
organization_id, actor_user_id = require_canonical_identity(identity)
|
||||
return (
|
||||
_service(services),
|
||||
organization_id,
|
||||
actor_user_id,
|
||||
_resolved_knowledge_base_ids(identity, services, permission),
|
||||
_resolved_document_ids(
|
||||
identity,
|
||||
services,
|
||||
AclPermission.READ,
|
||||
include_deleted=True,
|
||||
)
|
||||
or (),
|
||||
)
|
||||
113
hy4/_kp/apps/api/src/ekp_api/dev_startup.py
Normal file
113
hy4/_kp/apps/api/src/ekp_api/dev_startup.py
Normal file
@ -0,0 +1,113 @@
|
||||
"""Dev-only API startup with canonical database migration."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import uvicorn
|
||||
from alembic import command
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import inspect, text
|
||||
|
||||
from ekp_core.config import Settings, get_settings, get_settings_profile
|
||||
from ekp_core.database import create_database_engine
|
||||
from ekp_core.observability import configure_json_logging
|
||||
|
||||
_DEV_PROFILE = "dev-image"
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DevMigrationError(RuntimeError):
|
||||
"""Report a safe dev migration failure without connection or SQL details."""
|
||||
|
||||
def __init__(self, *, error_type: str) -> None:
|
||||
super().__init__("dev canonical migration failed")
|
||||
self.error_type = error_type
|
||||
|
||||
|
||||
def _run_alembic(database_url: str) -> None:
|
||||
"""Upgrade the canonical schema using an explicit, non-rendered URL."""
|
||||
# The non-editable image installs this module into site-packages while the
|
||||
# Docker WORKDIR remains /app, where Alembic config and migrations live.
|
||||
config = Config(str(Path.cwd() / "alembic.ini"))
|
||||
config.attributes["database_url"] = database_url
|
||||
command.upgrade(config, "head")
|
||||
|
||||
|
||||
def migrate_dev_database(
|
||||
settings: Settings,
|
||||
*,
|
||||
migration_runner: Callable[[str], None] = _run_alembic,
|
||||
) -> None:
|
||||
"""Upgrade dev canonical tables before serving traffic.
|
||||
|
||||
The central dev pipeline can only update one default image and provides no
|
||||
Migration Job. Production targets retain the Helm hook and never call this
|
||||
entrypoint. The central dev deployment is constrained to one API replica;
|
||||
multi-replica environments must use the independent Migration Job instead.
|
||||
"""
|
||||
if settings.app_env != "development" or get_settings_profile() != _DEV_PROFILE:
|
||||
raise DevMigrationError(error_type="invalid_profile")
|
||||
|
||||
database_url = settings.database_url.get_secret_value()
|
||||
try:
|
||||
migration_runner(database_url)
|
||||
except Exception as error:
|
||||
raise DevMigrationError(error_type=type(error).__name__) from error
|
||||
|
||||
|
||||
def _read_canonical_revision(settings: Settings) -> str:
|
||||
"""Return only a safe revision identifier for dev startup diagnostics."""
|
||||
engine = create_database_engine(settings)
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
if not inspect(connection).has_table("alembic_version"):
|
||||
return "unversioned"
|
||||
revision = connection.scalar(text("SELECT version_num FROM alembic_version"))
|
||||
return revision if isinstance(revision, str) and revision else "unversioned"
|
||||
except Exception:
|
||||
return "unavailable"
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Migrate the authorized dev database, then start the API process."""
|
||||
settings = get_settings()
|
||||
configure_json_logging(settings.log_level)
|
||||
try:
|
||||
migrate_dev_database(settings)
|
||||
except DevMigrationError as error:
|
||||
os.environ["EKP_DEV_MIGRATION_STATUS"] = "failed"
|
||||
os.environ["EKP_DEV_MIGRATION_REVISION"] = _read_canonical_revision(settings)
|
||||
os.environ["EKP_DEV_MIGRATION_ERROR_TYPE"] = error.error_type
|
||||
logger.error(
|
||||
"Dev canonical migration failed",
|
||||
extra={
|
||||
"event": "dev_canonical_migration_failed",
|
||||
"dependency": "canonical_database",
|
||||
},
|
||||
)
|
||||
else:
|
||||
os.environ["EKP_DEV_MIGRATION_STATUS"] = "completed"
|
||||
os.environ["EKP_DEV_MIGRATION_REVISION"] = _read_canonical_revision(settings)
|
||||
os.environ.pop("EKP_DEV_MIGRATION_ERROR_TYPE", None)
|
||||
logger.info(
|
||||
"Dev canonical migration completed",
|
||||
extra={
|
||||
"event": "dev_canonical_migration_completed",
|
||||
"dependency": "canonical_database",
|
||||
},
|
||||
)
|
||||
uvicorn.run(
|
||||
"ekp_api.main:app",
|
||||
host=settings.api_host,
|
||||
port=settings.api_port,
|
||||
access_log=False,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
85
hy4/_kp/apps/api/src/ekp_api/index_rebuild.py
Normal file
85
hy4/_kp/apps/api/src/ekp_api/index_rebuild.py
Normal file
@ -0,0 +1,85 @@
|
||||
"""Controlled CLI for the ARCH-005 shadow index cutover workflow."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import asdict
|
||||
|
||||
from ekp_api.providers import (
|
||||
create_embedding_provider,
|
||||
create_index_profile,
|
||||
create_object_storage,
|
||||
create_vector_store,
|
||||
)
|
||||
from ekp_core.chunking import StructureAwareChunker
|
||||
from ekp_core.config import Settings
|
||||
from ekp_core.database import create_database_engine, create_session_factory
|
||||
from ekp_core.document_chunking import CanonicalChunkRebuildService
|
||||
from ekp_core.index_rebuild import IndexBuildSummary, IndexRebuildService
|
||||
|
||||
|
||||
def _parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Build, verify, activate, or roll back a versioned vector index."
|
||||
)
|
||||
parser.add_argument(
|
||||
"command",
|
||||
choices=("build", "verify", "activate", "rollback"),
|
||||
help="Lifecycle step to execute for INDEX_PROFILE_ID.",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _create_service(settings: Settings) -> IndexRebuildService:
|
||||
"""Compose the command from the same provider settings as the API runtime."""
|
||||
engine = create_database_engine(settings)
|
||||
session_factory = create_session_factory(engine)
|
||||
target_profile = create_index_profile(settings)
|
||||
return IndexRebuildService(
|
||||
session_factory=session_factory,
|
||||
embedding_provider=create_embedding_provider(settings),
|
||||
vector_store=create_vector_store(settings, target_profile),
|
||||
target_profile=target_profile,
|
||||
chunk_rebuilder=CanonicalChunkRebuildService(
|
||||
session_factory,
|
||||
create_object_storage(settings),
|
||||
StructureAwareChunker(
|
||||
max_tokens=target_profile.chunk_max_tokens,
|
||||
overlap_tokens=target_profile.chunk_overlap_tokens,
|
||||
max_characters=target_profile.chunk_max_characters,
|
||||
strategy_version=target_profile.chunk_strategy_version,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _execute(service: IndexRebuildService, command: str) -> IndexBuildSummary:
|
||||
"""Run exactly one explicit lifecycle step."""
|
||||
if command == "build":
|
||||
return service.rebuild()
|
||||
if command == "verify":
|
||||
return service.verify()
|
||||
if command == "activate":
|
||||
return service.activate()
|
||||
if command == "rollback":
|
||||
return service.rollback()
|
||||
raise ValueError("unsupported index rebuild command")
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
"""Return a shell-safe status without printing endpoints, SQL, or credentials."""
|
||||
args = _parser().parse_args(argv)
|
||||
try:
|
||||
summary = _execute(_create_service(Settings()), args.command)
|
||||
except Exception:
|
||||
print("index profile command failed; inspect sanitized service logs", file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(asdict(summary), sort_keys=True, separators=(",", ":")))
|
||||
# Shell automation must not interpret an incomplete build/verification as
|
||||
# a successful cutover prerequisite merely because safe JSON was emitted.
|
||||
return 0 if summary.complete else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
275
hy4/_kp/apps/api/src/ekp_api/live_acceptance.py
Normal file
275
hy4/_kp/apps/api/src/ekp_api/live_acceptance.py
Normal file
@ -0,0 +1,275 @@
|
||||
"""Controlled end-to-end acceptance for the target provider chain."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import asdict, dataclass, replace
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from time import perf_counter_ns
|
||||
from uuid import uuid4
|
||||
|
||||
from ekp_api.providers import ApiServices, create_api_services
|
||||
from ekp_core.config import Settings
|
||||
from ekp_core.evaluation import (
|
||||
EvaluationCaseResult,
|
||||
GoldenDataset,
|
||||
GoldenDatasetError,
|
||||
GoldenSource,
|
||||
RankedSource,
|
||||
RetrievalMetrics,
|
||||
evaluate_rankings,
|
||||
load_golden_dataset,
|
||||
)
|
||||
from ekp_core.search import SearchItem, SourceResult
|
||||
|
||||
MIN_GOLDEN_QUERIES = 30
|
||||
MAX_GOLDEN_QUERIES = 500
|
||||
DEFAULT_MIN_RECALL_AT_1 = 0.90
|
||||
DEFAULT_MIN_RECALL_AT_5 = 1.0
|
||||
DEFAULT_MIN_MRR = 0.90
|
||||
DEFAULT_MIN_NDCG_AT_10 = 0.90
|
||||
DEFAULT_MAX_P95_MS = 10_000.0
|
||||
|
||||
|
||||
class LiveAcceptanceError(RuntimeError):
|
||||
"""Raised for a safe precondition or acceptance failure."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AcceptanceReport:
|
||||
"""Provider-neutral quality and latency metrics without retrieved content."""
|
||||
|
||||
query_count: int
|
||||
recall_at_1: float
|
||||
recall_at_3: float
|
||||
recall_at_5: float
|
||||
recall_at_10: float
|
||||
mrr: float
|
||||
ndcg_at_10: float
|
||||
latency_p50_ms: float
|
||||
latency_p95_ms: float
|
||||
dataset_id: str
|
||||
dataset_version: str
|
||||
index_profile_id: str
|
||||
passed: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AcceptanceThresholds:
|
||||
"""Explicit gates for controlled live acceptance."""
|
||||
|
||||
min_recall_at_1: float = DEFAULT_MIN_RECALL_AT_1
|
||||
min_recall_at_5: float = DEFAULT_MIN_RECALL_AT_5
|
||||
min_mrr: float = DEFAULT_MIN_MRR
|
||||
min_ndcg_at_10: float = DEFAULT_MIN_NDCG_AT_10
|
||||
max_p95_ms: float = DEFAULT_MAX_P95_MS
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
quality_values = (
|
||||
self.min_recall_at_1,
|
||||
self.min_recall_at_5,
|
||||
self.min_mrr,
|
||||
self.min_ndcg_at_10,
|
||||
)
|
||||
if any(not 0 <= value <= 1 for value in quality_values):
|
||||
raise ValueError("live quality thresholds must be between zero and one")
|
||||
if self.max_p95_ms <= 0:
|
||||
raise ValueError("live latency threshold must be positive")
|
||||
|
||||
|
||||
def _project_root() -> Path:
|
||||
return Path(__file__).resolve().parents[4]
|
||||
|
||||
|
||||
def _load_live_dataset(path: Path) -> GoldenDataset:
|
||||
"""Load the shared manifest so live and offline evaluation cannot drift."""
|
||||
try:
|
||||
dataset = load_golden_dataset(path)
|
||||
except GoldenDatasetError as error:
|
||||
raise LiveAcceptanceError("live Golden Dataset is invalid") from error
|
||||
if not MIN_GOLDEN_QUERIES <= len(dataset.cases) <= MAX_GOLDEN_QUERIES:
|
||||
raise LiveAcceptanceError("live Golden Dataset must contain 30 to 500 queries")
|
||||
return dataset
|
||||
|
||||
|
||||
def _ranked_sources(results: Sequence[SearchItem]) -> tuple[RankedSource, ...]:
|
||||
"""Project live canonical results onto the provider-neutral scoring schema."""
|
||||
if any(not isinstance(result, SourceResult) for result in results):
|
||||
raise LiveAcceptanceError("Raw provider acceptance received a Compiled result")
|
||||
return tuple(
|
||||
RankedSource(
|
||||
filename=result.filename,
|
||||
section=result.section,
|
||||
sheet_name=result.sheet_name,
|
||||
row_range=result.row_range,
|
||||
column_range=result.column_range,
|
||||
)
|
||||
for result in results
|
||||
if isinstance(result, SourceResult)
|
||||
)
|
||||
|
||||
|
||||
def _live_expected_source(source: GoldenSource, live_filename: str) -> GoldenSource:
|
||||
"""Apply the random run prefix to filename-derived CSV sheet identities."""
|
||||
sheet_name = source.sheet_name
|
||||
original_path = Path(source.filename)
|
||||
if original_path.suffix.casefold() == ".csv" and sheet_name == original_path.stem:
|
||||
sheet_name = Path(live_filename).stem
|
||||
return replace(source, filename=live_filename, sheet_name=sheet_name)
|
||||
|
||||
|
||||
def _build_report(
|
||||
results: Sequence[EvaluationCaseResult],
|
||||
thresholds: AcceptanceThresholds,
|
||||
*,
|
||||
dataset: GoldenDataset,
|
||||
index_profile_id: str,
|
||||
) -> AcceptanceReport:
|
||||
"""Calculate the shared metrics and apply explicit live acceptance gates."""
|
||||
metrics = evaluate_rankings(tuple(results))
|
||||
passed = (
|
||||
metrics.recall_at_1 >= thresholds.min_recall_at_1
|
||||
and metrics.recall_at_5 >= thresholds.min_recall_at_5
|
||||
and metrics.mrr >= thresholds.min_mrr
|
||||
and metrics.ndcg_at_10 >= thresholds.min_ndcg_at_10
|
||||
and metrics.latency_p95_ms <= thresholds.max_p95_ms
|
||||
)
|
||||
return _acceptance_report(metrics, dataset, index_profile_id, passed)
|
||||
|
||||
|
||||
def _acceptance_report(
|
||||
metrics: RetrievalMetrics,
|
||||
dataset: GoldenDataset,
|
||||
index_profile_id: str,
|
||||
passed: bool,
|
||||
) -> AcceptanceReport:
|
||||
"""Attach immutable evaluation identities to the shared metric values."""
|
||||
return AcceptanceReport(
|
||||
query_count=metrics.query_count,
|
||||
recall_at_1=metrics.recall_at_1,
|
||||
recall_at_3=metrics.recall_at_3,
|
||||
recall_at_5=metrics.recall_at_5,
|
||||
recall_at_10=metrics.recall_at_10,
|
||||
mrr=metrics.mrr,
|
||||
ndcg_at_10=metrics.ndcg_at_10,
|
||||
latency_p50_ms=metrics.latency_p50_ms,
|
||||
latency_p95_ms=metrics.latency_p95_ms,
|
||||
dataset_id=dataset.dataset_id,
|
||||
dataset_version=dataset.version,
|
||||
index_profile_id=index_profile_id,
|
||||
passed=passed,
|
||||
)
|
||||
|
||||
|
||||
def _thresholds_from_environment() -> AcceptanceThresholds:
|
||||
return AcceptanceThresholds(
|
||||
min_recall_at_1=float(os.getenv("EKP_LIVE_MIN_RECALL_AT_1", str(DEFAULT_MIN_RECALL_AT_1))),
|
||||
min_recall_at_5=float(os.getenv("EKP_LIVE_MIN_RECALL_AT_5", str(DEFAULT_MIN_RECALL_AT_5))),
|
||||
min_mrr=float(os.getenv("EKP_LIVE_MIN_MRR", str(DEFAULT_MIN_MRR))),
|
||||
min_ndcg_at_10=float(os.getenv("EKP_LIVE_MIN_NDCG_AT_10", str(DEFAULT_MIN_NDCG_AT_10))),
|
||||
max_p95_ms=float(os.getenv("EKP_LIVE_MAX_P95_MS", str(DEFAULT_MAX_P95_MS))),
|
||||
)
|
||||
|
||||
|
||||
def _require_safe_live_settings(settings: Settings) -> None:
|
||||
"""Refuse external writes unless rotation and resource authorization are explicit."""
|
||||
required_flags = (
|
||||
"EKP_RUN_LIVE_TESTS",
|
||||
"EKP_CREDENTIALS_AUTHORIZED",
|
||||
"EKP_LIVE_RESOURCE_ACK",
|
||||
)
|
||||
if any(os.getenv(name) != "1" for name in required_flags):
|
||||
raise LiveAcceptanceError("live acceptance safety flags are not enabled")
|
||||
if settings.app_env != "test":
|
||||
raise LiveAcceptanceError("live acceptance requires APP_ENV=test")
|
||||
if settings.object_storage_provider != "bos" or settings.vector_store_provider != "oceanbase":
|
||||
raise LiveAcceptanceError("live acceptance requires the target provider baseline")
|
||||
if not settings.bos_key_prefix.startswith("live-acceptance/"):
|
||||
raise LiveAcceptanceError("BOS_KEY_PREFIX must use a dedicated live-acceptance prefix")
|
||||
|
||||
|
||||
def _cleanup_run_documents(services: ApiServices, filename_prefix: str) -> None:
|
||||
"""Delete only documents created under this random acceptance-run prefix."""
|
||||
documents = tuple(
|
||||
document
|
||||
for document in services.documents.list_documents()
|
||||
if document.filename.startswith(filename_prefix)
|
||||
)
|
||||
for document in documents:
|
||||
services.documents.delete_document(document.id)
|
||||
|
||||
|
||||
def _run_live(settings: Settings) -> AcceptanceReport:
|
||||
"""Upload synthetic sources, execute every query, and clean run-owned data."""
|
||||
_require_safe_live_settings(settings)
|
||||
dataset = _load_live_dataset(_project_root() / "fixtures/evaluation/manifest.json")
|
||||
services = create_api_services(settings)
|
||||
filename_prefix = f"live-{uuid4().hex[:12]}-"
|
||||
uploaded_names: dict[str, str] = {}
|
||||
cleanup_error: Exception | None = None
|
||||
try:
|
||||
for corpus_file in dataset.corpus:
|
||||
live_name = f"{filename_prefix}{corpus_file.filename}"
|
||||
services.ingestion.ingest(
|
||||
live_name,
|
||||
corpus_file.media_type,
|
||||
BytesIO(corpus_file.path.read_bytes()),
|
||||
)
|
||||
uploaded_names[corpus_file.filename] = live_name
|
||||
|
||||
case_results: list[EvaluationCaseResult] = []
|
||||
index_profile_id: str | None = None
|
||||
for case in dataset.cases:
|
||||
started = perf_counter_ns()
|
||||
result = services.search.search(case.query, top_k=10)
|
||||
latency_ms = (perf_counter_ns() - started) / 1_000_000
|
||||
if index_profile_id is None:
|
||||
index_profile_id = result.index_profile_id
|
||||
elif result.index_profile_id != index_profile_id:
|
||||
raise LiveAcceptanceError("live search crossed the selected index profile")
|
||||
live_case = replace(
|
||||
case,
|
||||
expected_sources=tuple(
|
||||
_live_expected_source(source, uploaded_names[source.filename])
|
||||
for source in case.expected_sources
|
||||
),
|
||||
)
|
||||
case_results.append(
|
||||
EvaluationCaseResult(
|
||||
case=live_case,
|
||||
ranked_sources=_ranked_sources(result.results),
|
||||
latency_ms=latency_ms,
|
||||
)
|
||||
)
|
||||
if index_profile_id is None:
|
||||
raise LiveAcceptanceError("live evaluation returned no index profile")
|
||||
return _build_report(
|
||||
case_results,
|
||||
_thresholds_from_environment(),
|
||||
dataset=dataset,
|
||||
index_profile_id=index_profile_id,
|
||||
)
|
||||
finally:
|
||||
try:
|
||||
_cleanup_run_documents(services, filename_prefix)
|
||||
except Exception as error:
|
||||
cleanup_error = error
|
||||
if cleanup_error is not None:
|
||||
raise LiveAcceptanceError("live acceptance cleanup failed") from cleanup_error
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Print only aggregate metrics and return non-zero for unsafe or failed runs."""
|
||||
try:
|
||||
report = _run_live(Settings())
|
||||
except Exception:
|
||||
print("live acceptance failed; no provider details were emitted", file=sys.stderr)
|
||||
return 1
|
||||
print(json.dumps(asdict(report), sort_keys=True, separators=(",", ":")))
|
||||
return 0 if report.passed else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
156
hy4/_kp/apps/api/src/ekp_api/main.py
Normal file
156
hy4/_kp/apps/api/src/ekp_api/main.py
Normal file
@ -0,0 +1,156 @@
|
||||
"""FastAPI application entry point."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from collections.abc import Awaitable, Callable
|
||||
from pathlib import Path
|
||||
from time import monotonic
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI, Request, Response
|
||||
from fastapi.responses import FileResponse, PlainTextResponse, RedirectResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ekp_api.admin_routes import router as administration_router
|
||||
from ekp_api.compiled_routes import router as compiled_knowledge_router
|
||||
from ekp_api.metrics import HttpMetrics
|
||||
from ekp_api.providers import ApiServices
|
||||
from ekp_api.routes import router
|
||||
from ekp_core.config import Settings, get_settings
|
||||
from ekp_core.observability import bind_request_id, configure_json_logging, reset_request_id
|
||||
|
||||
ADMIN_WEB_ROOT = Path(__file__).resolve().parents[3] / "admin-web"
|
||||
REQUEST_ID_PATTERN = re.compile(r"^[A-Za-z0-9._:-]{1,64}$")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
"""Health endpoint response."""
|
||||
|
||||
status: str
|
||||
application: str
|
||||
environment: str
|
||||
migration_status: str | None = None
|
||||
migration_revision: str | None = None
|
||||
migration_error_type: str | None = None
|
||||
|
||||
|
||||
def create_app(
|
||||
settings: Settings | None = None,
|
||||
api_services: ApiServices | None = None,
|
||||
) -> FastAPI:
|
||||
"""Create the API application with explicit, testable settings."""
|
||||
resolved_settings = settings or get_settings()
|
||||
configure_json_logging(resolved_settings.log_level)
|
||||
application = FastAPI(
|
||||
title=resolved_settings.app_name,
|
||||
version="0.1.0",
|
||||
docs_url=f"{resolved_settings.api_prefix}/docs",
|
||||
openapi_url=f"{resolved_settings.api_prefix}/openapi.json",
|
||||
)
|
||||
application.state.settings = resolved_settings
|
||||
application.state.api_services = api_services
|
||||
application.state.http_metrics = HttpMetrics()
|
||||
application.include_router(router, prefix=resolved_settings.api_prefix)
|
||||
application.include_router(administration_router, prefix=resolved_settings.api_prefix)
|
||||
application.include_router(compiled_knowledge_router, prefix=resolved_settings.api_prefix)
|
||||
admin_web_root = (resolved_settings.admin_web_root or ADMIN_WEB_ROOT).resolve()
|
||||
|
||||
# The Demo console is a deliberately thin, same-origin client. It contains
|
||||
# no business logic and calls only the versioned public REST API.
|
||||
application.mount(
|
||||
"/admin/assets",
|
||||
StaticFiles(directory=admin_web_root / "assets"),
|
||||
name="admin-assets",
|
||||
)
|
||||
|
||||
@application.get("/", include_in_schema=False)
|
||||
async def admin_console_entry() -> RedirectResponse:
|
||||
"""Send the public web entry to the canonical Admin Console route."""
|
||||
return RedirectResponse(url="/admin", status_code=307)
|
||||
|
||||
@application.middleware("http")
|
||||
async def observe_http(
|
||||
request: Request,
|
||||
call_next: Callable[[Request], Awaitable[Response]],
|
||||
) -> Response:
|
||||
"""Correlate requests and emit bounded metrics without body/query data."""
|
||||
incoming_request_id = request.headers.get("x-request-id", "")
|
||||
request_id = (
|
||||
incoming_request_id
|
||||
if REQUEST_ID_PATTERN.fullmatch(incoming_request_id)
|
||||
else uuid4().hex
|
||||
)
|
||||
token = bind_request_id(request_id)
|
||||
started_at = monotonic()
|
||||
status_code = 500
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status_code = response.status_code
|
||||
response.headers["x-request-id"] = request_id
|
||||
return response
|
||||
finally:
|
||||
duration_seconds = monotonic() - started_at
|
||||
route = request.scope.get("route")
|
||||
route_path = getattr(route, "path", "unmatched")
|
||||
application.state.http_metrics.observe(
|
||||
request.method,
|
||||
route_path,
|
||||
status_code,
|
||||
duration_seconds,
|
||||
)
|
||||
logger.info(
|
||||
"HTTP request completed",
|
||||
extra={
|
||||
"event": "http_request_completed",
|
||||
"method": request.method,
|
||||
"path": route_path,
|
||||
"status_code": status_code,
|
||||
"duration_ms": round(duration_seconds * 1000, 3),
|
||||
},
|
||||
)
|
||||
reset_request_id(token)
|
||||
|
||||
@application.get("/admin", include_in_schema=False)
|
||||
@application.get("/admin/overview", include_in_schema=False)
|
||||
@application.get("/admin/upload", include_in_schema=False)
|
||||
@application.get("/admin/documents", include_in_schema=False)
|
||||
@application.get("/admin/compiled-knowledge", include_in_schema=False)
|
||||
@application.get("/admin/search", include_in_schema=False)
|
||||
@application.get("/admin/knowledge-bases", include_in_schema=False)
|
||||
@application.get("/admin/permissions", include_in_schema=False)
|
||||
async def admin_console() -> FileResponse:
|
||||
"""Serve the API-first Admin Console shell for every core client route."""
|
||||
return FileResponse(admin_web_root / "index.html")
|
||||
|
||||
@application.get(
|
||||
"/healthz",
|
||||
response_model=HealthResponse,
|
||||
response_model_exclude_none=True,
|
||||
tags=["system"],
|
||||
)
|
||||
async def health() -> HealthResponse:
|
||||
migration_status = os.getenv("EKP_DEV_MIGRATION_STATUS")
|
||||
return HealthResponse(
|
||||
status="degraded" if migration_status == "failed" else "ok",
|
||||
application=resolved_settings.app_name,
|
||||
environment=resolved_settings.app_env,
|
||||
migration_status=migration_status,
|
||||
migration_revision=os.getenv("EKP_DEV_MIGRATION_REVISION"),
|
||||
migration_error_type=os.getenv("EKP_DEV_MIGRATION_ERROR_TYPE"),
|
||||
)
|
||||
|
||||
@application.get("/metrics", include_in_schema=False)
|
||||
async def metrics() -> PlainTextResponse:
|
||||
"""Expose aggregate process/HTTP metrics without provider or user data."""
|
||||
return PlainTextResponse(
|
||||
application.state.http_metrics.render(),
|
||||
media_type="text/plain; version=0.0.4",
|
||||
)
|
||||
|
||||
return application
|
||||
|
||||
|
||||
app = create_app()
|
||||
59
hy4/_kp/apps/api/src/ekp_api/metrics.py
Normal file
59
hy4/_kp/apps/api/src/ekp_api/metrics.py
Normal file
@ -0,0 +1,59 @@
|
||||
"""Small dependency-free Prometheus endpoint for cluster smoke monitoring."""
|
||||
|
||||
from collections import defaultdict
|
||||
from threading import Lock
|
||||
from time import monotonic
|
||||
|
||||
|
||||
class HttpMetrics:
|
||||
"""Keep bounded per-process HTTP counters using route templates as labels."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._started_at = monotonic()
|
||||
self._lock = Lock()
|
||||
self._counts: defaultdict[tuple[str, str, int], int] = defaultdict(int)
|
||||
self._durations: defaultdict[tuple[str, str], tuple[int, float]] = defaultdict(
|
||||
lambda: (0, 0.0)
|
||||
)
|
||||
|
||||
def observe(self, method: str, path: str, status_code: int, duration_seconds: float) -> None:
|
||||
"""Record one request without retaining query strings or user content."""
|
||||
with self._lock:
|
||||
self._counts[(method, path, status_code)] += 1
|
||||
count, total = self._durations[(method, path)]
|
||||
self._durations[(method, path)] = (count + 1, total + duration_seconds)
|
||||
|
||||
def render(self) -> str:
|
||||
"""Return Prometheus text exposition with stable low-cardinality labels."""
|
||||
with self._lock:
|
||||
counts = tuple(sorted(self._counts.items()))
|
||||
durations = tuple(sorted(self._durations.items()))
|
||||
lines = [
|
||||
"# HELP ekp_process_uptime_seconds Process uptime in seconds.",
|
||||
"# TYPE ekp_process_uptime_seconds gauge",
|
||||
f"ekp_process_uptime_seconds {monotonic() - self._started_at:.6f}",
|
||||
"# HELP ekp_http_requests_total HTTP requests handled by this API process.",
|
||||
"# TYPE ekp_http_requests_total counter",
|
||||
]
|
||||
for (method, path, status_code), value in counts:
|
||||
labels = _labels(method=method, path=path, status=str(status_code))
|
||||
lines.append(f"ekp_http_requests_total{{{labels}}} {value}")
|
||||
lines.extend(
|
||||
(
|
||||
"# HELP ekp_http_request_duration_seconds HTTP request duration.",
|
||||
"# TYPE ekp_http_request_duration_seconds summary",
|
||||
)
|
||||
)
|
||||
for (method, path), (count, total) in durations:
|
||||
labels = _labels(method=method, path=path)
|
||||
lines.append(f"ekp_http_request_duration_seconds_count{{{labels}}} {count}")
|
||||
lines.append(f"ekp_http_request_duration_seconds_sum{{{labels}}} {total:.6f}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def _labels(**values: str) -> str:
|
||||
"""Escape the limited label set according to Prometheus text rules."""
|
||||
return ",".join(
|
||||
f'{name}="{value.replace(chr(92), chr(92) * 2).replace(chr(10), chr(92) + "n").replace(chr(34), chr(92) + chr(34))}"'
|
||||
for name, value in values.items()
|
||||
)
|
||||
250
hy4/_kp/apps/api/src/ekp_api/offline_evaluation.py
Normal file
250
hy4/_kp/apps/api/src/ekp_api/offline_evaluation.py
Normal file
@ -0,0 +1,250 @@
|
||||
"""Deterministic, provider-free RAG evaluation for the versioned Golden Dataset."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import asdict, dataclass
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
from time import perf_counter_ns
|
||||
from uuid import UUID
|
||||
|
||||
from sqlalchemy import create_engine, select
|
||||
|
||||
from ekp_core.chunking import StructureAwareChunker
|
||||
from ekp_core.database import create_session_factory
|
||||
from ekp_core.dense_retrieval import DenseRetrievalService
|
||||
from ekp_core.document_chunking import DocumentChunkingService, DocumentPipelineService
|
||||
from ekp_core.document_indexing import DocumentIndexingPipelineService, DocumentIndexingService
|
||||
from ekp_core.document_parsing import DocumentParsingService
|
||||
from ekp_core.document_upload import (
|
||||
DOCLING_DOCUMENT_EXTENSIONS,
|
||||
STRUCTURED_DOCUMENT_EXTENSIONS,
|
||||
DocumentUploadService,
|
||||
)
|
||||
from ekp_core.evaluation import (
|
||||
EvaluationCaseResult,
|
||||
GoldenDataset,
|
||||
GoldenKeywordEmbeddingProvider,
|
||||
RankedSource,
|
||||
RetrievalMetrics,
|
||||
evaluate_rankings,
|
||||
load_golden_dataset,
|
||||
)
|
||||
from ekp_core.hybrid_retrieval import HybridRetrievalService
|
||||
from ekp_core.search import SearchService, SourceResult
|
||||
from ekp_core.sparse_retrieval import SparseRetrievalService
|
||||
from ekp_domain import Base, Chunk, IndexProfile
|
||||
from ekp_object_storage import LocalObjectStorage
|
||||
from ekp_parser import DoclingParser, ParserRouter
|
||||
from ekp_reranker import FakeRerankerProvider
|
||||
from ekp_vector_store import FakeVectorStore
|
||||
|
||||
DEFAULT_MANIFEST_PATH = Path("fixtures/evaluation/manifest.json")
|
||||
OFFLINE_PROFILE_ID = "golden-structure-v2-offline-v1"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class EvaluationStages:
|
||||
"""Required metrics for every decision point in the retrieval pipeline."""
|
||||
|
||||
dense: RetrievalMetrics
|
||||
sparse: RetrievalMetrics
|
||||
hybrid: RetrievalMetrics
|
||||
reranked: RetrievalMetrics
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class OfflineEvaluationReport:
|
||||
"""Aggregate-only report safe to persist in CI logs."""
|
||||
|
||||
dataset_id: str
|
||||
dataset_version: str
|
||||
index_profile_id: str
|
||||
query_count: int
|
||||
stages: EvaluationStages
|
||||
|
||||
|
||||
def run_offline_evaluation(manifest_path: Path = DEFAULT_MANIFEST_PATH) -> OfflineEvaluationReport:
|
||||
"""Ingest the hashed fixture corpus and evaluate every retrieval stage."""
|
||||
dataset = load_golden_dataset(manifest_path)
|
||||
with TemporaryDirectory(prefix="ekp-evaluation-") as temporary_directory:
|
||||
return _run_with_storage(dataset, Path(temporary_directory))
|
||||
|
||||
|
||||
def _run_with_storage(dataset: GoldenDataset, storage_root: Path) -> OfflineEvaluationReport:
|
||||
"""Build one isolated canonical/index state for a complete evaluation run."""
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
session_factory = create_session_factory(engine)
|
||||
storage = LocalObjectStorage(storage_root)
|
||||
docling = DoclingParser()
|
||||
parser = ParserRouter({extension: docling for extension in DOCLING_DOCUMENT_EXTENSIONS})
|
||||
upload = DocumentUploadService(
|
||||
session_factory,
|
||||
storage,
|
||||
max_upload_bytes=50 * 1024 * 1024,
|
||||
allowed_extensions=STRUCTURED_DOCUMENT_EXTENSIONS,
|
||||
)
|
||||
embedding = GoldenKeywordEmbeddingProvider()
|
||||
vector_store = FakeVectorStore()
|
||||
profile = IndexProfile(
|
||||
OFFLINE_PROFILE_ID,
|
||||
"structure_v2",
|
||||
embedding.model_name,
|
||||
embedding.dimensions,
|
||||
hybrid_top_n=10,
|
||||
reranker_model="golden-reranker-v1",
|
||||
final_top_k=10,
|
||||
)
|
||||
pipeline = DocumentIndexingPipelineService(
|
||||
DocumentPipelineService(
|
||||
upload,
|
||||
DocumentParsingService(session_factory, storage, parser),
|
||||
DocumentChunkingService(session_factory, storage, StructureAwareChunker()),
|
||||
),
|
||||
DocumentIndexingService(session_factory, embedding, vector_store, profile),
|
||||
)
|
||||
for corpus_file in dataset.corpus:
|
||||
pipeline.ingest(
|
||||
corpus_file.filename,
|
||||
corpus_file.media_type,
|
||||
BytesIO(corpus_file.path.read_bytes()),
|
||||
)
|
||||
|
||||
dense = DenseRetrievalService(embedding, vector_store, profile)
|
||||
sparse = SparseRetrievalService(vector_store, profile)
|
||||
hybrid = HybridRetrievalService(
|
||||
dense,
|
||||
sparse,
|
||||
index_profile_id=profile.profile_id,
|
||||
default_top_n=10,
|
||||
)
|
||||
search = SearchService(
|
||||
session_factory,
|
||||
hybrid,
|
||||
FakeRerankerProvider(model_name="golden-reranker-v1"),
|
||||
profile,
|
||||
)
|
||||
with session_factory() as session:
|
||||
chunks = {chunk.chunk_id: chunk for chunk in session.scalars(select(Chunk)).all()}
|
||||
|
||||
dense_results: list[EvaluationCaseResult] = []
|
||||
sparse_results: list[EvaluationCaseResult] = []
|
||||
hybrid_results: list[EvaluationCaseResult] = []
|
||||
reranked_results: list[EvaluationCaseResult] = []
|
||||
for case in dataset.cases:
|
||||
started = perf_counter_ns()
|
||||
dense_result = dense.search(case.query, top_n=10)
|
||||
dense_results.append(
|
||||
EvaluationCaseResult(
|
||||
case,
|
||||
_ranked_sources(
|
||||
tuple(candidate.chunk_id for candidate in dense_result.candidates),
|
||||
chunks,
|
||||
),
|
||||
_elapsed_ms(started),
|
||||
)
|
||||
)
|
||||
|
||||
started = perf_counter_ns()
|
||||
sparse_result = sparse.search(case.query, top_n=10)
|
||||
sparse_results.append(
|
||||
EvaluationCaseResult(
|
||||
case,
|
||||
_ranked_sources(
|
||||
tuple(candidate.chunk_id for candidate in sparse_result.candidates),
|
||||
chunks,
|
||||
),
|
||||
_elapsed_ms(started),
|
||||
)
|
||||
)
|
||||
|
||||
started = perf_counter_ns()
|
||||
hybrid_result = hybrid.search(case.query, top_n=10)
|
||||
hybrid_results.append(
|
||||
EvaluationCaseResult(
|
||||
case,
|
||||
_ranked_sources(
|
||||
tuple(candidate.chunk_id for candidate in hybrid_result.candidates),
|
||||
chunks,
|
||||
),
|
||||
_elapsed_ms(started),
|
||||
)
|
||||
)
|
||||
|
||||
started = perf_counter_ns()
|
||||
search_result = search.search(case.query, top_k=10)
|
||||
if any(not isinstance(result, SourceResult) for result in search_result.results):
|
||||
raise RuntimeError("offline Raw evaluation received a Compiled result")
|
||||
reranked_results.append(
|
||||
EvaluationCaseResult(
|
||||
case,
|
||||
tuple(
|
||||
RankedSource(
|
||||
filename=result.filename,
|
||||
section=result.section,
|
||||
sheet_name=result.sheet_name,
|
||||
row_range=result.row_range,
|
||||
column_range=result.column_range,
|
||||
)
|
||||
for result in search_result.results
|
||||
if isinstance(result, SourceResult)
|
||||
),
|
||||
_elapsed_ms(started),
|
||||
)
|
||||
)
|
||||
|
||||
return OfflineEvaluationReport(
|
||||
dataset_id=dataset.dataset_id,
|
||||
dataset_version=dataset.version,
|
||||
index_profile_id=profile.profile_id,
|
||||
query_count=len(dataset.cases),
|
||||
stages=EvaluationStages(
|
||||
dense=evaluate_rankings(tuple(dense_results)),
|
||||
sparse=evaluate_rankings(tuple(sparse_results)),
|
||||
hybrid=evaluate_rankings(tuple(hybrid_results)),
|
||||
reranked=evaluate_rankings(tuple(reranked_results)),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _ranked_sources(
|
||||
chunk_ids: tuple[UUID, ...],
|
||||
chunks: dict[UUID, Chunk],
|
||||
) -> tuple[RankedSource, ...]:
|
||||
"""Enrich rebuildable index candidates with canonical location metadata."""
|
||||
return tuple(
|
||||
RankedSource(
|
||||
filename=chunks[chunk_id].filename,
|
||||
section=chunks[chunk_id].section,
|
||||
sheet_name=chunks[chunk_id].sheet_name,
|
||||
row_range=chunks[chunk_id].row_range,
|
||||
column_range=chunks[chunk_id].column_range,
|
||||
)
|
||||
for chunk_id in chunk_ids
|
||||
)
|
||||
|
||||
|
||||
def _elapsed_ms(started_ns: int) -> float:
|
||||
"""Convert monotonic elapsed time to the report's millisecond unit."""
|
||||
return (perf_counter_ns() - started_ns) / 1_000_000
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> None:
|
||||
"""Write a stable aggregate JSON report for local and CI evaluation runs."""
|
||||
parser = argparse.ArgumentParser(description="Run the offline EKP Golden Dataset")
|
||||
parser.add_argument(
|
||||
"--manifest",
|
||||
type=Path,
|
||||
default=DEFAULT_MANIFEST_PATH,
|
||||
help="path to the versioned Golden Dataset manifest",
|
||||
)
|
||||
arguments = parser.parse_args(argv)
|
||||
report = run_offline_evaluation(arguments.manifest)
|
||||
print(json.dumps(asdict(report), ensure_ascii=False, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
624
hy4/_kp/apps/api/src/ekp_api/providers.py
Normal file
624
hy4/_kp/apps/api/src/ekp_api/providers.py
Normal file
@ -0,0 +1,624 @@
|
||||
"""Runtime provider and application-service composition from safe settings."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
from ekp_api.celery_queue import (
|
||||
CeleryCompilationDispatcher,
|
||||
CeleryIngestionDispatcher,
|
||||
create_celery_app,
|
||||
)
|
||||
from ekp_core.administration import AdministrationService
|
||||
from ekp_core.archive_ingestion import ZipIngestionService
|
||||
from ekp_core.chunking import DEFAULT_CHUNK_STRATEGY_VERSION, StructureAwareChunker
|
||||
from ekp_core.compilation_jobs import (
|
||||
AsyncCompilationSubmissionService,
|
||||
CompilationJobRunner,
|
||||
CompilationJobService,
|
||||
)
|
||||
from ekp_core.compiled_knowledge import CompiledKnowledgeService
|
||||
from ekp_core.config import Settings
|
||||
from ekp_core.database import create_database_engine, create_session_factory
|
||||
from ekp_core.dense_retrieval import DenseRetrievalService
|
||||
from ekp_core.document_chunking import DocumentChunkingService, DocumentPipelineService
|
||||
from ekp_core.document_indexing import (
|
||||
DocumentIndexingPipelineService,
|
||||
DocumentIndexingService,
|
||||
)
|
||||
from ekp_core.document_management import DocumentManagementService
|
||||
from ekp_core.document_parsing import DocumentParsingService
|
||||
from ekp_core.document_upload import (
|
||||
DOCLING_DOCUMENT_EXTENSIONS,
|
||||
MINERU_DOCUMENT_EXTENSIONS,
|
||||
DocumentUploadService,
|
||||
)
|
||||
from ekp_core.hybrid_retrieval import HybridRetrievalService
|
||||
from ekp_core.index_rebuild import resolve_active_index_profile
|
||||
from ekp_core.ingestion_batches import IngestionBatchService
|
||||
from ekp_core.ingestion_jobs import (
|
||||
AsyncIngestionSubmissionService,
|
||||
DerivedDocumentCleanupService,
|
||||
IngestionJobRunner,
|
||||
IngestionJobService,
|
||||
)
|
||||
from ekp_core.knowledge_access import AuthorizedKnowledgeAccessService
|
||||
from ekp_core.permission_resolver import PermissionResolver
|
||||
from ekp_core.search import AuthorizedSearchService, SearchService
|
||||
from ekp_core.sparse_retrieval import SparseRetrievalService
|
||||
from ekp_domain import IndexProfile
|
||||
from ekp_domain.index_profile import DEFAULT_SPARSE_STRATEGY
|
||||
from ekp_embedding import (
|
||||
BGE_LARGE_ZH_DIMENSIONS,
|
||||
EmbeddingProvider,
|
||||
QianfanEmbeddingProvider,
|
||||
)
|
||||
from ekp_llm import LLMProvider, OpenAICompatibleLLMProvider
|
||||
from ekp_object_storage import ObjectStorage, create_bos_object_storage
|
||||
from ekp_object_storage.local import LocalObjectStorage
|
||||
from ekp_parser import MINERU_EXTENSIONS, DoclingParser, MinerUParser, Parser, ParserRouter
|
||||
from ekp_reranker import QianfanRerankerProvider, RerankerProvider
|
||||
from ekp_vector_store import (
|
||||
RetrievalStore,
|
||||
SparseStore,
|
||||
VectorStore,
|
||||
create_oceanbase_vector_store,
|
||||
validate_retrieval_store,
|
||||
)
|
||||
|
||||
DEMO_CHUNK_STRATEGY_VERSION = DEFAULT_CHUNK_STRATEGY_VERSION
|
||||
|
||||
|
||||
class ProviderConfigurationError(ValueError):
|
||||
"""Raised when a runtime provider lacks required injected configuration."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ApiServices:
|
||||
"""Business services injected into REST routes without provider leakage."""
|
||||
|
||||
ingestion: DocumentIndexingPipelineService
|
||||
submissions: AsyncIngestionSubmissionService
|
||||
batches: IngestionBatchService
|
||||
jobs: IngestionJobService
|
||||
documents: DocumentManagementService
|
||||
search: SearchService
|
||||
authorized_search: AuthorizedSearchService | None = None
|
||||
knowledge_access: AuthorizedKnowledgeAccessService | None = None
|
||||
permission_resolver: PermissionResolver | None = None
|
||||
administration: AdministrationService | None = None
|
||||
compiled_knowledge: CompiledKnowledgeService | None = None
|
||||
compilation_jobs: CompilationJobService | None = None
|
||||
compilation_submissions: AsyncCompilationSubmissionService | None = None
|
||||
archives: ZipIngestionService | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RuntimeSearchServices:
|
||||
"""Shared public and ACL-aware search graph for independent transports."""
|
||||
|
||||
search: SearchService
|
||||
authorized_search: AuthorizedSearchService
|
||||
knowledge_access: AuthorizedKnowledgeAccessService
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WorkerServices:
|
||||
"""Provider graph required by Celery without REST search-only services."""
|
||||
|
||||
runner: IngestionJobRunner
|
||||
jobs: IngestionJobService
|
||||
dispatcher: CeleryIngestionDispatcher
|
||||
compilation_runner: CompilationJobRunner
|
||||
compilation_jobs: CompilationJobService
|
||||
compilation_dispatcher: CeleryCompilationDispatcher
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _IngestionComponents:
|
||||
"""Shared composition result used by API and independent worker processes."""
|
||||
|
||||
session_factory: sessionmaker[Session]
|
||||
storage: ObjectStorage
|
||||
vector_store: RetrievalStore
|
||||
embedding_provider: EmbeddingProvider
|
||||
index_profile: IndexProfile
|
||||
upload: DocumentUploadService
|
||||
parsing: DocumentParsingService
|
||||
chunking: DocumentChunkingService
|
||||
indexing: DocumentIndexingService
|
||||
jobs: IngestionJobService
|
||||
dispatcher: CeleryIngestionDispatcher
|
||||
compilation_dispatcher: CeleryCompilationDispatcher
|
||||
|
||||
|
||||
def create_index_profile(
|
||||
settings: Settings,
|
||||
session_factory: sessionmaker[Session] | None = None,
|
||||
) -> IndexProfile:
|
||||
"""Resolve the canonical active profile or the fresh-deployment baseline."""
|
||||
configured = IndexProfile(
|
||||
profile_id=settings.index_profile_id,
|
||||
chunk_strategy_version=DEMO_CHUNK_STRATEGY_VERSION,
|
||||
embedding_model=settings.embedding_model,
|
||||
vector_dimension=BGE_LARGE_ZH_DIMENSIONS,
|
||||
chunk_max_tokens=settings.chunk_max_tokens,
|
||||
chunk_overlap_tokens=settings.chunk_overlap_tokens,
|
||||
chunk_max_characters=settings.chunk_max_characters,
|
||||
dense_top_n=settings.dense_top_n,
|
||||
sparse_top_n=settings.sparse_top_n,
|
||||
hybrid_top_n=settings.hybrid_top_n,
|
||||
hybrid_rrf_k=settings.hybrid_rrf_k,
|
||||
hybrid_dense_weight=settings.hybrid_dense_weight,
|
||||
hybrid_sparse_weight=settings.hybrid_sparse_weight,
|
||||
reranker_model=settings.reranker_model,
|
||||
final_top_k=settings.search_top_k,
|
||||
)
|
||||
if session_factory is None:
|
||||
return configured
|
||||
return resolve_active_index_profile(session_factory, configured)
|
||||
|
||||
|
||||
def create_embedding_provider(settings: Settings) -> QianfanEmbeddingProvider:
|
||||
"""Create Qianfan only when its API key is supplied by the runtime."""
|
||||
if settings.baidu_qianfan_api_key is None:
|
||||
raise ProviderConfigurationError("BAIDU_QIANFAN_API_KEY is required")
|
||||
return QianfanEmbeddingProvider(
|
||||
api_key=settings.baidu_qianfan_api_key.get_secret_value(),
|
||||
endpoint=settings.embedding_endpoint,
|
||||
timeout_seconds=settings.embedding_timeout_seconds,
|
||||
)
|
||||
|
||||
|
||||
def create_object_storage(settings: Settings) -> ObjectStorage:
|
||||
"""Create local or BOS storage while keeping credentials out of core services."""
|
||||
if settings.object_storage_provider == "local":
|
||||
return LocalObjectStorage(settings.object_storage_path)
|
||||
if settings.bos_endpoint is None:
|
||||
raise ProviderConfigurationError("BOS_ENDPOINT is required")
|
||||
if settings.bos_access_key_id is None:
|
||||
raise ProviderConfigurationError("BOS_ACCESS_KEY_ID is required")
|
||||
if settings.bos_secret_access_key is None:
|
||||
raise ProviderConfigurationError("BOS_SECRET_ACCESS_KEY is required")
|
||||
if settings.bos_bucket is None:
|
||||
raise ProviderConfigurationError("BOS_BUCKET is required")
|
||||
return create_bos_object_storage(
|
||||
endpoint=settings.bos_endpoint,
|
||||
access_key_id=settings.bos_access_key_id.get_secret_value(),
|
||||
secret_access_key=settings.bos_secret_access_key.get_secret_value(),
|
||||
bucket_name=settings.bos_bucket,
|
||||
key_prefix=settings.bos_key_prefix,
|
||||
timeout_seconds=settings.bos_timeout_seconds,
|
||||
)
|
||||
|
||||
|
||||
def create_mineru_parser(settings: Settings) -> MinerUParser:
|
||||
"""Build the optional MinerU adapter from validated runtime settings."""
|
||||
if settings.mineru_endpoint is None:
|
||||
raise ProviderConfigurationError("MINERU_ENDPOINT is required")
|
||||
api_token = (
|
||||
settings.mineru_api_token.get_secret_value()
|
||||
if settings.mineru_api_token is not None
|
||||
else None
|
||||
)
|
||||
return MinerUParser(
|
||||
endpoint=str(settings.mineru_endpoint),
|
||||
backend=settings.mineru_backend,
|
||||
parse_method=settings.mineru_parse_method,
|
||||
language=settings.mineru_language,
|
||||
effort=settings.mineru_effort,
|
||||
api_token=api_token,
|
||||
timeout_seconds=settings.mineru_timeout_seconds,
|
||||
max_source_bytes=settings.max_upload_bytes,
|
||||
low_ocr_confidence_threshold=settings.mineru_low_ocr_confidence_threshold,
|
||||
)
|
||||
|
||||
|
||||
def create_docling_parser(settings: Settings) -> DoclingParser:
|
||||
"""Build the bounded local Docling adapter used by structured v2 ingestion."""
|
||||
return DoclingParser(max_source_bytes=settings.max_upload_bytes)
|
||||
|
||||
|
||||
def create_parser_router(
|
||||
settings: Settings,
|
||||
mineru_parser: Parser | None = None,
|
||||
docling_parser: Parser | None = None,
|
||||
*,
|
||||
enable_structured: bool = True,
|
||||
) -> ParserRouter:
|
||||
"""Route OCR and structured formats while retaining lightweight text parsing."""
|
||||
routes: dict[str, Parser] = {}
|
||||
if enable_structured:
|
||||
selected_docling = docling_parser or create_docling_parser(settings)
|
||||
routes.update({extension: selected_docling for extension in DOCLING_DOCUMENT_EXTENSIONS})
|
||||
if settings.parser_provider == "mineru":
|
||||
selected_mineru = mineru_parser or create_mineru_parser(settings)
|
||||
routes.update(
|
||||
{
|
||||
extension: selected_mineru
|
||||
for extension in MINERU_EXTENSIONS
|
||||
if extension in MINERU_DOCUMENT_EXTENSIONS
|
||||
}
|
||||
)
|
||||
return ParserRouter(routes)
|
||||
|
||||
|
||||
def create_vector_store(
|
||||
settings: Settings,
|
||||
index_profile: IndexProfile | None = None,
|
||||
) -> RetrievalStore:
|
||||
"""Create the OceanBase adapter from explicit, secret-safe settings."""
|
||||
if settings.oceanbase_vector_uri is None:
|
||||
raise ProviderConfigurationError("OCEANBASE_VECTOR_URI is required")
|
||||
if settings.oceanbase_vector_user is None:
|
||||
raise ProviderConfigurationError("OCEANBASE_VECTOR_USER is required")
|
||||
if settings.oceanbase_vector_password is None:
|
||||
raise ProviderConfigurationError("OCEANBASE_VECTOR_PASSWORD is required")
|
||||
if settings.oceanbase_vector_database is None:
|
||||
raise ProviderConfigurationError("OCEANBASE_VECTOR_DATABASE is required")
|
||||
return validate_retrieval_store(
|
||||
create_oceanbase_vector_store(
|
||||
uri=settings.oceanbase_vector_uri,
|
||||
user=settings.oceanbase_vector_user,
|
||||
password=settings.oceanbase_vector_password.get_secret_value(),
|
||||
database_name=settings.oceanbase_vector_database,
|
||||
table_name=settings.oceanbase_vector_table,
|
||||
index_name=settings.oceanbase_vector_index,
|
||||
vector_dimension=(
|
||||
BGE_LARGE_ZH_DIMENSIONS if index_profile is None else index_profile.vector_dimension
|
||||
),
|
||||
batch_size=settings.oceanbase_vector_batch_size,
|
||||
timeout_seconds=settings.oceanbase_vector_timeout_seconds,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def create_dense_retrieval_service(
|
||||
embedding_provider: EmbeddingProvider,
|
||||
vector_store: VectorStore,
|
||||
index_profile: IndexProfile,
|
||||
) -> DenseRetrievalService:
|
||||
"""Compose the profile-scoped dense branch used by hybrid retrieval."""
|
||||
return DenseRetrievalService(
|
||||
embedding_provider=embedding_provider,
|
||||
vector_store=vector_store,
|
||||
index_profile=index_profile,
|
||||
default_top_n=index_profile.dense_top_n,
|
||||
)
|
||||
|
||||
|
||||
def create_sparse_retrieval_service(
|
||||
sparse_store: SparseStore,
|
||||
index_profile: IndexProfile,
|
||||
) -> SparseRetrievalService:
|
||||
"""Compose the profile-scoped sparse branch used by hybrid retrieval."""
|
||||
if index_profile.sparse_strategy != DEFAULT_SPARSE_STRATEGY:
|
||||
raise ProviderConfigurationError("active index profile uses an unsupported sparse strategy")
|
||||
return SparseRetrievalService(
|
||||
sparse_store=sparse_store,
|
||||
index_profile=index_profile,
|
||||
default_top_n=index_profile.sparse_top_n,
|
||||
)
|
||||
|
||||
|
||||
def create_hybrid_retrieval_service(
|
||||
dense_retriever: DenseRetrievalService,
|
||||
sparse_retriever: SparseRetrievalService,
|
||||
index_profile: IndexProfile,
|
||||
) -> HybridRetrievalService:
|
||||
"""Compose weighted rank fusion without leaking provider-native scores."""
|
||||
return HybridRetrievalService(
|
||||
dense_retriever,
|
||||
sparse_retriever,
|
||||
index_profile_id=index_profile.profile_id,
|
||||
default_top_n=index_profile.hybrid_top_n,
|
||||
rrf_k=index_profile.hybrid_rrf_k,
|
||||
dense_weight=index_profile.hybrid_dense_weight,
|
||||
sparse_weight=index_profile.hybrid_sparse_weight,
|
||||
)
|
||||
|
||||
|
||||
def create_reranker_provider(
|
||||
settings: Settings,
|
||||
index_profile: IndexProfile,
|
||||
) -> QianfanRerankerProvider:
|
||||
"""Create the Demo BCE reranker using an explicit or shared Qianfan key."""
|
||||
secret = settings.reranker_api_key or settings.baidu_qianfan_api_key
|
||||
if secret is None:
|
||||
raise ProviderConfigurationError("RERANKER_API_KEY or BAIDU_QIANFAN_API_KEY is required")
|
||||
return QianfanRerankerProvider(
|
||||
api_key=secret.get_secret_value(),
|
||||
endpoint=settings.reranker_endpoint,
|
||||
model_name=index_profile.reranker_model,
|
||||
timeout_seconds=settings.reranker_timeout_seconds,
|
||||
)
|
||||
|
||||
|
||||
def create_llm_provider(settings: Settings) -> LLMProvider | None:
|
||||
"""Create optional knowledge-compilation generation without blocking manual drafts."""
|
||||
if settings.llm_provider == "disabled":
|
||||
return None
|
||||
if settings.llm_endpoint is None or settings.llm_api_key is None or settings.llm_model is None:
|
||||
raise ProviderConfigurationError("LLM provider configuration is incomplete")
|
||||
try:
|
||||
return OpenAICompatibleLLMProvider(
|
||||
endpoint=settings.llm_endpoint,
|
||||
api_key=settings.llm_api_key.get_secret_value(),
|
||||
model_name=settings.llm_model,
|
||||
processing_location=settings.llm_processing_location,
|
||||
timeout_seconds=settings.llm_timeout_seconds,
|
||||
)
|
||||
except ValueError as error:
|
||||
raise ProviderConfigurationError("LLM provider configuration is invalid") from error
|
||||
|
||||
|
||||
def create_search_service(
|
||||
session_factory: sessionmaker[Session],
|
||||
hybrid_retriever: HybridRetrievalService,
|
||||
reranker_provider: RerankerProvider,
|
||||
index_profile: IndexProfile,
|
||||
) -> SearchService:
|
||||
"""Compose reranked hybrid search independently from REST schemas."""
|
||||
return SearchService(
|
||||
session_factory=session_factory,
|
||||
hybrid_retriever=hybrid_retriever,
|
||||
reranker_provider=reranker_provider,
|
||||
index_profile=index_profile,
|
||||
)
|
||||
|
||||
|
||||
def create_runtime_search_services(settings: Settings) -> RuntimeSearchServices:
|
||||
"""Compose one read graph with both compatibility and ACL-aware boundaries."""
|
||||
engine = create_database_engine(settings)
|
||||
session_factory = create_session_factory(engine)
|
||||
index_profile = create_index_profile(settings, session_factory)
|
||||
embedding_provider = create_embedding_provider(settings)
|
||||
vector_store = create_vector_store(settings, index_profile)
|
||||
dense_retrieval = create_dense_retrieval_service(
|
||||
embedding_provider,
|
||||
vector_store,
|
||||
index_profile,
|
||||
)
|
||||
sparse_retrieval = create_sparse_retrieval_service(
|
||||
vector_store,
|
||||
index_profile,
|
||||
)
|
||||
hybrid_retrieval = create_hybrid_retrieval_service(
|
||||
dense_retrieval,
|
||||
sparse_retrieval,
|
||||
index_profile,
|
||||
)
|
||||
search = create_search_service(
|
||||
session_factory,
|
||||
hybrid_retrieval,
|
||||
create_reranker_provider(settings, index_profile),
|
||||
index_profile,
|
||||
)
|
||||
permission_resolver = PermissionResolver(session_factory)
|
||||
return RuntimeSearchServices(
|
||||
search=search,
|
||||
authorized_search=AuthorizedSearchService(
|
||||
permission_resolver,
|
||||
search,
|
||||
),
|
||||
knowledge_access=AuthorizedKnowledgeAccessService(
|
||||
session_factory,
|
||||
permission_resolver,
|
||||
active_index_profile_id=index_profile.profile_id,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def create_runtime_search_service(settings: Settings) -> SearchService:
|
||||
"""Preserve the public SearchService factory for existing callers."""
|
||||
return create_runtime_search_services(settings).search
|
||||
|
||||
|
||||
def _create_ingestion_components(settings: Settings) -> _IngestionComponents:
|
||||
"""Compose provider-neutral ingestion stages and the Celery delivery adapter."""
|
||||
if settings.rabbitmq_url is None or not settings.rabbitmq_url.get_secret_value().strip():
|
||||
raise ProviderConfigurationError("RABBITMQ_URL is required")
|
||||
engine = create_database_engine(settings)
|
||||
session_factory = create_session_factory(engine)
|
||||
storage = create_object_storage(settings)
|
||||
index_profile = create_index_profile(settings, session_factory)
|
||||
embedding_provider = create_embedding_provider(settings)
|
||||
vector_store = create_vector_store(settings, index_profile)
|
||||
structured_chunks_enabled = (
|
||||
index_profile.chunk_strategy_version == DEFAULT_CHUNK_STRATEGY_VERSION
|
||||
)
|
||||
parser_router = create_parser_router(
|
||||
settings,
|
||||
enable_structured=structured_chunks_enabled,
|
||||
)
|
||||
|
||||
upload_service = DocumentUploadService(
|
||||
session_factory,
|
||||
storage,
|
||||
max_upload_bytes=settings.max_upload_bytes,
|
||||
# Derive admission from the exact runtime router so upload and Worker
|
||||
# capabilities cannot drift as providers are added or replaced.
|
||||
allowed_extensions=parser_router.supported_extensions,
|
||||
)
|
||||
parsing_service = DocumentParsingService(
|
||||
session_factory,
|
||||
storage,
|
||||
parser_router,
|
||||
)
|
||||
chunking_service = DocumentChunkingService(
|
||||
session_factory,
|
||||
storage,
|
||||
StructureAwareChunker(
|
||||
max_tokens=index_profile.chunk_max_tokens,
|
||||
overlap_tokens=index_profile.chunk_overlap_tokens,
|
||||
max_characters=index_profile.chunk_max_characters,
|
||||
strategy_version=index_profile.chunk_strategy_version,
|
||||
),
|
||||
)
|
||||
indexing_service = DocumentIndexingService(
|
||||
session_factory,
|
||||
embedding_provider,
|
||||
vector_store,
|
||||
index_profile,
|
||||
)
|
||||
jobs = IngestionJobService(session_factory, settings.ingestion_max_attempts)
|
||||
celery_app = create_celery_app(settings)
|
||||
dispatcher = CeleryIngestionDispatcher(celery_app)
|
||||
compilation_dispatcher = CeleryCompilationDispatcher(celery_app)
|
||||
return _IngestionComponents(
|
||||
session_factory=session_factory,
|
||||
storage=storage,
|
||||
vector_store=vector_store,
|
||||
embedding_provider=embedding_provider,
|
||||
index_profile=index_profile,
|
||||
upload=upload_service,
|
||||
parsing=parsing_service,
|
||||
chunking=chunking_service,
|
||||
indexing=indexing_service,
|
||||
jobs=jobs,
|
||||
dispatcher=dispatcher,
|
||||
compilation_dispatcher=compilation_dispatcher,
|
||||
)
|
||||
|
||||
|
||||
def _create_runner(settings: Settings, components: _IngestionComponents) -> IngestionJobRunner:
|
||||
"""Compose retry cleanup and execution policy from non-sensitive settings."""
|
||||
cleanup = DerivedDocumentCleanupService(
|
||||
components.session_factory,
|
||||
components.storage,
|
||||
components.vector_store,
|
||||
components.index_profile.profile_id,
|
||||
)
|
||||
return IngestionJobRunner(
|
||||
components.session_factory,
|
||||
components.parsing,
|
||||
components.chunking,
|
||||
components.indexing,
|
||||
cleanup,
|
||||
retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
|
||||
retry_backoff_max_seconds=settings.ingestion_retry_backoff_max_seconds,
|
||||
lease_seconds=settings.ingestion_worker_lease_seconds,
|
||||
)
|
||||
|
||||
|
||||
def create_worker_services(settings: Settings) -> WorkerServices:
|
||||
"""Build the independent worker runtime from the same application boundaries."""
|
||||
components = _create_ingestion_components(settings)
|
||||
permission_resolver = PermissionResolver(components.session_factory)
|
||||
compiler = CompiledKnowledgeService(
|
||||
components.session_factory,
|
||||
create_llm_provider(settings),
|
||||
)
|
||||
compilation_jobs = CompilationJobService(
|
||||
components.session_factory,
|
||||
settings.compilation_max_attempts,
|
||||
)
|
||||
return WorkerServices(
|
||||
runner=_create_runner(settings, components),
|
||||
jobs=components.jobs,
|
||||
dispatcher=components.dispatcher,
|
||||
compilation_runner=CompilationJobRunner(
|
||||
components.session_factory,
|
||||
compiler,
|
||||
permission_resolver,
|
||||
retry_backoff_seconds=settings.compilation_retry_backoff_seconds,
|
||||
retry_backoff_max_seconds=settings.compilation_retry_backoff_max_seconds,
|
||||
lease_seconds=settings.compilation_worker_lease_seconds,
|
||||
),
|
||||
compilation_jobs=compilation_jobs,
|
||||
compilation_dispatcher=components.compilation_dispatcher,
|
||||
)
|
||||
|
||||
|
||||
def create_api_services(settings: Settings) -> ApiServices:
|
||||
"""Compose REST services while ingestion execution remains in workers."""
|
||||
components = _create_ingestion_components(settings)
|
||||
document_pipeline = DocumentPipelineService(
|
||||
components.upload,
|
||||
components.parsing,
|
||||
components.chunking,
|
||||
)
|
||||
ingestion = DocumentIndexingPipelineService(document_pipeline, components.indexing)
|
||||
submissions = AsyncIngestionSubmissionService(
|
||||
components.upload,
|
||||
components.jobs,
|
||||
components.dispatcher,
|
||||
)
|
||||
batches = IngestionBatchService(
|
||||
components.session_factory,
|
||||
submissions,
|
||||
max_files=settings.batch_max_files,
|
||||
max_total_bytes=settings.batch_max_total_bytes,
|
||||
max_archive_files=settings.zip_max_files,
|
||||
)
|
||||
archives = ZipIngestionService(
|
||||
batches,
|
||||
max_archive_bytes=settings.max_upload_bytes,
|
||||
max_files=settings.zip_max_files,
|
||||
max_entry_bytes=settings.max_upload_bytes,
|
||||
max_total_bytes=settings.batch_max_total_bytes,
|
||||
max_compression_ratio=settings.zip_max_compression_ratio,
|
||||
)
|
||||
dense_retrieval = create_dense_retrieval_service(
|
||||
components.embedding_provider,
|
||||
components.vector_store,
|
||||
components.index_profile,
|
||||
)
|
||||
sparse_retrieval = create_sparse_retrieval_service(
|
||||
components.vector_store,
|
||||
components.index_profile,
|
||||
)
|
||||
hybrid_retrieval = create_hybrid_retrieval_service(
|
||||
dense_retrieval,
|
||||
sparse_retrieval,
|
||||
components.index_profile,
|
||||
)
|
||||
reranker_provider = create_reranker_provider(settings, components.index_profile)
|
||||
search = create_search_service(
|
||||
components.session_factory,
|
||||
hybrid_retrieval,
|
||||
reranker_provider,
|
||||
components.index_profile,
|
||||
)
|
||||
documents = DocumentManagementService(
|
||||
components.session_factory,
|
||||
components.storage,
|
||||
components.vector_store,
|
||||
active_index_profile_id=components.index_profile.profile_id,
|
||||
)
|
||||
permission_resolver = PermissionResolver(components.session_factory)
|
||||
compiled_knowledge = CompiledKnowledgeService(
|
||||
components.session_factory,
|
||||
create_llm_provider(settings),
|
||||
)
|
||||
compilation_jobs = CompilationJobService(
|
||||
components.session_factory,
|
||||
settings.compilation_max_attempts,
|
||||
)
|
||||
return ApiServices(
|
||||
ingestion=ingestion,
|
||||
submissions=submissions,
|
||||
batches=batches,
|
||||
jobs=components.jobs,
|
||||
documents=documents,
|
||||
search=search,
|
||||
authorized_search=AuthorizedSearchService(
|
||||
permission_resolver,
|
||||
search,
|
||||
),
|
||||
knowledge_access=AuthorizedKnowledgeAccessService(
|
||||
components.session_factory,
|
||||
permission_resolver,
|
||||
active_index_profile_id=components.index_profile.profile_id,
|
||||
),
|
||||
permission_resolver=permission_resolver,
|
||||
administration=AdministrationService(components.session_factory),
|
||||
compiled_knowledge=compiled_knowledge,
|
||||
compilation_jobs=compilation_jobs,
|
||||
compilation_submissions=AsyncCompilationSubmissionService(
|
||||
compiled_knowledge,
|
||||
compilation_jobs,
|
||||
components.compilation_dispatcher,
|
||||
),
|
||||
archives=archives,
|
||||
)
|
||||
838
hy4/_kp/apps/api/src/ekp_api/routes.py
Normal file
838
hy4/_kp/apps/api/src/ekp_api/routes.py
Normal file
@ -0,0 +1,838 @@
|
||||
"""Versioned document and search routes for the retrieval Demo."""
|
||||
|
||||
import logging
|
||||
from collections.abc import Iterator
|
||||
from typing import Annotated, BinaryIO
|
||||
from urllib.parse import quote
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
Depends,
|
||||
File,
|
||||
Form,
|
||||
HTTPException,
|
||||
Query,
|
||||
Request,
|
||||
Response,
|
||||
UploadFile,
|
||||
status,
|
||||
)
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from ekp_api.auth import ApiIdentity, get_api_identity, require_canonical_identity, require_scope
|
||||
from ekp_api.providers import ApiServices, ProviderConfigurationError, create_api_services
|
||||
from ekp_api.schemas import (
|
||||
DocumentContentResponse,
|
||||
DocumentResponse,
|
||||
FolderResponse,
|
||||
IngestionAcceptedResponse,
|
||||
IngestionBatchResponse,
|
||||
IngestionJobResponse,
|
||||
KnowledgeBaseResponse,
|
||||
SearchDiagnosticsResponse,
|
||||
SearchRequest,
|
||||
SearchResponse,
|
||||
)
|
||||
from ekp_core.administration import AdministrationNotFoundError
|
||||
from ekp_core.archive_ingestion import (
|
||||
EmptyZipArchiveError,
|
||||
InvalidZipArchiveError,
|
||||
UnsafeZipArchiveError,
|
||||
UnsupportedZipArchiveError,
|
||||
ZipArchiveLimitError,
|
||||
)
|
||||
from ekp_core.dense_retrieval import InvalidRetrievalQueryError
|
||||
from ekp_core.document_management import (
|
||||
DocumentArtifactCleanupError,
|
||||
DocumentFolderScope,
|
||||
DocumentIngestionActiveError,
|
||||
DocumentNotFoundError,
|
||||
DocumentScopeNotFoundError,
|
||||
DocumentSourceUnavailableError,
|
||||
DocumentVersionNotFoundError,
|
||||
)
|
||||
from ekp_core.document_upload import (
|
||||
DocumentTooLargeError,
|
||||
DocumentUploadError,
|
||||
DuplicateDocumentSourceError,
|
||||
InvalidDocumentScopeError,
|
||||
MediaTypeMismatchError,
|
||||
UnsupportedDocumentTypeError,
|
||||
)
|
||||
from ekp_core.ingestion_batches import (
|
||||
BatchDirectoryScopeError,
|
||||
BatchFileCountLimitError,
|
||||
BatchFolderScopeError,
|
||||
BatchNotFoundError,
|
||||
BatchSource,
|
||||
)
|
||||
from ekp_core.ingestion_jobs import IngestionJobNotFoundError
|
||||
from ekp_core.permission_resolver import PermissionIdentityError
|
||||
from ekp_core.search import SearchConfigurationError
|
||||
from ekp_domain import AclPermission
|
||||
from ekp_vector_store import OceanBaseConfigurationError, OceanBaseVectorStoreError
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_api_services(request: Request) -> ApiServices:
|
||||
"""Resolve provider composition lazily so health checks need no cloud credentials."""
|
||||
configured = getattr(request.app.state, "api_services", None)
|
||||
if isinstance(configured, ApiServices):
|
||||
return configured
|
||||
try:
|
||||
services = create_api_services(request.app.state.settings)
|
||||
except ProviderConfigurationError as error:
|
||||
logger.error(
|
||||
"API provider composition failed",
|
||||
extra={"event": "api_provider_configuration_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="API providers are not fully configured",
|
||||
) from error
|
||||
except SQLAlchemyError as error:
|
||||
# Return only the stable dependency boundary. SQLAlchemy messages may
|
||||
# contain SQL or endpoints and therefore remain out of HTTP responses.
|
||||
logger.error(
|
||||
"API dependency initialization failed",
|
||||
extra={
|
||||
"event": "api_dependency_initialization_failed",
|
||||
"dependency": "canonical_database",
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="API canonical database is unavailable",
|
||||
headers={"X-EKP-Dependency": "canonical-database"},
|
||||
) from error
|
||||
except (OceanBaseConfigurationError, OceanBaseVectorStoreError) as error:
|
||||
logger.error(
|
||||
"API dependency initialization failed",
|
||||
extra={
|
||||
"event": "api_dependency_initialization_failed",
|
||||
"dependency": "vector_index",
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="API vector index is unavailable",
|
||||
headers={"X-EKP-Dependency": "vector-index"},
|
||||
) from error
|
||||
request.app.state.api_services = services
|
||||
return services
|
||||
|
||||
|
||||
Services = Annotated[ApiServices, Depends(get_api_services)]
|
||||
Identity = Annotated[ApiIdentity, Depends(get_api_identity)]
|
||||
|
||||
|
||||
@router.get(
|
||||
"/knowledge-bases",
|
||||
response_model=list[KnowledgeBaseResponse],
|
||||
tags=["knowledge-bases"],
|
||||
)
|
||||
def list_accessible_knowledge_bases(
|
||||
services: Services, identity: Identity
|
||||
) -> list[KnowledgeBaseResponse]:
|
||||
"""List only active Knowledge Bases readable by the current canonical User."""
|
||||
require_scope(identity, "knowledge:read")
|
||||
if not identity.is_canonical:
|
||||
return []
|
||||
allowed = frozenset(_resolved_knowledge_base_ids(identity, services, AclPermission.READ))
|
||||
if services.administration is None:
|
||||
raise HTTPException(status_code=503, detail="administration service is unavailable")
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
return [
|
||||
KnowledgeBaseResponse.from_knowledge_base(value)
|
||||
for value in services.administration.list_knowledge_bases(organization_id, user_id)
|
||||
if value.id in allowed
|
||||
]
|
||||
|
||||
|
||||
@router.get(
|
||||
"/knowledge-bases/{knowledge_base_id}/folders",
|
||||
response_model=list[FolderResponse],
|
||||
tags=["knowledge-bases"],
|
||||
)
|
||||
def list_accessible_folders(
|
||||
knowledge_base_id: UUID, services: Services, identity: Identity
|
||||
) -> list[FolderResponse]:
|
||||
"""List folder metadata only after resolving READ access to its Knowledge Base."""
|
||||
require_scope(identity, "knowledge:read")
|
||||
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
|
||||
if knowledge_base_id not in allowed:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found")
|
||||
if services.administration is None:
|
||||
raise HTTPException(status_code=503, detail="administration service is unavailable")
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
try:
|
||||
values = services.administration.list_folders(organization_id, user_id, knowledge_base_id)
|
||||
except AdministrationNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found") from error
|
||||
return [FolderResponse.from_folder(value) for value in values]
|
||||
|
||||
|
||||
@router.post(
|
||||
"/documents",
|
||||
response_model=IngestionAcceptedResponse,
|
||||
status_code=status.HTTP_202_ACCEPTED,
|
||||
tags=["documents"],
|
||||
)
|
||||
def upload_document(
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
file: Annotated[
|
||||
UploadFile,
|
||||
File(description="One configured P0 document, spreadsheet, HTML, or image file"),
|
||||
],
|
||||
knowledge_base_id: Annotated[UUID | None, Form()] = None,
|
||||
folder_id: Annotated[UUID | None, Form()] = None,
|
||||
) -> IngestionAcceptedResponse:
|
||||
"""Persist one immutable source and durable Job, then return immediately."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:write")
|
||||
organization_id = _authorize_upload_scope(
|
||||
identity, services, knowledge_base_id, AclPermission.WRITE
|
||||
)
|
||||
submission = services.submissions.submit(
|
||||
file.filename or "",
|
||||
file.content_type,
|
||||
file.file,
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=knowledge_base_id,
|
||||
folder_id=folder_id,
|
||||
)
|
||||
except DuplicateDocumentSourceError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail={
|
||||
"code": "document_source_already_exists",
|
||||
"document_id": str(error.document_id),
|
||||
"document_version_id": str(error.document_version_id),
|
||||
},
|
||||
) from error
|
||||
except DocumentTooLargeError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
|
||||
detail="document exceeds the configured upload limit",
|
||||
) from error
|
||||
except (UnsupportedDocumentTypeError, MediaTypeMismatchError) as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||
detail="unsupported document type or media type",
|
||||
) from error
|
||||
except InvalidDocumentScopeError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="knowledge base or folder not found",
|
||||
) from error
|
||||
except DocumentUploadError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="document could not be accepted",
|
||||
) from error
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as error:
|
||||
# Provider and persistence exceptions are intentionally not reflected to clients.
|
||||
logger.error(
|
||||
"Document acceptance failed",
|
||||
extra={"event": "document_acceptance_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="document acceptance failed",
|
||||
) from error
|
||||
logger.info(
|
||||
"Document accepted for ingestion",
|
||||
extra={
|
||||
"event": "document_accepted",
|
||||
"document_id": str(submission.document.id),
|
||||
"document_version_id": str(submission.version.id),
|
||||
"job_id": str(submission.job.id),
|
||||
},
|
||||
)
|
||||
return IngestionAcceptedResponse.from_submission(submission)
|
||||
|
||||
|
||||
@router.get("/jobs/{job_id}", response_model=IngestionJobResponse, tags=["jobs"])
|
||||
def get_ingestion_job(job_id: UUID, services: Services, identity: Identity) -> IngestionJobResponse:
|
||||
"""Return durable stage, attempt and safe failure state for one Job."""
|
||||
try:
|
||||
job = services.jobs.get(job_id)
|
||||
except IngestionJobNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="job not found"
|
||||
) from error
|
||||
require_scope(identity, "knowledge:read")
|
||||
_ensure_document_access(identity, services, job.document_id, AclPermission.READ)
|
||||
return IngestionJobResponse.from_job(job)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/ingestion-batches",
|
||||
response_model=IngestionBatchResponse,
|
||||
status_code=status.HTTP_202_ACCEPTED,
|
||||
tags=["jobs"],
|
||||
)
|
||||
def upload_ingestion_batch(
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
files: Annotated[
|
||||
list[UploadFile],
|
||||
File(description="One or more configured P0 documents, spreadsheets, HTML, or images"),
|
||||
],
|
||||
knowledge_base_id: Annotated[UUID | None, Form()] = None,
|
||||
folder_id: Annotated[UUID | None, Form()] = None,
|
||||
relative_paths: Annotated[
|
||||
list[str] | None,
|
||||
Form(description="Ordered browser-relative paths for a directory upload"),
|
||||
] = None,
|
||||
) -> IngestionBatchResponse:
|
||||
"""Accept a bounded file or directory batch with per-file isolation."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:write")
|
||||
if relative_paths is not None and len(relative_paths) != len(files):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="relative paths must match the ordered file list",
|
||||
)
|
||||
organization_id = _authorize_upload_scope(
|
||||
identity, services, knowledge_base_id, AclPermission.WRITE
|
||||
)
|
||||
view = services.batches.submit(
|
||||
tuple(
|
||||
BatchSource(
|
||||
filename=file.filename or "",
|
||||
media_type=file.content_type,
|
||||
source=file.file,
|
||||
relative_path=(relative_paths[index] if relative_paths is not None else None),
|
||||
)
|
||||
for index, file in enumerate(files)
|
||||
),
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=knowledge_base_id,
|
||||
folder_id=folder_id,
|
||||
)
|
||||
except BatchFileCountLimitError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
|
||||
detail="batch file count exceeds the configured limit",
|
||||
) from error
|
||||
except BatchDirectoryScopeError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="directory upload requires a knowledge base",
|
||||
) from error
|
||||
except BatchFolderScopeError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="folder not found",
|
||||
) from error
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"Batch acceptance failed",
|
||||
extra={"event": "ingestion_batch_acceptance_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="batch could not be accepted",
|
||||
) from error
|
||||
logger.info(
|
||||
"Ingestion batch accepted",
|
||||
extra={"event": "ingestion_batch_accepted", "batch_id": str(view.batch.id)},
|
||||
)
|
||||
return IngestionBatchResponse.from_view(view)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/ingestion-archives",
|
||||
response_model=IngestionBatchResponse,
|
||||
status_code=status.HTTP_202_ACCEPTED,
|
||||
tags=["jobs"],
|
||||
)
|
||||
def upload_ingestion_archive(
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
archive: Annotated[
|
||||
UploadFile,
|
||||
File(description="One ZIP archive containing a bounded document tree"),
|
||||
],
|
||||
knowledge_base_id: Annotated[UUID | None, Form()] = None,
|
||||
folder_id: Annotated[UUID | None, Form()] = None,
|
||||
) -> IngestionBatchResponse:
|
||||
"""Safely expand one ZIP into independent canonical document Jobs."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:write")
|
||||
organization_id = _authorize_upload_scope(
|
||||
identity, services, knowledge_base_id, AclPermission.WRITE
|
||||
)
|
||||
if organization_id is None or knowledge_base_id is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="ZIP upload requires a knowledge base",
|
||||
)
|
||||
if services.archives is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
|
||||
detail="ZIP ingestion is unavailable",
|
||||
)
|
||||
view = services.archives.submit(
|
||||
archive.filename or "",
|
||||
archive.content_type,
|
||||
archive.file,
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=knowledge_base_id,
|
||||
folder_id=folder_id,
|
||||
)
|
||||
except UnsupportedZipArchiveError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE,
|
||||
detail="only ZIP archives are supported",
|
||||
) from error
|
||||
except (ZipArchiveLimitError, BatchFileCountLimitError) as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_413_CONTENT_TOO_LARGE,
|
||||
detail="ZIP archive exceeds a configured safety limit",
|
||||
) from error
|
||||
except (UnsafeZipArchiveError, InvalidZipArchiveError, EmptyZipArchiveError) as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="ZIP archive is invalid or unsafe",
|
||||
) from error
|
||||
except BatchFolderScopeError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="folder not found",
|
||||
) from error
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"ZIP ingestion failed",
|
||||
extra={"event": "zip_ingestion_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="ZIP archive could not be accepted",
|
||||
) from error
|
||||
logger.info(
|
||||
"ZIP ingestion accepted",
|
||||
extra={"event": "zip_ingestion_accepted", "batch_id": str(view.batch.id)},
|
||||
)
|
||||
return IngestionBatchResponse.from_view(view)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/ingestion-batches/{batch_id}",
|
||||
response_model=IngestionBatchResponse,
|
||||
tags=["jobs"],
|
||||
)
|
||||
def get_ingestion_batch(
|
||||
batch_id: UUID, services: Services, identity: Identity
|
||||
) -> IngestionBatchResponse:
|
||||
"""Return aggregate and per-file status for Admin Console polling."""
|
||||
try:
|
||||
view = services.batches.get(batch_id)
|
||||
except BatchNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail="ingestion batch not found",
|
||||
) from error
|
||||
require_scope(identity, "knowledge:read")
|
||||
_ensure_batch_access(
|
||||
identity,
|
||||
services,
|
||||
view.batch.organization_id,
|
||||
view.batch.knowledge_base_id,
|
||||
)
|
||||
return IngestionBatchResponse.from_view(view)
|
||||
|
||||
|
||||
@router.get("/documents", response_model=list[DocumentResponse], tags=["documents"])
|
||||
def list_documents(
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
knowledge_base_id: Annotated[
|
||||
UUID | None,
|
||||
Query(description="Optional readable Knowledge Base boundary"),
|
||||
] = None,
|
||||
folder_scope: Annotated[
|
||||
DocumentFolderScope,
|
||||
Query(description="all, root, or one selected folder subtree"),
|
||||
] = DocumentFolderScope.ALL,
|
||||
folder_id: Annotated[
|
||||
UUID | None,
|
||||
Query(description="Folder root required only when folder_scope=subtree"),
|
||||
] = None,
|
||||
) -> list[DocumentResponse]:
|
||||
"""List canonical documents after intersecting ACL and optional folder scope."""
|
||||
require_scope(identity, "knowledge:read")
|
||||
document_ids = _resolved_document_ids(identity, services, AclPermission.READ)
|
||||
if knowledge_base_id is None:
|
||||
if folder_scope is not DocumentFolderScope.ALL or folder_id is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="folder scope requires knowledge_base_id",
|
||||
)
|
||||
organization_id = None
|
||||
else:
|
||||
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
|
||||
if knowledge_base_id not in allowed:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found")
|
||||
organization_id, _ = require_canonical_identity(identity)
|
||||
if folder_scope is DocumentFolderScope.SUBTREE and folder_id is None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="folder_id is required for subtree scope",
|
||||
)
|
||||
if folder_scope is not DocumentFolderScope.SUBTREE and folder_id is not None:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="folder_id is only valid for subtree scope",
|
||||
)
|
||||
try:
|
||||
documents = services.documents.list_documents(
|
||||
document_ids,
|
||||
organization_id=organization_id,
|
||||
knowledge_base_id=knowledge_base_id,
|
||||
folder_scope=folder_scope,
|
||||
folder_id=folder_id,
|
||||
)
|
||||
except DocumentScopeNotFoundError as error:
|
||||
raise HTTPException(status_code=404, detail="folder not found") from error
|
||||
return [DocumentResponse.from_document(document) for document in documents]
|
||||
|
||||
|
||||
@router.get("/documents/{document_id}", response_model=DocumentResponse, tags=["documents"])
|
||||
def get_document(document_id: UUID, services: Services, identity: Identity) -> DocumentResponse:
|
||||
"""Return one document's status and safe processing diagnostic."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:read")
|
||||
_ensure_document_access(identity, services, document_id, AclPermission.READ)
|
||||
document = services.documents.get_document(document_id)
|
||||
except DocumentNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
|
||||
) from error
|
||||
return DocumentResponse.from_document(document)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/documents/{document_id}/content",
|
||||
response_model=DocumentContentResponse,
|
||||
tags=["documents"],
|
||||
)
|
||||
def get_document_content(
|
||||
document_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> DocumentContentResponse:
|
||||
"""Return canonical indexed content for the selected immutable source version."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:read")
|
||||
_ensure_document_access(identity, services, document_id, AclPermission.READ)
|
||||
content = services.documents.get_document_content(document_id, document_version_id)
|
||||
except DocumentNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
|
||||
) from error
|
||||
except DocumentVersionNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document version not found"
|
||||
) from error
|
||||
return DocumentContentResponse.from_view(content)
|
||||
|
||||
|
||||
@router.get("/documents/{document_id}/source", tags=["documents"])
|
||||
def get_document_source(
|
||||
document_id: UUID,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> StreamingResponse:
|
||||
"""Stream immutable source bytes without exposing storage keys or Provider details."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:read")
|
||||
_ensure_document_access(identity, services, document_id, AclPermission.READ)
|
||||
source = services.documents.open_document_source(document_id, document_version_id)
|
||||
except DocumentNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
|
||||
) from error
|
||||
except DocumentVersionNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document version not found"
|
||||
) from error
|
||||
except DocumentSourceUnavailableError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="document source is unavailable",
|
||||
) from error
|
||||
|
||||
inline_media_types = {
|
||||
"application/pdf",
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"text/markdown",
|
||||
"text/plain",
|
||||
}
|
||||
disposition = "inline" if source.media_type in inline_media_types else "attachment"
|
||||
response_media_type = (
|
||||
source.media_type if source.media_type in inline_media_types else "application/octet-stream"
|
||||
)
|
||||
encoded_filename = quote(source.filename, safe="")
|
||||
return StreamingResponse(
|
||||
_read_source_chunks(source.stream),
|
||||
media_type=response_media_type,
|
||||
headers={
|
||||
"Content-Disposition": (
|
||||
f"{disposition}; filename=\"source\"; filename*=UTF-8''{encoded_filename}"
|
||||
),
|
||||
"Content-Length": str(source.size_bytes),
|
||||
"X-Content-Type-Options": "nosniff",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _read_source_chunks(stream: BinaryIO, chunk_size: int = 64 * 1024) -> Iterator[bytes]:
|
||||
"""Close Provider streams deterministically after bounded response reads."""
|
||||
try:
|
||||
while True:
|
||||
chunk = stream.read(chunk_size)
|
||||
if not chunk:
|
||||
return
|
||||
yield chunk
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/documents/{document_id}",
|
||||
status_code=status.HTTP_204_NO_CONTENT,
|
||||
tags=["documents"],
|
||||
)
|
||||
def delete_document(document_id: UUID, services: Services, identity: Identity) -> Response:
|
||||
"""Delete an unreferenced document or tombstone an L2-referenced source."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:write")
|
||||
_ensure_document_access(identity, services, document_id, AclPermission.WRITE)
|
||||
services.documents.delete_document(document_id)
|
||||
except DocumentNotFoundError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND, detail="document not found"
|
||||
) from error
|
||||
except DocumentIngestionActiveError as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail="document ingestion is still active",
|
||||
) from error
|
||||
except DocumentArtifactCleanupError as error:
|
||||
logger.error(
|
||||
"Document artifact cleanup failed",
|
||||
extra={
|
||||
"event": "document_artifact_cleanup_failed",
|
||||
"document_id": str(document_id),
|
||||
},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail="document metadata was deleted but artifact cleanup failed",
|
||||
) from error
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"Document deletion failed",
|
||||
extra={"event": "document_deletion_failed", "document_id": str(document_id)},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="document deletion failed",
|
||||
) from error
|
||||
return Response(status_code=status.HTTP_204_NO_CONTENT)
|
||||
|
||||
|
||||
@router.post("/search", response_model=SearchResponse, tags=["search"])
|
||||
def search_knowledge(
|
||||
payload: SearchRequest, services: Services, identity: Identity
|
||||
) -> SearchResponse:
|
||||
"""Return reranked Top-K sources without leaking provider or index details."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:search")
|
||||
if identity.is_canonical:
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.authorized_search is None:
|
||||
raise HTTPException(status_code=503, detail="authorized search is unavailable")
|
||||
result = services.authorized_search.search(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
query=payload.query,
|
||||
top_k=payload.top_k,
|
||||
)
|
||||
else:
|
||||
result = services.search.search(payload.query, payload.top_k)
|
||||
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="search request is invalid for the active index profile",
|
||||
) from error
|
||||
except PermissionIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"Knowledge search provider chain failed",
|
||||
extra={"event": "knowledge_search_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="search providers failed",
|
||||
) from error
|
||||
return SearchResponse.from_result(result)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/search/diagnostics",
|
||||
response_model=SearchDiagnosticsResponse,
|
||||
tags=["search"],
|
||||
)
|
||||
def diagnose_knowledge_search(
|
||||
payload: SearchRequest,
|
||||
services: Services,
|
||||
identity: Identity,
|
||||
) -> SearchDiagnosticsResponse:
|
||||
"""Expose safe stage evidence for the same pipeline used by public Search."""
|
||||
try:
|
||||
require_scope(identity, "knowledge:search")
|
||||
if identity.is_canonical:
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.authorized_search is None:
|
||||
raise HTTPException(status_code=503, detail="authorized search is unavailable")
|
||||
diagnostics = services.authorized_search.diagnose(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
query=payload.query,
|
||||
top_k=payload.top_k,
|
||||
)
|
||||
else:
|
||||
diagnostics = services.search.diagnose(payload.query, payload.top_k)
|
||||
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
detail="search request is invalid for the active index profile",
|
||||
) from error
|
||||
except PermissionIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"Knowledge search diagnostics failed",
|
||||
extra={"event": "knowledge_search_diagnostics_failed"},
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_502_BAD_GATEWAY,
|
||||
detail="search diagnostics failed",
|
||||
) from error
|
||||
return SearchDiagnosticsResponse.from_result(diagnostics)
|
||||
|
||||
|
||||
def _resolved_document_ids(
|
||||
identity: ApiIdentity,
|
||||
services: ApiServices,
|
||||
permission: AclPermission,
|
||||
*,
|
||||
include_deleted: bool = False,
|
||||
) -> tuple[UUID, ...] | None:
|
||||
"""Resolve canonical ACL scope while retaining explicit legacy compatibility."""
|
||||
if not identity.is_canonical:
|
||||
return None
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.permission_resolver is None:
|
||||
raise HTTPException(status_code=503, detail="permission resolver is unavailable")
|
||||
try:
|
||||
return services.permission_resolver.resolve_documents(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
permission=permission,
|
||||
include_deleted=include_deleted,
|
||||
).document_ids
|
||||
except PermissionIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
|
||||
|
||||
def _ensure_document_access(
|
||||
identity: ApiIdentity,
|
||||
services: ApiServices,
|
||||
document_id: UUID,
|
||||
permission: AclPermission,
|
||||
) -> None:
|
||||
"""Use a uniform not-found response for resources outside canonical scope."""
|
||||
allowed = _resolved_document_ids(identity, services, permission)
|
||||
if allowed is not None and document_id not in allowed:
|
||||
raise HTTPException(status_code=404, detail="document not found")
|
||||
|
||||
|
||||
def _authorize_upload_scope(
|
||||
identity: ApiIdentity,
|
||||
services: ApiServices,
|
||||
knowledge_base_id: UUID | None,
|
||||
permission: AclPermission,
|
||||
) -> UUID | None:
|
||||
"""Require canonical uploads to select one writable Knowledge Base."""
|
||||
if not identity.is_canonical:
|
||||
return None
|
||||
organization_id, _ = require_canonical_identity(identity)
|
||||
if knowledge_base_id is None:
|
||||
raise HTTPException(status_code=422, detail="knowledge_base_id is required")
|
||||
allowed = _resolved_knowledge_base_ids(identity, services, permission)
|
||||
if knowledge_base_id not in allowed:
|
||||
raise HTTPException(status_code=404, detail="knowledge base not found")
|
||||
return organization_id
|
||||
|
||||
|
||||
def _resolved_knowledge_base_ids(
|
||||
identity: ApiIdentity,
|
||||
services: ApiServices,
|
||||
permission: AclPermission,
|
||||
) -> tuple[UUID, ...]:
|
||||
"""Resolve exact Knowledge Base scope for one canonical REST identity."""
|
||||
organization_id, user_id = require_canonical_identity(identity)
|
||||
if services.permission_resolver is None:
|
||||
raise HTTPException(status_code=503, detail="permission resolver is unavailable")
|
||||
try:
|
||||
return services.permission_resolver.resolve_knowledge_bases(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
permission=permission,
|
||||
).knowledge_base_ids
|
||||
except PermissionIdentityError as error:
|
||||
raise HTTPException(status_code=403, detail="operation is forbidden") from error
|
||||
|
||||
|
||||
def _ensure_batch_access(
|
||||
identity: ApiIdentity,
|
||||
services: ApiServices,
|
||||
batch_organization_id: UUID | None,
|
||||
batch_knowledge_base_id: UUID | None,
|
||||
) -> None:
|
||||
"""Authorize even all-rejected batches from their persisted submission scope."""
|
||||
if not identity.is_canonical:
|
||||
if batch_organization_id is not None or batch_knowledge_base_id is not None:
|
||||
raise HTTPException(status_code=404, detail="ingestion batch not found")
|
||||
return
|
||||
organization_id, _ = require_canonical_identity(identity)
|
||||
allowed = _resolved_knowledge_base_ids(identity, services, AclPermission.READ)
|
||||
if (
|
||||
batch_organization_id != organization_id
|
||||
or batch_knowledge_base_id is None
|
||||
or batch_knowledge_base_id not in allowed
|
||||
):
|
||||
raise HTTPException(status_code=404, detail="ingestion batch not found")
|
||||
1060
hy4/_kp/apps/api/src/ekp_api/schemas.py
Normal file
1060
hy4/_kp/apps/api/src/ekp_api/schemas.py
Normal file
File diff suppressed because it is too large
Load Diff
124
hy4/_kp/apps/api/src/ekp_api/worker.py
Normal file
124
hy4/_kp/apps/api/src/ekp_api/worker.py
Normal file
@ -0,0 +1,124 @@
|
||||
"""Celery worker entry point for durable ingestion Jobs."""
|
||||
|
||||
import logging
|
||||
from functools import lru_cache
|
||||
from typing import Any
|
||||
from uuid import UUID
|
||||
|
||||
from ekp_api.celery_queue import (
|
||||
COMPILATION_RECOVERY_TASK_NAME,
|
||||
COMPILATION_TASK_NAME,
|
||||
INGESTION_TASK_NAME,
|
||||
RECOVERY_TASK_NAME,
|
||||
create_celery_app,
|
||||
)
|
||||
from ekp_api.providers import WorkerServices, create_worker_services
|
||||
from ekp_core.compilation_jobs import CompilationRunAction, dispatch_pending_compilation_jobs
|
||||
from ekp_core.config import get_settings
|
||||
from ekp_core.ingestion_jobs import JobRunAction, dispatch_pending_jobs
|
||||
from ekp_core.observability import configure_json_logging
|
||||
|
||||
_settings = get_settings()
|
||||
configure_json_logging(_settings.log_level)
|
||||
celery_app = create_celery_app(_settings)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@lru_cache
|
||||
def _worker_services() -> WorkerServices:
|
||||
"""Build one provider/service graph per worker process."""
|
||||
return create_worker_services(_settings)
|
||||
|
||||
|
||||
@celery_app.task( # type: ignore[untyped-decorator]
|
||||
bind=True,
|
||||
name=INGESTION_TASK_NAME,
|
||||
max_retries=None,
|
||||
)
|
||||
def run_ingestion(self: Any, job_id: str) -> None:
|
||||
"""Execute one delivery and ask Celery to redeliver only when required.
|
||||
|
||||
Celery delivery retries are unbounded because valid-lease deferrals are not
|
||||
business Attempts; the durable runner independently enforces max_attempts.
|
||||
"""
|
||||
logger.info(
|
||||
"Ingestion delivery received",
|
||||
extra={"event": "ingestion_delivery_received", "job_id": job_id},
|
||||
)
|
||||
result = _worker_services().runner.run(UUID(job_id))
|
||||
logger.info(
|
||||
"Ingestion delivery completed",
|
||||
extra={
|
||||
"event": "ingestion_delivery_completed",
|
||||
"job_id": job_id,
|
||||
"job_status": result.status.value,
|
||||
"retry_delay_seconds": result.retry_delay_seconds,
|
||||
},
|
||||
)
|
||||
if result.action == JobRunAction.RETRY:
|
||||
raise self.retry(countdown=result.retry_delay_seconds or 1, max_retries=None)
|
||||
|
||||
|
||||
@celery_app.task(name=RECOVERY_TASK_NAME) # type: ignore[untyped-decorator]
|
||||
def recover_ingestion_jobs() -> int:
|
||||
"""Redeliver due durable Jobs after API/broker or worker interruptions."""
|
||||
services = _worker_services()
|
||||
dispatch_count = dispatch_pending_jobs(services.jobs, services.dispatcher)
|
||||
logger.info(
|
||||
"Durable ingestion recovery scan completed",
|
||||
extra={
|
||||
"event": "ingestion_recovery_scan_completed",
|
||||
"dispatch_count": dispatch_count,
|
||||
},
|
||||
)
|
||||
return dispatch_count
|
||||
|
||||
|
||||
@celery_app.task( # type: ignore[untyped-decorator]
|
||||
bind=True,
|
||||
name=COMPILATION_TASK_NAME,
|
||||
max_retries=None,
|
||||
)
|
||||
def run_compilation(self: Any, job_id: str) -> None:
|
||||
"""Execute one durable AI generation delivery outside the API process."""
|
||||
logger.info(
|
||||
"Compilation delivery received",
|
||||
extra={"event": "compilation_delivery_received", "job_id": job_id},
|
||||
)
|
||||
result = _worker_services().compilation_runner.run(UUID(job_id))
|
||||
logger.info(
|
||||
"Compilation delivery completed",
|
||||
extra={
|
||||
"event": "compilation_delivery_completed",
|
||||
"job_id": job_id,
|
||||
"job_status": result.status.value,
|
||||
"retry_delay_seconds": result.retry_delay_seconds,
|
||||
},
|
||||
)
|
||||
if result.action == CompilationRunAction.RETRY:
|
||||
raise self.retry(countdown=result.retry_delay_seconds or 1, max_retries=None)
|
||||
|
||||
|
||||
@celery_app.task(name=COMPILATION_RECOVERY_TASK_NAME) # type: ignore[untyped-decorator]
|
||||
def recover_compilation_jobs() -> int:
|
||||
"""Redeliver due durable Compilation Jobs after process interruptions."""
|
||||
services = _worker_services()
|
||||
dispatch_count = dispatch_pending_compilation_jobs(
|
||||
services.compilation_jobs,
|
||||
services.compilation_dispatcher,
|
||||
)
|
||||
logger.info(
|
||||
"Durable compilation recovery scan completed",
|
||||
extra={
|
||||
"event": "compilation_recovery_scan_completed",
|
||||
"dispatch_count": dispatch_count,
|
||||
},
|
||||
)
|
||||
return dispatch_count
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Run one worker with the embedded Demo recovery scheduler."""
|
||||
celery_app.worker_main(
|
||||
["worker", "--loglevel=INFO", "--queues=ingestion,compilation", "--beat"]
|
||||
)
|
||||
0
hy4/_kp/apps/mcp-server/.gitkeep
Normal file
0
hy4/_kp/apps/mcp-server/.gitkeep
Normal file
5
hy4/_kp/apps/mcp-server/src/ekp_mcp/__init__.py
Normal file
5
hy4/_kp/apps/mcp-server/src/ekp_mcp/__init__.py
Normal file
@ -0,0 +1,5 @@
|
||||
"""Enterprise Knowledge MCP gateway."""
|
||||
|
||||
from ekp_mcp.server import create_mcp_http_app, create_mcp_server
|
||||
|
||||
__all__ = ["create_mcp_http_app", "create_mcp_server"]
|
||||
157
hy4/_kp/apps/mcp-server/src/ekp_mcp/auth.py
Normal file
157
hy4/_kp/apps/mcp-server/src/ekp_mcp/auth.py
Normal file
@ -0,0 +1,157 @@
|
||||
"""Request-scoped MCP service identity and bearer-token verification."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from secrets import compare_digest
|
||||
from typing import Literal, cast
|
||||
from uuid import UUID
|
||||
|
||||
from mcp.server.auth.middleware.auth_context import get_access_token
|
||||
from mcp.server.auth.provider import AccessToken
|
||||
from mcp.server.mcpserver.exceptions import ToolError
|
||||
from pydantic import SecretStr
|
||||
|
||||
from ekp_core.config import McpIdentityCredential
|
||||
|
||||
McpCredentialType = Literal["legacy_public", "user_token", "service_api_key"]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class McpIdentity:
|
||||
"""Authenticated principal and its canonical ACL subject for one request."""
|
||||
|
||||
client_id: str
|
||||
subject: str
|
||||
scopes: frozenset[str]
|
||||
credential_type: McpCredentialType = "legacy_public"
|
||||
organization_id: UUID | None = None
|
||||
acl_user_id: UUID | None = None
|
||||
|
||||
|
||||
class StaticServiceTokenVerifier:
|
||||
"""Validate one runtime-injected, least-privilege Demo service credential.
|
||||
|
||||
The verifier is intentionally small for DEMO-012. It establishes the MCP
|
||||
identity boundary without pretending to be the V1 OAuth or ACL solution.
|
||||
"""
|
||||
|
||||
__slots__ = (
|
||||
"_client_id",
|
||||
"_expected_token",
|
||||
"_issuer",
|
||||
"_scope",
|
||||
"_subject",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
expected_token: SecretStr,
|
||||
*,
|
||||
client_id: str,
|
||||
subject: str,
|
||||
scope: str,
|
||||
issuer: str,
|
||||
) -> None:
|
||||
self._expected_token = expected_token
|
||||
self._client_id = client_id
|
||||
self._subject = subject
|
||||
self._scope = scope
|
||||
self._issuer = issuer
|
||||
|
||||
async def verify_token(self, token: str) -> AccessToken | None:
|
||||
"""Return scoped identity metadata only for a constant-time token match."""
|
||||
if not compare_digest(token, self._expected_token.get_secret_value()):
|
||||
return None
|
||||
return AccessToken(
|
||||
token=token,
|
||||
client_id=self._client_id,
|
||||
subject=self._subject,
|
||||
scopes=[self._scope],
|
||||
claims={"iss": self._issuer},
|
||||
)
|
||||
|
||||
|
||||
class ConfiguredIdentityTokenVerifier:
|
||||
"""Verify runtime-secret user/service credentials without logging their values."""
|
||||
|
||||
__slots__ = ("_credentials", "_issuer")
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials: tuple[McpIdentityCredential, ...],
|
||||
*,
|
||||
issuer: str,
|
||||
) -> None:
|
||||
if not credentials:
|
||||
raise ValueError("at least one MCP identity credential is required")
|
||||
tokens = [credential.token.get_secret_value() for credential in credentials]
|
||||
if len(set(tokens)) != len(tokens):
|
||||
raise ValueError("MCP identity credential tokens must be unique")
|
||||
self._credentials = credentials
|
||||
self._issuer = issuer
|
||||
|
||||
async def verify_token(self, token: str) -> AccessToken | None:
|
||||
"""Map a matching bearer secret to its configured canonical identity."""
|
||||
selected: McpIdentityCredential | None = None
|
||||
# Compare every configured secret so match position does not become a
|
||||
# useful timing signal when multiple principals share one gateway.
|
||||
for credential in self._credentials:
|
||||
if compare_digest(token, credential.token.get_secret_value()):
|
||||
selected = credential
|
||||
if selected is None:
|
||||
return None
|
||||
return AccessToken(
|
||||
token=token,
|
||||
client_id=selected.client_id,
|
||||
subject=selected.subject,
|
||||
scopes=list(selected.scopes),
|
||||
claims={
|
||||
"iss": self._issuer,
|
||||
"ekp_credential_type": selected.credential_type,
|
||||
"ekp_organization_id": str(selected.organization_id),
|
||||
"ekp_acl_user_id": str(selected.acl_user_id),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def current_mcp_identity() -> McpIdentity:
|
||||
"""Resolve the SDK-authenticated principal for the active tool request."""
|
||||
access_token = get_access_token()
|
||||
if access_token is None:
|
||||
# HTTP middleware normally rejects this earlier; retaining the guard
|
||||
# prevents accidental unauthenticated in-process tool invocation.
|
||||
raise ToolError("authentication required")
|
||||
subject = access_token.subject or access_token.client_id
|
||||
claims = access_token.claims or {}
|
||||
raw_credential_type = claims.get("ekp_credential_type", "legacy_public")
|
||||
credential_type = cast(McpCredentialType, raw_credential_type)
|
||||
if credential_type not in {"legacy_public", "user_token", "service_api_key"}:
|
||||
raise ToolError("authentication identity is invalid")
|
||||
organization_id = _optional_uuid_claim(claims, "ekp_organization_id")
|
||||
acl_user_id = _optional_uuid_claim(claims, "ekp_acl_user_id")
|
||||
if (organization_id is None) != (acl_user_id is None):
|
||||
raise ToolError("authentication identity is invalid")
|
||||
if credential_type != "legacy_public" and organization_id is None:
|
||||
raise ToolError("authentication identity is invalid")
|
||||
if credential_type == "legacy_public" and organization_id is not None:
|
||||
raise ToolError("authentication identity is invalid")
|
||||
return McpIdentity(
|
||||
client_id=access_token.client_id,
|
||||
subject=subject,
|
||||
scopes=frozenset(access_token.scopes),
|
||||
credential_type=credential_type,
|
||||
organization_id=organization_id,
|
||||
acl_user_id=acl_user_id,
|
||||
)
|
||||
|
||||
|
||||
def _optional_uuid_claim(claims: dict[str, object], name: str) -> UUID | None:
|
||||
"""Parse an optional UUID claim without reflecting attacker-controlled text."""
|
||||
value = claims.get(name)
|
||||
if value is None:
|
||||
return None
|
||||
if not isinstance(value, str):
|
||||
raise ToolError("authentication identity is invalid")
|
||||
try:
|
||||
return UUID(value)
|
||||
except ValueError as error:
|
||||
raise ToolError("authentication identity is invalid") from error
|
||||
308
hy4/_kp/apps/mcp-server/src/ekp_mcp/knowledge.py
Normal file
308
hy4/_kp/apps/mcp-server/src/ekp_mcp/knowledge.py
Normal file
@ -0,0 +1,308 @@
|
||||
"""Identity-aware MCP read boundary for documents, sections, sources, and spaces."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal, Protocol
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from ekp_core.knowledge_access import (
|
||||
KnowledgeBaseReadView,
|
||||
KnowledgeDocumentReadView,
|
||||
KnowledgeSectionReadView,
|
||||
KnowledgeSourceReadView,
|
||||
)
|
||||
from ekp_domain import DocumentStatus, KnowledgeBaseVisibility
|
||||
from ekp_mcp.auth import McpIdentity
|
||||
|
||||
|
||||
class KnowledgeReader(Protocol):
|
||||
"""Subset of the shared authorized read service consumed by MCP."""
|
||||
|
||||
def list_knowledge_bases(
|
||||
self,
|
||||
*,
|
||||
organization_id: UUID,
|
||||
user_id: UUID,
|
||||
) -> tuple[KnowledgeBaseReadView, ...]: ...
|
||||
|
||||
def get_document(
|
||||
self,
|
||||
*,
|
||||
organization_id: UUID,
|
||||
user_id: UUID,
|
||||
document_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> KnowledgeDocumentReadView: ...
|
||||
|
||||
def get_source(
|
||||
self,
|
||||
*,
|
||||
organization_id: UUID,
|
||||
user_id: UUID,
|
||||
document_id: UUID,
|
||||
chunk_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> KnowledgeSourceReadView: ...
|
||||
|
||||
def get_document_section(
|
||||
self,
|
||||
*,
|
||||
organization_id: UUID,
|
||||
user_id: UUID,
|
||||
document_id: UUID,
|
||||
anchor_chunk_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
offset: int = 0,
|
||||
max_chunks: int = 12,
|
||||
) -> KnowledgeSectionReadView: ...
|
||||
|
||||
|
||||
class McpKnowledgeBaseItem(BaseModel):
|
||||
"""One authorized MCP-enabled Knowledge Base."""
|
||||
|
||||
knowledge_base_id: UUID
|
||||
name: str
|
||||
description: str | None
|
||||
visibility: KnowledgeBaseVisibility
|
||||
readable_document_count: int
|
||||
|
||||
@classmethod
|
||||
def from_view(cls, view: KnowledgeBaseReadView) -> "McpKnowledgeBaseItem":
|
||||
return cls(
|
||||
knowledge_base_id=view.id,
|
||||
name=view.name,
|
||||
description=view.description,
|
||||
visibility=view.visibility,
|
||||
readable_document_count=view.readable_document_count,
|
||||
)
|
||||
|
||||
|
||||
class McpKnowledgeBaseListResponse(BaseModel):
|
||||
"""Stable structured response for `list_knowledge_bases`."""
|
||||
|
||||
knowledge_bases: list[McpKnowledgeBaseItem]
|
||||
|
||||
|
||||
class McpDocumentResponse(BaseModel):
|
||||
"""Document metadata and bounded outline without bulk body transfer."""
|
||||
|
||||
document_id: UUID
|
||||
knowledge_base_id: UUID
|
||||
knowledge_base_name: str
|
||||
document_version_id: UUID
|
||||
document_version: int
|
||||
filename: str
|
||||
media_type: str
|
||||
size_bytes: int
|
||||
status: DocumentStatus
|
||||
is_active: bool
|
||||
chunk_count: int
|
||||
section_paths: list[tuple[str, ...]]
|
||||
outline_truncated: bool
|
||||
|
||||
@classmethod
|
||||
def from_view(cls, view: KnowledgeDocumentReadView) -> "McpDocumentResponse":
|
||||
return cls(
|
||||
document_id=view.document_id,
|
||||
knowledge_base_id=view.knowledge_base_id,
|
||||
knowledge_base_name=view.knowledge_base_name,
|
||||
document_version_id=view.document_version_id,
|
||||
document_version=view.document_version,
|
||||
filename=view.filename,
|
||||
media_type=view.media_type,
|
||||
size_bytes=view.size_bytes,
|
||||
status=view.status,
|
||||
is_active=view.is_active,
|
||||
chunk_count=view.chunk_count,
|
||||
section_paths=list(view.section_paths),
|
||||
outline_truncated=view.outline_truncated,
|
||||
)
|
||||
|
||||
|
||||
class McpKnowledgeChunk(BaseModel):
|
||||
"""Exact canonical Chunk and its source coordinates."""
|
||||
|
||||
chunk_id: UUID
|
||||
chunk_index: int
|
||||
page: int | None
|
||||
page_end: int | None
|
||||
section: str | None
|
||||
section_path: tuple[str, ...]
|
||||
sheet_name: str | None
|
||||
row_range: tuple[int, int] | None
|
||||
column_range: tuple[int, int] | None
|
||||
content: str
|
||||
source_type: Literal["raw"]
|
||||
|
||||
|
||||
class McpSourceResponse(BaseModel):
|
||||
"""One exact source passage addressable from a Search result."""
|
||||
|
||||
document_id: UUID
|
||||
knowledge_base_id: UUID
|
||||
document_version_id: UUID
|
||||
document_version: int
|
||||
filename: str
|
||||
chunk: McpKnowledgeChunk
|
||||
|
||||
@classmethod
|
||||
def from_view(cls, view: KnowledgeSourceReadView) -> "McpSourceResponse":
|
||||
return cls(
|
||||
document_id=view.document_id,
|
||||
knowledge_base_id=view.knowledge_base_id,
|
||||
document_version_id=view.document_version_id,
|
||||
document_version=view.document_version,
|
||||
filename=view.filename,
|
||||
chunk=McpKnowledgeChunk.model_validate(view.chunk, from_attributes=True),
|
||||
)
|
||||
|
||||
|
||||
class McpDocumentSectionResponse(BaseModel):
|
||||
"""Bounded source-order page of one canonical section."""
|
||||
|
||||
document_id: UUID
|
||||
knowledge_base_id: UUID
|
||||
document_version_id: UUID
|
||||
document_version: int
|
||||
filename: str
|
||||
anchor_chunk_id: UUID
|
||||
section_path: tuple[str, ...]
|
||||
offset: int
|
||||
total_chunk_count: int
|
||||
chunks: list[McpKnowledgeChunk]
|
||||
next_offset: int | None
|
||||
|
||||
@classmethod
|
||||
def from_view(cls, view: KnowledgeSectionReadView) -> "McpDocumentSectionResponse":
|
||||
return cls(
|
||||
document_id=view.document_id,
|
||||
knowledge_base_id=view.knowledge_base_id,
|
||||
document_version_id=view.document_version_id,
|
||||
document_version=view.document_version,
|
||||
filename=view.filename,
|
||||
anchor_chunk_id=view.anchor_chunk_id,
|
||||
section_path=view.section_path,
|
||||
offset=view.offset,
|
||||
total_chunk_count=view.total_chunk_count,
|
||||
chunks=[
|
||||
McpKnowledgeChunk.model_validate(chunk, from_attributes=True)
|
||||
for chunk in view.chunks
|
||||
],
|
||||
next_offset=view.next_offset,
|
||||
)
|
||||
|
||||
|
||||
class McpKnowledgeReader(Protocol):
|
||||
"""Transport-facing application contract registered by the MCP server."""
|
||||
|
||||
def list_knowledge_bases(self, identity: McpIdentity) -> McpKnowledgeBaseListResponse: ...
|
||||
|
||||
def get_document(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
) -> McpDocumentResponse: ...
|
||||
|
||||
def get_source(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
chunk_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
) -> McpSourceResponse: ...
|
||||
|
||||
def get_document_section(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
anchor_chunk_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
offset: int,
|
||||
max_chunks: int,
|
||||
) -> McpDocumentSectionResponse: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class McpKnowledgeApplication:
|
||||
"""Require a scoped canonical identity for every non-search knowledge read."""
|
||||
|
||||
reader: KnowledgeReader
|
||||
required_scope: str
|
||||
|
||||
def list_knowledge_bases(self, identity: McpIdentity) -> McpKnowledgeBaseListResponse:
|
||||
organization_id, user_id = self._authorize(identity)
|
||||
return McpKnowledgeBaseListResponse(
|
||||
knowledge_bases=[
|
||||
McpKnowledgeBaseItem.from_view(view)
|
||||
for view in self.reader.list_knowledge_bases(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
def get_document(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
) -> McpDocumentResponse:
|
||||
organization_id, user_id = self._authorize(identity)
|
||||
return McpDocumentResponse.from_view(
|
||||
self.reader.get_document(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
document_id=document_id,
|
||||
document_version_id=document_version_id,
|
||||
)
|
||||
)
|
||||
|
||||
def get_source(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
chunk_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
) -> McpSourceResponse:
|
||||
organization_id, user_id = self._authorize(identity)
|
||||
return McpSourceResponse.from_view(
|
||||
self.reader.get_source(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
document_id=document_id,
|
||||
chunk_id=chunk_id,
|
||||
document_version_id=document_version_id,
|
||||
)
|
||||
)
|
||||
|
||||
def get_document_section(
|
||||
self,
|
||||
identity: McpIdentity,
|
||||
document_id: UUID,
|
||||
anchor_chunk_id: UUID,
|
||||
document_version_id: UUID | None,
|
||||
offset: int,
|
||||
max_chunks: int,
|
||||
) -> McpDocumentSectionResponse:
|
||||
organization_id, user_id = self._authorize(identity)
|
||||
return McpDocumentSectionResponse.from_view(
|
||||
self.reader.get_document_section(
|
||||
organization_id=organization_id,
|
||||
user_id=user_id,
|
||||
document_id=document_id,
|
||||
anchor_chunk_id=anchor_chunk_id,
|
||||
document_version_id=document_version_id,
|
||||
offset=offset,
|
||||
max_chunks=max_chunks,
|
||||
)
|
||||
)
|
||||
|
||||
def _authorize(self, identity: McpIdentity) -> tuple[UUID, UUID]:
|
||||
"""Fail closed for legacy or partially mapped identities."""
|
||||
if self.required_scope not in identity.scopes:
|
||||
raise PermissionError("MCP identity lacks the knowledge read scope")
|
||||
if identity.organization_id is None or identity.acl_user_id is None:
|
||||
raise PermissionError("MCP identity is not bound to a canonical ACL subject")
|
||||
return identity.organization_id, identity.acl_user_id
|
||||
32
hy4/_kp/apps/mcp-server/src/ekp_mcp/main.py
Normal file
32
hy4/_kp/apps/mcp-server/src/ekp_mcp/main.py
Normal file
@ -0,0 +1,32 @@
|
||||
"""Command-line entry point for the independent MCP gateway process."""
|
||||
|
||||
from mcp.server.transport_security import TransportSecuritySettings
|
||||
|
||||
from ekp_core.config import get_settings
|
||||
from ekp_core.observability import configure_json_logging
|
||||
from ekp_mcp.server import create_mcp_server
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Run the authenticated Streamable HTTP server from environment settings."""
|
||||
settings = get_settings()
|
||||
configure_json_logging(settings.log_level)
|
||||
server = create_mcp_server(settings)
|
||||
server.run(
|
||||
transport="streamable-http",
|
||||
host=settings.mcp_host,
|
||||
port=settings.mcp_port,
|
||||
streamable_http_path=settings.mcp_path,
|
||||
json_response=True,
|
||||
stateless_http=True,
|
||||
max_request_body_size=settings.mcp_max_request_body_bytes,
|
||||
transport_security=TransportSecuritySettings(
|
||||
enable_dns_rebinding_protection=True,
|
||||
allowed_hosts=list(settings.mcp_allowed_hosts),
|
||||
allowed_origins=list(settings.mcp_allowed_origins),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover - exercised through the console script.
|
||||
main()
|
||||
168
hy4/_kp/apps/mcp-server/src/ekp_mcp/search.py
Normal file
168
hy4/_kp/apps/mcp-server/src/ekp_mcp/search.py
Normal file
@ -0,0 +1,168 @@
|
||||
"""MCP-specific application boundary over the provider-neutral SearchService."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Annotated, Literal, Protocol
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from ekp_core.search import CompiledSourceResult, SearchResult
|
||||
from ekp_domain import CompiledKnowledgeType
|
||||
from ekp_mcp.auth import McpIdentity
|
||||
|
||||
|
||||
class KnowledgeSearcher(Protocol):
|
||||
"""Subset of SearchService required by the MCP gateway."""
|
||||
|
||||
def search(self, query: str, top_k: int | None = None) -> SearchResult:
|
||||
"""Return reranked canonical sources for a normalized query."""
|
||||
|
||||
|
||||
class AuthorizedKnowledgeSearcher(Protocol):
|
||||
"""Subset of AuthorizedSearchService required by the MCP gateway."""
|
||||
|
||||
def search(
|
||||
self,
|
||||
*,
|
||||
organization_id: UUID,
|
||||
user_id: UUID,
|
||||
query: str,
|
||||
top_k: int | None = None,
|
||||
mcp_enabled_only: bool = False,
|
||||
) -> SearchResult:
|
||||
"""Resolve ACL scope before either retrieval Top-N branch."""
|
||||
|
||||
|
||||
class McpSourceResponse(BaseModel):
|
||||
"""Provider-neutral source shape exposed by `search_knowledge`."""
|
||||
|
||||
document_id: UUID
|
||||
document_version_id: UUID
|
||||
document_version: int
|
||||
filename: str
|
||||
page: int | None
|
||||
page_end: int | None = None
|
||||
section: str | None
|
||||
section_path: tuple[str, ...]
|
||||
sheet_name: str | None = None
|
||||
row_range: tuple[int, int] | None = None
|
||||
column_range: tuple[int, int] | None = None
|
||||
chunk_id: UUID
|
||||
content: str
|
||||
context: str
|
||||
score: float
|
||||
rerank_score: float
|
||||
source_type: Literal["raw"]
|
||||
|
||||
|
||||
class McpCompiledUnderlyingSourceResponse(BaseModel):
|
||||
"""Exact source evidence beneath a published Compiled result."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
document_id: UUID
|
||||
document_version_id: UUID
|
||||
document_version: int
|
||||
filename: str
|
||||
page: int | None
|
||||
page_end: int | None
|
||||
section: str | None
|
||||
section_path: tuple[str, ...]
|
||||
sheet_name: str | None
|
||||
row_range: tuple[int, int] | None
|
||||
column_range: tuple[int, int] | None
|
||||
chunk_id: UUID
|
||||
chunk_index: int
|
||||
content: str
|
||||
|
||||
|
||||
class McpCompiledSearchResponse(BaseModel):
|
||||
"""Published L2 result aligned exactly with the REST response shape."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
compiled_knowledge_id: UUID
|
||||
knowledge_base_id: UUID
|
||||
title: str
|
||||
knowledge_type: CompiledKnowledgeType
|
||||
content: str
|
||||
context: str
|
||||
underlying_sources: tuple[McpCompiledUnderlyingSourceResponse, ...]
|
||||
score: float
|
||||
rerank_score: float
|
||||
source_type: Literal["compiled"]
|
||||
|
||||
|
||||
McpSearchSourceResponse = Annotated[
|
||||
McpSourceResponse | McpCompiledSearchResponse,
|
||||
Field(discriminator="source_type"),
|
||||
]
|
||||
|
||||
|
||||
class McpSearchResponse(BaseModel):
|
||||
"""Structured MCP result aligned with the REST search source fields."""
|
||||
|
||||
results: list[McpSearchSourceResponse]
|
||||
|
||||
|
||||
class McpKnowledgeSearch(Protocol):
|
||||
"""Identity-aware search boundary consumed by the protocol server."""
|
||||
|
||||
def search(self, identity: McpIdentity, query: str, top_k: int) -> McpSearchResponse:
|
||||
"""Return authorized, structured knowledge sources."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class McpSearchApplication:
|
||||
"""Enforce MCP scope and canonical Document ACL before shared retrieval."""
|
||||
|
||||
searcher: KnowledgeSearcher
|
||||
required_scope: str
|
||||
authorized_searcher: AuthorizedKnowledgeSearcher | None = None
|
||||
allow_legacy_public: bool = False
|
||||
|
||||
def search(self, identity: McpIdentity, query: str, top_k: int) -> McpSearchResponse:
|
||||
"""Authorize the principal, fail closed, and serialize canonical results."""
|
||||
if self.required_scope not in identity.scopes:
|
||||
raise PermissionError("MCP identity lacks the knowledge search scope")
|
||||
if identity.organization_id is not None and identity.acl_user_id is not None:
|
||||
if self.authorized_searcher is None:
|
||||
raise PermissionError("MCP ACL search service is unavailable")
|
||||
result = self.authorized_searcher.search(
|
||||
organization_id=identity.organization_id,
|
||||
user_id=identity.acl_user_id,
|
||||
query=query,
|
||||
top_k=top_k,
|
||||
mcp_enabled_only=True,
|
||||
)
|
||||
elif identity.credential_type == "legacy_public" and self.allow_legacy_public:
|
||||
result = self.searcher.search(query, top_k)
|
||||
else:
|
||||
raise PermissionError("MCP identity is not bound to a canonical ACL subject")
|
||||
responses: list[McpSourceResponse | McpCompiledSearchResponse] = []
|
||||
for source in result.results:
|
||||
if isinstance(source, CompiledSourceResult):
|
||||
responses.append(McpCompiledSearchResponse.model_validate(source))
|
||||
else:
|
||||
responses.append(
|
||||
McpSourceResponse(
|
||||
document_id=source.document_id,
|
||||
document_version_id=source.document_version_id,
|
||||
document_version=source.document_version,
|
||||
filename=source.filename,
|
||||
page=source.page,
|
||||
page_end=source.page_end,
|
||||
section=source.section,
|
||||
section_path=source.section_path,
|
||||
sheet_name=source.sheet_name,
|
||||
row_range=source.row_range,
|
||||
column_range=source.column_range,
|
||||
chunk_id=source.chunk_id,
|
||||
content=source.content,
|
||||
context=source.context,
|
||||
score=source.score,
|
||||
rerank_score=source.rerank_score,
|
||||
source_type=source.source_type,
|
||||
)
|
||||
)
|
||||
return McpSearchResponse(results=responses)
|
||||
323
hy4/_kp/apps/mcp-server/src/ekp_mcp/server.py
Normal file
323
hy4/_kp/apps/mcp-server/src/ekp_mcp/server.py
Normal file
@ -0,0 +1,323 @@
|
||||
"""Authenticated Streamable HTTP MCP server exposing Demo business tools."""
|
||||
|
||||
import logging
|
||||
from functools import partial
|
||||
from typing import Annotated
|
||||
from uuid import UUID
|
||||
|
||||
import anyio
|
||||
from mcp.server import MCPServer
|
||||
from mcp.server.auth.provider import TokenVerifier
|
||||
from mcp.server.auth.settings import AuthSettings
|
||||
from mcp.server.mcpserver.exceptions import ToolError
|
||||
from mcp.server.transport_security import TransportSecuritySettings
|
||||
from pydantic import Field
|
||||
from starlette.applications import Starlette
|
||||
|
||||
from ekp_api.providers import ApiServices, create_runtime_search_services
|
||||
from ekp_core.config import Settings, get_settings
|
||||
from ekp_core.dense_retrieval import InvalidRetrievalQueryError
|
||||
from ekp_core.knowledge_access import (
|
||||
KnowledgeAccessNotFoundError,
|
||||
KnowledgeSectionOffsetError,
|
||||
)
|
||||
from ekp_core.permission_resolver import PermissionResolutionError
|
||||
from ekp_core.search import SearchConfigurationError
|
||||
from ekp_mcp.auth import (
|
||||
ConfiguredIdentityTokenVerifier,
|
||||
StaticServiceTokenVerifier,
|
||||
current_mcp_identity,
|
||||
)
|
||||
from ekp_mcp.knowledge import (
|
||||
McpDocumentResponse,
|
||||
McpDocumentSectionResponse,
|
||||
McpKnowledgeApplication,
|
||||
McpKnowledgeBaseListResponse,
|
||||
McpKnowledgeReader,
|
||||
McpSourceResponse,
|
||||
)
|
||||
from ekp_mcp.search import McpKnowledgeSearch, McpSearchApplication, McpSearchResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class McpConfigurationError(ValueError):
|
||||
"""Raised when the gateway lacks its dedicated runtime credential."""
|
||||
|
||||
|
||||
def _create_applications(
|
||||
settings: Settings,
|
||||
api_services: ApiServices | None,
|
||||
) -> tuple[McpSearchApplication, McpKnowledgeApplication]:
|
||||
if api_services is not None:
|
||||
searcher = api_services.search
|
||||
authorized_searcher = api_services.authorized_search
|
||||
knowledge_access = api_services.knowledge_access
|
||||
else:
|
||||
runtime = create_runtime_search_services(settings)
|
||||
searcher = runtime.search
|
||||
authorized_searcher = runtime.authorized_search
|
||||
knowledge_access = runtime.knowledge_access
|
||||
if knowledge_access is None:
|
||||
raise McpConfigurationError("MCP knowledge access service is required")
|
||||
return (
|
||||
McpSearchApplication(
|
||||
searcher,
|
||||
settings.mcp_required_scope,
|
||||
authorized_searcher=authorized_searcher,
|
||||
allow_legacy_public=(
|
||||
settings.mcp_auth_token is not None and not settings.mcp_identity_credentials
|
||||
),
|
||||
),
|
||||
McpKnowledgeApplication(knowledge_access, settings.mcp_required_scope),
|
||||
)
|
||||
|
||||
|
||||
def _create_token_verifier(settings: Settings) -> TokenVerifier:
|
||||
"""Build the local secret verifier while keeping external IdP injection open."""
|
||||
if settings.mcp_identity_credentials:
|
||||
return ConfiguredIdentityTokenVerifier(
|
||||
tuple(settings.mcp_identity_credentials),
|
||||
issuer=str(settings.mcp_issuer_url),
|
||||
)
|
||||
legacy_token = settings.mcp_auth_token
|
||||
if legacy_token is None or len(legacy_token.get_secret_value()) < 32:
|
||||
raise McpConfigurationError(
|
||||
"MCP_IDENTITY_CREDENTIALS or a legacy MCP_AUTH_TOKEN of at least "
|
||||
"32 characters is required"
|
||||
)
|
||||
return StaticServiceTokenVerifier(
|
||||
legacy_token,
|
||||
client_id=settings.mcp_client_id,
|
||||
subject=settings.mcp_service_identity,
|
||||
scope=settings.mcp_required_scope,
|
||||
issuer=str(settings.mcp_issuer_url),
|
||||
)
|
||||
|
||||
|
||||
def create_mcp_server(
|
||||
settings: Settings | None = None,
|
||||
*,
|
||||
api_services: ApiServices | None = None,
|
||||
search_application: McpKnowledgeSearch | None = None,
|
||||
knowledge_application: McpKnowledgeReader | None = None,
|
||||
token_verifier: TokenVerifier | None = None,
|
||||
) -> MCPServer[dict[str, object]]:
|
||||
"""Create the authenticated MCP gateway with all five V1 P0 read tools."""
|
||||
resolved_settings = settings or get_settings()
|
||||
resolved_token_verifier = token_verifier or _create_token_verifier(resolved_settings)
|
||||
if search_application is None or knowledge_application is None:
|
||||
default_search, default_knowledge = _create_applications(
|
||||
resolved_settings,
|
||||
api_services,
|
||||
)
|
||||
application = search_application or default_search
|
||||
knowledge = knowledge_application or default_knowledge
|
||||
else:
|
||||
application = search_application
|
||||
knowledge = knowledge_application
|
||||
server: MCPServer[dict[str, object]] = MCPServer(
|
||||
name="enterprise-knowledge",
|
||||
title="Enterprise Knowledge MCP Server",
|
||||
description="Authenticated enterprise knowledge search backed by canonical sources.",
|
||||
version="0.1.0",
|
||||
token_verifier=resolved_token_verifier,
|
||||
auth=AuthSettings(
|
||||
issuer_url=resolved_settings.mcp_issuer_url,
|
||||
resource_server_url=resolved_settings.mcp_public_url,
|
||||
required_scopes=[resolved_settings.mcp_required_scope],
|
||||
),
|
||||
log_level=resolved_settings.log_level,
|
||||
)
|
||||
|
||||
@server.tool(
|
||||
name="search_knowledge",
|
||||
description=(
|
||||
"Search authorized enterprise knowledge and return traceable source passages. "
|
||||
"Use for factual questions that should be grounded in indexed company documents."
|
||||
),
|
||||
structured_output=True,
|
||||
)
|
||||
async def search_knowledge(
|
||||
query: Annotated[str, Field(min_length=1, max_length=1600)],
|
||||
top_k: Annotated[int, Field(ge=1, le=64, strict=True)] = 5,
|
||||
) -> McpSearchResponse:
|
||||
"""Run the shared synchronous search pipeline outside the ASGI event loop."""
|
||||
identity = current_mcp_identity()
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(
|
||||
partial(application.search, identity, query, top_k)
|
||||
)
|
||||
except (InvalidRetrievalQueryError, SearchConfigurationError) as error:
|
||||
raise ToolError("search request is invalid for the active index profile") from error
|
||||
except (PermissionError, PermissionResolutionError) as error:
|
||||
raise ToolError("knowledge search is not authorized") from error
|
||||
except Exception:
|
||||
# Do not reflect provider, database, endpoint, or credential details
|
||||
# into MCP content or logs. Operations can correlate this generic
|
||||
# event with upstream sanitized service telemetry.
|
||||
logger.error(
|
||||
"MCP knowledge search failed",
|
||||
extra={"event": "mcp_knowledge_search_failed"},
|
||||
)
|
||||
raise ToolError("knowledge search failed") from None
|
||||
|
||||
@server.tool(
|
||||
name="list_knowledge_bases",
|
||||
description=(
|
||||
"List Knowledge Bases the current identity may read through MCP, including "
|
||||
"the count of readable documents in each space."
|
||||
),
|
||||
structured_output=True,
|
||||
)
|
||||
async def list_knowledge_bases() -> McpKnowledgeBaseListResponse:
|
||||
"""List authorized spaces without exposing ACL rows or infrastructure."""
|
||||
identity = current_mcp_identity()
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(partial(knowledge.list_knowledge_bases, identity))
|
||||
except (PermissionError, PermissionResolutionError) as error:
|
||||
raise ToolError("knowledge access is not authorized") from error
|
||||
except Exception:
|
||||
_log_knowledge_failure("mcp_list_knowledge_bases_failed")
|
||||
raise ToolError("knowledge access failed") from None
|
||||
|
||||
@server.tool(
|
||||
name="get_document",
|
||||
description=(
|
||||
"Get authorized document/version metadata and a bounded section outline. "
|
||||
"Use get_document_section for readable body text."
|
||||
),
|
||||
structured_output=True,
|
||||
)
|
||||
async def get_document(
|
||||
document_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> McpDocumentResponse:
|
||||
"""Return metadata for one authorized document without bulk source transfer."""
|
||||
identity = current_mcp_identity()
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(
|
||||
partial(
|
||||
knowledge.get_document,
|
||||
identity,
|
||||
document_id,
|
||||
document_version_id,
|
||||
)
|
||||
)
|
||||
except KnowledgeAccessNotFoundError as error:
|
||||
raise ToolError("knowledge resource is unavailable") from error
|
||||
except (PermissionError, PermissionResolutionError) as error:
|
||||
raise ToolError("knowledge access is not authorized") from error
|
||||
except Exception:
|
||||
_log_knowledge_failure("mcp_get_document_failed")
|
||||
raise ToolError("knowledge access failed") from None
|
||||
|
||||
@server.tool(
|
||||
name="get_document_section",
|
||||
description=(
|
||||
"Read one authorized canonical section in source order. Pass an anchor chunk "
|
||||
"from search/get_source and use next_offset until it is null."
|
||||
),
|
||||
structured_output=True,
|
||||
)
|
||||
async def get_document_section(
|
||||
document_id: UUID,
|
||||
anchor_chunk_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
offset: Annotated[int, Field(ge=0, le=1_000_000, strict=True)] = 0,
|
||||
max_chunks: Annotated[int, Field(ge=1, le=32, strict=True)] = 12,
|
||||
) -> McpDocumentSectionResponse:
|
||||
"""Return a bounded page of the anchor Chunk's exact section."""
|
||||
identity = current_mcp_identity()
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(
|
||||
partial(
|
||||
knowledge.get_document_section,
|
||||
identity,
|
||||
document_id,
|
||||
anchor_chunk_id,
|
||||
document_version_id,
|
||||
offset,
|
||||
max_chunks,
|
||||
)
|
||||
)
|
||||
except KnowledgeAccessNotFoundError as error:
|
||||
raise ToolError("knowledge resource is unavailable") from error
|
||||
except KnowledgeSectionOffsetError as error:
|
||||
raise ToolError("document section request is invalid") from error
|
||||
except (PermissionError, PermissionResolutionError) as error:
|
||||
raise ToolError("knowledge access is not authorized") from error
|
||||
except Exception:
|
||||
_log_knowledge_failure("mcp_get_document_section_failed")
|
||||
raise ToolError("knowledge access failed") from None
|
||||
|
||||
@server.tool(
|
||||
name="get_source",
|
||||
description=(
|
||||
"Resolve one authorized search chunk to its exact canonical text and source "
|
||||
"coordinates. This tool returns source evidence, not raw file bytes."
|
||||
),
|
||||
structured_output=True,
|
||||
)
|
||||
async def get_source(
|
||||
document_id: UUID,
|
||||
chunk_id: UUID,
|
||||
document_version_id: UUID | None = None,
|
||||
) -> McpSourceResponse:
|
||||
"""Return exact canonical source evidence for one authorized Chunk."""
|
||||
identity = current_mcp_identity()
|
||||
try:
|
||||
return await anyio.to_thread.run_sync(
|
||||
partial(
|
||||
knowledge.get_source,
|
||||
identity,
|
||||
document_id,
|
||||
chunk_id,
|
||||
document_version_id,
|
||||
)
|
||||
)
|
||||
except KnowledgeAccessNotFoundError as error:
|
||||
raise ToolError("knowledge resource is unavailable") from error
|
||||
except (PermissionError, PermissionResolutionError) as error:
|
||||
raise ToolError("knowledge access is not authorized") from error
|
||||
except Exception:
|
||||
_log_knowledge_failure("mcp_get_source_failed")
|
||||
raise ToolError("knowledge access failed") from None
|
||||
|
||||
return server
|
||||
|
||||
|
||||
def create_mcp_http_app(
|
||||
settings: Settings | None = None,
|
||||
*,
|
||||
api_services: ApiServices | None = None,
|
||||
search_application: McpKnowledgeSearch | None = None,
|
||||
knowledge_application: McpKnowledgeReader | None = None,
|
||||
token_verifier: TokenVerifier | None = None,
|
||||
) -> Starlette:
|
||||
"""Build the stateless Streamable HTTP app used by WorkBuddy and tests."""
|
||||
resolved_settings = settings or get_settings()
|
||||
server = create_mcp_server(
|
||||
resolved_settings,
|
||||
api_services=api_services,
|
||||
search_application=search_application,
|
||||
knowledge_application=knowledge_application,
|
||||
token_verifier=token_verifier,
|
||||
)
|
||||
return server.streamable_http_app(
|
||||
streamable_http_path=resolved_settings.mcp_path,
|
||||
json_response=True,
|
||||
stateless_http=True,
|
||||
max_request_body_size=resolved_settings.mcp_max_request_body_bytes,
|
||||
host=resolved_settings.mcp_host,
|
||||
transport_security=TransportSecuritySettings(
|
||||
enable_dns_rebinding_protection=True,
|
||||
allowed_hosts=list(resolved_settings.mcp_allowed_hosts),
|
||||
allowed_origins=list(resolved_settings.mcp_allowed_origins),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _log_knowledge_failure(event: str) -> None:
|
||||
"""Emit only an allowlisted event name for non-search MCP failures."""
|
||||
logger.error("MCP knowledge access failed", extra={"event": event})
|
||||
36
hy4/_kp/deploy/docker/docker-bake.hcl
Normal file
36
hy4/_kp/deploy/docker/docker-bake.hcl
Normal file
@ -0,0 +1,36 @@
|
||||
# Reproducible target names consumed by local and CI buildx invocations.
|
||||
variable "IMAGE_REGISTRY" {
|
||||
default = "knowledge-platform"
|
||||
}
|
||||
|
||||
variable "IMAGE_TAG" {
|
||||
default = "dev"
|
||||
}
|
||||
|
||||
group "default" {
|
||||
targets = ["api", "mcp", "worker"]
|
||||
}
|
||||
|
||||
target "common" {
|
||||
context = "."
|
||||
dockerfile = "Dockerfile"
|
||||
platforms = ["linux/amd64", "linux/arm64"]
|
||||
}
|
||||
|
||||
target "api" {
|
||||
inherits = ["common"]
|
||||
target = "api"
|
||||
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-api:${IMAGE_TAG}"]
|
||||
}
|
||||
|
||||
target "mcp" {
|
||||
inherits = ["common"]
|
||||
target = "mcp"
|
||||
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-mcp:${IMAGE_TAG}"]
|
||||
}
|
||||
|
||||
target "worker" {
|
||||
inherits = ["common"]
|
||||
target = "worker"
|
||||
tags = ["${IMAGE_REGISTRY}/enterprise-knowledge-worker:${IMAGE_TAG}"]
|
||||
}
|
||||
73
hy4/_kp/deploy/helm/environments/values.example.yaml
Normal file
73
hy4/_kp/deploy/helm/environments/values.example.yaml
Normal file
@ -0,0 +1,73 @@
|
||||
# Copy this file outside the repository for one environment. It contains only
|
||||
# non-sensitive settings and Kubernetes Secret references; never add values for
|
||||
# DATABASE_URL, RABBITMQ_URL, provider keys, passwords or MCP_AUTH_TOKEN here.
|
||||
existingSecret: knowledge-platform-runtime
|
||||
|
||||
images:
|
||||
api:
|
||||
repository: registry.example.internal/knowledge/enterprise-knowledge-api
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
mcp:
|
||||
repository: registry.example.internal/knowledge/enterprise-knowledge-mcp
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
worker:
|
||||
repository: registry.example.internal/knowledge/enterprise-knowledge-worker
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
edge:
|
||||
repository: registry.example.internal/mirror/nginx
|
||||
tag: "1.29.1-alpine"
|
||||
digest: ""
|
||||
|
||||
podDisruptionBudget:
|
||||
enabled: true
|
||||
apiMinAvailable: 1
|
||||
mcpMinAvailable: 1
|
||||
edgeMinAvailable: 1
|
||||
|
||||
observability:
|
||||
metrics:
|
||||
enabled: true
|
||||
path: /metrics
|
||||
|
||||
ingress:
|
||||
enabled: true
|
||||
className: nginx
|
||||
api:
|
||||
host: knowledge.example.internal
|
||||
tlsSecretName: knowledge-platform-api-tls
|
||||
mcp:
|
||||
host: mcp.example.internal
|
||||
tlsSecretName: knowledge-platform-mcp-tls
|
||||
|
||||
# Use this mode instead of Ingress on clusters whose external gateway already
|
||||
# targets one NodePort. Stage with an unused port before taking over the live one.
|
||||
edge:
|
||||
enabled: false
|
||||
host: knowledge.example.internal
|
||||
service:
|
||||
nodePort: 0
|
||||
|
||||
config:
|
||||
objectStorage:
|
||||
endpoint: https://s3.example.internal
|
||||
bucket: knowledge-platform
|
||||
vectorStore:
|
||||
uri: oceanbase.example.internal:2881
|
||||
user: knowledge-platform
|
||||
database: knowledge_platform
|
||||
mcp:
|
||||
publicUrl: https://mcp.example.internal/mcp
|
||||
issuerUrl: https://mcp.example.internal
|
||||
allowedHosts:
|
||||
- mcp.example.internal
|
||||
allowedOrigins:
|
||||
- https://mcp.example.internal
|
||||
|
||||
# External RabbitMQ is the production default. If this local dependency is
|
||||
# explicitly enabled for a demo cluster, create its Secret before deployment.
|
||||
rabbitmq:
|
||||
enabled: false
|
||||
existingSecret: knowledge-platform-rabbitmq
|
||||
15
hy4/_kp/deploy/helm/knowledge-platform/Chart.yaml
Normal file
15
hy4/_kp/deploy/helm/knowledge-platform/Chart.yaml
Normal file
@ -0,0 +1,15 @@
|
||||
apiVersion: v2
|
||||
name: knowledge-platform
|
||||
description: Enterprise Knowledge Platform public-knowledge deployment
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "0.1.0"
|
||||
|
||||
# This local dependency is disabled by default. Production deployments should
|
||||
# use an externally managed RabbitMQ service with an independently owned data
|
||||
# lifecycle; the subchart exists only for explicit development/demo use.
|
||||
dependencies:
|
||||
- name: rabbitmq
|
||||
version: 0.1.0
|
||||
repository: file://charts/rabbitmq
|
||||
condition: rabbitmq.enabled
|
||||
@ -0,0 +1,6 @@
|
||||
apiVersion: v2
|
||||
name: rabbitmq
|
||||
description: Minimal single-node RabbitMQ dependency for local demo clusters only
|
||||
type: application
|
||||
version: 0.1.0
|
||||
appVersion: "4.1.4"
|
||||
@ -0,0 +1,11 @@
|
||||
{{- define "rabbitmq.fullname" -}}
|
||||
{{- printf "%s-rabbitmq" .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "rabbitmq.image" -}}
|
||||
{{- if .Values.image.digest -}}
|
||||
{{- printf "%s@%s" .Values.image.repository .Values.image.digest -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s:%s" .Values.image.repository .Values.image.tag -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
@ -0,0 +1,16 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "rabbitmq.fullname" . }}
|
||||
labels:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
spec:
|
||||
clusterIP: None
|
||||
ports:
|
||||
- name: amqp
|
||||
port: {{ .Values.service.port }}
|
||||
targetPort: amqp
|
||||
selector:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
@ -0,0 +1,8 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "rabbitmq.fullname" . }}
|
||||
labels:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
automountServiceAccountToken: false
|
||||
@ -0,0 +1,109 @@
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: {{ include "rabbitmq.fullname" . }}
|
||||
labels:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
spec:
|
||||
serviceName: {{ include "rabbitmq.fullname" . }}
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: rabbitmq
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "rabbitmq.fullname" . }}
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 999
|
||||
runAsGroup: 999
|
||||
fsGroup: 999
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: rabbitmq
|
||||
image: {{ include "rabbitmq.image" . | quote }}
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
ports:
|
||||
- name: amqp
|
||||
containerPort: 5672
|
||||
env:
|
||||
- name: RABBITMQ_DEFAULT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: {{ .Values.secretKeys.username }}
|
||||
- name: RABBITMQ_DEFAULT_PASS
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: {{ .Values.secretKeys.password }}
|
||||
- name: RABBITMQ_ERLANG_COOKIE
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: {{ .Values.secretKeys.erlangCookie }}
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["rabbitmq-diagnostics", "-q", "ping"]
|
||||
periodSeconds: 5
|
||||
failureThreshold: 24
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["rabbitmq-diagnostics", "-q", "check_running"]
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["rabbitmq-diagnostics", "-q", "ping"]
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/lib/rabbitmq
|
||||
- name: config
|
||||
mountPath: /etc/rabbitmq/conf.d
|
||||
- name: log
|
||||
mountPath: /var/log/rabbitmq
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: config
|
||||
emptyDir: {}
|
||||
- name: log
|
||||
emptyDir: {}
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
{{- if not .Values.persistence.enabled }}
|
||||
- name: data
|
||||
emptyDir: {}
|
||||
{{- end }}
|
||||
{{- if .Values.persistence.enabled }}
|
||||
volumeClaimTemplates:
|
||||
- metadata:
|
||||
name: data
|
||||
spec:
|
||||
accessModes: [{{ .Values.persistence.accessMode | quote }}]
|
||||
{{- if .Values.persistence.storageClass }}
|
||||
storageClassName: {{ .Values.persistence.storageClass | quote }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.persistence.size }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,69 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft-07/schema#",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "existingSecret", "secretKeys", "image", "service", "persistence", "resources"],
|
||||
"properties": {
|
||||
"global": {"type": "object"},
|
||||
"enabled": {"type": "boolean"},
|
||||
"existingSecret": {"type": "string", "minLength": 1},
|
||||
"secretKeys": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["username", "password", "erlangCookie"],
|
||||
"properties": {
|
||||
"username": {"type": "string", "minLength": 1},
|
||||
"password": {"type": "string", "minLength": 1},
|
||||
"erlangCookie": {"type": "string", "minLength": 1}
|
||||
}
|
||||
},
|
||||
"image": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repository", "tag", "digest", "pullPolicy"],
|
||||
"properties": {
|
||||
"repository": {"type": "string", "minLength": 1},
|
||||
"tag": {"type": "string", "minLength": 1, "not": {"enum": ["latest"]}},
|
||||
"digest": {"type": "string", "pattern": "^$|^sha256:[a-f0-9]{64}$"},
|
||||
"pullPolicy": {"enum": ["Always", "IfNotPresent", "Never"]}
|
||||
}
|
||||
},
|
||||
"service": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["port"],
|
||||
"properties": {"port": {"type": "integer", "minimum": 1, "maximum": 65535}}
|
||||
},
|
||||
"persistence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "storageClass", "accessMode", "size"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"storageClass": {"type": "string"},
|
||||
"accessMode": {"enum": ["ReadWriteOnce", "ReadWriteMany"]},
|
||||
"size": {"type": "string", "minLength": 2}
|
||||
}
|
||||
},
|
||||
"resources": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["requests", "limits"],
|
||||
"properties": {
|
||||
"requests": {"$ref": "#/definitions/resourcePair"},
|
||||
"limits": {"$ref": "#/definitions/resourcePair"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"resourcePair": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["cpu", "memory"],
|
||||
"properties": {
|
||||
"cpu": {"type": ["string", "number"]},
|
||||
"memory": {"type": "string", "minLength": 2}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,25 @@
|
||||
enabled: false
|
||||
existingSecret: knowledge-platform-rabbitmq
|
||||
secretKeys:
|
||||
username: username
|
||||
password: password
|
||||
erlangCookie: erlang-cookie
|
||||
image:
|
||||
repository: rabbitmq
|
||||
tag: 4.1.4-alpine
|
||||
digest: sha256:b736d649308e1b3e1a116c3f36986b605ee3d03e88f10166be2900083d2e63f2
|
||||
pullPolicy: IfNotPresent
|
||||
service:
|
||||
port: 5672
|
||||
persistence:
|
||||
enabled: true
|
||||
storageClass: ""
|
||||
accessMode: ReadWriteOnce
|
||||
size: 8Gi
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
17
hy4/_kp/deploy/helm/knowledge-platform/templates/NOTES.txt
Normal file
17
hy4/_kp/deploy/helm/knowledge-platform/templates/NOTES.txt
Normal file
@ -0,0 +1,17 @@
|
||||
Enterprise Knowledge Platform {{ .Chart.AppVersion }} is managed by Helm release {{ .Release.Name }}.
|
||||
|
||||
Runtime credentials are read only from Secret {{ .Values.existingSecret }}.
|
||||
{{- if .Values.ingress.enabled }}
|
||||
Admin/API: https://{{ .Values.ingress.api.host }}{{ .Values.ingress.api.path }}
|
||||
MCP: https://{{ .Values.ingress.mcp.host }}{{ .Values.ingress.mcp.path }}
|
||||
{{- else if .Values.edge.enabled }}
|
||||
Host: https://{{ .Values.edge.host }}
|
||||
Edge Service: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
NodePort: {{ if gt (int .Values.edge.service.nodePort) 0 }}{{ .Values.edge.service.nodePort }}{{ else }}allocated by Kubernetes{{ end }}
|
||||
Routes: /mcp -> MCP; all other paths -> Admin/API
|
||||
{{- else }}
|
||||
Ingress and edge are disabled. Enable one entry mode before remote use.
|
||||
{{- end }}
|
||||
|
||||
Inspect rollout state without printing environment variables:
|
||||
kubectl -n {{ .Release.Namespace }} get deploy,job,pod
|
||||
126
hy4/_kp/deploy/helm/knowledge-platform/templates/_helpers.tpl
Normal file
126
hy4/_kp/deploy/helm/knowledge-platform/templates/_helpers.tpl
Normal file
@ -0,0 +1,126 @@
|
||||
{{/* Stable resource name helpers keep every workload under one release identity. */}}
|
||||
{{- define "knowledge-platform.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.fullname" -}}
|
||||
{{- if .Values.fullnameOverride -}}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- $name := include "knowledge-platform.name" . -}}
|
||||
{{- if contains $name .Release.Name -}}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.labels" -}}
|
||||
helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | quote }}
|
||||
app.kubernetes.io/name: {{ include "knowledge-platform.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "knowledge-platform.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create -}}
|
||||
{{- default (include "knowledge-platform.fullname" .) .Values.serviceAccount.name -}}
|
||||
{{- else -}}
|
||||
{{- required "serviceAccount.name is required when serviceAccount.create=false" .Values.serviceAccount.name -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* A supplied digest takes precedence over the immutable release tag. */}}
|
||||
{{- define "knowledge-platform.image" -}}
|
||||
{{- if .digest -}}
|
||||
{{- printf "%s@%s" .repository .digest -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s:%s" .repository .tag -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.configMapName" -}}
|
||||
{{- printf "%s-runtime" (include "knowledge-platform.fullname" .) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* Cross-field rules complement values.schema.json's structural checks. */}}
|
||||
{{- define "knowledge-platform.validateValues" -}}
|
||||
{{- if lt (float64 .Values.config.ingestion.retryBackoffMaxSeconds) (float64 .Values.config.ingestion.retryBackoffSeconds) -}}
|
||||
{{- fail "config.ingestion.retryBackoffMaxSeconds must be >= retryBackoffSeconds" -}}
|
||||
{{- end -}}
|
||||
{{- if lt (float64 .Values.config.ingestion.batchMaxTotalBytes) (float64 .Values.config.ingestion.maxUploadBytes) -}}
|
||||
{{- fail "config.ingestion.batchMaxTotalBytes must be >= maxUploadBytes" -}}
|
||||
{{- end -}}
|
||||
{{- if ge (float64 .Values.config.chunking.overlapTokens) (float64 .Values.config.chunking.maxTokens) -}}
|
||||
{{- fail "config.chunking.overlapTokens must be < maxTokens" -}}
|
||||
{{- end -}}
|
||||
{{- if gt (float64 .Values.config.retrieval.searchTopK) (float64 .Values.config.retrieval.hybridTopN) -}}
|
||||
{{- fail "config.retrieval.searchTopK must be <= hybridTopN" -}}
|
||||
{{- end -}}
|
||||
{{- if eq .Values.config.parser.provider "mineru" -}}
|
||||
{{- $mineruEndpoint := required "config.parser.mineruEndpoint is required when provider=mineru" .Values.config.parser.mineruEndpoint -}}
|
||||
{{- if le (float64 .Values.config.ingestion.workerLeaseSeconds) (float64 .Values.config.parser.mineruTimeoutSeconds) -}}
|
||||
{{- fail "config.ingestion.workerLeaseSeconds must exceed config.parser.mineruTimeoutSeconds" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- if and .Values.ingress.enabled .Values.edge.enabled -}}
|
||||
{{- fail "ingress.enabled and edge.enabled cannot both be true" -}}
|
||||
{{- end -}}
|
||||
{{- if .Values.podDisruptionBudget.enabled -}}
|
||||
{{- if ge (int .Values.podDisruptionBudget.apiMinAvailable) (int .Values.api.replicaCount) -}}
|
||||
{{- fail "podDisruptionBudget.apiMinAvailable must be < api.replicaCount" -}}
|
||||
{{- end -}}
|
||||
{{- if ge (int .Values.podDisruptionBudget.mcpMinAvailable) (int .Values.mcp.replicaCount) -}}
|
||||
{{- fail "podDisruptionBudget.mcpMinAvailable must be < mcp.replicaCount" -}}
|
||||
{{- end -}}
|
||||
{{- if and .Values.edge.enabled (ge (int .Values.podDisruptionBudget.edgeMinAvailable) (int .Values.edge.replicaCount)) -}}
|
||||
{{- fail "podDisruptionBudget.edgeMinAvailable must be < edge.replicaCount" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.podSecurityContext" -}}
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
fsGroupChangePolicy: OnRootMismatch
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.containerSecurityContext" -}}
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.envFrom" -}}
|
||||
- configMapRef:
|
||||
name: {{ include "knowledge-platform.configMapName" . }}
|
||||
- secretRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.tmpVolume" -}}
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 1Gi
|
||||
{{- end -}}
|
||||
|
||||
{{- define "knowledge-platform.tmpMount" -}}
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
{{- end -}}
|
||||
@ -0,0 +1,80 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-api
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: api
|
||||
spec:
|
||||
replicas: {{ .Values.api.replicaCount }}
|
||||
minReadySeconds: 5
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: api
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: api
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
|
||||
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
|
||||
{{- if .Values.observability.metrics.enabled }}
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "8000"
|
||||
prometheus.io/path: {{ .Values.observability.metrics.path | quote }}
|
||||
{{- end }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: api
|
||||
image: {{ include "knowledge-platform.image" .Values.images.api | quote }}
|
||||
imagePullPolicy: {{ .Values.images.api.pullPolicy }}
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8000
|
||||
protocol: TCP
|
||||
envFrom:
|
||||
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
{{- toYaml .Values.api.probes.startup | nindent 12 }}
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
{{- toYaml .Values.api.probes.readiness | nindent 12 }}
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
{{- toYaml .Values.api.probes.liveness | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.api.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
@ -0,0 +1,17 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-api
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: api
|
||||
spec:
|
||||
type: {{ .Values.api.service.type }}
|
||||
ports:
|
||||
- name: http
|
||||
port: {{ .Values.api.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
selector:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: api
|
||||
@ -0,0 +1,84 @@
|
||||
{{- include "knowledge-platform.validateValues" . }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.configMapName" . }}
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
data:
|
||||
# Only the explicitly enumerated non-sensitive settings are rendered here.
|
||||
APP_ENV: "production"
|
||||
APP_NAME: {{ .Values.config.appName | quote }}
|
||||
LOG_LEVEL: {{ .Values.config.logLevel | quote }}
|
||||
API_HOST: "0.0.0.0"
|
||||
API_PORT: "8000"
|
||||
API_PREFIX: "/api/v1"
|
||||
DATABASE_POOL_SIZE: {{ .Values.config.database.poolSize | quote }}
|
||||
DATABASE_MAX_OVERFLOW: {{ .Values.config.database.maxOverflow | quote }}
|
||||
DATABASE_POOL_RECYCLE_SECONDS: {{ .Values.config.database.poolRecycleSeconds | quote }}
|
||||
DATABASE_POOL_TIMEOUT_SECONDS: {{ .Values.config.database.poolTimeoutSeconds | quote }}
|
||||
DATABASE_CONNECT_TIMEOUT_SECONDS: {{ .Values.config.database.connectTimeoutSeconds | quote }}
|
||||
INGESTION_MAX_ATTEMPTS: {{ .Values.config.ingestion.maxAttempts | quote }}
|
||||
INGESTION_RETRY_BACKOFF_SECONDS: {{ .Values.config.ingestion.retryBackoffSeconds | quote }}
|
||||
INGESTION_RETRY_BACKOFF_MAX_SECONDS: {{ .Values.config.ingestion.retryBackoffMaxSeconds | quote }}
|
||||
INGESTION_WORKER_LEASE_SECONDS: {{ .Values.config.ingestion.workerLeaseSeconds | quote }}
|
||||
INGESTION_RECOVERY_INTERVAL_SECONDS: {{ .Values.config.ingestion.recoveryIntervalSeconds | quote }}
|
||||
COMPILATION_MAX_ATTEMPTS: {{ .Values.config.compilation.maxAttempts | quote }}
|
||||
COMPILATION_RETRY_BACKOFF_SECONDS: {{ .Values.config.compilation.retryBackoffSeconds | quote }}
|
||||
COMPILATION_RETRY_BACKOFF_MAX_SECONDS: {{ .Values.config.compilation.retryBackoffMaxSeconds | quote }}
|
||||
COMPILATION_WORKER_LEASE_SECONDS: {{ .Values.config.compilation.workerLeaseSeconds | quote }}
|
||||
COMPILATION_RECOVERY_INTERVAL_SECONDS: {{ .Values.config.compilation.recoveryIntervalSeconds | quote }}
|
||||
BATCH_MAX_FILES: {{ .Values.config.ingestion.batchMaxFiles | quote }}
|
||||
BATCH_MAX_TOTAL_BYTES: {{ printf "%.0f" .Values.config.ingestion.batchMaxTotalBytes | quote }}
|
||||
MAX_UPLOAD_BYTES: {{ printf "%.0f" .Values.config.ingestion.maxUploadBytes | quote }}
|
||||
OBJECT_STORAGE_PROVIDER: {{ .Values.config.objectStorage.provider | quote }}
|
||||
BOS_ENDPOINT: {{ .Values.config.objectStorage.endpoint | quote }}
|
||||
BOS_BUCKET: {{ .Values.config.objectStorage.bucket | quote }}
|
||||
BOS_KEY_PREFIX: {{ .Values.config.objectStorage.keyPrefix | quote }}
|
||||
BOS_TIMEOUT_SECONDS: {{ .Values.config.objectStorage.timeoutSeconds | quote }}
|
||||
PARSER_PROVIDER: {{ .Values.config.parser.provider | quote }}
|
||||
MINERU_ENDPOINT: {{ .Values.config.parser.mineruEndpoint | quote }}
|
||||
MINERU_BACKEND: {{ .Values.config.parser.mineruBackend | quote }}
|
||||
MINERU_PARSE_METHOD: {{ .Values.config.parser.mineruParseMethod | quote }}
|
||||
MINERU_LANGUAGE: {{ .Values.config.parser.mineruLanguage | quote }}
|
||||
MINERU_EFFORT: {{ .Values.config.parser.mineruEffort | quote }}
|
||||
MINERU_TIMEOUT_SECONDS: {{ .Values.config.parser.mineruTimeoutSeconds | quote }}
|
||||
MINERU_LOW_OCR_CONFIDENCE_THRESHOLD: {{ .Values.config.parser.lowOcrConfidenceThreshold | quote }}
|
||||
CHUNK_MAX_TOKENS: {{ .Values.config.chunking.maxTokens | quote }}
|
||||
CHUNK_OVERLAP_TOKENS: {{ .Values.config.chunking.overlapTokens | quote }}
|
||||
CHUNK_MAX_CHARACTERS: {{ .Values.config.chunking.maxCharacters | quote }}
|
||||
EMBEDDING_PROVIDER: {{ .Values.config.embedding.provider | quote }}
|
||||
EMBEDDING_MODEL: {{ .Values.config.embedding.model | quote }}
|
||||
EMBEDDING_ENDPOINT: {{ .Values.config.embedding.endpoint | quote }}
|
||||
EMBEDDING_TIMEOUT_SECONDS: {{ .Values.config.embedding.timeoutSeconds | quote }}
|
||||
VECTOR_STORE_PROVIDER: {{ .Values.config.vectorStore.provider | quote }}
|
||||
OCEANBASE_VECTOR_URI: {{ .Values.config.vectorStore.uri | quote }}
|
||||
OCEANBASE_VECTOR_USER: {{ .Values.config.vectorStore.user | quote }}
|
||||
OCEANBASE_VECTOR_DATABASE: {{ .Values.config.vectorStore.database | quote }}
|
||||
OCEANBASE_VECTOR_TABLE: {{ .Values.config.vectorStore.table | quote }}
|
||||
OCEANBASE_VECTOR_INDEX: {{ .Values.config.vectorStore.index | quote }}
|
||||
OCEANBASE_VECTOR_BATCH_SIZE: {{ .Values.config.vectorStore.batchSize | quote }}
|
||||
OCEANBASE_VECTOR_TIMEOUT_SECONDS: {{ .Values.config.vectorStore.timeoutSeconds | quote }}
|
||||
INDEX_PROFILE_ID: {{ .Values.config.retrieval.indexProfileId | quote }}
|
||||
DENSE_TOP_N: {{ .Values.config.retrieval.denseTopN | quote }}
|
||||
SPARSE_TOP_N: {{ .Values.config.retrieval.sparseTopN | quote }}
|
||||
HYBRID_TOP_N: {{ .Values.config.retrieval.hybridTopN | quote }}
|
||||
HYBRID_RRF_K: {{ .Values.config.retrieval.hybridRrfK | quote }}
|
||||
HYBRID_DENSE_WEIGHT: {{ .Values.config.retrieval.hybridDenseWeight | quote }}
|
||||
HYBRID_SPARSE_WEIGHT: {{ .Values.config.retrieval.hybridSparseWeight | quote }}
|
||||
SEARCH_TOP_K: {{ .Values.config.retrieval.searchTopK | quote }}
|
||||
RERANKER_PROVIDER: {{ .Values.config.reranker.provider | quote }}
|
||||
RERANKER_MODEL: {{ .Values.config.reranker.model | quote }}
|
||||
RERANKER_ENDPOINT: {{ .Values.config.reranker.endpoint | quote }}
|
||||
RERANKER_TIMEOUT_SECONDS: {{ .Values.config.reranker.timeoutSeconds | quote }}
|
||||
MCP_HOST: "0.0.0.0"
|
||||
MCP_PORT: "8766"
|
||||
MCP_PATH: "/mcp"
|
||||
MCP_PUBLIC_URL: {{ .Values.config.mcp.publicUrl | quote }}
|
||||
MCP_ISSUER_URL: {{ .Values.config.mcp.issuerUrl | quote }}
|
||||
MCP_CLIENT_ID: {{ .Values.config.mcp.clientId | quote }}
|
||||
MCP_SERVICE_IDENTITY: {{ .Values.config.mcp.serviceIdentity | quote }}
|
||||
MCP_REQUIRED_SCOPE: {{ .Values.config.mcp.requiredScope | quote }}
|
||||
MCP_ALLOWED_HOSTS: {{ .Values.config.mcp.allowedHosts | toJson | quote }}
|
||||
MCP_ALLOWED_ORIGINS: {{ .Values.config.mcp.allowedOrigins | toJson | quote }}
|
||||
MCP_MAX_REQUEST_BODY_BYTES: {{ printf "%.0f" .Values.config.mcp.maxRequestBodyBytes | quote }}
|
||||
@ -0,0 +1,117 @@
|
||||
{{- if .Values.edge.enabled }}
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: edge
|
||||
data:
|
||||
# The edge is intentionally limited to path routing. Authentication and all
|
||||
# business behavior remain in the API and MCP applications behind it.
|
||||
nginx.conf: |
|
||||
pid /tmp/nginx.pid;
|
||||
error_log /dev/stderr info;
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
client_body_temp_path /tmp/client_temp;
|
||||
proxy_temp_path /tmp/proxy_temp;
|
||||
fastcgi_temp_path /tmp/fastcgi_temp;
|
||||
uwsgi_temp_path /tmp/uwsgi_temp;
|
||||
scgi_temp_path /tmp/scgi_temp;
|
||||
|
||||
server_tokens off;
|
||||
client_max_body_size {{ .Values.edge.clientMaxBodySize }};
|
||||
|
||||
map $http_x_request_id $correlation_id {
|
||||
default $http_x_request_id;
|
||||
"" $request_id;
|
||||
}
|
||||
map $http_x_forwarded_proto $forwarded_proto {
|
||||
default $http_x_forwarded_proto;
|
||||
"" $scheme;
|
||||
}
|
||||
|
||||
log_format edge_json escape=json
|
||||
'{"timestamp":"$time_iso8601","event":"edge_access",'
|
||||
'"request_id":"$correlation_id","method":"$request_method",'
|
||||
'"path":"$uri","status":$status,"duration_seconds":$request_time, '
|
||||
'"upstream_status":"$upstream_status"}';
|
||||
access_log /dev/stdout edge_json;
|
||||
|
||||
upstream api_backend {
|
||||
server {{ include "knowledge-platform.fullname" . }}-api:{{ .Values.api.service.port }};
|
||||
keepalive 32;
|
||||
}
|
||||
upstream mcp_backend {
|
||||
server {{ include "knowledge-platform.fullname" . }}-mcp:{{ .Values.mcp.service.port }};
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
|
||||
location = /edge-healthz {
|
||||
access_log off;
|
||||
default_type text/plain;
|
||||
return 200 "ok\n";
|
||||
}
|
||||
|
||||
# The public health endpoint follows API readiness so an external load
|
||||
# balancer cannot direct traffic to an edge with no usable API backend.
|
||||
location = /healthz {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Request-ID $correlation_id;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $forwarded_proto;
|
||||
}
|
||||
|
||||
# Streamable HTTP may keep a request open, so buffering is disabled only
|
||||
# for the MCP route while the gateway keeps ownership of authentication.
|
||||
location = /mcp {
|
||||
proxy_pass http://mcp_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
proxy_read_timeout {{ .Values.edge.proxyReadTimeoutSeconds }}s;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Request-ID $correlation_id;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $forwarded_proto;
|
||||
}
|
||||
|
||||
location ^~ /mcp/ {
|
||||
proxy_pass http://mcp_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
proxy_read_timeout {{ .Values.edge.proxyReadTimeoutSeconds }}s;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Request-ID $correlation_id;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $forwarded_proto;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://api_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Request-ID $correlation_id;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $forwarded_proto;
|
||||
}
|
||||
}
|
||||
}
|
||||
{{- end }}
|
||||
@ -0,0 +1,81 @@
|
||||
{{- if .Values.edge.enabled }}
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: edge
|
||||
spec:
|
||||
replicas: {{ .Values.edge.replicaCount }}
|
||||
minReadySeconds: 5
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: edge
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: edge
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
checksum/edge-config: {{ include (print $.Template.BasePath "/edge-configmap.yaml") . | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: edge
|
||||
image: {{ include "knowledge-platform.image" .Values.images.edge | quote }}
|
||||
imagePullPolicy: {{ .Values.images.edge.pullPolicy }}
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 8080
|
||||
protocol: TCP
|
||||
startupProbe:
|
||||
httpGet:
|
||||
path: /edge-healthz
|
||||
port: http
|
||||
{{- toYaml .Values.edge.probes.startup | nindent 12 }}
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: http
|
||||
{{- toYaml .Values.edge.probes.readiness | nindent 12 }}
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /edge-healthz
|
||||
port: http
|
||||
{{- toYaml .Values.edge.probes.liveness | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.edge.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: edge-config
|
||||
mountPath: /etc/nginx/nginx.conf
|
||||
subPath: nginx.conf
|
||||
readOnly: true
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
- name: edge-config
|
||||
configMap:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,22 @@
|
||||
{{- if .Values.edge.enabled }}
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: edge
|
||||
spec:
|
||||
type: {{ .Values.edge.service.type }}
|
||||
ports:
|
||||
- name: http
|
||||
port: {{ .Values.edge.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
{{- if gt (int .Values.edge.service.nodePort) 0 }}
|
||||
nodePort: {{ .Values.edge.service.nodePort }}
|
||||
{{- end }}
|
||||
selector:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: edge
|
||||
{{- end }}
|
||||
@ -0,0 +1,42 @@
|
||||
{{- if .Values.ingress.enabled }}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
ingressClassName: {{ .Values.ingress.className }}
|
||||
tls:
|
||||
- hosts:
|
||||
- {{ .Values.ingress.api.host | quote }}
|
||||
secretName: {{ .Values.ingress.api.tlsSecretName }}
|
||||
- hosts:
|
||||
- {{ .Values.ingress.mcp.host | quote }}
|
||||
secretName: {{ .Values.ingress.mcp.tlsSecretName }}
|
||||
rules:
|
||||
- host: {{ .Values.ingress.api.host | quote }}
|
||||
http:
|
||||
paths:
|
||||
- path: {{ .Values.ingress.api.path }}
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-api
|
||||
port:
|
||||
number: {{ .Values.api.service.port }}
|
||||
- host: {{ .Values.ingress.mcp.host | quote }}
|
||||
http:
|
||||
paths:
|
||||
- path: {{ .Values.ingress.mcp.path }}
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-mcp
|
||||
port:
|
||||
number: {{ .Values.mcp.service.port }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,72 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-mcp
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
spec:
|
||||
replicas: {{ .Values.mcp.replicaCount }}
|
||||
minReadySeconds: 5
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
|
||||
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: mcp
|
||||
image: {{ include "knowledge-platform.image" .Values.images.mcp | quote }}
|
||||
imagePullPolicy: {{ .Values.images.mcp.pullPolicy }}
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
ports:
|
||||
- name: mcp
|
||||
containerPort: 8766
|
||||
protocol: TCP
|
||||
envFrom:
|
||||
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
|
||||
startupProbe:
|
||||
tcpSocket:
|
||||
port: mcp
|
||||
{{- toYaml .Values.mcp.probes.startup | nindent 12 }}
|
||||
readinessProbe:
|
||||
tcpSocket:
|
||||
port: mcp
|
||||
{{- toYaml .Values.mcp.probes.readiness | nindent 12 }}
|
||||
livenessProbe:
|
||||
tcpSocket:
|
||||
port: mcp
|
||||
{{- toYaml .Values.mcp.probes.liveness | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.mcp.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
@ -0,0 +1,17 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-mcp
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
spec:
|
||||
type: {{ .Values.mcp.service.type }}
|
||||
ports:
|
||||
- name: mcp
|
||||
port: {{ .Values.mcp.service.port }}
|
||||
targetPort: mcp
|
||||
protocol: TCP
|
||||
selector:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
@ -0,0 +1,59 @@
|
||||
{{- if .Values.migration.enabled }}
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-migration
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: migration
|
||||
annotations:
|
||||
# Helm waits for pre-install/pre-upgrade hooks. A failed migration blocks
|
||||
# the application rollout, and the previous Job remains until the next run.
|
||||
helm.sh/hook: pre-install,pre-upgrade
|
||||
helm.sh/hook-weight: "-10"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
spec:
|
||||
backoffLimit: {{ .Values.migration.backoffLimit }}
|
||||
activeDeadlineSeconds: {{ .Values.migration.activeDeadlineSeconds }}
|
||||
ttlSecondsAfterFinished: {{ .Values.migration.ttlSecondsAfterFinished }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: migration
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# Pre-install hooks run before the release ServiceAccount exists. The
|
||||
# namespace default account is used with token mounting disabled; the
|
||||
# migration needs no Kubernetes API permissions.
|
||||
serviceAccountName: default
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: migration
|
||||
image: {{ include "knowledge-platform.image" .Values.images.api | quote }}
|
||||
imagePullPolicy: {{ .Values.images.api.pullPolicy }}
|
||||
command: ["alembic"]
|
||||
args: ["upgrade", "head"]
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
env:
|
||||
- name: APP_ENV
|
||||
value: production
|
||||
- name: DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: {{ .Values.existingSecret }}
|
||||
key: DATABASE_URL
|
||||
resources:
|
||||
{{- toYaml .Values.migration.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,49 @@
|
||||
{{- if .Values.podDisruptionBudget.enabled }}
|
||||
# DEPLOY-003 protects both user-facing paths during voluntary disruptions. PDBs
|
||||
# do not cover involuntary failures, which remain handled by replica recovery.
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-api
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: api
|
||||
spec:
|
||||
minAvailable: {{ .Values.podDisruptionBudget.apiMinAvailable }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: api
|
||||
---
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-mcp
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
spec:
|
||||
minAvailable: {{ .Values.podDisruptionBudget.mcpMinAvailable }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: mcp
|
||||
{{- if .Values.edge.enabled }}
|
||||
---
|
||||
# The edge proxy is the single external path on clusters without Ingress, so it
|
||||
# receives the same voluntary-disruption protection as API and MCP.
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-edge
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: edge
|
||||
spec:
|
||||
minAvailable: {{ .Values.podDisruptionBudget.edgeMinAvailable }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: edge
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,71 @@
|
||||
{{- if .Values.scheduler.enabled }}
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-scheduler
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: scheduler
|
||||
spec:
|
||||
# A single Celery Beat owns periodic durable-Job recovery. Recreate prevents
|
||||
# two schedulers from overlapping during upgrades.
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: scheduler
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: scheduler
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
|
||||
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: scheduler
|
||||
image: {{ include "knowledge-platform.image" .Values.images.worker | quote }}
|
||||
imagePullPolicy: {{ .Values.images.worker.pullPolicy }}
|
||||
workingDir: /tmp
|
||||
command: ["celery"]
|
||||
args:
|
||||
- --app=ekp_api.worker:celery_app
|
||||
- beat
|
||||
- --loglevel=INFO
|
||||
- --schedule=/tmp/celerybeat-schedule
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
envFrom:
|
||||
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
{{- toYaml .Values.scheduler.probes.startup | nindent 12 }}
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
{{- toYaml .Values.scheduler.probes.liveness | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.scheduler.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
{{- end }}
|
||||
@ -0,0 +1,13 @@
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
automountServiceAccountToken: false
|
||||
{{- end }}
|
||||
@ -0,0 +1,29 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-preflight-references
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
# The Make preflight renders this test hook to discover only resource
|
||||
# names. It never renders or reads Secret data.
|
||||
helm.sh/hook: test
|
||||
data:
|
||||
runtimeSecret: {{ .Values.existingSecret | quote }}
|
||||
ingressEnabled: {{ .Values.ingress.enabled | quote }}
|
||||
edgeEnabled: {{ .Values.edge.enabled | quote }}
|
||||
edgeHost: {{ .Values.edge.host | quote }}
|
||||
edgeService: {{ printf "%s-edge" (include "knowledge-platform.fullname" .) | quote }}
|
||||
edgeDeployment: {{ printf "%s-edge" (include "knowledge-platform.fullname" .) | quote }}
|
||||
edgeNodePort: {{ .Values.edge.service.nodePort | quote }}
|
||||
apiHost: {{ .Values.ingress.api.host | quote }}
|
||||
mcpHost: {{ .Values.ingress.mcp.host | quote }}
|
||||
apiDeployment: {{ printf "%s-api" (include "knowledge-platform.fullname" .) | quote }}
|
||||
mcpDeployment: {{ printf "%s-mcp" (include "knowledge-platform.fullname" .) | quote }}
|
||||
workerDeployment: {{ printf "%s-worker" (include "knowledge-platform.fullname" .) | quote }}
|
||||
schedulerDeployment: {{ printf "%s-scheduler" (include "knowledge-platform.fullname" .) | quote }}
|
||||
metricsPath: {{ .Values.observability.metrics.path | quote }}
|
||||
apiTlsSecret: {{ .Values.ingress.api.tlsSecretName | quote }}
|
||||
mcpTlsSecret: {{ .Values.ingress.mcp.tlsSecretName | quote }}
|
||||
rabbitmqEnabled: {{ .Values.rabbitmq.enabled | quote }}
|
||||
rabbitmqSecret: {{ .Values.rabbitmq.existingSecret | quote }}
|
||||
@ -0,0 +1,71 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "knowledge-platform.fullname" . }}-worker
|
||||
labels:
|
||||
{{- include "knowledge-platform.labels" . | nindent 4 }}
|
||||
app.kubernetes.io/component: worker
|
||||
spec:
|
||||
replicas: {{ .Values.worker.replicaCount }}
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 1
|
||||
maxSurge: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 6 }}
|
||||
app.kubernetes.io/component: worker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "knowledge-platform.selectorLabels" . | nindent 8 }}
|
||||
app.kubernetes.io/component: worker
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
annotations:
|
||||
checksum/runtime-config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
|
||||
checksum/runtime-secret-reference: {{ .Values.existingSecret | sha256sum }}
|
||||
{{- with .Values.podAnnotations }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
serviceAccountName: {{ include "knowledge-platform.serviceAccountName" . }}
|
||||
automountServiceAccountToken: false
|
||||
terminationGracePeriodSeconds: {{ .Values.worker.terminationGracePeriodSeconds }}
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.podSecurityContext" . | nindent 8 }}
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: worker
|
||||
image: {{ include "knowledge-platform.image" .Values.images.worker | quote }}
|
||||
imagePullPolicy: {{ .Values.images.worker.pullPolicy }}
|
||||
workingDir: /tmp
|
||||
command: ["celery"]
|
||||
args:
|
||||
- --app=ekp_api.worker:celery_app
|
||||
- worker
|
||||
- --loglevel=INFO
|
||||
- --queues=ingestion,compilation
|
||||
securityContext:
|
||||
{{- include "knowledge-platform.containerSecurityContext" . | nindent 12 }}
|
||||
envFrom:
|
||||
{{- include "knowledge-platform.envFrom" . | nindent 12 }}
|
||||
startupProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
{{- toYaml .Values.worker.probes.startup | nindent 12 }}
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["python", "-c", "import os; os.kill(1, 0)"]
|
||||
{{- toYaml .Values.worker.probes.liveness | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.worker.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
{{- include "knowledge-platform.tmpMount" . | nindent 12 }}
|
||||
volumes:
|
||||
{{- include "knowledge-platform.tmpVolume" . | nindent 8 }}
|
||||
450
hy4/_kp/deploy/helm/knowledge-platform/values.schema.json
Normal file
450
hy4/_kp/deploy/helm/knowledge-platform/values.schema.json
Normal file
@ -0,0 +1,450 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft-07/schema#",
|
||||
"title": "Enterprise Knowledge Platform Helm values",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"existingSecret",
|
||||
"images",
|
||||
"serviceAccount",
|
||||
"podDisruptionBudget",
|
||||
"observability",
|
||||
"api",
|
||||
"mcp",
|
||||
"edge",
|
||||
"worker",
|
||||
"scheduler",
|
||||
"migration",
|
||||
"ingress",
|
||||
"config",
|
||||
"rabbitmq"
|
||||
],
|
||||
"properties": {
|
||||
"nameOverride": {"type": "string"},
|
||||
"fullnameOverride": {"type": "string"},
|
||||
"existingSecret": {"$ref": "#/definitions/kubernetesName"},
|
||||
"imagePullSecrets": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name"],
|
||||
"properties": {"name": {"$ref": "#/definitions/kubernetesName"}}
|
||||
}
|
||||
},
|
||||
"images": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["api", "mcp", "worker", "edge"],
|
||||
"properties": {
|
||||
"api": {"$ref": "#/definitions/image"},
|
||||
"mcp": {"$ref": "#/definitions/image"},
|
||||
"worker": {"$ref": "#/definitions/image"},
|
||||
"edge": {"$ref": "#/definitions/image"}
|
||||
}
|
||||
},
|
||||
"serviceAccount": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["create", "name", "annotations"],
|
||||
"properties": {
|
||||
"create": {"type": "boolean"},
|
||||
"name": {"type": "string"},
|
||||
"annotations": {"$ref": "#/definitions/stringMap"}
|
||||
}
|
||||
},
|
||||
"podAnnotations": {"$ref": "#/definitions/stringMap"},
|
||||
"podLabels": {"$ref": "#/definitions/stringMap"},
|
||||
"podDisruptionBudget": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "apiMinAvailable", "mcpMinAvailable", "edgeMinAvailable"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"apiMinAvailable": {"type": "integer", "minimum": 1},
|
||||
"mcpMinAvailable": {"type": "integer", "minimum": 1},
|
||||
"edgeMinAvailable": {"type": "integer", "minimum": 1}
|
||||
}
|
||||
},
|
||||
"observability": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["metrics"],
|
||||
"properties": {
|
||||
"metrics": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "path"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"path": {"const": "/metrics"}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"api": {"$ref": "#/definitions/httpWorkload"},
|
||||
"mcp": {"$ref": "#/definitions/httpWorkload"},
|
||||
"edge": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "host", "replicaCount", "service", "clientMaxBodySize", "proxyReadTimeoutSeconds", "resources", "probes"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"host": {"type": "string", "minLength": 1},
|
||||
"replicaCount": {"type": "integer", "minimum": 2, "maximum": 100},
|
||||
"service": {"$ref": "#/definitions/edgeService"},
|
||||
"clientMaxBodySize": {"type": "string", "pattern": "^[1-9][0-9]*[kKmMgG]$"},
|
||||
"proxyReadTimeoutSeconds": {"type": "integer", "minimum": 1, "maximum": 86400},
|
||||
"resources": {"$ref": "#/definitions/resources"},
|
||||
"probes": {"$ref": "#/definitions/httpProbes"}
|
||||
}
|
||||
},
|
||||
"worker": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["replicaCount", "terminationGracePeriodSeconds", "resources", "probes"],
|
||||
"properties": {
|
||||
"replicaCount": {"type": "integer", "minimum": 1, "maximum": 100},
|
||||
"terminationGracePeriodSeconds": {"type": "integer", "minimum": 30, "maximum": 3600},
|
||||
"resources": {"$ref": "#/definitions/resources"},
|
||||
"probes": {"$ref": "#/definitions/processProbes"}
|
||||
}
|
||||
},
|
||||
"scheduler": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "resources", "probes"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"resources": {"$ref": "#/definitions/resources"},
|
||||
"probes": {"$ref": "#/definitions/processProbes"}
|
||||
}
|
||||
},
|
||||
"migration": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "activeDeadlineSeconds", "backoffLimit", "ttlSecondsAfterFinished", "resources"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"activeDeadlineSeconds": {"type": "integer", "minimum": 60, "maximum": 3600},
|
||||
"backoffLimit": {"type": "integer", "minimum": 0, "maximum": 3},
|
||||
"ttlSecondsAfterFinished": {"type": "integer", "minimum": 0},
|
||||
"resources": {"$ref": "#/definitions/resources"}
|
||||
}
|
||||
},
|
||||
"ingress": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["enabled", "className", "annotations", "api", "mcp"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"className": {"type": "string", "minLength": 1},
|
||||
"annotations": {"$ref": "#/definitions/stringMap"},
|
||||
"api": {"$ref": "#/definitions/ingressRoute"},
|
||||
"mcp": {"$ref": "#/definitions/ingressRoute"}
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["appName", "logLevel", "database", "ingestion", "compilation", "objectStorage", "parser", "chunking", "embedding", "vectorStore", "retrieval", "reranker", "mcp"],
|
||||
"properties": {
|
||||
"appName": {"type": "string", "minLength": 1},
|
||||
"logLevel": {"enum": ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]},
|
||||
"database": {"$ref": "#/definitions/databaseConfig"},
|
||||
"ingestion": {"$ref": "#/definitions/ingestionConfig"},
|
||||
"compilation": {"$ref": "#/definitions/compilationConfig"},
|
||||
"objectStorage": {"$ref": "#/definitions/objectStorageConfig"},
|
||||
"parser": {"$ref": "#/definitions/parserConfig"},
|
||||
"chunking": {"$ref": "#/definitions/chunkingConfig"},
|
||||
"embedding": {"$ref": "#/definitions/embeddingConfig"},
|
||||
"vectorStore": {"$ref": "#/definitions/vectorStoreConfig"},
|
||||
"retrieval": {"$ref": "#/definitions/retrievalConfig"},
|
||||
"reranker": {"$ref": "#/definitions/rerankerConfig"},
|
||||
"mcp": {"$ref": "#/definitions/mcpConfig"}
|
||||
}
|
||||
},
|
||||
"rabbitmq": {
|
||||
"type": "object",
|
||||
"required": ["enabled", "existingSecret"],
|
||||
"properties": {
|
||||
"enabled": {"type": "boolean"},
|
||||
"existingSecret": {"$ref": "#/definitions/kubernetesName"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"definitions": {
|
||||
"kubernetesName": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 253,
|
||||
"pattern": "^[a-z0-9]([-a-z0-9.]*[a-z0-9])?$"
|
||||
},
|
||||
"stringMap": {
|
||||
"type": "object",
|
||||
"additionalProperties": {"type": "string"}
|
||||
},
|
||||
"image": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repository", "tag", "digest", "pullPolicy"],
|
||||
"properties": {
|
||||
"repository": {"type": "string", "minLength": 1},
|
||||
"tag": {"type": "string", "minLength": 1, "not": {"enum": ["latest"]}},
|
||||
"digest": {"type": "string", "pattern": "^$|^sha256:[a-f0-9]{64}$"},
|
||||
"pullPolicy": {"enum": ["Always", "IfNotPresent", "Never"]}
|
||||
}
|
||||
},
|
||||
"probeTiming": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["failureThreshold", "periodSeconds"],
|
||||
"properties": {
|
||||
"failureThreshold": {"type": "integer", "minimum": 1},
|
||||
"periodSeconds": {"type": "integer", "minimum": 1},
|
||||
"timeoutSeconds": {"type": "integer", "minimum": 1}
|
||||
}
|
||||
},
|
||||
"httpProbes": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["startup", "readiness", "liveness"],
|
||||
"properties": {
|
||||
"startup": {"$ref": "#/definitions/probeTiming"},
|
||||
"readiness": {"$ref": "#/definitions/probeTiming"},
|
||||
"liveness": {"$ref": "#/definitions/probeTiming"}
|
||||
}
|
||||
},
|
||||
"processProbes": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["startup", "liveness"],
|
||||
"properties": {
|
||||
"startup": {"$ref": "#/definitions/probeTiming"},
|
||||
"liveness": {"$ref": "#/definitions/probeTiming"}
|
||||
}
|
||||
},
|
||||
"resources": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["requests", "limits"],
|
||||
"properties": {
|
||||
"requests": {"$ref": "#/definitions/resourcePair"},
|
||||
"limits": {"$ref": "#/definitions/resourcePair"}
|
||||
}
|
||||
},
|
||||
"resourcePair": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["cpu", "memory"],
|
||||
"properties": {
|
||||
"cpu": {"type": ["string", "number"]},
|
||||
"memory": {"type": "string", "minLength": 2}
|
||||
}
|
||||
},
|
||||
"httpWorkload": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["replicaCount", "service", "resources", "probes"],
|
||||
"properties": {
|
||||
"replicaCount": {"type": "integer", "minimum": 2, "maximum": 100},
|
||||
"service": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["type", "port"],
|
||||
"properties": {
|
||||
"type": {"enum": ["ClusterIP"]},
|
||||
"port": {"type": "integer", "minimum": 1, "maximum": 65535}
|
||||
}
|
||||
},
|
||||
"resources": {"$ref": "#/definitions/resources"},
|
||||
"probes": {"$ref": "#/definitions/httpProbes"}
|
||||
}
|
||||
},
|
||||
"edgeService": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["type", "port", "nodePort"],
|
||||
"properties": {
|
||||
"type": {"enum": ["NodePort"]},
|
||||
"port": {"type": "integer", "minimum": 1, "maximum": 65535},
|
||||
"nodePort": {
|
||||
"oneOf": [
|
||||
{"const": 0},
|
||||
{"type": "integer", "minimum": 30000, "maximum": 32767}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"ingressRoute": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["host", "path", "tlsSecretName"],
|
||||
"properties": {
|
||||
"host": {"type": "string", "minLength": 1},
|
||||
"path": {"type": "string", "pattern": "^/"},
|
||||
"tlsSecretName": {"$ref": "#/definitions/kubernetesName"}
|
||||
}
|
||||
},
|
||||
"databaseConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["poolSize", "maxOverflow", "poolRecycleSeconds", "poolTimeoutSeconds", "connectTimeoutSeconds"],
|
||||
"properties": {
|
||||
"poolSize": {"type": "integer", "minimum": 1, "maximum": 100},
|
||||
"maxOverflow": {"type": "integer", "minimum": 0, "maximum": 100},
|
||||
"poolRecycleSeconds": {"type": "integer", "minimum": 1},
|
||||
"poolTimeoutSeconds": {"type": "number", "exclusiveMinimum": 0},
|
||||
"connectTimeoutSeconds": {"type": "integer", "minimum": 1}
|
||||
}
|
||||
},
|
||||
"ingestionConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["maxAttempts", "retryBackoffSeconds", "retryBackoffMaxSeconds", "workerLeaseSeconds", "recoveryIntervalSeconds", "batchMaxFiles", "batchMaxTotalBytes", "maxUploadBytes"],
|
||||
"properties": {
|
||||
"maxAttempts": {"type": "integer", "minimum": 1, "maximum": 10},
|
||||
"retryBackoffSeconds": {"type": "integer", "minimum": 1},
|
||||
"retryBackoffMaxSeconds": {"type": "integer", "minimum": 1},
|
||||
"workerLeaseSeconds": {"type": "integer", "minimum": 30},
|
||||
"recoveryIntervalSeconds": {"type": "integer", "minimum": 5},
|
||||
"batchMaxFiles": {"type": "integer", "minimum": 1, "maximum": 100},
|
||||
"batchMaxTotalBytes": {"type": "integer", "minimum": 1},
|
||||
"maxUploadBytes": {"type": "integer", "minimum": 1}
|
||||
}
|
||||
},
|
||||
"compilationConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["maxAttempts", "retryBackoffSeconds", "retryBackoffMaxSeconds", "workerLeaseSeconds", "recoveryIntervalSeconds"],
|
||||
"properties": {
|
||||
"maxAttempts": {"type": "integer", "minimum": 1, "maximum": 10},
|
||||
"retryBackoffSeconds": {"type": "integer", "minimum": 1},
|
||||
"retryBackoffMaxSeconds": {"type": "integer", "minimum": 1},
|
||||
"workerLeaseSeconds": {"type": "integer", "minimum": 30},
|
||||
"recoveryIntervalSeconds": {"type": "integer", "minimum": 5}
|
||||
}
|
||||
},
|
||||
"objectStorageConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["provider", "endpoint", "bucket", "keyPrefix", "timeoutSeconds"],
|
||||
"properties": {
|
||||
"provider": {"const": "bos"},
|
||||
"endpoint": {"type": "string", "minLength": 1},
|
||||
"bucket": {"type": "string", "minLength": 1},
|
||||
"keyPrefix": {"type": "string", "minLength": 1},
|
||||
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
|
||||
}
|
||||
},
|
||||
"parserConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["provider", "mineruEndpoint", "mineruBackend", "mineruParseMethod", "mineruLanguage", "mineruEffort", "mineruTimeoutSeconds", "lowOcrConfidenceThreshold"],
|
||||
"properties": {
|
||||
"provider": {"enum": ["lightweight", "mineru"]},
|
||||
"mineruEndpoint": {"type": "string"},
|
||||
"mineruBackend": {"enum": ["pipeline", "vlm-engine", "hybrid-engine"]},
|
||||
"mineruParseMethod": {"enum": ["auto", "txt", "ocr"]},
|
||||
"mineruLanguage": {"type": "string", "minLength": 1, "maxLength": 32, "pattern": "^[A-Za-z0-9_-]+$"},
|
||||
"mineruEffort": {"enum": ["medium", "high"]},
|
||||
"mineruTimeoutSeconds": {"type": "number", "exclusiveMinimum": 0, "maximum": 3600},
|
||||
"lowOcrConfidenceThreshold": {"type": "number", "minimum": 0, "maximum": 1}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {"properties": {"provider": {"const": "mineru"}}},
|
||||
"then": {
|
||||
"properties": {
|
||||
"mineruEndpoint": {"type": "string", "pattern": "^https?://[^?#]+/file_parse$"}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"chunkingConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["maxTokens", "overlapTokens", "maxCharacters"],
|
||||
"properties": {
|
||||
"maxTokens": {"type": "integer", "minimum": 1},
|
||||
"overlapTokens": {"type": "integer", "minimum": 0},
|
||||
"maxCharacters": {"type": "integer", "minimum": 4}
|
||||
}
|
||||
},
|
||||
"embeddingConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["provider", "model", "endpoint", "timeoutSeconds"],
|
||||
"properties": {
|
||||
"provider": {"const": "qianfan"},
|
||||
"model": {"const": "bge-large-zh"},
|
||||
"endpoint": {"type": "string", "minLength": 1},
|
||||
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
|
||||
}
|
||||
},
|
||||
"vectorStoreConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["provider", "uri", "user", "database", "table", "index", "batchSize", "timeoutSeconds"],
|
||||
"properties": {
|
||||
"provider": {"const": "oceanbase"},
|
||||
"uri": {"type": "string", "minLength": 1},
|
||||
"user": {"type": "string", "minLength": 1},
|
||||
"database": {"type": "string", "minLength": 1},
|
||||
"table": {"type": "string", "minLength": 1},
|
||||
"index": {"type": "string", "minLength": 1},
|
||||
"batchSize": {"type": "integer", "minimum": 1, "maximum": 1000},
|
||||
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
|
||||
}
|
||||
},
|
||||
"retrievalConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"not": {
|
||||
"required": ["hybridDenseWeight", "hybridSparseWeight"],
|
||||
"properties": {
|
||||
"hybridDenseWeight": {"const": 0},
|
||||
"hybridSparseWeight": {"const": 0}
|
||||
}
|
||||
},
|
||||
"required": ["indexProfileId", "denseTopN", "sparseTopN", "hybridTopN", "hybridRrfK", "hybridDenseWeight", "hybridSparseWeight", "searchTopK"],
|
||||
"properties": {
|
||||
"indexProfileId": {"type": "string", "minLength": 1},
|
||||
"denseTopN": {"type": "integer", "minimum": 1, "maximum": 64},
|
||||
"sparseTopN": {"type": "integer", "minimum": 1, "maximum": 64},
|
||||
"hybridTopN": {"type": "integer", "minimum": 1, "maximum": 64},
|
||||
"hybridRrfK": {"type": "integer", "minimum": 1, "maximum": 1000},
|
||||
"hybridDenseWeight": {"type": "number", "minimum": 0, "maximum": 10},
|
||||
"hybridSparseWeight": {"type": "number", "minimum": 0, "maximum": 10},
|
||||
"searchTopK": {"type": "integer", "minimum": 1, "maximum": 64}
|
||||
}
|
||||
},
|
||||
"rerankerConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["provider", "model", "endpoint", "timeoutSeconds"],
|
||||
"properties": {
|
||||
"provider": {"const": "qianfan"},
|
||||
"model": {"const": "bce-reranker-base"},
|
||||
"endpoint": {"type": "string", "minLength": 1},
|
||||
"timeoutSeconds": {"type": "number", "exclusiveMinimum": 0}
|
||||
}
|
||||
},
|
||||
"mcpConfig": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["publicUrl", "issuerUrl", "clientId", "serviceIdentity", "requiredScope", "allowedHosts", "allowedOrigins", "maxRequestBodyBytes"],
|
||||
"properties": {
|
||||
"publicUrl": {"type": "string", "pattern": "^https://"},
|
||||
"issuerUrl": {"type": "string", "pattern": "^https://"},
|
||||
"clientId": {"type": "string", "minLength": 1},
|
||||
"serviceIdentity": {"type": "string", "minLength": 1},
|
||||
"requiredScope": {"const": "knowledge:search"},
|
||||
"allowedHosts": {"type": "array", "minItems": 1, "items": {"type": "string", "minLength": 1}},
|
||||
"allowedOrigins": {"type": "array", "minItems": 1, "items": {"type": "string", "pattern": "^https://"}},
|
||||
"maxRequestBodyBytes": {"type": "integer", "minimum": 1024, "maximum": 4194304}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
294
hy4/_kp/deploy/helm/knowledge-platform/values.yaml
Normal file
294
hy4/_kp/deploy/helm/knowledge-platform/values.yaml
Normal file
@ -0,0 +1,294 @@
|
||||
# DEPLOY-002 safe defaults. Real endpoints, immutable image coordinates and
|
||||
# Secret names are supplied in an environment-specific values file.
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
existingSecret: knowledge-platform-runtime
|
||||
imagePullSecrets: []
|
||||
|
||||
images:
|
||||
api:
|
||||
repository: knowledge-platform/enterprise-knowledge-api
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
pullPolicy: IfNotPresent
|
||||
mcp:
|
||||
repository: knowledge-platform/enterprise-knowledge-mcp
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
pullPolicy: IfNotPresent
|
||||
worker:
|
||||
repository: knowledge-platform/enterprise-knowledge-worker
|
||||
tag: "0.1.0"
|
||||
digest: ""
|
||||
pullPolicy: IfNotPresent
|
||||
# The optional edge proxy uses a standard upstream image. Production
|
||||
# environments should mirror it into their approved Registry and pin a digest.
|
||||
edge:
|
||||
repository: nginx
|
||||
tag: "1.29.1-alpine"
|
||||
digest: "sha256:42a516af16b852e33b7682d5ef8acbd5d13fe08fecadc7ed98605ba5e3b26ab8"
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: ""
|
||||
annotations: {}
|
||||
|
||||
podAnnotations: {}
|
||||
podLabels: {}
|
||||
|
||||
# DEPLOY-003 keeps one API and MCP replica available during voluntary node
|
||||
# maintenance. This complements, rather than replaces, RollingUpdate settings.
|
||||
podDisruptionBudget:
|
||||
enabled: true
|
||||
apiMinAvailable: 1
|
||||
mcpMinAvailable: 1
|
||||
edgeMinAvailable: 1
|
||||
|
||||
observability:
|
||||
metrics:
|
||||
enabled: true
|
||||
path: /metrics
|
||||
|
||||
api:
|
||||
replicaCount: 2
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 80
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
probes:
|
||||
startup:
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
readiness:
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
liveness:
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
|
||||
mcp:
|
||||
replicaCount: 2
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 80
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
probes:
|
||||
startup:
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
readiness:
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
liveness:
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
|
||||
# For clusters without an Ingress Controller, this infrastructure-only proxy
|
||||
# preserves one external endpoint while API and MCP remain separate workloads.
|
||||
# Use a temporary NodePort for staging, then change it to the existing public
|
||||
# NodePort only during the controlled cutover described in the runbook.
|
||||
edge:
|
||||
enabled: false
|
||||
host: knowledge.example.invalid
|
||||
replicaCount: 2
|
||||
service:
|
||||
type: NodePort
|
||||
port: 80
|
||||
nodePort: 0
|
||||
clientMaxBodySize: 210m
|
||||
proxyReadTimeoutSeconds: 3600
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
probes:
|
||||
startup:
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
readiness:
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
liveness:
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
|
||||
worker:
|
||||
replicaCount: 2
|
||||
terminationGracePeriodSeconds: 60
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2Gi
|
||||
probes:
|
||||
startup:
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
liveness:
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
|
||||
# Celery Beat is separated from scalable workers so only one recovery
|
||||
# scheduler publishes due durable Jobs at a time.
|
||||
scheduler:
|
||||
enabled: true
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
probes:
|
||||
startup:
|
||||
failureThreshold: 30
|
||||
periodSeconds: 2
|
||||
liveness:
|
||||
periodSeconds: 20
|
||||
timeoutSeconds: 2
|
||||
failureThreshold: 3
|
||||
|
||||
migration:
|
||||
enabled: true
|
||||
activeDeadlineSeconds: 600
|
||||
backoffLimit: 0
|
||||
ttlSecondsAfterFinished: 86400
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: nginx
|
||||
annotations: {}
|
||||
api:
|
||||
host: knowledge.example.invalid
|
||||
path: /
|
||||
tlsSecretName: knowledge-platform-api-tls
|
||||
mcp:
|
||||
host: mcp.example.invalid
|
||||
path: /mcp
|
||||
tlsSecretName: knowledge-platform-mcp-tls
|
||||
|
||||
# Only non-sensitive runtime settings belong here. The templates enumerate
|
||||
# keys explicitly so credentials cannot be added to this ConfigMap by mistake.
|
||||
config:
|
||||
appName: enterprise-knowledge-platform
|
||||
logLevel: INFO
|
||||
database:
|
||||
poolSize: 10
|
||||
maxOverflow: 10
|
||||
poolRecycleSeconds: 600
|
||||
poolTimeoutSeconds: 30
|
||||
connectTimeoutSeconds: 30
|
||||
ingestion:
|
||||
maxAttempts: 3
|
||||
retryBackoffSeconds: 2
|
||||
retryBackoffMaxSeconds: 60
|
||||
workerLeaseSeconds: 300
|
||||
recoveryIntervalSeconds: 30
|
||||
batchMaxFiles: 20
|
||||
batchMaxTotalBytes: 209715200
|
||||
maxUploadBytes: 52428800
|
||||
compilation:
|
||||
maxAttempts: 3
|
||||
retryBackoffSeconds: 2
|
||||
retryBackoffMaxSeconds: 60
|
||||
workerLeaseSeconds: 300
|
||||
recoveryIntervalSeconds: 30
|
||||
objectStorage:
|
||||
provider: bos
|
||||
endpoint: https://s3.example.invalid
|
||||
bucket: knowledge-platform
|
||||
keyPrefix: knowledge-platform
|
||||
timeoutSeconds: 30
|
||||
parser:
|
||||
# Enabling MinerU also requires a reachable endpoint and a worker lease
|
||||
# strictly greater than mineruTimeoutSeconds.
|
||||
provider: lightweight
|
||||
mineruEndpoint: ""
|
||||
mineruBackend: hybrid-engine
|
||||
mineruParseMethod: auto
|
||||
mineruLanguage: ch
|
||||
mineruEffort: medium
|
||||
mineruTimeoutSeconds: 600
|
||||
lowOcrConfidenceThreshold: 0.70
|
||||
chunking:
|
||||
maxTokens: 450
|
||||
overlapTokens: 75
|
||||
maxCharacters: 2000
|
||||
embedding:
|
||||
provider: qianfan
|
||||
model: bge-large-zh
|
||||
endpoint: https://qianfan.baidubce.com/v2/embeddings
|
||||
timeoutSeconds: 30
|
||||
vectorStore:
|
||||
provider: oceanbase
|
||||
uri: oceanbase.example.invalid:2881
|
||||
user: knowledge-platform
|
||||
database: knowledge_platform
|
||||
table: knowledge_chunks_v1
|
||||
index: vidx_knowledge_chunks_embedding
|
||||
batchSize: 100
|
||||
timeoutSeconds: 30
|
||||
retrieval:
|
||||
indexProfileId: demo-bge-large-zh-structure-v2-oceanbase-v1
|
||||
denseTopN: 20
|
||||
sparseTopN: 20
|
||||
hybridTopN: 20
|
||||
hybridRrfK: 60
|
||||
hybridDenseWeight: 1.0
|
||||
hybridSparseWeight: 1.0
|
||||
searchTopK: 5
|
||||
reranker:
|
||||
provider: qianfan
|
||||
model: bce-reranker-base
|
||||
endpoint: https://qianfan.baidubce.com/v2/rerank
|
||||
timeoutSeconds: 30
|
||||
mcp:
|
||||
publicUrl: https://mcp.example.invalid/mcp
|
||||
issuerUrl: https://mcp.example.invalid
|
||||
clientId: workbuddy-enterprise
|
||||
serviceIdentity: workbuddy-public-knowledge
|
||||
requiredScope: knowledge:search
|
||||
allowedHosts:
|
||||
- mcp.example.invalid
|
||||
allowedOrigins:
|
||||
- https://mcp.example.invalid
|
||||
maxRequestBodyBytes: 1048576
|
||||
|
||||
# Optional local dependency. Keep disabled for production and provide
|
||||
# RABBITMQ_URL through existingSecret for an externally managed broker.
|
||||
rabbitmq:
|
||||
enabled: false
|
||||
existingSecret: knowledge-platform-rabbitmq
|
||||
40
hy4/_kp/deploy/oceanbase/001_knowledge_chunks_v1.sql
Normal file
40
hy4/_kp/deploy/oceanbase/001_knowledge_chunks_v1.sql
Normal file
@ -0,0 +1,40 @@
|
||||
-- ARCH-003/EKP-052 rebuildable OceanBase dense and sparse chunk index.
|
||||
-- Canonical metadata is co-located in OceanBase relational tables; this vector
|
||||
-- table remains a derived index and may be rebuilt from canonical chunks.
|
||||
-- Requires OceanBase 4.3.5.4+; ARCH-006 verifies PRE-FILTER in the target test cluster.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS knowledge_chunks_v1 (
|
||||
index_record_id VARCHAR(36) NOT NULL,
|
||||
chunk_id VARCHAR(36) NOT NULL,
|
||||
document_id VARCHAR(36) NOT NULL,
|
||||
document_version_id VARCHAR(36) NOT NULL,
|
||||
filename VARCHAR(255) NOT NULL,
|
||||
page INT NULL,
|
||||
page_end INT NULL,
|
||||
section VARCHAR(1024) NULL,
|
||||
chunk_index INT NOT NULL,
|
||||
content MEDIUMTEXT NOT NULL,
|
||||
embedding VECTOR(1024) NOT NULL,
|
||||
embedding_model VARCHAR(128) NOT NULL,
|
||||
index_profile_id VARCHAR(128) NOT NULL,
|
||||
-- Applied by PRE-FILTER before ANN Top-K; historical rows remain rebuildable.
|
||||
is_active TINYINT(1) NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (index_record_id),
|
||||
KEY idx_knowledge_chunks_document_profile (document_id, index_profile_id),
|
||||
KEY idx_knowledge_chunks_version_profile (document_version_id, index_profile_id),
|
||||
KEY idx_knowledge_chunks_active_profile (index_profile_id, is_active),
|
||||
KEY idx_knowledge_chunks_profile (index_profile_id),
|
||||
-- NGRAM size 2 gives deterministic Chinese and mixed-term keyword recall.
|
||||
FULLTEXT INDEX ftidx_knowledge_chunks_content (content)
|
||||
WITH PARSER NGRAM
|
||||
PARSER_PROPERTIES=(ngram_token_size=2),
|
||||
VECTOR KEY vidx_knowledge_chunks_embedding (embedding)
|
||||
WITH (
|
||||
DISTANCE = COSINE,
|
||||
TYPE = HNSW,
|
||||
LIB = VSAG,
|
||||
M = 16,
|
||||
EF_CONSTRUCTION = 256,
|
||||
EF_SEARCH = 200
|
||||
) BLOCK_SIZE 16384
|
||||
) ORGANIZATION = HEAP;
|
||||
19
hy4/_kp/deploy/oceanbase/002_add_document_version_scope.sql
Normal file
19
hy4/_kp/deploy/oceanbase/002_add_document_version_scope.sql
Normal file
@ -0,0 +1,19 @@
|
||||
-- EKP-022 forward migration for an existing knowledge_chunks_v1 vector table.
|
||||
-- Apply once before deploying application code that emits version-scoped rows.
|
||||
-- Legacy version 1 deliberately reused document_id, so this backfill is exact.
|
||||
|
||||
ALTER TABLE knowledge_chunks_v1
|
||||
ADD COLUMN document_version_id VARCHAR(36) NULL AFTER document_id,
|
||||
ADD COLUMN is_active TINYINT(1) NOT NULL DEFAULT 1 AFTER index_profile_id;
|
||||
|
||||
UPDATE knowledge_chunks_v1
|
||||
SET document_version_id = document_id
|
||||
WHERE document_version_id IS NULL;
|
||||
|
||||
ALTER TABLE knowledge_chunks_v1
|
||||
MODIFY COLUMN document_version_id VARCHAR(36) NOT NULL,
|
||||
ADD KEY idx_knowledge_chunks_version_profile (
|
||||
document_version_id,
|
||||
index_profile_id
|
||||
),
|
||||
ADD KEY idx_knowledge_chunks_active_profile (index_profile_id, is_active);
|
||||
@ -0,0 +1,8 @@
|
||||
-- EKP-052 forward upgrade for an existing knowledge_chunks_v1 table.
|
||||
-- Apply once before enabling sparse retrieval. The index is derived entirely
|
||||
-- from existing content rows and does not change canonical Chunk ownership.
|
||||
|
||||
CREATE FULLTEXT INDEX ftidx_knowledge_chunks_content
|
||||
ON knowledge_chunks_v1(content)
|
||||
WITH PARSER NGRAM
|
||||
PARSER_PROPERTIES=(ngram_token_size=2);
|
||||
12
hy4/_kp/deploy/workbuddy-mcp.example.json
Normal file
12
hy4/_kp/deploy/workbuddy-mcp.example.json
Normal file
@ -0,0 +1,12 @@
|
||||
{
|
||||
"mcpServers": {
|
||||
"enterprise-knowledge": {
|
||||
"type": "http",
|
||||
"url": "${EKP_MCP_URL:-http://127.0.0.1:8766/mcp}",
|
||||
"headers": {
|
||||
"Authorization": "Bearer ${EKP_MCP_TOKEN}"
|
||||
},
|
||||
"description": "企业知识检索、来源定位与章节阅读"
|
||||
}
|
||||
}
|
||||
}
|
||||
4
hy4/_kp/docker-compose.dev.yml
Normal file
4
hy4/_kp/docker-compose.dev.yml
Normal file
@ -0,0 +1,4 @@
|
||||
services:
|
||||
rabbitmq:
|
||||
image: rabbitmq:4-management
|
||||
ports: ["5672:5672", "15672:15672"]
|
||||
154
hy4/_kp/docs/README.md
Normal file
154
hy4/_kp/docs/README.md
Normal file
@ -0,0 +1,154 @@
|
||||
# 文档中心
|
||||
|
||||
本文档是 Enterprise Knowledge Platform 的文档导航与治理入口。阅读或修改文档时,先确认文档的职责与优先级,避免在多个文件中重复维护同一结论。
|
||||
|
||||
## 权威性顺序
|
||||
|
||||
发生冲突时,按以下顺序判断:
|
||||
|
||||
1. `AGENTS.md`:仓库协作规则与不可违反的工程约束。
|
||||
2. `product/PRD.md`:产品范围、需求和 V1 验收标准。
|
||||
3. `architecture/ADR.md`:已接受、已取代的架构决策。
|
||||
4. `architecture/ARCHITECTURE.md`:当前架构及模块边界。
|
||||
5. `api/API_CONTRACT.md`、`api/MCP_CONTRACT.md`:公开接口契约。
|
||||
6. `plans/TASKS.md` 与当前活动计划:任务状态、依赖和当前执行顺序。
|
||||
7. `development/`:实现、部署和运维操作说明,不得重新定义产品或架构。
|
||||
8. 历史计划与进度记录:只用于追溯,不得覆盖当前基线。
|
||||
|
||||
如果 PRD 与 ADR 冲突,应先明确是产品范围调整还是架构调整,不得在操作手册中自行消解冲突。
|
||||
|
||||
## 核心文档
|
||||
|
||||
| 文档 | 职责 | 生命周期 |
|
||||
|---|---|---|
|
||||
| [`../README.md`](../README.md) | 项目简介、启动方式和最短阅读路径 | 活跃 |
|
||||
| [`product/PRD.md`](product/PRD.md) | 产品需求与 V1 验收基线 | 权威 |
|
||||
| [`architecture/ADR.md`](architecture/ADR.md) | 架构决策状态 | 权威 |
|
||||
| [`architecture/ARCHITECTURE.md`](architecture/ARCHITECTURE.md) | 当前技术架构 | 权威 |
|
||||
| [`api/API_CONTRACT.md`](api/API_CONTRACT.md) | REST API 公开契约 | 权威 |
|
||||
| [`api/MCP_CONTRACT.md`](api/MCP_CONTRACT.md) | MCP 公开契约 | 权威 |
|
||||
| [`plans/TASKS.md`](plans/TASKS.md) | Task ID、状态和依赖 | 权威 |
|
||||
| [`plans/PROGRESS.md`](plans/PROGRESS.md) | 当前阶段摘要和最近进展 | 活跃 |
|
||||
| [`development/DEFINITION_OF_DONE.md`](development/DEFINITION_OF_DONE.md) | 任务完成标准 | 权威 |
|
||||
| [`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md) | 文档、分支、Tag、stash 与 worktree 治理规范 | 权威 |
|
||||
|
||||
## 当前活动文档
|
||||
|
||||
`DEPLOY-003` 已按开发侧范围闭环。`BUILD-001` 已完成根 Dockerfile、Bake、静态测试和 `dev`
|
||||
分支差异收敛;`EKP-015~022` 已完成 DocumentVersion 模型、Source Hash 去重和版本激活,
|
||||
`EKP-052~053` 已完成 BM25/Sparse 候选检索与 Hybrid Fusion;`EKP-025~027` 已完成可选
|
||||
MinerU OCR 路由、质量证据与清洗流水线;`EKP-026` 已完成未接线的 Docling Adapter。当前执行
|
||||
链中的 `EKP-028~029` 已完成规范解析模型、结构化表格 Chunk 和 P0 文件路由;`EKP-100` 已完成
|
||||
版本化 Golden Dataset 与统一指标基线,`EKP-042` 已补齐 Qwen3 候选 Adapter。由于尚无真实
|
||||
Qwen Endpoint/GPU 环境,`EKP-101` 保持未完成并暂停;`EKP-081` 已补齐高可读性 Source/解析
|
||||
双视图、页范围、页级质量和 Job/Attempt Inspector,用户确认权限视图合并到后续 User/KB/ACL
|
||||
任务。`EKP-010~017` 已完成领域包、OceanBase Migration 门禁、Organization/User/Group/Role、
|
||||
KnowledgeBase、Folder、ACL、持久化 Ingestion Job/Attempt 及 AuditLog 规范模型;`EKP-020~021`
|
||||
已补齐 ObjectStorage 与 Document 上传 API,`EKP-022~024` 已完成版本语义、异步 Job 和 Parser
|
||||
抽象,`EKP-040~046` 已闭环 EmbeddingProvider、千帆/Qwen Adapter、完整 Index Profile、
|
||||
VectorStore/Fake、OceanBase Vector Adapter 与完整 Shadow Index 生命周期,`EKP-050~056` 已完成
|
||||
内部 Permission Resolver、Document Folder 归属约束、Dense/Sparse 检索前 ACL 过滤、完整
|
||||
Dense/Sparse 候选、Hybrid Fusion、Reranker Provider、规范 Source 补全和 Search Service/API,
|
||||
以及五阶段检索诊断/Search Lab;`EKP-070` 已完成 MCP User Token/Service API Key、规范身份映射、
|
||||
ACL 与 `mcp_enabled` 检索前过滤;`EKP-071` 已补齐文档、大纲、章节、精确 Source 与 Knowledge
|
||||
Base 列表五个 P0 Tool,`EKP-072` 已完成官方 MCP Client 五 Tool 来源链兼容验收。`EKP-080` 已
|
||||
增加实时知识流运营总览并闭环 Overview、上传、文档、Search Lab 四个核心操作面;`EKP-082` 已
|
||||
完成可信 REST User/Service 身份、首身份 Bootstrap、User/Group/Role 关系、KB/Folder/ACL 管理、
|
||||
KB-aware Document 归属和权限操作面,不创建占位权限数据。`EKP-090` 已完成可追溯 Compiled
|
||||
Knowledge 模型、手工/AI 草稿工作流、REST 和管理端操作面;`EKP-091` 已完成 Source 变化传播、
|
||||
冲突/Stale、审核与发布;`EKP-092` 已完成 Published-only Compiled 检索、Raw/Compiled 联合精排、
|
||||
REST/MCP 来源链及 Search Lab 六阶段;`EKP-103` 已完成持久化 Compilation Job、后台重试、租约
|
||||
恢复以及安全/性能/重建门禁。`EKP-001~004` 已完成交付;`EKP-005` 结构化日志已完成实现与验证:
|
||||
字段白名单覆盖现有业务事件,无效/非结构化依赖消息失败关闭,请求关联 ID、标量边界和幂等配置由
|
||||
测试保护,并已合并至 `main@010c0a2`;`EKP-006` CI 基线已完成实现与验证:Provider-free
|
||||
`make ci` 串联锁文件、Web、静态检查、Unit/Integration、Migration、离线 RAG 与真实进程 E2E,
|
||||
且不改变中央发布模板,已合并至 `main@4327ede`;`EKP-007` Fake Provider 框架已完成实现与验证:
|
||||
四类公共 Fake 复用 Provider-neutral 契约、禁止外部 I/O 和生产回退,并明确不形成 Benchmark 结论;
|
||||
实现与首轮追溯文档提交为 `c312364`,任务分支已推送并合并至 `main@5957a5f`。原 V1 开发主队列
|
||||
已完成;2026-09-15 根据新增产品需求重新打开补充开发队列,依次处理 `EKP-083` 文档范围筛选、
|
||||
`EKP-084` 文件夹上传、`EKP-085` 安全 ZIP 导入和 `EKP-086` Knowledge Base/Folder 层级树均已
|
||||
完成;新增管理体验补充开发队列已闭环。
|
||||
文档治理、Benchmark、部署与现场验收继续独立排队,不阻塞该开发队列。
|
||||
|
||||
- [`plans/TASKS.md`](plans/TASKS.md):当前 Task 状态、开发/运维边界和后续顺序。
|
||||
- [`plans/PROGRESS.md`](plans/PROGRESS.md):最近验证、风险和下一步。
|
||||
- [`plans/DEPLOY_003_REPLAN.md`](plans/DEPLOY_003_REPLAN.md):历史部署门禁与运维交接依据。
|
||||
- [`development/DEPLOY_003_ACCEPTANCE.md`](development/DEPLOY_003_ACCEPTANCE.md):开发侧公网验收与运维待确认项。
|
||||
- [`development/CLUSTER_OPERATIONS.md`](development/CLUSTER_OPERATIONS.md):部署、故障恢复和运维操作。
|
||||
|
||||
完整 V1 MCP 业务能力保留在开发主队列;远程 MCP/WorkBuddy 与集群现场验收独立排队。任务状态
|
||||
以 `plans/TASKS.md` 为准,详细现场证据以验收记录为准,`PROGRESS.md` 不重复保存完整流水账。
|
||||
|
||||
## 开发与运行手册
|
||||
|
||||
| 主题 | 文档 |
|
||||
|---|---|
|
||||
| 本地配置 | [`development/LOCAL_PERSISTENT_CONFIG.md`](development/LOCAL_PERSISTENT_CONFIG.md) |
|
||||
| CI 基线 | [`development/CI_BASELINE.md`](development/CI_BASELINE.md) |
|
||||
| 仓库治理 | [`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md) |
|
||||
| Fake Provider | [`development/FAKE_PROVIDERS.md`](development/FAKE_PROVIDERS.md) |
|
||||
| REST API | [`development/REST_API_SETUP.md`](development/REST_API_SETUP.md) |
|
||||
| MCP Gateway | [`development/MCP_SETUP.md`](development/MCP_SETUP.md) |
|
||||
| 管理控制台 | [`development/ADMIN_CONSOLE.md`](development/ADMIN_CONSOLE.md) |
|
||||
| Compiled Knowledge | [`development/COMPILED_KNOWLEDGE.md`](development/COMPILED_KNOWLEDGE.md) |
|
||||
| 异步入库 | [`development/ASYNC_INGESTION_SETUP.md`](development/ASYNC_INGESTION_SETUP.md) |
|
||||
| BOS | [`development/BOS_SETUP.md`](development/BOS_SETUP.md) |
|
||||
| OceanBase Vector | [`development/OCEANBASE_VECTOR_SETUP.md`](development/OCEANBASE_VECTOR_SETUP.md) |
|
||||
| Sparse 检索 | [`development/SPARSE_RETRIEVAL.md`](development/SPARSE_RETRIEVAL.md) |
|
||||
| Dense 检索 | [`development/DENSE_RETRIEVAL.md`](development/DENSE_RETRIEVAL.md) |
|
||||
| Hybrid Fusion | [`development/HYBRID_RETRIEVAL.md`](development/HYBRID_RETRIEVAL.md) |
|
||||
| 规范来源补全 | [`development/SOURCE_ENRICHMENT.md`](development/SOURCE_ENRICHMENT.md) |
|
||||
| Search 诊断 | [`development/SEARCH_DIAGNOSTICS.md`](development/SEARCH_DIAGNOSTICS.md) |
|
||||
| Embedding Provider | [`development/EMBEDDING_PROVIDER.md`](development/EMBEDDING_PROVIDER.md) |
|
||||
| 千帆 BGE Embedding | [`development/QIANFAN_EMBEDDING_SETUP.md`](development/QIANFAN_EMBEDDING_SETUP.md) |
|
||||
| Index Profile | [`development/INDEX_PROFILE.md`](development/INDEX_PROFILE.md) |
|
||||
| VectorStore | [`development/VECTOR_STORE.md`](development/VECTOR_STORE.md) |
|
||||
| Parser 抽象 | [`development/PARSER_ABSTRACTION.md`](development/PARSER_ABSTRACTION.md) |
|
||||
| MinerU Parser | [`development/MINERU_SETUP.md`](development/MINERU_SETUP.md) |
|
||||
| Docling Parser | [`development/DOCLING_SETUP.md`](development/DOCLING_SETUP.md) |
|
||||
| Parsed Artifact | [`development/PARSED_DOCUMENT_SCHEMA.md`](development/PARSED_DOCUMENT_SCHEMA.md) |
|
||||
| 结构化 Chunk | [`development/STRUCTURED_CHUNKING.md`](development/STRUCTURED_CHUNKING.md) |
|
||||
| Reranker | [`development/RERANKER_SETUP.md`](development/RERANKER_SETUP.md) |
|
||||
| Shadow Index | [`development/INDEX_REBUILD.md`](development/INDEX_REBUILD.md) |
|
||||
| 权限解析与检索前过滤 | [`development/PERMISSION_RESOLUTION.md`](development/PERMISSION_RESOLUTION.md) |
|
||||
| Qwen3 Embedding | [`development/QWEN_EMBEDDING_SETUP.md`](development/QWEN_EMBEDDING_SETUP.md) |
|
||||
| 容器镜像 | [`development/CONTAINER_IMAGES.md`](development/CONTAINER_IMAGES.md) |
|
||||
| Kubernetes | [`development/KUBERNETES_DEPLOYMENT.md`](development/KUBERNETES_DEPLOYMENT.md) |
|
||||
| 集群运维 | [`development/CLUSTER_OPERATIONS.md`](development/CLUSTER_OPERATIONS.md) |
|
||||
| 安全 | [`development/SECURITY.md`](development/SECURITY.md) |
|
||||
| 测试 | [`development/TEST_STRATEGY.md`](development/TEST_STRATEGY.md) |
|
||||
| RAG 评估 | [`evaluation/RAG_EVALUATION.md`](evaluation/RAG_EVALUATION.md) |
|
||||
|
||||
## 历史与冻结文档
|
||||
|
||||
以下文档记录已经完成的阶段或过去的技术迁移,只保留追溯价值:
|
||||
|
||||
- `demo/DEMO_SCOPE.md`
|
||||
- `plans/ARCHITECTURE_MIGRATION_PLAN.md`
|
||||
- `plans/ASYNC_INGESTION_MCP_PLAN.md`
|
||||
- `plans/PUBLIC_KNOWLEDGE_CLUSTER_PLAN.md`
|
||||
|
||||
冻结文档不再同步当前实现细节。若其中内容与核心文档冲突,以核心文档和当前活动计划为准。旧版本的逐任务实现细节通过 Git 历史查询,不继续堆积到活跃文档。
|
||||
|
||||
## 文档维护规则
|
||||
|
||||
1. 每个概念只设一个事实源;其他文档使用链接引用,不复制整段内容。
|
||||
2. PRD 只描述产品需求,Architecture 只描述技术结构,Contract 只描述外部行为,Runbook 只描述操作步骤。
|
||||
3. `TASKS.md` 只记录任务状态和依赖;`PROGRESS.md` 只保留当前阶段、最近结果、风险和下一步。
|
||||
4. 已完成的计划标记为“历史/冻结”,不继续加入新状态。
|
||||
5. 行为变化必须同步对应契约;仅实现细节变化不应改写 PRD。
|
||||
6. 不在文档中保存未获授权的密码、Token、Key、内部 Endpoint 或真实业务正文。
|
||||
7. 命令、路径、配置键、API 字段、状态值和 Task ID 保持原样,避免翻译导致不可执行。
|
||||
8. 删除或合并文档前,先更新所有内部引用并确认权威信息已有归属。
|
||||
9. 文档精简、任务分支归档、里程碑 Tag 和 worktree 清理统一遵循
|
||||
[`development/REPOSITORY_GOVERNANCE.md`](development/REPOSITORY_GOVERNANCE.md)。
|
||||
|
||||
## 变更检查
|
||||
|
||||
文档变更至少执行:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
```
|
||||
|
||||
涉及代码行为、Migration、REST/MCP 契约或检索结果的变更,仍须按 `development/DEFINITION_OF_DONE.md` 运行相应工程门禁。
|
||||
431
hy4/_kp/docs/api/API_CONTRACT.md
Normal file
431
hy4/_kp/docs/api/API_CONTRACT.md
Normal file
@ -0,0 +1,431 @@
|
||||
# REST API 契约
|
||||
|
||||
所有公开 API 均在 `/api/v1` 下进行版本化。
|
||||
|
||||
初始领域包括:认证、用户/组、知识库、文档、Compiled Knowledge、检索、评估、Job、审计日志、Provider 和 MCP 管理。
|
||||
|
||||
实现开始后,以 OpenAPI 作为可执行事实源。Provider 特有概念不得泄漏到公开契约中。
|
||||
|
||||
## Demo v0.1 P0 范围
|
||||
|
||||
DEMO-009 公开以下最小接口。`docs/demo/DEMO_SCOPE.md` 中的 `/api/*` 逻辑路径按本契约统一
|
||||
版本化为 `/api/v1/*`。
|
||||
|
||||
`DEMO-015` 已将单文件上传切换为持久化异步 Job;`DEMO-016` 已增加批量提交和聚合查询。
|
||||
|
||||
| 方法 | 路径 | 行为 |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/v1/documents` | multipart 单文件接收;持久化 Source、Document、Job 后返回 `202` |
|
||||
| `POST` | `/api/v1/ingestion-batches` | multipart `files` 多文件接收;可选同序 `relative_paths` 保留文件夹层级;返回批次与逐文件状态 |
|
||||
| `POST` | `/api/v1/ingestion-archives` | multipart `archive` 安全展开单个 ZIP;保留多层目录并返回批次与逐文件状态 |
|
||||
| `GET` | `/api/v1/ingestion-batches/{batch_id}` | 返回批次汇总和逐文件最新 Job 状态 |
|
||||
| `GET` | `/api/v1/documents` | 按创建时间倒序返回 ACL 交集内的文档列表,可选 KB/Folder 范围 |
|
||||
| `GET` | `/api/v1/documents/{document_id}` | 返回单个文档状态与安全错误信息 |
|
||||
| `GET` | `/api/v1/documents/{document_id}/content` | 返回指定或当前版本的规范 Chunk 正文与来源位置 |
|
||||
| `GET` | `/api/v1/documents/{document_id}/source` | 流式返回指定或当前版本的不可变原始文件 |
|
||||
| `DELETE` | `/api/v1/documents/{document_id}` | 无活动入库 Job 时删除;有 Compiled/Compilation Job 引用则保留不可变证据并逻辑删除,否则清理全部产物 |
|
||||
| `GET` | `/api/v1/jobs/{job_id}` | 返回阶段、状态、attempt、最大尝试次数和安全错误信息 |
|
||||
| `GET` | `/api/v1/knowledge-bases` | 按当前 User 的有效 Read ACL 列出可读 Knowledge Base |
|
||||
| `GET` | `/api/v1/knowledge-bases/{knowledge_base_id}/folders` | 在可读 KB 内列出 Folder Metadata |
|
||||
| `GET` | `/api/v1/compiled-knowledge` | 列出当前身份可读来源交集内的 Compiled Knowledge |
|
||||
| `GET` | `/api/v1/compiled-knowledge/{compiled_id}` | 返回一条可追溯 Compiled Knowledge |
|
||||
| `POST` | `/api/v1/compiled-knowledge/drafts` | 从真实 Chunk 创建手工草稿 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/drafts:generate` | 校验来源并持久化 Compilation Job,返回 `202` |
|
||||
| `GET` | `/api/v1/compiled-knowledge/jobs/{job_id}` | 按 KB 与全部来源权限交集返回后台生成状态与 Attempt |
|
||||
| `PUT` | `/api/v1/compiled-knowledge/{compiled_id}/draft` | 使用期望修订号替换草稿编辑字段 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:submit-review` | 核验来源后提交人工审核 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}/conflicts` | 记录绑定两个确切来源的冲突主张 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}/conflicts/{conflict_id}:resolve` | 以说明解决一个冲突 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:publish` | 发布已核验且无未解决冲突的审核版本 |
|
||||
| `POST` | `/api/v1/compiled-knowledge/{compiled_id}:reject` | 携带原因退回审核版本 |
|
||||
| `POST` | `/api/v1/search` | Dense/Sparse Top20、Weighted RRF Top20、Reranker、TopK、来源补全 |
|
||||
| `POST` | `/api/v1/search/diagnostics` | 执行同一检索链一次并返回 Scope/Profile、各阶段候选、耗时和最终来源 |
|
||||
|
||||
除 `/healthz`、OpenAPI 和同源 Admin 静态资源外,REST 业务接口必须携带
|
||||
`Authorization: Bearer <token>`。运行时 `API_IDENTITY_CREDENTIALS` 将 User Token 或 Service API
|
||||
Key 映射到规范 Organization/User,并显式声明 `knowledge:read`、`knowledge:write`、
|
||||
`knowledge:search`、`knowledge:manage` Scope;Token 不进入 URL、响应或日志。缺失/无效凭据返回
|
||||
`401`,有效凭据缺少操作 Scope 或映射到失效 User 返回 `403`,无权读取的具体资源与不存在统一返回
|
||||
`404`。`API_ALLOW_LEGACY_PUBLIC=true` 只在未配置任何正式凭据时保留旧公共知识兼容,不具备管理权限。
|
||||
|
||||
`EKP-053` 不改变普通 Search 请求或响应字段。响应中的 `score` 表示融合分数,`rerank_score` 表示
|
||||
精排分数;Dense/Sparse 原始分数、名次、权重和 Index Profile 只通过 `EKP-056` 的管理端诊断
|
||||
接口提供,普通 Search 与 MCP 不返回这些字段。Provider Endpoint、Credential 和厂商响应在任何
|
||||
接口中都不可见。
|
||||
`EKP-051` 只补强内部 Dense Query/Top-N、Profile Query Instruction、空 ACL Scope 和候选失败关闭
|
||||
边界,不改变上述公开路径、请求、响应或状态码。
|
||||
`EKP-054` 同样只补强内部 Reranker 协议、输入/输出门禁、Active Profile 模型/候选规模对齐及
|
||||
千帆 BCE 安全故障边界;不新增或改变公开字段、路径、状态码或客户端入口。
|
||||
`EKP-055` 将最终结果严格绑定 Active Index Profile 与可验证的规范 Chunk/Index State/Ready Active
|
||||
Version,并补充 `section_path` 和固定为 `raw` 的 `source_type`。客户端请求的 `top_k` 不得大于
|
||||
Active Profile 的 `final_top_k`,禁止一次请求绕过已评测配置扩大最终结果数。
|
||||
|
||||
上传字段名为 `file`;正式身份还必须提交 `knowledge_base_id`,可选 `folder_id` 必须属于同一知识库。
|
||||
上传前按当前 User 的有效 `write` 权限验证知识库,随后把 Organization/KB/Folder 原样写入 Document
|
||||
与 DocumentVersion。`structure_v2` Active 时支持 PDF、DOCX、PPTX、XLSX、CSV、HTML/HTM、
|
||||
TXT 和 Markdown;PPTX/XLSX/CSV/HTML 由本地 Docling 解析。`PARSER_PROVIDER=mineru` 时额外
|
||||
接受 JPG/JPEG/PNG,并将 PDF、DOCX 和图片交给 MinerU `auto` 解析,其他格式保持各自路由。
|
||||
数据库仍为 `structure_v1` Active 时新增 Docling 格式返回 `415`,直至完成 Shadow Profile 切换。
|
||||
上传准入由实际 Worker Parser Router 的不可变能力集合生成,不维护第二份运行时白名单;该内部
|
||||
加固不改变支持格式、请求/响应字段或状态码。
|
||||
成功返回 `202`、Document 范围字段 `organization_id`、`knowledge_base_id`、`folder_id` 及
|
||||
`document_version_id`、`document_version`、`source_hash`、`job_id`、`job_status=queued`、
|
||||
`max_attempts`;响应不等待 Parse、Clean、Chunk、Embedding 或 Index。Document 响应不暴露
|
||||
`storage_key`、Provider 配置或索引实现。schema v4 Page/Sheet/Table 结构及页级 OCR
|
||||
Confidence/警告保存在内部 Parsed Artifact,不属于 Demo Document 响应;检索来源只公开已命中
|
||||
Chunk 的 Sheet 与行列范围,完整 Inspector API/UI 由 `EKP-081` 定义。
|
||||
|
||||
上传响应中的 `id` 是稳定逻辑 Document ID;`filename`、`media_type`、`size_bytes`、`status`、
|
||||
`error_message`、`created_at` 和 `updated_at` 描述本次接收的不可变 DocumentVersion。上传同名新版本
|
||||
时,旧 Active Version 继续作为 `GET /documents/{id}` 的当前投影,`POST` 的 `202` 响应不得混入
|
||||
旧版大小或 `ready` 状态。Job、Document 与 Version 标识不一致或 Source Hash 缺失时失败关闭。
|
||||
|
||||
服务流式计算 SHA-256 Source Hash。同一 Knowledge Base 内 Hash 相同返回 `409` 和稳定错误码
|
||||
`document_source_already_exists`,并给出既有 `document_id`、`document_version_id`;同名但 Hash
|
||||
不同且位于同一 Folder 时复用逻辑 Document ID 并创建递增的 inactive Version;不同 Folder 中的
|
||||
同名文件是独立逻辑 Document。不同 Knowledge Base 可接收相同文件名和 Hash。旧公共兼容数据保留
|
||||
在独立 `public/public` 唯一范围。Job、Chunk 和派生产物绑定该 Version,
|
||||
索引成功后才激活。失败候选不会改变列表/详情所投影的旧 Active Version。
|
||||
首次并发接收同名不同内容时,数据库唯一约束决定逻辑 Document 胜者,失败请求在确认胜者存在后
|
||||
执行有界 Metadata 重试并创建后续 Version;不会把正常创建竞争直接映射为 `502`。
|
||||
|
||||
`GET /api/v1/documents` 可省略筛选参数以列出全部可读文档,也可提交 `knowledge_base_id` 缩小到一个
|
||||
可读知识库。知识库范围内的 `folder_scope` 为 `all`(默认,包含根目录与全部 Folder)、`root`
|
||||
(只含 `folder_id=null` 的知识库根目录文档)或 `subtree`(指定 Folder 及全部后代);`subtree`
|
||||
必须同时提交 `folder_id`,其他范围禁止提交 `folder_id`。ACL Document ID 集合始终先解析,KB/
|
||||
Folder 条件只能与该集合取交集,不能扩大权限。未授权 Knowledge Base、未知 Folder 或跨 KB Folder
|
||||
统一返回 `404`;缺少 KB、缺少 Subtree Folder 或参数组合冲突返回 `422`。旧公共兼容身份只能使用
|
||||
无筛选列表,不得借筛选参数进入规范 Knowledge Base。
|
||||
|
||||
## Compiled Knowledge 生命周期与检索契约(EKP-090~092)
|
||||
|
||||
V1 类型为 `summary | entity | concept | faq | sop | comparison | overview | custom_wiki_page`。
|
||||
EKP-090 创建 `draft`;EKP-091 实现 `draft → review_required → published|rejected`,以及由 Source
|
||||
变化产生的 `stale`;EKP-092 把仍可核验的 `published` 内容接入正式 Search。所有写操作携带
|
||||
`expected_revision`,过期修订返回 `409`;AI 始终只创建 Draft,不存在自动发布路径。
|
||||
|
||||
手工草稿请求包含 `knowledge_base_id`、`title`、`knowledge_type`、`content` 和 1~50 个去重的
|
||||
`source_chunk_ids`。所有 Chunk 必须真实存在、属于同一 Organization/Knowledge Base,且调用者对
|
||||
其 Document 具有 Read 权限、对目标 KB 具有 Write 权限;不接受客户端上传的来源正文、Document
|
||||
Version 或 Provider 信息。AI 请求以同一来源集合加可选 `title_hint`/`instruction` 先完成 Provider、
|
||||
来源权限与数据策略校验,再持久化 Compilation Job 并返回 `202 Accepted`;API 请求不调用 LLM。
|
||||
未配置 Provider 返回 `503`;`local_only` KB 在接收 Job 前拒绝 cloud Provider 并返回 `409`。
|
||||
|
||||
Compilation Job 状态为 `queued | running | retrying | succeeded | failed | cancelled`。查询响应包含
|
||||
稳定 `job_id`、知识库/类型、有序来源 Chunk ID、Attempt 次数/上限、安全错误、下次执行时间、
|
||||
Attempt 历史和成功后的 `compiled_knowledge_id`,不返回 instruction、来源正文、Provider 响应或租约
|
||||
Token。Worker 每次执行前重新解析提交者的 KB Write 与全部 Document Read 权限,再由编纂服务重新
|
||||
核验 Chunk 所属 Active Version 和数据处理策略。瞬时 LLM/数据库故障有界退避重试;权限、来源、
|
||||
契约或策略变化永久失败关闭。Broker 只携带 Job ID,定时恢复扫描补投 queued、到期 retrying 和
|
||||
租约过期 running Job;`generation_job_id` 唯一约束确保重复投递只产生一条 Draft。
|
||||
|
||||
响应至少包含 `compiled_id`、标题/类型/正文、`source_document_ids`、`source_chunk_ids`、明确的
|
||||
`source_document_versions` Document/version 对、有序 `sources`、`created_by`、创建/更新时间、
|
||||
状态、`origin=manual|ai`、可空 `provider_model`、`revision`、`last_verified_at` 和
|
||||
`permissions={mode: source_intersection, resolved: true}`。每个 Source 还返回稳定 `source_id`、文件名、
|
||||
Chunk 顺序/章节、Source Version 创建时间和 `state=current|outdated|deleted`;生命周期返回
|
||||
submitter/reviewer、提交/
|
||||
审核/发布时间、退回原因和稳定 Stale 原因。读取时必须同时满足 KB Read 权限和全部
|
||||
来源 Document Read 权限,任一来源无权时整条结果隐藏;禁止先返回 Compiled 内容再过滤来源。
|
||||
|
||||
更新使用完整 title/type/content 与 `expected_revision`;`draft` 可直接更新,`rejected|stale` 可修订
|
||||
回新 Draft。修订 Stale 时可用 1~50 个 `source_chunk_ids` 替换整组来源;旧冲突配对随旧来源集合
|
||||
失效,生命周期 AuditLog 仍保留动作证据。
|
||||
|
||||
提交审核与发布都会核验每个精确 Source 仍属于未删除 Document 的 Ready Active Version;失败会
|
||||
原子标记 `stale` 并返回 `409`。新 DocumentVersion 激活、引用文档逻辑删除、ACL 新增或删除会在
|
||||
同一规范事务内自动使受影响知识进入 `stale`,原因分别为 `source_version_changed`、
|
||||
`source_deleted`、`source_permissions_changed`。引用文档的 DELETE 返回 `204`,但只移出检索并
|
||||
保留 Source/Version/Chunk 作为不可变证据;普通 Document API 隐藏该逻辑删除项。无 Compiled
|
||||
引用的 Document 暂时沿用既有直接清理路径。
|
||||
|
||||
冲突由审核人明确确认:请求提供有界 Claim 和同一知识项内两个不同 `source_id`;响应保留 Claim、
|
||||
两侧 Source/Version/创建时间、报告人与解决状态。未解决冲突阻断 Publish,解决必须提供有界说明。
|
||||
系统不会让 LLM 自动折中冲突,也不会在日志中保存 Claim、正文或标题。提交、冲突、解决、发布、
|
||||
退回、修订和自动 Stale 均写入脱敏 AuditLog。
|
||||
|
||||
Search 请求:
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "产品A退款周期是多少?",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
|
||||
Search 响应:
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"document_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
|
||||
"document_version_id": "a23c91d7-6a8c-4a66-b4ca-6220d8e3fb09",
|
||||
"document_version": 2,
|
||||
"filename": "产品A说明书.pdf",
|
||||
"page": 12,
|
||||
"page_end": 13,
|
||||
"section": "退款政策",
|
||||
"section_path": ["产品A", "售后服务", "退款政策"],
|
||||
"sheet_name": null,
|
||||
"row_range": null,
|
||||
"column_range": null,
|
||||
"chunk_id": "76ba1c30-bff7-44e5-9989-fb8372f54f46",
|
||||
"content": "……",
|
||||
"context": "命中 Chunk 前后各一个规范 Chunk 组成的有界可读上下文……",
|
||||
"score": 0.82,
|
||||
"rerank_score": 0.94,
|
||||
"source_type": "raw"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`page`、`page_end` 是命中 Chunk 覆盖的一基起止页;单页命中两者相同,非分页来源均为 `null`。
|
||||
跨页命中只能把原始 PDF 从 `page` 打开,客户端必须展示完整页范围,不得把起始页描述成精确页。
|
||||
Spreadsheet/Table 命中时,`sheet_name` 为 Sheet 名或 `null`,`row_range`、`column_range` 为
|
||||
一基闭区间二元数组,例如 `[2, 8]`;非表格 Chunk 三个字段均为 `null`。这些来源字段来自
|
||||
OceanBase 规范 Chunk,不信任向量索引中的冗余 Metadata。
|
||||
|
||||
`content` 始终是参与召回和 Reranker 的精确命中 Chunk;`context` 从同一 DocumentVersion、同一
|
||||
Chunk 策略的前后各一个规范 Chunk 拼接,最多 2400 字符,用于避免标题或极短结构节点脱离正文。
|
||||
上下文不改变候选排序和分数,也不从向量 Metadata 恢复内容。
|
||||
`section_path` 是从顶层到命中节点的完整规范章节路径;`section` 保留为兼容摘要字段。
|
||||
`source_type=raw` 明示结果来自不可变 Source 的派生 Chunk,后续 Compiled 检索不得复用该值伪装
|
||||
成 Raw 结果。所有来源字段以关系库中的规范记录为准;缺少当前 Profile 的 `ChunkIndexState`、
|
||||
模型/维度或 Chunk 策略不一致、Version 非 Ready Active、文件名或正文漂移时整次检索失败关闭。
|
||||
|
||||
正式身份 Search 还可返回 `source_type=compiled`。Compiled 候选只从已经通过检索前 ACL Filter 的
|
||||
Raw Source 候选扩展,其所属 Knowledge Base 和全部底层 Document 都必须位于调用者已解析 Read
|
||||
Scope;旧公共兼容入口
|
||||
不返回 Compiled 内容。只有 `status=published`、`published_at`/`last_verified_at` 完整且每个精确
|
||||
Source 仍为未删除 Ready Active Version 的知识可以进入联合 Reranker。Draft、ReviewRequired、
|
||||
Rejected、Stale 和 Archived 全部排除。
|
||||
|
||||
Compiled Result 不伪造 Raw Document/Chunk 字段,返回 `compiled_knowledge_id`、
|
||||
`knowledge_base_id`、`title`、`knowledge_type`、已发布 `content`、有界 `context`、排序分数和
|
||||
`underlying_sources[]`。每个底层来源包含 Document/Version、文件名、页/章节/Sheet/行列范围、
|
||||
Chunk ID/顺序与精确正文,供客户端进入同一 Document Inspector 或不可变 Source。`score` 是触发
|
||||
该知识的最强授权 Raw 证据的 Fusion Score;标题与正文随后和 Raw Chunk 一起进入同一次 Reranker,
|
||||
联合候选总数仍不超过 Active Profile 的 `hybrid_top_n`,最终结果不超过请求/Profile TopK。
|
||||
|
||||
`GET /api/v1/documents/{document_id}/content` 可通过 `document_version_id` 查询参数选择精确不可变
|
||||
版本;省略时优先读取 Active Version,尚未激活时读取最新候选。响应包含 Version、状态、Active
|
||||
标记、Index Profile 及按 `chunk_index` 排序的 Chunk 正文、Page/Section/Sheet/行列位置。只返回
|
||||
当前 Active Index Profile 已成功索引的规范 Chunk,处理中尚无完整索引时 `chunks` 为空。
|
||||
`quality_status` 区分 `available`、`not_recorded` 和 `unavailable`;`page_quality` 只返回稳定的页码、
|
||||
OCR Confidence 和警告,不暴露 Provider 原始响应。`processing_job` 返回该 Version 的持久化 Job
|
||||
及按次数排序的 Attempt 历史,不暴露 Broker 或 Provider 细节。质量 Artifact 缺失或损坏时仍
|
||||
返回规范 Chunk,并以 `unavailable` 明示降级。该接口是解析/检索溯源视图,不保留 PDF 或 Office
|
||||
排版,也不应被客户端描述为完整原文。
|
||||
|
||||
`GET /api/v1/documents/{document_id}/source` 使用相同版本选择规则,流式读取 ObjectStorage 中的
|
||||
不可变 Source,不回显 `storage_key`。PDF、纯文本、Markdown 和图片使用安全 inline 响应;HTML、
|
||||
Office 与其他格式强制下载并使用 `X-Content-Type-Options: nosniff`,避免同源主动内容执行。
|
||||
|
||||
公开错误语义:请求校验失败返回 `422`,不支持的上传类型返回 `415`,文件过大返回 `413`,
|
||||
文档、Version、Job 或批次不存在返回 `404`;重复 Source 或活动 Job 返回 `409`,前者的
|
||||
`detail.code` 为 `document_source_already_exists`;运行时 Provider 未配置返回 `503`。上游 Provider
|
||||
或对象存储异常只返回经过收敛的 `5xx` 信息,不回显凭证、存储 Key、请求正文或厂商响应。MinerU 瞬时网络、
|
||||
限流和服务端错误进入既有有界 Job 重试;内容或结构错误按永久失败处理。
|
||||
业务服务首次装配时,规范数据库异常返回 `503 API canonical database is unavailable` 并附带
|
||||
`X-EKP-Dependency: canonical-database`;向量索引初始化、配置兼容或 Schema 检查异常返回
|
||||
`503 API vector index is unavailable` 并附带 `X-EKP-Dependency: vector-index`。这些稳定分类仅用于
|
||||
部署诊断,不返回 SQL、连接地址、表名、Provider 响应或底层异常文本。
|
||||
Search Query 非法或 `top_k` 超过 Active Profile 的 `final_top_k` 时返回安全、稳定的 `422`,响应
|
||||
不披露 Profile ID、模型名或内部候选规模。
|
||||
|
||||
### Search 诊断响应
|
||||
|
||||
`POST /api/v1/search/diagnostics` 复用相同 `SearchRequest`,只供受控网络内的管理端 Search Lab。
|
||||
它必须只执行一次完整链路,不能为收集阶段信息再次发送 Query 或产生与普通 Search 不同的排序。
|
||||
响应包含:
|
||||
|
||||
- 规范化 Query;正式身份返回 `document_filter` Scope、`acl_filter_applied=true` 及已解析 Document
|
||||
数量;显式旧公共兼容入口才返回 `public_knowledge`/`false`,不得混淆两种安全边界;
|
||||
- Active Profile ID、Chunk/Embedding/Sparse/Reranker 标识、Dense/Sparse/Hybrid/Final 数量、RRF
|
||||
常量和权重;不包含 Endpoint、Key 或 Provider 原始配置;
|
||||
- `dense_candidates`、`sparse_candidates`、`fusion_candidates`、`compiled_candidates`、
|
||||
`reranked_candidates` 与最终
|
||||
`results`。阶段候选提供规范身份、最多 1200 字符的显式截断正文、名次和适用的分支/Fusion/
|
||||
Reranker 分数;最终 `results.content` 仍是完整精确命中 Chunk;
|
||||
- Dense、Sparse、Fusion、Compiled Expansion、Reranker、Source Enrichment 和总耗时,单位为
|
||||
毫秒且使用单调时钟。
|
||||
|
||||
阶段候选仍受检索前 Scope 约束。诊断接口不允许客户端修改 Profile、权重、模型或候选规模,也不
|
||||
提供独立 Provider 调用能力;非法 Query/Profile TopK 返回与普通 Search 一致的 `422`,其他错误
|
||||
收敛为不含内部异常的 `502`。
|
||||
|
||||
## Demo 管理端客户端路由
|
||||
|
||||
DEMO-010 提供 `/admin/upload`、`/admin/documents` 和 `/admin/search` 三个客户端路由;`EKP-080`
|
||||
新增 `/admin/overview` 运营总览,以同一个 `GET /documents` 响应聚合 Source、Processing、Ready
|
||||
库存、就绪率和最近文档,不引入第二套统计事实源。`EKP-081`
|
||||
在既有路由内增加 Document Inspector、规范 Chunk 来源轨道、原始文件入口,以及 Search Lab 的
|
||||
可读上下文、精确命中片段和命中来源定位。Document Inspector 提供“原始文件预览”和“解析与
|
||||
溯源”双视图:PDF 从命中起始页打开,Markdown、纯文本和图片安全在线预览,Office/HTML 等主动
|
||||
或浏览器不稳定格式保持下载回退。跨页 Chunk 展示完整起止页范围,不把起始页误述为精确页;
|
||||
Inspector 同时展示持久化处理流水线、Attempt 历史和低置信度 OCR 页警告。`EKP-056` 将 Search
|
||||
Lab 接到 `/search/diagnostics`,显示 Query/Scope/ACL、Profile、Dense、Sparse、Fusion、Reranker
|
||||
和 Final 信号轨;最终来源操作仍复用同一 Inspector 与不可变 Source 入口。
|
||||
`EKP-082` 新增 `/admin/knowledge-bases` 和 `/admin/permissions`,前者管理 Knowledge Base/Folder,
|
||||
后者管理 User/Group/Role、成员/角色关系和显式 ACL。浏览器 Token 仅保存在当前标签页的
|
||||
`sessionStorage`;原文预览与新窗口打开先发送带认证 Header 的请求,再使用短生命周期 Blob URL,
|
||||
不会把 Token 写入链接。
|
||||
`EKP-083` 在 `/admin/documents` 增加两级范围筛选:先选择一个可读 Knowledge Base,再选择全部
|
||||
文件夹、知识库根目录或显示完整祖先路径的 Folder 子树;列表同步显示结果数量、当前范围和筛选空
|
||||
状态。页面直接调用上述 Document 列表查询参数,不在浏览器下载全部文档后自行模拟权限筛选。
|
||||
`EKP-090~091` 的 `/admin/compiled-knowledge` 从可读 KB、Ready Document 和真实 Chunk 选择来源,
|
||||
创建手工/AI 草稿、按 revision 编辑,展示当前/历史/删除来源轨道,并提供核验提交、双来源冲突记录/
|
||||
解决、发布、退回和 Stale 修订入口;来源跳转仍进入 Document Inspector,不把顺序 Chunk 描述为原文。
|
||||
`/admin` 默认展示运营总览;服务根路径 `/` 以 `307 Temporary Redirect` 同源跳转到 `/admin`,
|
||||
使用户直接打开部署域名即可进入服务台。它们是同源静态页面,不是新增业务 API,也不改变
|
||||
上述公开 REST 契约;所有业务操作仍调用 `/api/v1`。
|
||||
|
||||
`EKP-092` 在 `/admin/search` 的诊断轨中增加“已发布知识”阶段。Final 中 Raw 结果继续显示上下文、
|
||||
精确命中和原始文件入口;Compiled 结果显示“已审核知识”、已发布正文与底层来源证据轨道,每个
|
||||
来源分别提供“定位底层来源”和“查看或下载原始文件”,不把 L2 正文描述成原文。
|
||||
|
||||
## V1 身份与管理契约(EKP-082)
|
||||
|
||||
`knowledge:manage` 是 Organization 管理员 Scope;除 Session 自省外,所有管理路径还必须具备该
|
||||
Scope 并绑定一个当前 Organization 内的 Active 规范 User。Session 向任意有效规范身份返回自身
|
||||
`scopes` 和 `credential_type`,供客户端收敛入口。管理接口只返回身份/资源 Metadata,不创建或
|
||||
返回 Token:
|
||||
|
||||
| 方法 | 路径 | 行为 |
|
||||
|---|---|---|
|
||||
| `GET` | `/api/v1/administration/session` | 验证当前 Organization/User 映射 |
|
||||
| `GET/POST` | `/api/v1/administration/users` | 列出或创建无凭据 User |
|
||||
| `GET/POST` | `/api/v1/administration/groups` | 列出或创建 Group,列表包含成员与 Role ID |
|
||||
| `PUT/DELETE` | `/api/v1/administration/groups/{group_id}/members/{user_id}` | 幂等添加/移除成员 |
|
||||
| `GET/POST` | `/api/v1/administration/roles` | 列出或创建 Role,列表包含直接 User/Group 分配 |
|
||||
| `PUT/DELETE` | `/api/v1/administration/roles/{role_id}/users/{user_id}` | 幂等分配/撤销 User Role |
|
||||
| `PUT/DELETE` | `/api/v1/administration/roles/{role_id}/groups/{group_id}` | 幂等分配/撤销 Group Role |
|
||||
| `GET/POST` | `/api/v1/administration/knowledge-bases` | 列出或创建 Knowledge Base |
|
||||
| `GET/POST` | `/api/v1/administration/knowledge-bases/{knowledge_base_id}/folders` | 列出或创建同 KB Folder |
|
||||
| `GET/POST` | `/api/v1/administration/knowledge-bases/{knowledge_base_id}/access-control` | 列出或创建 User/Group/Role ACL |
|
||||
| `DELETE` | `/api/v1/administration/access-control/{entry_id}` | 删除一个显式 ACL |
|
||||
|
||||
Permission Resolver 在 Document 列表、详情、内容、原文、Job、Batch、删除、Search 和 Search
|
||||
Diagnostics 前解析同一规范 User/Group/Role/ACL 数据。Document 无权与不存在统一隐藏,空 Scope
|
||||
不调用检索 Provider。旧未分配 Document 仍属于兼容公共范围,不会自动迁入任意知识库或 ACL。
|
||||
面向上传选择器的非管理 `GET /knowledge-bases` 与 `GET /knowledge-bases/{id}/folders` 只要求
|
||||
`knowledge:read`,且只投影当前 User 可读的 Active KB/Folder;无权 KB 返回 `404`。
|
||||
首次配置可通过 `ekp-api-identity-bootstrap` 按已存在于运行时 Secret 的映射显式创建精确
|
||||
Organization/User;命令行不接受 Token 参数。
|
||||
|
||||
`EKP-017` 复用既有持久化 Ingestion Job/Attempt 并新增内部 AuditLog 模型;本任务不新增公开
|
||||
审计接口,也未把登录、文档、检索、权限或 MCP 操作接入 AuditLog。
|
||||
|
||||
集群重启、扩缩容和滚动部署不得改变本契约或要求浏览器重新注入 Provider 配置。数据库、BOS、
|
||||
模型与队列凭据由服务端 Secret 提供,任何 API 响应都不得返回这些值。
|
||||
|
||||
DEPLOY-003 增加两个不进入 `/api/v1` OpenAPI 的运维端点:`GET /healthz` 只表示 API 进程可用;
|
||||
`GET /metrics` 使用 Prometheus text format 返回进程存活时间、按 HTTP method/路由模板/status
|
||||
聚合的请求量和按 method/路由模板聚合的耗时。两者不得包含 Query、文件名、正文、身份、
|
||||
Provider Endpoint 或凭据。响应附带安全 `X-Request-ID`,客户端提供的值只有匹配
|
||||
`[A-Za-z0-9._:-]{1,64}` 时才会沿用。
|
||||
默认 dev 镜像执行规范库迁移时,`/healthz` 额外返回 `migration_status`、安全的
|
||||
`migration_revision` 和 `migration_error_type`;迁移失败时 `status=degraded`,但仍返回 `200` 供
|
||||
中央部署链诊断,所有业务入口继续失败关闭。显式生产 target 不返回这些 dev 专用字段。
|
||||
|
||||
## 异步入库契约(DEMO-015~016 已实现)
|
||||
|
||||
| 方法 | 路径 | 行为 |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/v1/documents` | 已实现:单文件接收并返回 Document/Job 标识 |
|
||||
| `POST` | `/api/v1/ingestion-batches` | multipart 多文件/文件夹接收,返回批次及逐文件 Document/Job 标识 |
|
||||
| `POST` | `/api/v1/ingestion-archives` | multipart 单 ZIP 安全展开,返回批次及逐文件 Document/Job 标识 |
|
||||
| `GET` | `/api/v1/ingestion-batches/{batch_id}` | 返回批次汇总和逐文件最新状态 |
|
||||
| `GET` | `/api/v1/jobs/{job_id}` | 已实现:返回持久化 Job 状态 |
|
||||
|
||||
单文件响应示意:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
|
||||
"organization_id": "11111111-1111-4111-8111-111111111111",
|
||||
"knowledge_base_id": "22222222-2222-4222-8222-222222222222",
|
||||
"folder_id": null,
|
||||
"filename": "退款政策.md",
|
||||
"media_type": "text/markdown",
|
||||
"size_bytes": 1024,
|
||||
"status": "uploaded",
|
||||
"error_message": null,
|
||||
"created_at": "2026-09-02T12:00:00Z",
|
||||
"updated_at": "2026-09-02T12:00:00Z",
|
||||
"job_id": "7adc4278-1157-4c3d-95a4-6926ad77a04e",
|
||||
"document_version_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
|
||||
"document_version": 1,
|
||||
"source_hash": "6ca13d52ca70c883e0f0bb101e425a89e8624de51db2d2392593af6a84118090",
|
||||
"job_status": "queued",
|
||||
"max_attempts": 3
|
||||
}
|
||||
```
|
||||
|
||||
普通批量字段名为 `files`。文件夹上传必须再按相同顺序重复提交 `relative_paths`,每个值使用浏览器
|
||||
提供的 POSIX 相对路径(例如 `产品手册/区域/退款.md`);两组数量不一致返回 `422`。目录模式必须
|
||||
指定正式 `knowledge_base_id`,可选 `folder_id` 是目录层级的基准父节点,跨 Knowledge Base 或
|
||||
不存在的基准 Folder 返回 `404`,且不会创建批次、Folder 或对象。成功返回 `202`。响应示意:
|
||||
|
||||
```json
|
||||
{
|
||||
"batch_id": "d21f8012-4c4e-4bb4-89af-e11c0f9f470e",
|
||||
"organization_id": "11111111-1111-4111-8111-111111111111",
|
||||
"knowledge_base_id": "22222222-2222-4222-8222-222222222222",
|
||||
"status": "processing",
|
||||
"total_files": 2,
|
||||
"accepted_files": 1,
|
||||
"rejected_files": 1,
|
||||
"completed_files": 1,
|
||||
"created_at": "2026-09-02T12:00:00Z",
|
||||
"items": [
|
||||
{
|
||||
"item_index": 0,
|
||||
"filename": "退款政策.md",
|
||||
"relative_path": "产品手册/华东/退款政策.md",
|
||||
"acceptance_status": "accepted",
|
||||
"document_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
|
||||
"document_version_id": "6e5ec370-5a45-4eb7-95d7-e84267ad8752",
|
||||
"job_id": "7adc4278-1157-4c3d-95a4-6926ad77a04e",
|
||||
"job_status": "queued",
|
||||
"stage": null,
|
||||
"attempt": 0,
|
||||
"max_attempts": 3,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Job 公开状态为 `queued | running | retrying | succeeded | failed | cancelled`,并通过 `stage`
|
||||
表示 `parsing | cleaning | chunking | embedding | indexing`。错误字段只返回有界、安全摘要。
|
||||
正式身份创建的批次保存并返回 `organization_id`/`knowledge_base_id`,即使所有文件均在准入阶段
|
||||
被拒绝,轮询仍按该 KB 的有效 Read Scope 鉴权;旧公共兼容批次两字段为 `null`。批次状态为
|
||||
`processing | completed | partial_failure | failed`;逐文件接收状态为
|
||||
`pending | accepted | rejected`。批次允许逐文件部分接收/失败,不以一个文件的处理结果回滚
|
||||
其他文件。文件数量、单文件大小和请求总大小分别由 `BATCH_MAX_FILES`、`MAX_UPLOAD_BYTES`
|
||||
和 `BATCH_MAX_TOTAL_BYTES` 限制。数量超限返回 `413`;类型、大小或内容不合规作为该文件的
|
||||
`rejected` 安全结果返回,其他合规文件继续接收。
|
||||
|
||||
服务端只持久化通过校验的 `relative_path`,普通文件上传和非法路径均返回 `null`。合法路径最长
|
||||
1024 字符、目录最多 32 层、每段最多 255 字符;绝对路径、反斜线、空段、`.`、`..`、控制字符、
|
||||
首尾空白以及路径末尾与 multipart 文件名不一致均只拒绝对应文件并返回固定安全错误。合法目录段
|
||||
在已授权基准 Folder 下逐级创建或复用;空目录不会产生 Folder 或 Document。文件类型、大小、共享
|
||||
预算、同 KB Source Hash 去重和同名版本语义与普通批次一致。
|
||||
|
||||
ZIP 字段名为 `archive`,必须同时指定正式 `knowledge_base_id`,可选 `folder_id` 作为包内目录的基准
|
||||
父节点。服务在创建 Batch、Folder、Document 或对象前验证完整中央目录、CRC 和压缩/展开预算;拒绝
|
||||
绝对路径、父级跳转、反斜线、重复路径、符号链接/特殊文件、加密条目、嵌套 ZIP 和损坏包,忽略
|
||||
`__MACOSX`、`.DS_Store` 与 AppleDouble 元数据。结构错误整包返回 `422`,格式错误返回 `415`,
|
||||
预算超限返回 `413`;结构安全后,不支持的普通文件类型作为逐项 `rejected`,其余文件继续入库。
|
||||
根层文件放入 ZIP 文件名对应的 Folder,包内既有多根目录保持原路径;空目录不落库,ZIP 本身不保存
|
||||
为 Source。压缩包、条目、展开总量、文件数和压缩比分别受 `MAX_UPLOAD_BYTES`、
|
||||
`MAX_UPLOAD_BYTES`、`BATCH_MAX_TOTAL_BYTES`、`ZIP_MAX_FILES` 和
|
||||
`ZIP_MAX_COMPRESSION_RATIO` 约束,目录仍遵守 32 层/1024 字符安全路径上限。
|
||||
|
||||
Job 查询响应包含 `document_version_id`。自动重试是服务端 Job 行为,不要求客户端再次上传,也
|
||||
不新增公开手动重试接口。每次重试前只清理该 Job/Version 的派生数据,保留原始文件、稳定的
|
||||
Document ID、历史 Version 和旧 Active 检索;达到次数上限或清理失败后进入 `failed`。内部
|
||||
Lease Token 只用于隔离过期 Worker,不进入 Job 响应,也不改变上述公开状态和字段。
|
||||
124
hy4/_kp/docs/api/MCP_CONTRACT.md
Normal file
124
hy4/_kp/docs/api/MCP_CONTRACT.md
Normal file
@ -0,0 +1,124 @@
|
||||
# MCP 契约
|
||||
|
||||
> 当前状态:`EKP-070~072` 已完成 MCP 身份、ACL-aware Search、五个 P0 Tool 与客户端兼容
|
||||
> 验收器的本地开发闭环。远程 MCP Gateway 与 WorkBuddy 现场验收仍作为独立非开发
|
||||
> 任务搁置,不得据此声称目标环境 MCP 已验收。
|
||||
|
||||
P0 业务 Tool:
|
||||
|
||||
- `search_knowledge`
|
||||
- `get_document`
|
||||
- `get_document_section`
|
||||
- `get_source`
|
||||
- `list_knowledge_bases`
|
||||
|
||||
每个请求都必须映射到用户或服务身份,并使用与 REST 相同的授权和应用服务。不得暴露 OceanBase Vector、BOS、模型名、Parser 或其他基础设施选型。
|
||||
|
||||
## P0 Tool 契约
|
||||
|
||||
所有 Tool 都要求 `knowledge:search` scope 和规范 Organization/ACL User;除历史
|
||||
`search_knowledge` 公共 Demo 兼容外,旧静态 Token 调用其他 Tool 必须失败关闭。Document、Version
|
||||
或 Chunk 不存在、跨归属或无权访问统一返回 `knowledge resource is unavailable`,不得帮助调用者
|
||||
枚举越权资源。
|
||||
|
||||
`list_knowledge_bases` 无参数,只返回当前身份可读且 `mcp_enabled=true` 的 Active Knowledge Base:
|
||||
`knowledge_base_id`、`name`、`description`、`visibility` 和 `readable_document_count`。即使空间暂无
|
||||
Document,只要调用者对 Knowledge Base 本身有 READ 权限也可见;仅通过 Folder/Document ACL
|
||||
获得下级内容时,对应容器也可见。响应不返回 Owner、ACL 行或数据处理内部配置。
|
||||
|
||||
`get_document` 输入 `document_id` 和可选 `document_version_id`。省略 Version 时优先选择 Active
|
||||
Version,否则选择最新候选;响应返回 Knowledge Base、Version、文件名、媒体类型、大小、状态、
|
||||
Active 标记、当前 Profile 已索引的 Chunk 数,以及最多 256 条去重 `section_paths`。
|
||||
`outline_truncated=true` 表示仍有未返回的大纲,不得把该响应描述为正文或原文件。
|
||||
|
||||
`get_source` 输入 `document_id`、`chunk_id` 和可选 `document_version_id`,用于把 Search 命中解析为
|
||||
一个精确规范 Chunk。响应返回 Document/Knowledge Base/Version/文件名,以及 Chunk 的
|
||||
`chunk_id`、`chunk_index`、Page/Section/Sheet/行列位置、`content` 和 `source_type=raw`。
|
||||
该 Tool 返回可引用的文本证据,不返回原文件字节;原始 PDF/Office/图片等下载继续使用受控 REST
|
||||
`GET /api/v1/documents/{document_id}/source`。
|
||||
|
||||
`get_document_section` 输入 `document_id`、`anchor_chunk_id`、可选 `document_version_id`、
|
||||
`offset`(默认 0)与 `max_chunks`(默认 12,范围 1~32)。它按 `chunk_index` 返回与 Anchor 具有
|
||||
相同完整 `section_path` 的规范 Chunk,并提供 `total_chunk_count` 和 `next_offset`;调用方应持续
|
||||
使用同一 Anchor 和 `next_offset`,直至其为 `null`。空章节路径只返回 Anchor 本身,避免把整份
|
||||
无结构文档误当成一个无界响应。
|
||||
|
||||
以上读取只接受当前 Active Index Profile 有 `ChunkIndexState` 证据的规范 Chunk;不从向量
|
||||
Metadata 恢复正文,不返回 Index Profile、Provider 或存储 Key。
|
||||
|
||||
## Demo 优先纵向切片(DEMO-012,已实现)
|
||||
|
||||
`DEMO-012` 已提升为当前最高实现优先级。首个可运行 Gateway 只注册
|
||||
`search_knowledge`,其输入与 `/api/v1/search` 对齐:
|
||||
|
||||
```json
|
||||
{
|
||||
"query": "产品A退款周期是多少?",
|
||||
"top_k": 5
|
||||
}
|
||||
```
|
||||
|
||||
输出保留业务来源字段:`document_id`、`document_version_id`、`document_version`、`filename`、
|
||||
`page`、`page_end`、`section`、`section_path`、`sheet_name`、`row_range`、`column_range`、
|
||||
`chunk_id`、`content`、`context`、排序分数和 `source_type`。`content` 是精确命中 Chunk;
|
||||
`context` 是与 REST 一致的有界相邻段落,只改善阅读与
|
||||
来源理解,不改变候选排序或分数。
|
||||
|
||||
`EKP-092` 后正式 User/Service 身份还可收到 `source_type=compiled` 的已发布知识;其响应与 REST
|
||||
一致,包含 `compiled_knowledge_id`、Knowledge Base、标题/类型/正文、排序分数和完整
|
||||
`underlying_sources[]`。每个底层来源保留精确 DocumentVersion/Chunk 与位置/正文,可继续调用
|
||||
`get_source`、`get_document_section` 或 REST Source 入口。Gateway 不自行复制扩展、排序或权限
|
||||
逻辑;Draft/ReviewRequired/Rejected/Stale/Archived、所属 Knowledge Base 或任一来源不在调用者
|
||||
READ Scope,以及未启用 MCP 的 Knowledge Base 都不会进入候选。旧公共 Demo 兼容身份继续只返回
|
||||
Raw 结果。
|
||||
分页范围与 REST 一样使用一基起止页,单页时两者相同;Spreadsheet 范围使用一基闭区间二元数组,
|
||||
非对应来源字段为 `null`。Gateway 必须复用
|
||||
现有 Search Service 或正式 REST API,不得复制检索、来源补全或排序逻辑;默认只返回 Active
|
||||
Version。
|
||||
|
||||
`EKP-053` 后共享 Search Service 使用 Dense/Sparse Weighted RRF + Reranker;输出 `score` 为融合
|
||||
分数,`rerank_score` 为精排分数。该行为复用不新增 MCP Tool,也不恢复已搁置的远程验收任务。
|
||||
|
||||
Demo 验证也必须建立 user/service identity 映射边界,禁止使用共享超级管理员 Token。由于
|
||||
`EKP-070` 在保留 Demo 兼容层的同时新增运行时 User Token 与 Service API Key 映射。每个正式
|
||||
凭据都必须绑定 `client_id`、`subject`、`organization_id`、规范 `acl_user_id` 与最小 scopes;
|
||||
Gateway 调用 `AuthorizedSearchService`,在 Dense/Sparse Top-N 前应用同一精确 Document Filter,
|
||||
并只纳入 `mcp_enabled=true` 的 Knowledge Base。身份缺失、User 不存在/禁用/跨 Organization、
|
||||
ACL 服务未装配或规范绑定不完整时全部失败关闭。Service API Key 的 `acl_user_id` 必须是为该
|
||||
服务专门配置的规范授权主体,不得映射到共享超级管理员。
|
||||
|
||||
旧 `MCP_AUTH_TOKEN` 只允许单一公共知识 Demo 兼容路径,不能与正式身份列表同时配置,也不能
|
||||
用于非公共知识、ACL 完成声明或远程生产接入。`DEMO-012` 只能证明身份传递、认证拒绝和业务
|
||||
服务复用,不能替代 `EKP-070` 的规范身份与 ACL 证据。错误响应和日志不得包含 Provider
|
||||
Endpoint、凭据、数据库表名或模型名。
|
||||
|
||||
开发验收覆盖:五个 P0 Tool 发现、参数校验、成功/空结果、User/Service 身份、ACL 越权隐藏、
|
||||
章节分页、认证失败和安全错误收敛。`EKP-072` 还提供可执行的官方 MCP Client 验收器,在同一
|
||||
会话中验证 Search → Source/Document/Section/Knowledge Base 的来源链;远程只接受 HTTPS,明文
|
||||
HTTP 仅允许本机回环地址。WorkBuddy Enterprise 的目标环境现场验收属于独立非开发验收,不由
|
||||
本地确定性 Provider 结果替代。
|
||||
|
||||
实现使用独立 Streamable HTTP Gateway,默认端点为 `http://127.0.0.1:8766/mcp`。请求必须
|
||||
携带 `Authorization: Bearer <token>`;正式凭据列表只从运行时 Secret 的
|
||||
`MCP_IDENTITY_CREDENTIALS` 读取,支持 `user_token` 和 `service_api_key`,并映射为显式
|
||||
Client、主体、Organization、ACL User 和 scopes。配置是本地受控凭据适配层,为 OIDC/OAuth2/
|
||||
企业 SSO Token Verifier 保留替换边界;不伪装成已经接入外部 IdP。
|
||||
|
||||
公开错误边界:缺失/无效 Token 为 HTTP `401`,scope 不足为 `403`;Tool 参数不合法以及
|
||||
`top_k` 超过 Active Profile 最终上限时返回安全的 MCP 请求错误;其他检索运行失败只返回
|
||||
`knowledge search failed`。Host/Origin allowlist 默认只允许
|
||||
本机,远程部署必须显式注入公开域名。
|
||||
|
||||
## 历史公共知识库部署边界(PLAN-002)
|
||||
|
||||
PLAN-002 历史集群切片只暴露只读 `search_knowledge`,并为 WorkBuddy 配置稳定的 HTTPS MCP 地址。
|
||||
Credential 由 Kubernetes Secret 在服务端运行时注入,客户端只保存管理员分发的最小
|
||||
`knowledge:search` Token;Pod 重启或滚动升级不得要求重新生成配置。公共知识域不等于匿名
|
||||
访问或超级管理员身份,未来引入非公共知识前必须升级为用户委托身份与 ACL-aware Search。
|
||||
|
||||
当前代码升级到 EKP-071 后应发现本契约列出的五个 P0 Tool;若目标环境仍只发现
|
||||
`search_knowledge`,表示其仍运行历史镜像,不构成当前版本兼容验收。恢复远程 MCP 任务后,验收
|
||||
必须从最终 HTTPS 地址发现且仅发现五个批准 Tool,并完成一次真实、有来源的完整读取链。Gateway
|
||||
失败日志只记录固定事件名,不写请求 Query、异常正文、Provider
|
||||
Endpoint 或 Token;具体请求量与 Pod 可用性由集群入口/Kubernetes 指标补充,当前不新增 MCP
|
||||
业务 Tool。
|
||||
40
hy4/_kp/docs/architecture/ADR.md
Normal file
40
hy4/_kp/docs/architecture/ADR.md
Normal file
@ -0,0 +1,40 @@
|
||||
# 架构决策记录
|
||||
|
||||
- ADR-001:API First + 管理控制台 — 已接受
|
||||
- ADR-002:V1 采用模块化单体 + Worker + MCP Gateway — 已接受
|
||||
- ADR-003:PostgreSQL + 对象存储保存规范数据 — 已由 ADR-012 取代
|
||||
- ADR-004:Mochow 是初始检索后端,但必须可替换 — 已由 ADR-010 取代
|
||||
- ADR-005:为 Embedding、Reranker、Parser 和 LLM 建立 Provider 抽象 — 已接受
|
||||
- ADR-006:RAG + Knowledge Compilation — 已接受
|
||||
- ADR-007:在检索前或检索过程中执行 ACL — 已接受
|
||||
- ADR-008:MCP 隐藏基础设施细节 — 已接受
|
||||
- ADR-009:百度 BOS 是主要对象存储 Provider;本地存储继续用于测试和本地开发 — 已接受
|
||||
- ADR-010:OceanBase Vector 是 `VectorStore` 背后的主要可重建检索索引;PostgreSQL 仍保存规范数据 — 已由 ADR-012 取代
|
||||
- ADR-011:千帆 `bge-large-zh` + `bce-reranker-base` 是初始 Demo 模型基线 — 已接受
|
||||
- ADR-012:OceanBase MySQL 关系表 + BOS 保存规范数据;OceanBase 向量表仍为可重建索引 — 已接受
|
||||
|
||||
## 架构迁移边界
|
||||
|
||||
ARCH-006 的只读环境核验确认,目标平台没有独立 PostgreSQL,OceanBase MySQL 模式同时承载
|
||||
关系数据和向量数据。因此 ADR-012 取代 ADR-003/010 中的 PostgreSQL 部署假设:OceanBase
|
||||
关系表保存 canonical metadata,BOS 保存不可变源文件,OceanBase 专用向量表仍只保存可由
|
||||
canonical Chunk 重建的检索行。共用数据库实例不改变两类数据的所有权边界,也不允许业务层
|
||||
绕过 `VectorStore` 直接依赖 `pyobvector`。
|
||||
|
||||
目标 test 集群为 OceanBase 4.3.5.4 MySQL 模式。ARCH-006 已用只读 `EXPLAIN` 验证
|
||||
`VECTOR_INDEX(... pre_filter)` 产生 `VECTOR INDEX ADAPTIVE SCAN (PRE-FILTER)`;版本下限与
|
||||
运行检查据此调整为 4.3.5.4。生产升级和重新验收仍由运维流程控制。
|
||||
|
||||
## 公共知识交付边界
|
||||
|
||||
PLAN-002 将当前交付阶段限定为企业受控网络内的单一公共知识可见域,并加入 Kubernetes/Helm
|
||||
集群交付任务。这是部署范围和实施顺序调整,不是新的架构选型:继续采用 ADR-002 的模块化
|
||||
单体 API、独立 Worker 和 MCP Gateway,以及架构既定的 Celery/RabbitMQ;OceanBase/BOS
|
||||
边界继续遵循 ADR-012。
|
||||
|
||||
ADR-007 仍为已接受状态。内部 Permission Resolver 和检索前 ACL 下推已实现;开始接收任何非公共
|
||||
知识前,还必须完成可信请求身份、KB-aware Document 归属及 REST/MCP/Source 授权接线。公共知识库也不等于匿名
|
||||
公网开放,Admin 写操作只允许经受控网络或统一接入网关访问,MCP 使用最小只读服务身份。
|
||||
|
||||
本次迁移保留 Demo 的 Dense Top20 → Reranker Top5 行为和当前轻量 Parser 范围。Top50 检索、
|
||||
距离阈值、双 Query RRF、Marker/Pandoc 和 LLM 派生索引内容,需要分别作出产品与评估决策。
|
||||
408
hy4/_kp/docs/architecture/ARCHITECTURE.md
Normal file
408
hy4/_kp/docs/architecture/ARCHITECTURE.md
Normal file
@ -0,0 +1,408 @@
|
||||
# 企业内部智能知识库 V1 技术架构
|
||||
|
||||
> 生命周期:权威、活跃。产品范围以 [`../product/PRD.md`](../product/PRD.md) 为准,架构决策状态以 [`ADR.md`](ADR.md) 为准。本文只描述当前技术结构和模块边界,不记录任务流水账或操作步骤。
|
||||
|
||||
## 1. 建设目标
|
||||
|
||||
平台提供两条相互关联的知识链路:
|
||||
|
||||
```text
|
||||
Source → Parse/OCR/Clean → Chunk → Index → Retrieval → REST/MCP
|
||||
|
||||
Source/L1 → Knowledge Compilation → Review/Publish → Compiled Retrieval
|
||||
```
|
||||
|
||||
V1 最终目标包括多格式入库、Hybrid Retrieval、Reranker、ACL、来源追踪、Knowledge Compilation、REST、MCP 和管理控制台。当前已从公共兼容域演进到可信 REST 身份与 Knowledge Base/ACL 运行边界;Knowledge Compilation 等差距见 [`../plans/TASKS.md`](../plans/TASKS.md)。
|
||||
|
||||
## 2. 架构不变量
|
||||
|
||||
1. API First:管理端、MCP 和其他系统复用同一业务服务。
|
||||
2. V1 采用模块化单体,不拆分无必要的微服务。
|
||||
3. OceanBase MySQL 关系表与 BOS 保存规范数据;OceanBase 向量表仅为可重建索引。
|
||||
4. Embedding、Reranker、VectorStore、Parser、OCR、ObjectStorage 和 LLM 均通过 Provider 接口隔离。
|
||||
5. 原始 Source 不可变;Parsed、Chunk、Index 和 Compiled Knowledge 均为可追溯派生数据。
|
||||
6. ACL 必须在检索前或检索过程中解析并下推,禁止先做全局 Top-K 再过滤。
|
||||
7. MCP 只暴露业务语义,不暴露数据库、存储、模型或 Parser 实现。
|
||||
8. Index Profile 必须版本化;模型或 Chunk 变更先构建 Shadow Index,验证后再激活。
|
||||
|
||||
这些约束由 [`ADR.md`](ADR.md) 和根目录 [`../../AGENTS.md`](../../AGENTS.md) 共同约束。
|
||||
|
||||
## 3. 总体拓扑
|
||||
|
||||
```text
|
||||
┌──────────────────┐
|
||||
│ Admin Console │
|
||||
└────────┬─────────┘
|
||||
│ REST
|
||||
▼
|
||||
WorkBuddy ─ MCP ─▶ MCP Gateway ─▶ Knowledge Application Services ◀─ REST ─ Other Systems
|
||||
│ │
|
||||
┌────────────┘ └────────────┐
|
||||
▼ ▼
|
||||
OceanBase MySQL + Vector BOS
|
||||
规范关系表 / 可重建索引 Source / 派生产物
|
||||
▲
|
||||
│
|
||||
Celery / RabbitMQ
|
||||
│
|
||||
Async Worker
|
||||
│
|
||||
Parser / Embedding / Index
|
||||
```
|
||||
|
||||
逻辑进程边界:
|
||||
|
||||
- API/Admin:FastAPI 提供 REST、静态管理端、健康检查与指标。
|
||||
- MCP Gateway:独立进程,负责协议、认证与业务 Tool 映射。
|
||||
- Async Worker:独立 Celery Worker,执行持久化 Ingestion 与 Compilation Job。
|
||||
- Recovery Scheduler:单实例调度过期租约和待重投递 Job。
|
||||
- Provider:BOS、OceanBase Vector、千帆等外部能力的 Adapter。
|
||||
|
||||
业务核心仍属于一个模块化单体,各进程复用同一 Domain 和 Application Service。
|
||||
|
||||
## 4. 代码模块
|
||||
|
||||
```text
|
||||
apps/
|
||||
├── api/ FastAPI、REST、Provider 装配、Celery 入口
|
||||
├── mcp-server/ MCP Gateway、认证、Tool 映射
|
||||
└── admin-web/ 同源静态管理端
|
||||
|
||||
packages/
|
||||
├── domain/ SQLAlchemy 领域模型与值对象
|
||||
├── core/ 应用服务和业务编排
|
||||
└── providers/ ObjectStorage、Parser、Embedding、Reranker、VectorStore Adapter
|
||||
|
||||
migrations/ OceanBase MySQL 规范关系表 Migration
|
||||
deploy/ 容器、Helm 与集群交付物
|
||||
tests/ 单元、集成和 RAG 回归
|
||||
```
|
||||
|
||||
`workers/`、`packages/shared/` 和 `packages/retrieval/` 当前是未来模块占位目录;现有 Worker 实现在 `apps/api` 中,不应把占位目录误认为独立服务。
|
||||
|
||||
`ekp_domain.base` 是全部规范模型唯一的 SQLAlchemy Metadata 根,统一约束命名和 OceanBase/SQLite
|
||||
可移植字符串 Enum。领域包只依赖 Python/SQLAlchemy 和自身模块,不得反向依赖 `apps`、`ekp_core`
|
||||
或 Provider;该边界由结构测试保护。新增领域模型及表结构仍分别进入对应模型与 Migration Task。
|
||||
|
||||
规范 Schema 使用单 Base、单 Head 的线性 Alembic 历史向前演进。已发布 Revision 由 SHA-256 清单
|
||||
显式锁定;新增表或字段必须追加 Revision,不得重写既有历史。离线门禁使用 MySQL Dialect 验证
|
||||
关系表 DDL,不把独立管理的 OceanBase Vector 表纳入 Alembic 所有权。
|
||||
|
||||
## 5. 数据所有权
|
||||
|
||||
| 存储 | 保存内容 | 是否事实源 |
|
||||
|---|---|---|
|
||||
| OceanBase MySQL 关系表 | Organization、User、Group、Role、成员/角色关系、KnowledgeBase、Folder、ACL、Document、DocumentVersion、Chunk、Job、Attempt、Batch、AuditLog、Index Profile、ChunkIndexState | 是 |
|
||||
| BOS | 不可变原始文件、解析 JSON 和其他大对象产物 | Source 是;派生产物可重建 |
|
||||
| OceanBase Vector 表 | Dense Vector 与检索过滤 Metadata | 否,可由规范 Chunk 重建 |
|
||||
| RabbitMQ | Job 投递提示 | 否,Job 状态以数据库为准 |
|
||||
|
||||
物理共库不改变所有权边界。业务服务不得从向量行反向恢复规范业务状态,也不得绕过 `VectorStore` 直接依赖 `pyobvector`。
|
||||
|
||||
## 6. 文档入库
|
||||
|
||||
### 6.1 接收边界
|
||||
|
||||
API 完成以下操作后返回 `202 Accepted`:
|
||||
|
||||
```text
|
||||
验证输入并流式计算 Source Hash
|
||||
→ 保存不可变 Source
|
||||
→ 持久化 DocumentVersion/Job
|
||||
→ 投递 Job
|
||||
```
|
||||
|
||||
当前支持 PDF、DOCX、PPTX、XLSX、CSV、HTML/HTM、TXT 和 Markdown;显式启用 MinerU 后额外
|
||||
支持 JPG/JPEG/PNG 与扫描件 OCR。文件大小、批量数量和批量总大小分别由运行时配置限制。批次
|
||||
只负责提交和聚合状态,每个文件拥有独立 Document 与 Job,不构成跨文件事务。浏览器文件夹上传
|
||||
额外携带与文件严格同序的安全相对路径;服务端在已授权 Knowledge Base/基准 Folder 内逐段创建或
|
||||
复用规范 Folder,只持久化校验通过的路径。路径逃逸只拒绝对应文件,跨 Knowledge Base 基准 Folder
|
||||
则在任何批次、目录和对象写入前失败关闭;空目录不创建占位资源。
|
||||
ZIP 导入在该 Batch 边界前增加完整结构与资源预算预检;包结构不安全时不产生任何规范写入,安全
|
||||
条目以延迟流分别进入既有上传服务,ZIP 自身与平台元数据不成为 Source。不同 Folder 的同名文件
|
||||
使用独立 Document 身份,同一 Folder 内继续使用不可变 DocumentVersion 演进。
|
||||
|
||||
`Document` 是稳定逻辑身份,`DocumentVersion` 是不可变、带序号的 Source 身份;`documents`
|
||||
只保留 Active Version 投影以兼容列表与详情 API。正式入库在同一 Knowledge Base/Folder 内以文件名归并
|
||||
逻辑 Document,并以 SHA-256 Source Hash 去重:Hash 相同返回已存在提示,同名而 Hash 不同创建 inactive 的下一
|
||||
版本。数据库生成列唯一索引保证每个 Document 最多一个 Active Version。Chunk、Parsed Artifact、
|
||||
Job 和向量行均绑定具体 Version;只有索引完整成功后才切换关系与向量 Active Marker,失败候选不
|
||||
影响旧 Active Version。不同 Knowledge Base 使用独立唯一范围;旧未分配数据保留在隔离的
|
||||
`public/public` 兼容范围,不自动迁入任意 KB。
|
||||
|
||||
同名候选的 `202` 响应描述新接收的 Version,同时保留稳定 Document ID;旧 Active Document 投影
|
||||
在新索引成功前不变。多 API 副本首次并发创建同名 Document 时,数据库唯一约束选出胜者,失败方
|
||||
只在确认同名胜者存在后有界重试 Version Metadata,Source Hash 相同则仍收敛为重复 Source。
|
||||
|
||||
### 6.2 Worker 流水线
|
||||
|
||||
```text
|
||||
Parsing → Cleaning → Chunking → Embedding → Indexing
|
||||
```
|
||||
|
||||
- 默认 PDF 使用 PyMuPDF,DOCX 使用 python-docx,TXT/Markdown 使用轻量 Parser。显式选择
|
||||
MinerU Provider 后,PDF、DOCX 和 JPG/JPEG/PNG 进入自托管 `/file_parse` 的 `auto` 路由,
|
||||
TXT/Markdown 保持轻量实现;不存在静默 Provider 回退。
|
||||
- Docling 本地 Adapter 负责 PPTX、XLSX、CSV 和 HTML/HTM;它保留 reading order、章节、Page、
|
||||
Sheet、Table Cell 坐标和 Image Reference。DOCX 继续由轻量 Parser 或显式 MinerU 处理,避免
|
||||
同一格式的隐式 Provider 切换。
|
||||
- `Parser` 是可运行时检查的结构协议;`ParserRouter` 持有不可变扩展名能力集合,上传准入直接
|
||||
使用该集合。Router 同时校验 Provider 输出的 Document ID 与文件名,拒绝错误 Source 绑定。
|
||||
- Parsed Artifact 使用 schema v4 版本化 JSON;每个结构节点显式绑定 Source Document 和数字
|
||||
Version,并保留 Page、Section、类型、内容、页级 OCR Confidence 和低置信度警告;旧 schema
|
||||
v1~v3 仍可读取且不伪造缺失结构。
|
||||
- Cleaning 处理 Unicode/控制字符、空白和异常换行、Latin 连字符断词、标点噪声与精确重复块;
|
||||
MinerU 辅助页眉页脚在映射阶段移除,Original Source 不被修改。
|
||||
- `structure_v2` 保留 Title/Heading/Section 硬边界和约 450 Token、75 Token Overlap;Table 不与
|
||||
叙述文本或其他 Sheet 混合,窗口重复列标题,并把一基闭区间 Sheet/Row/Column 来源写入规范
|
||||
Chunk。`structure_v1` 只用于既有 Active Profile 的兼容读取与安全切换。
|
||||
- 文档只有在全部 Chunk 完成索引后才进入 `ready`。
|
||||
- 流水线失败不会删除不可变原始文件。
|
||||
|
||||
### 6.3 Job、重试与恢复
|
||||
|
||||
OceanBase 中的 `IngestionJob`/`IngestionAttempt` 与 `CompilationJob`/`CompilationAttempt` 分别是
|
||||
入库和编纂执行事实源。Broker 重投递、Worker 重启和租约恢复必须收敛到同一状态。每次 Worker
|
||||
Claim 都写入新的 Lease Fencing Token;只有仍持有当前 Token 且 Attempt Number 匹配的执行者可以
|
||||
结束 Job,过期 Worker 恢复运行时只能安全退出。Compilation Job 仅持久化规范化意图和有序 Chunk
|
||||
标识;执行时重新解析提交者的 KB Write、全部来源 Document Read、Active Version 与数据策略。
|
||||
生成 Draft 通过唯一 `generation_job_id` 与 Job 一对一关联,从数据库层收敛重复投递。
|
||||
|
||||
瞬时错误执行有界重试。下一次 Attempt 开始前按以下顺序清理派生数据:
|
||||
|
||||
```text
|
||||
Vector/Sparse rows
|
||||
→ ChunkIndexState
|
||||
→ Chunk
|
||||
→ Parsed Artifact / parsed_storage_key
|
||||
→ 重置 Document 状态
|
||||
```
|
||||
|
||||
清理只作用于 Job 绑定的 DocumentVersion,保留全部不可变 Source、Document、历史 Version、Job
|
||||
和 Attempt。清理失败时必须终止,禁止在未知脏状态继续执行。详细运行方式见
|
||||
[`../development/ASYNC_INGESTION_SETUP.md`](../development/ASYNC_INGESTION_SETUP.md)。
|
||||
|
||||
AuditLog 规范模型保存 User/Service/System/Anonymous Actor、REST/MCP/Worker/Scheduler/System
|
||||
来源、Action、Outcome、稳定资源引用和脱敏详情。资源引用不设删除级联,以保留历史证据;
|
||||
Compiled 生命周期及 Compilation Job 接收、成功和失败已接入业务写点,完整查询 API 与
|
||||
Append-only 数据库权限仍属于后续治理范围。
|
||||
|
||||
## 7. 索引与检索
|
||||
|
||||
### 7.1 当前实现
|
||||
|
||||
```text
|
||||
Query
|
||||
→ Normalize
|
||||
→ Active Profile + Active DocumentVersion 预过滤范围内 Dense Top20 + Sparse Top20
|
||||
→ Weighted Reciprocal Rank Fusion Top20
|
||||
→ Reranker
|
||||
→ TopK
|
||||
→ 从规范 Chunk 补全来源
|
||||
```
|
||||
|
||||
默认基线:
|
||||
|
||||
| 能力 | 当前实现 |
|
||||
|---|---|
|
||||
| Embedding | 千帆 `bge-large-zh`,1024 维 |
|
||||
| VectorStore | OceanBase Vector |
|
||||
| SparseStore | OceanBase `MATCH ... AGAINST` + NGRAM Full-Text Index |
|
||||
| Fusion | Weighted RRF,`k=60`,Dense/Sparse 默认等权,输出 Top20 |
|
||||
| Reranker | 千帆 `bce-reranker-base` |
|
||||
| 来源 | Document/Version、文件名、页码、完整章节路径、Sheet/行列范围、精确 Chunk、有界上下文、Raw 类型、Fusion/Reranker 分数 |
|
||||
|
||||
`index_profile_id + is_active` 必须在 Dense ANN 或 Sparse 排序的 TopN 截断前过滤;候选的 Version
|
||||
非 Active,或索引内容与规范 Chunk 不一致时失败关闭,不能返回历史版本或无法验证的来源。
|
||||
Dense 与 Sparse 复用 `knowledge_chunks_v1` 可重建行及其激活、删除和 Shadow Profile 生命周期;
|
||||
关系 Chunk 仍是规范数据。测试 Fake 使用确定性 BM25,OceanBase 使用原生 Full-Text 相关度,
|
||||
两者分数只在各自候选列表内排序,不作为跨 Provider 的绝对值契约。Fusion 使用名次而不是原始
|
||||
分数,并按 Chunk 去重;跨路来源标识或正文冲突时失败关闭。
|
||||
|
||||
Provider-neutral VectorStore 对 Row、Query、Profile/Version 范围和运行时实现形状执行统一校验;
|
||||
确定性 Fake 在 Profile + Active Version 预过滤后执行稳定精确 Cosine,并拒绝维度/模型漂移、
|
||||
Source 身份改绑和同 Document/Profile 的多个 Active Version。厂商 Adapter 不得降低这些边界。
|
||||
Dense 应用服务还会在任何 Provider 调用前校验 Query/Top-N,空授权范围不发送 Query,并要求
|
||||
Query Instruction 与 Active Profile 一致;返回后再次验证来源身份、唯一 Chunk、Cosine 范围、
|
||||
降序、Profile 和 ACL Scope。
|
||||
OceanBase Adapter 在启动时只读验证最低版本与预建 Dense/Sparse Schema,版本切换先锁定并确认
|
||||
目标行再于同一事务更新 Active Marker;ANN 查询显式要求 `pre_filter`,所有值均使用绑定参数,
|
||||
厂商输出在进入业务服务前校验来源、分数、排序和唯一身份。
|
||||
|
||||
最终 Search Service 固定使用 Active Index Profile 的 Hybrid TopN、Reranker Model 与 Final TopK;
|
||||
单次请求只能缩小、不能扩大 Profile 的最终结果数。选中候选必须在规范关系库中同时找到匹配的
|
||||
Chunk、该 Profile 的 ChunkIndexState 和 Ready Active DocumentVersion,并复验模型、维度、Chunk
|
||||
策略、来源身份、文件名与正文。任一证据缺失或漂移即失败关闭。`context` 只读取同 Version、同
|
||||
策略相邻 Chunk,最多 2400 字符且保留命中正文;它改善可读性,但不参与排序或替代不可变原文。
|
||||
|
||||
离线质量门禁读取哈希绑定的版本化 Golden Dataset,通过与生产相同的 Parser、Chunk、Dense、
|
||||
Sparse、Fusion、Reranker 和规范来源补全边界生成聚合报告;基础设施由 SQLite、Local Object
|
||||
Storage 和确定性 Fake Provider 替代。评测模块不依赖具体生产模型,报告用 Dataset 版本和 Index
|
||||
Profile 标识输入,避免不同语料、Chunk 或模型结果被错误横向比较。
|
||||
|
||||
### 7.2 V1 目标
|
||||
|
||||
完整检索链仍需实现:
|
||||
|
||||
```text
|
||||
Query Normalize
|
||||
→ Permission Resolver
|
||||
→ Dense + BM25/Sparse
|
||||
→ Fusion
|
||||
→ Candidate TopN
|
||||
→ Reranker
|
||||
→ TopK
|
||||
→ Source Enrichment
|
||||
```
|
||||
|
||||
Dense、Sparse、Hybrid Fusion 及 Knowledge Base/Folder/Document ACL Scope 已完成;MCP 与 REST
|
||||
正式身份均接入授权 Search,REST 上传显式接收 KB/Folder Scope,管理入口维护规范身份与 ACL。
|
||||
|
||||
### 7.3 Index Profile
|
||||
|
||||
Index Profile 是完整且不可变的检索配置快照,绑定 Chunk 窗口、Embedding 模型/输入上限/Query
|
||||
Instruction/向量维度与 Float32 存储、Sparse 策略、Dense/Sparse/Hybrid 候选规模、Weighted RRF、
|
||||
Reranker 模型和最终 TopK。运行时从数据库 Active Profile 读取这些值,避免环境变量与已建索引
|
||||
语义漂移;Credential、Endpoint、业务 Query 和正文不进入 Profile。生命周期为:
|
||||
|
||||
```text
|
||||
shadow → active → retired
|
||||
```
|
||||
|
||||
新配置必须创建新 Profile,使用规范 Chunk 构建 Shadow Index,核对全部 `ChunkIndexState` 后原子激活。任一时刻最多一个 Active Profile。操作和回滚边界见 [`../development/INDEX_REBUILD.md`](../development/INDEX_REBUILD.md)。
|
||||
字段和数据库约束见 [`../development/INDEX_PROFILE.md`](../development/INDEX_PROFILE.md)。
|
||||
|
||||
`structure_v2` 切换时,受控 Build 会先从 Active Version 的 Parsed Artifact 幂等生成目标策略
|
||||
Chunk,再构建 Shadow Index;Verify 必须把缺少目标 Chunk 的 Active Version 计为不完整。运行时
|
||||
始终使用数据库登记的 Active Profile,v1 尚未激活到 v2 时不得接收新增 Docling 格式。
|
||||
|
||||
## 8. Provider 边界
|
||||
|
||||
| 接口 | 当前 Provider | 后续能力 |
|
||||
|---|---|---|
|
||||
| `ObjectStorage` | 百度 BOS;本地 Fake/Adapter 用于测试 | 其他对象存储 |
|
||||
| `Parser` | 运行时可检查协议与不可变 Router;PyMuPDF、python-docx、文本/Markdown、本地 Docling Office/Spreadsheet/HTML;可选 MinerU HTTP OCR | Parser Benchmark 与性能加固 |
|
||||
| `EmbeddingProvider` | 运行时可检查协议、统一能力/向量门禁与确定性 Fake;千帆 BGE 的安全 HTTPS/错误分类 Adapter;未激活的 Qwen3 OpenAI-compatible Adapter | 本地模型与 Provider Benchmark |
|
||||
| `RerankerProvider` | 运行时可检查协议、统一请求/结果门禁与确定性 Fake;千帆 BCE 的安全 HTTPS/错误分类 Adapter | Qwen 或本地 Reranker Benchmark |
|
||||
| `VectorStore` | OceanBase Vector | 其他向量数据库 Adapter |
|
||||
| `LLMProvider` | Provider-neutral 协议、OpenAI-compatible Adapter、确定性 Fake;运行时可选 | 后续 Provider 扩展与评测 |
|
||||
|
||||
配置、Credential 和厂商响应只能停留在运行时装配与 Provider Adapter 中,不得进入 Domain、REST 或 MCP 契约。
|
||||
|
||||
## 9. 权限与身份
|
||||
|
||||
V1 权限主体包括 User、Group、Role,资源层级包括 Organization、Knowledge Base、Folder 和 Document。标准链路为:
|
||||
|
||||
```text
|
||||
Identity → Permission Resolver → Allowed Scope → Provider-neutral Retrieval Filter
|
||||
```
|
||||
|
||||
REST、MCP 和管理端必须复用同一权限服务。
|
||||
|
||||
当前已建立 Organization、User、Group、Role 及同 Organization 成员/角色关系的规范模型,并以
|
||||
类型化复合外键保存 Organization/KB/Folder/Document ACL。内部 `PermissionResolver` 已展开 Active
|
||||
User/Group/Role、解析同 KB Folder 父链和 Deny/Allow,形成精确 Document Scope;Dense/Sparse
|
||||
Adapter 在各自 Top-N 前消费同一个 Provider-neutral Filter,`AuthorizedSearchService` 固化授权编排。
|
||||
`EKP-082` 提供 REST User Token/Service API Key 映射、显式首身份 Bootstrap、User/Group/Role 与
|
||||
成员/角色关系、KB/Folder/ACL 管理 API,并在 Document/Batch/Job/Source/Search 前复用 Permission
|
||||
Resolver。`EKP-070` 的 MCP 身份继续使用同一规范授权数据并限制 `mcp_enabled` Knowledge Base;
|
||||
外部 IdP 仍是可替换的 Token Verifier 扩展点,当前运行配置不负责签发 Token。
|
||||
|
||||
KnowledgeBase 已保存同 Organization Owner、状态、可见性意图、数据处理策略、MCP Availability
|
||||
与 Metadata Schema。Folder 已以 Organization/KB 复合范围保存树结构,并拒绝跨 KB 父子关系。
|
||||
Document 提供 Organization/KB/Folder 复合范围,使 ACL 无法引用未分配的旧公共 Document;正式
|
||||
上传必须选择有 Write 权限的 KB,并把同一范围复制到不可变 DocumentVersion。未分配范围不是授权
|
||||
默认值,旧公共数据不会自动归入任意 Organization,也不会创建占位授权数据。
|
||||
|
||||
## 10. 对外接口
|
||||
|
||||
### REST
|
||||
|
||||
公开 API 统一位于 `/api/v1`。当前提供 Document 上传/查询/删除、规范 Chunk 正文读取、不可变
|
||||
Source 访问、Batch、Job 和 Search 接口;错误响应不得回显 Provider、数据库、存储或 Credential。
|
||||
完整行为以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
|
||||
|
||||
### MCP
|
||||
|
||||
独立 Streamable HTTP Gateway 已注册 `search_knowledge`、`get_document`、
|
||||
`get_document_section`、`get_source` 与 `list_knowledge_bases` 五个 P0 Tool,复用与 REST 相同的
|
||||
Search、Permission 和 Knowledge Access 应用服务。正式运行时 Bearer Token 支持 User Token/
|
||||
Service API Key,并绑定规范 Organization/ACL User;旧静态凭据只保留公共 Search Demo 兼容。
|
||||
`get_source` 返回精确规范 Chunk 证据而非原文件字节,原文件仍由受控 REST Source 接口流式读取。
|
||||
外部 IdP 和远程现场验收尚未完成。完整契约见 [`../api/MCP_CONTRACT.md`](../api/MCP_CONTRACT.md)。
|
||||
|
||||
### 管理控制台
|
||||
|
||||
当前管理端是同源静态客户端,提供运营总览、批量上传、文档列表、Document/Chunk Inspector、
|
||||
Search Lab、Knowledge Base/Folder 及 User/Group/Role/ACL 管理。运营总览复用公开 Document 列表
|
||||
按状态投影知识流库存和最近来源,不保存独立聚合状态;
|
||||
Inspector 以不可变 Source 预览和规范 Chunk 溯源组成双视图:PDF 保持原版式并从命中起始页打开,
|
||||
Markdown/纯文本连续阅读,图片在线预览,Office/HTML 安全下载;跨页命中同时显示起止页和打开页。
|
||||
Search Lab 区分可读上下文与精确命中 Chunk,并通过一次实际 Search 执行展示 Query、Scope/ACL、
|
||||
Active Profile、Dense、Sparse、Fusion、Reranker、Final 与各阶段耗时。它不承载或复制检索业务
|
||||
逻辑,所有操作均调用 `/api/v1`;诊断响应不含 Endpoint、Credential 或厂商原始数据。
|
||||
页级质量来自 Version 绑定的 Parsed Artifact,Job/Attempt 来自规范关系表;质量派生数据不可读
|
||||
不会隐藏规范 Chunk。管理端 Token 只保存于当前标签页,并以认证 Fetch/Blob 打开不可变 Source。
|
||||
Compiled Knowledge 页面允许从可读规范 Chunk 创建/生成草稿、编辑和回到 Document Inspector 核对
|
||||
来源;EKP-091 已接入来源当前性、冲突证据、审核、退回、发布与 Stale 修订。EKP-092 已把仍可
|
||||
核验的 Published 内容接入正式 REST/MCP Search,并在六阶段 Search Lab 中保留全部底层来源。
|
||||
Benchmark 作为独立非开发队列记录。
|
||||
|
||||
## 11. Knowledge Compilation
|
||||
|
||||
长期知识模型:
|
||||
|
||||
```text
|
||||
L0 Source:不可变原始事实
|
||||
L1 Structured:Parsed / Chunk / Index
|
||||
L2 Compiled:Wiki / FAQ / SOP / Summary / Comparison
|
||||
```
|
||||
|
||||
EKP-090 已建立 Organization/Knowledge Base 范围的 `CompiledKnowledge` 和有序
|
||||
`CompiledKnowledgeSource`,每个草稿至少绑定一个确切 Document、DocumentVersion 与 Chunk。读路径
|
||||
先解析 KB 和全部来源 Document 的权限交集;写路径要求目标 KB Write 加来源 Read。手工草稿与可选
|
||||
LLMProvider 生成均只进入 Draft,AI 结果不会覆盖既有内容。`local_only` KB 不得调用声明为 cloud 的
|
||||
Provider;OpenAI-compatible Adapter 仅接受安全 Endpoint、Bearer Secret 与严格 JSON title/content,
|
||||
厂商字段不进入 Domain/API。
|
||||
|
||||
EKP-091 在同一规范事务内把 Source 新版本激活、被引用 Document 删除及 ACL 增删传播为稳定的
|
||||
Stale 状态;被引用 Document 使用 tombstone 逻辑删除,退出检索和普通 Document API,但保留不可变
|
||||
Source/Version/Chunk 证据。审核人把冲突主张绑定到两个确切 Source,未解决冲突阻断发布;AI 不会
|
||||
自动折中或发布。Compiled 读取继续动态执行 KB 与全部来源 Document 的权限交集。EKP-092 只从
|
||||
检索前已授权的 Raw Source 候选扩展 Published L2,按全部来源权限交集过滤后与 Raw 共同精排;
|
||||
最终结果再次核验规范 Source,并返回精确 `underlying_sources[]`。当前不建立未版本化的独立 L2
|
||||
向量索引,Compiled Recall 继承底层 Raw Source 召回。
|
||||
|
||||
## 12. 部署与可观测性
|
||||
|
||||
生产逻辑角色:API/Admin、MCP Gateway、Async Worker、Recovery Scheduler,以及可选 Edge Proxy。外部依赖包括 OceanBase、BOS、RabbitMQ 和 AI Provider。
|
||||
|
||||
当前开发交付已完成 API/Admin、Worker 和检索公网烟测;远程 MCP/WorkBuddy 现场验收进入非开发
|
||||
独立队列。这不删除 MCP Gateway 进程边界,也不改变 ADR-002。实际 GitOps Values、Secret、
|
||||
Namespace、工作负载副本、滚动/回滚、监控和备份由运维人员管控,开发仓库继续提供对应 Helm
|
||||
能力、运行契约和验收工具。
|
||||
|
||||
Helm 基线要求:
|
||||
|
||||
- API/MCP 默认多副本,Worker 可独立扩缩;
|
||||
- 串行 Migration Job;
|
||||
- 生产 target 只引用预建 Secret,不在 Values 或镜像中保存凭据;当前中央 dev 流水线的默认
|
||||
`dev` target 是经明确授权、限制在 dev Registry/环境的例外;
|
||||
- Readiness/Liveness、资源限制、非 root、只读根文件系统和 PDB;
|
||||
- Ingress/TLS,或在无 Ingress 场景使用仅负责路径代理的 Edge;
|
||||
- 字段白名单 JSON 日志、请求关联 ID 和低基数 `/metrics`。
|
||||
|
||||
部署命令和运维步骤分别见 [`../development/KUBERNETES_DEPLOYMENT.md`](../development/KUBERNETES_DEPLOYMENT.md) 与 [`../development/CLUSTER_OPERATIONS.md`](../development/CLUSTER_OPERATIONS.md)。当前远程验收状态见 [`../development/DEPLOY_003_ACCEPTANCE.md`](../development/DEPLOY_003_ACCEPTANCE.md)。
|
||||
|
||||
## 13. 验收边界
|
||||
|
||||
完整 V1 只有同时满足以下条件才可完成:
|
||||
|
||||
- P0 文件完成 Upload → Parse/OCR → Clean → Chunk → Embed → Dense/BM25 Index;
|
||||
- Hybrid + Reranker + ACL 检索通过质量与性能门禁;
|
||||
- 每条结果可追踪 Source、Version、Page、Section 和 Chunk;
|
||||
- REST、MCP、管理端和 Source 下载执行同一授权边界;
|
||||
- Knowledge Compilation 完成 Source Link、冲突/Stale、审核和发布;
|
||||
- Golden Dataset、Provider Benchmark、安全、可观测性、备份和重建通过验收。
|
||||
|
||||
当前 Demo/Deploy 纵向切片不能自动完成对应 `EKP-*` 任务。完成标准以 [`../development/DEFINITION_OF_DONE.md`](../development/DEFINITION_OF_DONE.md) 和 [`../plans/TASKS.md`](../plans/TASKS.md) 为准。
|
||||
165
hy4/_kp/docs/demo/DEMO_SCOPE.md
Normal file
165
hy4/_kp/docs/demo/DEMO_SCOPE.md
Normal file
@ -0,0 +1,165 @@
|
||||
# EKP 知识检索 Demo v0.1
|
||||
|
||||
> 生命周期:历史基线、冻结。本文只定义 Demo 验收边界,不再同步当前部署状态。当前任务和实现状态分别以 [`../plans/TASKS.md`](../plans/TASKS.md) 与 [`../plans/PROGRESS.md`](../plans/PROGRESS.md) 为准。
|
||||
|
||||
## 1. 目标
|
||||
|
||||
Demo 用于验证最小知识入库与检索闭环:
|
||||
|
||||
```text
|
||||
文件上传
|
||||
→ 文档解析与基础清洗
|
||||
→ 结构感知 Chunk
|
||||
→ Embedding
|
||||
→ OceanBase Vector
|
||||
→ Dense Top20
|
||||
→ Reranker Top5
|
||||
→ 来源结果
|
||||
→ 简单管理页面
|
||||
```
|
||||
|
||||
成功标准是核心链路可用并具备演进基础,不是完整企业级知识平台完成。
|
||||
|
||||
## 2. 架构
|
||||
|
||||
```text
|
||||
Admin Web ─ REST ─ FastAPI
|
||||
├─ Upload → Parser → Cleaner → Chunker → Embedding → VectorStore
|
||||
└─ Search → Query Embedding → Dense TopN → Reranker → Source
|
||||
|
||||
OceanBase MySQL:规范 Document、Chunk、Job 与索引状态
|
||||
BOS / Local:不可变原始文件和解析产物
|
||||
OceanBase Vector:可重建 Dense 检索索引
|
||||
```
|
||||
|
||||
Provider 迁移后,目标运行时为 OceanBase MySQL + BOS + OceanBase Vector + 千帆模型。业务层必须通过 Provider 接口访问基础设施。
|
||||
|
||||
## 3. P0 范围
|
||||
|
||||
### 3.1 文件与解析
|
||||
|
||||
支持:
|
||||
|
||||
- 文本型 PDF:提取正文并尽量保留页码;
|
||||
- DOCX:提取标题、段落和基础章节结构;
|
||||
- UTF-8 TXT/Markdown:直接读取并保留 Markdown 标题层级;
|
||||
- 单文件与有界批量上传;
|
||||
- 原始文件不可覆盖。
|
||||
|
||||
暂不支持 PPTX、XLSX、CSV、HTML、图片、扫描 PDF、OCR 和复杂版面恢复。
|
||||
|
||||
### 3.2 清洗与 Chunk
|
||||
|
||||
基础清洗处理多余空行、连续空格、异常换行和首尾空白,不修改原始 Source。
|
||||
|
||||
Chunk 优先按标题、章节和段落组合,再按 Token 限制切分;默认约 400~500 Token,并保留 50~100 Token Overlap。每个 Chunk 至少包含:
|
||||
|
||||
```text
|
||||
chunk_id
|
||||
document_id
|
||||
filename
|
||||
page / page_end
|
||||
section / section_path
|
||||
chunk_index
|
||||
content
|
||||
strategy_version
|
||||
```
|
||||
|
||||
### 3.3 Embedding、索引与检索
|
||||
|
||||
- `EmbeddingProvider` 当前使用千帆 `bge-large-zh`;
|
||||
- `VectorStore` 当前使用 OceanBase Vector;
|
||||
- Dense Retrieval 默认召回 Top20;
|
||||
- `RerankerProvider` 当前使用千帆 `bce-reranker-base`;
|
||||
- 最终返回 TopK,并保留 Dense 与 Reranker 分数;
|
||||
- 来源必须从规范关系表中的 Chunk 补全,陈旧或不一致索引失败关闭。
|
||||
|
||||
结果至少包含:
|
||||
|
||||
```text
|
||||
document_id
|
||||
filename
|
||||
page
|
||||
section
|
||||
chunk_id
|
||||
content
|
||||
score
|
||||
rerank_score
|
||||
```
|
||||
|
||||
### 3.4 异步 Job 与批量接收
|
||||
|
||||
Source、Document 和 Job 持久化后,API 返回 `202`,不等待解析和索引。Celery/RabbitMQ Worker 执行流水线,OceanBase Job/Attempt 是执行事实源。
|
||||
|
||||
瞬时失败执行有界重试;重试前清理 Parsed Artifact、Chunk、ChunkIndexState 和 Vector Row,同时保留原始 Source、Document、Job 与 Attempt 历史。批次按文件隔离,单个文件失败不得回滚其他文件。
|
||||
|
||||
## 4. REST API
|
||||
|
||||
当前 Demo 接口统一位于 `/api/v1`:
|
||||
|
||||
| 方法 | 路径 | 行为 |
|
||||
|---|---|---|
|
||||
| `POST` | `/documents` | 单文件异步接收 |
|
||||
| `POST` | `/ingestion-batches` | 多文件异步接收 |
|
||||
| `GET` | `/ingestion-batches/{batch_id}` | 批次与逐文件状态 |
|
||||
| `GET` | `/jobs/{job_id}` | Job、Stage、Attempt 和安全错误信息 |
|
||||
| `GET` | `/documents` | 文档列表 |
|
||||
| `GET` | `/documents/{document_id}` | 文档详情 |
|
||||
| `DELETE` | `/documents/{document_id}` | Demo 直接删除 |
|
||||
| `POST` | `/search` | Dense → Reranker → 来源结果 |
|
||||
|
||||
公开契约以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
|
||||
|
||||
## 5. 管理控制台与 MCP
|
||||
|
||||
管理控制台只提供:
|
||||
|
||||
1. 批量上传与处理状态;
|
||||
2. 文档列表、详情和删除;
|
||||
3. Search Lab,展示来源、Chunk、Dense 分数和 Reranker 分数。
|
||||
|
||||
MCP Gateway 只暴露 `search_knowledge`,复用 REST 相同的 `SearchService`,并使用最小服务身份。该能力不代表完整 MCP P0 Tool 或用户委托 ACL 已完成。MCP 契约见 [`../api/MCP_CONTRACT.md`](../api/MCP_CONTRACT.md)。
|
||||
|
||||
## 6. 明确后置
|
||||
|
||||
Demo 不完成以下 V1 能力:
|
||||
|
||||
- 多租户、User/Group/Role、Knowledge Base、Folder 和 Document ACL;
|
||||
- LDAP、OIDC、SSO、审计日志和 Source 下载授权;
|
||||
- DocumentVersion、Source Hash 去重、软删除;
|
||||
- BM25、Hybrid Fusion 和查询路由;
|
||||
- OCR、MinerU、Docling 和复杂表格/版面;
|
||||
- Knowledge Compilation、审核、冲突和 Stale;
|
||||
- 多 Embedding/Reranker Benchmark 平台;
|
||||
- MCP 其余 P0 Tool;
|
||||
- 完整 Observability、跨区域灾备和企业级生产加固。
|
||||
|
||||
公共知识域不等于匿名公网。开始接收任何非公共知识前,必须先完成 Permission Resolver 和检索前/检索中 ACL 下推。
|
||||
|
||||
## 7. Task 与完成状态
|
||||
|
||||
Demo 纵向切片包括:
|
||||
|
||||
```text
|
||||
DEMO-001~010
|
||||
DEMO-012
|
||||
DEMO-014~016
|
||||
ARCH-001~006
|
||||
PLAN-001~002
|
||||
DEPLOY-001~003
|
||||
```
|
||||
|
||||
截至本基线冻结时,Demo、Provider 迁移、MCP、异步批量入库和 DEPLOY-001~002 已完成;DEPLOY-003 仍进行中。最新状态只在 [`../plans/TASKS.md`](../plans/TASKS.md) 更新。
|
||||
|
||||
## 8. Demo 完成定义
|
||||
|
||||
- PDF、DOCX、TXT、Markdown 可以成功接收;
|
||||
- Upload → Parse → Clean → Chunk → Embed → Index 全链路完成;
|
||||
- Query → Dense Top20 → Reranker Top5 完成;
|
||||
- 每条结果包含文件名、页码、章节、Chunk 内容和两阶段分数;
|
||||
- 批量接收具备逐文件隔离、持久化状态和失败恢复;
|
||||
- 非研发用户可以通过管理端完成上传、等待处理、检索和查看来源;
|
||||
- 默认自动化测试不依赖真实云服务、互联网或 GPU;
|
||||
- 检索变更运行 Golden Dataset 回归。
|
||||
|
||||
完整任务完成标准以 [`../development/DEFINITION_OF_DONE.md`](../development/DEFINITION_OF_DONE.md) 为准。
|
||||
207
hy4/_kp/docs/development/ADMIN_CONSOLE.md
Normal file
207
hy4/_kp/docs/development/ADMIN_CONSOLE.md
Normal file
@ -0,0 +1,207 @@
|
||||
# Admin Console 使用说明
|
||||
|
||||
管理端在 FastAPI 服务中提供一个轻量、同源的管理客户端。页面只负责展示和交互,知识总览、上传、
|
||||
文档管理与检索仍全部通过 `/api/v1` 公开业务接口完成,不复制服务端业务逻辑。
|
||||
|
||||
## 启动与访问
|
||||
|
||||
完成运行时配置后启动 Worker 与 API:
|
||||
|
||||
```bash
|
||||
make run-worker
|
||||
make run
|
||||
```
|
||||
|
||||
页面地址:
|
||||
|
||||
| 页面 | 地址 | 对应业务接口 |
|
||||
|---|---|---|
|
||||
| 知识总览 | `/admin`、`/admin/overview` | `GET /api/v1/documents` |
|
||||
| 文档导入 | `/admin/upload` | `POST/GET /api/v1/ingestion-batches*` |
|
||||
| 文档管理 | `/admin/documents` | `GET/DELETE /api/v1/documents*` |
|
||||
| 知识检索 | `/admin/search` | `POST /api/v1/search/diagnostics` |
|
||||
| 知识库管理 | `/admin/knowledge-bases` | `GET/POST /api/v1/administration/knowledge-bases*` |
|
||||
| 权限管理 | `/admin/permissions` | `GET/POST/PUT/DELETE /api/v1/administration/*` |
|
||||
|
||||
`/admin` 默认展示知识总览。OpenAPI 文档仍位于 `/api/v1/docs`。
|
||||
|
||||
## 品牌视觉基线
|
||||
|
||||
管理台延续地瓜机器人公开官网的设计语言,使用品牌橙 `#FF3C00`、黑白高对比、中文
|
||||
无衬线标题、紧凑数据字体、低圆角容器和宽松留白。贯穿页面标题的“计算信号轨”是唯一的
|
||||
强调性图形。`EKP-080` 用 Source → Processing → Ready → Retrieval 运行轨道编码真实知识处理顺序,
|
||||
一级导航按“知识运营”和“系统管理”分组,不把并列功能伪装成固定步骤。
|
||||
|
||||
品牌图形使用仓库内的代码原生 SVG/CSS 绘制,不复制或热链官网图片、字体和脚本;页面
|
||||
运行时不依赖品牌官网。前端设计方法参考 GitHub `anthropics/skills` 的 Apache-2.0
|
||||
`frontend-design` Skill,并以当前产品信息架构、可访问性和业务正确性为优先约束。
|
||||
|
||||
## 核心操作闭环
|
||||
|
||||
1. 从知识总览查看全部、处理中、可检索文档和就绪率;处理失败时直接打开最近文档的证据链,
|
||||
或进入文档管理页;
|
||||
2. 首次进入时输入运行环境分配的 REST Token;Token 只保存在当前标签页 `sessionStorage`,经
|
||||
`/administration/session` 验证后,页面显示规范 User 和 Organization,并按返回的 Scope 隐藏
|
||||
上传、删除、知识库和权限管理入口;
|
||||
3. 在知识库管理页展开“知识库根目录”和有子项的 Folder,沿缩进与分支轨道查看真实父子关系;
|
||||
目录树保留知识库可见性、数据处理策略、MCP 状态与文件夹计数。“新建知识库”或知识库卡片中的
|
||||
“新建文件夹”继续打开独立弹窗;在权限管理页通过对应操作按钮打开用户、用户组、角色、成员
|
||||
关系、角色关系和授权规则弹窗。页面主体只用于查看资源和关系,不与新增表单混排;
|
||||
4. 在上传页选择目标 Knowledge Base/Folder,再使用独立的“选择文件”或“选择文件夹”入口提交;
|
||||
文件夹模式保留所选根目录及其子目录层级,每个文件仍独立进入 Job。支持 PDF、DOCX、PPTX、
|
||||
XLSX、CSV、HTML/HTM、TXT 或 Markdown;服务端显式启用 `PARSER_PROVIDER=mineru` 后还可选择
|
||||
JPG/JPEG/PNG,扫描 PDF 由 MinerU `auto` 判定;
|
||||
5. 接口返回 `202` 后,页面自动轮询批次,逐文件显示接收状态、处理阶段、尝试次数和安全错误;
|
||||
6. 批次进入 `Completed`、`Partial failure` 或 `Failed` 后停止轮询;在文档管理页可先按可读
|
||||
Knowledge Base 联动选择全部文件夹、知识库根目录或某个 Folder 子树,核对当前范围与结果数量,
|
||||
再点击“查看文档”。Inspector 默认展示安全的原始文件预览,也可切换到按来源顺序排列的已索引
|
||||
解析 Chunk;
|
||||
7. 在知识检索页输入问题并查看 Query、Scope/ACL 实际状态、Active Profile、Dense、Sparse、
|
||||
Fusion、Reranker、Final 候选和阶段耗时;最终 Top-K 继续展示文件名、页码、章节、Sheet/行列
|
||||
范围、相邻 Chunk 上下文和精确命中;“定位命中来源”进入双视图 Inspector,
|
||||
“查看或下载原始文件”在新窗口读取可预览的不可变 Source,不支持浏览器预览的二进制格式保留
|
||||
原始文件名下载;
|
||||
8. 如需删除文档,在文档管理页确认后直接删除。
|
||||
|
||||
## 范围与安全边界
|
||||
|
||||
- 页面静态文件位于 `apps/admin-web`,由同一个 FastAPI 进程提供,避免额外前端运行时;
|
||||
- 页面保持零构建、无第三方 npm 依赖;Node/npm 只用于开发期 JavaScript 模块语法、锁文件和本地
|
||||
版本化资源引用检查,不进入 API 或生产浏览器运行时;
|
||||
- API 返回内容仅通过 DOM `textContent` 渲染,不作为 HTML 注入;
|
||||
- 所有业务请求发送 Bearer Header;浏览器不持久保存 Token,原文预览/打开使用认证 Fetch 与短生命
|
||||
周期 Blob URL,不把 Token 放入 URL;关闭标签页即清除身份;
|
||||
- 低权限用户的上传/管理导航和文档删除按钮使用 Session Scope 真实收敛;CSS 对语义 `hidden`
|
||||
使用强制隐藏规则,静态资源带任务版本标识,避免旧缓存重新暴露已隐藏入口;
|
||||
- 文档范围筛选只使用 `/documents` 的 ACL 交集查询,不在客户端缓存全量资源后过滤;Folder 选项
|
||||
来自已授权 KB 的 Metadata 接口,选项显示祖先路径并明确“含子文件夹”。切换身份会重新解析可读
|
||||
Knowledge Base,未知、跨 KB 或越权范围不会降级为无约束列表;
|
||||
- 知识库管理把现有 Folder API 的 `parent_folder_id` 平面投影在浏览器中派生为只读目录树,不复制
|
||||
服务端层级事实。知识库根节点和含子项 Folder 使用原生 `details/summary`,支持鼠标与键盘展开;
|
||||
深层和长名称保持单行省略并通过完整路径提示保留语义,移动端隐藏次要计数而不丢失层级;
|
||||
- 页面不保存或展示 Provider 凭据、内部 Endpoint 或 storage key;知识检索页只展示诊断所需的
|
||||
Active Index Profile 与模型标识,不返回 Provider 原始配置;
|
||||
- Console 上传页只轮询当前批次的聚合接口;Document Inspector 展示所选 Version 的持久化 Job、
|
||||
Attempt 与安全错误,但不读取 Provider/队列细节,也不提供人工重试按钮;
|
||||
- Inspector 从内部 Parsed Artifact 投影页级 OCR Confidence 和稳定警告,不返回完整 Artifact 或
|
||||
Provider 原始响应;证据不可读取时明确显示降级状态,不隐藏仍有效的规范 Chunk;
|
||||
- “查看文档”和“定位命中来源”打开 Source 预览/解析溯源双视图。PDF 使用浏览器原版式阅读器并
|
||||
从命中起始页打开;Markdown、纯文本和图片安全在线预览;Office/HTML 强制下载。跨页 Chunk
|
||||
显示完整页范围和实际打开的起始页,不把起始页误述为精确页;
|
||||
- 检索过程使用候选数量标签而非无说明序号;Dense、Sparse、Fusion、Published Compiled Expansion、
|
||||
Reranker 和 Final 形成六阶段证据轨。Final 中 Raw 继续展示精确命中与原文入口,Compiled 使用
|
||||
“已审核知识”标识和底层来源轨道。文本 Source 的 Blob 显式声明 UTF-8,二进制附件保持下载语义,
|
||||
避免浏览器把原始字节渲染为乱码;
|
||||
- 文件夹上传依赖浏览器目录选择 API,只传输文件及其相对路径,空目录不会创建占位 Folder;服务端
|
||||
校验路径并在已授权目标下创建/复用目录。ZIP 入口在完整安全预检后保留包内多级目录并逐文件
|
||||
入队;当前仍不包含 RAR/7z/tar、断点续传或跨批次去重;
|
||||
- 当前页面已接入知识总览、Document Version、Hybrid/BM25、检索诊断、可信 REST 身份、User/Group/
|
||||
Role 关系、Knowledge Base/Folder、ACL 管理,以及 Compiled Knowledge 草稿、来源当前性、冲突证据、
|
||||
审核、退回、发布和 Published 检索;审计查询等后续能力不创建空白占位入口;
|
||||
- 删除无引用文档时清理 canonical Metadata、派生产物、源文件与向量行。被 Compiled Knowledge 引用
|
||||
时改为 tombstone 逻辑删除:从检索和普通文档入口隐藏、移除向量并标记关联 L2 为 Stale,同时保留
|
||||
Source/Version/Chunk 追溯证据。
|
||||
|
||||
`/admin/compiled-knowledge` 把真实来源片段整理成可审核知识:编辑器从可读知识库、Ready Document
|
||||
和规范 Chunk 中选择来源,手工保存或调用可选 LLMProvider 生成 Draft;库存通过来源证据轨道展示每个
|
||||
Document Version 的当前性,并可回到 Inspector 核对。Draft 可提交审核;Manage 身份可记录/解决
|
||||
双来源冲突、发布或填写原因退回。Stale 项修订时重新选择来源,所有写操作携带当前 revision。
|
||||
|
||||
`/admin/search` 对 Compiled Final 展示完整已发布正文,并把每个 `underlying_sources[]` 渲染为可读
|
||||
证据轨道:文件名、Version、Chunk 顺序、章节/页、精确正文摘要,以及“定位底层来源”和“查看或
|
||||
下载原始文件”。这些入口继续使用认证 Fetch/Blob 和现有 Inspector,不把 L2 正文或 Chunk 拼接
|
||||
描述成原始文件。
|
||||
|
||||
## 验证
|
||||
|
||||
开发期先运行 `make web-check`。该命令通过锁定且无第三方依赖的 `package-lock.json` 建立干净
|
||||
npm 环境,再检查 `app.js` 模块语法、`index.html` 是否只引用存在的本地版本化运行资源。Node/npm
|
||||
版本分别由根 `.node-version` 与 `package.json#packageManager`/`engines` 约束。
|
||||
|
||||
静态路由和范围契约由 `tests/integration/test_admin_console.py` 覆盖;REST 全链路测试覆盖上传、
|
||||
列表、解析内容、原始文件、搜索与删除。新增或改变 API/前端入口还必须启动真实服务并通过浏览器
|
||||
执行入口、内容读取、命中定位和 Source 访问,不能只依赖 `TestClient` 或静态断言。浏览器验收可
|
||||
使用确定性本地 Provider 和合成文档,不读取真实企业文档;外部 Provider/目标集群验收独立记录。
|
||||
|
||||
2026-09-15 的 `EKP-083` 验收使用真实 Uvicorn/TCP、SQLite、Local ObjectStorage、合成规范身份及
|
||||
两级 Folder 数据。桌面浏览器选择 `Browser Filter KB` 后显示 3 份文档;选择 `Runbooks` 子树只
|
||||
显示父级和 `Runbooks / East` 中的 2 份文档,选择知识库根目录只显示 1 份,选择空 Folder 后显示
|
||||
“当前范围没有文档”及清除筛选入口。Folder 选项按父级在前排列并展示完整祖先路径。390×844
|
||||
复核为单列筛选、页面无横向溢出,浏览器 Console warning/error 为 0。真实 TCP 自动测试同时覆盖
|
||||
ACL 交集、根目录、后代展开、跨 KB Folder `404` 和非法参数 `422`;Fake Provider 只替代外部模型,
|
||||
不替代浏览器、网络、应用服务、规范数据库或对象存储边界。
|
||||
|
||||
2026-09-15 的 `EKP-084` 自动 E2E 在真实 Uvicorn/TCP 进程中通过管理 API 创建 Knowledge Base 与
|
||||
基准 Folder,经 multipart `relative_paths` 上传两级目录,实际 `IngestionJobRunner` 完成解析、
|
||||
Chunk 与索引后批次直接进入 `completed`;随后读取规范 Folder 层级、不可变 Source 字节和 Content
|
||||
Chunk,证明 API、Worker、SQLite 与 Local ObjectStorage 边界贯通。可见 Chromium 进一步读取用户
|
||||
授权的 `/Users/d-robotics/Downloads/知识库`:首次验收发现目录 File 默认把完整相对路径作为
|
||||
multipart `filename`,修复为显式 basename 后,同一批次 6 个 Markdown/DOCX 全部完成,`.DS_Store`
|
||||
单项安全拒绝;页面显示 `6 / 1` 和每项完整相对路径,文档管理页可见规范“知识库” Folder 与 6 份
|
||||
Ready 文档,Console Error 为 0。两级嵌套目录另由真实进程自动 E2E 覆盖。
|
||||
|
||||
同日 `EKP-086` 验收启动真实 Uvicorn/TCP、SQLite 和规范管理身份:先从页面“新建文件夹”创建根
|
||||
Folder,再经真实管理 API 写入 4 层目录、并行根目录、兄弟叶子及长名称,刷新后目录树显示 6 个
|
||||
Folder 的准确父子关系。浏览器以鼠标展开深层节点,并聚焦相同原生 Summary 后使用 Enter 完成
|
||||
收起/再次展开;空树提示、中文状态标签、完整路径辅助信息均可见,Console warning/error 为 0。
|
||||
真实 Safari 窄窗口验收触发 820px 单列导航后,再以浏览器放大触发 540px 断点:卡片标题与状态
|
||||
纵向排列、隐藏叶子次要计数,三级长名称保持在树容器内并保留层级轨道,不改变目录顺序或可展开
|
||||
操作。浏览器窗口、缩放和测试标签页在验收后均已恢复。
|
||||
|
||||
2026-09-09 的 `EKP-080` 验收启动进程隔离 Uvicorn,并通过真实 TCP 读取 `/admin/overview`、静态
|
||||
资源和 `GET /api/v1/documents`;合成 Markdown 完成实际入库后,总览显示 `1` 份 Source、`0`
|
||||
份处理中、`1` 份 Ready 和 `100%` 就绪率。可见浏览器分别以 1440×900 与 390×844 复核桌面/
|
||||
移动布局,并从最近文档点击“查看证据链”打开真实 Source 预览;浏览器控制台无错误。
|
||||
|
||||
2026-09-09 的 `EKP-081` 运行态验收使用本机真实 FastAPI 进程、SQLite 持久化、Local
|
||||
ObjectStorage 和确定性 Provider,加载合成 Markdown 与三页 PDF Source。浏览器中完成了:文档
|
||||
列表“查看文档”默认连续渲染 Markdown 并可切换查看 4 个顺序 Chunk;知识检索页经真实 HTTP
|
||||
返回带上下文和精确命中片段的结果;三页 PDF 命中显示 `Page 1–3`,“定位命中来源”打开真实 PDF
|
||||
阅读器,明确提示“命中范围第 1–3 页、从第 1 页打开”,Source URL 带 `#page=1`。同一运行态还
|
||||
使用真实持久化 Job/Attempt 和 schema v4 页质量 Artifact 验证:展开“处理流水线”可见五阶段及
|
||||
成功 Attempt;第 2 页 OCR 42% 显示为需复核警告。该记录证明本地用户路径与持久化/存储接线,
|
||||
不代表外部 Provider、GPU 或目标集群现场验收。
|
||||
|
||||
2026-09-14 的 `EKP-091` 运行态验收使用隔离 Uvicorn、SQLite、Local ObjectStorage 和合成身份/
|
||||
Markdown,在管理端完成双来源草稿创建、来源核验与提交审核、双来源冲突记录、未解决时隐藏发布、
|
||||
填写解决说明后发布,以及第二条知识的退回原因和修订为新草稿。来源轨道和冲突配对均显示真实
|
||||
文件名、Version、Chunk 顺序与章节,并能进入 Document Inspector 定位精确 Chunk。桌面操作后以
|
||||
390×844 复核完整列表和 Inspector 弹窗,控制台 warning/error 为空。首轮浏览器操作发现序号来源
|
||||
难辨认及同文档同版本来源歧义,已在本任务内修正后重新完成全链路验收。
|
||||
|
||||
同日 `EKP-092` 运行态验收使用隔离 Uvicorn、真实 TCP、SQLite、Local ObjectStorage 和确定性
|
||||
Provider。可见浏览器登录后检索“审核发布的产品 A 退款统一结论”,页面显示 Dense、Sparse、
|
||||
Fusion、Published Compiled Expansion、Reranker、Final 六阶段;Final 首位展示已发布正文和
|
||||
“已审核知识”标识。展开底层来源可见 `refund-policy.md · V1 · Chunk 2`、章节位置和精确 Source
|
||||
摘要;点击“定位底层来源”后,Document Inspector 打开确切 Chunk 2。桌面路径完成后以 390×844
|
||||
复核诊断轨、Compiled 卡片和来源轨道,浏览器控制台 warning/error 为空。该证据只证明本地真实
|
||||
应用/网络/存储接线和确定性 Provider 契约,不替代外部模型 Benchmark 或目标集群验收。
|
||||
|
||||
同日 `EKP-056` 验收启动更新后的真实 Uvicorn 进程并使用可见浏览器打开 `/admin/search`:提交
|
||||
“产品 A 的退款周期是多少?”后,页面显示 `public_knowledge / ACL 未接线`、`e2e-profile-v1`、
|
||||
Dense/Sparse/Fusion/Reranker/Final 五阶段真实候选数量与单调时钟耗时;展开 Dense 可见候选名次、
|
||||
分数和安全正文摘要,Final 结果可继续打开 Inspector 并定位精确 Chunk。窄屏信号轨支持横向滚动,
|
||||
折叠阶段不被展开列强制等高。该过程实际调用 `/search/diagnostics`、Document 内容和 Source API。
|
||||
|
||||
2026-09-10 的 `EKP-082` 验收启动真实 Uvicorn 进程,通过 TCP 完成严格 `401`、规范 Session、
|
||||
Knowledge Base/Folder、User/Group/Role、成员与角色关系、Role ACL、KB-scoped multipart 上传和
|
||||
ACL-aware Search/Document 读取。可见浏览器在真实 `/admin/permissions`、`/admin/knowledge-bases`、
|
||||
`/admin/upload` 与 `/admin/documents` 页面完成身份连接、资源创建、授权、范围选择和文档阅读;
|
||||
“新窗口打开原始文件”产生认证读取后的 Blob 页面,地址不含 Token。Fake Provider 只替代外部
|
||||
Embedding/Vector/Reranker,不替代浏览器、网络、应用服务、SQLite 规范数据或 Local ObjectStorage。
|
||||
最终低权限复验使用只含 Read/Search Scope 的 Alice:侧栏只保留总览、文档和知识检索,删除
|
||||
按钮不可见,列表只显示 Role ACL 授权的一个文档;点击“查看文档”连续呈现 Markdown 标题和正文。
|
||||
|
||||
2026-09-10 的 `EKP-080` 体验补充验收继续使用真实 Uvicorn、TCP、SQLite、Local ObjectStorage
|
||||
和确定性 Provider。可见浏览器完成身份登录、知识检索、页面刷新后的查询/结构化诊断响应恢复,
|
||||
并从恢复结果成功打开“定位命中来源”;“清空查询”只清除输入,不删除当前结果。知识库新增弹窗和
|
||||
390×844 权限授权弹窗均可读,浏览器 Console 无 warning/error。会话证据在切换访问身份时清除,
|
||||
避免跨身份保留。自动门禁为:管理台静态集成 `3 passed`、真实 TCP E2E `4 passed`、完整
|
||||
Unit/Integration `709 passed`、Evaluation `8 passed`、Mypy 166 个源文件以及 Migration head
|
||||
`20260909_0020` 全部通过。
|
||||
|
||||
- 品牌命名边界:左上品牌主体使用“地瓜机器人”,工作台标题和左下平台标识使用“地瓜知识平台”;左上品牌组合按比例放大图标、主名称、英文副标题和间距。
|
||||
- 文档导入区提供文件、多层文件夹和单 ZIP 三个明确入口;普通批次提示 `BATCH_MAX_FILES=20`,ZIP
|
||||
提示 `ZIP_MAX_FILES=1000`。两种目录来源均显示保留后的相对路径,结构安全但格式不支持的条目
|
||||
单项拒绝。
|
||||
- 查询留存、清空按钮和诊断恢复已收敛到 `app.js` 与 `index.html`,不再依赖已删除的 `ux-enhancements.js`。
|
||||
157
hy4/_kp/docs/development/ASYNC_INGESTION_SETUP.md
Normal file
157
hy4/_kp/docs/development/ASYNC_INGESTION_SETUP.md
Normal file
@ -0,0 +1,157 @@
|
||||
# 异步入库、编纂与失败恢复说明
|
||||
|
||||
## 范围
|
||||
|
||||
DEMO-015 将单文件上传从 API 请求线程迁移到独立 Celery Worker,DEMO-016 在其上增加逐文件
|
||||
隔离的批量接收和状态聚合。OceanBase 中的
|
||||
`ingestion_jobs` 和 `ingestion_attempts` 是执行状态事实源;每个 Job 固定绑定一个
|
||||
`document_version_id`,RabbitMQ 消息只携带 Job ID,
|
||||
不携带文件内容、Provider 凭据或业务状态。
|
||||
|
||||
批次只聚合接收与 Job 状态,不改变每个文件的独立执行和重试边界。
|
||||
|
||||
`EKP-103` 复用同一数据库事实源与消息提示原则,为 AI 草稿新增独立
|
||||
`compilation_jobs`、`compilation_job_sources`、`compilation_attempts` 和 `compilation` queue。
|
||||
Compilation 消息同样只携带 Job ID,不携带 instruction、来源正文或 Provider 响应。
|
||||
|
||||
## 运行时配置
|
||||
|
||||
以下值通过环境变量、部署 Secret 或用户明确授权提交的具体环境文件注入:
|
||||
|
||||
```text
|
||||
RABBITMQ_URL=amqp://<user>:<password>@<host>:5672/<vhost>
|
||||
INGESTION_MAX_ATTEMPTS=3
|
||||
INGESTION_RETRY_BACKOFF_SECONDS=2
|
||||
INGESTION_RETRY_BACKOFF_MAX_SECONDS=60
|
||||
INGESTION_WORKER_LEASE_SECONDS=300
|
||||
INGESTION_RECOVERY_INTERVAL_SECONDS=30
|
||||
COMPILATION_MAX_ATTEMPTS=3
|
||||
COMPILATION_RETRY_BACKOFF_SECONDS=2
|
||||
COMPILATION_RETRY_BACKOFF_MAX_SECONDS=60
|
||||
COMPILATION_WORKER_LEASE_SECONDS=300
|
||||
COMPILATION_RECOVERY_INTERVAL_SECONDS=30
|
||||
BATCH_MAX_FILES=20
|
||||
BATCH_MAX_TOTAL_BYTES=209715200
|
||||
ZIP_MAX_FILES=1000
|
||||
ZIP_MAX_COMPRESSION_RATIO=100
|
||||
```
|
||||
|
||||
Worker 与 API 还需要相同的 OceanBase、BOS、Embedding、VectorStore 和 Index Profile 配置。
|
||||
RabbitMQ 用户应只具备指定 vhost/queue 的最小权限。
|
||||
|
||||
启用 `PARSER_PROVIDER=mineru` 时,Worker 还必须获得相同的 `MINERU_*` 配置,并令
|
||||
`INGESTION_WORKER_LEASE_SECONDS` 严格大于 `MINERU_TIMEOUT_SECONDS` 和其他 Provider Timeout。
|
||||
默认 MinerU Timeout 为 600 秒,因此租约至少设置为 601 秒;Settings 与 Helm 会拒绝不安全值。
|
||||
启用 `LLM_PROVIDER=openai_compatible` 时,`COMPILATION_WORKER_LEASE_SECONDS` 必须严格大于
|
||||
`LLM_TIMEOUT_SECONDS`。
|
||||
|
||||
## 启动顺序
|
||||
|
||||
```bash
|
||||
uv run alembic upgrade head
|
||||
make run-worker
|
||||
make run
|
||||
```
|
||||
|
||||
`make run-worker` 在当前 Demo 中同时启动 Worker 和恢复调度器。恢复调度器周期扫描:
|
||||
|
||||
- 尚未成功投递的 `queued` Job;
|
||||
- 已到重试时间的 `retrying` Job;
|
||||
- Worker 丢失后租约已过期的 `running` Job。
|
||||
|
||||
任务投递使用 late acknowledgement、worker-lost reject 和单任务 prefetch。重复消息会先锁定
|
||||
Job 行;有效租约内不会创建第二个 Attempt。租约过期后,旧 Attempt 以安全错误关闭,再按重试
|
||||
规则恢复。每次 Claim 同时写入不可公开的 Lease Fencing Token;Stage 续租、Attempt 结束和清理
|
||||
失败收敛都必须携带当前 Token。若旧 Worker 在新 Claim 后恢复,它会被识别为已替换执行并安全
|
||||
确认该旧消息,不能覆盖新 Attempt 的状态。
|
||||
|
||||
Compilation Worker 在每次 Attempt 执行时重新解析提交 User 当前的 Knowledge Base Write 和全部
|
||||
来源 Document Read 权限,并重新核验 Active Version 与 Data Processing Policy。只有当前租约
|
||||
持有者能写入通过唯一 `generation_job_id` 绑定的 Draft;重复投递或旧 Worker 恢复最多复用既有
|
||||
结果,不能创建第二条 Draft 或覆盖新 Attempt。
|
||||
任一历史 Compilation Job 引用的 Chunk 会随 Document 逻辑删除保留,避免删除未生成 Draft 的失败
|
||||
Job 时破坏执行证据;向量/稀疏索引仍会移除,已生成的 Compiled Knowledge 同时进入 Stale。
|
||||
|
||||
## 重试与清理
|
||||
|
||||
确定性的内容错误(无法解析、清洗后为空、无 Chunk、派生数据身份不一致)直接进入 `failed`。
|
||||
其余运行时/Provider 错误按指数退避重试,默认最多三次总 Attempt。
|
||||
|
||||
第二次及后续 Attempt 创建前严格执行:
|
||||
|
||||
```text
|
||||
该 DocumentVersion 的 OceanBase Vector rows
|
||||
→ ChunkIndexState
|
||||
→ Chunk
|
||||
→ parsed JSON object
|
||||
→ reset DocumentVersion processing state
|
||||
```
|
||||
|
||||
不可变源文件、Document、历史 Version、Job 和已结束 Attempt 始终保留。旧 Active Version 的
|
||||
派生数据和检索状态不受候选 Version 重试影响。任一清理步骤失败时不创建新 Attempt,Job 直接以
|
||||
`retry cleanup failed` 终止,禁止在未知脏状态继续执行。
|
||||
|
||||
## API 状态
|
||||
|
||||
`POST /api/v1/documents` 成功返回 `202`、`document_version_id` 和 `job_id`。查询:
|
||||
|
||||
```text
|
||||
GET /api/v1/jobs/{job_id}
|
||||
```
|
||||
|
||||
Job 状态为 `queued | running | retrying | succeeded | failed | cancelled`;阶段为
|
||||
`parsing | cleaning | chunking | embedding | indexing`。公开错误只返回固定安全摘要,不保存或
|
||||
返回 Broker/Provider 异常正文。活动 Job 的 Document 暂不允许删除,API 返回 `409`。
|
||||
|
||||
批量提交与查询:
|
||||
|
||||
```text
|
||||
POST /api/v1/ingestion-batches # multipart 字段 files;目录模式再提供同序 relative_paths
|
||||
POST /api/v1/ingestion-archives # multipart 字段 archive;单个安全 ZIP
|
||||
GET /api/v1/ingestion-batches/{batch_id}
|
||||
```
|
||||
|
||||
批次创建后逐文件返回 `accepted` 或 `rejected`;已接收文件继续通过上述 Job 状态异步推进。
|
||||
批次聚合状态为 `processing | completed | partial_failure | failed`。请求文件数超过
|
||||
`BATCH_MAX_FILES` 时整体返回 `413`;单文件或共享总字节预算超限只拒绝当前文件,不删除此前
|
||||
已接收文件。默认总预算 200 MiB,且配置不得小于单文件 `MAX_UPLOAD_BYTES`。
|
||||
|
||||
文件夹上传由浏览器为每个 `files` 提供 `webkitRelativePath`,并以重复 `relative_paths` 字段按同序
|
||||
提交。服务端拒绝绝对路径、父级跳转、反斜线、空段、控制字符和文件名错位,只把验证后的相对路径
|
||||
写入 `ingestion_batch_items.relative_path`。目录层级在已授权 Knowledge Base/目标 Folder 下创建或
|
||||
复用;基准 Folder 跨 KB 时在对象存储前返回 `404`。文件数量、单文件/共享字节预算、去重、Job
|
||||
重试和逐文件部分失败语义不因目录模式改变。
|
||||
|
||||
ZIP 入口使用独立文件数上限,并在任何规范写入前完成 CRC、路径、链接/特殊文件、加密、嵌套包、
|
||||
压缩包大小、单文件/展开总量和压缩比预检。平台元数据和空目录不入库;不支持的普通文件类型仍按
|
||||
批次逐项拒绝。不同 Folder 的重复文件名是独立 Document,同一 Folder 内保持版本演进。
|
||||
|
||||
AI 编纂提交与查询:
|
||||
|
||||
```text
|
||||
POST /api/v1/compiled-knowledge/drafts:generate # 返回 202 + Job
|
||||
GET /api/v1/compiled-knowledge/jobs/{job_id} # 权限交集内轮询
|
||||
```
|
||||
|
||||
成功响应通过 `compiled_knowledge_id` 指向新 Draft。查询不返回 instruction、来源内容、租约 Token
|
||||
或原始错误;管理端的轮询最多等待 120 秒,超时不会取消仍由 Worker 持续推进的规范 Job。
|
||||
|
||||
## 故障检查
|
||||
|
||||
1. Job 长期为 `queued`:检查 RabbitMQ 连通性、恢复调度器以及 `ingestion` queue;
|
||||
2. Job 长期为 `running`:检查 Worker,租约到期后恢复调度器会重新投递;
|
||||
3. Job 为 `retrying`:等待 `next_run_at`,客户端不应重新上传同一文件;
|
||||
4. Job 为 `failed` 且错误为 `retry cleanup failed`:停止人工重试,先核对向量、关系数据与 BOS
|
||||
派生对象的一致性;
|
||||
5. 不得通过日志打印 `RABBITMQ_URL`、数据库 URL、BOS/模型凭据或原始 Provider 响应。
|
||||
6. Compilation Job 长期等待时同时检查 `compilation` queue、LLM 配置、Worker 与恢复调度器;失败
|
||||
后只按公开安全错误定位,原始 Provider 正文不得写入 Job、Attempt、Audit 或日志。
|
||||
|
||||
`EKP-023` 的本地集成回归覆盖 Job/Attempt 持久化、Broker 失败恢复、有效租约重复投递、过期
|
||||
租约恢复、Fencing Token 拒绝旧 Worker、有界瞬时重试、永久错误、清理前置、清理失败关闭及
|
||||
Source 保留。该任务未新增或改变 API、页面、按钮或用户入口,因此不触发新的运行态 E2E;
|
||||
`EKP-021` 已记录的真实 HTTP 上传链路继续证明 Job 创建入口。
|
||||
|
||||
`EKP-103` 回归覆盖 Compilation Job/Attempt 持久化、瞬时故障有界重试、执行时权限重验、有效与
|
||||
过期租约 Fencing、重复投递单 Draft、固定查询数、Shadow Index 重建保留 L2/L1 规范数据,以及
|
||||
真实 TCP 和浏览器 `202 → Job → Draft` 链路。
|
||||
73
hy4/_kp/docs/development/BOS_SETUP.md
Normal file
73
hy4/_kp/docs/development/BOS_SETUP.md
Normal file
@ -0,0 +1,73 @@
|
||||
# ObjectStorage 与 BOS 配置
|
||||
|
||||
`ARCH-002` 新增百度 BOS Adapter,`EKP-020` 补齐并关闭完整 `ObjectStorage` Provider 范围。
|
||||
业务服务仍只使用逻辑 key,不知道 Bucket、Prefix、SDK 类型或凭据。`ARCH-005` 已将 BOS 设为
|
||||
正式运行默认值;Local Adapter 继续用于单元测试和显式配置的本地开发。
|
||||
|
||||
## 运行配置
|
||||
|
||||
配置由部署环境、Secret Manager、本机 `.env` 或用户明确授权提交的环境文件注入;真实值不得
|
||||
进入镜像、日志或测试快照,提交范围必须记录并限定到获授权的环境文件。
|
||||
|
||||
```text
|
||||
OBJECT_STORAGE_PROVIDER=bos
|
||||
BOS_ENDPOINT=<已授权的 BOS Host 或 HTTP(S) Origin>
|
||||
BOS_ACCESS_KEY_ID=<运行时注入>
|
||||
BOS_SECRET_ACCESS_KEY=<运行时注入>
|
||||
BOS_BUCKET=<预创建的 Bucket>
|
||||
BOS_KEY_PREFIX=knowledge-platform
|
||||
BOS_TIMEOUT_SECONDS=30
|
||||
```
|
||||
|
||||
- Endpoint 必须是无用户名、密码、Path、Query 和 Fragment 的 Host 或 HTTP(S) Origin;旧配置
|
||||
的纯 Host 会在 Provider 边界强制规范化为 HTTPS,禁止沿用 SDK 的默认 HTTP;
|
||||
- Bucket 名必须符合 BOS 的小写字母、数字和连字符规则;
|
||||
- Prefix 可为空;非空时使用安全的 POSIX 相对 key,不允许空段、`.`、`..` 或反斜杠;
|
||||
- 应用不会创建、删除或修改 Bucket 配置。
|
||||
|
||||
## 对象与不可变约束
|
||||
|
||||
canonical `Document.storage_key` 保存与 Provider 无关的逻辑 key,例如:
|
||||
|
||||
```text
|
||||
documents/<document_id>/<filename>
|
||||
```
|
||||
|
||||
BOS Adapter 发起请求时才映射为:
|
||||
|
||||
```text
|
||||
<BOS_KEY_PREFIX>/documents/<document_id>/<filename>
|
||||
```
|
||||
|
||||
上传先写入可回卷的临时流,以满足官方 SDK 对长度和 CRC 计算的要求;随后使用
|
||||
`If-None-Match: *` 执行单对象 Put,服务端返回 412 时统一映射为
|
||||
`ObjectAlreadyExistsError`。这样不会用“先 HEAD、再 PUT”的竞态检查破坏不可变来源约束。
|
||||
|
||||
下载返回 SDK 的流式 Response Body,由调用方负责关闭;存在性检查只发 HEAD,不下载正文。
|
||||
404、412 和其他 SDK 异常映射为稳定的 Provider-neutral 错误,外层错误不包含 Endpoint、
|
||||
SDK Response Body 或凭据。
|
||||
|
||||
Local 与 BOS 必须通过同一份 Provider 契约测试:安全逻辑 Key、Put/Open/Exists/Delete 生命周期、
|
||||
缺失对象、不可覆盖和写入期间不可见。Local Put 先在目标目录写入临时文件并落盘,再使用不覆盖的
|
||||
原子硬链接发布;因此读取方不会看到部分 Source,并发写入也不能替换已经成功的对象。输入流或
|
||||
Provider I/O 失败统一收敛为不包含本地路径、Endpoint 或上游响应的 `ObjectStorageError`。
|
||||
|
||||
Worker 租约只计算当前启用的 Provider Timeout。显式选择 Local 时,不再被未使用的
|
||||
`BOS_TIMEOUT_SECONDS` 限制;生产默认 BOS 仍要求租约严格长于 BOS、Embedding、VectorStore 及
|
||||
已启用 Parser 的最长 Timeout。
|
||||
|
||||
## 权限与验收边界
|
||||
|
||||
运行身份只授予目标 Bucket/Prefix 所需的 Put、Get/Head 和 Delete 权限,不授予 Bucket 管理或
|
||||
全局 List 权限。若生产策略要求来源不可物理删除,应在后续 Document Version/Retention
|
||||
设计中单独处理;当前 Demo 的直接删除行为保持不变。
|
||||
|
||||
默认测试使用 Local Adapter 和 Fake BOS Transport,不连接 BOS。真实连接、权限和临时对象清理
|
||||
已在 `ARCH-006` 的受控 test 资源中验证;该现场证据不把凭据写入仓库、日志或测试快照。后续目标
|
||||
环境复验属于独立部署验收,不阻塞 `EKP-020` 的 Provider 开发闭环。
|
||||
|
||||
官方参考:
|
||||
|
||||
- [BOS Python SDK 上传文件](https://cloud.baidu.com/doc/BOS/s/yl5jfscvi)
|
||||
- [BOS 获取 Object](https://cloud.baidu.com/doc/BOS/s/Mk4xtzv8v)
|
||||
- [BOS 访问控制](https://cloud.baidu.com/doc/BOS/s/Tjwvysda9)
|
||||
30
hy4/_kp/docs/development/CI_BASELINE.md
Normal file
30
hy4/_kp/docs/development/CI_BASELINE.md
Normal file
@ -0,0 +1,30 @@
|
||||
# CI 基线
|
||||
|
||||
`EKP-006` 提供与具体托管平台无关的单一入口:
|
||||
|
||||
```bash
|
||||
make ci
|
||||
```
|
||||
|
||||
该命令按顺序、失败即停止地执行:Python 锁文件/工具链检查、管理端依赖锁定与静态资源检查、
|
||||
Ruff Format/Lint、Mypy、Unit/Integration、Migration 契约、离线 RAG Evaluation,以及通过本机
|
||||
回环网络启动真实 API/MCP 进程的 E2E。所有 Python 命令使用 `uv run --locked`,Web 检查使用
|
||||
锁文件驱动的 `npm ci`。
|
||||
|
||||
## 安全边界
|
||||
|
||||
默认 CI 不读取真实 Provider 凭据,不连接目标数据库、对象存储、模型服务或 GPU,也不执行
|
||||
`live-acceptance`、Embedding/Reranker Benchmark、镜像推送、集群部署、远程 MCP/WorkBuddy 验收或
|
||||
依赖锁重写。上述操作分别属于显式验收、Benchmark 或部署/运维任务,必须单独授权和记录。
|
||||
|
||||
仓库当前的中央 dev 发布由受管 Drone 模板和 `dev` 分支负责。`make ci` 是可由本地或任意托管
|
||||
Runner 调用的开发质量契约,不复制、不猜测中央模板,也不会因为目标环境暂不可用而阻塞开发主
|
||||
队列。托管 Runner 的接线与运行证据属于独立部署/运维队列。
|
||||
|
||||
## 维护规则
|
||||
|
||||
- 新增通用开发门禁时,同步加入 `ci` 目标及其契约测试。
|
||||
- CI 只能调用只读、确定性、无真实 Provider 依赖的目标;发布和现场验收不得混入默认入口。
|
||||
- 新增或改变 API、页面、按钮或用户入口时,必须扩展 `tests/e2e` 的真实运行路径;`make ci` 会自动
|
||||
执行该套件。
|
||||
- 本地复现必须直接运行 `make ci`,不得用一次成功的目标环境部署替代工程门禁。
|
||||
179
hy4/_kp/docs/development/CLUSTER_OPERATIONS.md
Normal file
179
hy4/_kp/docs/development/CLUSTER_OPERATIONS.md
Normal file
@ -0,0 +1,179 @@
|
||||
# 公共知识库集群运行与验收手册
|
||||
|
||||
## 1. 范围与安全边界
|
||||
|
||||
本手册覆盖 DEPLOY-003 的目标集群安装、幂等升级、滚动恢复、Worker 故障恢复、最小日志/指标、
|
||||
备份责任和远程 MCP/WorkBuddy 验收。该 DEPLOY-003 现场基线仍按企业受控网络内的单一公共知识库
|
||||
验收;应用代码已在 `EKP-082` 支持可信 REST 身份、多 Knowledge Base 与 Document ACL,目标环境
|
||||
启用这些能力前必须配置规范身份/授权 Secret 并完成独立迁移验收。本手册不提供完整 Observability
|
||||
平台和跨区域灾备。
|
||||
|
||||
所有敏感值必须由 Secret 管理系统写入 Kubernetes Secret。Values、Git、镜像、命令行参数、
|
||||
验收记录和日志中不得出现数据库 URL、RabbitMQ URL、云密钥、Provider Key 或 MCP Token。
|
||||
|
||||
## 2. 交付责任边界
|
||||
|
||||
| 资源 | 平台仓库负责 | 环境/运维负责 |
|
||||
|---|---|---|
|
||||
| 应用镜像 | 可重复构建的 API、MCP、Worker target | 目标 Registry、不可变 tag/digest、镜像扫描和保留策略 |
|
||||
| Kubernetes | Helm Chart、Migration、探针、PDB、资源和 Pod 安全默认值 | 集群、Namespace、Ingress Controller、配额和网络策略 |
|
||||
| 敏感配置 | Secret 引用和必需键名预检 | Secret 创建、轮换、审计和访问控制 |
|
||||
| HTTPS | Ingress Host/TLS Secret 引用 | DNS、证书签发/续期与 TLS Secret |
|
||||
| 数据 | canonical/可重建边界和恢复顺序 | OceanBase、BOS、RabbitMQ 的备份、容量、HA 和恢复演练 |
|
||||
| 监控 | JSON 事件、API `/metrics`、验收检查 | 日志/指标采集、留存、查询、告警和 On-call |
|
||||
|
||||
## 3. 部署前检查
|
||||
|
||||
操作者需要准备仓库外的环境 Values、Namespace、运行时 Secret、TLS Secret、ImagePullSecret,
|
||||
并确认集群可访问 OceanBase、BOS、RabbitMQ、Embedding 和 Reranker。先显式选择并核对 context:
|
||||
|
||||
```bash
|
||||
kubectl config current-context
|
||||
kubectl auth can-i get deployments -n knowledge-platform
|
||||
kubectl auth can-i patch deployments -n knowledge-platform
|
||||
```
|
||||
|
||||
镜像必须使用 Registry 中已存在的不可变 tag 或 digest,禁止 `latest`。部署前执行:
|
||||
|
||||
```bash
|
||||
make deploy-lint VALUES=/secure/path/environment-values.yaml
|
||||
make deploy-render VALUES=/secure/path/environment-values.yaml >/tmp/ekp-rendered.yaml
|
||||
```
|
||||
|
||||
渲染文件只能包含非敏感 ConfigMap 和 Secret 引用;完成检查后应删除临时文件。
|
||||
|
||||
## 4. 安装、幂等升级与只读验收
|
||||
|
||||
首次安装或普通升级使用同一入口:
|
||||
|
||||
```bash
|
||||
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
|
||||
make deploy VALUES=/secure/path/environment-values.yaml
|
||||
```
|
||||
|
||||
只读验收不会触发安装、重启或故障注入,但强制 context 与操作者声明完全一致:
|
||||
|
||||
```bash
|
||||
ACCEPTANCE_KUBE_CONTEXT="$(kubectl config current-context)" \
|
||||
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
|
||||
make cluster-acceptance VALUES=/secure/path/environment-values.yaml
|
||||
```
|
||||
|
||||
## 5. 完整可用性验收
|
||||
|
||||
完整模式会连续执行两次相同 Helm 入口、滚动重启 API/MCP,并在一个合成入库 Job 运行时强制
|
||||
删除一个明确选中的 Worker Pod。只能在已批准的验收 Namespace 执行:
|
||||
|
||||
```bash
|
||||
context="$(kubectl config current-context)"
|
||||
ACCEPTANCE_MODE=full \
|
||||
ACCEPTANCE_KUBE_CONTEXT="${context}" \
|
||||
ACCEPTANCE_CONFIRM="DEPLOY-003:${context}:knowledge-platform:knowledge-platform" \
|
||||
NAMESPACE=knowledge-platform RELEASE=knowledge-platform \
|
||||
make cluster-acceptance VALUES=/secure/path/environment-values.yaml
|
||||
```
|
||||
|
||||
脚本验证以下内容:
|
||||
|
||||
- 相同 Values 的安装/升级可以重复成功,Migration Hook 先于工作负载更新完成;
|
||||
- API/MCP 以及启用的 Edge 至少两个副本全部 Ready/Available,PDB 存在;
|
||||
- Admin、健康检查、指标和 MCP 均通过稳定 HTTPS Host 到达;
|
||||
- API/MCP/Edge 滚动期间连续健康探测无中断;
|
||||
- Worker 被非优雅中断后,同一 canonical Job 经租约到期和恢复调度器完成,不要求重新上传;
|
||||
- 合成文档验收完成后调用公开删除 API 清理,脚本不访问 Secret 值。
|
||||
|
||||
若合成文档处理过快,脚本会失败而不是虚报 Worker 中断已验证。可在受控验收环境调整 Provider
|
||||
延迟或使用批准的非敏感验收文件重跑,不得加入测试后门。
|
||||
|
||||
### 5.1 复用现有域名和 NodePort 的替换流程
|
||||
|
||||
当集群没有 Ingress、外部 HTTPS 网关已固定转发现网 NodePort 时,启用 Edge 模式并按以下顺序
|
||||
切换。以下旧入口切换步骤仅保留为历史回滚参考;DEPLOY-003 当前使用独立开发域名
|
||||
`knowledge-dev.d-robotics.cc` 和 `dev` 流水线发布,不执行 NodePort 替换。命令中的资源名必须先
|
||||
从目标集群核对,不能
|
||||
直接套用到其他环境。
|
||||
|
||||
1. 在与现网入口相同的 Namespace 内安装新 Release,Edge 使用自动分配或确认空闲的临时
|
||||
NodePort;旧 Deployment/Service 保持不变。
|
||||
2. 通过集群内请求或受控端口转发验证 `/healthz`、Admin/API、带认证 `/mcp`、批量异步入库和
|
||||
Worker 故障恢复;确认 API/MCP/Edge PDB 与滚动升级均通过。
|
||||
3. 将旧 Deployment、Service、ConfigMap 引用和公开的镜像坐标保存到权限受控的仓库外目录。
|
||||
不导出 Secret data,不把快照提交 Git。
|
||||
4. 先把旧入口 Service 从 NodePort 改为 ClusterIP 以释放 `30081`,随后立即把新 Edge Service
|
||||
更新为 `30081`。只切换 Service,旧 Deployment 暂时保留。
|
||||
5. 从正式域名复验 `/healthz`、`/admin/upload`、REST、MCP 和真实检索,并观察错误率与 Pod
|
||||
状态。失败时先释放新 Edge 的 `30081`,再按快照恢复旧 Service。
|
||||
6. 稳定观察窗口结束并确认 WorkBuddy 远程验收后,才清理旧 Deployment/Service;数据库
|
||||
Migration 为 forward-only,回切应用前必须确认版本兼容。
|
||||
|
||||
整个切换窗口禁止同时创建两个占用 `30081` 的 Service,也禁止通过删除旧 Deployment 来释放
|
||||
NodePort。外部 HTTPS 网关和证书保持不变,因此用户端无需重新配置域名。
|
||||
|
||||
## 6. 远程 MCP 与 WorkBuddy
|
||||
|
||||
先用官方 MCP Python Client 验证远程协议、严格五个 P0 Tool 和一次完整来源读取链。Token 通过静默输入进入
|
||||
当前进程环境,不写入命令参数或配置文件:
|
||||
|
||||
```bash
|
||||
export EKP_MCP_URL=https://mcp.example.internal/mcp
|
||||
export EKP_MCP_ACCEPTANCE_QUERY='已批准的公共知识验收问题'
|
||||
read -rsp 'MCP token: ' EKP_MCP_TOKEN && export EKP_MCP_TOKEN && echo
|
||||
make mcp-remote-acceptance
|
||||
unset EKP_MCP_TOKEN
|
||||
```
|
||||
|
||||
随后在与最终用户相同网络路径的 WorkBuddy 客户端加载
|
||||
[`workbuddy-mcp.example.json`](../../deploy/workbuddy-mcp.example.json),只通过客户端安全环境注入
|
||||
`EKP_MCP_URL` 和 `EKP_MCP_TOKEN`。验收必须确认:连接状态正常、只发现五个批准的 P0 Tool、
|
||||
真实问题返回至少一个带来源字段的结果,并可继续读取 Source、Document、Section 和所属 Knowledge
|
||||
Base。记录客户端版本、时间、网络区域、
|
||||
Tool 名和成功/失败即可,禁止复制 Token、正文或内部 Provider 信息。
|
||||
|
||||
## 7. 日志、指标与最小告警
|
||||
|
||||
应用事件为单行 JSON,固定基础字段为 `timestamp`、`level`、`logger`、`message` 和 `event`;可选
|
||||
白名单字段包括 `request_id`、HTTP 方法/路由模板/状态/耗时、Job/Document/Version/Batch ID、
|
||||
`attempt_number`、`stage`、`dependency`、`job_status`、`retry_delay_seconds`、`retryable` 和
|
||||
`dispatch_count`。事件名必须是最长 64 字符的小写 `snake_case`;请求上下文中的已校验
|
||||
`request_id` 优先于日志调用附加值。应用日志消息必须是代码字面量并显式声明 `event`,异常正文、
|
||||
插值参数、嵌套对象和非白名单字段不会被格式化器写出;没有有效结构化 `event` 的依赖日志只输出
|
||||
`unstructured_log_suppressed` 安全标记。常用检查:
|
||||
|
||||
```bash
|
||||
kubectl -n knowledge-platform logs deployment/knowledge-platform-api --since=15m
|
||||
kubectl -n knowledge-platform logs deployment/knowledge-platform-worker --since=15m
|
||||
curl --fail https://knowledge.example.internal/metrics
|
||||
kubectl -n knowledge-platform get deployments,pods,poddisruptionbudgets
|
||||
```
|
||||
|
||||
环境监控至少采集并告警:API/MCP Ready 副本不足、Pod 重启持续增加、HTTP 5xx、P95 请求耗时、
|
||||
Worker 无 Ready 副本、`ingestion_attempt_failed`/`ingestion_retry_cleanup_failed`、恢复扫描持续
|
||||
失败、Migration Job 失败,以及 OceanBase/BOS/RabbitMQ/模型服务在应用日志中的安全失败事件。
|
||||
阈值由真实流量基线确定,不在本任务中固化完整监控平台。
|
||||
|
||||
## 8. 故障处置
|
||||
|
||||
API/MCP 不可用时先检查 Deployment/Endpoint/Ingress/TLS,再按单个 Deployment 滚动重启;禁止
|
||||
直接打印 Pod 环境变量。Job 卡住时使用 Admin/API 的 Job ID 查询状态,再按相同 Job ID 搜索
|
||||
Worker JSON 事件。处于有效租约的 Job 不应手工重复提交;租约到期后恢复调度器会重投递。
|
||||
|
||||
更新同名 Secret 不会自动改变已运行进程。由运维完成 Secret 轮换后,依次滚动重启 API、MCP、
|
||||
Worker 和 Scheduler,并复跑只读验收。稳定 Host 和客户端 Token 未轮换时,Admin/WorkBuddy
|
||||
无需重新配置;MCP Token 轮换时必须通过客户端 Secret 分发流程更新。
|
||||
|
||||
## 9. 备份与恢复顺序
|
||||
|
||||
OceanBase 关系表和 BOS 不可变源文件共同构成事实数据,必须由资源所有者执行一致性备份策略。
|
||||
OceanBase 向量表是可重建索引,不得代替 canonical 备份。RabbitMQ 消息只提供执行提示,
|
||||
canonical Job/Attempt 才是恢复依据,但生产 Broker 仍应按其 SLA 做持久化和 HA。
|
||||
|
||||
恢复顺序为:恢复/验证 OceanBase canonical 表 → 恢复并核对 BOS 对象 → 部署相同或兼容应用版本
|
||||
并执行 forward-only Migration → 恢复 RabbitMQ 连通 → 启动 Worker/Scheduler → 按 active Index
|
||||
Profile 重建缺失向量 → 验证来源追踪和真实检索。Helm 回滚只能回退兼容的应用清单,不能反向
|
||||
撤销数据库 Migration。完整灾备演练仍是后置任务。
|
||||
|
||||
## 10. 验收记录
|
||||
|
||||
每次目标集群验收按 [`DEPLOY_003_ACCEPTANCE.md`](DEPLOY_003_ACCEPTANCE.md) 记录非敏感结果。
|
||||
只有完整自动化验收、真实 MCP 协议调用和远程 WorkBuddy 三部分全部通过,DEPLOY-003 才能
|
||||
标记完成并合并到 `main`。
|
||||
33
hy4/_kp/docs/development/CODING_RULES.md
Normal file
33
hy4/_kp/docs/development/CODING_RULES.md
Normal file
@ -0,0 +1,33 @@
|
||||
# 编码规则
|
||||
|
||||
- 基础设施 SDK 必须封装在 Provider 接口之后。
|
||||
- 不得硬编码凭据、租户 ID、Endpoint、模型名或 Collection 名。
|
||||
- 优先使用带类型的契约和高内聚模块。
|
||||
- 数据库 Migration 只允许向前演进。
|
||||
- 核心测试必须能在没有外部 SaaS、互联网或 GPU 的情况下运行。
|
||||
- 避免过早拆分微服务。
|
||||
|
||||
## 仓库协作
|
||||
|
||||
- 一个 Task ID 对应一个临时任务分支;默认只使用仓库根目录这一处 worktree。
|
||||
- 切换任务前必须提交修改或保存并核验命名 stash,禁止覆盖、reset 或删除用户工作。
|
||||
- 完成分支先推送、验证并合并 `main`,再按里程碑归档;`dev` 只承担开发环境发布。
|
||||
- 文档事实源、生命周期、分支/Tag 清理和恢复步骤统一遵循
|
||||
[`REPOSITORY_GOVERNANCE.md`](REPOSITORY_GOVERNANCE.md)。
|
||||
|
||||
## Python 依赖与工具链
|
||||
|
||||
- Python 基线由 `.python-version`、`pyproject.toml` 的 `requires-python`、Ruff/Mypy 目标版本、
|
||||
`tool.uv.required-version` 和 `uv.lock` 共同约束;调整任一版本时必须同步更新其他契约并运行
|
||||
`make toolchain-check`。
|
||||
- 日常安装、运行和质量门禁统一使用冻结的 `uv.lock`,不得由普通 `make` 入口隐式改写依赖解析。
|
||||
- 只有在有意修改 `pyproject.toml` 依赖后才运行 `make lock`,审查 `uv.lock` 差异并再次执行完整
|
||||
工程门禁。
|
||||
|
||||
## 持续集成
|
||||
|
||||
- 本地和托管 Runner 统一执行 `make ci`;具体门禁与外部操作排除项见
|
||||
[`CI_BASELINE.md`](CI_BASELINE.md)。
|
||||
- 默认 CI 必须保持确定性和 Provider-free,不读取真实凭据,不执行发布、部署、Benchmark 或目标
|
||||
环境验收。
|
||||
- API/MCP 真实进程回环 E2E 属于默认 CI;新增或改变用户入口时必须同步扩展该套件。
|
||||
91
hy4/_kp/docs/development/COMPILED_KNOWLEDGE.md
Normal file
91
hy4/_kp/docs/development/COMPILED_KNOWLEDGE.md
Normal file
@ -0,0 +1,91 @@
|
||||
# Compiled Knowledge 审核与发布
|
||||
|
||||
EKP-090 提供 L2 草稿纵向切片;EKP-091 在同一模型上补齐 Source 当前性、冲突证据、人工审核和发布;
|
||||
EKP-092 将仍可核验的 Published 内容接入正式检索。
|
||||
公开字段与状态码以 [`../api/API_CONTRACT.md`](../api/API_CONTRACT.md) 为准。
|
||||
|
||||
## 数据与权限边界
|
||||
|
||||
- 每条草稿属于一个 Organization/Knowledge Base,并至少引用一个真实 Chunk;来源行同时保存
|
||||
Document、DocumentVersion 与 Chunk ID,顺序稳定;
|
||||
- 创建要求目标 KB Write 和每个来源 Document Read;读取要求 KB Read 与全部来源 Document Read
|
||||
的交集,任一来源不可读时整条知识隐藏;
|
||||
- `permissions.mode=source_intersection` 描述请求时动态解析结果,不保存或伪造一份独立 ACL;
|
||||
- `draft` 可提交审核,`review_required` 只能由 Manage 身份发布或退回;`rejected|stale` 修订后回到
|
||||
新 Draft。Stale 修订可替换完整来源集合,所有变化使用 expected revision 防止覆盖并发编辑;
|
||||
- Source 新版本激活、被引用 Document 删除及 ACL 增删会自动标记 Stale。引用文档删除保留不可变
|
||||
Source/Version/Chunk 作为证据,但退出原始检索和普通 Document API;
|
||||
- 冲突是审核人明确记录的 Claim 与两个确切 Source 的配对,未解决冲突阻断发布。系统不自动推断、
|
||||
折中或让 AI 发布;只有 Published 内容允许进入检索。
|
||||
|
||||
## Published 检索
|
||||
|
||||
正式身份的 Raw Dense/Sparse/Weighted RRF 已先应用精确 Document Filter。Search Service 只从这些
|
||||
授权 Raw 候选引用到的 Compiled Knowledge 扩展 L2 候选,并以“所属 Knowledge Base 与全部来源
|
||||
Document 都在调用者 Read Scope”作为 SQL 前置条件;MCP 还继承 `mcp_enabled` Scope。没有规范身份 Filter 的历史公共入口
|
||||
不扩展 L2,避免把 Organization 内容带入兼容域。
|
||||
|
||||
扩展只接受 `published` 且具有发布时间、最后核验时间和非空来源的条目。触发分数采用最强命中
|
||||
Source 的 Fusion Score;L2 标题和正文与 Raw Chunk 一起进入同一次 Reranker,联合池按 Active
|
||||
Profile `hybrid_top_n` 截断,分数完全相同时 Raw 优先。最终回查再次核验每个 Source 的 Organization/
|
||||
KB/Document/Version/Chunk 归属、未删除、Ready Active 状态、文件名和调用者 Scope,异常时整次
|
||||
Search 失败关闭。
|
||||
|
||||
本实现没有把 L2 伪装成 Raw Chunk,也没有建立未版本化的第二套向量索引。Compiled Recall 继承其
|
||||
底层 Source 的 Raw 召回:只有至少一个底层 Source 进入 Hybrid 候选时才扩展 L2;后续如引入独立
|
||||
Compiled Embedding/Index,必须纳入版本化 Index Profile、Shadow Index 与 RAG 回归,不能绕过既有
|
||||
索引边界。
|
||||
|
||||
Compiled Search Result 返回完整已发布正文和底层来源数组;每个来源包含精确 Chunk 正文与位置,
|
||||
支持 Search Lab、REST 和 MCP 走同一来源链核对。非 Published 状态全部排除。
|
||||
|
||||
## LLMProvider
|
||||
|
||||
`LLM_PROVIDER=disabled` 是默认值,手工草稿不依赖外部模型。启用 OpenAI-compatible Chat
|
||||
Completions 时配置:
|
||||
|
||||
```dotenv
|
||||
LLM_PROVIDER=openai_compatible
|
||||
LLM_ENDPOINT=https://llm.example.test/v1/chat/completions
|
||||
LLM_API_KEY=<runtime-secret>
|
||||
LLM_MODEL=<model-id>
|
||||
LLM_PROCESSING_LOCATION=cloud
|
||||
LLM_TIMEOUT_SECONDS=60
|
||||
```
|
||||
|
||||
Secret 只能进入授权环境文件或运行时 Secret。Endpoint 必须为 HTTPS,本机回环模型可使用 HTTP。
|
||||
`LLM_PROCESSING_LOCATION` 是部署者对处理位置的明确声明;`local_only` KB 会在发送正文前拒绝 cloud
|
||||
Provider。Adapter 使用 Bearer Header、禁止重定向转发、限制响应大小,并只接受一个 JSON
|
||||
`title`/`content` 对象。确定性 Fake 只用于测试,不能作为模型质量或目标环境验收结果。
|
||||
`FakeLLMProvider` 在构造时执行同一能力校验,并在直接返回前执行公共 Compilation Result 校验,
|
||||
因此测试替身不能绕过生产服务消费的非空和长度边界。
|
||||
|
||||
## 操作路径
|
||||
|
||||
管理端打开 `/admin/compiled-knowledge`:选择知识库 → 选择 Ready 文档 → 勾选一个或多个规范 Chunk
|
||||
→ 填写类型、标题和正文后“保存手工草稿”,或使用“AI 生成草稿”。库存用来源证据轨道逐项显示
|
||||
文件名、Version、Chunk 顺序/章节、创建时间与 `current|outdated|deleted`;可从轨道进入 Inspector 核对仍可读取的
|
||||
来源。草稿提交审核前由服务端重新核验全部精确来源。
|
||||
|
||||
审核态允许记录两个确切来源之间的冲突、填写解决说明、发布或携带原因退回。未解决冲突时管理端
|
||||
不展示可执行的发布按钮,服务端也以 `409` 失败关闭。退回或 Stale 项可修订;Stale 修订要求重新
|
||||
选择来源,避免无意继续引用已删除或过期 Chunk。
|
||||
|
||||
AI 入口返回 `503 compilation provider is unavailable` 时,表示运行时保持默认 disabled 或配置未
|
||||
进入 API 进程;这不影响手工草稿。数据策略冲突返回 `409`。请求通过后返回 `202` 和持久化 Job,
|
||||
管理端轮询 `/compiled-knowledge/jobs/{job_id}`;成功后以 `compiled_knowledge_id` 读取 Draft,失败仅
|
||||
展示稳定脱敏错误。
|
||||
|
||||
`EKP-103` 已接入独立 `compilation_jobs`、有序 `compilation_job_sources` 和
|
||||
`compilation_attempts`。RabbitMQ 消息不携带 Prompt 或来源正文;Worker 重新解析提交者当前权限并
|
||||
复核来源与数据策略,使用租约 Fencing、有界指数退避和恢复扫描收敛重复/中断执行。Job 对生成结果
|
||||
的一对一唯一引用保证重复投递不创建重复草稿。真实外部 LLM 的质量、成本与目标环境可用性仍必须
|
||||
另行验收,不能用本地 Fake 测试替代。
|
||||
|
||||
## 验证
|
||||
|
||||
自动测试覆盖模型/迁移、Job/Attempt、重复投递、瞬时失败重试、租约恢复、执行时权限重验、来源
|
||||
范围、来源权限交集、revision 冲突、local-only 阻断、Adapter 输入输出、
|
||||
审核/冲突/发布/退回、Source 自动失效及真实 Uvicorn/TCP REST。每次改变页面或按钮还必须用真实浏览器
|
||||
完成身份进入、来源选择、创建/编辑、提交审核、冲突记录与解决、发布/退回和 Inspector 来源跳转,
|
||||
并确认控制台无错误。
|
||||
143
hy4/_kp/docs/development/CONTAINER_IMAGES.md
Normal file
143
hy4/_kp/docs/development/CONTAINER_IMAGES.md
Normal file
@ -0,0 +1,143 @@
|
||||
# 生产容器镜像说明
|
||||
|
||||
## 范围
|
||||
|
||||
DEPLOY-001 使用同一个锁定运行时构建三个独立镜像目标:
|
||||
|
||||
| Target | 默认镜像名 | 入口 | 端口 | 镜像健康检查 |
|
||||
|---|---|---|---|---|
|
||||
| `api` | `enterprise-knowledge-api` | Uvicorn + API/Admin | `8000` | `GET /healthz` |
|
||||
| `mcp` | `enterprise-knowledge-mcp` | `ekp-mcp` | `8766` | PID 1 存活 |
|
||||
| `worker` | `enterprise-knowledge-worker` | `ekp-worker` | 无 | PID 1 存活 |
|
||||
|
||||
Dockerfile 另有默认最终 `dev` target:它继承 `api`,只用于当前中央 dev 流水线,并自动加载已获
|
||||
明确授权的 `.env.dev`。显式构建上述三个 target 时不会包含该文件。
|
||||
|
||||
镜像没有内置数据库迁移、Helm、Ingress 或 Secret 对象;DEPLOY-002 已在独立 Chart 中提供这些
|
||||
编排能力。API 镜像包含 Alembic 和 migrations,由一次性 Migration Job 覆盖命令后复用。详见
|
||||
[`KUBERNETES_DEPLOYMENT.md`](KUBERNETES_DEPLOYMENT.md)。
|
||||
|
||||
`BUILD-001` 已将应用镜像定义收敛到仓库根 `Dockerfile`。Bake 与静态契约测试均引用该文件,
|
||||
`deploy/docker/` 只保存多架构 Bake 计划,不再维护第二份 Dockerfile。
|
||||
|
||||
## 可重复构建基线
|
||||
|
||||
- Python 基础镜像固定为 `3.12.14-slim-bookworm`,uv 固定为 `0.12.5`,两者同时锁定 manifest
|
||||
digest;
|
||||
- Python 依赖只使用提交的 `uv.lock`,以 `uv sync --frozen --no-dev --no-editable` 安装;
|
||||
- 三个目标共享相同依赖层,不复制不同版本的业务运行时;
|
||||
- 进程均以固定 UID/GID `10001:10001` 运行,不具备 root 权限;
|
||||
- `.dockerignore` 排除 `.env*`、Git、虚拟环境、缓存、本地存储、测试和开发产物,只对白名单
|
||||
`.env.dev` 例外供默认 `dev` target 显式复制;
|
||||
- Bake 计划同时声明 `linux/amd64` 与 `linux/arm64`,不下载或打包平台专属模型。
|
||||
|
||||
基础镜像 tag/digest 与应用依赖均已固定;镜像发布系统仍应保存本项目镜像的构建 digest,并让
|
||||
Helm 按 digest 部署。本项目镜像 digest 由实际 Registry 发布产生,不把某个环境的 Registry
|
||||
地址写入仓库。
|
||||
|
||||
## 构建命令
|
||||
|
||||
先确认 Docker Buildx builder 支持两个目标平台:
|
||||
|
||||
```bash
|
||||
make images-print
|
||||
```
|
||||
|
||||
构建全部目标:
|
||||
|
||||
```bash
|
||||
IMAGE_REGISTRY=registry.example.internal/knowledge \
|
||||
IMAGE_TAG=<immutable-release-tag> \
|
||||
make images
|
||||
```
|
||||
|
||||
只有发布流水线才执行推送:
|
||||
|
||||
```bash
|
||||
IMAGE_REGISTRY=registry.example.internal/knowledge \
|
||||
IMAGE_TAG=<immutable-release-tag> \
|
||||
make images-push
|
||||
```
|
||||
|
||||
不要把 Registry 密码放入变量或命令行;构建节点应提前通过其凭据助手登录。禁止使用
|
||||
`latest` 作为集群部署版本。
|
||||
|
||||
## 运行时配置
|
||||
|
||||
显式 `api`、`mcp`、`worker` target 只内置非敏感进程默认值:`APP_ENV=production`、容器监听
|
||||
地址、静态页面绝对路径和临时缓存目录;OceanBase、BOS、RabbitMQ、Embedding、Reranker 与
|
||||
MCP Token 必须在启动时注入。默认 `dev` target 是受控例外:它设置 `APP_ENV=development` 和
|
||||
固定 Profile,并以 UID `10001`、权限 `0400` 保存已授权 `.env.dev`;不得将该 target 发布到
|
||||
非 dev 环境,也不得把其中值写入构建日志、标签或镜像检查输出。
|
||||
|
||||
API 的 `/healthz` 只证明进程存活,不读取云端凭据或探测依赖。Kubernetes readiness、资源限制、
|
||||
只读根文件系统所需临时卷及依赖就绪策略由 DEPLOY-002 Chart 配置。MCP endpoint 需要认证,Worker
|
||||
没有 HTTP server,因此这两个镜像的 Docker 健康检查只验证主进程;业务可恢复性仍以持久化
|
||||
Job/Attempt 和 DEPLOY-003 故障验收为准。
|
||||
|
||||
## 本地单目标检查
|
||||
|
||||
需要加载到本地 Docker 时,可直接从根 Dockerfile 构建当前平台的一个 target:
|
||||
|
||||
```bash
|
||||
docker build --target api -f Dockerfile -t ekp-api:test .
|
||||
docker build --target mcp -f Dockerfile -t ekp-mcp:test .
|
||||
docker build --target worker -f Dockerfile -t ekp-worker:test .
|
||||
```
|
||||
|
||||
Worker target 保持非 root,并使用 `/tmp` 作为工作目录,使本地 `ekp-worker` 入口携带的 Celery
|
||||
Beat 可以写入调度状态。Helm 仍将 Worker 与单例 Scheduler 作为独立工作负载运行。
|
||||
|
||||
## `dev` 分支差异审查
|
||||
|
||||
`BUILD-001` 对 `decef23`、`a314dff`、`3cbe42b` 的历史结论如下:
|
||||
|
||||
- 不吸收基于 `ENV` 隐式选择并打包 `.env.<环境>` 的实现;环境配置继续由运行时环境或 Secret
|
||||
注入,镜像只复制明确的代码与构建输入;
|
||||
- 不吸收 API/Worker `combined` target;两者继续独立部署、扩缩容和恢复;
|
||||
- 不吸收 root 运行;所有应用 target 继续使用 UID/GID `10001:10001`。
|
||||
|
||||
启动时使用 `--env-file` 或编排系统的 Secret 引用,不在命令中展开敏感值。API/Admin smoke test
|
||||
至少检查 `/healthz`、`/admin/upload` 和 `/api/v1/openapi.json`;MCP 与 Worker 的真实依赖启动
|
||||
验收在具备 RabbitMQ 和目标 Provider 的受控环境执行。
|
||||
|
||||
`BUILD-002` 根据当前 dev 流水线实际约束增加受限的默认 `dev` target,不恢复 `ENV` 到任意
|
||||
`.env.<环境>` 的路径映射,也不改变显式生产 target、非 root 运行或 API/Worker 独立进程边界。
|
||||
`main@f03c48e` 合入 `dev@5c72218` 后,公网 `/healthz` 返回 `environment=development`,已证明
|
||||
中央流水线发布的是该默认 target 且固定 Profile 生效。随后业务 API 的 500 属于依赖 Schema
|
||||
验收问题,不是 dotenv 选择回退为失败。
|
||||
|
||||
中央 dev 模板只更新默认镜像和 GitOps tag,未提供独立 Migration 步骤。默认 `dev` target 因此
|
||||
使用 `python -m ekp_api.dev_startup`:严格要求 `APP_ENV=development` 和
|
||||
`EKP_SETTINGS_PROFILE=dev-image`,先执行 `alembic upgrade head`,成功后才启动 Uvicorn;失败则
|
||||
安全记录 `canonical_database` 分类,并以 `degraded` 健康响应暴露当前 Revision 和异常类型,
|
||||
使无目标集群日志权限时仍可诊断,业务入口继续失败关闭。该兼容路径只适用于当前中央 dev
|
||||
单副本部署;显式生产 `api` target 仍直接启动 Uvicorn,并继续由 Helm Migration Job 串行迁移。
|
||||
若 dev 扩为多副本,必须先接入独立 Migration Job,禁止并发依赖应用启动迁移。
|
||||
|
||||
## BUILD-001 CI 实证
|
||||
|
||||
Drone #27 已从 `codex/build-001@416a1c0` 成功检出任务提交,但任务分支只执行仓库名校验和通知,
|
||||
没有执行镜像构建,因此该记录不产生发布证据。完成合并和 `dev` 差异治理后,Drone #30 从
|
||||
`dev@974c276c` 执行 `image-build-push-dev` 和 `gitops-dev`,并确认:
|
||||
|
||||
- 构建入口为根 `Dockerfile`,基础镜像按 digest 拉取;
|
||||
- Admin 静态资源断言通过,最终用户为 `10001:10001`;
|
||||
- Kaniko 解析并构建 `mcp`、`worker`、`api` 阶段,Worker 使用 `/tmp` 工作目录;
|
||||
- 发布 tag 为 `974c276c-1788852979`,OCI digest 为
|
||||
`sha256:5d18409db5ef2fd310fc00c11b7e6bb7dd68ea16408c186f9f623b818b8a56f2`;
|
||||
- GitOps 更新成功,开发环境 `/healthz` 返回正常。
|
||||
|
||||
中央模板当前只推送根 Dockerfile 的默认最终 target,即 API 镜像;Bake 仍是三个独立 target 的
|
||||
权威多架构构建入口。若发布环境要求分别推送 MCP/Worker 镜像,应在后续流水线治理中显式增加
|
||||
target,而不能把单个默认镜像误写成三个独立镜像产物。
|
||||
|
||||
## DEPLOY-001 构建验收
|
||||
|
||||
2026-09-02 已使用 Buildx 分别构建并加载三个 `linux/arm64` target 和三个 `linux/amd64`
|
||||
target。API/Admin 以非 root 用户完成健康、静态页面和 OpenAPI smoke test;MCP/Worker 完成非
|
||||
root 安装入口检查。构建历史与容器文件系统检查未发现 `.env`、Git、测试文件或敏感配置。
|
||||
|
||||
这次验收只生成本地 `deploy001-*` 测试 tag,没有登录或推送 Registry,因此不产生可供集群
|
||||
引用的项目镜像 digest。DEPLOY-002 必须使用操作者指定的不可变 Registry tag/digest,不得把
|
||||
本地测试 tag 当作发布版本。
|
||||
38
hy4/_kp/docs/development/DEFINITION_OF_DONE.md
Normal file
38
hy4/_kp/docs/development/DEFINITION_OF_DONE.md
Normal file
@ -0,0 +1,38 @@
|
||||
# 完成定义
|
||||
|
||||
只有同时满足以下条件,任务才算完成:任务范围已经实现;架构不变量得到遵守;格式检查、Lint 和类型检查通过;相关测试通过;按需更新 Migration、契约和文档;未提交任何未经批准的 Secret;已记录所有经明确授权的凭据文件及其授权范围;检索变更包含 RAG 回归结果;已记录已知限制。
|
||||
|
||||
Task 的进度说明必须与仓库文档保持同一状态:每次向用户报告开发中、阻塞、恢复或完成等变化前,
|
||||
先同步 `PROGRESS.md`、`TASKS.md` 以及受影响的契约/开发文档。会话上下文接近需要切换时,先写明
|
||||
当前 Task、分支/工作树、已完成验证、未完成步骤和阻塞,再明确提醒用户切换会话。
|
||||
|
||||
日常开发默认只使用仓库根目录这一处工作区,在同一目录内按 Task 顺序切换任务分支。除非用户
|
||||
明确要求并行隔离或授权额外 worktree,不得为每个 Task 创建同级项目目录。存在未提交修改时,
|
||||
必须先提交或保存为带来源说明的命名 stash,并核对其覆盖的文件后才能切换分支;禁止使用 reset、
|
||||
覆盖文件或删除工作区来换取干净状态。
|
||||
|
||||
新增或改变任何 API、前端页面、按钮或用户入口时,必须启动真实应用进程,通过网络 HTTP 或真实
|
||||
浏览器完成用户路径测试,并贯穿对应应用服务及数据库/对象存储边界。必须记录入口、操作、结果和
|
||||
非敏感证据。单元测试、进程内 `TestClient`、静态资源字符串断言、Mock/Fake 结果或截图只能作为
|
||||
补充,不能单独满足端到端完成条件;确定性本地 Provider 可以用于可重复测试,但不得伪装成外部
|
||||
Provider 或目标环境验收。
|
||||
|
||||
对于异步入库或重试任务,相关测试还必须证明:Job/Attempt 状态可持久化、有界重试分类正确、
|
||||
重复投递具备幂等性、重试前执行清理、不可变 Source 得到保留,以及清理失败时能够正确终止。
|
||||
批量入库必须覆盖逐文件部分失败和配置限制。
|
||||
|
||||
对于 MCP 任务,契约测试必须覆盖 Tool 发现、身份传播、授权/错误边界、REST 与应用服务的一致性、
|
||||
基础设施信息脱敏,以及明确指定的 MCP Client 兼容性检查。
|
||||
|
||||
对于开发团队负责的集群交付任务,完成条件还包括:可重复构建的生产镜像;明确的环境配置契约;
|
||||
串行 Migration、Readiness/Liveness Probe、资源与 Pod 安全默认值等交付清单;开发侧测试通过;
|
||||
目标分支 CI/CD 成功;已批准的公网 Admin/API、异步 Worker 与检索烟测通过。实际 GitOps Values、
|
||||
Secret、Namespace、镜像拉取、集群副本/PDB、幂等安装、滚动重启、回滚、监控、告警和备份由运维
|
||||
负责时,应形成明确交接,但不作为开发 Task 的完成阻塞项。
|
||||
|
||||
文档治理、Benchmark、部署/运维和目标环境验收属于独立非开发任务类型。它们必须在 Task 与进度
|
||||
文档中同步状态、结论和风险,但除非开发功能本身依赖其产出,不得阻塞 V1 开发主队列。
|
||||
|
||||
只有明确包含 MCP 的任务才要求稳定 MCP Endpoint 与远程 WorkBuddy 验收;搁置状态不得写成完成。
|
||||
必须在各自责任范围内检查 Manifest、日志和镜像是否泄漏未经批准的凭据。公共知识交付不会完成或
|
||||
豁免 V1 用户/ACL 任务。
|
||||
50
hy4/_kp/docs/development/DENSE_RETRIEVAL.md
Normal file
50
hy4/_kp/docs/development/DENSE_RETRIEVAL.md
Normal file
@ -0,0 +1,50 @@
|
||||
# EKP-051 Dense 检索
|
||||
|
||||
`EKP-051` 提供独立、Provider-neutral 的 Dense Top-N 候选服务。它复用 `EKP-040~046` 的
|
||||
Embedding、Index Profile、VectorStore 与 Active/Shadow 生命周期,并在 `EKP-050` 的精确
|
||||
Document Scope 内检索;本任务不新增或改变 REST/MCP 字段、页面、按钮或用户入口。
|
||||
|
||||
## 执行顺序
|
||||
|
||||
```text
|
||||
规范化并校验 Query / Top-N
|
||||
→ 空授权范围直接返回
|
||||
→ Query Embedding
|
||||
→ 向量数量、维度、有限值和正范数校验
|
||||
→ Profile + Active Version + Document Scope 预过滤
|
||||
→ Cosine Top-N
|
||||
→ 候选身份、来源、分数、顺序和范围复验
|
||||
```
|
||||
|
||||
Query 去除首尾空白后必须非空、不含 NUL 且不超过 1600 字符;Top-N 必须是正整数且不得超过
|
||||
Active Profile 固化的 `dense_top_n`,布尔值不作为整数接受。诊断可以缩小候选集,不能在请求时
|
||||
扩大版本化 Pipeline。验证在调用外部 Embedding 前完成。授权 Document Scope 为空时不会调用
|
||||
Embedding 或 VectorStore,避免无结果请求仍向外部 Provider 披露 Query。
|
||||
|
||||
Embedding Provider 的模型和维度必须与 Index Profile 一致。Profile 配置了
|
||||
`embedding_query_instruction` 时,Query Provider 必须公开完全一致的 `query_instruction`;缺失、
|
||||
非法或漂移均在装配时失败关闭。文档 Embedding 不添加 Instruction,Qwen Query Provider 在自身
|
||||
Adapter 内按已固化 Instruction 格式化输入,公开检索服务不感知厂商格式。
|
||||
|
||||
Dense 候选保留 `index_record_id`、Document/Version/Chunk、文件名、Page Range、Section、正文、
|
||||
Chunk Index、Profile 和 Cosine Score。业务服务再次拒绝非 Tuple、超额、重复 Chunk/Index Row、
|
||||
无效来源字段、跨 Profile/ACL、非有限或超出 `[-1, 1]` 的分数,以及非降序结果,不能依赖某个
|
||||
Provider 恰好正确。
|
||||
|
||||
## 配置与验证
|
||||
|
||||
默认候选数来自 Active Index Profile 的 `dense_top_n`;当前运行配置 `DENSE_TOP_N` 和 Helm
|
||||
`config.retrieval.denseTopN` 收敛为 `1~64`。Profile 或模型/维度/Instruction 变化必须创建并构建
|
||||
新的 Shadow Profile,禁止原地改变 Active Index。
|
||||
|
||||
离线验证:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/unit/test_dense_retrieval.py -q
|
||||
uv run pytest tests/integration/test_dense_retrieval_integration.py -q
|
||||
uv run pytest tests/evaluation/test_dense_retrieval_regression.py -q
|
||||
```
|
||||
|
||||
集成测试覆盖真实 SQLite 规范关系表、Local ObjectStorage 与 Upload→Parse→Chunk→Embed→Index→
|
||||
Dense 链路;Golden Dataset 覆盖 40 条合成 Query 的 Recall/MRR 门禁。Fake 的精确 Cosine 不代表
|
||||
目标 OceanBase HNSW 的近似召回、延迟或容量,真实查询计划和性能由 `EKP-103`/现场验收独立记录。
|
||||
255
hy4/_kp/docs/development/DEPLOY_003_ACCEPTANCE.md
Normal file
255
hy4/_kp/docs/development/DEPLOY_003_ACCEPTANCE.md
Normal file
@ -0,0 +1,255 @@
|
||||
# DEPLOY-003 目标集群验收记录
|
||||
|
||||
**状态:** dev 500 已修复;API 身份凭据与集群资源级事项由独立环境队列接管;远程 MCP/WorkBuddy 搁置
|
||||
|
||||
**Task:** DEPLOY-003
|
||||
|
||||
**记录规则:** 只记录非敏感标识和聚合结果,不记录 Secret、业务正文或内部 Provider 响应。
|
||||
|
||||
## 2026-09-10 受控发现性部署
|
||||
|
||||
- 远程 `dev` 从 `974c276` 以普通 merge 推进到 `8141f23`,合并父提交为 `dev@974c276` 与
|
||||
`main@83f55fb`;推送前确认最终文件树与 `main` 完全一致,未强推或改写 `dev` 历史;
|
||||
- 推送前 Ruff Format/Lint、Mypy、27 个部署与 Migration 相关测试、Migration Contract、Helm
|
||||
Lint 和 Admin JavaScript 语法检查通过;Migration Contract 的唯一 Head 为 `20260909_0020`;
|
||||
- 推送后公网 `/healthz` 与 `/api/v1/openapi.json` 返回 `200`,Admin HTML 出现 `ekp-082`
|
||||
资源标记,浏览器实际加载了含知识库、身份与权限入口的新运营台,证明 API 默认镜像和 Admin
|
||||
静态资源已发生滚动更新;
|
||||
- 业务接口未通过运行验收:`/api/v1/administration/session` 与 `/api/v1/documents` 均返回
|
||||
`503`,公开错误为 `API providers are not fully configured`。这表明当前 GitOps Runtime
|
||||
Config/Secret 尚未满足新版本 Provider 装配要求,不能将本次部署记为业务可用;
|
||||
- 健康响应仍报告 `environment=production`,与 dev 域名语义不一致;HTTPS `/admin/` 还会生成
|
||||
指向 HTTP 的 `307`,均作为环境配置问题移交;
|
||||
- `/mcp` 继续返回 `404`。中央 Drone 模板不在本仓库内,且当前仍缺少 MCP/Worker 独立目标镜像
|
||||
的发布证据,因此本次只确认 API/Admin 更新,不确认 Migration Job、MCP 或 Worker 已更新;
|
||||
- 本次发现性部署只形成可复现的环境问题清单,不撤销下方历史闭环结论,也不阻塞 V1 开发主队列。
|
||||
|
||||
## 2026-09-10 BUILD-002 配置加载复验
|
||||
|
||||
- 用户明确授权仓库 `.env.dev` 的现有配置进入默认 dev 镜像及对应 Registry,授权范围仅为 dev;
|
||||
显式 `api`、`mcp`、`worker` 生产 target 继续不包含 dotenv;
|
||||
- `BUILD-002` 以 `main@f03c48e` 合入 `dev@5c72218` 并触发发布。新实例 `/healthz` 稳定返回
|
||||
`200` 和 `environment=development`,证明默认 dev target 已固定选择镜像内 `.env.dev`;
|
||||
- 原先两个业务入口的 `503 API providers are not fully configured` 已消失,说明 Provider 必填
|
||||
配置门禁已通过;`/api/v1/documents` 与 `/api/v1/administration/session` 当前改为返回 `500`;
|
||||
- 一次无敏感信息的聚合指标快照记录到 85 次 `/healthz` 200、7 次 `/documents` 500 和 1 次
|
||||
`/administration/session` 500;业务请求约在 `0.08~0.16s` 内失败,不符合外部 Provider
|
||||
连接超时特征;
|
||||
- 当前代码在旧可用 dev 基线之后新增规范库 Migration `0009~0020`、完整 Index Profile 字段、
|
||||
向量表版本字段和 Sparse 全文索引,而中央流水线仍无 Migration Job 运行证据。因此旧 Schema
|
||||
是当前最强嫌疑,但没有 Pod 异常日志和 dev 数据库只读检查结果,不能写成已证实根因;
|
||||
- 后续由独立部署队列先核对 Pod 的结构化错误事件、`alembic_version`、`index_profiles` 列集合
|
||||
以及向量表 DDL。任何真实 Migration、向量 DDL 或数据修复均属于外部写操作,必须在备份与
|
||||
明确授权后执行。该 500 不否定 BUILD-002 的配置选择验收,也不阻塞 V1 开发主队列。
|
||||
|
||||
## 2026-09-10 规范库故障定位与修复
|
||||
|
||||
- Drone #126 从 `dev@ed69886` 成功构建并更新 GitOps;新业务响应由原始 500 收敛为
|
||||
`503 API canonical database is unavailable`,且响应头为
|
||||
`X-EKP-Dependency: canonical-database`。真实异常发生在规范库初始化,向量 Schema 尚未成为
|
||||
当前首个故障点;
|
||||
- Drone #121/#126 的可见阶段均只有镜像构建和 GitOps tag 更新,没有执行 Alembic 或独立
|
||||
Migration Job。结合旧版本曾使用同一数据库连接、当前代码要求 Head `20260909_0020` 且失败
|
||||
快速发生,规范库未随代码升级是当前具备完整证据链的修复方向;实际 Head 仍需迁移启动器在
|
||||
目标网络内读取并由运行结果最终确认;
|
||||
- 中央 GitOps 仓库和实际运行集群不在当前授权 kubeconfig 范围内,不能直接创建 Job。为恢复
|
||||
dev,默认 dev 镜像增加受限启动器:只允许 `development + dev-image`,在 Uvicorn 监听前执行
|
||||
`alembic upgrade head`;失败时健康响应标记 `degraded` 并仅公开当前 Revision/异常类型,业务
|
||||
服务继续失败关闭。生产 target 和 Helm Migration Job 保持不变;
|
||||
- OceanBase 实测不支持 MySQL `GET_LOCK/RELEASE_LOCK`,因此未引入未经支持的命名锁。兼容启动器
|
||||
仅适用于中央模板当前的单 API 副本;扩容前必须把迁移恢复为独立、串行的 Migration Job。
|
||||
- Drone #132 发布安全诊断后,`/healthz` 报告 `migration_revision=20260902_0008`、
|
||||
`migration_error_type=CommandError`,直接证实规范库落后 12 个 Revision。该错误发生在 Alembic
|
||||
Config 定位阶段:非 editable 镜像中的模块位于 site-packages,不能从 `__file__` 反推 `/app`;
|
||||
Revision 保持 `0008`,未发生部分 DDL。启动器改为使用 Docker 明确保证的 `/app` 工作目录后
|
||||
再执行真实迁移。
|
||||
- Drone #135 从 `dev@8ec40b7` 发布路径修复后,Alembic 已真实执行,健康诊断推进到
|
||||
`migration_revision=20260908_0009`、`migration_error_type=OperationalError`。只读结构检查确认
|
||||
OceanBase 的非事务 DDL 已完成 `0010` 的 Version、Chunk 和 Job 回填,但旧 Job 唯一索引仍被
|
||||
外键使用;2 条 Job 均已正确回填且没有孤儿。先补普通外键索引,再按 `0010` 目标完成非空、
|
||||
外键和唯一约束,经结构校验后将 Revision 推进到 `0010`;
|
||||
- `0011~0013` 正常升级。OceanBase `4.3.5.4` 在 `0014` 拒绝复合自引用 Folder 外键,即使补齐
|
||||
子表索引或使用会话级 `foreign_key_checks=0` 仍返回 `1215`。目标表为空,因此先创建完整列、
|
||||
Check、Unique 和 Organization/KB 外键;自引用兼容为 `parent_folder_id → id ON DELETE CASCADE`。
|
||||
现有 Folder 创建服务仍按 Organization + KB 查询父目录,跨域父目录会在应用写入前失败;数据库
|
||||
复合同域自引用约束须在 OceanBase 升级并实测支持后恢复。结构校验后 `0015~0020` 全部成功;
|
||||
- 向量派生表共有 73 条既有记录,仍缺少仓库已定义的 `002` 版本/激活列及 `003` NGRAM 全文索引。
|
||||
先验证 `document_id` 全部有效,再按既有 DDL 精确回填并建索引。真实 `SHOW CREATE TABLE` 使用
|
||||
`INT(11) DEFAULT NULL` 表示与 `INT NULL` 等价的列,Adapter 原校验未归一化该厂商渲染,且
|
||||
`OceanBaseConfigurationError` 未进入安全 503 分类;两项均已补充回归测试并修复;
|
||||
- Drone #138 从 `dev@f4be521` 成功构建并更新 GitOps。滚动后 `/healthz` 返回 `200`、
|
||||
`migration_status=completed`、`migration_revision=20260909_0020`;本地使用当前 `main` 和授权配置
|
||||
对真实 dev 依赖执行只读服务组合成功,并列出 2 条文档;
|
||||
- 公网 `/api/v1/documents` 与 `/api/v1/administration/session` 已由原始 `500` 收敛为稳定的
|
||||
`401 authentication required`。该轮仓库 `.env`、`.env.dev` 尚未配置
|
||||
`API_IDENTITY_CREDENTIALS`,因此没有伪造认证成功验收;该身份配置缺口与本次
|
||||
canonical/vector 500 根因相互独立,后续处置见下节。
|
||||
|
||||
## 2026-09-10 dev 首个 API 身份
|
||||
|
||||
- 用户明确授权为 dev 生成 API Token、写入仓库 `.env.dev` 并创建首个管理员;授权仅覆盖默认
|
||||
dev 镜像及对应 Registry,不扩大到生产环境、MCP 或其他文件;
|
||||
- 本地生成 64 位十六进制随机 Token 和独立 Organization/User UUID,不在命令、日志、文档或
|
||||
回复中回显。Credential 使用 `user_token`、`client_id=admin-console`、
|
||||
`subject=dev-platform-admin`,并声明 `knowledge:read/write/search/manage`;
|
||||
- Bootstrap 已在真实 dev 规范库创建 `D-Robotics` Organization 和 Active
|
||||
`Platform Administrator` User。随后通过真实 Uvicorn/TCP、真实 canonical/vector 依赖和生成
|
||||
Token 验证 `/api/v1/administration/session` 与 `/api/v1/documents` 均为 `200`;当前管理员尚无
|
||||
Knowledge Base/ACL,文档列表按权限失败关闭为空,不把两条旧公共文档自动归属给新身份。
|
||||
- Drone #143 从 `dev@a172524` 成功构建并更新 GitOps。公网携带生成 Token 后,`/healthz`、
|
||||
`/api/v1/administration/session`、`/api/v1/documents` 和 `/api/v1/knowledge-bases` 均返回 `200`,
|
||||
Revision 为 `20260909_0020`;Document/Knowledge Base 数量均为 0。相同 Document 请求不携带
|
||||
Token 时仍返回 `401`,证明认证成功路径和匿名失败关闭路径均已真实验证。
|
||||
|
||||
## 2026-09-08 开发侧闭环结论
|
||||
|
||||
- `main` 与 `dev` 已推进到开发交付基线;Drone #19 的镜像构建、推送和 `gitops-dev` 均成功;
|
||||
- 公网 `knowledge-dev.d-robotics.cc` 的 `/`、Admin 页面与静态资源、`/healthz`、`/metrics`、
|
||||
Document API 和 Search API 均正常;
|
||||
- 远程两个文档均为 `ready`,真实搜索返回 Top5,证明 API、OceanBase、BOS、Embedding、Vector、
|
||||
Reranker 和 Worker 主链路可用;
|
||||
- `/mcp` 当前返回 `404`,远程 MCP/WorkBuddy 按用户决定搁置,不计入本次开发闭环,也不得标记
|
||||
为完整 V1 MCP 已完成;
|
||||
- 实际 GitOps Values/Secret、Namespace、Migration 运行记录、镜像拉取、工作负载副本/PDB、
|
||||
滚动/回滚、监控、告警和备份转由运维确认。下方未勾选项目作为运维交接记录保留,不再阻塞
|
||||
开发 Task;
|
||||
- 双 Dockerfile 构建来源问题转为独立开发维护任务 `BUILD-001`。
|
||||
|
||||
闭环后核对发现远程 `dev` 又新增 `decef23`、`a314dff`、`3cbe42b` 三个未进入 `main` 的部署
|
||||
提交。它们支撑当前公网运行,但包含 API/Worker 同容器监管和 root 运行,不作为已接受架构或
|
||||
开发基线,也不在本记录中追认。差异审查和安全收敛由 `BUILD-001` 在下一次开发代码发布前完成;
|
||||
当前运维人员不得假定后续 `main → dev` 推送会自动保留这些分支独有改动。
|
||||
|
||||
## 环境信息
|
||||
|
||||
- Kubernetes context:`2c1284bae51b4bf3a2e623557844c6d6@kubernetes`
|
||||
- 集群版本/节点:Kubernetes `v1.34.2`,两个 `linux/amd64` 节点均 Ready
|
||||
- Namespace / Release:等待 `dev` 流水线/Argo CD 实证,不沿用旧 `d-wiki-service` 假设
|
||||
- API HTTPS Host:`knowledge-dev.d-robotics.cc`
|
||||
- MCP HTTPS Host:`knowledge-dev.d-robotics.cc/mcp`
|
||||
- 镜像仓库:`gitea.d-robotics.cc/d-cloud/knowledge-platform`
|
||||
- 代码仓库:`d-cloud/knowledge-platform` 的 DEPLOY-003 分支已持续推送
|
||||
- 首次 `dev` 触发提交:`3776bbf`;最新 dev 500 修复基线:`f4be521`
|
||||
- 验收时间与操作者:2026-09-03,Codex 只读预检
|
||||
|
||||
目标 Namespace 已存在 `d-wiki-http-service` 与 `d-wiki-service` 两个 Deployment,以及同名
|
||||
ClusterIP/NodePort Service,运行的镜像为既有 `d-robotics/d-wiki-service:1.0.13`,不是本项目
|
||||
镜像。集群没有列出 IngressClass 或 Ingress;正式域名看起来由集群外入口转发至现有 NodePort。
|
||||
最初提供的 kubeconfig 身份只有 `get/list/watch`。后续 `agent.conf` 已确认可创建 Namespace,
|
||||
并对 Deployment、Service、ConfigMap、Secret、ServiceAccount、Job、PodDisruptionBudget 和
|
||||
Ingress 具有 create/patch/delete 权限。为复用现有 ImagePullSecret 并进行最小中断回切,新
|
||||
Release 计划部署到已有 `d-wiki-service` Namespace;入口和运行时依赖未就绪前仍未执行 Helm
|
||||
安装或故障注入。
|
||||
|
||||
2026-09-03 曾确认复用 `d-wiki.d-robotics.cc` 并直接替换现有服务;随后目标更新为独立开发域名
|
||||
`knowledge-dev.d-robotics.cc`,远程 `dev` 分支推送会触发流水线部署。后者是当前有效方案,原
|
||||
NodePort 直接替换方案停止执行,旧 `d-wiki` Service/Deployment 不得修改。正式路径仍保持
|
||||
`/mcp` 进入独立 MCP Gateway,其余路径进入 API/Admin,不合并两个应用进程。
|
||||
|
||||
仓库已为无 Ingress 环境增加可选双副本 Edge Proxy 和 NodePort Service。Edge 与 Ingress 互斥,
|
||||
包含零不可用滚动策略、PDB、非 root/只读根文件系统、关联 ID、安全 JSON 访问日志和 MCP 流式
|
||||
代理配置。`nginx:1.29.1-alpine` 的 manifest digest 已固定;实际部署时仍需镜像到批准的目标
|
||||
Registry。Edge 模式通过 Helm 单测、16 个资源的 strict Kubernetes Schema 校验,以及 UID/GID
|
||||
`10001` + 只读根文件系统的容器启动烟测。
|
||||
|
||||
最终 Edge 清单使用临时 NodePort `30083` 完成目标 Kubernetes API server-side dry-run,16 个资源
|
||||
全部通过且未创建对象;`30082` 已由 `agent-hub/data-dashboard-web` 占用,已明确排除。集群内没有
|
||||
监听 `5671/5672/15672` 的 Service,也没有 RabbitMQ Deployment/StatefulSet,不能复用现存 Broker。
|
||||
|
||||
用户提供的 `d-cloud/knowledge-platform.git` 已确认是 Git 代码仓库,不是可直接交给 Helm 的镜像
|
||||
坐标;同域 `/v2/` 端点要求鉴权并支持 OCI。初次核对时本机 Docker 未登录,Git Keychain 凭据
|
||||
没有在未授权状态下复用于容器 Registry。获得授权后已用实际 push 验证单仓库镜像坐标
|
||||
`gitea.d-robotics.cc/d-cloud/knowledge-platform` 及写权限,API/MCP/Worker/Edge 使用独立不可变 tag。
|
||||
|
||||
获得用户明确授权后,使用本机 Gitea Keychain 凭据临时登录同域 OCI Registry,以下
|
||||
`linux/amd64` 镜像均已成功推送,临时 Docker 配置随后已删除:
|
||||
|
||||
- `api-c3529ce`:`sha256:133678975e232add5c0e84aa3f67a0d05a02608161c1c9ec9691b738598c7bd8`
|
||||
- `mcp-c3529ce`:`sha256:6bd6e142832063238561264d73b06f8ab2ad7d4ce1372933ad8d7be992449ec7`
|
||||
- `worker-c3529ce`:`sha256:e2036b9c8c0bf119417a27e5615721a36d7bd71e9dd2e3a2acd6a8d505d03e18`
|
||||
- `edge-nginx-1.29.1`:`sha256:77d740efa8f9c4753f2a7212d8422b8c77411681971f400ea03d07fe38476cac`
|
||||
|
||||
远程 `main` 随后新增 `.drone.yml` 与根 `Dockerfile`,并由用户指定为最新配置依据;本任务分支
|
||||
已合入这两个提交。`.drone.yml` 加载仓库外的中央模板 `argocd-python.jsonnet`,且设置字符串
|
||||
`DisableDockerfile: "true"`;仓库内无法静态确认该参数的真实语义。同时,现有 Bake 与镜像静态
|
||||
测试仍指向 `deploy/docker/Dockerfile`,而上述三个应用镜像正是由该旧路径构建。因此三个应用
|
||||
digest 降级为候选记录,在完成 Drone 模板实证、根 Dockerfile 单一来源和 CI 重新构建前,不得
|
||||
用于正式 Values 或现场切流。Edge digest 不依赖应用 Dockerfile,但仍须完成集群拉取验证。
|
||||
|
||||
重新规划后的严格门禁记录在
|
||||
[`DEPLOY_003_REPLAN.md`](../plans/DEPLOY_003_REPLAN.md):D3-G0 Drone 模板实证、D3-G1
|
||||
Dockerfile 收敛、D3-G2 CI 新镜像、D3-G3 集群拉取、D3-G4 外部依赖、D3-G5 开发域名部署、D3-G6
|
||||
可用性、D3-G7 发布回滚、D3-G8 远程验收。D3-G5 已调整为开发域名流水线部署而非临时入口;
|
||||
下面的现场复选项只有在前四个门禁完成后才开始执行。
|
||||
|
||||
2026-09-03 已在确认远程 `dev` 仍位于 `ceca075` 且为当前提交祖先后,将 `3776bbf` 以快进方式
|
||||
推送至 `dev`,远程接受该引用更新。推送后立即及约 30 秒后,从最终用户路径探测
|
||||
`knowledge-dev.d-robotics.cc` 的 `/healthz`、`/admin/`、`/mcp`,两次均在 DNS 阶段失败,HTTP
|
||||
状态为 `000`;因此尚不能验证 TLS 或应用路由。使用现有 `agent.conf` 只读检查 Deployment、Pod、
|
||||
Service 和 Ingress,也没有发现本项目新资源,旧 `d-wiki` 工作负载保持原状。该观察可能表示
|
||||
流水线尚未完成、流水线失败、DNS 尚未配置,或流水线部署到另一个集群;在取得实际 Drone 构建
|
||||
状态、日志与 Argo CD 目标前不能区分,D3-G0 保持未完成。
|
||||
|
||||
2026-09-07 用户确认该开发域名只在公网环境解析。因此本机或受限执行环境 DNS 失败不再作为
|
||||
流水线失败或服务不可用的结论;HTTPS、根路径、Admin、API 与 MCP 必须由公网探针或最终用户
|
||||
网络路径验收。当前分支相对首次触发的 `dev@3776bbf` 增加 7 个提交,包含持久开发配置、根路径
|
||||
重定向以及真实 OceanBase 单文件上传响应修复;须合并 `main` 后重新推进 `dev`,以新流水线产物
|
||||
作为后续门禁依据。
|
||||
|
||||
运行字段复核确认受控提交的 `.env.dev` 已覆盖 OceanBase、BOS、Embedding、Reranker、Vector
|
||||
Store、共享外部 RabbitMQ 和 Admin/MCP 地址。共享 RabbitMQ 已在本地完成认证及真实异步入库,
|
||||
不再属于“Broker 未提供”阻塞项;远程 Pod 到该 Broker 的网络与认证仍待验证。`MCP_AUTH_TOKEN`
|
||||
仍为占位值,必须在远程启用前替换为固定 token,并与 WorkBuddy 客户端一致。BUILD-002 后,
|
||||
`.dockerignore` 排除 `.env*` 并仅对白名单 `.env.dev` 例外;默认最终 `dev` target 固定设置
|
||||
`EKP_SETTINGS_PROFILE=dev-image` 并以 `0400` 权限复制该文件。真实发布后的
|
||||
`environment=development` 已证明默认 dev 镜像自动加载生效。显式生产 target 仍必须从实际
|
||||
GitOps `values.yaml` 或渲染 Deployment 证明 Runtime Secret/ConfigMap 注入,不得复用此例外。
|
||||
|
||||
创建 Namespace 级 `knowledge-platform-registry` ImagePullSecret 属于凭据长期存储,尚未获得单独
|
||||
明确授权,因此未创建,集群拉取烟测也尚未执行。
|
||||
|
||||
目标集群没有 StorageClass,现有 PersistentVolume 全部 Bound,因此不在该集群补建 RabbitMQ。
|
||||
项目已采用用户提供的共享外部 RabbitMQ;本地认证和异步任务执行成功,目标 Pod 的网络连通、
|
||||
凭据注入、队列隔离与恢复行为仍须在 D3-G4~D3-G6 验证。
|
||||
|
||||
## 运维接管的自动化验收
|
||||
|
||||
以下项目由运维人员确认,保留未勾选状态表示开发侧没有替运维作出结论;它们不再阻塞
|
||||
`DEPLOY-003` 开发侧闭环。
|
||||
|
||||
- [ ] 同一 Values 连续两次 `upgrade --install` 成功
|
||||
- [ ] Migration Hook 成功且失败保护已核对
|
||||
- [ ] API/MCP/Edge 各至少两个 Ready/Available 副本
|
||||
- [ ] API/MCP/Edge PodDisruptionBudget 生效
|
||||
- [ ] Admin、健康检查、`/metrics` 和 MCP 使用有效 HTTPS
|
||||
- [ ] API/MCP/Edge 滚动重启期间连续健康探测无失败
|
||||
- [ ] Worker 非优雅中断后同一 Job 自动恢复成功
|
||||
- [ ] 验收合成文档已清理
|
||||
- [ ] Manifest、镜像与验收日志凭据扫描为 0
|
||||
|
||||
仓库与清单预检:Ruff Format/Lint、Mypy、Bash 语法、三种 Helm lint、Alembic Head/离线 SQL、
|
||||
Unit + Integration `202 passed`、RAG Evaluation `5 passed`;目标 API Server dry-run 16 个资源通过。
|
||||
真实安装、升级与故障注入的执行结果仍待现场依赖就绪后记录。
|
||||
|
||||
## 远程 MCP / WorkBuddy(搁置)
|
||||
|
||||
- [ ] 官方 MCP Client 只发现当前五个批准的 P0 Tool
|
||||
- [ ] 官方 MCP Client 完成一次真实、有来源的检索与 Source/Document/Section/Knowledge Base 读取链
|
||||
- [ ] 最终用户网络路径上的 WorkBuddy 连接成功
|
||||
- [ ] WorkBuddy 只发现当前五个批准 Tool 并完成真实检索与来源读取链
|
||||
|
||||
客户端版本、网络区域和非敏感结果:按用户决定搁置,恢复任务后再执行。
|
||||
|
||||
## 运维接管的运行保障
|
||||
|
||||
- [ ] 日志采集可按 `request_id` / `job_id` 定位
|
||||
- [ ] 最小指标已被环境监控系统抓取
|
||||
- [ ] Ready 副本、5xx、耗时、Pod 重启和 Job 失败告警责任人已确认
|
||||
- [ ] OceanBase/BOS/RabbitMQ 备份责任人、周期和恢复顺序已确认
|
||||
|
||||
## 结论
|
||||
|
||||
开发侧实现、工程门禁、`dev` 流水线和公网 Admin/API/Worker/Search 验证均已完成,`DEPLOY-003`
|
||||
按调整后的开发责任范围闭环。未勾选的集群资源、滚动/回滚、监控和备份项目由运维接管,不表示
|
||||
开发任务失败,也不得伪装为开发已验证。远程 MCP/WorkBuddy 继续搁置;完整 V1 MCP 目标不变。
|
||||
构建来源收敛转入 `BUILD-001`,不再回写本验收记录的历史门禁。
|
||||
58
hy4/_kp/docs/development/DOCLING_SETUP.md
Normal file
58
hy4/_kp/docs/development/DOCLING_SETUP.md
Normal file
@ -0,0 +1,58 @@
|
||||
# Docling Parser Adapter
|
||||
|
||||
本文记录 `EKP-026` 的本地 Docling Provider 边界。Adapter 已通过真实格式样例验证,`EKP-028`
|
||||
已建立规范解析模型,`EKP-029` 已在 `structure_v2` 中消费表格结构并接入 P0 上传/Worker 路由。
|
||||
|
||||
## 版本与依赖
|
||||
|
||||
- 依赖:`docling-slim[convert-core,format-html,format-office,format-pdf]>=2.126,<3`;
|
||||
- 当前锁定版本:`docling-slim 2.126.0`;
|
||||
- `format-pdf` 是 Docling `DocumentConverter` 当前导入链所需的兼容依赖,不表示本 Adapter 接管
|
||||
PDF;PDF 仍由轻量 Parser 或显式 MinerU 路由处理;
|
||||
- 未安装 `models-local`,因此不会把 Torch、Transformers 或 Docling 本地 OCR 模型栈带入应用镜像。
|
||||
|
||||
上游依据:
|
||||
|
||||
- [Docling 支持格式](https://docling-project.github.io/docling/usage/supported_formats/)
|
||||
- [DocumentConverter 与 DocumentStream](https://github.com/docling-project/docling/blob/main/docs/v2.md)
|
||||
- [DoclingDocument 结构和遍历](https://docling-project.github.io/docling/concepts/docling_document/)
|
||||
- [docling-slim 可选依赖](https://github.com/docling-project/docling/blob/main/packages/docling-slim/README.md)
|
||||
|
||||
## 当前 Adapter 边界
|
||||
|
||||
`DoclingParser` 通过现有 `Parser` 协议接收内存流,支持:
|
||||
|
||||
- DOCX;
|
||||
- PPTX;
|
||||
- XLSX;
|
||||
- CSV;
|
||||
- HTML/HTM。
|
||||
|
||||
输入受 `MAX_UPLOAD_BYTES` 约束,不写临时文件。HTML 的远程资源、本地文件和图片抓取均显式
|
||||
关闭,避免文档内容触发外部或宿主文件访问。Docling 实现异常会统一转换为不包含底层详情的
|
||||
`DoclingConversionError`;该本地格式错误属于永久解析失败,不进入瞬时重试。
|
||||
|
||||
Adapter 按 Docling body reading order 映射 Title、Heading、Paragraph、List、Table 和
|
||||
Image Reference,并保留一基 Page。XLSX/CSV 进一步保留 Sheet、Header、Cell 与一基闭区间行列
|
||||
坐标;可读 Markdown Table 只是兼容内容,结构事实以 schema v4 Cell Grid 为准。
|
||||
|
||||
## 已完成启用顺序
|
||||
|
||||
1. `EKP-026`(已完成):稳定 Docling Adapter 与上游结构读取边界;
|
||||
2. `EKP-028`(已完成):扩展版本化规范解析文档模型,覆盖 PRD 要求的节点及来源位置;
|
||||
3. `EKP-029`(已完成):让结构感知与 Spreadsheet Chunk 消费规范结构;
|
||||
4. PPTX、XLSX、CSV 和 HTML/HTM 已加入 `structure_v2` 上传白名单并路由到 Docling;DOCX
|
||||
保持轻量 Parser 或显式 MinerU 路由。
|
||||
|
||||
数据库仍为 `structure_v1` Active Profile 时,新格式保持关闭;完成 v2 Shadow Build/Verify/Activate
|
||||
并重启运行进程后才启用,详见 [`STRUCTURED_CHUNKING.md`](STRUCTURED_CHUNKING.md)。不得绕过
|
||||
schema v4 Cell Grid,将 Markdown 表格文本直接作为电子表格 Chunk 完成证据。
|
||||
|
||||
## 验证
|
||||
|
||||
离线测试使用内存生成的 DOCX、PPTX、XLSX,以及 CSV、HTML 样例调用真实 Docling 转换器,同时
|
||||
覆盖读取顺序、章节路径、表格可读中间文本、HTML 抓取禁用、大小边界和安全错误信息。运行:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/unit/test_docling_parser.py tests/unit/test_provider_wiring.py -q
|
||||
```
|
||||
47
hy4/_kp/docs/development/EMBEDDING_PROVIDER.md
Normal file
47
hy4/_kp/docs/development/EMBEDDING_PROVIDER.md
Normal file
@ -0,0 +1,47 @@
|
||||
# Embedding Provider 抽象
|
||||
|
||||
本文记录 `EKP-040` 的 Provider-neutral Embedding 契约。千帆和 Qwen3 的具体运行配置分别见
|
||||
[`QIANFAN_EMBEDDING_SETUP.md`](QIANFAN_EMBEDDING_SETUP.md) 与
|
||||
[`QWEN_EMBEDDING_SETUP.md`](QWEN_EMBEDDING_SETUP.md),不进入 REST/MCP 契约。
|
||||
|
||||
## 运行时能力
|
||||
|
||||
`EmbeddingProvider` 是可运行时检查的结构协议,固定公开 `model_name`、`dimensions`、
|
||||
`max_batch_size` 和按输入顺序返回向量的 `embed()`。装配到索引或查询服务前必须满足:
|
||||
|
||||
- 模型名为无首尾空白的非空稳定字符串;
|
||||
- 向量维度和单次最大批量均为正整数,布尔值不作为整数接受;
|
||||
- Provider 模型名、维度与所绑定的版本化 Index Profile 严格一致。
|
||||
|
||||
Index Profile 同时记录模型最大输入 Token、可选 Query Instruction、Float32 数据类型及由维度推导
|
||||
的原始存储需求;Dense 装配要求 Query Provider 的可选 `query_instruction` 与 Profile 精确一致,
|
||||
缺失或漂移时失败关闭。完整配置边界见 [`INDEX_PROFILE.md`](INDEX_PROFILE.md)。
|
||||
|
||||
Credential、Endpoint、厂商请求/响应和 SDK 类型只存在于具体 Adapter;Domain、REST、MCP 与向量
|
||||
存储只依赖该中立协议。
|
||||
|
||||
## 向量输出门禁
|
||||
|
||||
Provider 输出在进入可重建索引或 Dense Query 前统一校验:外层与单向量均为不可变 Tuple,数量与
|
||||
输入一致,维度与 Profile 一致,全部值为有限数值,且每个向量的余弦范数为正。任一条件不满足均
|
||||
失败关闭,不写入部分索引,也不向 VectorStore 发起查询。
|
||||
|
||||
空批量返回空 Tuple;非空输入必须保持顺序。具体 Provider 可以附加输入长度、批量、Query
|
||||
Instruction 和安全 Endpoint 限制,但不能弱化上述公共边界。
|
||||
|
||||
## 确定性 Fake
|
||||
|
||||
`FakeEmbeddingProvider` 仅用于单元测试、集成测试和离线开发验证。它将模型身份与文本共同写入
|
||||
分块 SHA-256 输入,扩展为指定维度并执行 L2 归一化,因此具有稳定、非零、非循环和输入顺序可
|
||||
验证的输出;它不联网、不读取 Credential,也不模拟真实模型语义质量。
|
||||
|
||||
Fake 结果不得用于 Embedding 选型、生产质量、延迟、吞吐、成本或 GPU 结论。`EKP-101` 仍必须在
|
||||
获得真实 Qwen Endpoint/GPU 环境后使用同一 Golden Dataset 执行真实候选 Benchmark。
|
||||
Fake 的非法批量、空文本和输入类型统一返回 `EmbeddingInputError`,与生产 Provider 的永久请求
|
||||
错误分类一致,避免异步 Job 把确定性测试输入错误误判为暂时故障。
|
||||
|
||||
## 验证边界
|
||||
|
||||
单元与集成测试覆盖运行时协议、非法能力、稳定性、顺序、跨摘要块维度、单位范数、批量/空文本
|
||||
错误,以及索引和 Dense Query 对错误输出的失败关闭。本任务不新增或改变 API、前端页面、按钮或
|
||||
用户入口,因此不触发新的真实运行态 E2E;检索核心变更仍执行完整 RAG 回归评估。
|
||||
30
hy4/_kp/docs/development/FAKE_PROVIDERS.md
Normal file
30
hy4/_kp/docs/development/FAKE_PROVIDERS.md
Normal file
@ -0,0 +1,30 @@
|
||||
# Fake Provider 框架
|
||||
|
||||
`EKP-007` 的 Fake 仅用于确定性测试和离线开发,不是运行时降级方案。当前公共替身包括:
|
||||
|
||||
| Provider 边界 | 确定性实现 | 共享契约 |
|
||||
|---|---|---|
|
||||
| Embedding | `FakeEmbeddingProvider` | 能力、请求错误分类、向量数量/维度/有限非零 Tuple |
|
||||
| VectorStore | `FakeVectorStore` | Row/Query、Profile/Version/ACL 预过滤、原子写入与稳定排序 |
|
||||
| Reranker | `FakeRerankerProvider` | Query/候选/Top-N、索引映射和有限降序分数 |
|
||||
| LLM | `FakeLLMProvider` | 模型/处理位置能力和有界非空 Compilation Result |
|
||||
|
||||
Parser 使用真实的轻量级/Docling 本地解析实现,对象存储使用真实的 `LocalObjectStorage` 文件语义,
|
||||
canonical 数据测试使用 SQLite 执行同一 SQLAlchemy 模型;这些组件不伪装成外部 Provider。
|
||||
|
||||
## 强制边界
|
||||
|
||||
- 公共 Fake 必须实现对应运行时 Protocol,并复用 Provider-neutral 输入/输出验证器和错误分类。
|
||||
- Fake 实现不得读取环境配置或凭据,不得导入网络客户端,也不得连接 SaaS、数据库、GPU 或模型。
|
||||
- 同一输入和显式模型身份必须得到稳定结果;默认模型名必须明确以 `fake-` 标识。
|
||||
- 生产 `apps/api/src/ekp_api/providers.py` 不得导入或回退到任何 `Fake*`。真实配置缺失时继续失败关闭。
|
||||
- 测试专用 HTTP Client、故障注入类可以留在具体测试文件中,不作为公共 Provider 实现导出。
|
||||
|
||||
上述边界由 `tests/unit/test_fake_provider_framework.py` 跨包检查;各 Provider 自身的详细契约仍由对应
|
||||
单元/集成测试负责。
|
||||
|
||||
## 结果使用限制
|
||||
|
||||
Fake 可以证明协议、业务编排、ACL、来源追溯、幂等性和错误处理的可重复行为;不得用于模型选型、
|
||||
生产质量、真实 Recall、延迟、吞吐、容量、成本或 GPU 结论。`EKP-101~102` 仍须等待真实候选环境,
|
||||
任何确定性测试数字都不能替代 Benchmark 报告。
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user