2026-09-02 16:40:00 +08:00
2026-08-31 20:53:53 +08:00

Enterprise Knowledge Platform

Enterprise internal knowledge infrastructure.

Core capabilities

  • Multi-format document ingestion
  • OCR and document parsing
  • Structure-aware chunking
  • Multiple embedding providers
  • Hybrid retrieval and reranking
  • ACL-aware search
  • Knowledge Compilation
  • REST API
  • MCP Server
  • Admin Console

Architecture baseline

  • API First + Admin Console
  • Modular Monolith + async workers + MCP gateway
  • OceanBase MySQL relational tables + object storage as canonical source of truth
  • Co-located OceanBase Vector tables as rebuildable retrieval indexes behind an adapter
  • Baidu BOS as target object storage; local storage remains for tests/development
  • Embedding / reranker / parser / LLM providers are replaceable
  • Raw source is immutable
  • Compiled Knowledge is derived and traceable
  • ACL is enforced before or during retrieval, never after Top-K

Start here

  1. Read AGENTS.md
  2. Read CODEX_START_HERE.md
  3. Read docs/product/PRD.md
  4. Read docs/architecture/ARCHITECTURE.md
  5. Read docs/plans/V1_EXECUTION_PLAN.md
  6. For the current milestone, read docs/demo/DEMO_SCOPE.md and pick exactly one DEMO-* task. The V1 tasks in docs/plans/TASKS.md remain the deferred product backlog.

Demo development

The current implementation target is the retrieval demo defined in docs/demo/DEMO_SCOPE.md.

Prerequisites:

  • uv
  • Docker (for optional local integration dependencies)

Install dependencies and run the API:

cp .env.example .env
make install
make run

Run the independent authenticated MCP Gateway after injecting a dedicated runtime token of at least 32 characters:

export MCP_AUTH_TOKEN="<runtime-secret-with-at-least-32-characters>"
make run-mcp

The Streamable HTTP endpoint defaults to http://127.0.0.1:8766/mcp and exposes only search_knowledge. See docs/development/MCP_SETUP.md for WorkBuddy configuration, identity boundaries, remote allowlists, and protocol verification.

The health endpoint is available at http://localhost:8000/healthz, and the versioned OpenAPI document is available at http://localhost:8000/api/v1/openapi.json. The Demo Admin Console is available at http://localhost:8000/admin and contains only document upload, document management, and Search Lab pages. See docs/development/ADMIN_CONSOLE.md for the operation and scope boundaries.

Run the development checks with:

make format-check lint typecheck test migration-check eval

Controlled target-provider acceptance uses 32 synthetic queries and is disabled by default. It requires rotated secrets, dedicated test resources, and all safety acknowledgements described in docs/development/LIVE_ACCEPTANCE.md; only then run make live-acceptance.

Document metadata is canonical in OceanBase MySQL relational tables. Demo source files use the local storage/ directory through the provider-neutral ObjectStorage interface; existing objects cannot be overwritten. The upload service accepts PDF, DOCX, TXT, and MD sources, streams at most MAX_UPLOAD_BYTES, and compensates stored bytes if metadata persistence fails. The ingestion service then routes PDF to PyMuPDF, DOCX to python-docx, and UTF-8 TXT/Markdown to lightweight parsers. It normalizes whitespace, retains available page/section context, and stores a versioned parsed JSON artifact under parsed/{document_id}/document.json. The structure_v1 chunk strategy keeps heading/section boundaries, combines paragraphs, and creates configurable 450-token windows with a 75-token overlap. Chunk metadata, including deterministic IDs, filename, page range, section, index, and strategy version, is stored atomically in canonical relational tables. The lightweight token counter is replaceable so DEMO-006 can inject model-specific behavior. Documents remain processing until embedding/indexing completes; pipeline failures become failed without deleting the immutable original. DEMO-009 exposes this pipeline through the versioned upload endpoint.

DEMO-006 added a provider-neutral EmbeddingProvider and VectorStore, plus a validated Qianfan bge-large-zh adapter. ARCH-005 now wires that boundary to OceanBase. Chunk vectors are upserted with source metadata under a versioned index profile; OceanBase relational tables record canonical index state while vector rows remain rebuildable. A document becomes ready only after every chunk is indexed. See docs/development/OCEANBASE_VECTOR_SETUP.md for the required environment variables and pre-provisioned table schema.

The accepted provider migration is tracked by ARCH-001 through ARCH-006: Baidu BOS, OceanBase Vector, and Qianfan bce-reranker-base replace the initial demo providers, and ARCH-006 aligns canonical ownership with the verified OceanBase target. BOS and OceanBase are now the runtime defaults; LocalObjectStorage and FakeVectorStore remain available for explicit local/offline tests. ARCH-005 also adds the controlled shadow rebuild workflow documented in docs/development/INDEX_REBUILD.md.

DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The default is Top 20; the in-memory fake uses exact cosine search for deterministic offline regression. Returned objects are raw dense candidates only.

DEMO-008 adds a provider-neutral reranker, and ARCH-004 aligns its validated Qianfan adapter with the bce-reranker-base baseline. The final Top20-to-Top5 pipeline and source results are rehydrated from canonical relational chunks and retain both Dense and reranker scores; stale vector content fails closed. See docs/development/RERANKER_SETUP.md for configuration and validation limits. DEMO-009 exposes the completed synchronous pipeline through versioned document and search REST endpoints under /api/v1. Upload, list, detail, direct delete, and reranked source search are documented in docs/development/REST_API_SETUP.md; provider and storage internals remain outside the public response contract.

DEMO-012 adds the independent authenticated MCP Gateway. It uses the official MCP Python SDK, maps a runtime-only Bearer credential to a least-privilege service identity, and delegates search_knowledge to the same SearchService used by REST. No provider, storage, database, model, or index-profile details are exposed.

Description
知识平台
Readme 1.9 MiB
Languages
Python 88.7%
CSS 3.8%
JavaScript 2.8%
HTML 1.6%
Shell 0.9%
Other 2.2%