2026-09-01 20:48:41 +08:00

4.8 KiB

Enterprise Knowledge Platform

Enterprise internal knowledge infrastructure.

Core capabilities

  • Multi-format document ingestion
  • OCR and document parsing
  • Structure-aware chunking
  • Multiple embedding providers
  • Hybrid retrieval and reranking
  • ACL-aware search
  • Knowledge Compilation
  • REST API
  • MCP Server
  • Admin Console

Architecture baseline

  • API First + Admin Console
  • Modular Monolith + async workers + MCP gateway
  • PostgreSQL + object storage as canonical source of truth
  • OceanBase Vector as the target retrieval index, behind a replaceable adapter
  • Baidu BOS as target object storage; local storage remains for tests/development
  • Embedding / reranker / parser / LLM providers are replaceable
  • Raw source is immutable
  • Compiled Knowledge is derived and traceable
  • ACL is enforced before or during retrieval, never after Top-K

Start here

  1. Read AGENTS.md
  2. Read CODEX_START_HERE.md
  3. Read docs/product/PRD.md
  4. Read docs/architecture/ARCHITECTURE.md
  5. Read docs/plans/V1_EXECUTION_PLAN.md
  6. For the current milestone, read docs/demo/DEMO_SCOPE.md and pick exactly one DEMO-* task. The V1 tasks in docs/plans/TASKS.md remain the deferred product backlog.

Demo development

The current implementation target is the retrieval demo defined in docs/demo/DEMO_SCOPE.md.

Prerequisites:

  • uv
  • Docker (for PostgreSQL and later integration dependencies)

Install dependencies and run the API:

cp .env.example .env
make install
make run

The health endpoint is available at http://localhost:8000/healthz, and the versioned OpenAPI document is available at http://localhost:8000/api/v1/openapi.json.

Run the development checks with:

make format-check lint typecheck test migration-check eval

Document metadata is canonical in PostgreSQL. Demo source files use the local storage/ directory through the provider-neutral ObjectStorage interface; existing objects cannot be overwritten. The upload service accepts PDF, DOCX, TXT, and MD sources, streams at most MAX_UPLOAD_BYTES, and compensates stored bytes if metadata persistence fails. The ingestion service then routes PDF to PyMuPDF, DOCX to python-docx, and UTF-8 TXT/Markdown to lightweight parsers. It normalizes whitespace, retains available page/section context, and stores a versioned parsed JSON artifact under parsed/{document_id}/document.json. The structure_v1 chunk strategy keeps heading/section boundaries, combines paragraphs, and creates configurable 450-token windows with a 75-token overlap. Chunk metadata, including deterministic IDs, filename, page range, section, index, and strategy version, is stored atomically in PostgreSQL. The lightweight token counter is replaceable so DEMO-006 can inject model-specific behavior. Documents remain processing until embedding/indexing completes; pipeline failures become failed without deleting the immutable original. DEMO-009 exposes this pipeline through the versioned upload endpoint.

DEMO-006 added a provider-neutral EmbeddingProvider and VectorStore, a validated Qianfan bge-large-zh adapter, and the initial isolated pymochow adapter. Chunk vectors are upserted with source metadata under a versioned index profile; PostgreSQL records canonical index state while Mochow remains rebuildable. A document becomes ready only after every chunk is indexed. See docs/development/MOCHOW_SETUP.md for the required environment variables and pre-provisioned table schema.

The accepted provider migration is tracked by ARCH-001 through ARCH-006: Baidu BOS, OceanBase Vector, and Qianfan bce-reranker-base replace the initial demo providers without changing PostgreSQL canonical ownership. Until the runtime cutover task is merged, the paragraphs above describe the current implementation. ARCH-002 adds the selectable BOS adapter and safe runtime configuration documented in docs/development/BOS_SETUP.md; local storage remains the default until ARCH-005.

DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The default is Top 20 with a configurable Mochow HNSW ef; the in-memory fake uses exact cosine search for deterministic offline regression. Returned objects are raw dense candidates only.

DEMO-008 adds a provider-neutral reranker, a validated Qianfan qwen3-reranker-0.6b adapter, and the final Top20-to-Top5 search pipeline. Final source results are rehydrated from canonical PostgreSQL chunks and retain both Dense and reranker scores; stale vector content fails closed. See docs/development/RERANKER_SETUP.md for configuration and validation limits. DEMO-009 exposes the completed synchronous pipeline through versioned document and search REST endpoints under /api/v1. Upload, list, detail, direct delete, and reranked source search are documented in docs/development/REST_API_SETUP.md; provider and storage internals remain outside the public response contract.