7.3 KiB
Enterprise Knowledge Platform
Enterprise internal knowledge infrastructure.
Core capabilities
- Multi-format document ingestion
- OCR and document parsing
- Structure-aware chunking
- Multiple embedding providers
- Hybrid retrieval and reranking
- ACL-aware search
- Knowledge Compilation
- REST API
- MCP Server
- Admin Console
Architecture baseline
- API First + Admin Console
- Modular Monolith + async workers + MCP gateway
- OceanBase MySQL relational tables + object storage as canonical source of truth
- Co-located OceanBase Vector tables as rebuildable retrieval indexes behind an adapter
- Baidu BOS as target object storage; local storage remains for tests/development
- Embedding / reranker / parser / LLM providers are replaceable
- Raw source is immutable
- Compiled Knowledge is derived and traceable
- ACL is enforced before or during retrieval, never after Top-K
Start here
- Read
AGENTS.md - Read
CODEX_START_HERE.md - Read
docs/product/PRD.md - Read
docs/architecture/ARCHITECTURE.md - Read
docs/plans/V1_EXECUTION_PLAN.md - For the current milestone, read
docs/demo/DEMO_SCOPE.mdand pick exactly oneDEMO-*task. The V1 tasks indocs/plans/TASKS.mdremain the deferred product backlog.
Demo development
The current implementation target is the retrieval demo defined in
docs/demo/DEMO_SCOPE.md.
Prerequisites:
uv- Docker (for optional local integration dependencies)
Install dependencies and run the API:
cp .env.example .env
make install
make run-worker
make run
Run the independent authenticated MCP Gateway after injecting a dedicated runtime token of at least 32 characters:
export MCP_AUTH_TOKEN="<runtime-secret-with-at-least-32-characters>"
make run-mcp
The Streamable HTTP endpoint defaults to http://127.0.0.1:8766/mcp and exposes
only search_knowledge. See docs/development/MCP_SETUP.md for WorkBuddy configuration,
identity boundaries, remote allowlists, and protocol verification.
Kubernetes installation, target-cluster acceptance, rolling recovery, metrics and backup
responsibility are documented in docs/development/KUBERNETES_DEPLOYMENT.md and
docs/development/CLUSTER_OPERATIONS.md. Cluster acceptance always requires an explicitly named
Kubernetes context; credentials remain in Kubernetes Secret or the client process environment.
The health endpoint is available at http://localhost:8000/healthz, and the
versioned OpenAPI document is available at
http://localhost:8000/api/v1/openapi.json.
The Demo Admin Console is available at http://localhost:8000/admin and contains
only document upload, document management, and Search Lab pages. See
docs/development/ADMIN_CONSOLE.md for the operation and scope boundaries.
DEPLOY-001 provides locked, non-root api, mcp, and worker production image targets with an
amd64/arm64 Buildx Bake plan. DEPLOY-002 adds the Kubernetes/Helm Chart, runtime Secret references,
serialized Migration, Ingress/TLS and the make deploy VALUES=<file> entrypoint. See
docs/development/CONTAINER_IMAGES.md and docs/development/KUBERNETES_DEPLOYMENT.md.
Run the development checks with:
make format-check lint typecheck test migration-check eval
Controlled target-provider acceptance uses 32 synthetic queries and is disabled by
default. It requires rotated secrets, dedicated test resources, and all safety
acknowledgements described in docs/development/LIVE_ACCEPTANCE.md; only then run
make live-acceptance.
Document metadata is canonical in OceanBase MySQL relational tables. Demo source files use the local
storage/ directory through the provider-neutral ObjectStorage interface;
existing objects cannot be overwritten. The upload service accepts PDF, DOCX,
TXT, and MD sources, streams at most MAX_UPLOAD_BYTES, and compensates stored
bytes if metadata persistence fails. The ingestion service then routes PDF to
PyMuPDF, DOCX to python-docx, and UTF-8 TXT/Markdown to lightweight parsers. It
normalizes whitespace, retains available page/section context, and stores a
versioned parsed JSON artifact under parsed/{document_id}/document.json.
The structure_v1 chunk strategy keeps heading/section boundaries, combines
paragraphs, and creates configurable 450-token windows with a 75-token overlap.
Chunk metadata, including deterministic IDs, filename, page range, section,
index, and strategy version, is stored atomically in canonical relational tables. The lightweight
token counter is replaceable so DEMO-006 can inject model-specific behavior.
Documents remain processing until embedding/indexing completes; pipeline
failures become failed without deleting the immutable original. DEMO-015 now
accepts a source plus durable Job with HTTP 202, while a Celery/RabbitMQ worker
runs the pipeline. Bounded transient retries clean every derived artifact before
starting the next Attempt; broker redelivery and expired worker leases converge on
OceanBase Job state. See docs/development/ASYNC_INGESTION_SETUP.md.
DEMO-006 added a provider-neutral EmbeddingProvider and VectorStore, plus a
validated Qianfan bge-large-zh adapter. ARCH-005 now wires that boundary to OceanBase.
Chunk vectors are upserted with source metadata under a versioned index profile;
OceanBase relational tables record canonical index state while vector rows remain rebuildable. A
document becomes ready only after every chunk is indexed. See
docs/development/OCEANBASE_VECTOR_SETUP.md for the required environment variables
and pre-provisioned table schema.
The accepted provider migration is tracked by ARCH-001 through ARCH-006:
Baidu BOS, OceanBase Vector, and Qianfan bce-reranker-base replace the initial
demo providers, and ARCH-006 aligns canonical ownership with the verified OceanBase target. BOS and OceanBase
are now the runtime defaults; LocalObjectStorage and FakeVectorStore remain available
for explicit local/offline tests. ARCH-005 also adds the controlled shadow rebuild
workflow documented in docs/development/INDEX_REBUILD.md.
DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The default is Top 20; the in-memory fake uses exact cosine search for deterministic offline regression. Returned objects are raw dense candidates only.
DEMO-008 adds a provider-neutral reranker, and ARCH-004 aligns its validated Qianfan
adapter with the bce-reranker-base baseline. The final Top20-to-Top5 pipeline and
source results are rehydrated from canonical relational chunks and retain both
Dense and reranker scores; stale vector content fails closed. See
docs/development/RERANKER_SETUP.md for configuration and validation limits.
DEMO-009 exposes document and search REST endpoints under /api/v1; DEMO-015
changes upload to asynchronous acceptance and adds Job status. DEMO-016 adds bounded multi-file
submission, batch/per-file status, and automatic Admin Console polling. Upload, list, detail, direct delete,
and reranked source search are documented in
docs/development/REST_API_SETUP.md; provider and storage internals remain
outside the public response contract.
DEMO-012 adds the independent authenticated MCP Gateway. It uses the official
MCP Python SDK, maps a runtime-only Bearer credential to a least-privilege service
identity, and delegates search_knowledge to the same SearchService used by REST.
No provider, storage, database, model, or index-profile details are exposed.