151 lines
6.9 KiB
Markdown
151 lines
6.9 KiB
Markdown
# Enterprise Knowledge Platform
|
|
|
|
Enterprise internal knowledge infrastructure.
|
|
|
|
## Core capabilities
|
|
|
|
- Multi-format document ingestion
|
|
- OCR and document parsing
|
|
- Structure-aware chunking
|
|
- Multiple embedding providers
|
|
- Hybrid retrieval and reranking
|
|
- ACL-aware search
|
|
- Knowledge Compilation
|
|
- REST API
|
|
- MCP Server
|
|
- Admin Console
|
|
|
|
## Architecture baseline
|
|
|
|
- API First + Admin Console
|
|
- Modular Monolith + async workers + MCP gateway
|
|
- OceanBase MySQL relational tables + object storage as canonical source of truth
|
|
- Co-located OceanBase Vector tables as rebuildable retrieval indexes behind an adapter
|
|
- Baidu BOS as target object storage; local storage remains for tests/development
|
|
- Embedding / reranker / parser / LLM providers are replaceable
|
|
- Raw source is immutable
|
|
- Compiled Knowledge is derived and traceable
|
|
- ACL is enforced before or during retrieval, never after Top-K
|
|
|
|
## Start here
|
|
|
|
1. Read `AGENTS.md`
|
|
2. Read `CODEX_START_HERE.md`
|
|
3. Read `docs/product/PRD.md`
|
|
4. Read `docs/architecture/ARCHITECTURE.md`
|
|
5. Read `docs/plans/V1_EXECUTION_PLAN.md`
|
|
6. For the current milestone, read `docs/demo/DEMO_SCOPE.md` and pick exactly
|
|
one `DEMO-*` task. The V1 tasks in `docs/plans/TASKS.md` remain the deferred
|
|
product backlog.
|
|
|
|
## Demo development
|
|
|
|
The current implementation target is the retrieval demo defined in
|
|
`docs/demo/DEMO_SCOPE.md`.
|
|
|
|
Prerequisites:
|
|
|
|
- `uv`
|
|
- Docker (for optional local integration dependencies)
|
|
|
|
Install dependencies and run the API:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
make install
|
|
make run-worker
|
|
make run
|
|
```
|
|
|
|
Run the independent authenticated MCP Gateway after injecting a dedicated
|
|
runtime token of at least 32 characters:
|
|
|
|
```bash
|
|
export MCP_AUTH_TOKEN="<runtime-secret-with-at-least-32-characters>"
|
|
make run-mcp
|
|
```
|
|
|
|
The Streamable HTTP endpoint defaults to `http://127.0.0.1:8766/mcp` and exposes
|
|
only `search_knowledge`. See `docs/development/MCP_SETUP.md` for WorkBuddy configuration,
|
|
identity boundaries, remote allowlists, and protocol verification.
|
|
|
|
The health endpoint is available at `http://localhost:8000/healthz`, and the
|
|
versioned OpenAPI document is available at
|
|
`http://localhost:8000/api/v1/openapi.json`.
|
|
The Demo Admin Console is available at `http://localhost:8000/admin` and contains
|
|
only document upload, document management, and Search Lab pages. See
|
|
`docs/development/ADMIN_CONSOLE.md` for the operation and scope boundaries.
|
|
|
|
DEPLOY-001 provides locked, non-root `api`, `mcp`, and `worker` production image targets with an
|
|
amd64/arm64 Buildx Bake plan. DEPLOY-002 adds the Kubernetes/Helm Chart, runtime Secret references,
|
|
serialized Migration, Ingress/TLS and the `make deploy VALUES=<file>` entrypoint. See
|
|
`docs/development/CONTAINER_IMAGES.md` and `docs/development/KUBERNETES_DEPLOYMENT.md`.
|
|
|
|
Run the development checks with:
|
|
|
|
```bash
|
|
make format-check lint typecheck test migration-check eval
|
|
```
|
|
|
|
Controlled target-provider acceptance uses 32 synthetic queries and is disabled by
|
|
default. It requires rotated secrets, dedicated test resources, and all safety
|
|
acknowledgements described in `docs/development/LIVE_ACCEPTANCE.md`; only then run
|
|
`make live-acceptance`.
|
|
|
|
Document metadata is canonical in OceanBase MySQL relational tables. Demo source files use the local
|
|
`storage/` directory through the provider-neutral `ObjectStorage` interface;
|
|
existing objects cannot be overwritten. The upload service accepts PDF, DOCX,
|
|
TXT, and MD sources, streams at most `MAX_UPLOAD_BYTES`, and compensates stored
|
|
bytes if metadata persistence fails. The ingestion service then routes PDF to
|
|
PyMuPDF, DOCX to python-docx, and UTF-8 TXT/Markdown to lightweight parsers. It
|
|
normalizes whitespace, retains available page/section context, and stores a
|
|
versioned parsed JSON artifact under `parsed/{document_id}/document.json`.
|
|
The `structure_v1` chunk strategy keeps heading/section boundaries, combines
|
|
paragraphs, and creates configurable 450-token windows with a 75-token overlap.
|
|
Chunk metadata, including deterministic IDs, filename, page range, section,
|
|
index, and strategy version, is stored atomically in canonical relational tables. The lightweight
|
|
token counter is replaceable so DEMO-006 can inject model-specific behavior.
|
|
Documents remain `processing` until embedding/indexing completes; pipeline
|
|
failures become `failed` without deleting the immutable original. DEMO-015 now
|
|
accepts a source plus durable Job with HTTP `202`, while a Celery/RabbitMQ worker
|
|
runs the pipeline. Bounded transient retries clean every derived artifact before
|
|
starting the next Attempt; broker redelivery and expired worker leases converge on
|
|
OceanBase Job state. See `docs/development/ASYNC_INGESTION_SETUP.md`.
|
|
|
|
DEMO-006 added a provider-neutral `EmbeddingProvider` and `VectorStore`, plus a
|
|
validated Qianfan `bge-large-zh` adapter. ARCH-005 now wires that boundary to OceanBase.
|
|
Chunk vectors are upserted with source metadata under a versioned index profile;
|
|
OceanBase relational tables record canonical index state while vector rows remain rebuildable. A
|
|
document becomes `ready` only after every chunk is indexed. See
|
|
`docs/development/OCEANBASE_VECTOR_SETUP.md` for the required environment variables
|
|
and pre-provisioned table schema.
|
|
|
|
The accepted provider migration is tracked by `ARCH-001` through `ARCH-006`:
|
|
Baidu BOS, OceanBase Vector, and Qianfan `bce-reranker-base` replace the initial
|
|
demo providers, and ARCH-006 aligns canonical ownership with the verified OceanBase target. BOS and OceanBase
|
|
are now the runtime defaults; LocalObjectStorage and FakeVectorStore remain available
|
|
for explicit local/offline tests. ARCH-005 also adds the controlled shadow rebuild
|
|
workflow documented in `docs/development/INDEX_REBUILD.md`.
|
|
|
|
DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The
|
|
default is Top 20; the in-memory fake uses
|
|
exact cosine search for deterministic offline regression. Returned objects are
|
|
raw dense candidates only.
|
|
|
|
DEMO-008 adds a provider-neutral reranker, and ARCH-004 aligns its validated Qianfan
|
|
adapter with the `bce-reranker-base` baseline. The final Top20-to-Top5 pipeline and
|
|
source results are rehydrated from canonical relational chunks and retain both
|
|
Dense and reranker scores; stale vector content fails closed. See
|
|
`docs/development/RERANKER_SETUP.md` for configuration and validation limits.
|
|
DEMO-009 exposes document and search REST endpoints under `/api/v1`; DEMO-015
|
|
changes upload to asynchronous acceptance and adds Job status. DEMO-016 adds bounded multi-file
|
|
submission, batch/per-file status, and automatic Admin Console polling. Upload, list, detail, direct delete,
|
|
and reranked source search are documented in
|
|
`docs/development/REST_API_SETUP.md`; provider and storage internals remain
|
|
outside the public response contract.
|
|
|
|
DEMO-012 adds the independent authenticated MCP Gateway. It uses the official
|
|
MCP Python SDK, maps a runtime-only Bearer credential to a least-privilege service
|
|
identity, and delegates `search_knowledge` to the same `SearchService` used by REST.
|
|
No provider, storage, database, model, or index-profile details are exposed.
|