2026-09-02 11:28:13 +08:00

116 lines
4.8 KiB
Markdown

# Enterprise Knowledge Platform
Enterprise internal knowledge infrastructure.
## Core capabilities
- Multi-format document ingestion
- OCR and document parsing
- Structure-aware chunking
- Multiple embedding providers
- Hybrid retrieval and reranking
- ACL-aware search
- Knowledge Compilation
- REST API
- MCP Server
- Admin Console
## Architecture baseline
- API First + Admin Console
- Modular Monolith + async workers + MCP gateway
- PostgreSQL + object storage as canonical source of truth
- OceanBase Vector as the target retrieval index, behind a replaceable adapter
- Baidu BOS as target object storage; local storage remains for tests/development
- Embedding / reranker / parser / LLM providers are replaceable
- Raw source is immutable
- Compiled Knowledge is derived and traceable
- ACL is enforced before or during retrieval, never after Top-K
## Start here
1. Read `AGENTS.md`
2. Read `CODEX_START_HERE.md`
3. Read `docs/product/PRD.md`
4. Read `docs/architecture/ARCHITECTURE.md`
5. Read `docs/plans/V1_EXECUTION_PLAN.md`
6. For the current milestone, read `docs/demo/DEMO_SCOPE.md` and pick exactly
one `DEMO-*` task. The V1 tasks in `docs/plans/TASKS.md` remain the deferred
product backlog.
## Demo development
The current implementation target is the retrieval demo defined in
`docs/demo/DEMO_SCOPE.md`.
Prerequisites:
- `uv`
- Docker (for PostgreSQL and later integration dependencies)
Install dependencies and run the API:
```bash
cp .env.example .env
make install
make run
```
The health endpoint is available at `http://localhost:8000/healthz`, and the
versioned OpenAPI document is available at
`http://localhost:8000/api/v1/openapi.json`.
Run the development checks with:
```bash
make format-check lint typecheck test migration-check eval
```
Document metadata is canonical in PostgreSQL. Demo source files use the local
`storage/` directory through the provider-neutral `ObjectStorage` interface;
existing objects cannot be overwritten. The upload service accepts PDF, DOCX,
TXT, and MD sources, streams at most `MAX_UPLOAD_BYTES`, and compensates stored
bytes if metadata persistence fails. The ingestion service then routes PDF to
PyMuPDF, DOCX to python-docx, and UTF-8 TXT/Markdown to lightweight parsers. It
normalizes whitespace, retains available page/section context, and stores a
versioned parsed JSON artifact under `parsed/{document_id}/document.json`.
The `structure_v1` chunk strategy keeps heading/section boundaries, combines
paragraphs, and creates configurable 450-token windows with a 75-token overlap.
Chunk metadata, including deterministic IDs, filename, page range, section,
index, and strategy version, is stored atomically in PostgreSQL. The lightweight
token counter is replaceable so DEMO-006 can inject model-specific behavior.
Documents remain `processing` until embedding/indexing completes; pipeline
failures become `failed` without deleting the immutable original. DEMO-009
exposes this pipeline through the versioned upload endpoint.
DEMO-006 added a provider-neutral `EmbeddingProvider` and `VectorStore`, plus a
validated Qianfan `bge-large-zh` adapter. ARCH-005 now wires that boundary to OceanBase.
Chunk vectors are upserted with source metadata under a versioned index profile;
PostgreSQL records canonical index state while OceanBase remains rebuildable. A
document becomes `ready` only after every chunk is indexed. See
`docs/development/OCEANBASE_VECTOR_SETUP.md` for the required environment variables
and pre-provisioned table schema.
The accepted provider migration is tracked by `ARCH-001` through `ARCH-006`:
Baidu BOS, OceanBase Vector, and Qianfan `bce-reranker-base` replace the initial
demo providers without changing PostgreSQL canonical ownership. BOS and OceanBase
are now the runtime defaults; LocalObjectStorage and FakeVectorStore remain available
for explicit local/offline tests. ARCH-005 also adds the controlled shadow rebuild
workflow documented in `docs/development/INDEX_REBUILD.md`.
DEMO-007 adds query embedding and profile-scoped Dense Top-N retrieval. The
default is Top 20; the in-memory fake uses
exact cosine search for deterministic offline regression. Returned objects are
raw dense candidates only.
DEMO-008 adds a provider-neutral reranker, and ARCH-004 aligns its validated Qianfan
adapter with the `bce-reranker-base` baseline. The final Top20-to-Top5 pipeline and
source results are rehydrated from canonical PostgreSQL chunks and retain both
Dense and reranker scores; stale vector content fails closed. See
`docs/development/RERANKER_SETUP.md` for configuration and validation limits.
DEMO-009 exposes the completed synchronous pipeline through versioned document
and search REST endpoints under `/api/v1`. Upload, list, detail, direct delete,
and reranked source search are documented in
`docs/development/REST_API_SETUP.md`; provider and storage internals remain
outside the public response contract.