EmboFlow/docs/plans/2026-03-26-emboflow-v1-foundation-and-mvp.md

631 lines
19 KiB
Markdown

# EmboFlow V1 Foundation And MVP Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Build the first usable EmboFlow increment: workspace-aware raw asset ingestion, workflow definition/versioning, local workflow execution, and the first web workflow authoring surfaces.
**Architecture:** Use a TypeScript monorepo with a React web app, a Node.js API control plane, and a separate Node.js worker. Use MongoDB as the only database, object storage abstraction for cloud storage or MinIO, and a local scheduler with Python and Docker executor contracts.
**Tech Stack:** pnpm workspace, React, TypeScript, React Flow, NestJS, Mongoose, MongoDB, Docker Compose, Python runtime hooks, Python unittest, and Node 22 built-in test runner with TypeScript stripping for early package-level tests
---
## Progress Notes
- `2026-03-26`: Tasks 1 and 2 are complete and committed.
- `2026-03-26`: Tasks 3 through 6 are implemented against in-memory V1 control-plane services so the API and worker contracts can stabilize before persistence and framework wiring are deepened.
- `2026-03-26`: Package-level verification continues to use the Node 22 built-in test runner with direct file targets such as `pnpm --filter api test test/projects.e2e-spec.ts` and `pnpm --filter worker test test/task-runner.spec.ts`.
- `2026-03-26`: Tasks 7 through 10 add the first web shell, workflow editor surfaces, artifact explore renderers, developer entry commands, and CI/pre-push test execution through `make test`.
---
### Task 1: Bootstrap The Monorepo And Runtime Skeleton
**Files:**
- Create: `package.json`
- Create: `pnpm-workspace.yaml`
- Create: `tsconfig.base.json`
- Create: `apps/web/package.json`
- Create: `apps/api/package.json`
- Create: `apps/worker/package.json`
- Create: `docker-compose.yml`
- Create: `.env.example`
- Test: `tests/test_repo_structure.py`
**Step 1: Write the failing test**
Create `tests/test_repo_structure.py` to assert the repository contains the expected top-level app folders and root workspace files.
**Step 2: Run test to verify it fails**
Run:
```bash
python3 -m unittest tests/test_repo_structure.py -v
```
Expected: FAIL because the monorepo files and app folders do not exist yet.
**Step 3: Write minimal implementation**
Create the pnpm workspace root, app package manifests, root TypeScript config, `.env.example`, and `docker-compose.yml` with services for:
- `web`
- `api`
- `worker`
- `mongo`
- `minio`
Keep the first version minimal. Do not add extra infra services that are not required by the design.
**Step 4: Run test to verify it passes**
Run:
```bash
python3 -m unittest tests/test_repo_structure.py -v
```
Expected: PASS
**Step 5: Commit**
```bash
git add package.json pnpm-workspace.yaml tsconfig.base.json apps docker-compose.yml .env.example tests/test_repo_structure.py
git commit -m ":tada: bootstrap workspace and runtime skeleton"
```
### Task 2: Create Shared Domain Contracts And Mongo Setup
**Files:**
- Create: `packages/contracts/package.json`
- Create: `packages/contracts/src/domain.ts`
- Create: `apps/api/src/common/mongo/mongo.module.ts`
- Create: `apps/api/src/common/mongo/schemas/workspace.schema.ts`
- Create: `apps/api/src/common/mongo/schemas/project.schema.ts`
- Create: `apps/api/src/common/mongo/schemas/asset.schema.ts`
- Create: `apps/api/src/common/mongo/schemas/workflow.schema.ts`
- Test: `apps/api/test/domain-contracts.spec.ts`
**Step 1: Write the failing test**
Create `apps/api/test/domain-contracts.spec.ts` asserting:
- workspace types include `personal` and `team`
- asset types include raw and dataset-style sources
- workflow status values match the design docs
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter api test test/domain-contracts.spec.ts
```
Expected: FAIL because contracts and schemas are missing.
**Step 3: Write minimal implementation**
Create shared domain enums and base Mongo schema definitions for:
- workspaces
- projects
- assets
- workflow definitions
Add a minimal Mongo module in the API app using environment-based connection config.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter api test test/domain-contracts.spec.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add packages/contracts apps/api/src/common apps/api/test/domain-contracts.spec.ts
git commit -m ":sparkles: add shared domain contracts and mongo setup"
```
### Task 3: Implement Identity, Workspace, And Project APIs
**Files:**
- Create: `apps/api/src/modules/auth/auth.module.ts`
- Create: `apps/api/src/modules/auth/auth.controller.ts`
- Create: `apps/api/src/modules/workspaces/workspaces.module.ts`
- Create: `apps/api/src/modules/workspaces/workspaces.controller.ts`
- Create: `apps/api/src/modules/projects/projects.module.ts`
- Create: `apps/api/src/modules/projects/projects.controller.ts`
- Create: `apps/api/src/modules/projects/projects.service.ts`
- Test: `apps/api/test/projects.e2e-spec.ts`
**Step 1: Write the failing test**
Create `apps/api/test/projects.e2e-spec.ts` covering:
- create personal workspace bootstrap flow
- create project under a workspace
- reject project creation without a workspace id
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter api test test/projects.e2e-spec.ts
```
Expected: FAIL because the modules and endpoints do not exist yet.
**Step 3: Write minimal implementation**
Implement:
- development-safe auth stub or local auth module
- workspace creation and listing
- project creation and listing
- basic membership checks sufficient for V1 local development
Do not build a full production auth stack before the API shape is stable.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter api test test/projects.e2e-spec.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/api/src/modules/auth apps/api/src/modules/workspaces apps/api/src/modules/projects apps/api/test/projects.e2e-spec.ts
git commit -m ":sparkles: add workspace and project APIs"
```
### Task 4: Implement Asset Ingestion, Storage Abstraction, And Probe Metadata
**Files:**
- Create: `apps/api/src/modules/storage/storage.module.ts`
- Create: `apps/api/src/modules/storage/storage.service.ts`
- Create: `apps/api/src/modules/assets/assets.module.ts`
- Create: `apps/api/src/modules/assets/assets.controller.ts`
- Create: `apps/api/src/modules/assets/assets.service.ts`
- Create: `apps/api/src/modules/assets/probe/probe.service.ts`
- Create: `apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts`
- Test: `apps/api/test/assets.e2e-spec.ts`
**Step 1: Write the failing test**
Create `apps/api/test/assets.e2e-spec.ts` covering:
- register an uploaded asset record
- create a probe report for a raw asset
- return recommended next actions from probe metadata
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter api test test/assets.e2e-spec.ts
```
Expected: FAIL because asset ingestion and probe services are missing.
**Step 3: Write minimal implementation**
Implement:
- storage abstraction interface
- MinIO/S3-compatible config contract
- asset create/list/detail endpoints
- probe-report persistence
- placeholder probe logic for directory and archive summaries
Do not build full binary upload optimization yet. First make the metadata contract stable.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter api test test/assets.e2e-spec.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/api/src/modules/storage apps/api/src/modules/assets apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts apps/api/test/assets.e2e-spec.ts
git commit -m ":truck: add asset ingestion and probe metadata flow"
```
### Task 5: Implement Workflow Definitions, Versions, Runs, And Tasks
**Files:**
- Create: `apps/api/src/modules/workflows/workflows.module.ts`
- Create: `apps/api/src/modules/workflows/workflows.controller.ts`
- Create: `apps/api/src/modules/workflows/workflows.service.ts`
- Create: `apps/api/src/modules/runs/runs.module.ts`
- Create: `apps/api/src/modules/runs/runs.controller.ts`
- Create: `apps/api/src/modules/runs/runs.service.ts`
- Create: `apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts`
- Create: `apps/api/src/common/mongo/schemas/workflow-run.schema.ts`
- Create: `apps/api/src/common/mongo/schemas/run-task.schema.ts`
- Test: `apps/api/test/workflow-runs.e2e-spec.ts`
**Step 1: Write the failing test**
Create `apps/api/test/workflow-runs.e2e-spec.ts` covering:
- create workflow definition
- save workflow version
- create workflow run from saved version
- generate initial run tasks for ready nodes
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter api test test/workflow-runs.e2e-spec.ts
```
Expected: FAIL because workflow versioning and run creation do not exist yet.
**Step 3: Write minimal implementation**
Implement:
- workflow definition head record
- immutable workflow version snapshots
- run creation from a workflow version
- initial DAG compilation for simple source-to-transform chains
- run task persistence
Keep V1 graph compilation simple. Support sequential edges first, then one-level branching.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter api test test/workflow-runs.e2e-spec.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/api/src/modules/workflows apps/api/src/modules/runs apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts apps/api/src/common/mongo/schemas/workflow-run.schema.ts apps/api/src/common/mongo/schemas/run-task.schema.ts apps/api/test/workflow-runs.e2e-spec.ts
git commit -m ":sparkles: add workflow versioning and run records"
```
### Task 6: Add The Worker, Local Scheduler, And Executor Contracts
**Files:**
- Create: `apps/worker/src/main.ts`
- Create: `apps/worker/src/runner/task-runner.ts`
- Create: `apps/worker/src/scheduler/local-scheduler.ts`
- Create: `apps/worker/src/executors/python-executor.ts`
- Create: `apps/worker/src/executors/docker-executor.ts`
- Create: `apps/worker/src/executors/http-executor.ts`
- Create: `apps/worker/src/contracts/execution-context.ts`
- Test: `apps/worker/test/task-runner.spec.ts`
**Step 1: Write the failing test**
Create `apps/worker/test/task-runner.spec.ts` covering:
- worker loads pending tasks
- worker marks task running then success
- worker chooses executor based on node runtime config
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter worker test task-runner.spec.ts
```
Expected: FAIL because the worker runtime does not exist yet.
**Step 3: Write minimal implementation**
Implement:
- worker bootstrap
- polling or queue-backed local scheduler
- execution context builder
- stub Python, Docker, and HTTP executors
- task status transitions
Do not implement full Docker isolation logic in one step. First lock the runtime interfaces and transitions.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter worker test task-runner.spec.ts
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/worker apps/api/src/modules/runs apps/worker/test/task-runner.spec.ts
git commit -m ":construction_worker: add local worker and executor contracts"
```
### Task 7: Build The Web Shell, Workspace Flow, And Asset Workspace
**Files:**
- Create: `apps/web/src/main.tsx`
- Create: `apps/web/src/app/router.tsx`
- Create: `apps/web/src/features/layout/app-shell.tsx`
- Create: `apps/web/src/features/workspaces/workspace-switcher.tsx`
- Create: `apps/web/src/features/projects/project-selector.tsx`
- Create: `apps/web/src/features/assets/assets-page.tsx`
- Create: `apps/web/src/features/assets/asset-detail-page.tsx`
- Create: `apps/web/src/features/assets/components/asset-list.tsx`
- Create: `apps/web/src/features/assets/components/asset-summary-panel.tsx`
- Test: `apps/web/src/features/assets/assets-page.test.tsx`
**Step 1: Write the failing test**
Create `apps/web/src/features/assets/assets-page.test.tsx` covering:
- app shell renders primary navigation
- assets page renders asset rows from API data
- asset detail page renders probe summary
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter web test assets-page.test.tsx
```
Expected: FAIL because the web app shell and pages do not exist yet.
**Step 3: Write minimal implementation**
Implement:
- web app bootstrap
- primary navigation matching the design docs
- workspace/project header controls
- asset list page
- asset detail page with summary and action buttons
Defer advanced preview renderers. Start with structured metadata and simple detail views.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter web test assets-page.test.tsx
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/web apps/web/src/features/assets/assets-page.test.tsx
git commit -m ":sparkles: add web shell and asset workspace"
```
### Task 8: Build Canvas Authoring, Run Detail, And First Workflow Actions
**Files:**
- Create: `apps/web/src/features/workflows/workflows-page.tsx`
- Create: `apps/web/src/features/workflows/workflow-editor-page.tsx`
- Create: `apps/web/src/features/workflows/components/node-library.tsx`
- Create: `apps/web/src/features/workflows/components/workflow-canvas.tsx`
- Create: `apps/web/src/features/workflows/components/node-config-panel.tsx`
- Create: `apps/web/src/features/runs/run-detail-page.tsx`
- Create: `apps/web/src/features/runs/components/run-graph-view.tsx`
- Create: `apps/web/src/features/runs/components/task-log-panel.tsx`
- Test: `apps/web/src/features/workflows/workflow-editor-page.test.tsx`
**Step 1: Write the failing test**
Create `apps/web/src/features/workflows/workflow-editor-page.test.tsx` covering:
- node library renders categories
- node config panel opens when a node is selected
- run detail view shows node status badges from run data
**Step 2: Run test to verify it fails**
Run:
```bash
pnpm --filter web test workflow-editor-page.test.tsx
```
Expected: FAIL because the workflow editor and run detail pages do not exist yet.
**Step 3: Write minimal implementation**
Implement:
- workflow list page
- workflow editor page using React Flow
- left node library, center canvas, right config panel
- save workflow version action
- trigger workflow run action
- run detail page with graph and selected-node log panel
Keep the first editor scoped to V1 node categories and schema-driven config rendering.
**Step 4: Run test to verify it passes**
Run:
```bash
pnpm --filter web test workflow-editor-page.test.tsx
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/web/src/features/workflows apps/web/src/features/runs apps/web/src/features/workflows/workflow-editor-page.test.tsx
git commit -m ":sparkles: add canvas workflow editor and run detail pages"
```
### Task 9: Add Preview Surface, Delivery Nodes, And MVP Integration
**Files:**
- Create: `apps/api/src/modules/artifacts/artifacts.module.ts`
- Create: `apps/api/src/modules/artifacts/artifacts.controller.ts`
- Create: `apps/api/src/modules/artifacts/artifacts.service.ts`
- Create: `apps/web/src/features/explore/explore-page.tsx`
- Create: `apps/web/src/features/explore/renderers/json-renderer.tsx`
- Create: `apps/web/src/features/explore/renderers/video-renderer.tsx`
- Create: `apps/web/src/features/explore/renderers/directory-renderer.tsx`
- Create: `apps/api/src/modules/plugins/builtin/delivery-nodes.ts`
- Test: `apps/api/test/artifacts.e2e-spec.ts`
- Test: `apps/web/src/features/explore/explore-page.test.tsx`
**Step 1: Write the failing tests**
Create:
- `apps/api/test/artifacts.e2e-spec.ts` for artifact retrieval by producer
- `apps/web/src/features/explore/explore-page.test.tsx` for opening and rendering supported artifact types
**Step 2: Run tests to verify they fail**
Run:
```bash
pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx
```
Expected: FAIL because artifact APIs and explore renderers do not exist yet.
**Step 3: Write minimal implementation**
Implement:
- artifact module and lookup endpoints
- explore page
- JSON, directory, and video renderers
- built-in delivery-normalization node definitions for the V1 business path
Do not implement the full renderer plugin platform yet. Start with built-ins and stable renderer contracts.
**Step 4: Run tests to verify they pass**
Run:
```bash
pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx
```
Expected: PASS
**Step 5: Commit**
```bash
git add apps/api/src/modules/artifacts apps/api/src/modules/plugins/builtin/delivery-nodes.ts apps/api/test/artifacts.e2e-spec.ts apps/web/src/features/explore apps/web/src/features/explore/explore-page.test.tsx
git commit -m ":package: add explore surface and delivery artifacts"
```
### Task 10: Harden Guardrails, Docs, And Developer Entry Commands
**Files:**
- Modify: `CONTRIBUTING.md`
- Modify: `docs/development-workflow.md`
- Modify: `design/03-workflows/workflow-execution-model.md`
- Modify: `design/05-data/mongodb-data-model.md`
- Create: `Makefile`
- Create: `README.md`
- Test: `tests/test_dev_commands.py`
**Step 1: Write the failing test**
Create `tests/test_dev_commands.py` asserting:
- `Makefile` exposes expected local commands
- `README.md` documents bootstrap, hooks, test, and local run commands
**Step 2: Run test to verify it fails**
Run:
```bash
python3 -m unittest tests/test_dev_commands.py -v
```
Expected: FAIL because developer entry commands are not documented yet.
**Step 3: Write minimal implementation**
Add:
- `make bootstrap`
- `make test`
- `make dev-api`
- `make dev-web`
- `make dev-worker`
- `make guardrails`
Document the developer flow in `README.md` and update design docs if implementation decisions changed during Tasks 1-9.
**Step 4: Run test to verify it passes**
Run:
```bash
python3 -m unittest tests/test_dev_commands.py -v
```
Expected: PASS
**Step 5: Commit**
```bash
git add CONTRIBUTING.md docs/development-workflow.md design/03-workflows/workflow-execution-model.md design/05-data/mongodb-data-model.md Makefile README.md tests/test_dev_commands.py
git commit -m ":memo: add developer entry commands and bootstrap docs"
```
## Exit Criteria
The first implementation pass is complete when:
- a user can create a workspace and project
- a raw asset can be registered and probed
- a workflow can be created, versioned, and executed locally
- run tasks produce observable status and artifacts
- the web app exposes assets, workflows, runs, and basic explore views
- guardrails for docs, hooks, commit messages, and CI remain green
## Notes
- Keep commits small and use the repository gitmoji + English commit policy.
- Update design files in the same task where behavior or architecture changes.
- Do not add training execution before the V1 data workflow loop is stable.