EmboFlow/docs/plans/2026-03-26-emboflow-v1-foundation-and-mvp.md

EmboFlow V1 Foundation And MVP Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Build the first usable EmboFlow increment: workspace-aware raw asset ingestion, workflow definition/versioning, local workflow execution, and the first web workflow authoring surfaces.

Architecture: Use a TypeScript monorepo with a React web app, a Node.js API control plane, and a separate Node.js worker. Use MongoDB as the only database, object storage abstraction for cloud storage or MinIO, and a local scheduler with Python and Docker executor contracts.

Tech Stack: pnpm workspace, React, TypeScript, React Flow, NestJS, Mongoose, MongoDB, Docker Compose, Python runtime hooks, Python unittest, and Node 22 built-in test runner with TypeScript stripping for early package-level tests

---

Progress Notes

• 2026-03-26: Tasks 1 and 2 are complete and committed.
• 2026-03-26: Tasks 3 through 6 are implemented against in-memory V1 control-plane services so the API and worker contracts can stabilize before persistence and framework wiring are deepened.
• 2026-03-26: Package-level verification continues to use the Node 22 built-in test runner with direct file targets such as pnpm --filter api test test/projects.e2e-spec.ts and pnpm --filter worker test test/task-runner.spec.ts.

---

Task 1: Bootstrap The Monorepo And Runtime Skeleton

Files:

• Create: package.json
• Create: pnpm-workspace.yaml
• Create: tsconfig.base.json
• Create: apps/web/package.json
• Create: apps/api/package.json
• Create: apps/worker/package.json
• Create: docker-compose.yml
• Create: .env.example
• Test: tests/test_repo_structure.py

Step 1: Write the failing test

Create tests/test_repo_structure.py to assert the repository contains the expected top-level app folders and root workspace files.

Step 2: Run test to verify it fails

Run:

python3 -m unittest tests/test_repo_structure.py -v

Expected: FAIL because the monorepo files and app folders do not exist yet.

Step 3: Write minimal implementation

Create the pnpm workspace root, app package manifests, root TypeScript config, .env.example, and docker-compose.yml with services for:

• web
• api
• worker
• mongo
• minio

Keep the first version minimal. Do not add extra infra services that are not required by the design.

Step 4: Run test to verify it passes

Run:

python3 -m unittest tests/test_repo_structure.py -v

Expected: PASS

Step 5: Commit

git add package.json pnpm-workspace.yaml tsconfig.base.json apps docker-compose.yml .env.example tests/test_repo_structure.py
git commit -m ":tada: bootstrap workspace and runtime skeleton"

Task 2: Create Shared Domain Contracts And Mongo Setup

Files:

• Create: packages/contracts/package.json
• Create: packages/contracts/src/domain.ts
• Create: apps/api/src/common/mongo/mongo.module.ts
• Create: apps/api/src/common/mongo/schemas/workspace.schema.ts
• Create: apps/api/src/common/mongo/schemas/project.schema.ts
• Create: apps/api/src/common/mongo/schemas/asset.schema.ts
• Create: apps/api/src/common/mongo/schemas/workflow.schema.ts
• Test: apps/api/test/domain-contracts.spec.ts

Step 1: Write the failing test

Create apps/api/test/domain-contracts.spec.ts asserting:

• workspace types include personal and team
• asset types include raw and dataset-style sources
• workflow status values match the design docs

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/domain-contracts.spec.ts

Expected: FAIL because contracts and schemas are missing.

Step 3: Write minimal implementation

Create shared domain enums and base Mongo schema definitions for:

• workspaces
• projects
• assets
• workflow definitions

Add a minimal Mongo module in the API app using environment-based connection config.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/domain-contracts.spec.ts

Expected: PASS

Step 5: Commit

git add packages/contracts apps/api/src/common apps/api/test/domain-contracts.spec.ts
git commit -m ":sparkles: add shared domain contracts and mongo setup"

Task 3: Implement Identity, Workspace, And Project APIs

Files:

• Create: apps/api/src/modules/auth/auth.module.ts
• Create: apps/api/src/modules/auth/auth.controller.ts
• Create: apps/api/src/modules/workspaces/workspaces.module.ts
• Create: apps/api/src/modules/workspaces/workspaces.controller.ts
• Create: apps/api/src/modules/projects/projects.module.ts
• Create: apps/api/src/modules/projects/projects.controller.ts
• Create: apps/api/src/modules/projects/projects.service.ts
• Test: apps/api/test/projects.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/projects.e2e-spec.ts covering:

• create personal workspace bootstrap flow
• create project under a workspace
• reject project creation without a workspace id

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/projects.e2e-spec.ts

Expected: FAIL because the modules and endpoints do not exist yet.

Step 3: Write minimal implementation

Implement:

• development-safe auth stub or local auth module
• workspace creation and listing
• project creation and listing
• basic membership checks sufficient for V1 local development

Do not build a full production auth stack before the API shape is stable.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/projects.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/auth apps/api/src/modules/workspaces apps/api/src/modules/projects apps/api/test/projects.e2e-spec.ts
git commit -m ":sparkles: add workspace and project APIs"

Task 4: Implement Asset Ingestion, Storage Abstraction, And Probe Metadata

Files:

• Create: apps/api/src/modules/storage/storage.module.ts
• Create: apps/api/src/modules/storage/storage.service.ts
• Create: apps/api/src/modules/assets/assets.module.ts
• Create: apps/api/src/modules/assets/assets.controller.ts
• Create: apps/api/src/modules/assets/assets.service.ts
• Create: apps/api/src/modules/assets/probe/probe.service.ts
• Create: apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts
• Test: apps/api/test/assets.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/assets.e2e-spec.ts covering:

• register an uploaded asset record
• create a probe report for a raw asset
• return recommended next actions from probe metadata

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/assets.e2e-spec.ts

Expected: FAIL because asset ingestion and probe services are missing.

Step 3: Write minimal implementation

Implement:

• storage abstraction interface
• MinIO/S3-compatible config contract
• asset create/list/detail endpoints
• probe-report persistence
• placeholder probe logic for directory and archive summaries

Do not build full binary upload optimization yet. First make the metadata contract stable.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/assets.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/storage apps/api/src/modules/assets apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts apps/api/test/assets.e2e-spec.ts
git commit -m ":truck: add asset ingestion and probe metadata flow"

Task 5: Implement Workflow Definitions, Versions, Runs, And Tasks

Files:

• Create: apps/api/src/modules/workflows/workflows.module.ts
• Create: apps/api/src/modules/workflows/workflows.controller.ts
• Create: apps/api/src/modules/workflows/workflows.service.ts
• Create: apps/api/src/modules/runs/runs.module.ts
• Create: apps/api/src/modules/runs/runs.controller.ts
• Create: apps/api/src/modules/runs/runs.service.ts
• Create: apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts
• Create: apps/api/src/common/mongo/schemas/workflow-run.schema.ts
• Create: apps/api/src/common/mongo/schemas/run-task.schema.ts
• Test: apps/api/test/workflow-runs.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/workflow-runs.e2e-spec.ts covering:

• create workflow definition
• save workflow version
• create workflow run from saved version
• generate initial run tasks for ready nodes

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/workflow-runs.e2e-spec.ts

Expected: FAIL because workflow versioning and run creation do not exist yet.

Step 3: Write minimal implementation

Implement:

• workflow definition head record
• immutable workflow version snapshots
• run creation from a workflow version
• initial DAG compilation for simple source-to-transform chains
• run task persistence

Keep V1 graph compilation simple. Support sequential edges first, then one-level branching.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/workflow-runs.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/workflows apps/api/src/modules/runs apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts apps/api/src/common/mongo/schemas/workflow-run.schema.ts apps/api/src/common/mongo/schemas/run-task.schema.ts apps/api/test/workflow-runs.e2e-spec.ts
git commit -m ":sparkles: add workflow versioning and run records"

Task 6: Add The Worker, Local Scheduler, And Executor Contracts

Files:

• Create: apps/worker/src/main.ts
• Create: apps/worker/src/runner/task-runner.ts
• Create: apps/worker/src/scheduler/local-scheduler.ts
• Create: apps/worker/src/executors/python-executor.ts
• Create: apps/worker/src/executors/docker-executor.ts
• Create: apps/worker/src/executors/http-executor.ts
• Create: apps/worker/src/contracts/execution-context.ts
• Test: apps/worker/test/task-runner.spec.ts

Step 1: Write the failing test

Create apps/worker/test/task-runner.spec.ts covering:

• worker loads pending tasks
• worker marks task running then success
• worker chooses executor based on node runtime config

Step 2: Run test to verify it fails

Run:

pnpm --filter worker test task-runner.spec.ts

Expected: FAIL because the worker runtime does not exist yet.

Step 3: Write minimal implementation

Implement:

• worker bootstrap
• polling or queue-backed local scheduler
• execution context builder
• stub Python, Docker, and HTTP executors
• task status transitions

Do not implement full Docker isolation logic in one step. First lock the runtime interfaces and transitions.

Step 4: Run test to verify it passes

Run:

pnpm --filter worker test task-runner.spec.ts

Expected: PASS

Step 5: Commit

git add apps/worker apps/api/src/modules/runs apps/worker/test/task-runner.spec.ts
git commit -m ":construction_worker: add local worker and executor contracts"

Task 7: Build The Web Shell, Workspace Flow, And Asset Workspace

Files:

• Create: apps/web/src/main.tsx
• Create: apps/web/src/app/router.tsx
• Create: apps/web/src/features/layout/app-shell.tsx
• Create: apps/web/src/features/workspaces/workspace-switcher.tsx
• Create: apps/web/src/features/projects/project-selector.tsx
• Create: apps/web/src/features/assets/assets-page.tsx
• Create: apps/web/src/features/assets/asset-detail-page.tsx
• Create: apps/web/src/features/assets/components/asset-list.tsx
• Create: apps/web/src/features/assets/components/asset-summary-panel.tsx
• Test: apps/web/src/features/assets/assets-page.test.tsx

Step 1: Write the failing test

Create apps/web/src/features/assets/assets-page.test.tsx covering:

• app shell renders primary navigation
• assets page renders asset rows from API data
• asset detail page renders probe summary

Step 2: Run test to verify it fails

Run:

pnpm --filter web test assets-page.test.tsx

Expected: FAIL because the web app shell and pages do not exist yet.

Step 3: Write minimal implementation

Implement:

• web app bootstrap
• primary navigation matching the design docs
• workspace/project header controls
• asset list page
• asset detail page with summary and action buttons

Defer advanced preview renderers. Start with structured metadata and simple detail views.

Step 4: Run test to verify it passes

Run:

pnpm --filter web test assets-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/web apps/web/src/features/assets/assets-page.test.tsx
git commit -m ":sparkles: add web shell and asset workspace"

Task 8: Build Canvas Authoring, Run Detail, And First Workflow Actions

Files:

• Create: apps/web/src/features/workflows/workflows-page.tsx
• Create: apps/web/src/features/workflows/workflow-editor-page.tsx
• Create: apps/web/src/features/workflows/components/node-library.tsx
• Create: apps/web/src/features/workflows/components/workflow-canvas.tsx
• Create: apps/web/src/features/workflows/components/node-config-panel.tsx
• Create: apps/web/src/features/runs/run-detail-page.tsx
• Create: apps/web/src/features/runs/components/run-graph-view.tsx
• Create: apps/web/src/features/runs/components/task-log-panel.tsx
• Test: apps/web/src/features/workflows/workflow-editor-page.test.tsx

Step 1: Write the failing test

Create apps/web/src/features/workflows/workflow-editor-page.test.tsx covering:

• node library renders categories
• node config panel opens when a node is selected
• run detail view shows node status badges from run data

Step 2: Run test to verify it fails

Run:

pnpm --filter web test workflow-editor-page.test.tsx

Expected: FAIL because the workflow editor and run detail pages do not exist yet.

Step 3: Write minimal implementation

Implement:

• workflow list page
• workflow editor page using React Flow
• left node library, center canvas, right config panel
• save workflow version action
• trigger workflow run action
• run detail page with graph and selected-node log panel

Keep the first editor scoped to V1 node categories and schema-driven config rendering.

Step 4: Run test to verify it passes

Run:

pnpm --filter web test workflow-editor-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/web/src/features/workflows apps/web/src/features/runs apps/web/src/features/workflows/workflow-editor-page.test.tsx
git commit -m ":sparkles: add canvas workflow editor and run detail pages"

Task 9: Add Preview Surface, Delivery Nodes, And MVP Integration

Files:

• Create: apps/api/src/modules/artifacts/artifacts.module.ts
• Create: apps/api/src/modules/artifacts/artifacts.controller.ts
• Create: apps/api/src/modules/artifacts/artifacts.service.ts
• Create: apps/web/src/features/explore/explore-page.tsx
• Create: apps/web/src/features/explore/renderers/json-renderer.tsx
• Create: apps/web/src/features/explore/renderers/video-renderer.tsx
• Create: apps/web/src/features/explore/renderers/directory-renderer.tsx
• Create: apps/api/src/modules/plugins/builtin/delivery-nodes.ts
• Test: apps/api/test/artifacts.e2e-spec.ts
• Test: apps/web/src/features/explore/explore-page.test.tsx

Step 1: Write the failing tests

Create:

• apps/api/test/artifacts.e2e-spec.ts for artifact retrieval by producer
• apps/web/src/features/explore/explore-page.test.tsx for opening and rendering supported artifact types

Step 2: Run tests to verify they fail

Run:

pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx

Expected: FAIL because artifact APIs and explore renderers do not exist yet.

Step 3: Write minimal implementation

Implement:

• artifact module and lookup endpoints
• explore page
• JSON, directory, and video renderers
• built-in delivery-normalization node definitions for the V1 business path

Do not implement the full renderer plugin platform yet. Start with built-ins and stable renderer contracts.

Step 4: Run tests to verify they pass

Run:

pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/artifacts apps/api/src/modules/plugins/builtin/delivery-nodes.ts apps/api/test/artifacts.e2e-spec.ts apps/web/src/features/explore apps/web/src/features/explore/explore-page.test.tsx
git commit -m ":package: add explore surface and delivery artifacts"

Task 10: Harden Guardrails, Docs, And Developer Entry Commands

Files:

• Modify: CONTRIBUTING.md
• Modify: docs/development-workflow.md
• Modify: design/03-workflows/workflow-execution-model.md
• Modify: design/05-data/mongodb-data-model.md
• Create: Makefile
• Create: README.md
• Test: tests/test_dev_commands.py

Step 1: Write the failing test

Create tests/test_dev_commands.py asserting:

• Makefile exposes expected local commands
• README.md documents bootstrap, hooks, test, and local run commands

Step 2: Run test to verify it fails

Run:

python3 -m unittest tests/test_dev_commands.py -v

Expected: FAIL because developer entry commands are not documented yet.

Step 3: Write minimal implementation

Add:

• make bootstrap
• make test
• make dev-api
• make dev-web
• make dev-worker
• make guardrails

Document the developer flow in README.md and update design docs if implementation decisions changed during Tasks 1-9.

Step 4: Run test to verify it passes

Run:

python3 -m unittest tests/test_dev_commands.py -v

Expected: PASS

Step 5: Commit

git add CONTRIBUTING.md docs/development-workflow.md design/03-workflows/workflow-execution-model.md design/05-data/mongodb-data-model.md Makefile README.md tests/test_dev_commands.py
git commit -m ":memo: add developer entry commands and bootstrap docs"

Exit Criteria

The first implementation pass is complete when:

• a user can create a workspace and project
• a raw asset can be registered and probed
• a workflow can be created, versioned, and executed locally
• run tasks produce observable status and artifacts
• the web app exposes assets, workflows, runs, and basic explore views
• guardrails for docs, hooks, commit messages, and CI remain green

Notes

• Keep commits small and use the repository gitmoji + English commit policy.
• Update design files in the same task where behavior or architecture changes.
• Do not add training execution before the V1 data workflow loop is stable.