EmboFlow/docs/plans/2026-03-26-emboflow-v1-foundation-and-mvp.md

EmboFlow V1 Foundation And MVP Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Build the first usable EmboFlow increment: workspace-aware raw asset ingestion, workflow definition/versioning, local workflow execution, and the first web workflow authoring surfaces.

Architecture: Use a TypeScript monorepo with a React web app, a Node.js API control plane, and a separate Node.js worker. Use MongoDB as the only database, object storage abstraction for cloud storage or MinIO, and a local scheduler with Python and Docker executor contracts.

Tech Stack: pnpm workspace, React, TypeScript, React Flow, NestJS, Mongoose, MongoDB, Docker Compose, Python runtime hooks, Python unittest, and tsx --test for package-level TypeScript tests, including Mongo-backed runtime integration coverage

---

Progress Notes

• 2026-03-26: Tasks 1 and 2 are complete and committed.
• 2026-03-26: Tasks 3 through 6 are implemented against in-memory V1 control-plane services so the API and worker contracts can stabilize before persistence and framework wiring are deepened.
• 2026-03-26: Package-level verification continues to use the Node 22 built-in test runner with direct file targets such as pnpm --filter api test test/projects.e2e-spec.ts and pnpm --filter worker test test/task-runner.spec.ts.
• 2026-03-26: Tasks 7 through 10 add the first web shell, workflow editor surfaces, artifact explore renderers, developer entry commands, and CI/pre-push test execution through make test.
• 2026-03-26: The next runtime pass adds a Mongo-backed HTTP API, a real React and Vite web runtime, and local data validation against /Users/longtaowu/workspace/emboldata/data.
• 2026-03-26: The follow-up runtime pass adds Mongo-backed HTTP integration tests and converts the workflow editor from a fixed sample graph to a persisted draft-and-version model.
• 2026-03-26: The current runtime pass binds workflow runs to registered project assets so run snapshots, run tasks, worker execution context, and the editor all agree on the concrete input asset being processed.
• 2026-03-26: The current UI/runtime pass turns Runs into a real project-scoped workspace with run history queries, active-run polling, and task artifact links into Explore.
• 2026-03-27: The current observability pass persists task execution summaries, timestamps, log lines, and result previews on Mongo-backed run_tasks, and surfaces those fields in the React run detail view.
• 2026-03-27: The current follow-up observability pass adds persisted stdout/stderr fields on run_tasks plus aggregated run summaries, durations, and task counts on workflow_runs.
• 2026-03-27: The current run-control pass adds run cancellation, run retry from immutable snapshots, and task retry for failed/cancelled nodes with downstream reset semantics.
• 2026-03-27: The current runtime-config pass freezes per-node executor config into workflow_runs and run_tasks, exposes runtime editing controls in the React workflow editor, and executes trusted-local Python code hooks from the task snapshot.

---

Task 1: Bootstrap The Monorepo And Runtime Skeleton

Files:

• Create: package.json
• Create: pnpm-workspace.yaml
• Create: tsconfig.base.json
• Create: apps/web/package.json
• Create: apps/api/package.json
• Create: apps/worker/package.json
• Create: docker-compose.yml
• Create: .env.example
• Test: tests/test_repo_structure.py

Step 1: Write the failing test

Create tests/test_repo_structure.py to assert the repository contains the expected top-level app folders and root workspace files.

Step 2: Run test to verify it fails

Run:

python3 -m unittest tests/test_repo_structure.py -v

Expected: FAIL because the monorepo files and app folders do not exist yet.

Step 3: Write minimal implementation

Create the pnpm workspace root, app package manifests, root TypeScript config, .env.example, and docker-compose.yml with services for:

• web
• api
• worker
• mongo
• minio

Keep the first version minimal. Do not add extra infra services that are not required by the design.

Step 4: Run test to verify it passes

Run:

python3 -m unittest tests/test_repo_structure.py -v

Expected: PASS

Step 5: Commit

git add package.json pnpm-workspace.yaml tsconfig.base.json apps docker-compose.yml .env.example tests/test_repo_structure.py
git commit -m ":tada: bootstrap workspace and runtime skeleton"

Task 2: Create Shared Domain Contracts And Mongo Setup

Files:

• Create: packages/contracts/package.json
• Create: packages/contracts/src/domain.ts
• Create: apps/api/src/common/mongo/mongo.module.ts
• Create: apps/api/src/common/mongo/schemas/workspace.schema.ts
• Create: apps/api/src/common/mongo/schemas/project.schema.ts
• Create: apps/api/src/common/mongo/schemas/asset.schema.ts
• Create: apps/api/src/common/mongo/schemas/workflow.schema.ts
• Test: apps/api/test/domain-contracts.spec.ts

Step 1: Write the failing test

Create apps/api/test/domain-contracts.spec.ts asserting:

• workspace types include personal and team
• asset types include raw and dataset-style sources
• workflow status values match the design docs

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/domain-contracts.spec.ts

Expected: FAIL because contracts and schemas are missing.

Step 3: Write minimal implementation

Create shared domain enums and base Mongo schema definitions for:

• workspaces
• projects
• assets
• workflow definitions

Add a minimal Mongo module in the API app using environment-based connection config.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/domain-contracts.spec.ts

Expected: PASS

Step 5: Commit

git add packages/contracts apps/api/src/common apps/api/test/domain-contracts.spec.ts
git commit -m ":sparkles: add shared domain contracts and mongo setup"

Task 3: Implement Identity, Workspace, And Project APIs

Files:

• Create: apps/api/src/modules/auth/auth.module.ts
• Create: apps/api/src/modules/auth/auth.controller.ts
• Create: apps/api/src/modules/workspaces/workspaces.module.ts
• Create: apps/api/src/modules/workspaces/workspaces.controller.ts
• Create: apps/api/src/modules/projects/projects.module.ts
• Create: apps/api/src/modules/projects/projects.controller.ts
• Create: apps/api/src/modules/projects/projects.service.ts
• Test: apps/api/test/projects.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/projects.e2e-spec.ts covering:

• create personal workspace bootstrap flow
• create project under a workspace
• reject project creation without a workspace id

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/projects.e2e-spec.ts

Expected: FAIL because the modules and endpoints do not exist yet.

Step 3: Write minimal implementation

Implement:

• development-safe auth stub or local auth module
• workspace creation and listing
• project creation and listing
• basic membership checks sufficient for V1 local development

Do not build a full production auth stack before the API shape is stable.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/projects.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/auth apps/api/src/modules/workspaces apps/api/src/modules/projects apps/api/test/projects.e2e-spec.ts
git commit -m ":sparkles: add workspace and project APIs"

Task 4: Implement Asset Ingestion, Storage Abstraction, And Probe Metadata

Files:

• Create: apps/api/src/modules/storage/storage.module.ts
• Create: apps/api/src/modules/storage/storage.service.ts
• Create: apps/api/src/modules/assets/assets.module.ts
• Create: apps/api/src/modules/assets/assets.controller.ts
• Create: apps/api/src/modules/assets/assets.service.ts
• Create: apps/api/src/modules/assets/probe/probe.service.ts
• Create: apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts
• Test: apps/api/test/assets.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/assets.e2e-spec.ts covering:

• register an uploaded asset record
• create a probe report for a raw asset
• return recommended next actions from probe metadata

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/assets.e2e-spec.ts

Expected: FAIL because asset ingestion and probe services are missing.

Step 3: Write minimal implementation

Implement:

• storage abstraction interface
• MinIO/S3-compatible config contract
• asset create/list/detail endpoints
• probe-report persistence
• placeholder probe logic for directory and archive summaries

Do not build full binary upload optimization yet. First make the metadata contract stable.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/assets.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/storage apps/api/src/modules/assets apps/api/src/common/mongo/schemas/asset-probe-report.schema.ts apps/api/test/assets.e2e-spec.ts
git commit -m ":truck: add asset ingestion and probe metadata flow"

Task 5: Implement Workflow Definitions, Versions, Runs, And Tasks

Files:

• Create: apps/api/src/modules/workflows/workflows.module.ts
• Create: apps/api/src/modules/workflows/workflows.controller.ts
• Create: apps/api/src/modules/workflows/workflows.service.ts
• Create: apps/api/src/modules/runs/runs.module.ts
• Create: apps/api/src/modules/runs/runs.controller.ts
• Create: apps/api/src/modules/runs/runs.service.ts
• Create: apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts
• Create: apps/api/src/common/mongo/schemas/workflow-run.schema.ts
• Create: apps/api/src/common/mongo/schemas/run-task.schema.ts
• Test: apps/api/test/workflow-runs.e2e-spec.ts

Step 1: Write the failing test

Create apps/api/test/workflow-runs.e2e-spec.ts covering:

• create workflow definition
• save workflow version
• create workflow run from saved version
• generate initial run tasks for ready nodes

Step 2: Run test to verify it fails

Run:

pnpm --filter api test test/workflow-runs.e2e-spec.ts

Expected: FAIL because workflow versioning and run creation do not exist yet.

Step 3: Write minimal implementation

Implement:

• workflow definition head record
• immutable workflow version snapshots
• run creation from a workflow version
• initial DAG compilation for simple source-to-transform chains
• run task persistence

Keep V1 graph compilation simple. Support sequential edges first, then one-level branching.

Step 4: Run test to verify it passes

Run:

pnpm --filter api test test/workflow-runs.e2e-spec.ts

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/workflows apps/api/src/modules/runs apps/api/src/common/mongo/schemas/workflow-definition-version.schema.ts apps/api/src/common/mongo/schemas/workflow-run.schema.ts apps/api/src/common/mongo/schemas/run-task.schema.ts apps/api/test/workflow-runs.e2e-spec.ts
git commit -m ":sparkles: add workflow versioning and run records"

Task 6: Add The Worker, Local Scheduler, And Executor Contracts

Files:

• Create: apps/worker/src/main.ts
• Create: apps/worker/src/runner/task-runner.ts
• Create: apps/worker/src/scheduler/local-scheduler.ts
• Create: apps/worker/src/executors/python-executor.ts
• Create: apps/worker/src/executors/docker-executor.ts
• Create: apps/worker/src/executors/http-executor.ts
• Create: apps/worker/src/contracts/execution-context.ts
• Test: apps/worker/test/task-runner.spec.ts

Step 1: Write the failing test

Create apps/worker/test/task-runner.spec.ts covering:

• worker loads pending tasks
• worker marks task running then success
• worker chooses executor based on node runtime config

Step 2: Run test to verify it fails

Run:

pnpm --filter worker test task-runner.spec.ts

Expected: FAIL because the worker runtime does not exist yet.

Step 3: Write minimal implementation

Implement:

• worker bootstrap
• polling or queue-backed local scheduler
• execution context builder
• stub Python, Docker, and HTTP executors
• task status transitions

Do not implement full Docker isolation logic in one step. First lock the runtime interfaces and transitions.

Step 4: Run test to verify it passes

Run:

pnpm --filter worker test task-runner.spec.ts

Expected: PASS

Step 5: Commit

git add apps/worker apps/api/src/modules/runs apps/worker/test/task-runner.spec.ts
git commit -m ":construction_worker: add local worker and executor contracts"

Task 7: Build The Web Shell, Workspace Flow, And Asset Workspace

Files:

• Create: apps/web/src/main.tsx
• Create: apps/web/src/app/router.tsx
• Create: apps/web/src/features/layout/app-shell.tsx
• Create: apps/web/src/features/workspaces/workspace-switcher.tsx
• Create: apps/web/src/features/projects/project-selector.tsx
• Create: apps/web/src/features/assets/assets-page.tsx
• Create: apps/web/src/features/assets/asset-detail-page.tsx
• Create: apps/web/src/features/assets/components/asset-list.tsx
• Create: apps/web/src/features/assets/components/asset-summary-panel.tsx
• Test: apps/web/src/features/assets/assets-page.test.tsx

Step 1: Write the failing test

Create apps/web/src/features/assets/assets-page.test.tsx covering:

• app shell renders primary navigation
• assets page renders asset rows from API data
• asset detail page renders probe summary

Step 2: Run test to verify it fails

Run:

pnpm --filter web test assets-page.test.tsx

Expected: FAIL because the web app shell and pages do not exist yet.

Step 3: Write minimal implementation

Implement:

• web app bootstrap
• primary navigation matching the design docs
• workspace/project header controls
• asset list page
• asset detail page with summary and action buttons

Defer advanced preview renderers. Start with structured metadata and simple detail views.

Step 4: Run test to verify it passes

Run:

pnpm --filter web test assets-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/web apps/web/src/features/assets/assets-page.test.tsx
git commit -m ":sparkles: add web shell and asset workspace"

Task 8: Build Canvas Authoring, Run Detail, And First Workflow Actions

Files:

• Create: apps/web/src/features/workflows/workflows-page.tsx
• Create: apps/web/src/features/workflows/workflow-editor-page.tsx
• Create: apps/web/src/features/workflows/components/node-library.tsx
• Create: apps/web/src/features/workflows/components/workflow-canvas.tsx
• Create: apps/web/src/features/workflows/components/node-config-panel.tsx
• Create: apps/web/src/features/runs/run-detail-page.tsx
• Create: apps/web/src/features/runs/components/run-graph-view.tsx
• Create: apps/web/src/features/runs/components/task-log-panel.tsx
• Test: apps/web/src/features/workflows/workflow-editor-page.test.tsx

Step 1: Write the failing test

Create apps/web/src/features/workflows/workflow-editor-page.test.tsx covering:

• node library renders categories
• node config panel opens when a node is selected
• run detail view shows node status badges from run data

Step 2: Run test to verify it fails

Run:

pnpm --filter web test workflow-editor-page.test.tsx

Expected: FAIL because the workflow editor and run detail pages do not exist yet.

Step 3: Write minimal implementation

Implement:

• workflow list page
• workflow editor page using React Flow
• left node library, center canvas, right config panel
• save workflow version action
• trigger workflow run action
• run detail page with graph and selected-node log panel

Keep the first editor scoped to V1 node categories and schema-driven config rendering.

Step 4: Run test to verify it passes

Run:

pnpm --filter web test workflow-editor-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/web/src/features/workflows apps/web/src/features/runs apps/web/src/features/workflows/workflow-editor-page.test.tsx
git commit -m ":sparkles: add canvas workflow editor and run detail pages"

Task 9: Add Preview Surface, Delivery Nodes, And MVP Integration

Files:

• Create: apps/api/src/modules/artifacts/artifacts.module.ts
• Create: apps/api/src/modules/artifacts/artifacts.controller.ts
• Create: apps/api/src/modules/artifacts/artifacts.service.ts
• Create: apps/web/src/features/explore/explore-page.tsx
• Create: apps/web/src/features/explore/renderers/json-renderer.tsx
• Create: apps/web/src/features/explore/renderers/video-renderer.tsx
• Create: apps/web/src/features/explore/renderers/directory-renderer.tsx
• Create: apps/api/src/modules/plugins/builtin/delivery-nodes.ts
• Test: apps/api/test/artifacts.e2e-spec.ts
• Test: apps/web/src/features/explore/explore-page.test.tsx

Step 1: Write the failing tests

Create:

• apps/api/test/artifacts.e2e-spec.ts for artifact retrieval by producer
• apps/web/src/features/explore/explore-page.test.tsx for opening and rendering supported artifact types

Step 2: Run tests to verify they fail

Run:

pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx

Expected: FAIL because artifact APIs and explore renderers do not exist yet.

Step 3: Write minimal implementation

Implement:

• artifact module and lookup endpoints
• explore page
• JSON, directory, and video renderers
• built-in delivery-normalization node definitions for the V1 business path

Do not implement the full renderer plugin platform yet. Start with built-ins and stable renderer contracts.

Step 4: Run tests to verify they pass

Run:

pnpm --filter api test test/artifacts.e2e-spec.ts
pnpm --filter web test src/features/explore/explore-page.test.tsx

Expected: PASS

Step 5: Commit

git add apps/api/src/modules/artifacts apps/api/src/modules/plugins/builtin/delivery-nodes.ts apps/api/test/artifacts.e2e-spec.ts apps/web/src/features/explore apps/web/src/features/explore/explore-page.test.tsx
git commit -m ":package: add explore surface and delivery artifacts"

Task 10: Harden Guardrails, Docs, And Developer Entry Commands

Files:

• Modify: CONTRIBUTING.md
• Modify: docs/development-workflow.md
• Modify: design/03-workflows/workflow-execution-model.md
• Modify: design/05-data/mongodb-data-model.md
• Create: Makefile
• Create: README.md
• Test: tests/test_dev_commands.py

Step 1: Write the failing test

Create tests/test_dev_commands.py asserting:

• Makefile exposes expected local commands
• README.md documents bootstrap, hooks, test, and local run commands

Step 2: Run test to verify it fails

Run:

python3 -m unittest tests/test_dev_commands.py -v

Expected: FAIL because developer entry commands are not documented yet.

Step 3: Write minimal implementation

Add:

• make bootstrap
• make test
• make dev-api
• make dev-web
• make dev-worker
• make guardrails

Document the developer flow in README.md and update design docs if implementation decisions changed during Tasks 1-9.

Step 4: Run test to verify it passes

Run:

python3 -m unittest tests/test_dev_commands.py -v

Expected: PASS

Step 5: Commit

git add CONTRIBUTING.md docs/development-workflow.md design/03-workflows/workflow-execution-model.md design/05-data/mongodb-data-model.md Makefile README.md tests/test_dev_commands.py
git commit -m ":memo: add developer entry commands and bootstrap docs"

Exit Criteria

The first implementation pass is complete when:

• a user can create a workspace and project
• a raw asset can be registered and probed
• a workflow can be created, versioned, and executed locally
• run tasks produce observable status and artifacts
• the web app exposes assets, workflows, runs, and basic explore views
• guardrails for docs, hooks, commit messages, and CI remain green

Notes

• Keep commits small and use the repository gitmoji + English commit policy.
• Update design files in the same task where behavior or architecture changes.
• Do not add training execution before the V1 data workflow loop is stable.