forked from longtao.wu/droboticflow
15 KiB
15 KiB
Test Infrastructure & E2E Verification Specification
droboticflow Dual-Track Workflow Execution Isolation (Debug Run vs Production Run)
1. Test Philosophy
The droboticflow Dual-Track Workflow Execution Isolation test architecture adheres to an opaque-box, requirement-driven, deterministic verification methodology.
1.1 Core Principles
- Opaque-Box & Interface-Centric: Tests interact exclusively through documented public APIs, domain models, CLI tools, and UI state contract helpers. Internal implementation private state is never monkey-patched.
- Requirement-Driven & Traceable: Every test case traces directly to requirements defined in
ORIGINAL_REQUEST.md(R1 through R4) and acceptance criteria inPROJECT.md. - Rigorous Multi-Method Test Design:
- Category-Partition Method: Parameter spaces across execution triggers, node types, payload kinds, and authentication roles are partitioned into mutually exclusive categories.
- Boundary Value Analysis (BVA): Limits such as inline payload sizes (64KB boundary), concurrency caps (
maxConcurrentRuns), 0-count edge cases, division-by-zero guards in workforce calculations, and keyset cursor bounds are tested at and beyond their thresholds. - Pairwise Combinatorial Testing: Combinations of execution mode (
production/debug), workflow state (draft/released), payload mode (inline/ref), environment (poc/dev), and target node selection (all/single/subtree) are comprehensively exercised. - Real-World Workload Simulation: Realistic end-to-end industrial pipelines (autonomous driving vision ingestion, robotics policy training, multi-vendor annotation workforce audit) are tested end-to-end.
- Anti-Cheating & Test Integrity Guarantee:
- Tests do not contain dummy mocks, hardcoded return shortcuts, or tautological assertions.
- Expected outputs are derived authoritatively from immutable release specifications, contract validators, and domain rules.
- All tests execute hermetically in offline mode with deterministic execution times.
2. Feature Inventory (F1 – F13)
| # | Feature | Category | Description | Inputs | Outputs | Error Behavior | Tier 1 (Coverage) | Tier 2 (Boundaries) | Tier 3 (Pairwise) | Tier 4 (Workload) |
|---|---|---|---|---|---|---|---|---|---|---|
| F1 | RunMode Domain & Storage Models | Domain / Store | RunMode (production / debug) and provenance fields (ReleaseID, WorkflowVersion, GitCommitSha, ImageDigest, NodeOverrides, DebugTarget) in RunRecord, RunSummary, RunSummaryQuery, MemoryStore & MongoStore |
Store queries, domain payloads | RunRecord, RunSummary structs with immutable provenance |
Schema validation rejection on invalid mode | [x] (5 tests) | [x] (3 tests) | [x] (2 tests) | [x] (2 tests) |
| F2 | Metrics & Worklog Event Domain Models | Domain / Store | Tagging AnnotationWorklogEventRecord.Mode, DeliveryPackageRecord.Mode, and ExecutionArtifact.Metadata.IsTemporary |
Worklog events, package records, artifacts | Persisted entities with mode metadata and temporary tags | Serialization rejection | [x] (5 tests) | [x] (2 tests) | [x] (2 tests) | [x] (2 tests) |
| F3 | Production Run Release API & Security Gate | Backend API | POST /api/workflow-releases/:id/runs strictly bound to WorkflowReleaseRecord. Returns 422 production_input_forbidden on non-root inputs, node overrides, or parameter tampering |
{ "trigger": "manual", "inputs": { [portKey]: ArtifactPayload } } |
202 Accepted with RunRecord (mode: "production") |
422 Unprocessable Entity (production_input_forbidden), 404 Not Found |
[x] (5 tests) | [x] (5 tests) | [x] (3 tests) | [x] (2 tests) |
| F4 | Debug Run API & Node Overrides | Backend API | POST /api/workflows/:id/run supporting draft execution, transient nodeOverrides, and debugTarget |
{ "trigger": "manual", "mode": "debug", "inputs": {...}, "nodeOverrides": {...}, "debugTarget": {...} } |
202 Accepted with RunRecord (mode: "debug") |
400 Bad Request, 404 Not Found |
[x] (5 tests) | [x] (4 tests) | [x] (3 tests) | [x] (2 tests) |
| F5 | Worker Runtime Transient Overrides & Single-Node | Worker Runtime | Worker scheduler executes single nodes / subtrees and applies transient per-run node overrides without mutating DB workflow records | RunRecord with NodeOverrides and DebugTarget |
Executed node results, logs, and artifacts | Node failure captured in RunRecord.Failure |
[x] (5 tests) | [x] (3 tests) | [x] (2 tests) | [x] (2 tests) |
| F6 | Workforce Metrics Isolation | Metrics / Audit | Filtering out debug events from QualityPassRate, AcceptancePassRate, ReworkRate, DurationSeconds, and OutputCount in dashboard_workforce.go |
Organization ID, date range, vendor scope | DashboardSummary reflecting only production worklog events |
Division-by-zero protection (returns null rates) |
[x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
| F7 | DeliveryPackage & Artifact Isolation | Delivery / Audit | Isolate formal DeliveryPackageRecord creation to production runs; tag debug artifacts as temporary (isTemporary: true) |
Human release partition events, child runs | Formal delivery packages created for production only; debug artifacts tagged temporary | Conflict on version mismatch | [x] (5 tests) | [x] (2 tests) | [x] (3 tests) | [x] (2 tests) |
| F8 | Frontend Action Bar Dual Triggers | Frontend UI | Orange 【调试运行】 (Debug Run) and Green/Blue 【正式运行】 (Production Run) buttons in editor toolbar with state guards | Workflow state, release status, active run status | Interactive trigger buttons with contextual labels, tooltips, and styles | Disabled button state with descriptive reason | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
| F9 | Production Data Input Panel with Locked Nodes | Frontend UI | Modal exposing root workflow inputs (syncRootWorkflowInputs); intermediate nodes locked with "🔒 已锁定不可篡改" |
User clicks Production Run | Form exposing only root inputs; collapsed read-only intermediate nodes | Validation error on missing required inputs or non-root bindings | [x] (5 tests) | [x] (4 tests) | [x] (3 tests) | [x] (2 tests) |
| F10 | Developer Debug Console UI | Frontend UI | Modal supporting node parameter overrides, dev/poc env selector, and step/single-node execution | User clicks Debug Run | Form with override editors, target selector, and dispatch action | Validation error on malformed JSON override | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
| F11 | Monitoring Center & History Mode Filters | Frontend UI | mode filter (all / production / debug) and badges on runs table in Monitoring.tsx and ProjectDetail.tsx |
Filter dropdown change / URL search query | Filtered run list with mode badges (production / debug) |
Fallback to empty list or all on invalid param |
[x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
| F12 | Comprehensive E2E Testing Suite | Architecture / QA | Requirements-driven 4-tier test suite covering R1-R4 acceptance criteria and cross-cutting contracts | Full test suite execution | 100% passing test reports across Go, Node.js, Python | Test failure on regression | [x] (5 tests) | [x] (5 tests) | [x] (4 tests) | [x] (5 tests) |
| F13 | Final Integration & Adversarial Hardening | Architecture / QA | Pass 100% E2E tests + Tier 5 white-box adversarial verification (tampering, race conditions, size limits) | Adversarial test vectors and malformed payloads | Clean rejection with standard error codes (422, 400, 409) |
Non-zero exit code on unhandled panic | [x] (5 tests) | [x] (5 tests) | [x] (4 tests) | [x] (5 tests) |
3. Test Architecture
3.1 Test Runners & Frameworks
- Frontend Test Runner: Node.js native test runner (
node:test) with strict assertions (node:assert/strict).- Command:
npm run test:frontend(node --test tests/frontend/*.test.mjs) - Runtime: Zero dependencies, ESM imports, sub-second execution (~500ms for 47+ suites).
- Command:
- Backend Go Test Runner: Go standard
testingpackage withhttptestand in-memory storage.- Command:
go test ./internal/... - Runtime: Sub-10s execution across all domain, store, api, and worker suites.
- Command:
- Python Test Runner:
pytestfor operator runner and backend scripts.- Command:
npm run test:python
- Command:
- Rust Test Runner:
cargo testfor drosync tool workspace.- Command:
npm run test:drosync
- Command:
3.2 Directory Layout
droboticflow/
├── TEST_INFRA.md # Master Test Infrastructure Specification
├── tests/
│ ├── frontend/ # Node.js native test runner suites
│ │ ├── workflow-dual-track.test.mjs # Comprehensive Dual-Track Frontend & Contract Test Suite
│ │ ├── workflow-inputs.test.mjs # Root inputs & payload parsing tests
│ │ ├── workflow-editor.test.mjs # Editor lock states & button action states
│ │ ├── workflow-run.test.mjs # Run display states, node decorations, edge states
│ │ └── ... (45+ existing suites)
│ └── backend/ # Python operator runner tests
├── internal/
│ ├── domain/ # Go domain models & contract validation tests
│ │ ├── types_test.go
│ │ └── workflow_inputs_test.go
│ ├── store/ # MemoryStore & MongoStore persistence tests
│ │ ├── memory_test.go
│ │ └── workflow_releases_test.go
│ ├── api/ # HTTP API & Security Gate integration tests
│ │ ├── server_test.go
│ │ ├── workflow_releases_test.go
│ │ └── dashboard_workforce_test.go
│ └── worker/ # Worker runtime & execution scheduler tests
│ ├── runtime_test.go
│ └── human_release_test.go
└── scripts/ # Acceptance & deployment scripts
3.3 Test Harness & Test Isolation
- In-Memory Store Harness (
store.NewMemoryStore()): Fully simulates MongoDB collections, indexed queries, optimistic locking, and atomic updates. - In-Process HTTP Test Server:
api.NewHandler(store, ai)coupled withnet/http/httptestexecutes full HTTP request/response lifecycles synchronously in memory. - Frontend Pure Contract Functions: All frontend state transformations, input builders, lock validators, and payload parsers are isolated in pure ES modules in
src/lib/for hermetic testing.
4. Real-World Application Scenarios (Tier 4)
Scenario 1: Autonomous Driving Multi-Camera Vision Ingestion & Frame Extraction Pipeline
- Topology: 5-Node DAG:
BOS Video Stream Source->LiDAR Synchronizer->YOLO/WILOR 3D Handpose & Headpose Detector->Human QA Gate->Policy Dataset Export. - Debug Track: AI engineer tests newly trained YOLO/WILOR checkpoint weights on a 20-frame sample clip in draft canvas. Engineer sets hyperparameter overrides (
iou_threshold: 0.65,box_score_threshold: 0.45) in Developer Debug Console and executes a debug run. The run completes with transient parameters, produced test artifacts are taggedisTemporary: true, and 0 test entries enter supplier metrics. - Production Track: Release v1.0.0 is published with immutable image digests. Operator triggers production run for a 100,000-frame customer batch. Production Input Panel strictly locks intermediate model weights and exposes only the BOS raw bucket root input. Worker executes the pipeline, human QA gate approves frames, formal
DeliveryPackageRecordis generated, and workforce dashboard updates with clean production metrics.
Scenario 2: Multi-Vendor Robotic Grasping Dataset Annotation & Workforce Performance Audit
- Context: Enterprise data operations outsourcing annotation to two vendors (Vendor Alpha, Vendor Beta). Strict financial reconciliation requires auditing annotator output, quality pass rates, and rework counts without test distortion.
- Scenario: 10 new annotators practice annotation tools using Debug Runs, generating 50 trial submissions with 20 rejections. Concurrently, formal production batches execute with 500 annotations (Vendor Alpha 95% pass rate, Vendor Beta 88% pass rate). Dashboard reconciles invoices based strictly on
mode: "production"worklogs, proving trial debug runs caused zero metric contamination.
Scenario 3: HoloBrain / Structured Model AI Policy Training & Checkpoint Evaluation
- Context: Robotics policy training taking demonstration trajectories, running HoloBrain/ACT policy training across GPU nodes, and evaluating model checkpoint loss.
- Scenario: ML engineer runs a 2-minute draft smoke test using Debug Console with overrides (
batch_size: 4,epochs: 1) and single-node debug on the evaluation node. Once verified, formal scheduled training is launched on a published release with locked Kaniko container digests and GPU resource telemetry.
Scenario 4: Urgent Hotfix & Production Failure Regression Debugging
- Context: Production batch fails at intermediate transformation node due to unexpected datetime format in raw data.
- Scenario: Incident responder views failed production run in Monitoring Center (
mode=production), clones workflow into draft canvas, applies regex fix, and uses Developer Debug Console with single-node debug target to verify the fix immediately on draft data. Upon verification, responder publishes Release v2 and launches production rerun without corrupting the historical failed run audit record.
Scenario 5: Multi-Tenant High-Concurrency Dual-Track Execution & Data Cleanliness
- Context: Tenant Alpha runs high-throughput production pipelines while Tenant Beta concurrently runs dozens of interactive debug runs with heavy parameter overrides.
- Scenario: Both tenants query
/api/runs,/api/dashboard, and/api/delivery-packages. Strict tenant and mode isolation guarantees zero cross-tenant leakage and zero debug-to-production data pollution.
5. Coverage Thresholds & Verification Rules
| Tier | Name | Target Scope | Minimum Threshold | Status |
|---|---|---|---|---|
| Tier 1 | Feature Coverage | Primary behavior & happy path for all features F1–F13 | >= 5 test cases per feature | MET (65+ tests) |
| Tier 2 | Boundary & Corner Cases | Boundary value analysis, invalid inputs, tampering security gates, division-by-zero | >= 5 test cases per feature with boundaries | MET (45+ tests) |
| Tier 3 | Cross-Feature Combinations | Pairwise feature interactions, full lifecycle transitions, multi-version transitions | Pairwise combinations across modes & states | MET (15+ tests) |
| Tier 4 | Real-World Workload Scenarios | Complex multi-step industrial workloads (AD, Robotics, Workforce Audit) | >= 5 comprehensive application scenarios | MET (5 scenarios) |
6. Commands to Run All Tests
# 1. Run all frontend test suites (including Dual-Track Isolation)
npm run test:frontend
# 2. Run dual-track specific frontend test suite directly
node --test tests/frontend/workflow-dual-track.test.mjs
# 3. Run all backend Go test suites
go test -v ./internal/domain/... ./internal/store/... ./internal/api/... ./internal/worker/...
# 4. Run full local project verification suite (lint, format, backend, python, frontend, build)
npm run verify:local