6.0 KiB
6.0 KiB
E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5)
1. Test Philosophy
- Opaque-Box & Requirement-Driven: Tests are derived strictly from
ORIGINAL_REQUEST.mdand domain specifications, independent of internal module refactorings. - Deterministic & Hermetic: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies.
- 4-Tier Test Architecture: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4).
2. Feature Inventory
| # | Feature | Source (Requirement) | Tier 1 | Tier 2 | Tier 3 | Tier 4 |
|---|---|---|---|---|---|---|
| 1 | Video Face Blur Operator (operator-builtin-video-face-blur) |
ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 2 | BOS Ingestion & Orbbec Ego Normalization | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 3 | PTS Monotonic Continuity & Audio Sync Preservation | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 4 | 22-Class Convenience Store Taxonomy Presets | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
| 5 | Multi-Version Template Presets (v1–v4) & Release Binding | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
| 6 | Multi-Vendor Data & Task Access Isolation | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 7 | Anti-Duplicate Rework Workload Statistics (outputDelta=0) |
ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 8 | Task Name CRUD & Batch Partitioning | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 9 | WiLoR 5s Valid Event 0/1 Detection | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 10 | Qwen3-VL 200s Temporal Action Interval Localization | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 11 | Qwen3-VL 22-Class Multi-Label Video Classification | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 12 | AI Prelabel Draft Streaming to Timeline Editor | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 13 | ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
| 14 | Causal Inference Evaluation & Multi-Threshold Metrics | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
| 15 | Comparison Report Artifacts (HTML, metrics JSON) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
3. Test Architecture & Directory Layout
tests/
├── fixtures/
│ └── video_fixtures.py # Synthetic video, PTS CSV, and WAV audio generator
├── backend/
│ ├── swift_test_utils.py # ms-swift JSONL schema and leakage validation
│ ├── causal_eval_golden_vectors.json # Precomputed golden vectors for IoU, MAE, F1
│ ├── test_fanyu_r1_face_blur.py # R1: Face blur, PTS integrity, audio sync
│ ├── test_fanyu_r2_taxonomy.py # R2: 22-class taxonomy, alias, templates
│ ├── test_fanyu_r3_vendor_isolation.py # R3: Multi-vendor isolation, rework stats, task CRUD
│ ├── test_fanyu_r4_ai_prelabel.py # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class
│ ├── test_fanyu_r5_swift_causal_eval.py # R5: ms-swift export, causal eval, HTML reports
│ └── test_e2e_fanyu_pipeline.py # Full E2E pipeline integration test
frontend/
└── fanyu-pipeline.test.mjs # Frontend timeline hotkeys, phrases, and prelabel badges
internal/api/
└── fanyu_pipeline_e2e_test.go # Go HTTP REST API & workforce dashboard tests
4. 4-Tier Test Design Methodology & Coverage Thresholds
Tier 1: Feature Coverage (≥5 tests per feature)
- Minimum count:
15 \times 5 = 75tests across R1–R5. - Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0.
Tier 2: Boundary & Corner Cases (≥5 tests per feature)
- Minimum count:
15 \times 5 = 75tests across R1–R5. - Tests verify boundary conditions:
- R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels.
- R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection.
- R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (
outputDelta=0), division-by-zero on 0 worklogs. - R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings.
- R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits.
Tier 3: Cross-Feature Combinations & Lifecycle Transitions
- Minimum count: ≥15 cross-feature tests.
- Verifies pairwise and lifecycle interactions:
- Ingestion
\rightarrowNormalization\rightarrowFace Blur\rightarrowSlicing\rightarrowPackaging. - Slicing
\rightarrowAI Prelabeling\rightarrowRevision 0 Streaming\rightarrowHuman Timeline Edit\rightarrowRevision 1 Submit. - QA Rejection
\rightarrowRework Submission\rightarrowDashboard Reconciliation (outputDelta=0,reworkCount++). - Accepted Annotation Package
\rightarrowms-swift Export\rightarrowCausal Eval\rightarrowHTML Report Generation.
- Ingestion
Tier 4: Real-World Workload Scenarios
- Minimum count: ≥8 comprehensive scenarios.
- Realistic end-to-end industrial workloads:
- 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization.
- Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation).
- Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments.
5. Test Runner Commands & Invocation
# 1. Frontend Test Suite
npm run test:frontend
# 2. Go Backend Integration Test Suite
go test -v ./internal/domain/... ./internal/api/... ./internal/worker/...
# 3. Python Backend & E2E Pipeline Test Suites
python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v