# E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5) ## 1. Test Philosophy - **Opaque-Box & Requirement-Driven**: Tests are derived strictly from `ORIGINAL_REQUEST.md` and domain specifications, independent of internal module refactorings. - **Deterministic & Hermetic**: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies. - **4-Tier Test Architecture**: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4). --- ## 2. Feature Inventory | # | Feature | Source (Requirement) | Tier 1 | Tier 2 | Tier 3 | Tier 4 | |---|---------|---------------------|:------:|:------:|:------:|:------:| | 1 | Video Face Blur Operator (`operator-builtin-video-face-blur`) | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ | | 2 | BOS Ingestion & Orbbec Ego Normalization | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ | | 3 | PTS Monotonic Continuity & Audio Sync Preservation | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ | | 4 | 22-Class Convenience Store Taxonomy Presets | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ | | 5 | Multi-Version Template Presets (v1–v4) & Release Binding | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ | | 6 | Multi-Vendor Data & Task Access Isolation | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ | | 7 | Anti-Duplicate Rework Workload Statistics (`outputDelta=0`) | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ | | 8 | Task Name CRUD & Batch Partitioning | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ | | 9 | WiLoR 5s Valid Event 0/1 Detection | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ | | 10 | Qwen3-VL 200s Temporal Action Interval Localization | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ | | 11 | Qwen3-VL 22-Class Multi-Label Video Classification | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ | | 12 | AI Prelabel Draft Streaming to Timeline Editor | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ | | 13 | ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ | | 14 | Causal Inference Evaluation & Multi-Threshold Metrics | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ | | 15 | Comparison Report Artifacts (HTML, metrics JSON) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ | --- ## 3. Test Architecture & Directory Layout ``` tests/ ├── fixtures/ │ └── video_fixtures.py # Synthetic video, PTS CSV, and WAV audio generator ├── backend/ │ ├── swift_test_utils.py # ms-swift JSONL schema and leakage validation │ ├── causal_eval_golden_vectors.json # Precomputed golden vectors for IoU, MAE, F1 │ ├── test_fanyu_r1_face_blur.py # R1: Face blur, PTS integrity, audio sync │ ├── test_fanyu_r2_taxonomy.py # R2: 22-class taxonomy, alias, templates │ ├── test_fanyu_r3_vendor_isolation.py # R3: Multi-vendor isolation, rework stats, task CRUD │ ├── test_fanyu_r4_ai_prelabel.py # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class │ ├── test_fanyu_r5_swift_causal_eval.py # R5: ms-swift export, causal eval, HTML reports │ └── test_e2e_fanyu_pipeline.py # Full E2E pipeline integration test frontend/ └── fanyu-pipeline.test.mjs # Frontend timeline hotkeys, phrases, and prelabel badges internal/api/ └── fanyu_pipeline_e2e_test.go # Go HTTP REST API & workforce dashboard tests ``` --- ## 4. 4-Tier Test Design Methodology & Coverage Thresholds ### Tier 1: Feature Coverage (≥5 tests per feature) - Minimum count: $15 \times 5 = 75$ tests across R1–R5. - Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0. ### Tier 2: Boundary & Corner Cases (≥5 tests per feature) - Minimum count: $15 \times 5 = 75$ tests across R1–R5. - Tests verify boundary conditions: - R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels. - R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection. - R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (`outputDelta=0`), division-by-zero on 0 worklogs. - R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings. - R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits. ### Tier 3: Cross-Feature Combinations & Lifecycle Transitions - Minimum count: ≥15 cross-feature tests. - Verifies pairwise and lifecycle interactions: - Ingestion $\rightarrow$ Normalization $\rightarrow$ Face Blur $\rightarrow$ Slicing $\rightarrow$ Packaging. - Slicing $\rightarrow$ AI Prelabeling $\rightarrow$ Revision 0 Streaming $\rightarrow$ Human Timeline Edit $\rightarrow$ Revision 1 Submit. - QA Rejection $\rightarrow$ Rework Submission $\rightarrow$ Dashboard Reconciliation (`outputDelta=0`, `reworkCount++`). - Accepted Annotation Package $\rightarrow$ ms-swift Export $\rightarrow$ Causal Eval $\rightarrow$ HTML Report Generation. ### Tier 4: Real-World Workload Scenarios - Minimum count: ≥8 comprehensive scenarios. - Realistic end-to-end industrial workloads: - 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization. - Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation). - Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments. --- ## 5. Test Runner Commands & Invocation ```bash # 1. Frontend Test Suite npm run test:frontend # 2. Go Backend Integration Test Suite go test -v ./internal/domain/... ./internal/api/... ./internal/worker/... # 3. Python Backend & E2E Pipeline Test Suites python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v ```