droboticflow-new/TEST_INFRA.md

6.0 KiB
Raw Permalink Blame History

E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5)

1. Test Philosophy

  • Opaque-Box & Requirement-Driven: Tests are derived strictly from ORIGINAL_REQUEST.md and domain specifications, independent of internal module refactorings.
  • Deterministic & Hermetic: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies.
  • 4-Tier Test Architecture: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4).

2. Feature Inventory

# Feature Source (Requirement) Tier 1 Tier 2 Tier 3 Tier 4
1 Video Face Blur Operator (operator-builtin-video-face-blur) ORIGINAL_REQUEST §R1 ≥5 ≥5 ✓ ✓
2 BOS Ingestion & Orbbec Ego Normalization ORIGINAL_REQUEST §R1 ≥5 ≥5 ✓ ✓
3 PTS Monotonic Continuity & Audio Sync Preservation ORIGINAL_REQUEST §R1 ≥5 ≥5 ✓ ✓
4 22-Class Convenience Store Taxonomy Presets ORIGINAL_REQUEST §R2 ≥5 ≥5 ✓ ✓
5 Multi-Version Template Presets (v1–v4) & Release Binding ORIGINAL_REQUEST §R2 ≥5 ≥5 ✓ ✓
6 Multi-Vendor Data & Task Access Isolation ORIGINAL_REQUEST §R3 ≥5 ≥5 ✓ ✓
7 Anti-Duplicate Rework Workload Statistics (outputDelta=0) ORIGINAL_REQUEST §R3 ≥5 ≥5 ✓ ✓
8 Task Name CRUD & Batch Partitioning ORIGINAL_REQUEST §R3 ≥5 ≥5 ✓ ✓
9 WiLoR 5s Valid Event 0/1 Detection ORIGINAL_REQUEST §R4 ≥5 ≥5 ✓ ✓
10 Qwen3-VL 200s Temporal Action Interval Localization ORIGINAL_REQUEST §R4 ≥5 ≥5 ✓ ✓
11 Qwen3-VL 22-Class Multi-Label Video Classification ORIGINAL_REQUEST §R4 ≥5 ≥5 ✓ ✓
12 AI Prelabel Draft Streaming to Timeline Editor ORIGINAL_REQUEST §R4 ≥5 ≥5 ✓ ✓
13 ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) ORIGINAL_REQUEST §R5 ≥5 ≥5 ✓ ✓
14 Causal Inference Evaluation & Multi-Threshold Metrics ORIGINAL_REQUEST §R5 ≥5 ≥5 ✓ ✓
15 Comparison Report Artifacts (HTML, metrics JSON) ORIGINAL_REQUEST §R5 ≥5 ≥5 ✓ ✓

3. Test Architecture & Directory Layout

tests/
├── fixtures/
│   └── video_fixtures.py                   # Synthetic video, PTS CSV, and WAV audio generator
├── backend/
│   ├── swift_test_utils.py                 # ms-swift JSONL schema and leakage validation
│   ├── causal_eval_golden_vectors.json     # Precomputed golden vectors for IoU, MAE, F1
│   ├── test_fanyu_r1_face_blur.py          # R1: Face blur, PTS integrity, audio sync
│   ├── test_fanyu_r2_taxonomy.py           # R2: 22-class taxonomy, alias, templates
│   ├── test_fanyu_r3_vendor_isolation.py   # R3: Multi-vendor isolation, rework stats, task CRUD
│   ├── test_fanyu_r4_ai_prelabel.py        # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class
│   ├── test_fanyu_r5_swift_causal_eval.py  # R5: ms-swift export, causal eval, HTML reports
│   └── test_e2e_fanyu_pipeline.py          # Full E2E pipeline integration test
frontend/
└── fanyu-pipeline.test.mjs                 # Frontend timeline hotkeys, phrases, and prelabel badges
internal/api/
└── fanyu_pipeline_e2e_test.go              # Go HTTP REST API & workforce dashboard tests

4. 4-Tier Test Design Methodology & Coverage Thresholds

Tier 1: Feature Coverage (≥5 tests per feature)

  • Minimum count: 15 \times 5 = 75 tests across R1–R5.
  • Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0.

Tier 2: Boundary & Corner Cases (≥5 tests per feature)

  • Minimum count: 15 \times 5 = 75 tests across R1–R5.
  • Tests verify boundary conditions:
    • R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels.
    • R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection.
    • R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (outputDelta=0), division-by-zero on 0 worklogs.
    • R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings.
    • R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits.

Tier 3: Cross-Feature Combinations & Lifecycle Transitions

  • Minimum count: ≥15 cross-feature tests.
  • Verifies pairwise and lifecycle interactions:
    • Ingestion \rightarrow Normalization \rightarrow Face Blur \rightarrow Slicing \rightarrow Packaging.
    • Slicing \rightarrow AI Prelabeling \rightarrow Revision 0 Streaming \rightarrow Human Timeline Edit \rightarrow Revision 1 Submit.
    • QA Rejection \rightarrow Rework Submission \rightarrow Dashboard Reconciliation (outputDelta=0, reworkCount++).
    • Accepted Annotation Package \rightarrow ms-swift Export \rightarrow Causal Eval \rightarrow HTML Report Generation.

Tier 4: Real-World Workload Scenarios

  • Minimum count: ≥8 comprehensive scenarios.
  • Realistic end-to-end industrial workloads:
    • 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization.
    • Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation).
    • Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments.

5. Test Runner Commands & Invocation

# 1. Frontend Test Suite
npm run test:frontend

# 2. Go Backend Integration Test Suite
go test -v ./internal/domain/... ./internal/api/... ./internal/worker/...

# 3. Python Backend & E2E Pipeline Test Suites
python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v