droboticflow/TEST_INFRA.md

97 lines
6.0 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5)
## 1. Test Philosophy
- **Opaque-Box & Requirement-Driven**: Tests are derived strictly from `ORIGINAL_REQUEST.md` and domain specifications, independent of internal module refactorings.
- **Deterministic & Hermetic**: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies.
- **4-Tier Test Architecture**: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4).
---
## 2. Feature Inventory
| # | Feature | Source (Requirement) | Tier 1 | Tier 2 | Tier 3 | Tier 4 |
|---|---------|---------------------|:------:|:------:|:------:|:------:|
| 1 | Video Face Blur Operator (`operator-builtin-video-face-blur`) | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 2 | BOS Ingestion & Orbbec Ego Normalization | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 3 | PTS Monotonic Continuity & Audio Sync Preservation | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
| 4 | 22-Class Convenience Store Taxonomy Presets | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
| 5 | Multi-Version Template Presets (v1–v4) & Release Binding | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
| 6 | Multi-Vendor Data & Task Access Isolation | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 7 | Anti-Duplicate Rework Workload Statistics (`outputDelta=0`) | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 8 | Task Name CRUD & Batch Partitioning | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
| 9 | WiLoR 5s Valid Event 0/1 Detection | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 10 | Qwen3-VL 200s Temporal Action Interval Localization | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 11 | Qwen3-VL 22-Class Multi-Label Video Classification | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 12 | AI Prelabel Draft Streaming to Timeline Editor | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
| 13 | ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
| 14 | Causal Inference Evaluation & Multi-Threshold Metrics | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
| 15 | Comparison Report Artifacts (HTML, metrics JSON) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
---
## 3. Test Architecture & Directory Layout
```
tests/
├── fixtures/
│ └── video_fixtures.py # Synthetic video, PTS CSV, and WAV audio generator
├── backend/
│ ├── swift_test_utils.py # ms-swift JSONL schema and leakage validation
│ ├── causal_eval_golden_vectors.json # Precomputed golden vectors for IoU, MAE, F1
│ ├── test_fanyu_r1_face_blur.py # R1: Face blur, PTS integrity, audio sync
│ ├── test_fanyu_r2_taxonomy.py # R2: 22-class taxonomy, alias, templates
│ ├── test_fanyu_r3_vendor_isolation.py # R3: Multi-vendor isolation, rework stats, task CRUD
│ ├── test_fanyu_r4_ai_prelabel.py # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class
│ ├── test_fanyu_r5_swift_causal_eval.py # R5: ms-swift export, causal eval, HTML reports
│ └── test_e2e_fanyu_pipeline.py # Full E2E pipeline integration test
frontend/
└── fanyu-pipeline.test.mjs # Frontend timeline hotkeys, phrases, and prelabel badges
internal/api/
└── fanyu_pipeline_e2e_test.go # Go HTTP REST API & workforce dashboard tests
```
---
## 4. 4-Tier Test Design Methodology & Coverage Thresholds
### Tier 1: Feature Coverage (≥5 tests per feature)
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
- Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0.
### Tier 2: Boundary & Corner Cases (≥5 tests per feature)
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
- Tests verify boundary conditions:
- R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels.
- R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection.
- R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (`outputDelta=0`), division-by-zero on 0 worklogs.
- R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings.
- R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits.
### Tier 3: Cross-Feature Combinations & Lifecycle Transitions
- Minimum count: ≥15 cross-feature tests.
- Verifies pairwise and lifecycle interactions:
- Ingestion $\rightarrow$ Normalization $\rightarrow$ Face Blur $\rightarrow$ Slicing $\rightarrow$ Packaging.
- Slicing $\rightarrow$ AI Prelabeling $\rightarrow$ Revision 0 Streaming $\rightarrow$ Human Timeline Edit $\rightarrow$ Revision 1 Submit.
- QA Rejection $\rightarrow$ Rework Submission $\rightarrow$ Dashboard Reconciliation (`outputDelta=0`, `reworkCount++`).
- Accepted Annotation Package $\rightarrow$ ms-swift Export $\rightarrow$ Causal Eval $\rightarrow$ HTML Report Generation.
### Tier 4: Real-World Workload Scenarios
- Minimum count: ≥8 comprehensive scenarios.
- Realistic end-to-end industrial workloads:
- 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization.
- Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation).
- Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments.
---
## 5. Test Runner Commands & Invocation
```bash
# 1. Frontend Test Suite
npm run test:frontend
# 2. Go Backend Integration Test Suite
go test -v ./internal/domain/... ./internal/api/... ./internal/worker/...
# 3. Python Backend & E2E Pipeline Test Suites
python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v
```