97 lines
6.0 KiB
Markdown
97 lines
6.0 KiB
Markdown
# E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5)
|
||
|
||
## 1. Test Philosophy
|
||
- **Opaque-Box & Requirement-Driven**: Tests are derived strictly from `ORIGINAL_REQUEST.md` and domain specifications, independent of internal module refactorings.
|
||
- **Deterministic & Hermetic**: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies.
|
||
- **4-Tier Test Architecture**: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4).
|
||
|
||
---
|
||
|
||
## 2. Feature Inventory
|
||
| # | Feature | Source (Requirement) | Tier 1 | Tier 2 | Tier 3 | Tier 4 |
|
||
|---|---------|---------------------|:------:|:------:|:------:|:------:|
|
||
| 1 | Video Face Blur Operator (`operator-builtin-video-face-blur`) | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 2 | BOS Ingestion & Orbbec Ego Normalization | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 3 | PTS Monotonic Continuity & Audio Sync Preservation | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 4 | 22-Class Convenience Store Taxonomy Presets | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 5 | Multi-Version Template Presets (v1–v4) & Release Binding | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 6 | Multi-Vendor Data & Task Access Isolation | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 7 | Anti-Duplicate Rework Workload Statistics (`outputDelta=0`) | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 8 | Task Name CRUD & Batch Partitioning | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 9 | WiLoR 5s Valid Event 0/1 Detection | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 10 | Qwen3-VL 200s Temporal Action Interval Localization | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 11 | Qwen3-VL 22-Class Multi-Label Video Classification | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 12 | AI Prelabel Draft Streaming to Timeline Editor | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 13 | ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 14 | Causal Inference Evaluation & Multi-Threshold Metrics | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||
| 15 | Comparison Report Artifacts (HTML, metrics JSON) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||
|
||
---
|
||
|
||
## 3. Test Architecture & Directory Layout
|
||
```
|
||
tests/
|
||
├── fixtures/
|
||
│ └── video_fixtures.py # Synthetic video, PTS CSV, and WAV audio generator
|
||
├── backend/
|
||
│ ├── swift_test_utils.py # ms-swift JSONL schema and leakage validation
|
||
│ ├── causal_eval_golden_vectors.json # Precomputed golden vectors for IoU, MAE, F1
|
||
│ ├── test_fanyu_r1_face_blur.py # R1: Face blur, PTS integrity, audio sync
|
||
│ ├── test_fanyu_r2_taxonomy.py # R2: 22-class taxonomy, alias, templates
|
||
│ ├── test_fanyu_r3_vendor_isolation.py # R3: Multi-vendor isolation, rework stats, task CRUD
|
||
│ ├── test_fanyu_r4_ai_prelabel.py # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class
|
||
│ ├── test_fanyu_r5_swift_causal_eval.py # R5: ms-swift export, causal eval, HTML reports
|
||
│ └── test_e2e_fanyu_pipeline.py # Full E2E pipeline integration test
|
||
frontend/
|
||
└── fanyu-pipeline.test.mjs # Frontend timeline hotkeys, phrases, and prelabel badges
|
||
internal/api/
|
||
└── fanyu_pipeline_e2e_test.go # Go HTTP REST API & workforce dashboard tests
|
||
```
|
||
|
||
---
|
||
|
||
## 4. 4-Tier Test Design Methodology & Coverage Thresholds
|
||
|
||
### Tier 1: Feature Coverage (≥5 tests per feature)
|
||
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
|
||
- Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0.
|
||
|
||
### Tier 2: Boundary & Corner Cases (≥5 tests per feature)
|
||
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
|
||
- Tests verify boundary conditions:
|
||
- R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels.
|
||
- R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection.
|
||
- R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (`outputDelta=0`), division-by-zero on 0 worklogs.
|
||
- R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings.
|
||
- R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits.
|
||
|
||
### Tier 3: Cross-Feature Combinations & Lifecycle Transitions
|
||
- Minimum count: ≥15 cross-feature tests.
|
||
- Verifies pairwise and lifecycle interactions:
|
||
- Ingestion $\rightarrow$ Normalization $\rightarrow$ Face Blur $\rightarrow$ Slicing $\rightarrow$ Packaging.
|
||
- Slicing $\rightarrow$ AI Prelabeling $\rightarrow$ Revision 0 Streaming $\rightarrow$ Human Timeline Edit $\rightarrow$ Revision 1 Submit.
|
||
- QA Rejection $\rightarrow$ Rework Submission $\rightarrow$ Dashboard Reconciliation (`outputDelta=0`, `reworkCount++`).
|
||
- Accepted Annotation Package $\rightarrow$ ms-swift Export $\rightarrow$ Causal Eval $\rightarrow$ HTML Report Generation.
|
||
|
||
### Tier 4: Real-World Workload Scenarios
|
||
- Minimum count: ≥8 comprehensive scenarios.
|
||
- Realistic end-to-end industrial workloads:
|
||
- 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization.
|
||
- Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation).
|
||
- Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments.
|
||
|
||
---
|
||
|
||
## 5. Test Runner Commands & Invocation
|
||
|
||
```bash
|
||
# 1. Frontend Test Suite
|
||
npm run test:frontend
|
||
|
||
# 2. Go Backend Integration Test Suite
|
||
go test -v ./internal/domain/... ./internal/api/... ./internal/worker/...
|
||
|
||
# 3. Python Backend & E2E Pipeline Test Suites
|
||
python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v
|
||
```
|