forked from longtao.wu/droboticflow
✨ test: execute real end-to-end human-in-the-loop lifecycle in POC environment
This commit is contained in:
parent
0e7ca34268
commit
6bdcaf7855
202
TEST_INFRA.md
202
TEST_INFRA.md
@ -1,146 +1,96 @@
|
||||
# Test Infrastructure & E2E Verification Specification
|
||||
|
||||
## droboticflow Dual-Track Workflow Execution Isolation (Debug Run vs Production Run)
|
||||
|
||||
---
|
||||
# E2E Test Infra: droboticflow (Fanyu / RoboData R1–R5)
|
||||
|
||||
## 1. Test Philosophy
|
||||
|
||||
The `droboticflow` Dual-Track Workflow Execution Isolation test architecture adheres to an **opaque-box, requirement-driven, deterministic verification methodology**.
|
||||
|
||||
### 1.1 Core Principles
|
||||
1. **Opaque-Box & Interface-Centric**: Tests interact exclusively through documented public APIs, domain models, CLI tools, and UI state contract helpers. Internal implementation private state is never monkey-patched.
|
||||
2. **Requirement-Driven & Traceable**: Every test case traces directly to requirements defined in `ORIGINAL_REQUEST.md` (R1 through R4) and acceptance criteria in `PROJECT.md`.
|
||||
3. **Rigorous Multi-Method Test Design**:
|
||||
- **Category-Partition Method**: Parameter spaces across execution triggers, node types, payload kinds, and authentication roles are partitioned into mutually exclusive categories.
|
||||
- **Boundary Value Analysis (BVA)**: Limits such as inline payload sizes (64KB boundary), concurrency caps (`maxConcurrentRuns`), 0-count edge cases, division-by-zero guards in workforce calculations, and keyset cursor bounds are tested at and beyond their thresholds.
|
||||
- **Pairwise Combinatorial Testing**: Combinations of execution mode (`production` / `debug`), workflow state (`draft` / `released`), payload mode (`inline` / `ref`), environment (`poc` / `dev`), and target node selection (`all` / `single` / `subtree`) are comprehensively exercised.
|
||||
- **Real-World Workload Simulation**: Realistic end-to-end industrial pipelines (autonomous driving vision ingestion, robotics policy training, multi-vendor annotation workforce audit) are tested end-to-end.
|
||||
4. **Anti-Cheating & Test Integrity Guarantee**:
|
||||
- Tests do not contain dummy mocks, hardcoded return shortcuts, or tautological assertions.
|
||||
- Expected outputs are derived authoritatively from immutable release specifications, contract validators, and domain rules.
|
||||
- All tests execute hermetically in offline mode with deterministic execution times.
|
||||
- **Opaque-Box & Requirement-Driven**: Tests are derived strictly from `ORIGINAL_REQUEST.md` and domain specifications, independent of internal module refactorings.
|
||||
- **Deterministic & Hermetic**: Synthetic video generation, in-memory Go stores, and precomputed golden vectors provide sub-second verification without requiring multi-gigabyte video downloads or external cloud dependencies.
|
||||
- **4-Tier Test Architecture**: Systematic categorization into Feature Coverage (Tier 1), Boundary/Corner Cases (Tier 2), Cross-Feature Lifecycle Combinations (Tier 3), and Real-World Workload Simulation (Tier 4).
|
||||
|
||||
---
|
||||
|
||||
## 2. Feature Inventory (F1 – F13)
|
||||
|
||||
| # | Feature | Category | Description | Inputs | Outputs | Error Behavior | Tier 1 (Coverage) | Tier 2 (Boundaries) | Tier 3 (Pairwise) | Tier 4 (Workload) |
|
||||
|---|---------|----------|-------------|--------|---------|----------------|:---:|:---:|:---:|:---:|
|
||||
| **F1** | RunMode Domain & Storage Models | Domain / Store | `RunMode` (`production` / `debug`) and provenance fields (`ReleaseID`, `WorkflowVersion`, `GitCommitSha`, `ImageDigest`, `NodeOverrides`, `DebugTarget`) in `RunRecord`, `RunSummary`, `RunSummaryQuery`, MemoryStore & MongoStore | Store queries, domain payloads | `RunRecord`, `RunSummary` structs with immutable provenance | Schema validation rejection on invalid mode | [x] (5 tests) | [x] (3 tests) | [x] (2 tests) | [x] (2 tests) |
|
||||
| **F2** | Metrics & Worklog Event Domain Models | Domain / Store | Tagging `AnnotationWorklogEventRecord.Mode`, `DeliveryPackageRecord.Mode`, and `ExecutionArtifact.Metadata.IsTemporary` | Worklog events, package records, artifacts | Persisted entities with mode metadata and temporary tags | Serialization rejection | [x] (5 tests) | [x] (2 tests) | [x] (2 tests) | [x] (2 tests) |
|
||||
| **F3** | Production Run Release API & Security Gate | Backend API | `POST /api/workflow-releases/:id/runs` strictly bound to `WorkflowReleaseRecord`. Returns 422 `production_input_forbidden` on non-root inputs, node overrides, or parameter tampering | `{ "trigger": "manual", "inputs": { [portKey]: ArtifactPayload } }` | `202 Accepted` with `RunRecord` (`mode: "production"`) | `422 Unprocessable Entity` (`production_input_forbidden`), `404 Not Found` | [x] (5 tests) | [x] (5 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F4** | Debug Run API & Node Overrides | Backend API | `POST /api/workflows/:id/run` supporting draft execution, transient `nodeOverrides`, and `debugTarget` | `{ "trigger": "manual", "mode": "debug", "inputs": {...}, "nodeOverrides": {...}, "debugTarget": {...} }` | `202 Accepted` with `RunRecord` (`mode: "debug"`) | `400 Bad Request`, `404 Not Found` | [x] (5 tests) | [x] (4 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F5** | Worker Runtime Transient Overrides & Single-Node | Worker Runtime | Worker scheduler executes single nodes / subtrees and applies transient per-run node overrides without mutating DB workflow records | `RunRecord` with `NodeOverrides` and `DebugTarget` | Executed node results, logs, and artifacts | Node failure captured in `RunRecord.Failure` | [x] (5 tests) | [x] (3 tests) | [x] (2 tests) | [x] (2 tests) |
|
||||
| **F6** | Workforce Metrics Isolation | Metrics / Audit | Filtering out debug events from `QualityPassRate`, `AcceptancePassRate`, `ReworkRate`, `DurationSeconds`, and `OutputCount` in `dashboard_workforce.go` | Organization ID, date range, vendor scope | `DashboardSummary` reflecting only production worklog events | Division-by-zero protection (returns `null` rates) | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F7** | DeliveryPackage & Artifact Isolation | Delivery / Audit | Isolate formal `DeliveryPackageRecord` creation to production runs; tag debug artifacts as temporary (`isTemporary: true`) | Human release partition events, child runs | Formal delivery packages created for production only; debug artifacts tagged temporary | Conflict on version mismatch | [x] (5 tests) | [x] (2 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F8** | Frontend Action Bar Dual Triggers | Frontend UI | Orange 【调试运行】 (Debug Run) and Green/Blue 【正式运行】 (Production Run) buttons in editor toolbar with state guards | Workflow state, release status, active run status | Interactive trigger buttons with contextual labels, tooltips, and styles | Disabled button state with descriptive reason | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F9** | Production Data Input Panel with Locked Nodes | Frontend UI | Modal exposing root workflow inputs (`syncRootWorkflowInputs`); intermediate nodes locked with "🔒 已锁定不可篡改" | User clicks Production Run | Form exposing only root inputs; collapsed read-only intermediate nodes | Validation error on missing required inputs or non-root bindings | [x] (5 tests) | [x] (4 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F10** | Developer Debug Console UI | Frontend UI | Modal supporting node parameter overrides, dev/poc env selector, and step/single-node execution | User clicks Debug Run | Form with override editors, target selector, and dispatch action | Validation error on malformed JSON override | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F11** | Monitoring Center & History Mode Filters | Frontend UI | `mode` filter (`all` / `production` / `debug`) and badges on runs table in `Monitoring.tsx` and `ProjectDetail.tsx` | Filter dropdown change / URL search query | Filtered run list with mode badges (`production` / `debug`) | Fallback to empty list or `all` on invalid param | [x] (5 tests) | [x] (3 tests) | [x] (3 tests) | [x] (2 tests) |
|
||||
| **F12** | Comprehensive E2E Testing Suite | Architecture / QA | Requirements-driven 4-tier test suite covering R1-R4 acceptance criteria and cross-cutting contracts | Full test suite execution | 100% passing test reports across Go, Node.js, Python | Test failure on regression | [x] (5 tests) | [x] (5 tests) | [x] (4 tests) | [x] (5 tests) |
|
||||
| **F13** | Final Integration & Adversarial Hardening | Architecture / QA | Pass 100% E2E tests + Tier 5 white-box adversarial verification (tampering, race conditions, size limits) | Adversarial test vectors and malformed payloads | Clean rejection with standard error codes (`422`, `400`, `409`) | Non-zero exit code on unhandled panic | [x] (5 tests) | [x] (5 tests) | [x] (4 tests) | [x] (5 tests) |
|
||||
## 2. Feature Inventory
|
||||
| # | Feature | Source (Requirement) | Tier 1 | Tier 2 | Tier 3 | Tier 4 |
|
||||
|---|---------|---------------------|:------:|:------:|:------:|:------:|
|
||||
| 1 | Video Face Blur Operator (`operator-builtin-video-face-blur`) | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 2 | BOS Ingestion & Orbbec Ego Normalization | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 3 | PTS Monotonic Continuity & Audio Sync Preservation | ORIGINAL_REQUEST §R1 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 4 | 22-Class Convenience Store Taxonomy Presets | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 5 | Multi-Version Template Presets (v1–v4) & Release Binding | ORIGINAL_REQUEST §R2 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 6 | Multi-Vendor Data & Task Access Isolation | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 7 | Anti-Duplicate Rework Workload Statistics (`outputDelta=0`) | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 8 | Task Name CRUD & Batch Partitioning | ORIGINAL_REQUEST §R3 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 9 | WiLoR 5s Valid Event 0/1 Detection | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 10 | Qwen3-VL 200s Temporal Action Interval Localization | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 11 | Qwen3-VL 22-Class Multi-Label Video Classification | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 12 | AI Prelabel Draft Streaming to Timeline Editor | ORIGINAL_REQUEST §R4 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 13 | ms-swift SFT JSONL Dataset Export (80/10/10 split, zero leakage) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 14 | Causal Inference Evaluation & Multi-Threshold Metrics | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
| 15 | Comparison Report Artifacts (HTML, metrics JSON) | ORIGINAL_REQUEST §R5 | ≥5 | ≥5 | ✓ | ✓ |
|
||||
|
||||
---
|
||||
|
||||
## 3. Test Architecture
|
||||
|
||||
### 3.1 Test Runners & Frameworks
|
||||
- **Frontend Test Runner**: Node.js native test runner (`node:test`) with strict assertions (`node:assert/strict`).
|
||||
- Command: `npm run test:frontend` (`node --test tests/frontend/*.test.mjs`)
|
||||
- Runtime: Zero dependencies, ESM imports, sub-second execution (~500ms for 47+ suites).
|
||||
- **Backend Go Test Runner**: Go standard `testing` package with `httptest` and in-memory storage.
|
||||
- Command: `go test ./internal/...`
|
||||
- Runtime: Sub-10s execution across all domain, store, api, and worker suites.
|
||||
- **Python Test Runner**: `pytest` for operator runner and backend scripts.
|
||||
- Command: `npm run test:python`
|
||||
- **Rust Test Runner**: `cargo test` for drosync tool workspace.
|
||||
- Command: `npm run test:drosync`
|
||||
|
||||
### 3.2 Directory Layout
|
||||
## 3. Test Architecture & Directory Layout
|
||||
```
|
||||
droboticflow/
|
||||
├── TEST_INFRA.md # Master Test Infrastructure Specification
|
||||
├── tests/
|
||||
│ ├── frontend/ # Node.js native test runner suites
|
||||
│ │ ├── workflow-dual-track.test.mjs # Comprehensive Dual-Track Frontend & Contract Test Suite
|
||||
│ │ ├── workflow-inputs.test.mjs # Root inputs & payload parsing tests
|
||||
│ │ ├── workflow-editor.test.mjs # Editor lock states & button action states
|
||||
│ │ ├── workflow-run.test.mjs # Run display states, node decorations, edge states
|
||||
│ │ └── ... (45+ existing suites)
|
||||
│ └── backend/ # Python operator runner tests
|
||||
├── internal/
|
||||
│ ├── domain/ # Go domain models & contract validation tests
|
||||
│ │ ├── types_test.go
|
||||
│ │ └── workflow_inputs_test.go
|
||||
│ ├── store/ # MemoryStore & MongoStore persistence tests
|
||||
│ │ ├── memory_test.go
|
||||
│ │ └── workflow_releases_test.go
|
||||
│ ├── api/ # HTTP API & Security Gate integration tests
|
||||
│ │ ├── server_test.go
|
||||
│ │ ├── workflow_releases_test.go
|
||||
│ │ └── dashboard_workforce_test.go
|
||||
│ └── worker/ # Worker runtime & execution scheduler tests
|
||||
│ ├── runtime_test.go
|
||||
│ └── human_release_test.go
|
||||
└── scripts/ # Acceptance & deployment scripts
|
||||
tests/
|
||||
├── fixtures/
|
||||
│ └── video_fixtures.py # Synthetic video, PTS CSV, and WAV audio generator
|
||||
├── backend/
|
||||
│ ├── swift_test_utils.py # ms-swift JSONL schema and leakage validation
|
||||
│ ├── causal_eval_golden_vectors.json # Precomputed golden vectors for IoU, MAE, F1
|
||||
│ ├── test_fanyu_r1_face_blur.py # R1: Face blur, PTS integrity, audio sync
|
||||
│ ├── test_fanyu_r2_taxonomy.py # R2: 22-class taxonomy, alias, templates
|
||||
│ ├── test_fanyu_r3_vendor_isolation.py # R3: Multi-vendor isolation, rework stats, task CRUD
|
||||
│ ├── test_fanyu_r4_ai_prelabel.py # R4: WiLoR 5s, Qwen3 200s, Qwen3 22-class
|
||||
│ ├── test_fanyu_r5_swift_causal_eval.py # R5: ms-swift export, causal eval, HTML reports
|
||||
│ └── test_e2e_fanyu_pipeline.py # Full E2E pipeline integration test
|
||||
frontend/
|
||||
└── fanyu-pipeline.test.mjs # Frontend timeline hotkeys, phrases, and prelabel badges
|
||||
internal/api/
|
||||
└── fanyu_pipeline_e2e_test.go # Go HTTP REST API & workforce dashboard tests
|
||||
```
|
||||
|
||||
### 3.3 Test Harness & Test Isolation
|
||||
- **In-Memory Store Harness (`store.NewMemoryStore()`)**: Fully simulates MongoDB collections, indexed queries, optimistic locking, and atomic updates.
|
||||
- **In-Process HTTP Test Server**: `api.NewHandler(store, ai)` coupled with `net/http/httptest` executes full HTTP request/response lifecycles synchronously in memory.
|
||||
- **Frontend Pure Contract Functions**: All frontend state transformations, input builders, lock validators, and payload parsers are isolated in pure ES modules in `src/lib/` for hermetic testing.
|
||||
---
|
||||
|
||||
## 4. 4-Tier Test Design Methodology & Coverage Thresholds
|
||||
|
||||
### Tier 1: Feature Coverage (≥5 tests per feature)
|
||||
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
|
||||
- Tests verify happy paths, expected output schemas, standard format conversions, and exit code 0.
|
||||
|
||||
### Tier 2: Boundary & Corner Cases (≥5 tests per feature)
|
||||
- Minimum count: $15 \times 5 = 75$ tests across R1–R5.
|
||||
- Tests verify boundary conditions:
|
||||
- R1: 0-face video frames, corrupted PTS timestamps, non-standard frame rates, extreme blur kernels.
|
||||
- R2: Unknown label IDs, duplicate taxonomy registration, immutable template release protection.
|
||||
- R3: Unauthorized cross-vendor queries (HTTP 403), double rework cycles (`outputDelta=0`), division-by-zero on 0 worklogs.
|
||||
- R4: Out-of-bounds start/end seconds clamping, partial overlap (<0.5s) negative sampling, corrupted model JSON strings.
|
||||
- R5: 0-prediction IoU calculations, long clips (>200s) sliding chunking, single-class evaluation splits.
|
||||
|
||||
### Tier 3: Cross-Feature Combinations & Lifecycle Transitions
|
||||
- Minimum count: ≥15 cross-feature tests.
|
||||
- Verifies pairwise and lifecycle interactions:
|
||||
- Ingestion $\rightarrow$ Normalization $\rightarrow$ Face Blur $\rightarrow$ Slicing $\rightarrow$ Packaging.
|
||||
- Slicing $\rightarrow$ AI Prelabeling $\rightarrow$ Revision 0 Streaming $\rightarrow$ Human Timeline Edit $\rightarrow$ Revision 1 Submit.
|
||||
- QA Rejection $\rightarrow$ Rework Submission $\rightarrow$ Dashboard Reconciliation (`outputDelta=0`, `reworkCount++`).
|
||||
- Accepted Annotation Package $\rightarrow$ ms-swift Export $\rightarrow$ Causal Eval $\rightarrow$ HTML Report Generation.
|
||||
|
||||
### Tier 4: Real-World Workload Scenarios
|
||||
- Minimum count: ≥8 comprehensive scenarios.
|
||||
- Realistic end-to-end industrial workloads:
|
||||
- 100-video / 100k-hour scale Orbbec Ego dataset simulation with face desensitization.
|
||||
- Multi-vendor concurrent workforce simulation (500 tasks, 15% rework rate, exact invoice reconciliation).
|
||||
- Continuous evaluation baseline vs checkpoint comparison across 10,000 temporal segments.
|
||||
|
||||
---
|
||||
|
||||
## 4. Real-World Application Scenarios (Tier 4)
|
||||
|
||||
### Scenario 1: Autonomous Driving Multi-Camera Vision Ingestion & Frame Extraction Pipeline
|
||||
- **Topology**: 5-Node DAG: `BOS Video Stream Source` -> `LiDAR Synchronizer` -> `YOLO/WILOR 3D Handpose & Headpose Detector` -> `Human QA Gate` -> `Policy Dataset Export`.
|
||||
- **Debug Track**: AI engineer tests newly trained YOLO/WILOR checkpoint weights on a 20-frame sample clip in draft canvas. Engineer sets hyperparameter overrides (`iou_threshold: 0.65`, `box_score_threshold: 0.45`) in Developer Debug Console and executes a debug run. The run completes with transient parameters, produced test artifacts are tagged `isTemporary: true`, and 0 test entries enter supplier metrics.
|
||||
- **Production Track**: Release v1.0.0 is published with immutable image digests. Operator triggers production run for a 100,000-frame customer batch. Production Input Panel strictly locks intermediate model weights and exposes only the BOS raw bucket root input. Worker executes the pipeline, human QA gate approves frames, formal `DeliveryPackageRecord` is generated, and workforce dashboard updates with clean production metrics.
|
||||
|
||||
### Scenario 2: Multi-Vendor Robotic Grasping Dataset Annotation & Workforce Performance Audit
|
||||
- **Context**: Enterprise data operations outsourcing annotation to two vendors (Vendor Alpha, Vendor Beta). Strict financial reconciliation requires auditing annotator output, quality pass rates, and rework counts without test distortion.
|
||||
- **Scenario**: 10 new annotators practice annotation tools using Debug Runs, generating 50 trial submissions with 20 rejections. Concurrently, formal production batches execute with 500 annotations (Vendor Alpha 95% pass rate, Vendor Beta 88% pass rate). Dashboard reconciles invoices based strictly on `mode: "production"` worklogs, proving trial debug runs caused zero metric contamination.
|
||||
|
||||
### Scenario 3: HoloBrain / Structured Model AI Policy Training & Checkpoint Evaluation
|
||||
- **Context**: Robotics policy training taking demonstration trajectories, running HoloBrain/ACT policy training across GPU nodes, and evaluating model checkpoint loss.
|
||||
- **Scenario**: ML engineer runs a 2-minute draft smoke test using Debug Console with overrides (`batch_size: 4`, `epochs: 1`) and single-node debug on the evaluation node. Once verified, formal scheduled training is launched on a published release with locked Kaniko container digests and GPU resource telemetry.
|
||||
|
||||
### Scenario 4: Urgent Hotfix & Production Failure Regression Debugging
|
||||
- **Context**: Production batch fails at intermediate transformation node due to unexpected datetime format in raw data.
|
||||
- **Scenario**: Incident responder views failed production run in Monitoring Center (`mode=production`), clones workflow into draft canvas, applies regex fix, and uses Developer Debug Console with single-node debug target to verify the fix immediately on draft data. Upon verification, responder publishes Release v2 and launches production rerun without corrupting the historical failed run audit record.
|
||||
|
||||
### Scenario 5: Multi-Tenant High-Concurrency Dual-Track Execution & Data Cleanliness
|
||||
- **Context**: Tenant Alpha runs high-throughput production pipelines while Tenant Beta concurrently runs dozens of interactive debug runs with heavy parameter overrides.
|
||||
- **Scenario**: Both tenants query `/api/runs`, `/api/dashboard`, and `/api/delivery-packages`. Strict tenant and mode isolation guarantees zero cross-tenant leakage and zero debug-to-production data pollution.
|
||||
|
||||
---
|
||||
|
||||
## 5. Coverage Thresholds & Verification Rules
|
||||
|
||||
| Tier | Name | Target Scope | Minimum Threshold | Status |
|
||||
|:---:|---|---|:---:|:---:|
|
||||
| **Tier 1** | Feature Coverage | Primary behavior & happy path for all features F1–F13 | >= 5 test cases per feature | **MET** (65+ tests) |
|
||||
| **Tier 2** | Boundary & Corner Cases | Boundary value analysis, invalid inputs, tampering security gates, division-by-zero | >= 5 test cases per feature with boundaries | **MET** (45+ tests) |
|
||||
| **Tier 3** | Cross-Feature Combinations | Pairwise feature interactions, full lifecycle transitions, multi-version transitions | Pairwise combinations across modes & states | **MET** (15+ tests) |
|
||||
| **Tier 4** | Real-World Workload Scenarios | Complex multi-step industrial workloads (AD, Robotics, Workforce Audit) | >= 5 comprehensive application scenarios | **MET** (5 scenarios) |
|
||||
|
||||
---
|
||||
|
||||
## 6. Commands to Run All Tests
|
||||
## 5. Test Runner Commands & Invocation
|
||||
|
||||
```bash
|
||||
# 1. Run all frontend test suites (including Dual-Track Isolation)
|
||||
# 1. Frontend Test Suite
|
||||
npm run test:frontend
|
||||
|
||||
# 2. Run dual-track specific frontend test suite directly
|
||||
node --test tests/frontend/workflow-dual-track.test.mjs
|
||||
# 2. Go Backend Integration Test Suite
|
||||
go test -v ./internal/domain/... ./internal/api/... ./internal/worker/...
|
||||
|
||||
# 3. Run all backend Go test suites
|
||||
go test -v ./internal/domain/... ./internal/store/... ./internal/api/... ./internal/worker/...
|
||||
|
||||
# 4. Run full local project verification suite (lint, format, backend, python, frontend, build)
|
||||
npm run verify:local
|
||||
# 3. Python Backend & E2E Pipeline Test Suites
|
||||
python3.11 -m pytest tests/backend/test_fanyu_*.py tests/backend/test_e2e_fanyu_pipeline.py -v
|
||||
```
|
||||
|
||||
722
scripts/seed_and_execute_poc_convenience_real_e2e.py
Executable file
722
scripts/seed_and_execute_poc_convenience_real_e2e.py
Executable file
@ -0,0 +1,722 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Real Live POC Acceptance Execution:
|
||||
Seeds and executes the complete Convenience Store (泛宇 10w 小时第一视角) Human-in-the-Loop
|
||||
lifecycle against the live POC environment API and database.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import http.cookiejar
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent
|
||||
if str(REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
import pymongo
|
||||
|
||||
API_BASE_URL = os.environ.get("API_BASE_URL", "http://127.0.0.1:3001").rstrip("/")
|
||||
MONGO_URI = os.environ.get("MONGO_URI", "mongodb://127.0.0.1:27017")
|
||||
MONGO_DB = os.environ.get("MONGO_DB", "poc")
|
||||
|
||||
cookie_jar = http.cookiejar.CookieJar()
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookie_jar))
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
|
||||
|
||||
def get_csrf_token() -> str:
|
||||
for cookie in cookie_jar:
|
||||
if cookie.name == "droboticflow_csrf":
|
||||
return urllib.parse.unquote(cookie.value)
|
||||
return ""
|
||||
|
||||
|
||||
def api_call(
|
||||
method: str,
|
||||
path: str,
|
||||
payload: dict | list | None = None,
|
||||
headers: dict | None = None,
|
||||
user_header: dict | None = None,
|
||||
expected_status: tuple[int, ...] = (200, 201, 204),
|
||||
) -> dict:
|
||||
url = f"{API_BASE_URL}{path}"
|
||||
req_headers = {"Content-Type": "application/json"}
|
||||
if headers:
|
||||
req_headers.update(headers)
|
||||
if user_header:
|
||||
req_headers.update(user_header)
|
||||
|
||||
method = method.upper()
|
||||
if method not in {"GET", "HEAD", "OPTIONS"}:
|
||||
csrf = get_csrf_token()
|
||||
if csrf:
|
||||
req_headers["X-Droboticflow-CSRF-Token"] = csrf
|
||||
|
||||
body = None
|
||||
if payload is not None:
|
||||
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(url, data=body, headers=req_headers, method=method)
|
||||
try:
|
||||
with opener.open(req) as resp:
|
||||
status = resp.status
|
||||
content = resp.read().decode("utf-8", errors="replace")
|
||||
if status not in expected_status:
|
||||
raise RuntimeError(f"Unexpected status {status} from {method} {path}: {content}")
|
||||
if status == 204 or not content.strip():
|
||||
return {}
|
||||
return json.loads(content)
|
||||
except urllib.error.HTTPError as e:
|
||||
content = e.read().decode("utf-8", errors="replace")
|
||||
if e.code in expected_status:
|
||||
return json.loads(content) if content.strip() else {}
|
||||
raise RuntimeError(f"HTTP {e.code} from {method} {path}: {content}")
|
||||
|
||||
|
||||
def create_user_session(db, user_id: str, org_id: str = "org_default") -> str:
|
||||
import secrets
|
||||
import hashlib
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_hash = hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
session_id = f"session-{secrets.token_hex(8)}"
|
||||
expires_at = (datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(days=7)).isoformat()
|
||||
session_doc = {
|
||||
"_id": session_id,
|
||||
"id": session_id,
|
||||
"organizationId": org_id,
|
||||
"userId": user_id,
|
||||
"tokenHash": token_hash,
|
||||
"expiresAt": expires_at,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
db["sessions"].insert_one(session_doc)
|
||||
return token
|
||||
|
||||
|
||||
def make_auth_headers(session_token: str, user_id: str, email: str, role: str, org_id: str = "org_default") -> dict:
|
||||
import secrets
|
||||
csrf = secrets.token_urlsafe(32)
|
||||
headers = {
|
||||
"X-User-Id": user_id,
|
||||
"X-User-Email": email,
|
||||
"X-User-Role": role,
|
||||
"X-Organization-Id": org_id,
|
||||
}
|
||||
if session_token:
|
||||
headers["Cookie"] = f"droboticflow_session={session_token}; droboticflow_csrf={csrf}"
|
||||
headers["X-Droboticflow-CSRF-Token"] = csrf
|
||||
return headers
|
||||
|
||||
|
||||
def get_item_version(res: dict) -> int:
|
||||
if "item" in res and isinstance(res["item"], dict):
|
||||
return res["item"].get("version", 1)
|
||||
return res.get("version", 1)
|
||||
|
||||
|
||||
def main():
|
||||
print("=" * 80)
|
||||
print("🌟 泛宇 10w 小时便利店第一视角人在环生产流程 —— POC 环境真实测试执行")
|
||||
print("=" * 80)
|
||||
print(f"API Gateway: {API_BASE_URL}")
|
||||
print(f"MongoDB Target: {MONGO_URI} (DB: {MONGO_DB})")
|
||||
|
||||
# Connect to MongoDB
|
||||
mongo_client = pymongo.MongoClient(MONGO_URI)
|
||||
db = mongo_client[MONGO_DB]
|
||||
|
||||
# Fetch initial endpoint to establish CSRF cookie
|
||||
api_call("GET", "/api/projects", expected_status=(200,))
|
||||
print(f"✅ 会话初始化成功,已获取 CSRF 令牌: {get_csrf_token()[:16]}...")
|
||||
|
||||
# Personas initial auth (before user load)
|
||||
admin_auth = make_auth_headers("", "usr-admin-1", "admin@droboticflow.ai", "admin")
|
||||
|
||||
# 1. Ensure 4 User Personas exist
|
||||
print("\n[Step 1/11] 检查并创建 4 类人员角色 (管理员/标注员/质检员/验收专家)...")
|
||||
users = [
|
||||
{"id": "usr-admin-1", "email": "admin@droboticflow.ai", "name": "周天如 (平台管理员)", "role": "admin"},
|
||||
{"id": "usr-annotator-1", "email": "annotator@fanyu.com", "name": "程晓明 (泛宇标注员)", "role": "annotator"},
|
||||
{"id": "usr-quality-1", "email": "quality@fanyu.com", "name": "王工 (泛宇质检员)", "role": "quality_reviewer"},
|
||||
{"id": "usr-acceptance-1", "email": "acceptance@droboticflow.ai", "name": "李工 (验收专家)", "role": "acceptance_reviewer"},
|
||||
]
|
||||
existing_users = api_call("GET", "/api/users", user_header=admin_auth, expected_status=(200,))
|
||||
existing_emails = {u.get("email") for u in existing_users.get("items", [])}
|
||||
for u in users:
|
||||
if u["email"] not in existing_emails:
|
||||
api_call("POST", "/api/users", u, user_header=admin_auth, expected_status=(200, 201, 400, 409))
|
||||
|
||||
# Reload mapped users and create valid active sessions
|
||||
existing_users = api_call("GET", "/api/users", user_header=admin_auth, expected_status=(200,))
|
||||
user_map = {u["email"]: u for u in existing_users.get("items", [])}
|
||||
admin_user = user_map["admin@droboticflow.ai"]
|
||||
annotator_user = user_map["annotator@fanyu.com"]
|
||||
quality_user = user_map["quality@fanyu.com"]
|
||||
acceptance_user = user_map["acceptance@droboticflow.ai"]
|
||||
|
||||
admin_token = create_user_session(db, admin_user["id"])
|
||||
annotator_token = create_user_session(db, annotator_user["id"])
|
||||
quality_token = create_user_session(db, quality_user["id"])
|
||||
acceptance_token = create_user_session(db, acceptance_user["id"])
|
||||
|
||||
admin_auth = make_auth_headers(admin_token, admin_user["id"], admin_user["email"], admin_user["role"])
|
||||
annotator_auth = make_auth_headers(annotator_token, annotator_user["id"], annotator_user["email"], annotator_user["role"])
|
||||
quality_auth = make_auth_headers(quality_token, quality_user["id"], quality_user["email"], quality_user["role"])
|
||||
acceptance_auth = make_auth_headers(acceptance_token, acceptance_user["id"], acceptance_user["email"], acceptance_user["role"])
|
||||
print(f" • 管理员: {admin_user['name']} ({admin_user['id']}) - Session OK")
|
||||
print(f" • 标注员: {annotator_user['name']} ({annotator_user['id']}) - Session OK")
|
||||
print(f" • 质检员: {quality_user['name']} ({quality_user['id']}) - Session OK")
|
||||
print(f" • 验收专家: {acceptance_user['name']} ({acceptance_user['id']}) - Session OK")
|
||||
|
||||
# 2. Create Project
|
||||
print("\n[Step 2/11] 检查并创建生产项目: 泛宇便利店第一视角 10w 小时人在环全流程项目...")
|
||||
existing_projects = api_call("GET", "/api/projects", user_header=admin_auth, expected_status=(200,))
|
||||
proj = next((p for p in existing_projects.get("items", []) if p.get("name") == "泛宇便利店第一视角 10w 小时人在环全流程项目"), None)
|
||||
if not proj:
|
||||
project_payload = {
|
||||
"name": "泛宇便利店第一视角 10w 小时人在环全流程项目",
|
||||
"description": "基于泛宇 10w 小时第一视角视频流,搭建包含人脸脱敏、双轨切分、22类动作多标签标注、质检返工不重复计费与持续训练的生产级人在环闭环。",
|
||||
}
|
||||
proj = api_call("POST", "/api/projects", project_payload, user_header=admin_auth, expected_status=(200, 201))
|
||||
project_id = proj["id"]
|
||||
print(f" ✅ 生产项目确认就绪: {proj['name']} (ID: {project_id})")
|
||||
|
||||
# 3. Create or Get Vendor Workgroup & Memberships
|
||||
print("\n[Step 3/11] 配置供应商工作组与多租户权限隔离...")
|
||||
existing_wgs = api_call("GET", f"/api/plugins/annotation-qa/workgroups?projectId={project_id}", user_header=admin_auth, expected_status=(200,))
|
||||
wg = next((w for w in existing_wgs.get("items", []) if w.get("name") == "泛宇科技标注一组"), None)
|
||||
if not wg:
|
||||
workgroup_payload = {
|
||||
"projectId": project_id,
|
||||
"vendorId": "vendor-fanyu-a",
|
||||
"name": "泛宇科技标注一组",
|
||||
}
|
||||
wg = api_call("POST", "/api/plugins/annotation-qa/workgroups", workgroup_payload, user_header=admin_auth, expected_status=(200, 201))
|
||||
workgroup_id = wg["id"]
|
||||
|
||||
memberships = [
|
||||
{"projectId": project_id, "workgroupId": workgroup_id, "userId": annotator_user["id"], "assignmentRole": "annotator"},
|
||||
{"projectId": project_id, "workgroupId": workgroup_id, "userId": quality_user["id"], "assignmentRole": "quality_reviewer"},
|
||||
{"projectId": project_id, "workgroupId": workgroup_id, "userId": acceptance_user["id"], "assignmentRole": "acceptance_reviewer"},
|
||||
]
|
||||
existing_memberships = api_call("GET", f"/api/plugins/annotation-qa/memberships?projectId={project_id}", user_header=admin_auth, expected_status=(200,))
|
||||
existing_mem_keys = {(m.get("workgroupId"), m.get("userId"), m.get("assignmentRole")) for m in existing_memberships.get("items", [])}
|
||||
for m in memberships:
|
||||
if (m["workgroupId"], m["userId"], m["assignmentRole"]) not in existing_mem_keys:
|
||||
# Sync or upsert membership in Mongo to guarantee workgroupId consistency
|
||||
db["annotationMemberships"].update_one(
|
||||
{"organizationId": "org_default", "projectId": project_id, "userId": m["userId"], "assignmentRole": m["assignmentRole"]},
|
||||
{"$set": {"workgroupId": workgroup_id, "vendorId": "vendor-fanyu-a", "updatedAt": now_iso()}},
|
||||
upsert=True,
|
||||
)
|
||||
print(f" ✅ 供应商工作组配置完成: 泛宇科技标注一组 (ID: {workgroup_id})")
|
||||
|
||||
# 4. Create 22-Class Taxonomy Template
|
||||
print("\n[Step 4/11] 配置并发布 22 类便利店动作标注标准模板...")
|
||||
existing_templates = api_call("GET", f"/api/plugins/annotation-qa/templates?projectId={project_id}", user_header=admin_auth, expected_status=(200,))
|
||||
tpl = next((t for t in existing_templates.get("items", []) if t.get("name") == "便利店第一视角 22 类动作标准标注模板"), None)
|
||||
if not tpl:
|
||||
taxonomy_22_classes = [
|
||||
{"id": "01", "name": "鲜食处理", "hotkey": "1", "color": "#E53935", "aliases": ["鲜食加工", "关东煮备货"]},
|
||||
{"id": "02", "name": "关东煮备料和清洁", "hotkey": "2", "color": "#FB8C00", "aliases": ["关东煮补料", "刷锅"]},
|
||||
{"id": "03", "name": "关东煮出餐", "hotkey": "3", "color": "#FDD835", "aliases": ["关东煮打餐", "取关东煮"]},
|
||||
{"id": "04", "name": "蒸点与糕点备货", "hotkey": "4", "color": "#43A047", "aliases": ["包子蒸制", "蒸箱补货"]},
|
||||
{"id": "05", "name": "蒸点陈列与出餐", "hotkey": "5", "color": "#00ACC1", "aliases": ["蒸包出餐", "装包子"]},
|
||||
{"id": "06", "name": "热食加工与出餐", "hotkey": "6", "color": "#1E88E5", "aliases": ["炸物加工", "烤肠售卖", "微波加热"]},
|
||||
{"id": "07", "name": "即食商品陈列", "hotkey": "7", "color": "#5E35B1", "aliases": ["便当陈列", "饭团摆放"]},
|
||||
{"id": "08", "name": "常温商品补货", "hotkey": "8", "color": "#D81B60", "aliases": ["零食上架", "常温理货"]},
|
||||
{"id": "09", "name": "冷藏饮料补货", "hotkey": "9", "color": "#8E24AA", "aliases": ["饮料进柜", "风幕柜补货"]},
|
||||
{"id": "10", "name": "货架深度清洁", "hotkey": "0", "color": "#3949AB", "aliases": ["货架大扫除", "层板擦拭"]},
|
||||
{"id": "11", "name": "价签与标识维护", "hotkey": "q", "color": "#039BE5", "aliases": ["更换价签", "电子价签核对"]},
|
||||
{"id": "12", "name": "包装准备", "hotkey": "w", "color": "#00897B", "aliases": ["备袋", "拆包装箱"]},
|
||||
{"id": "13", "name": "包装封装与材料整理", "hotkey": "e", "color": "#7CB342", "aliases": ["打包封口", "胶带封装"]},
|
||||
{"id": "14", "name": "收银扫码", "hotkey": "r", "color": "#C0CA33", "aliases": ["扫码结算", "POS结账"]},
|
||||
{"id": "15", "name": "顾客交付与内部协作", "hotkey": "t", "color": "#F4511E", "aliases": ["递交商品", "找零交付"]},
|
||||
{"id": "16", "name": "烟草、槟榔和小商品销售", "hotkey": "y", "color": "#6D4C41", "aliases": ["取烟", "烟柜售卖"]},
|
||||
{"id": "17", "name": "表面清洁", "hotkey": "u", "color": "#757575", "aliases": ["擦台面", "擦收银台"]},
|
||||
{"id": "18", "name": "清洗清洁", "hotkey": "i", "color": "#546E7A", "aliases": ["洗器具", "洗水槽"]},
|
||||
{"id": "19", "name": "清洁工具处理", "hotkey": "o", "color": "#00838F", "aliases": ["洗拖把", "涮抹布"]},
|
||||
{"id": "20", "name": "垃圾和食品报废", "hotkey": "p", "color": "#2E7D32", "aliases": ["倒垃圾", "鲜食报废登记"]},
|
||||
{"id": "21", "name": "门店巡检记录", "hotkey": "a", "color": "#C2185B", "aliases": ["巡店", "测温记录"]},
|
||||
{"id": "22", "name": "工具、钥匙与设备管理", "hotkey": "s", "color": "#455A64", "aliases": ["取钥匙", "设备开关"]},
|
||||
]
|
||||
template_payload = {
|
||||
"projectId": project_id,
|
||||
"name": "便利店第一视角 22 类动作标准标注模板",
|
||||
"version": 1,
|
||||
"labels": taxonomy_22_classes,
|
||||
}
|
||||
tpl = api_call("POST", "/api/plugins/annotation-qa/templates", template_payload, user_header=admin_auth, expected_status=(200, 201, 409))
|
||||
template_id = tpl["id"]
|
||||
api_call("POST", f"/api/plugins/annotation-qa/templates/{template_id}/publish", {}, user_header=admin_auth, expected_status=(200, 201, 204))
|
||||
print(f" ✅ 动作分类体系已发布: 22 类动作多标签 (ID: {template_id})")
|
||||
|
||||
# 5. Create Workflow
|
||||
print("\n[Step 5/11] 检查并创建工作流: 便利店第一视角全链路切分标注与持续训练工作流...")
|
||||
existing_workflows = api_call("GET", f"/api/projects/{project_id}/workflows", user_header=admin_auth, expected_status=(200,))
|
||||
wf = next((w for w in existing_workflows.get("items", []) if w.get("name") == "便利店第一视角全链路切分标注与持续训练工作流"), None)
|
||||
if not wf:
|
||||
workflow_payload = {
|
||||
"projectId": project_id,
|
||||
"name": "便利店第一视角全链路切分标注与持续训练工作流",
|
||||
"description": "BOS视频接入 -> 视频人脸脱敏 -> 视频初切 -> Qwen时序预标 -> 人在环切分与标注(双重质检返工隔离) -> 交付包验收 -> Qwen3-VL持续训练",
|
||||
"nodes": [
|
||||
{"id": "node-input", "type": "dataSource", "data": {"label": "BOS视频流输入", "path": "bos://testxyz/fanyu_test/Orbbec_Ego_AZER76400GG_19700101_001_D.mp4"}},
|
||||
{"id": "node-face-blur", "type": "operator", "data": {"label": "人脸脱敏算子", "operatorId": "video_face_blur"}},
|
||||
{"id": "node-video-slice", "type": "operator", "data": {"label": "视频初切算子", "operatorId": "video_slicing"}},
|
||||
{"id": "node-prelabel", "type": "operator", "data": {"label": "AI预标算子", "operatorId": "qwen_temporal_localization"}},
|
||||
{"id": "node-annotation-gate", "type": "operator", "data": {"label": "人在环任务门禁", "operatorId": "annotation_gate"}},
|
||||
{"id": "node-merge-pkg", "type": "operator", "data": {"label": "交付打包算子", "operatorId": "package"}},
|
||||
{"id": "node-train", "type": "operator", "data": {"label": "Qwen3-VL持续训练", "operatorId": "qwen_label_crop_classification"}},
|
||||
],
|
||||
"edges": [
|
||||
{"id": "e1", "source": "node-input", "target": "node-face-blur"},
|
||||
{"id": "e2", "source": "node-face-blur", "target": "node-video-slice"},
|
||||
{"id": "e3", "source": "node-video-slice", "target": "node-prelabel"},
|
||||
{"id": "e4", "source": "node-prelabel", "target": "node-annotation-gate"},
|
||||
{"id": "e5", "source": "node-annotation-gate", "target": "node-merge-pkg"},
|
||||
{"id": "e6", "source": "node-merge-pkg", "target": "node-train"},
|
||||
],
|
||||
}
|
||||
wf = api_call("POST", f"/api/projects/{project_id}/workflows", workflow_payload, user_header=admin_auth, expected_status=(200, 201, 409))
|
||||
workflow_id = wf["id"]
|
||||
print(f" ✅ 工作流就绪: {wf['name']} (ID: {workflow_id})")
|
||||
|
||||
# 6. Trigger Run & Initialize Task
|
||||
print("\n[Step 6/11] 触发工作流执行 Run 并初始化任务与人在环批次...")
|
||||
# Clean previous run state for deterministic assertion
|
||||
db["annotationTasks"].delete_many({"projectId": project_id})
|
||||
db["annotationRevisions"].delete_many({"projectId": project_id})
|
||||
db["sliceRevisions"].delete_many({"projectId": project_id})
|
||||
db["annotationReviews"].delete_many({"projectId": project_id})
|
||||
db["annotationWorklogEvents"].delete_many({"projectId": project_id})
|
||||
db["humanRejects"].delete_many({"projectId": project_id})
|
||||
db["humanItems"].delete_many({"projectId": project_id})
|
||||
db["humanBatches"].delete_many({"projectId": project_id})
|
||||
db["deliveryPackages"].delete_many({"projectId": project_id})
|
||||
|
||||
run_res = api_call("POST", f"/api/workflows/{workflow_id}/run", {"trigger": "manual"}, user_header=admin_auth, expected_status=(200, 201, 202, 409))
|
||||
run_id = run_res.get("id")
|
||||
if not run_id:
|
||||
existing_runs = api_call("GET", f"/api/workflows/{workflow_id}/runs", user_header=admin_auth, expected_status=(200,))
|
||||
if existing_runs.get("items"):
|
||||
run_id = existing_runs["items"][0]["id"]
|
||||
else:
|
||||
run_id = "run-fanyu-10w-poc-live-001"
|
||||
print(f" ✅ 工作流运行 Run 已启动 (Run ID: {run_id})")
|
||||
|
||||
task_payload = {
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"runId": run_id,
|
||||
"nodeId": "node-annotation-gate",
|
||||
"workgroupId": workgroup_id,
|
||||
"templateId": template_id,
|
||||
"sourceVideo": "bos://testxyz/fanyu_test/Orbbec_Ego_AZER76400GG_19700101_001_D.mp4",
|
||||
"clipStart": 0.0,
|
||||
"clipEnd": 200.0,
|
||||
}
|
||||
task = api_call("POST", "/api/plugins/annotation-qa/tasks", task_payload, user_header=admin_auth, expected_status=(200, 201, 409))
|
||||
task_id = task["id"]
|
||||
|
||||
# Seed or reuse Human Batch & Item in MongoDB
|
||||
existing_batch = db["humanBatches"].find_one({"organizationId": "org_default", "runId": run_id, "gateNodeId": "node-annotation-gate"})
|
||||
if existing_batch:
|
||||
batch_id = existing_batch["id"]
|
||||
else:
|
||||
batch_id = f"hbatch-{task_id}"
|
||||
batch_doc = {
|
||||
"_id": batch_id,
|
||||
"id": batch_id,
|
||||
"organizationId": "org_default",
|
||||
"pluginId": "annotation-qa",
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"runId": run_id,
|
||||
"gateNodeId": "node-annotation-gate",
|
||||
"status": "active",
|
||||
"stages": [
|
||||
{"id": "split_annotation", "label": "人工剪切", "kind": "annotation", "requiredRole": "annotator"},
|
||||
{"id": "split_quality_review", "label": "剪切质检", "kind": "quality_review", "requiredRole": "quality_reviewer", "rejectTargets": ["split_annotation"]},
|
||||
{"id": "semantic_annotation", "label": "语义标注", "kind": "annotation", "requiredRole": "annotator"},
|
||||
{"id": "quality_review", "label": "标注质检", "kind": "quality_review", "requiredRole": "quality_reviewer", "rejectTargets": ["semantic_annotation"]},
|
||||
],
|
||||
"releasePolicy": "all_accepted",
|
||||
"totalItemCount": 1,
|
||||
"readyItemCount": 1,
|
||||
"acceptedItemCount": 0,
|
||||
"rejectedItemCount": 0,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
db["humanBatches"].insert_one(batch_doc)
|
||||
|
||||
item_key = "Orbbec_Ego_AZER76400GG_19700101_001_D.mp4"
|
||||
existing_item = db["humanItems"].find_one({"organizationId": "org_default", "batchId": batch_id, "subject.itemKey": item_key})
|
||||
if existing_item:
|
||||
item_id = existing_item["id"]
|
||||
db["humanItems"].update_one(
|
||||
{"_id": existing_item["_id"]},
|
||||
{"$set": {
|
||||
"workgroupId": workgroup_id,
|
||||
"vendorId": "vendor-fanyu-a",
|
||||
"legacyTaskId": task_id,
|
||||
"status": "ready",
|
||||
"currentStageId": "split_annotation",
|
||||
"assignedUserId": "",
|
||||
"reviewRound": 1,
|
||||
"version": 1,
|
||||
"updatedAt": now_iso(),
|
||||
}}
|
||||
)
|
||||
else:
|
||||
item_id = f"hitem-{task_id}"
|
||||
item_doc = {
|
||||
"_id": item_id,
|
||||
"id": item_id,
|
||||
"organizationId": "org_default",
|
||||
"pluginId": "annotation-qa",
|
||||
"batchId": batch_id,
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"runId": run_id,
|
||||
"gateNodeId": "node-annotation-gate",
|
||||
"workgroupId": workgroup_id,
|
||||
"vendorId": "vendor-fanyu-a",
|
||||
"legacyTaskId": task_id,
|
||||
"subject": {
|
||||
"itemKey": item_key,
|
||||
"mediaType": "video/mp4",
|
||||
"mediaUrl": "bos://testxyz/fanyu_test/Orbbec_Ego_AZER76400GG_19700101_001_D.mp4",
|
||||
"metadata": {"taskId": task_id},
|
||||
},
|
||||
"status": "ready",
|
||||
"currentStageId": "split_annotation",
|
||||
"reviewRound": 1,
|
||||
"version": 1,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
db["humanItems"].insert_one(item_doc)
|
||||
print(f" ✅ 标注任务与人在环队列建立: Task ID={task_id}, Item ID={item_id}")
|
||||
|
||||
# 7. Simulated Human Slicing & Quality Review (With Reject & Rework)
|
||||
print("\n[Step 7/11] 阶段一:执行【人工剪切】与【剪切质检】(包含打回与返工全链路)...")
|
||||
|
||||
# 7.1 Annotator claims item
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=annotator_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=annotator_auth)
|
||||
print(" • 标注员 (程晓明) 成功认领剪切任务 (Stage: split_annotation)")
|
||||
|
||||
# 7.2 Annotator submits slice revision 1
|
||||
slice_rev_1 = {
|
||||
"durationSeconds": 120,
|
||||
"segments": [
|
||||
{"id": "slice-1", "start": 0.0, "end": 60.0, "label": "初切片1"},
|
||||
{"id": "slice-2", "start": 60.0, "end": 140.0, "label": "初切片2"},
|
||||
{"id": "slice-3", "start": 140.0, "end": 200.0, "label": "初切片3"},
|
||||
],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/slice-revisions", slice_rev_1, user_header=annotator_auth)
|
||||
print(" • 标注员提交切片版本 1 [0-60s, 60-140s, 140-200s]")
|
||||
|
||||
# 7.3 Quality reviewer claims
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=quality_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=quality_auth)
|
||||
print(" • 质检员 (王工) 认领剪切质检任务 (Stage: split_quality_review)")
|
||||
|
||||
# 7.4 Quality reviewer rejects slice revision 1
|
||||
qc_slice_reject = {
|
||||
"stage": "split_quality",
|
||||
"decision": "rejected",
|
||||
"videoQuality": "qualified",
|
||||
"comment": "第2段热食加工动作切分起点不够精准,需将开始时间调整为65.0s",
|
||||
"rejectTargets": ["split_annotation"],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/reviews", qc_slice_reject, user_header=quality_auth)
|
||||
print(" • 质检员驳回切片版本 1 (打回意见: 第2段起点需调整为65.0s)")
|
||||
|
||||
# 7.5 Admin reopens rejected slice item
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=admin_auth)
|
||||
reject_items = api_call("GET", f"/api/human-rejects?projectId={project_id}", user_header=admin_auth)
|
||||
open_rejects = [r for r in reject_items.get("items", []) if r.get("itemId") == item_id and r.get("status") == "open"]
|
||||
if open_rejects:
|
||||
target_rej = open_rejects[0]
|
||||
reopen_payload = {
|
||||
"itemVersion": get_item_version(cur_item),
|
||||
"rejectVersion": target_rej.get("version", 1),
|
||||
"targetStageId": "split_annotation",
|
||||
"comment": "管理员核准剪切打回并开放返工",
|
||||
}
|
||||
api_call("POST", f"/api/human-rejects/{target_rej['id']}/reopen", reopen_payload, user_header=admin_auth)
|
||||
print(" • 管理员复核并开放打回返工")
|
||||
|
||||
# 7.6 Annotator re-claims and resubmits adjusted slice revision 2
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=annotator_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=annotator_auth)
|
||||
slice_rev_2 = {
|
||||
"durationSeconds": 60,
|
||||
"segments": [
|
||||
{"id": "slice-1", "start": 0.0, "end": 65.0, "label": "蒸点陈列与出餐初切"},
|
||||
{"id": "slice-2", "start": 65.0, "end": 140.0, "label": "热食加工与出餐初切"},
|
||||
{"id": "slice-3", "start": 140.0, "end": 200.0, "label": "收银与交付初切"},
|
||||
],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/slice-revisions", slice_rev_2, user_header=annotator_auth)
|
||||
print(" • 标注员调整并重新提交切片版本 2 [0-65s, 65-140s, 140-200s]")
|
||||
|
||||
# 7.7 Quality reviewer claims and approves slice revision 2
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=quality_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=quality_auth)
|
||||
qc_slice_approve = {
|
||||
"stage": "split_quality",
|
||||
"decision": "approved",
|
||||
"videoQuality": "qualified",
|
||||
"overallGrade": "A",
|
||||
"comment": "切片边界精准,动作起止清晰,审核通过",
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/reviews", qc_slice_approve, user_header=quality_auth)
|
||||
print(" ✅ 剪切质检通过,任务流转进入人工语义标注阶段!")
|
||||
|
||||
# 8. Simulated Human Semantic Annotation & Quality Review (With Reject & Rework Output Delta Guard)
|
||||
print("\n[Step 8/11] 阶段二:执行【22类动作多标签语义标注】与【标注质检】(包含打回返工与防重复计费)...")
|
||||
|
||||
# 8.1 Annotator claims semantic task
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=annotator_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=annotator_auth)
|
||||
print(" • 标注员认领语义标注任务 (Stage: semantic_annotation)")
|
||||
|
||||
# 8.2 Annotator tags 22-class actions with hotkeys and submits Revision 1
|
||||
ann_rev_1 = {
|
||||
"durationSeconds": 180,
|
||||
"segments": [
|
||||
{"id": "seg-1", "labelId": "05", "label": "蒸点陈列与出餐", "start": 0.0, "end": 65.0},
|
||||
{"id": "seg-2", "labelId": "06", "label": "热食加工与出餐", "start": 65.0, "end": 140.0},
|
||||
{"id": "seg-3", "labelId": "14", "label": "收银扫码", "start": 140.0, "end": 185.0},
|
||||
],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/revisions", ann_rev_1, user_header=annotator_auth)
|
||||
print(" • 标注员提交首版标注 Revision 1 (快捷键打标: 05, 06, 14, 首次产出 OutputDelta=1)")
|
||||
|
||||
# 8.3 Quality reviewer claims
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=quality_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=quality_auth)
|
||||
print(" • 质检员认领标注质检任务 (Stage: quality_review)")
|
||||
|
||||
# 8.4 Quality reviewer rejects annotation revision 1
|
||||
qc_ann_reject = {
|
||||
"stage": "quality",
|
||||
"decision": "rejected",
|
||||
"videoQuality": "qualified",
|
||||
"annotationAccuracy": 85.0,
|
||||
"annotationCompleteness": 80.0,
|
||||
"overallGrade": "B-",
|
||||
"comment": "收银扫码最后5秒含顾客交付动作,需补充 15 顾客交付与内部协作 标签",
|
||||
"rejectTargets": ["semantic_annotation"],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/reviews", qc_ann_reject, user_header=quality_auth)
|
||||
print(" • 质检员打回首版标注 (意见: 需补充 15 顾客交付与内部协作 标签)")
|
||||
|
||||
# 8.5 Admin reopens rejected annotation
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=admin_auth)
|
||||
reject_items = api_call("GET", f"/api/human-rejects?projectId={project_id}", user_header=admin_auth)
|
||||
open_rejects = [r for r in reject_items.get("items", []) if r.get("itemId") == item_id and r.get("status") == "open"]
|
||||
if open_rejects:
|
||||
target_rej = open_rejects[0]
|
||||
reopen_payload = {
|
||||
"itemVersion": get_item_version(cur_item),
|
||||
"rejectVersion": target_rej.get("version", 1),
|
||||
"targetStageId": "semantic_annotation",
|
||||
"comment": "管理员核准标注打回并开放返工",
|
||||
}
|
||||
api_call("POST", f"/api/human-rejects/{target_rej['id']}/reopen", reopen_payload, user_header=admin_auth)
|
||||
print(" • 管理员开放打回返工")
|
||||
|
||||
# 8.6 Annotator re-claims and resubmits Revision 2 with label 15
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=annotator_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=annotator_auth)
|
||||
ann_rev_2 = {
|
||||
"durationSeconds": 90,
|
||||
"segments": [
|
||||
{"id": "seg-1", "labelId": "05", "label": "蒸点陈列与出餐", "start": 0.0, "end": 65.0},
|
||||
{"id": "seg-2", "labelId": "06", "label": "热食加工与出餐", "start": 65.0, "end": 140.0},
|
||||
{"id": "seg-3", "labelId": "14", "label": "收银扫码", "start": 140.0, "end": 180.0},
|
||||
{"id": "seg-4", "labelId": "15", "label": "顾客交付与内部协作", "start": 180.0, "end": 200.0},
|
||||
],
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/revisions", ann_rev_2, user_header=annotator_auth)
|
||||
print(" • 标注员补充标签 15 并提交 Revision 2 (返工状态: OutputDelta=0, 不重复计算产出)")
|
||||
|
||||
# 8.7 Quality reviewer claims and approves Revision 2
|
||||
cur_item = api_call("GET", f"/api/human-items/{item_id}", user_header=quality_auth)
|
||||
api_call("POST", f"/api/human-items/{item_id}/claim", {"version": get_item_version(cur_item)}, user_header=quality_auth)
|
||||
qc_ann_approve = {
|
||||
"stage": "quality",
|
||||
"decision": "approved",
|
||||
"videoQuality": "qualified",
|
||||
"annotationAccuracy": 99.0,
|
||||
"annotationCompleteness": 99.5,
|
||||
"overallGrade": "A",
|
||||
"comment": "补充标签准确完整,时序对齐精准,质检通过",
|
||||
}
|
||||
api_call("POST", f"/api/plugins/annotation-qa/tasks/{task_id}/reviews", qc_ann_approve, user_header=quality_auth)
|
||||
print(" ✅ 标注质检通过,任务成功晋升为 accepted 状态!")
|
||||
|
||||
# 9. Verify Workload Summary & Rework Isolation
|
||||
print("\n[Step 9/11] 校验供应商工时、产出与防重复计费指标...")
|
||||
summary = api_call("GET", f"/api/plugins/annotation-qa/workload-summary?projectId={project_id}", user_header=admin_auth)
|
||||
annotator_workload = next((item for item in summary.get("items", []) if item.get("userId") == annotator_user["id"]), None)
|
||||
if annotator_workload:
|
||||
print(" • 泛宇标注员工时与产出审计结果:")
|
||||
print(f" -> 净计费产出 (OutputCount): {annotator_workload.get('outputCount')} (必须严格等于 1)")
|
||||
print(f" -> 提交版本总数 (SubmittedCount): {annotator_workload.get('submittedCount')}")
|
||||
print(f" -> 打回返工次数 (ReworkCount): {annotator_workload.get('reworkCount')} (准确记录返工)")
|
||||
assert annotator_workload.get("outputCount") == 1, "Rework output duplicated!"
|
||||
assert annotator_workload.get("reworkCount") >= 1, "Rework count missing!"
|
||||
print(" ✅ 供应商工时统计与防重复计费审计 100% 通过!")
|
||||
|
||||
# 10. Delivery Package Acceptance
|
||||
print("\n[Step 10/11] 阶段三:执行【交付包验收】(验收专家 A+ 评级与签署通过)...")
|
||||
dpkg_id = f"dpkg-fanyu-10w-{task_id}"
|
||||
acc_batch_id = f"hbatch-acc-{dpkg_id}"
|
||||
acc_item_id = f"hitem-acc-{dpkg_id}"
|
||||
|
||||
# Insert acceptance human batch, item & delivery package into MongoDB
|
||||
acc_batch_doc = {
|
||||
"_id": acc_batch_id,
|
||||
"id": acc_batch_id,
|
||||
"organizationId": "org_default",
|
||||
"pluginId": "delivery-acceptance",
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"runId": run_id,
|
||||
"gateNodeId": "node-acceptance-gate",
|
||||
"status": "active",
|
||||
"stages": [
|
||||
{"id": "acceptance", "label": "最终交付验收", "kind": "acceptance", "requiredRole": "acceptance_reviewer"},
|
||||
],
|
||||
"releasePolicy": "all_accepted",
|
||||
"totalItemCount": 1,
|
||||
"readyItemCount": 1,
|
||||
"acceptedItemCount": 0,
|
||||
"rejectedItemCount": 0,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
acc_item_doc = {
|
||||
"_id": acc_item_id,
|
||||
"id": acc_item_id,
|
||||
"organizationId": "org_default",
|
||||
"pluginId": "delivery-acceptance",
|
||||
"batchId": acc_batch_id,
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"runId": run_id,
|
||||
"gateNodeId": "node-acceptance-gate",
|
||||
"workgroupId": workgroup_id,
|
||||
"vendorId": "vendor-fanyu-a",
|
||||
"subject": {"itemKey": f"{dpkg_id}.tar.gz", "mediaType": "application/json"},
|
||||
"status": "ready",
|
||||
"currentStageId": "acceptance",
|
||||
"reviewRound": 1,
|
||||
"version": 1,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
dpkg_doc = {
|
||||
"_id": dpkg_id,
|
||||
"id": dpkg_id,
|
||||
"organizationId": "org_default",
|
||||
"projectId": project_id,
|
||||
"workflowId": workflow_id,
|
||||
"parentRunId": run_id,
|
||||
"childRunId": f"{run_id}-delivery",
|
||||
"releasePartitionId": "part-1",
|
||||
"manifestDigest": "sha256:fanyu10wmanifest1234567890",
|
||||
"itemIds": [item_id],
|
||||
"workgroupId": workgroup_id,
|
||||
"vendorId": "vendor-fanyu-a",
|
||||
"status": "pending_acceptance",
|
||||
"acceptanceHumanItemId": acc_item_id,
|
||||
"artifactIds": ["art-face-blur", "art-convenience-json", "art-rrd"],
|
||||
"version": 1,
|
||||
"createdAt": now_iso(),
|
||||
"updatedAt": now_iso(),
|
||||
}
|
||||
db["humanBatches"].replace_one({"_id": acc_batch_id}, acc_batch_doc, upsert=True)
|
||||
db["humanItems"].replace_one({"_id": acc_item_id}, acc_item_doc, upsert=True)
|
||||
db["deliveryPackages"].replace_one({"_id": dpkg_id}, dpkg_doc, upsert=True)
|
||||
|
||||
# Acceptance expert claims acceptance human item & delivery package
|
||||
cur_acc_item = api_call("GET", f"/api/human-items/{acc_item_id}", user_header=acceptance_auth)
|
||||
api_call("POST", f"/api/human-items/{acc_item_id}/claim", {"version": get_item_version(cur_acc_item)}, user_header=acceptance_auth)
|
||||
cur_dpkg = api_call("GET", f"/api/delivery-packages/{dpkg_id}", user_header=acceptance_auth)
|
||||
api_call("POST", f"/api/delivery-packages/{dpkg_id}/claim", {"version": cur_dpkg["version"]}, user_header=acceptance_auth)
|
||||
print(" • 验收专家 (李工) 认领交付包与验收卡片")
|
||||
|
||||
# Acceptance expert decision
|
||||
cur_dpkg = api_call("GET", f"/api/delivery-packages/{dpkg_id}", user_header=acceptance_auth)
|
||||
decision = {
|
||||
"version": cur_dpkg["version"],
|
||||
"decision": "approved",
|
||||
"videoQuality": "qualified",
|
||||
"annotationAccuracy": 99.5,
|
||||
"annotationCompleteness": 100.0,
|
||||
"overallGrade": "A+",
|
||||
"comment": "泛宇 10w 小时首批便利店第一视角数据已通过全流程验收,准予交付并触发下游持续训练",
|
||||
}
|
||||
api_call("POST", f"/api/delivery-packages/{dpkg_id}/decisions", decision, user_header=acceptance_auth, expected_status=(200, 201))
|
||||
print(" ✅ 验收专家评级 A+ 签署交付通过!")
|
||||
|
||||
# 11. Trigger Downstream Continuous Training
|
||||
print("\n[Step 11/11] 阶段四:触发下游持续训练算子 (Qwen3-VL 22 类动作分类持续训练)...")
|
||||
from backend.operators.builtin.qwen_label_crop_classification.operator import run as run_qwen_crop
|
||||
|
||||
train_dir = pathlib.Path("./build/continuous_training_output")
|
||||
train_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
inputs = [
|
||||
{
|
||||
"name": "delivery_package",
|
||||
"data": {
|
||||
"sourceVideo": "bos://testxyz/fanyu_test/Orbbec_Ego_AZER76400GG_19700101_001_D.mp4",
|
||||
"segments": [
|
||||
{"labelId": "05", "label": "蒸点陈列与出餐", "start": 0.0, "end": 65.0},
|
||||
{"labelId": "06", "label": "热食加工与出餐", "start": 65.0, "end": 140.0},
|
||||
{"labelId": "14", "label": "收银扫码", "start": 140.0, "end": 180.0},
|
||||
{"labelId": "15", "label": "顾客交付与内部协作", "start": 180.0, "end": 200.0},
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
artifacts = run_qwen_crop(
|
||||
run_id=run_id,
|
||||
node_id="node-train",
|
||||
output_dir=train_dir,
|
||||
node_data={"hyperparameters": {"TASK_MODE": "continuous_train", "MAX_CLIP_SECONDS": 200.0}},
|
||||
inputs=inputs,
|
||||
helpers={},
|
||||
)
|
||||
print(f" ✅ 持续训练算子执行完成!已在 {train_dir} 生成 {len(artifacts)} 个模型与评测产物:")
|
||||
for art in artifacts:
|
||||
print(f" • {art.get('name')}: {art.get('uri')}")
|
||||
|
||||
print("\n" + "=" * 80)
|
||||
print("🎉🎉🎉 泛宇便利店第一视角 10w 小时人在环全流程在 POC 环境真实执行 100% 成功!")
|
||||
print("=" * 80)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
x
Reference in New Issue
Block a user