forked from longtao.wu/droboticflow
506 lines
50 KiB
Markdown
506 lines
50 KiB
Markdown
# Operations Guide
|
|
|
|
## Service Responsibilities
|
|
|
|
### API
|
|
|
|
- validates incoming payloads
|
|
- persists projects, workflows, operators, templates, settings, runs, and audit tasks
|
|
- serves the built SPA in production
|
|
- runs as a single Go binary
|
|
|
|
### Worker
|
|
|
|
- polls MongoDB for queued runs
|
|
- auto-enqueues streaming workflows based on workflow-level schedule modes:
|
|
- `schedule.interval`
|
|
- `schedule.intervalHours`
|
|
- `schedule.dayInterval + schedule.timeOfDay`
|
|
- executes workflow nodes in topological order
|
|
- pauses runs on human-review nodes
|
|
- resumes approved or rejected audit branches
|
|
- creates Kubernetes Jobs for every operator execution
|
|
- writes node input manifests into the shared workspace PVC before launching Jobs
|
|
- runs as a single Go binary
|
|
|
|
### Operator Runner
|
|
|
|
- receives `RUN_ID`, `WORKFLOW_ID`, `NODE_ID`, and `NODE_LABEL`
|
|
- executes a generic Python operator contract
|
|
- reads input manifests from the shared workspace PVC
|
|
- writes `result.json` and `stage-manifest.json` back to the shared workspace PVC
|
|
- preserves catalog defaults only when optional hyperparams are omitted; explicit malformed JSON, boolean, comparison-operator, sampling-rule, or exact-positive-integer hyperparams fail the node instead of falling back to defaults
|
|
|
|
## MongoDB Collections
|
|
|
|
- `projects`
|
|
- `operators`
|
|
- `workflows`
|
|
- `templates`
|
|
- `runs`
|
|
- `auditTasks`
|
|
- `users`
|
|
- `settings`
|
|
|
|
API and worker startup both call `EnsureIndexes` on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups.
|
|
|
|
Incremental source-state checkpoints are treated as authoritative runtime state, not best-effort metadata. Scan-workflow `DATA_SOURCE` accepts only `BOS`, `SCAN_MODE` accepts only `FULL` or `INCREMENTAL` when explicitly configured, and `INCREMENTAL_UNIT` accepts only `OBJECT` or `DIRECTORY`. The shared source runner accepts explicit `DATA_SOURCE` only as `INLINE`, `LOCAL`, `BOS`, or `UPSTREAM`; unknown values fail instead of producing an empty manifest. `SOURCE_STATE_JSON` must be a JSON object whose `seenFingerprints` entries are strings, `incrementalUnit` is `OBJECT` or `DIRECTORY`, and `directoryDepth` is a positive integer. When a completed run exposes `source-checkpoint.json`, the worker reads that file as the source of truth; unreadable, malformed, empty, missing-fingerprint, or lossy-count checkpoints fail checkpoint commit instead of falling back to stale inline artifact data.
|
|
|
|
Data-asset APIs apply the same stale-data rule for local external payloads. If `artifact.data.dataRef.payloadPath` points to a local file, that payload must be readable JSON object data; unreadable, malformed, or non-object payloads are skipped instead of using older inline `artifact.data`. Collection/package integer summaries only accept exact integer values, so lossy numeric fields are ignored rather than truncated.
|
|
|
|
## Run Lifecycle
|
|
|
|
1. frontend calls `POST /api/workflows/:id/run`
|
|
2. API inserts a `queued` run document
|
|
3. worker claims the run and marks it `running`
|
|
4. worker executes start nodes and records logs, resource samples, and artifacts
|
|
5. if a node requires review, the run moves to `waiting_for_audit`
|
|
6. audit action updates the task to `approved` or `rejected`
|
|
7. worker resumes the graph from the appropriate edge
|
|
8. run ends in `completed` or `failed`
|
|
|
|
## Local Verification
|
|
|
|
Run local verification and the security audit as separate gates:
|
|
|
|
```bash
|
|
npm run verify:local
|
|
npm run verify:security
|
|
```
|
|
|
|
`verify:local` includes TypeScript/Go/script linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, and Kubernetes manifest dry-run validation. `verify:security` runs `npm audit --audit-level=moderate` through a retry wrapper so transient registry failures do not prevent local validation from running first. `npm run verify` runs both in that order.
|
|
|
|
## Runtime Metrics
|
|
|
|
The worker exposes Prometheus text metrics at `/worker-api/metrics`. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the most recent dispatch queue depth. Kubernetes deployments annotate the worker Service for Prometheus scraping at this path.
|
|
|
|
Runs that execute on NVIDIA hardware also persist GPU utilization, GPU memory used/total in MiB, GPU count, and GPU model into `run.resourceSeries`. The monitoring page shows these fields in the resource modal. A missing GPU memory value means the operator-runner could not execute `nvidia-smi`, not that the workflow requested no GPU.
|
|
|
|
When a node requests GPU resources, the worker checks Kubernetes Node allocatable `nvidia.com/gpu` before creating the operator-runner Job. If no schedulable node can satisfy the requested GPU count and configured GPU node selector, the node fails immediately with a `gpu request cannot be scheduled` message instead of leaving a Pending Job to time out. This is a real cluster-capacity check; it does not emulate GPU execution.
|
|
|
|
The workflow editor also reads the same `GET /worker-api/capacity` snapshot. If the current workflow contains GPU-requesting nodes and the loaded capacity proves that no schedulable GPU exists, or that a single node requests more GPUs than any schedulable node can provide, the run button is disabled before a run is created. This prevents predictable POC failures while keeping the worker-side Kubernetes preflight as the authoritative runtime guard.
|
|
|
|
## Job Retention
|
|
|
|
New operator-runner Jobs use `OPERATOR_JOB_TTL_SECONDS` as `ttlSecondsAfterFinished`, defaulting to one day. For clusters upgraded from older manifests, clean pre-existing finished Jobs with:
|
|
|
|
```bash
|
|
DRY_RUN=true npm run k8s:cleanup-jobs
|
|
npm run k8s:cleanup-jobs
|
|
```
|
|
|
|
Set `OPERATOR_JOB_CLEANUP_MIN_AGE_SECONDS` and `OPERATOR_JOB_CLEANUP_STATUSES=succeeded,failed` to adjust the retention window and terminal states.
|
|
|
|
## Account Security
|
|
|
|
The API supports HttpOnly cookie-session auth through `/api/auth/*` and super-admin organization/user management under `/api/admin/*`. Existing deployments keep their current behavior unless `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` is set. Before enabling it, bootstrap the first super admin and verify login through `/api/auth/login`.
|
|
|
|
## Shell Command Policy
|
|
|
|
Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Set `OPERATOR_ALLOW_SHELL_COMMANDS=true` only for deployments that need shell operators, and constrain commands with `OPERATOR_COMMAND_ALLOWLIST_REGEX` and `OPERATOR_COMMAND_DENYLIST_REGEX`. These are platform-controlled variables passed from the worker to operator-runner Jobs, and workflow hyperparams cannot override them.
|
|
|
|
## Acceptance Checks
|
|
|
|
Run acceptance checks after a deployment from a local shell or external CI. Live workflow acceptance requires `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. GPU acceptance loads `scripts/load-deploy-env.sh`, so on `dev` it uses the POC kubeconfig when `DEPLOY_ENV` is unset; other environments still require a kubeconfig with permission to create a temporary Job in `K8S_NAMESPACE`. The GPU acceptance wrapper validates namespace, image, timeout, GPU count, selector, tolerations JSON, resource name, Job name, and runtime class before any `kubectl` call; invalid acceptance inputs fail locally instead of creating a malformed Job. It also checks for a selector-matched, non-unschedulable Node with enough allocatable GPU before creating its probe Job, so a CPU-only cluster or selector mismatch reports a hardware-capacity blocker explicitly. The deployed worker exposes the same allocatable-node view at `GET /worker-api/capacity`; API callers reach it through the normal worker API proxy and should expect `gpu.available=true` only after the target cluster has schedulable `nvidia.com/gpu` resources.
|
|
|
|
`npm run acceptance:poc-gpu-capacity` verifies the deployed POC worker capacity endpoint from inside the cluster. It creates a short-lived Pod with the deployed `OPERATOR_RUNNER_IMAGE`, calls `http://droboticflow-worker:3002/worker-api/capacity`, and requires node-level capacity evidence. The default expectation is CPU-only POC (`gpu.available=false`, zero schedulable GPU, zero max single-node GPU); set `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` only after a real GPU node pool and NVIDIA runtime/device plugin are installed.
|
|
|
|
`npm run acceptance:poc-gpu-workflow` is the POC workflow-level GPU check. With `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, it logs in and exercises the normal project/workflow/run APIs. Without session credentials, it uses `DROBOTICFLOW_API_TOKEN` from the environment or reads it from `droboticflow-auth-secret`, calls the bounded `/api/acceptance/poc-gpu-workflow` route to create or reuse one fixed GPU acceptance workflow, port-forwards the deployed worker service, and triggers `/worker-api/process-once`. The default CPU-only mode requires a failed run log containing `gpu request cannot be scheduled`. After real GPU hardware is installed, run with `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true` or `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true`; the same script then requires a completed run with artifacts. Each invocation creates a fresh run for evidence but reuses the fixed POC GPU acceptance project/workflow to avoid cluttering the platform with duplicate acceptance containers. This verifies the worker path, not only the standalone `nvidia-smi` probe, while regular business writes still require a user session. `GET /api/acceptance/poc-readiness` and the dashboard expose this latest workflow-level proof as `gpuWorkflowPreflight`, separate from the hardware-capacity `gpu` gate. `npm run acceptance:poc-readiness` also calls that bounded platform readiness endpoint by default and requires matching CPU-only blocked evidence or GPU-available completed evidence unless `POC_PLATFORM_READINESS_ENABLED=false` is explicitly set for isolated diagnostics.
|
|
|
|
## Video Preview Performance
|
|
|
|
Monitoring artifact previews, data-asset artifact previews, workflow edge previews, sampling previews, and object-storage video previews lazy-load browser media. The preview card initially renders a play action without a `<video src>`; the real Range-backed, `transcode=browser` URL is attached only after the user starts the preview. This keeps artifact lists and run detail pages from opening multiple large MP4 streams while the operator is only browsing metadata.
|
|
|
|
## Generated Artifacts
|
|
|
|
Build output, local Playwright screenshots/videos, pytest caches, and zip bundles should stay out of git. Regenerate validation artifacts locally or publish them as CI/release artifacts when they need to be shared.
|
|
|
|
Start the stack locally:
|
|
|
|
```bash
|
|
npm run dev:full
|
|
```
|
|
|
|
Run live workflow acceptance against the API and worker:
|
|
|
|
```bash
|
|
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
|
|
./scripts/live-workflow-acceptance.sh
|
|
```
|
|
|
|
`WORKFLOW_ACCEPTANCE_SOURCE_PATH` may be a filesystem/PVC path or a `bos://bucket/prefix` path. The `真实数据源` runner infers `LOCAL` or `BOS` from the path and rejects a run that has no `SOURCE_TEXT`, `SOURCE_JSON`, `SOURCE_PATH`, configured BOS source, local `STORAGE_PREFIX`, or upstream artifact input. This prevents an unconfigured source node from completing with an empty manifest.
|
|
|
|
For deployed workflow acceptance, pass the deployed URL or set `BASE_URL` / `API_BASE_URL`, and set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` so localhost defaults are disabled.
|
|
|
|
Run the production-style WJZH real-data regression:
|
|
|
|
```bash
|
|
API_BASE_URL=http://<node-ip>:30080 \
|
|
SOURCE_PATH=bos://<bucket>/<input-prefix> \
|
|
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
|
|
STORAGE_CONNECTION_ID=<storage-connection-id> \
|
|
MEDIUM_VIDEO=<medium-video-filename> \
|
|
HUGE_VIDEO=<huge-video-filename> \
|
|
ROUTE_DEVICE_ID=<device-id-present-in-source> \
|
|
FILTER_DATE_START=<yyyy-mm-dd> \
|
|
FILTER_DATE_END=<yyyy-mm-dd> \
|
|
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
|
|
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
|
|
INCREMENTAL_PLACE=<real-place-label> \
|
|
INCREMENTAL_CAMERA_MODEL=<camera-model> \
|
|
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
|
|
QUALITY_MIN_CLARITY=<threshold> \
|
|
QUALITY_MAX_SHAKE=<threshold> \
|
|
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
|
|
HUGE_WINDOW_SECONDS=<seconds> \
|
|
MEDIUM_WINDOW_SECONDS=<seconds> \
|
|
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
BATCH_WINDOW_SECONDS=<seconds> \
|
|
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
|
|
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
INCREMENTAL_WINDOW_SECONDS=<seconds> \
|
|
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
|
|
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
|
|
python3 scripts/wjzh_operator_regression.py --preflight-only
|
|
```
|
|
|
|
The WJZH regression script does not provide built-in quality thresholds, video timing, or sampling thresholds; `QUALITY_MIN_CLARITY`, `QUALITY_MAX_SHAKE`, `QUALITY_MAX_HAND_OCCLUSION`, the `*_SECONDS` timing values, and `SAMPLING_RULES_JSON` must match the target dataset before the run is accepted as evidence. Quality thresholds are normalized acceptance gates and must be in `[0, 1]`; integer timing values must be exact safe positive integers; out-of-range values fail preflight instead of weakening the run evidence. See the full runbook in [`docs/wjzh-operator-regression.md`](wjzh-operator-regression.md).
|
|
|
|
Run browser object-storage acceptance against a live storage connection:
|
|
|
|
```bash
|
|
BASE_URL=http://<node-ip>:30080 \
|
|
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
|
|
OBJECT_STORAGE_BUCKET=<bucket> \
|
|
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
|
|
npm run acceptance:object-storage
|
|
```
|
|
|
|
`OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` must point to a non-empty UTF-8 text or JSON file that represents real acceptance content. The script uses that same payload for browser create, preview, rename, upload, and delete checks; it no longer has a built-in object body fallback. Set `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE` when the file extension is ambiguous, and set `OBJECT_STORAGE_VIDEO_KEY=<existing-video-key>` when video playback should be included. `PLAYWRIGHT_IGNORE_HTTPS_ERRORS` accepts only `true/false`, `1/0`, `yes/no`, or `on/off` when explicitly set. The script creates temporary objects under an `object-storage-acceptance-*` prefix and removes them at the end. For session-authenticated public APIs, provide an organization admin through `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`; for token-protected non-session APIs, provide `DROBOTICFLOW_API_TOKEN`.
|
|
|
|
Node/Playwright acceptance scripts share `scripts/lib/acceptance-client.mjs` for API token headers, Playwright session-cookie CSRF propagation, JSON requests, retries, and optional session login. Keep new browser acceptance scripts on that helper instead of carrying local request wrappers. `scripts/playwright-functional-check.mjs` and `scripts/playwright-preview-operators-check.mjs` also forward `DROBOTICFLOW_API_TOKEN` into browser contexts so UI fetches and preflight API setup use the same token-authenticated path.
|
|
|
|
Run operator preview acceptance with explicit real payloads and source paths:
|
|
|
|
```bash
|
|
BASE_URL=http://<node-ip>:30080 \
|
|
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
|
|
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
|
|
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
|
|
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
|
|
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
|
|
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
|
|
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
|
|
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
|
|
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
|
|
npm run acceptance:operators-preview
|
|
```
|
|
|
|
The CSV, text, and JSON payloads must be supplied by the caller; the script materializes those payloads inside operator-runner jobs and no longer creates built-in payload data. `PREVIEW_CHECK_JSON_PAYLOAD_FILE` must contain a non-empty object array or an object with `items[]`; the extract/filter/map variables define how the JSON data-operator chain is validated. `PREVIEW_CHECK_JSON_FIELD_MAP_JSON` must use non-empty string source/destination fields and must not map multiple input fields to the same output field. The scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs, not just from the machine launching the script. Optional checks: `PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON`, `PREVIEW_CHECK_SCAN_MIN_FILES`, `PREVIEW_CHECK_ROSBAG_TOPIC`, and `PREVIEW_CHECK_LEROBOT_ROBOT_TYPE`. `PREVIEW_CHECK_SCAN_MIN_FILES` must be an exact safe positive integer.
|
|
|
|
Run the deployed POC operator-runner full builtin-operator acceptance Job:
|
|
|
|
```bash
|
|
npm run acceptance:poc-operators
|
|
```
|
|
|
|
Build and push all three POC images from the cluster when local Docker or Docker Hub pull limits would make a workstation build unreliable:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc \
|
|
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
|
REGISTRY_NAMESPACE=dataflow-poc \
|
|
npm run images:poc-kaniko
|
|
```
|
|
|
|
The Kaniko wrapper uploads the current tracked worktree files into the shared `/workspace` PVC by default, creates one Kubernetes Job each for API, worker, and operator-runner, and mounts the existing `droboticflow-registry` Secret as `/kaniko/.docker/config.json`. Set `POC_KANIKO_CONTEXT_SOURCE=head` when a reproducible build from committed `HEAD` is required. `IMAGE_TAG` defaults to `<short-sha>-poc-<timestamp>` for rollback-friendly POC pushes. The uploaded build context is removed on success, failure, or timeout by default; set `POC_KANIKO_CLEAN_CONTEXT=false` only when retaining the context for cluster-side diagnostics. `POC_KANIKO_REGISTRY_MIRRORS` defaults to `mirror.gcr.io` so base image pulls do not rely on unauthenticated Docker Hub quota; set it to a comma-separated mirror list or an empty value only when the cluster registry path is already reliable. POC also sets `POC_KANIKO_GOPROXY=https://goproxy.cn,direct` by default for Go module downloads inside the cluster; override it when the cluster has a different approved module proxy.
|
|
|
|
Before launching the Job, run the read-only readiness gate:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc npm run acceptance:poc-readiness
|
|
```
|
|
|
|
The readiness gate emits JSON and exits non-zero when the deployed API/worker images are not ready, the configured operator-runner image is missing, the deployed worker GPU capacity endpoint cannot be verified, the platform `gpuWorkflowPreflight` gate is missing or stale, generated fixtures are enabled, or the real fixture manifest is not usable on the shared workspace PVC. GPU capacity readiness defaults to a local `kubectl port-forward` probe against `service/droboticflow-worker`, disables local HTTP proxies for that loopback request, and expects CPU-only POC capacity unless `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` is set. Platform GPU workflow readiness resolves `API_BASE_URL` / `BASE_URL` first, otherwise reads `service/droboticflow-api-public`, and authenticates with `DROBOTICFLOW_API_TOKEN` or the token in `droboticflow-auth-secret`; run `npm run acceptance:poc-gpu-workflow` before final CPU-only readiness, or `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true npm run acceptance:poc-gpu-workflow` before final GPU-available readiness. Use `POC_GPU_CAPACITY_PROBE_MODE=pod` only when port-forward is unavailable and a short-lived in-cluster Pod probe is acceptable. Its default manifest path is `${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json`, and every manifest or fixture path must stay under the same `WORKSPACE_MOUNT_PATH` used by the deployed Pods. Manifest readiness means the JSON is parseable, declares `xplatform`, `csv`, `json`, `lerobot`, `rosbag`, `model`, and `shellCommand`, and every declared fixture path is readable with the expected file or directory type from the deployed Pod. `POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true` is diagnostic-only and blocks strict POC readiness instead of producing a `ready` result. The gate decodes secret keys only to check that required values are non-empty and never prints token, cookie, password, AK, SK, or kubeconfig data. A `ready` result is only a precondition for final acceptance; the strict pass signal still comes from `npm run acceptance:poc-operators` returning `fullExternalAcceptance=true`.
|
|
|
|
For the final no-mock POC gate, prefer the strict orchestrator:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc npm run acceptance:poc-strict
|
|
```
|
|
|
|
`acceptance:poc-strict` refuses dependency-isolation scopes by requiring `POC_ACCEPTANCE_REQUIRED_EXTERNALS` to include `bos,mongo`. It runs `acceptance:poc-readiness` and only then launches `acceptance:poc-operators`. Readiness failures emit a standard `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` preflight-failure result and stop before the Kubernetes acceptance Job is created. With `POC_ACCEPTANCE_IMPORT_RESULT=true`, POC strict acceptance imports those blocked preflights into the `POC 算子验收` platform history exactly like Job results; it uses `POC_ACCEPTANCE_IMPORT_API_BASE_URL`, `API_BASE_URL`, or `BASE_URL` when provided, otherwise it discovers the `droboticflow-api-public` LoadBalancer URL and loads `DROBOTICFLOW_API_TOKEN` from `droboticflow-auth-secret`. Use narrower `POC_ACCEPTANCE_REQUIRED_EXTERNALS` only when intentionally running `acceptance:poc-operators` as dependency-isolation diagnostics, not as final strict evidence.
|
|
|
|
`scripts/poc-operator-full-acceptance.sh` loads the `poc` deployment environment on `dev`, reads the current `OPERATOR_RUNNER_IMAGE`, workspace PVC, Mongo secret, and BOS workspace secret references from Kubernetes, mounts `scripts/poc_operator_full_acceptance_job.py` into a temporary Job, and prints one `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` result. The Job executes builtin modules from the deployed runner image and writes evidence under `/workspace/acceptance/<run-id>/`; `<run-id>` is set from `POC_OPERATOR_ACCEPTANCE_JOB_NAME` so the raw result, Kubernetes Job, imported platform run source, and workspace evidence directory stay traceable to the same acceptance execution. If the PVC is misconfigured and the runner cannot write there, the run fails by default instead of silently accepting temporary evidence. Set `POC_ACCEPTANCE_ALLOW_TMP_FALLBACK=true` only for diagnostic runs that intentionally write to `/tmp/droboticflow-acceptance/<run-id>/` and report `workspaceFallbackReason`.
|
|
|
|
Final POC operator acceptance requires real input fixtures that already exist inside the Job container, usually on the shared workspace PVC mounted at `${WORKSPACE_MOUNT_PATH:-/workspace}`. Prefer the default `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH=/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json` file so the platform run records one reproducible fixture contract:
|
|
|
|
```json
|
|
{
|
|
"fixtures": {
|
|
"xplatform": "/workspace/fixtures/xplatform",
|
|
"csv": "/workspace/fixtures/records.csv",
|
|
"json": "/workspace/fixtures/records.json",
|
|
"lerobot": "/workspace/fixtures/lerobot",
|
|
"rosbag": "/workspace/fixtures/rosbag2",
|
|
"model": "/workspace/fixtures/model.json"
|
|
},
|
|
"json": {
|
|
"filterField": "label",
|
|
"filterExpected": "keep",
|
|
"extractField": "items.0.decision",
|
|
"fieldMap": {"label": "decision", "score": "confidence"},
|
|
"templateText": "accepted={items.0.id}; decision={items.0.decision}"
|
|
},
|
|
"video": {
|
|
"minClarity": 0,
|
|
"maxShake": 1,
|
|
"maxHandOcclusion": 1,
|
|
"clarityScale": 35,
|
|
"segmentDurationSeconds": 1,
|
|
"frameIntervalSeconds": 0.5
|
|
},
|
|
"xplatform": {
|
|
"dateStart": "",
|
|
"dateEnd": "",
|
|
"place": "",
|
|
"type": ""
|
|
},
|
|
"audit": {
|
|
"rules": [{"id": "all-items", "field": "collectionId", "operator": "exists", "samplePercent": 100}]
|
|
},
|
|
"shellCommand": "python3 /workspace/fixtures/shell_acceptance.py"
|
|
}
|
|
```
|
|
|
|
The legacy explicit variables remain supported and take precedence over the manifest: `POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH`, `POC_ACCEPTANCE_CSV_INPUT_PATH`, `POC_ACCEPTANCE_JSON_INPUT_PATH`, `POC_ACCEPTANCE_LEROBOT_SOURCE_PATH`, `POC_ACCEPTANCE_ROSBAG_SOURCE_PATH`, `POC_ACCEPTANCE_MODEL_JSON_PATH`, and `POC_ACCEPTANCE_SHELL_COMMAND`. The script fails by default when any required real fixture path is missing or unreadable. `POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true` exists only for explicitly labeled diagnostics; generated video, CSV, JSON, LeRobot, ROSBag, model, or shell-command fixtures are not valid final POC acceptance evidence. Tune real fixture semantics with either the manifest fields above or the equivalent environment variables: `POC_ACCEPTANCE_JSON_FILTER_FIELD`, `POC_ACCEPTANCE_JSON_FILTER_EXPECTED`, `POC_ACCEPTANCE_JSON_EXTRACT_FIELD`, `POC_ACCEPTANCE_JSON_FIELD_MAP_JSON`, `POC_ACCEPTANCE_TEMPLATE_TEXT`, `POC_ACCEPTANCE_MIN_CLARITY`, `POC_ACCEPTANCE_MAX_SHAKE`, `POC_ACCEPTANCE_MAX_HAND_OCCLUSION`, `POC_ACCEPTANCE_CLARITY_SCALE`, `POC_ACCEPTANCE_XPLATFORM_DATE_START`, `POC_ACCEPTANCE_XPLATFORM_DATE_END`, `POC_ACCEPTANCE_XPLATFORM_PLACE`, `POC_ACCEPTANCE_XPLATFORM_TYPE`, `POC_ACCEPTANCE_SEGMENT_DURATION_SECONDS`, `POC_ACCEPTANCE_FRAME_INTERVAL_SECONDS`, and `POC_ACCEPTANCE_AUDIT_RULES_JSON`.
|
|
|
|
By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, or `MONGO_URI` is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with `POC_ACCEPTANCE_BOS_ENDPOINT` only when validating a different bucket region. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose the bucket name. D-Cloud FAT operators were removed from the builtin catalog; historical D-Cloud result artifacts may still render in Monitoring, but current strict POC evidence is based on BOS, Mongo, real fixtures, operator coverage, and repository policy.
|
|
|
|
The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `externalVerificationPassed`, `externalVerificationMissing`, `externalVerificationPassedCount`, `externalVerificationRequiredCount`, `operatorRepositoryCount`, `operatorRepositoryRequiredCount`, `operatorRepositoryCoverageComplete`, `operatorRepositoryPolicyClean`, `operatorRepositoryPolicyViolations`, `statusCounts`, and skipped-count fields. The Job sets `fullExternalAcceptance=true` only when the same strict evidence set is already present in the raw result: scope is `full`, required externals include `bos,mongo`, every required external system is verified, every operator is covered and passed, every builtin operator has a matching local repository manifest, repository policy violations are empty, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal; any `dependency-isolation` result can prove the configured dependencies but not the omitted external systems. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, and Mongo requires a write followed by an independent count/drop verification. The Monitoring summary card applies the same strict display rules.
|
|
|
|
When the Job emits a `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` marker, the wrapper extracts it from the Job logs even if the Job exits failed. If `POC_ACCEPTANCE_IMPORT_API_BASE_URL`, `API_BASE_URL`, or `BASE_URL` is set, it posts that result to `/api/acceptance/poc-operator-result` and records it as a normal platform run under the `POC 算子验收` project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, or the shared `DROBOTICFLOW_API_TOKEN` for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a `poc-operator-full-acceptance-result.json` artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Automation can read the narrowed POC gate summary with `GET /api/acceptance/poc-readiness` and `Authorization: Bearer ${DROBOTICFLOW_API_TOKEN}`; this endpoint returns only GPU, D-Cloud, strict-acceptance, fixture, coverage, and next-action evidence, while the full `/api/dashboard` remains session-only in public POC deployments. For `fullExternalAcceptance=true`, the import must also carry Kubernetes source evidence (`source.namespace`, `source.jobName`, `source.image`, and `source.imageId`); for every imported result, `result.runId` should equal `source.jobName`, any provided `source.imageId` must be a full `@sha256:<64 hex>` image digest, must identify the same repository as `source.image`, and any provided `source.image` must not contain whitespace. The Monitoring card displays these fields so a final pass is tied to the exact POC Job and operator-runner image digest that produced it. Set `POC_ACCEPTANCE_IMPORT_RESULT=false` for one-off diagnostics that should not create or update platform acceptance history.
|
|
|
|
After importing a POC operator result, verify the platform record with the same Kubernetes/Mongo evidence path instead of relying on the import response alone:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc \
|
|
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
|
|
OPERATOR_RUNNER_IMAGE="${OPERATOR_RUNNER_IMAGE}" \
|
|
npm run acceptance:poc-import-verify -- \
|
|
--run-id "run-poc-operator-acceptance-<imported-run-id>" \
|
|
--expected-runner-image "${OPERATOR_RUNNER_IMAGE}" \
|
|
--expected-scope dependency-isolation \
|
|
--expected-external mongo \
|
|
--expected-human-review-task-count 10
|
|
```
|
|
|
|
`acceptance:poc-import-verify` creates a short-lived `mongo:7` Pod in the target namespace, injects `MONGO_URI` from `droboticflow-mongo-secret`, reads `MONGO_DB` from `droboticflow-config`, asserts that the imported run contains `poc-operator-full-acceptance-result.json`, checks the runner image, requires a sha256 image digest, requires Kubernetes source evidence (`namespace`, `jobName`, `image`, and `imageId`), requires real fixture evidence by default (`allFixturesReal=true`, `generatedFixtureCount=0`, positive `realFixtureCount`, absolute fixture paths, and no generated fixture sources), verifies every `--expected-external` system is in `externalVerificationPassed`, enforces clean repository-policy evidence, and optionally checks `human_review` task evidence. It deletes the probe Pod by default and prints only the bounded verification summary, never the Mongo URI or secret values. `--result-file <file>` can be used instead of `--run-id`; the script derives the platform run id from the raw `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` marker.
|
|
|
|
If strict acceptance cannot even create the Kubernetes Job because required readiness inputs are missing, the wrapper still emits a standard failed `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` preflight result. A single blocker keeps its specific `preflightStage` such as `fixture-readiness`; multiple blockers use `preflightStage=preflight-readiness`, list each specific stage in `preflightStages`, and include one failed `records` entry per blocker. Fixture-readiness failures also include a bounded `fixtureDiscovery` evidence summary from the deployed API Pod's shared workspace mount: scanned roots, missing fixture categories, selected paths if any, candidate counts, and next actions. The preflight result also includes the static builtin-operator repository evidence fields (`static`, `operatorRepositoryCount`, `operatorRepositoryRequiredCount`, `operatorRepositoryCoverageComplete`, `operatorRepositoryPolicyClean`, and `operatorRepositoryPolicyViolations`) so a blocked platform run still proves whether local independent operator repositories and no-mock/no-simulation/no-hardcoded policy metadata are complete. When platform import is enabled, the failure is imported as a failed POC acceptance run with `preflightFailure=true`, `acceptanceScope=full`, `fullExternalAcceptance=false`, and blocking coverage derived from the affected repository manifests. This keeps the platform history honest: missing real fixture contracts are visible before a Job is created.
|
|
|
|
Verify an imported strict preflight blocker with the same Mongo probe by naming the expected stage. This switches the verifier to failed-run semantics and checks `preflightFailure`, `preflightStages`, failed preflight records, blocking coverage, and failure messages instead of requiring passed-run fixture/image-digest evidence:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc \
|
|
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
|
|
npm run acceptance:poc-import-verify -- \
|
|
--run-id "run-poc-operator-acceptance-<strict-preflight-run-id>" \
|
|
--expected-scope full \
|
|
--expected-preflight-stage fixture-readiness
|
|
```
|
|
|
|
Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH` or all explicit real fixture inputs, including `POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH`, `POC_ACCEPTANCE_CSV_INPUT_PATH`, `POC_ACCEPTANCE_JSON_INPUT_PATH`, `POC_ACCEPTANCE_LEROBOT_SOURCE_PATH`, `POC_ACCEPTANCE_ROSBAG_SOURCE_PATH`, `POC_ACCEPTANCE_MODEL_JSON_PATH`, and `POC_ACCEPTANCE_SHELL_COMMAND`. Manifest and explicit fixture paths must point under `WORKSPACE_MOUNT_PATH`; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with `fixture-readiness` coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later.
|
|
|
|
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, `requiredExternals` containing `bos,mongo`, `externalVerificationPassed` containing both systems, empty `externalVerificationMissing`, `externalVerificationPassedCount>=2`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, `realFixtureCount>0`, structured `fixtureSources.items` covering `csv`, `json`, `lerobot`, `model`, `rosbag`, and `xplatform` with absolute fixture paths and `generated=false` for every item, fixture counts that match the `fixtureSources.items` evidence, `static.operatorFileCount>0`, `static.repositoryManifestCount==static.operatorFileCount`, empty static manifest/duplicate/legacy/helper/hardcoded-source violation lists, `operatorRepositoryCoverageComplete=true`, and `operatorRepositoryPolicyClean=true`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
|
|
|
|
Configure or refresh the POC real fixture manifest from local fixture files and directories:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover
|
|
```
|
|
|
|
`acceptance:poc-fixtures:discover` scans existing files and directories on the deployed POC shared `/workspace` PVC through the API Pod mount. It reports candidate `xplatform`, `csv`, `json`, `lerobot`, `rosbag`, and `model` paths, but it does not generate data, infer a shell validation command, or write a manifest unless `--write-manifest` is explicitly passed and `POC_ACCEPTANCE_SHELL_COMMAND` is configured. Historical `/workspace/acceptance/...` and `/workspace/droboticflow-acceptance/...` output, build contexts, image-build scratch directories, workflow run output under `/workspace/runs`, and `poc-workflow-acceptance-*` scratch inputs are excluded by default so generated artifacts and deployment scratch trees cannot be selected as final fixture input or slow down a broad `/workspace` scan. The discovery result records `excludedRoots` next to `roots` for auditability. Set `POC_ACCEPTANCE_DISCOVERY_INCLUDE_ACCEPTANCE_OUTPUT=true` only when auditing old runs; override the full prune list with `POC_ACCEPTANCE_DISCOVERY_EXCLUDE_ROOTS=/workspace/archive,/workspace/tmp` when a POC cluster has different large non-fixture trees. Narrow scanning with `POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets` when the PVC contains large unrelated trees.
|
|
|
|
When discovery finds every real fixture and the shell validation command is known, write the manifest in place:
|
|
|
|
```bash
|
|
POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets \
|
|
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
|
|
DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover -- --write-manifest
|
|
```
|
|
|
|
If the real fixtures are still local to the machine running the command, copy them to the POC PVC and write the manifest with explicit paths:
|
|
|
|
```bash
|
|
POC_ACCEPTANCE_LOCAL_XPLATFORM_SOURCE_PATH=/path/to/xplatform-root \
|
|
POC_ACCEPTANCE_LOCAL_CSV_INPUT_PATH=/path/to/records.csv \
|
|
POC_ACCEPTANCE_LOCAL_JSON_INPUT_PATH=/path/to/records.json \
|
|
POC_ACCEPTANCE_LOCAL_LEROBOT_SOURCE_PATH=/path/to/lerobot-root \
|
|
POC_ACCEPTANCE_LOCAL_ROSBAG_SOURCE_PATH=/path/to/rosbag-or-dir \
|
|
POC_ACCEPTANCE_LOCAL_MODEL_JSON_PATH=/path/to/model.json \
|
|
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
|
|
npm run acceptance:poc-fixtures
|
|
```
|
|
|
|
`acceptance:poc-fixtures` copies only caller-provided non-empty local files/directories, or validates existing paths under `${WORKSPACE_MOUNT_PATH:-/workspace}` when those are already present on the shared PVC. It writes the manifest to `${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json` by default and prints the corresponding `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH` export line. It does not create mock, simulated, generated, or hardcoded acceptance data; dry-run mode (`POC_ACCEPTANCE_FIXTURE_DRY_RUN=true`) validates local inputs and prints the resolved workspace paths without touching Kubernetes.
|
|
|
|
The historical D-Cloud FAT secret helper is retained only for old deployments that still run external D-Cloud operators. The current builtin catalog does not register those operators, and strict POC acceptance does not require this secret:
|
|
|
|
```bash
|
|
DCLOUD_FAT_USERNAME=... \
|
|
DCLOUD_FAT_PASSWORD=... \
|
|
DCLOUD_FAT_SIGNIN_URL=https://sso-fat.d-robotics.cc/api/signin \
|
|
DCLOUD_FAT_BASE_URL=https://cloud-fat.d-robotics.cc \
|
|
DCLOUD_ORG_SYMBOL=... \
|
|
DCLOUD_SOURCEAPP=... \
|
|
scripts/configure-poc-dcloud-secret.sh
|
|
```
|
|
|
|
The script also accepts `DCLOUD_FAT_AUTH_TOKEN`/`DCLOUD_API_TOKEN` or `DCLOUD_FAT_COOKIE`/`DCLOUD_COOKIE` instead of username/password. When username/password auth is used, set `DCLOUD_FAT_SIGNIN_URL` for SSO-hosted FAT login endpoints such as `https://sso-fat.d-robotics.cc/api/signin`; otherwise the runtime falls back to `${DCLOUD_FAT_BASE_URL}/${DCLOUD_FAT_SIGNIN_ENDPOINT:-/api/signin}`. JWT-style auth tokens are decoded locally and rejected when the `exp` claim is missing, invalid, or already expired. The script writes canonical secret keys through `kubectl apply -f -` without placing secret values in kubectl command-line arguments. Use `POC_DCLOUD_SECRET_DRY_RUN=true scripts/configure-poc-dcloud-secret.sh` to validate the configured key set without printing values or touching the cluster.
|
|
|
|
Every builtin operator directory carries a local release manifest at `operator-repos/builtin/<slug>/release.json` and a generated repository manifest at `operator-repos/builtin/<slug>/repository.json`. The release manifest is the production image truth: it records the digest-pinned CCR image, source commit, source tree hash, environment release evidence, and acceptance evidence. Regenerate repository manifests with `npm run operators:manifests` after contract changes, and enforce drift checks with `npm run operators:manifests:check` plus `npm run operators:images:check`. These checks require every builtin operator to have a matching `release.json`, require `release.json.imageId == repository.json.imageId == catalog.imageId`, and fail when the operator source tree changes without a refreshed release manifest. The POC acceptance Job promotes this static check into top-level `operatorRepository*` evidence fields, and the platform import gate refuses a `fullExternalAcceptance=true` payload when repository coverage is incomplete or policy violations are present. A strict full POC acceptance run must include `bos,mongo`; D-Cloud FAT API operators are intentionally absent from the current builtin catalog.
|
|
|
|
The POC static operator-layout check derives the builtin operator count from the deployed runner image's `backend/operators/builtin/*/operator.json` files. It fails when no builtin metadata exists, when any operator lacks `repository.json`, when repository manifests do not match their paired metadata, or when builtin operator IDs/names are duplicated. Do not add fixed operator-count assertions to the acceptance Job; adding or removing a real builtin operator should update the catalog, metadata, generated repository manifest, docs, and tests together.
|
|
|
|
Run X-platform operator-chain acceptance against an explicit real dataset:
|
|
|
|
```bash
|
|
BASE_URL=http://<node-ip>:30080 \
|
|
XPLATFORM_SOURCE_PATH=/workspace/datasets/xplatform/<collection-root> \
|
|
XPLATFORM_MIN_CLARITY=<threshold> \
|
|
XPLATFORM_MAX_SHAKE=<threshold> \
|
|
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
|
|
XPLATFORM_CLARITY_SCALE=<scale> \
|
|
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
|
|
XPLATFORM_RULES_JSON='[{"id":"...", "field":"quality.clarity", "operator":"gte", "value":0.3, "samplePercent":100}]' \
|
|
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
|
|
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
|
|
XPLATFORM_MIN_VIDEO_COUNT=<count> \
|
|
XPLATFORM_MIN_FILTERED_COUNT=<count> \
|
|
XPLATFORM_MIN_CLIP_COUNT=<count> \
|
|
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
|
|
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
|
|
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
|
|
npm run acceptance:xplatform -- --preflight-only
|
|
```
|
|
|
|
For BOS-backed datasets, provide a storage connection selector and use a full `bos://bucket/prefix` source path:
|
|
|
|
```bash
|
|
BASE_URL=http://<node-ip>:30080 \
|
|
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
|
|
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
|
|
XPLATFORM_MIN_CLARITY=<threshold> \
|
|
XPLATFORM_MAX_SHAKE=<threshold> \
|
|
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
|
|
XPLATFORM_CLARITY_SCALE=<scale> \
|
|
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
|
|
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
|
|
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
|
|
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
|
|
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
|
|
XPLATFORM_MIN_VIDEO_COUNT=<count> \
|
|
XPLATFORM_MIN_FILTERED_COUNT=<count> \
|
|
XPLATFORM_MIN_CLIP_COUNT=<count> \
|
|
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
|
|
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
|
|
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
|
|
npm run acceptance:xplatform -- --preflight-only
|
|
```
|
|
|
|
The X-platform acceptance script does not generate synthetic videos, fallback data, sampling rules, quality thresholds, slice/annotation parameters, or minimum-count assertions. It creates a temporary project/workflow that runs `BOS 增量扫描` when the source is BOS, then `X 平台数据集索引`, collection validation, video quality, filtering, slicing, annotation, packaging, sampling, and result upload. `XPLATFORM_RULES_JSON`, `XPLATFORM_MIN_*`, `XPLATFORM_MIN_CLARITY`, `XPLATFORM_MAX_SHAKE`, `XPLATFORM_MAX_HAND_OCCLUSION`, `XPLATFORM_CLARITY_SCALE`, `XPLATFORM_SEGMENT_DURATION_SECONDS`, and `XPLATFORM_FRAME_INTERVAL_SECONDS` must match the target real dataset before the run is accepted as evidence. X-platform quality thresholds are finite normalized `[0, 1]` gates, matching the runtime `clarity`, `shake`, and `handOcclusion` metrics; out-of-range or non-finite values fail before any live workflow is created. `XPLATFORM_CLARITY_SCALE` and `XPLATFORM_FRAME_INTERVAL_SECONDS` must be finite positive numbers, while `XPLATFORM_SEGMENT_DURATION_SECONDS` and every `XPLATFORM_MIN_*` count must be exact safe positive integers to match operator-runner runtime validation and acceptance assertions. `XPLATFORM_RULES_JSON` is validated with the same rule-shape gate as the WJZH regression: every rule needs `id`, `field`, `operator`, `value`, and `samplePercent`, ids must be unique, operators must be supported, and `samplePercent` must be finite numeric data in `(0, 100]`; booleans are rejected instead of being coerced. Optional filters such as `XPLATFORM_DATE_START`, `XPLATFORM_DATE_END`, `XPLATFORM_PLACE`, and `XPLATFORM_TYPE` further narrow the target dataset. For protected APIs, use `DROBOTICFLOW_API_TOKEN` or `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD`.
|
|
|
|
Python live-acceptance and reconciliation scripts share `scripts/lib/droboticflow_acceptance.py` for API token headers, session login cookies, CSRF headers, JSON requests, optional transient-error retries, and binary object-storage requests. Keep new Python acceptance, regression, or operational API scripts on that shared client so protected deployments are tested through the same real auth path as `scripts/xplatform-live-acceptance.py`, `scripts/wjzh_operator_regression.py`, and `scripts/reconcile_system_data.py`.
|
|
|
|
Reconcile persistent system data and remove legacy demo records:
|
|
|
|
```bash
|
|
export K8S_ENDPOINT=https://<api-server>:6443
|
|
# Optional for token-protected APIs when DROBOTICFLOW_REQUIRE_SESSION_AUTH=false:
|
|
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
|
|
# Required for session-authenticated public APIs; use an organization admin:
|
|
export DROBOTICFLOW_ACCEPTANCE_EMAIL=<admin-email>
|
|
export DROBOTICFLOW_ACCEPTANCE_PASSWORD=<admin-password>
|
|
API_BASE_URL=http://<node-ip>:30080 ./scripts/reconcile-system-data.sh
|
|
```
|
|
|
|
`scripts/reconcile-system-data.sh` is only the environment wrapper. The Python implementation reads the current builtin operator catalog from `/api/operators` and updates those live records to the digest-pinned image declared by `operator-repos/builtin/<slug>/release.json` for the target environment. Registry lookup is a validation step only: if the registry digest differs from the committed release manifest, reconcile fails instead of silently replacing the source of truth. The cleanup step skips builtin operators even when their image tag is `:latest`, so old custom/demo operators can be removed without tombstoning platform-provided builtins.
|
|
|
|
The source of truth for platform-provided builtin operators is `operator-repos/builtin/<slug>/operator.json` plus `operator-repos/builtin/<slug>/release.json`. `internal/api/builtin_operators_catalog.json` is a generated API seed artifact and must not be hand-edited as the only change. The API embeds the generated catalog and still validates required fields and duplicate IDs/names at startup. `scripts/generate_operator_docs.py` defaults to the local generated catalog and contacts a live API only when `--api-base-url` is provided, so adding or changing a builtin operator should update the operator repo directory, refresh the release manifest after image publication, and regenerate the catalog/manifests.
|
|
|
|
The Python operator-runner keeps the video quality and data-filter fallback thresholds aligned with that catalog. If an old workflow node is missing `MIN_CLARITY`, `MAX_SHAKE`, or `MAX_HAND_OCCLUSION`, runtime falls back to `0.8`, `0.1`, and `0.1`; it should not silently accept every video with `0.0/1.0/1.0`. If a workflow node explicitly sets any of those thresholds, the value must be finite, numeric, and in `[0, 1]`; invalid values fail the node instead of being silently replaced with defaults.
|
|
|
|
The same fail-fast rule applies to video processing timing and normalization parameters. Missing `CLARITY_SCALE`, `SEGMENT_DURATION_SECONDS`, or `FRAME_INTERVAL_SECONDS` use catalog defaults `35`, `300`, and `1.0`; explicit overrides must be finite positive numbers, and `SEGMENT_DURATION_SECONDS` must be an exact safe positive integer. Invalid overrides fail the node instead of being clamped or replaced with defaults.
|
|
|
|
## Kubernetes Verification
|
|
|
|
Check workloads:
|
|
|
|
```bash
|
|
kubectl -n droboticflow get pods
|
|
kubectl -n droboticflow get jobs
|
|
kubectl -n droboticflow get svc
|
|
```
|
|
|
|
Inspect logs:
|
|
|
|
```bash
|
|
kubectl -n droboticflow logs deploy/droboticflow-api
|
|
kubectl -n droboticflow logs deploy/droboticflow-worker
|
|
kubectl -n droboticflow logs job/<job-name>
|
|
```
|
|
|
|
Check the shared workspace:
|
|
|
|
```bash
|
|
kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200
|
|
```
|
|
|
|
## Common Failure Modes
|
|
|
|
### API starts but returns Mongo errors
|
|
|
|
- confirm `mongodb` pod is healthy
|
|
- confirm the app secret and database name match the Mongo init script
|
|
- inspect `kubectl logs deploy/mongodb`
|
|
|
|
### Worker cannot launch Jobs
|
|
|
|
- confirm the worker pod is running with service account `droboticflow-worker`
|
|
- confirm Role and RoleBinding exist in the namespace
|
|
- inspect worker logs for Kubernetes API errors
|
|
- confirm the operator-runner image pull secret name matches `K8S_IMAGE_PULL_SECRET_NAME`
|
|
- confirm the shared PVC exists and is bound
|
|
- confirm worker and API pods both mount the same workspace path
|
|
- for GPU nodes, confirm `GET /worker-api/capacity` returns matching schedulable nodes before submitting GPU workloads
|
|
|
|
### Job starts but downstream nodes cannot see outputs
|
|
|
|
- confirm the worker pod and operator-runner Job mount the same PVC and mount path
|
|
- confirm the node directory contains `input-artifacts.json`, `result.json`, `stage-manifest.json`, and `outputs/`
|
|
- inspect worker logs for `DROBOTICFLOW_RESULT_PATH` parse failures
|
|
|
|
### Image pull failure
|
|
|
|
- recreate `droboticflow-registry` secret with valid registry credentials
|
|
- confirm the image tags pushed to the registry match the deployment values
|
|
- confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest
|
|
|
|
### UI cannot load data
|
|
|
|
- confirm the API pod is healthy at `/api/health`
|
|
- confirm the NodePort is reachable from the target network
|
|
- inspect browser network traffic for failing endpoints
|
|
|
|
### Old demo data still appears after an upgrade
|
|
|
|
- run `K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080`
|
|
- include `DROBOTICFLOW_API_TOKEN` when token auth is enabled without required session auth
|
|
- include `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` for public session-auth deployments, using an organization admin account
|
|
- confirm `/api/users`, `/api/projects`, and `/api/templates` no longer return legacy seeded records
|
|
- confirm `/api/operators` only returns the current built-in operator catalog with the active operator-runner image tag
|