droboticflow/docs/operations.md
2026-06-18 01:39:27 +08:00

21 KiB

Operations Guide

Service Responsibilities

API

  • validates incoming payloads
  • persists projects, workflows, operators, templates, settings, runs, and audit tasks
  • serves the built SPA in production
  • runs as a single Go binary

Worker

  • polls MongoDB for queued runs
  • auto-enqueues streaming workflows based on workflow-level schedule modes:
    • schedule.interval
    • schedule.intervalHours
    • schedule.dayInterval + schedule.timeOfDay
  • executes workflow nodes in topological order
  • pauses runs on human-review nodes
  • resumes approved or rejected audit branches
  • creates Kubernetes Jobs for every operator execution
  • writes node input manifests into the shared workspace PVC before launching Jobs
  • runs as a single Go binary

Operator Runner

  • receives RUN_ID, WORKFLOW_ID, NODE_ID, and NODE_LABEL
  • executes a generic Python operator contract
  • reads input manifests from the shared workspace PVC
  • writes result.json and stage-manifest.json back to the shared workspace PVC
  • preserves catalog defaults only when optional hyperparams are omitted; explicit malformed JSON, boolean, comparison-operator, sampling-rule, or exact-positive-integer hyperparams fail the node instead of falling back to defaults

MongoDB Collections

  • projects
  • operators
  • workflows
  • templates
  • runs
  • auditTasks
  • users
  • settings

API and worker startup both call EnsureIndexes on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups.

Incremental source-state checkpoints are treated as authoritative runtime state, not best-effort metadata. Scan-workflow DATA_SOURCE accepts only BOS, SCAN_MODE accepts only FULL or INCREMENTAL when explicitly configured, and INCREMENTAL_UNIT accepts only OBJECT or DIRECTORY. The shared source runner accepts explicit DATA_SOURCE only as INLINE, LOCAL, BOS, or UPSTREAM; unknown values fail instead of producing an empty manifest. SOURCE_STATE_JSON must be a JSON object whose seenFingerprints entries are strings, incrementalUnit is OBJECT or DIRECTORY, and directoryDepth is a positive integer. When a completed run exposes source-checkpoint.json, the worker reads that file as the source of truth; unreadable, malformed, empty, missing-fingerprint, or lossy-count checkpoints fail checkpoint commit instead of falling back to stale inline artifact data.

Data-asset APIs apply the same stale-data rule for local external payloads. If artifact.data.dataRef.payloadPath points to a local file, that payload must be readable JSON object data; unreadable, malformed, or non-object payloads are skipped instead of using older inline artifact.data. Collection/package integer summaries only accept exact integer values, so lossy numeric fields are ignored rather than truncated.

Run Lifecycle

  1. frontend calls POST /api/workflows/:id/run
  2. API inserts a queued run document
  3. worker claims the run and marks it running
  4. worker executes start nodes and records logs, resource samples, and artifacts
  5. if a node requires review, the run moves to waiting_for_audit
  6. audit action updates the task to approved or rejected
  7. worker resumes the graph from the appropriate edge
  8. run ends in completed or failed

Local Verification

Run local verification and the security audit as separate gates:

npm run verify:local
npm run verify:security

verify:local includes TypeScript/Go/script linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, and Kubernetes manifest dry-run validation. verify:security runs npm audit --audit-level=moderate through a retry wrapper so transient registry failures do not prevent local validation from running first. npm run verify runs both in that order.

Runtime Metrics

The worker exposes Prometheus text metrics at /worker-api/metrics. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the most recent dispatch queue depth. Kubernetes deployments annotate the worker Service for Prometheus scraping at this path.

Runs that execute on NVIDIA hardware also persist GPU utilization, GPU memory used/total in MiB, GPU count, and GPU model into run.resourceSeries. The monitoring page shows these fields in the resource modal. A missing GPU memory value means the operator-runner could not execute nvidia-smi, not that the workflow requested no GPU.

Job Retention

New operator-runner Jobs use OPERATOR_JOB_TTL_SECONDS as ttlSecondsAfterFinished, defaulting to one day. For clusters upgraded from older manifests, clean pre-existing finished Jobs with:

DRY_RUN=true npm run k8s:cleanup-jobs
npm run k8s:cleanup-jobs

Set OPERATOR_JOB_CLEANUP_MIN_AGE_SECONDS and OPERATOR_JOB_CLEANUP_STATUSES=succeeded,failed to adjust the retention window and terminal states.

Account Security

The API supports HttpOnly cookie-session auth through /api/auth/* and super-admin organization/user management under /api/admin/*. Existing deployments keep their current behavior unless DROBOTICFLOW_REQUIRE_SESSION_AUTH=true is set. Before enabling it, bootstrap the first super admin and verify login through /api/auth/login.

Shell Command Policy

Explicit COMMAND / SCRIPT execution is disabled by default. Set OPERATOR_ALLOW_SHELL_COMMANDS=true only for deployments that need shell operators, and constrain commands with OPERATOR_COMMAND_ALLOWLIST_REGEX and OPERATOR_COMMAND_DENYLIST_REGEX. These are platform-controlled variables passed from the worker to operator-runner Jobs, and workflow hyperparams cannot override them.

Acceptance Checks

Run acceptance checks after a deployment from a local shell or external CI. API smoke requires SMOKE_SOURCE_PATH, a real path visible to operator-runner Jobs. GPU smoke requires a kubeconfig with permission to create a temporary Job in K8S_NAMESPACE. The GPU smoke wrapper validates namespace, image, timeout, GPU count, selector, tolerations JSON, resource name, Job name, and runtime class before any kubectl call; invalid acceptance inputs fail locally instead of creating a malformed Job.

Generated Artifacts

Build output, local Playwright screenshots/videos, pytest caches, and zip bundles should stay out of git. Regenerate validation artifacts locally or publish them as CI/release artifacts when they need to be shared.

Start the stack locally:

npm run dev:full

Smoke test the API and worker:

SMOKE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/smoke-test.sh

SMOKE_SOURCE_PATH may be a filesystem/PVC path or a bos://bucket/prefix path. The 真实数据源 runner infers LOCAL or BOS from the path and rejects a run that has no SOURCE_TEXT, SOURCE_JSON, SOURCE_PATH, configured BOS source, local STORAGE_PREFIX, or upstream artifact input. This prevents an unconfigured source node from completing with an empty manifest.

For deployed smoke, pass the deployed URL or set BASE_URL / API_BASE_URL, and set SMOKE_REQUIRE_EXPLICIT_ENDPOINT=true so localhost defaults are disabled.

Run the production-style WJZH real-data regression:

API_BASE_URL=http://<node-ip>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

The WJZH regression script does not provide built-in quality thresholds, video timing, or sampling thresholds; QUALITY_MIN_CLARITY, QUALITY_MAX_SHAKE, QUALITY_MAX_HAND_OCCLUSION, the *_SECONDS timing values, and SAMPLING_RULES_JSON must match the target dataset before the run is accepted as evidence. Quality thresholds are normalized acceptance gates and must be in [0, 1]; integer timing values must be exact safe positive integers; out-of-range values fail preflight instead of weakening the run evidence. See the full runbook in docs/wjzh-operator-regression.md.

Run browser object-storage acceptance against a live storage connection:

BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage

OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE must point to a non-empty UTF-8 text or JSON file that represents real acceptance content. The script uses that same payload for browser create, preview, rename, upload, and delete checks; it no longer has a built-in object body fallback. Set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when the file extension is ambiguous, and set OBJECT_STORAGE_VIDEO_KEY=<existing-video-key> when video playback should be included. PLAYWRIGHT_IGNORE_HTTPS_ERRORS accepts only true/false, 1/0, yes/no, or on/off when explicitly set. The script creates temporary objects under an object-storage-acceptance-* prefix and removes them at the end. For session-authenticated public APIs, provide an organization admin through DROBOTICFLOW_SMOKE_EMAIL and DROBOTICFLOW_SMOKE_PASSWORD; for token-protected non-session APIs, provide DROBOTICFLOW_API_TOKEN.

Node/Playwright acceptance scripts share scripts/lib/acceptance-client.mjs for API token headers, Playwright session-cookie CSRF propagation, JSON requests, retries, and optional session login. Keep new browser acceptance scripts on that helper instead of carrying local request wrappers. scripts/playwright-functional-check.mjs and scripts/playwright-preview-operators-check.mjs also forward DROBOTICFLOW_API_TOKEN into browser contexts so UI fetches and preflight API setup use the same token-authenticated path.

Run operator preview acceptance with explicit real payloads and source paths:

BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview

The CSV, text, and JSON payloads must be supplied by the caller; the script materializes those payloads inside operator-runner jobs and no longer creates built-in payload data. PREVIEW_CHECK_JSON_PAYLOAD_FILE must contain a non-empty object array or an object with items[]; the extract/filter/map variables define how the JSON data-operator chain is validated. PREVIEW_CHECK_JSON_FIELD_MAP_JSON must use non-empty string source/destination fields and must not map multiple input fields to the same output field. The scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs, not just from the machine launching the script. Optional checks: PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON, PREVIEW_CHECK_SCAN_MIN_FILES, PREVIEW_CHECK_ROSBAG_TOPIC, and PREVIEW_CHECK_LEROBOT_ROBOT_TYPE. PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.

Run X-platform operator-chain acceptance against an explicit real dataset:

BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=/workspace/datasets/xplatform/<collection-root> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='[{"id":"...", "field":"quality.clarity", "operator":"gte", "value":0.3, "samplePercent":100}]' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only

For BOS-backed datasets, provide a storage connection selector and use a full bos://bucket/prefix source path:

BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only

The X-platform acceptance script does not generate synthetic videos, fallback data, sampling rules, quality thresholds, slice/annotation parameters, or minimum-count assertions. It creates a temporary project/workflow that runs BOS 增量扫描 when the source is BOS, then X 平台数据集索引, collection validation, video quality, filtering, slicing, annotation, packaging, sampling, and result upload. XPLATFORM_RULES_JSON, XPLATFORM_MIN_*, XPLATFORM_MIN_CLARITY, XPLATFORM_MAX_SHAKE, XPLATFORM_MAX_HAND_OCCLUSION, XPLATFORM_CLARITY_SCALE, XPLATFORM_SEGMENT_DURATION_SECONDS, and XPLATFORM_FRAME_INTERVAL_SECONDS must match the target real dataset before the run is accepted as evidence. X-platform quality thresholds are finite normalized [0, 1] gates, matching the runtime clarity, shake, and handOcclusion metrics; out-of-range or non-finite values fail before any live workflow is created. XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS must be finite positive numbers, while XPLATFORM_SEGMENT_DURATION_SECONDS and every XPLATFORM_MIN_* count must be exact safe positive integers to match operator-runner runtime validation and acceptance assertions. XPLATFORM_RULES_JSON is validated with the same rule-shape gate as the WJZH regression: every rule needs id, field, operator, value, and samplePercent, ids must be unique, operators must be supported, and samplePercent must be finite numeric data in (0, 100]; booleans are rejected instead of being coerced. Optional filters such as XPLATFORM_DATE_START, XPLATFORM_DATE_END, XPLATFORM_PLACE, and XPLATFORM_TYPE further narrow the target dataset. For protected APIs, use DROBOTICFLOW_API_TOKEN or DROBOTICFLOW_SMOKE_EMAIL / DROBOTICFLOW_SMOKE_PASSWORD.

Python live-acceptance and reconciliation scripts share scripts/lib/droboticflow_acceptance.py for API token headers, session login cookies, CSRF headers, JSON requests, optional transient-error retries, and binary object-storage requests. Keep new Python acceptance, regression, or operational API scripts on that shared client so protected deployments are tested through the same real auth path as scripts/xplatform-live-acceptance.py, scripts/wjzh_operator_regression.py, and scripts/reconcile_system_data.py.

Reconcile persistent system data and remove legacy demo records:

export K8S_ENDPOINT=https://<api-server>:6443
# Optional for token-protected APIs when DROBOTICFLOW_REQUIRE_SESSION_AUTH=false:
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
# Required for session-authenticated public APIs; use an organization admin:
export DROBOTICFLOW_SMOKE_EMAIL=<admin-email>
export DROBOTICFLOW_SMOKE_PASSWORD=<admin-password>
API_BASE_URL=http://<node-ip>:30080 ./scripts/reconcile-system-data.sh

scripts/reconcile-system-data.sh is only the environment wrapper. The Python implementation reads the current builtin operator catalog from /api/operators and updates those live records to the requested OPERATOR_RUNNER_IMAGE; it does not carry a second static builtin-operator list. The cleanup step skips builtin operators even when their image tag is :latest, so old custom/demo operators can be removed without tombstoning platform-provided builtins.

The source catalog for platform-provided builtin operators is internal/api/builtin_operators_catalog.json. The API embeds this file, validates required fields and duplicate IDs/names at startup, and injects only runtime deployment fields such as the current operator-runner image and timestamps. scripts/generate_operator_docs.py defaults to the same local file and contacts a live API only when --api-base-url is provided, so adding or changing a builtin operator should update the catalog first instead of editing separate Go and Python maps.

The Python operator-runner keeps the video quality and data-filter fallback thresholds aligned with that catalog. If an old workflow node is missing MIN_CLARITY, MAX_SHAKE, or MAX_HAND_OCCLUSION, runtime falls back to 0.8, 0.1, and 0.1; it should not silently accept every video with 0.0/1.0/1.0. If a workflow node explicitly sets any of those thresholds, the value must be finite, numeric, and in [0, 1]; invalid values fail the node instead of being silently replaced with defaults.

The same fail-fast rule applies to video processing timing and normalization parameters. Missing CLARITY_SCALE, SEGMENT_DURATION_SECONDS, or FRAME_INTERVAL_SECONDS use catalog defaults 35, 300, and 1.0; explicit overrides must be finite positive numbers, and SEGMENT_DURATION_SECONDS must be an exact safe positive integer. Invalid overrides fail the node instead of being clamped or replaced with defaults.

Kubernetes Verification

Check workloads:

kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc

Inspect logs:

kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>

Check the shared workspace:

kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200

Common Failure Modes

API starts but returns Mongo errors

  • confirm mongodb pod is healthy
  • confirm the app secret and database name match the Mongo init script
  • inspect kubectl logs deploy/mongodb

Worker cannot launch Jobs

  • confirm the worker pod is running with service account droboticflow-worker
  • confirm Role and RoleBinding exist in the namespace
  • inspect worker logs for Kubernetes API errors
  • confirm the operator-runner image pull secret name matches K8S_IMAGE_PULL_SECRET_NAME
  • confirm the shared PVC exists and is bound
  • confirm worker and API pods both mount the same workspace path

Job starts but downstream nodes cannot see outputs

  • confirm the worker pod and operator-runner Job mount the same PVC and mount path
  • confirm the node directory contains input-artifacts.json, result.json, stage-manifest.json, and outputs/
  • inspect worker logs for DROBOTICFLOW_RESULT_PATH parse failures

Image pull failure

  • recreate droboticflow-registry secret with valid registry credentials
  • confirm the image tags pushed to the registry match the deployment values
  • confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest

UI cannot load data

  • confirm the API pod is healthy at /api/health
  • confirm the NodePort is reachable from the target network
  • inspect browser network traffic for failing endpoints

Old demo data still appears after an upgrade

  • run K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080
  • include DROBOTICFLOW_API_TOKEN when token auth is enabled without required session auth
  • include DROBOTICFLOW_SMOKE_EMAIL and DROBOTICFLOW_SMOKE_PASSWORD for public session-auth deployments, using an organization admin account
  • confirm /api/users, /api/projects, and /api/templates no longer return legacy seeded records
  • confirm /api/operators only returns the current built-in operator catalog with the active operator-runner image tag