droboticflow-new/docs/production-readiness.md
2026-07-14 16:14:39 +08:00

25 KiB

Production Readiness Notes

This note tracks the production-facing safeguards that must stay true for Drobodata deployments.

Runtime Boundaries

  • Production execution uses the Go worker and Python operator-runner Jobs. The API does not execute workflow nodes in-process.
  • The API proxies /worker-api/* through WORKER_API_BASE_URL; the default is the in-cluster worker Service.
  • Worker proxy health, stats, metrics, and retry calls use short upstream timeouts. Manual process-once uses a long timeout because it can execute real queued work.
  • Explicit shell operators are disabled by default through OPERATOR_ALLOW_SHELL_COMMANDS=false; allowlist and denylist regexes are platform-controlled and cannot be overridden by workflow hyperparams.
  • Environment-specific values such as K8S_ENDPOINT, registry host/namespace, registry username, deploy image references, and workspace BOS bucket have no private repository defaults; provide them from local deploy secrets or CI secrets.
  • Manifest rendering, image build, and deployment wrappers reject invalid Kubernetes namespace/Secret/PVC/Service names, non-absolute workspace mount paths, malformed worker/AI/BOS service URLs, and malformed image references before emitting manifests, building images, or calling kubectl. Multi-line ConfigMap block values are indented before manifest output so formatted GPU tolerations JSON and multi-line shell command policy regexes remain valid YAML.
  • Local development and acceptance endpoints are resolved through shared helpers instead of per-script localhost literals. Vite dev proxy targets can be overridden with VITE_DEV_API_BASE_URL and VITE_DEV_WORKER_BASE_URL; browser-oriented acceptance scripts accept BASE_URL or API_BASE_URL, while reconcile/deploy wrappers keep API_BASE_URL as the primary operational variable. Explicit endpoint values must be absolute http:// or https:// URLs and fail before Vite, curl, Playwright, or urllib requests are attempted. Browser-side VITE_API_BASE_URL follows the same absolute http(s) rule, with the empty value reserved for same-origin API calls. Deployed workflow acceptance should set WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true so an omitted endpoint cannot fall back to localhost.
  • Data reconciliation uses real API calls and supports protected deployments through DROBOTICFLOW_API_TOKEN or session login credentials plus CSRF headers. Public session-auth environments must use an organization admin account for reconcile because it updates settings and management resources.
  • Session-auth management surfaces are admin-only on both read and write paths. /api/users, /api/settings, database browser routes, object-storage management routes, Mongo connection tests, and external business-DB token management require an organization admin; session-auth user creation and password updates require a non-blank password so new accounts are immediately login-capable and blank password updates cannot appear successful. User names cannot be blank, and user status accepts only active or disabled. Project mutations reject blank names and unknown statuses; project status accepts only active or archived so dashboard counters and project-list filters cannot drift. Workflow and template mutations reject blank names and unknown metadata enum values so UI filters, worker dispatch, and stored records stay aligned. Operator mutations reject blank metadata, unknown operator/port enum values, duplicate port IDs, and invalid resource defaults so the operator library cannot persist records the workflow editor or Kubernetes launcher cannot interpret. Reusable storage/Mongo connection list/get routes and workflow storage browse helpers require admin/operator role, not auditor access. /api/public/settings exposes only the runtime timezone so non-admin pages can render timestamps without receiving K8s, AI, Mongo, or token metadata. Users update their own sidebar profile through PUT /api/users/me, which accepts only name and email; user storage-path favorites are self-or-admin scoped. Model-backed AI generation routes require admin/operator role rather than auditor access.
  • Python live-acceptance and WJZH regression scripts use the shared DroboticflowClient helper for API token headers, session-login cookies, CSRF propagation, JSON calls, and binary object-storage calls. WJZH quality thresholds are caller-supplied through QUALITY_MIN_CLARITY, QUALITY_MAX_SHAKE, and QUALITY_MAX_HAND_OCCLUSION; window, slicing, and annotation timing are caller-supplied through the documented *_SECONDS values; sampling thresholds are caller-supplied through SAMPLING_RULES_JSON. The scripts do not carry built-in audit rules. New Python acceptance scripts should reuse that helper instead of carrying their own auth/request stack.
  • Python WJZH and X-platform acceptance scripts validate quality thresholds through the shared normalized-threshold gate. QUALITY_MIN_CLARITY, QUALITY_MAX_SHAKE, QUALITY_MAX_HAND_OCCLUSION, XPLATFORM_MIN_CLARITY, XPLATFORM_MAX_SHAKE, and XPLATFORM_MAX_HAND_OCCLUSION must be finite values in [0, 1]; invalid values fail before live workflows are created.
  • Python WJZH and X-platform acceptance scripts also share the same sampling-rule validator. SAMPLING_RULES_JSON and XPLATFORM_RULES_JSON must be non-empty rule arrays with unique id values, supported operators, required field/value fields, and finite numeric samplePercent values in (0, 100]; boolean samplePercent values are rejected before creating live workflows instead of being coerced to 1 or 0.
  • Python live-acceptance positive integer gates are exact and safe. WJZH timing values, X-platform XPLATFORM_SEGMENT_DURATION_SECONDS, every XPLATFORM_MIN_* count, and wait-time knobs reject signs, leading zeros, decimals, units, unsafe integers, and wait values below 60 seconds instead of coercing or clamping them.
  • Node/Playwright acceptance scripts use the shared createAcceptanceAPIClient helper for bearer token headers, Playwright context CSRF propagation, JSON calls, retries, exact integer and boolean env parsing, and optional session login. Functional and operator-preview checks also pass DROBOTICFLOW_API_TOKEN through browser contexts so protected UI requests exercise the same token-authenticated path as API setup calls.
  • Browser API calls use a shared URL builder and request-header helper. JSON calls, object-storage text previews, and multipart object uploads all send credentials: include; unsafe methods, including logout, attach the double-submit CSRF token, while FormData uploads avoid forcing a JSON Content-Type.
  • Object-storage text preview metadata accepts byte-count response headers only as exact safe non-negative integers. Corrupt values such as decimals, units, signs, leading-zero strings, or unsafe integers are ignored instead of being truncated into plausible sizes.
  • Data-asset aggregation treats local artifact.data.dataRef.payloadPath values as authoritative external payloads. If the referenced local payload is unreadable, malformed JSON, or not a JSON object, source collections and package previews omit it instead of falling back to stale inline data. Data-asset summary integers such as counts, byte sizes, and frame counts must be exact integer values; lossy floats, NaN, invalid json.Number, and overflowing unsigned values are ignored rather than truncated.
  • Incremental scans require scan-workflow DATA_SOURCE to be BOS, SCAN_MODE to be either FULL or INCREMENTAL, INCREMENTAL_UNIT to be either OBJECT or DIRECTORY, and INCREMENTAL_DIRECTORY_DEPTH to be a positive integer. The workflow editor, API, worker, and operator-runner all reject lossy or invalid values instead of truncating or silently falling back; invalid source-node hyperparams are rejected before workflow creation/update and fail the worker run as a normal node error if stale persisted data reaches execution.
  • Incremental source checkpoints are committed only from authoritative, readable checkpoint payloads. If source-checkpoint.json is unreadable, malformed, empty, missing sourceFingerprint, or contains lossy directoryDepth / listedCount / newCount values, worker checkpoint commit fails instead of falling back to stale inline artifact data or truncating counts.
  • Video quality and data-filter runtime fallbacks match the builtin operator catalog defaults: MIN_CLARITY=0.8, MAX_SHAKE=0.1, and MAX_HAND_OCCLUSION=0.1. Missing node hyperparams must not silently widen the filter into 0.0/1.0/1.0, and explicit threshold hyperparams must be finite numeric [0, 1] values or the node fails.
  • Video quality, slicing, and annotation runtime defaults match the builtin operator catalog for CLARITY_SCALE=35, SEGMENT_DURATION_SECONDS=300, and FRAME_INTERVAL_SECONDS=1.0. Explicit timing and scale overrides must be finite positive values; SEGMENT_DURATION_SECONDS is an exact safe positive integer and rejects signs, leading zeros, decimals, units, and unsafe integers, so bad workflow hyperparams fail instead of being silently clamped.
  • Builtin operator hyperparams fail fast when callers provide malformed structured values. EXPECTED_JSON, FIELD_MAP_JSON, and RULES_JSON must be valid JSON; field maps must be JSON objects with non-empty string source/destination fields and no output-field collisions; sampling rules must be JSON arrays whose entries include id, field, operator, value, and finite numeric (0, 100] samplePercent values; comparison operators must be supported; source DATA_SOURCE values must be INLINE, LOCAL, BOS, or UPSTREAM; boolean samplePercent values are not numeric percentages; boolean hyperparams such as REQUIRE_INTRINSICS accept only true/false, 1/0, yes/no, or on/off spellings.
  • Workflow schedule and execution-guard numeric fields are integer-gated in the editor before save. Schedule intervals must be positive while active, execution guard values such as concurrency, timeouts, retry count, and retry backoff must be non-negative, and lossy values like 60s, 1.5, or 2days are rejected instead of being coerced into defaults.
  • System settings worker numeric fields are positive-integer gated in the Settings UI, the shared settings payload builder, and PUT /api/settings. Worker poll interval and default max concurrency reject blank strings, zero, decimals, units, unsafe integers, and other lossy values instead of ignoring invalid explicit updates. Default CPU/MEM settings are also validated server-side as positive Kubernetes resource quantities, and default GPU must be a positive integer when present.
  • Database document browsing validates explicit limit and skip query values in both the UI and API before it reaches platform or business Mongo stores. limit must be a positive integer, skip must be a non-negative integer, and oversized values are capped to the documented bounds.
  • Explicit startup environment values fail fast when malformed. MONGO_URI is required for API/worker startup and must be a concrete mongodb:// or mongodb+srv:// URI without whitespace or unreplaced placeholder markers; there is no hidden localhost runtime fallback. API/worker ports must be valid TCP ports, Mongo retry count and delay must be positive integers, worker polling must be positive, boolean auth flags must use supported boolean spellings, and manifest/deploy preflight rejects invalid replicas, NodePort, and Job TTL values before Kubernetes dry-run or cluster writes. String startup values, including worker, AI, BOS, dist, token, proxy, GPU, and workspace settings, are trimmed before use; whitespace-only optional values are treated as unset rather than silently entering runtime config. AI generation has no hardcoded provider/model fallback: without AI_API_KEY it stays disabled, and with AI_API_KEY both AI_API_BASE_URL and AI_MODEL must be explicit. The API command passes the validated config into the HTTP handler, so worker proxy routing and the optional global BOS endpoint are bound at startup and embedded/test handler construction reports invalid config through a controlled error response instead of a process panic. API and worker use the validated BOS_ENDPOINT startup value only when a scan config or BOS storage connection omits endpoint; if both are empty, real BOS operations fail with an explicit missing-endpoint error rather than using a hardcoded region.

Kubernetes Jobs

  • Operator-runner Jobs receive CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities; malformed values fail before Job creation instead of being silently omitted.
  • A positive GPU value emits nvidia.com/gpu in both requests and limits.
  • GPU node selector, tolerations, and runtime class are only injected for Jobs that request GPU.
  • Finished Jobs receive ttlSecondsAfterFinished from OPERATOR_JOB_TTL_SECONDS, default 86400.
  • Use DRY_RUN=true npm run k8s:cleanup-jobs before deleting old finished Jobs that predate TTL.

Data Plane Limits

  • Object-storage JSON object creation is capped at 4 MiB.
  • Browser multipart uploads stream to BOS and are capped at 2 GiB.
  • Directory uploads keep browser webkitRelativePath values, prepare at most 5000 files/placeholders per batch, and upload four files concurrently through the same presigned direct/multipart path. Conflict policy is explicit (overwrite, skip, or stop with error), failed files can be retried independently, and explicitly listed empty directories are preserved as .keep objects because browser directory inputs do not expose truly empty directories.
  • Large inline object reads use preview or presigned URL behavior instead of loading full objects into API memory.
  • Browser video previews are lazy-loaded in monitoring, artifact preview, and object-storage browsing surfaces. The UI does not attach the <video> source until the user starts the preview, so large MP4 artifacts do not trigger Range requests and decoder work during initial page render or artifact switching.
  • Run logs, resource samples, and artifacts support strict limit and offset / skip pagination. Empty values use defaults, explicit limit must be positive, explicit offsets must be non-negative, and invalid values return validation errors instead of silently falling back to default pages. Logs and resources default to the latest 1000 records for large runs.
  • Data-asset aggregation uses artifact-focused run projections so list endpoints do not read complete logs and execution traces.

GPU Observability

  • Operator-runner resource samples include CPU seconds, process memory, GPU utilization, GPU memory used/total, GPU count, and GPU model when nvidia-smi is available.
  • The monitoring page displays GPU utilization, GPU memory, and GPU device information.
  • GET /worker-api/capacity exposes the worker's current Kubernetes Node allocatable GPU snapshot, including matched nodes, schedulable allocatable count, max single-node GPU count, and available. The Settings page displays this snapshot, workflow node GPU inputs surface a warning when the current cluster has no schedulable GPU, and the workflow run action blocks GPU-requesting workflows when the loaded worker capacity proves they cannot be scheduled.
  • npm run acceptance:gpu loads the deployment environment resolver first, so dev runs against POC by default when DEPLOY_ENV is unset. It remains the real acceptance check for allocatable GPU resources, scheduling, runtime, and nvidia-smi. It rejects invalid namespace, image, timeout, GPU count, selector, tolerations, resource-name, Job-name, and runtime-class inputs before it calls kubectl, verifies that at least one selector-matched and non-unschedulable node has the requested allocatable GPU count before creating a Job, and quotes dynamic manifest values before rendering the temporary Job. A cluster with no matching allocatable GPU resources is GPU-not-accepted, not a skipped success.
  • npm run acceptance:poc-gpu-capacity validates the deployed POC worker capacity endpoint from inside the cluster with the currently configured operator-runner image. The default accepted result is CPU-only capacity (gpu.available=false); set POC_GPU_CAPACITY_EXPECT_AVAILABLE=true after POC has a real GPU node pool.
  • npm run acceptance:poc-readiness includes deployed worker GPU capacity evidence through a default kubectl port-forward probe and verifies the platform gpuWorkflowPreflight gate through GET /api/acceptance/poc-readiness, so every POC readiness report states both whether the current hardware expectation is CPU-only or GPU-available and whether the latest real GPU-requesting workflow matches that expectation. The API check uses DROBOTICFLOW_API_TOKEN from the environment or droboticflow-auth-secret; set POC_PLATFORM_READINESS_ENABLED=false only for isolated script tests that intentionally bypass platform evidence.
  • npm run acceptance:poc-gpu-workflow verifies the worker GPU path through a real POC workflow run. It uses a session-authenticated acceptance user when provided; otherwise it uses DROBOTICFLOW_API_TOKEN from the environment or droboticflow-auth-secret only for the bounded POC GPU route plus run reads, then triggers the deployed worker through a local port-forward. It expects gpu request cannot be scheduled while POC is CPU-only; with POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true or POC_GPU_CAPACITY_EXPECT_AVAILABLE=true, it expects a completed run with artifacts. The latest proof is exposed in POC readiness as gpuWorkflowPreflight.

Acceptance

Run local verification before merging:

npm run verify:local
npm run verify:security

The local gate includes shell syntax checks for operational scripts, so GPU acceptance, deployment, cleanup, and reconciliation wrappers are parsed before a change is accepted. The security gate runs npm audit --audit-level=moderate through the retry wrapper; npm run verify runs both gates in order.

Run deployed live workflow acceptance after rollout:

WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true \
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh http://<node-ip>:30080

The workflow acceptance source path may be local/PVC-backed or bos://bucket/prefix. The source node now fails fast when no real input source is configured, so a misconfigured workflow cannot pass by producing an empty source manifest. WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true keeps deployed acceptance from silently using local defaults.

Run browser object-storage acceptance after rollout:

BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage

OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE is required and must point to a non-empty real UTF-8 text or JSON payload. The script requires an explicit storage connection selector, uses the supplied payload for create/upload/preview checks, and will not modify connection metadata or fall back to a built-in object body to make the check pass. Set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when the file extension is ambiguous. Set OBJECT_STORAGE_VIDEO_KEY to validate browser playback for a known existing video object.

Run operator preview acceptance against supplied payloads and real source paths:

BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview

The operator preview script has no built-in CSV/text/JSON/ROSBag/LeRobot dataset fallback. Payload files are caller supplied; scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs. Set optional PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON, PREVIEW_CHECK_SCAN_MIN_FILES, PREVIEW_CHECK_ROSBAG_TOPIC, and PREVIEW_CHECK_LEROBOT_ROBOT_TYPE when the target dataset needs stricter assertions. PREVIEW_CHECK_JSON_FIELD_MAP_JSON uses the same strict field-map rules as the runtime operator, and PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.

Run X-platform operator-chain acceptance against a real dataset:

BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform

The script has no synthetic dataset fallback and no built-in acceptance criteria. For bos:// input or upload paths, provide an existing storage connection selector. Set XPLATFORM_RULES_JSON, every XPLATFORM_MIN_* count, quality thresholds, and slice/annotation timing explicitly to match the target dataset before treating the run as production acceptance evidence. The X-platform quality thresholds must be finite normalized [0, 1] values, XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS must be finite positive numbers, and XPLATFORM_SEGMENT_DURATION_SECONDS plus every XPLATFORM_MIN_* count must be exact safe positive integers. Python acceptance and reconciliation scripts use the shared protected API client for token/session auth, CSRF propagation, JSON requests, and optional transient-error retries.

Reconcile persistent system data after rollout:

K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080

The reconcile wrapper delegates to scripts/reconcile_system_data.py. It removes legacy demo records, updates cluster settings through the real API, and derives builtin operators from the live /api/operators catalog before refreshing each builtin to the digest-pinned image declared in operator-repos/builtin/<slug>/release.json for the target environment. Registry digest lookup is used only as a guardrail: a mismatch between the committed release manifest and the live registry digest blocks reconciliation instead of becoming an implicit production truth.

Builtin operator source lives in operator-repos/builtin/<slug>/operator.json and the paired release.json. internal/api/builtin_operators_catalog.json is generated from those source and release files before being embedded by the API. Startup validation rejects malformed or duplicate generated catalog entries before they can drift into the database, and documentation generation defaults to the local generated catalog; live API export requires an explicit --api-base-url.

Run GPU acceptance on a real GPU cluster:

npm run acceptance:gpu

Run the POC workflow-level GPU preflight check with a real acceptance user:

DROBOTICFLOW_ACCEPTANCE_EMAIL=<operator@example.com> \
DROBOTICFLOW_ACCEPTANCE_PASSWORD=<password> \
npm run acceptance:poc-gpu-workflow

Live workflow acceptance, GPU acceptance, object-storage acceptance, operator preview acceptance, and X-platform real-data acceptance can run against a deployed environment from a local shell or external CI. Provide WORKFLOW_ACCEPTANCE_SOURCE_PATH for workflow acceptance, configure a kubeconfig when GPU acceptance targets a non-default environment, and provide explicit live data paths plus storage connection selectors when storage-backed acceptance is enabled. For object-storage acceptance, provide OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE as a non-empty real UTF-8 payload file before invoking the browser check. For operator preview acceptance, provide PREVIEW_CHECK_CSV_PAYLOAD_FILE, PREVIEW_CHECK_TEXT_PAYLOAD_FILE, PREVIEW_CHECK_JSON_PAYLOAD_FILE, the JSON field settings, and the real scan/ROSBag/LeRobot source paths. For X-platform acceptance, provide the real source path plus XPLATFORM_RULES_JSON, all XPLATFORM_MIN_* counts, quality thresholds, and slice/annotation timing through environment variables or external CI secrets.

Remaining Cluster Responsibilities

Drobodata now exposes the necessary application controls, but production readiness still depends on the target cluster:

  • TLS ingress or trusted private network exposure
  • network policies around MongoDB, API, worker, and operator-runner Jobs
  • MongoDB backup, restore, and credential rotation
  • registry credential rotation
  • Prometheus scraping and alerting for worker metrics and failed Jobs