25 KiB
Production Readiness Notes
This note tracks the production-facing safeguards that must stay true for Drobodata deployments.
Runtime Boundaries
- Production execution uses the Go worker and Python operator-runner Jobs. The API does not execute workflow nodes in-process.
- The API proxies
/worker-api/*throughWORKER_API_BASE_URL; the default is the in-cluster worker Service. - Worker proxy health, stats, metrics, and retry calls use short upstream timeouts. Manual
process-onceuses a long timeout because it can execute real queued work. - Explicit shell operators are disabled by default through
OPERATOR_ALLOW_SHELL_COMMANDS=false; allowlist and denylist regexes are platform-controlled and cannot be overridden by workflow hyperparams. - Environment-specific values such as
K8S_ENDPOINT, registry host/namespace, registry username, deploy image references, and workspace BOS bucket have no private repository defaults; provide them from local deploy secrets or CI secrets. - Manifest rendering, image build, and deployment wrappers reject invalid Kubernetes namespace/Secret/PVC/Service names, non-absolute workspace mount paths, malformed worker/AI/BOS service URLs, and malformed image references before emitting manifests, building images, or calling
kubectl. Multi-line ConfigMap block values are indented before manifest output so formatted GPU tolerations JSON and multi-line shell command policy regexes remain valid YAML. - Local development and acceptance endpoints are resolved through shared helpers instead of per-script localhost literals. Vite dev proxy targets can be overridden with
VITE_DEV_API_BASE_URLandVITE_DEV_WORKER_BASE_URL; browser-oriented acceptance scripts acceptBASE_URLorAPI_BASE_URL, while reconcile/deploy wrappers keepAPI_BASE_URLas the primary operational variable. Explicit endpoint values must be absolutehttp://orhttps://URLs and fail before Vite, curl, Playwright, or urllib requests are attempted. Browser-sideVITE_API_BASE_URLfollows the same absolute http(s) rule, with the empty value reserved for same-origin API calls. Deployed workflow acceptance should setWORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=trueso an omitted endpoint cannot fall back to localhost. - Data reconciliation uses real API calls and supports protected deployments through
DROBOTICFLOW_API_TOKENor session login credentials plus CSRF headers. Public session-auth environments must use an organization admin account for reconcile because it updates settings and management resources. - Session-auth management surfaces are admin-only on both read and write paths.
/api/users,/api/settings, database browser routes, object-storage management routes, Mongo connection tests, and external business-DB token management require an organization admin; session-auth user creation and password updates require a non-blank password so new accounts are immediately login-capable and blank password updates cannot appear successful. User names cannot be blank, and user status accepts onlyactiveordisabled. Project mutations reject blank names and unknown statuses; project status accepts onlyactiveorarchivedso dashboard counters and project-list filters cannot drift. Workflow and template mutations reject blank names and unknown metadata enum values so UI filters, worker dispatch, and stored records stay aligned. Operator mutations reject blank metadata, unknown operator/port enum values, duplicate port IDs, and invalid resource defaults so the operator library cannot persist records the workflow editor or Kubernetes launcher cannot interpret. Reusable storage/Mongo connection list/get routes and workflow storage browse helpers require admin/operator role, not auditor access./api/public/settingsexposes only the runtime timezone so non-admin pages can render timestamps without receiving K8s, AI, Mongo, or token metadata. Users update their own sidebar profile throughPUT /api/users/me, which accepts onlynameandemail; user storage-path favorites are self-or-admin scoped. Model-backed AI generation routes require admin/operator role rather than auditor access. - Python live-acceptance and WJZH regression scripts use the shared
DroboticflowClienthelper for API token headers, session-login cookies, CSRF propagation, JSON calls, and binary object-storage calls. WJZH quality thresholds are caller-supplied throughQUALITY_MIN_CLARITY,QUALITY_MAX_SHAKE, andQUALITY_MAX_HAND_OCCLUSION; window, slicing, and annotation timing are caller-supplied through the documented*_SECONDSvalues; sampling thresholds are caller-supplied throughSAMPLING_RULES_JSON. The scripts do not carry built-in audit rules. New Python acceptance scripts should reuse that helper instead of carrying their own auth/request stack. - Python WJZH and X-platform acceptance scripts validate quality thresholds through the shared normalized-threshold gate.
QUALITY_MIN_CLARITY,QUALITY_MAX_SHAKE,QUALITY_MAX_HAND_OCCLUSION,XPLATFORM_MIN_CLARITY,XPLATFORM_MAX_SHAKE, andXPLATFORM_MAX_HAND_OCCLUSIONmust be finite values in[0, 1]; invalid values fail before live workflows are created. - Python WJZH and X-platform acceptance scripts also share the same sampling-rule validator.
SAMPLING_RULES_JSONandXPLATFORM_RULES_JSONmust be non-empty rule arrays with uniqueidvalues, supported operators, requiredfield/valuefields, and finite numericsamplePercentvalues in(0, 100]; booleansamplePercentvalues are rejected before creating live workflows instead of being coerced to1or0. - Python live-acceptance positive integer gates are exact and safe. WJZH timing values, X-platform
XPLATFORM_SEGMENT_DURATION_SECONDS, everyXPLATFORM_MIN_*count, and wait-time knobs reject signs, leading zeros, decimals, units, unsafe integers, and wait values below 60 seconds instead of coercing or clamping them. - Node/Playwright acceptance scripts use the shared
createAcceptanceAPIClienthelper for bearer token headers, Playwright context CSRF propagation, JSON calls, retries, exact integer and boolean env parsing, and optional session login. Functional and operator-preview checks also passDROBOTICFLOW_API_TOKENthrough browser contexts so protected UI requests exercise the same token-authenticated path as API setup calls. - Browser API calls use a shared URL builder and request-header helper. JSON calls, object-storage text previews, and multipart object uploads all send
credentials: include; unsafe methods, including logout, attach the double-submit CSRF token, while FormData uploads avoid forcing a JSONContent-Type. - Object-storage text preview metadata accepts byte-count response headers only as exact safe non-negative integers. Corrupt values such as decimals, units, signs, leading-zero strings, or unsafe integers are ignored instead of being truncated into plausible sizes.
- Data-asset aggregation treats local
artifact.data.dataRef.payloadPathvalues as authoritative external payloads. If the referenced local payload is unreadable, malformed JSON, or not a JSON object, source collections and package previews omit it instead of falling back to stale inline data. Data-asset summary integers such as counts, byte sizes, and frame counts must be exact integer values; lossy floats,NaN, invalidjson.Number, and overflowing unsigned values are ignored rather than truncated. - Incremental scans require scan-workflow
DATA_SOURCEto beBOS,SCAN_MODEto be eitherFULLorINCREMENTAL,INCREMENTAL_UNITto be eitherOBJECTorDIRECTORY, andINCREMENTAL_DIRECTORY_DEPTHto be a positive integer. The workflow editor, API, worker, and operator-runner all reject lossy or invalid values instead of truncating or silently falling back; invalid source-node hyperparams are rejected before workflow creation/update and fail the worker run as a normal node error if stale persisted data reaches execution. - Incremental source checkpoints are committed only from authoritative, readable checkpoint payloads. If
source-checkpoint.jsonis unreadable, malformed, empty, missingsourceFingerprint, or contains lossydirectoryDepth/listedCount/newCountvalues, worker checkpoint commit fails instead of falling back to stale inline artifact data or truncating counts. - Video quality and data-filter runtime fallbacks match the builtin operator catalog defaults:
MIN_CLARITY=0.8,MAX_SHAKE=0.1, andMAX_HAND_OCCLUSION=0.1. Missing node hyperparams must not silently widen the filter into0.0/1.0/1.0, and explicit threshold hyperparams must be finite numeric[0, 1]values or the node fails. - Video quality, slicing, and annotation runtime defaults match the builtin operator catalog for
CLARITY_SCALE=35,SEGMENT_DURATION_SECONDS=300, andFRAME_INTERVAL_SECONDS=1.0. Explicit timing and scale overrides must be finite positive values;SEGMENT_DURATION_SECONDSis an exact safe positive integer and rejects signs, leading zeros, decimals, units, and unsafe integers, so bad workflow hyperparams fail instead of being silently clamped. - Builtin operator hyperparams fail fast when callers provide malformed structured values.
EXPECTED_JSON,FIELD_MAP_JSON, andRULES_JSONmust be valid JSON; field maps must be JSON objects with non-empty string source/destination fields and no output-field collisions; sampling rules must be JSON arrays whose entries includeid,field,operator,value, and finite numeric(0, 100]samplePercentvalues; comparison operators must be supported; sourceDATA_SOURCEvalues must beINLINE,LOCAL,BOS, orUPSTREAM; booleansamplePercentvalues are not numeric percentages; boolean hyperparams such asREQUIRE_INTRINSICSaccept onlytrue/false,1/0,yes/no, oron/offspellings. - Workflow schedule and execution-guard numeric fields are integer-gated in the editor before save. Schedule intervals must be positive while active, execution guard values such as concurrency, timeouts, retry count, and retry backoff must be non-negative, and lossy values like
60s,1.5, or2daysare rejected instead of being coerced into defaults. - System settings worker numeric fields are positive-integer gated in the Settings UI, the shared settings payload builder, and
PUT /api/settings. Worker poll interval and default max concurrency reject blank strings, zero, decimals, units, unsafe integers, and other lossy values instead of ignoring invalid explicit updates. Default CPU/MEM settings are also validated server-side as positive Kubernetes resource quantities, and default GPU must be a positive integer when present. - Database document browsing validates explicit
limitandskipquery values in both the UI and API before it reaches platform or business Mongo stores.limitmust be a positive integer,skipmust be a non-negative integer, and oversized values are capped to the documented bounds. - Explicit startup environment values fail fast when malformed.
MONGO_URIis required for API/worker startup and must be a concretemongodb://ormongodb+srv://URI without whitespace or unreplaced placeholder markers; there is no hidden localhost runtime fallback. API/worker ports must be valid TCP ports, Mongo retry count and delay must be positive integers, worker polling must be positive, boolean auth flags must use supported boolean spellings, and manifest/deploy preflight rejects invalid replicas, NodePort, and Job TTL values before Kubernetes dry-run or cluster writes. String startup values, including worker, AI, BOS, dist, token, proxy, GPU, and workspace settings, are trimmed before use; whitespace-only optional values are treated as unset rather than silently entering runtime config. AI generation has no hardcoded provider/model fallback: withoutAI_API_KEYit stays disabled, and withAI_API_KEYbothAI_API_BASE_URLandAI_MODELmust be explicit. The API command passes the validated config into the HTTP handler, so worker proxy routing and the optional global BOS endpoint are bound at startup and embedded/test handler construction reports invalid config through a controlled error response instead of a process panic. API and worker use the validatedBOS_ENDPOINTstartup value only when a scan config or BOS storage connection omits endpoint; if both are empty, real BOS operations fail with an explicit missing-endpoint error rather than using a hardcoded region.
Kubernetes Jobs
- Operator-runner Jobs receive CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities; malformed values fail before Job creation instead of being silently omitted.
- A positive GPU value emits
nvidia.com/gpuin both requests and limits. - GPU node selector, tolerations, and runtime class are only injected for Jobs that request GPU.
- Finished Jobs receive
ttlSecondsAfterFinishedfromOPERATOR_JOB_TTL_SECONDS, default86400. - Use
DRY_RUN=true npm run k8s:cleanup-jobsbefore deleting old finished Jobs that predate TTL.
Data Plane Limits
- Object-storage JSON object creation is capped at 4 MiB.
- Browser multipart uploads stream to BOS and are capped at 2 GiB.
- Directory uploads keep browser
webkitRelativePathvalues, prepare at most 5000 files/placeholders per batch, and upload four files concurrently through the same presigned direct/multipart path. Conflict policy is explicit (overwrite,skip, or stop witherror), failed files can be retried independently, and explicitly listed empty directories are preserved as.keepobjects because browser directory inputs do not expose truly empty directories. - Large inline object reads use preview or presigned URL behavior instead of loading full objects into API memory.
- Browser video previews are lazy-loaded in monitoring, artifact preview, and object-storage browsing surfaces. The UI does not attach the
<video>source until the user starts the preview, so large MP4 artifacts do not trigger Range requests and decoder work during initial page render or artifact switching. - Run logs, resource samples, and artifacts support strict
limitandoffset/skippagination. Empty values use defaults, explicitlimitmust be positive, explicit offsets must be non-negative, and invalid values return validation errors instead of silently falling back to default pages. Logs and resources default to the latest 1000 records for large runs. - Data-asset aggregation uses artifact-focused run projections so list endpoints do not read complete logs and execution traces.
GPU Observability
- Operator-runner resource samples include CPU seconds, process memory, GPU utilization, GPU memory used/total, GPU count, and GPU model when
nvidia-smiis available. - The monitoring page displays GPU utilization, GPU memory, and GPU device information.
GET /worker-api/capacityexposes the worker's current Kubernetes Node allocatable GPU snapshot, including matched nodes, schedulable allocatable count, max single-node GPU count, andavailable. The Settings page displays this snapshot, workflow node GPU inputs surface a warning when the current cluster has no schedulable GPU, and the workflow run action blocks GPU-requesting workflows when the loaded worker capacity proves they cannot be scheduled.npm run acceptance:gpuloads the deployment environment resolver first, sodevruns against POC by default whenDEPLOY_ENVis unset. It remains the real acceptance check for allocatable GPU resources, scheduling, runtime, andnvidia-smi. It rejects invalid namespace, image, timeout, GPU count, selector, tolerations, resource-name, Job-name, and runtime-class inputs before it callskubectl, verifies that at least one selector-matched and non-unschedulable node has the requested allocatable GPU count before creating a Job, and quotes dynamic manifest values before rendering the temporary Job. A cluster with no matching allocatable GPU resources is GPU-not-accepted, not a skipped success.npm run acceptance:poc-gpu-capacityvalidates the deployed POC worker capacity endpoint from inside the cluster with the currently configured operator-runner image. The default accepted result is CPU-only capacity (gpu.available=false); setPOC_GPU_CAPACITY_EXPECT_AVAILABLE=trueafter POC has a real GPU node pool.npm run acceptance:poc-readinessincludes deployed worker GPU capacity evidence through a defaultkubectl port-forwardprobe and verifies the platformgpuWorkflowPreflightgate throughGET /api/acceptance/poc-readiness, so every POC readiness report states both whether the current hardware expectation is CPU-only or GPU-available and whether the latest real GPU-requesting workflow matches that expectation. The API check usesDROBOTICFLOW_API_TOKENfrom the environment ordroboticflow-auth-secret; setPOC_PLATFORM_READINESS_ENABLED=falseonly for isolated script tests that intentionally bypass platform evidence.npm run acceptance:poc-gpu-workflowverifies the worker GPU path through a real POC workflow run. It uses a session-authenticated acceptance user when provided; otherwise it usesDROBOTICFLOW_API_TOKENfrom the environment ordroboticflow-auth-secretonly for the bounded POC GPU route plus run reads, then triggers the deployed worker through a local port-forward. It expectsgpu request cannot be scheduledwhile POC is CPU-only; withPOC_GPU_WORKFLOW_EXPECT_AVAILABLE=trueorPOC_GPU_CAPACITY_EXPECT_AVAILABLE=true, it expects a completed run with artifacts. The latest proof is exposed in POC readiness asgpuWorkflowPreflight.
Acceptance
Run local verification before merging:
npm run verify:local
npm run verify:security
The local gate includes shell syntax checks for operational scripts, so GPU acceptance, deployment, cleanup, and reconciliation wrappers are parsed before a change is accepted. The security gate runs npm audit --audit-level=moderate through the retry wrapper; npm run verify runs both gates in order.
Run deployed live workflow acceptance after rollout:
WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true \
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh http://<node-ip>:30080
The workflow acceptance source path may be local/PVC-backed or bos://bucket/prefix. The source node now fails fast when no real input source is configured, so a misconfigured workflow cannot pass by producing an empty source manifest. WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true keeps deployed acceptance from silently using local defaults.
Run browser object-storage acceptance after rollout:
BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE is required and must point to a non-empty real UTF-8 text or JSON payload. The script requires an explicit storage connection selector, uses the supplied payload for create/upload/preview checks, and will not modify connection metadata or fall back to a built-in object body to make the check pass. Set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when the file extension is ambiguous. Set OBJECT_STORAGE_VIDEO_KEY to validate browser playback for a known existing video object.
Run operator preview acceptance against supplied payloads and real source paths:
BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview
The operator preview script has no built-in CSV/text/JSON/ROSBag/LeRobot dataset fallback. Payload files are caller supplied; scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs. Set optional PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON, PREVIEW_CHECK_SCAN_MIN_FILES, PREVIEW_CHECK_ROSBAG_TOPIC, and PREVIEW_CHECK_LEROBOT_ROBOT_TYPE when the target dataset needs stricter assertions. PREVIEW_CHECK_JSON_FIELD_MAP_JSON uses the same strict field-map rules as the runtime operator, and PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.
Run X-platform operator-chain acceptance against a real dataset:
BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform
The script has no synthetic dataset fallback and no built-in acceptance criteria. For bos:// input or upload paths, provide an existing storage connection selector. Set XPLATFORM_RULES_JSON, every XPLATFORM_MIN_* count, quality thresholds, and slice/annotation timing explicitly to match the target dataset before treating the run as production acceptance evidence. The X-platform quality thresholds must be finite normalized [0, 1] values, XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS must be finite positive numbers, and XPLATFORM_SEGMENT_DURATION_SECONDS plus every XPLATFORM_MIN_* count must be exact safe positive integers. Python acceptance and reconciliation scripts use the shared protected API client for token/session auth, CSRF propagation, JSON requests, and optional transient-error retries.
Reconcile persistent system data after rollout:
K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080
The reconcile wrapper delegates to scripts/reconcile_system_data.py. It removes legacy demo records, updates cluster settings through the real API, and derives builtin operators from the live /api/operators catalog before refreshing each builtin to the digest-pinned image declared in operator-repos/builtin/<slug>/release.json for the target environment. Registry digest lookup is used only as a guardrail: a mismatch between the committed release manifest and the live registry digest blocks reconciliation instead of becoming an implicit production truth.
Builtin operator source lives in operator-repos/builtin/<slug>/operator.json and the paired release.json. internal/api/builtin_operators_catalog.json is generated from those source and release files before being embedded by the API. Startup validation rejects malformed or duplicate generated catalog entries before they can drift into the database, and documentation generation defaults to the local generated catalog; live API export requires an explicit --api-base-url.
Run GPU acceptance on a real GPU cluster:
npm run acceptance:gpu
Run the POC workflow-level GPU preflight check with a real acceptance user:
DROBOTICFLOW_ACCEPTANCE_EMAIL=<operator@example.com> \
DROBOTICFLOW_ACCEPTANCE_PASSWORD=<password> \
npm run acceptance:poc-gpu-workflow
Live workflow acceptance, GPU acceptance, object-storage acceptance, operator preview acceptance, and X-platform real-data acceptance can run against a deployed environment from a local shell or external CI. Provide WORKFLOW_ACCEPTANCE_SOURCE_PATH for workflow acceptance, configure a kubeconfig when GPU acceptance targets a non-default environment, and provide explicit live data paths plus storage connection selectors when storage-backed acceptance is enabled. For object-storage acceptance, provide OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE as a non-empty real UTF-8 payload file before invoking the browser check. For operator preview acceptance, provide PREVIEW_CHECK_CSV_PAYLOAD_FILE, PREVIEW_CHECK_TEXT_PAYLOAD_FILE, PREVIEW_CHECK_JSON_PAYLOAD_FILE, the JSON field settings, and the real scan/ROSBag/LeRobot source paths. For X-platform acceptance, provide the real source path plus XPLATFORM_RULES_JSON, all XPLATFORM_MIN_* counts, quality thresholds, and slice/annotation timing through environment variables or external CI secrets.
Remaining Cluster Responsibilities
Drobodata now exposes the necessary application controls, but production readiness still depends on the target cluster:
- TLS ingress or trusted private network exposure
- network policies around MongoDB, API, worker, and operator-runner Jobs
- MongoDB backup, restore, and credential rotation
- registry credential rotation
- Prometheus scraping and alerting for worker metrics and failed Jobs