2026-09-27 04:27:54 -07:00
2026-08-19 03:08:37 +08:00
2026-08-19 03:08:37 +08:00
2026-07-06 15:54:27 +08:00
2026-07-30 01:29:39 +08:00
2026-07-13 19:21:12 +08:00

Drobodata

Drobodata is a workflow orchestration platform for embodied AI and vision pipelines. This repository now contains:

  • a React/Vite frontend
  • a Go-based MongoDB-backed REST API
  • a Go-based worker service that executes workflow DAGs
  • a Python-based Kubernetes Job operator runner
  • deployment assets for Kubernetes and Baidu Cloud Container Registry

Architecture

  • cmd/api: Go API entrypoint and SPA host
  • cmd/worker: Go worker entrypoint and worker control API
  • internal/api: REST handlers and request validation
  • internal/worker: run scheduler, DAG executor, audit pause/resume, worker control API
  • internal/store: Mongo-backed repositories plus in-memory test store
  • internal/k8s: Kubernetes Job launcher for operator nodes
  • backend/operator-runner: Python entrypoint used by Kubernetes Jobs
  • src: existing frontend, now wired to real APIs
  • k8s: Kubernetes manifest template
  • scripts: build, render, deploy, and live acceptance scripts
  • docs: API, deployment, operations, and design documents

Local Development

Prerequisites

  • Node.js 22.12+ or 24+
  • Go 1.24+
  • Python 3.9+
  • MongoDB 7+ reachable at MONGO_URI

Setup

  1. Install dependencies:

    npm install
    
  2. Copy the example environment file and replace the MongoDB placeholder with a real URI:

    cp .env.example .env.local
    
  3. Start the Go API and frontend:

    npm run dev
    
  4. Start the full stack including Go worker:

    npm run dev:full
    

Useful Scripts

  • npm run lint: type-check frontend and verify Go formatting
  • npm run format: format frontend source and frontend tests with Prettier
  • npm run format:check: check frontend formatting without writing files
  • npm test: Go unit and integration tests
  • npm run build: build frontend assets and Go binaries
  • npm run verify:local: web/Go/script lint, Go tests, Python operator-runner tests, frontend node tests, build, and Kubernetes manifest dry-run validation
  • npm run verify:security: dependency audit with retry
  • npm run verify: run local verification first, then dependency audit
  • npm run images:poc-kaniko: build and push API, worker, and operator-runner images inside the POC cluster with Kaniko, using the shared workspace PVC and registry secret
  • npm run acceptance:workflow: live workflow acceptance against the running API and worker; requires an explicit real source path
  • npm run acceptance:gpu: real Kubernetes GPU scheduling and nvidia-smi acceptance test
  • npm run k8s:cleanup-jobs: delete old finished operator-runner Jobs that predate TTL cleanup
  • npm run reconcile:data -- <api-base-url>: remove legacy demo data, reset settings, and bootstrap built-in operators through the real API; honors DROBOTICFLOW_API_TOKEN and optional session login env vars for protected deployments
  • npm run acceptance:object-storage: browser-driven object-storage acceptance against a configured live storage connection; requires OBJECT_STORAGE_CONNECTION_ID, OBJECT_STORAGE_CONNECTION_NAME, or OBJECT_STORAGE_BUCKET
  • npm run acceptance:poc-strict: run the strict POC gate: readiness, D-Cloud FAT probe, then full operator acceptance with bos,dcloud,mongo required; blocked preflights emit the same standard platform-importable result marker as the Kubernetes acceptance Job
  • npm run acceptance:poc-operators: run the POC Kubernetes Job that executes the deployed builtin operator modules with real files, video, MongoDB, BOS, and D-Cloud FAT calls when the required POC external credentials are configured
  • npm run acceptance:poc-readiness: preflight POC deployment image, D-Cloud FAT secret, and real fixture manifest readiness as JSON without launching an acceptance Job or printing secret values
  • npm run acceptance:dcloud-fat-probe: non-destructively probe the two D-Cloud FAT API contracts from droboticflow-dcloud-secret before enabling strict full POC acceptance; it reports status, response shape, redacted diagnostic fields, and header names, never secret values or user/task payloads
  • npm run acceptance:poc-fixtures:discover: scan existing real fixture candidates on the shared POC /workspace PVC and optionally write a manifest when every required fixture and shell validation command is present
  • npm run acceptance:poc-fixtures: copy caller-provided real POC acceptance fixture files/directories into the shared workspace PVC and write the fixture manifest consumed by acceptance:poc-operators; it does not generate mock, simulated, or hardcoded fixtures
  • npm run acceptance:xplatform: X-platform operator-chain acceptance against an explicit real dataset; requires XPLATFORM_SOURCE_PATH, explicit XPLATFORM_RULES_JSON, all XPLATFORM_MIN_* counts, quality/slicing thresholds, and a storage connection selector when the source or upload path is bos://...; count and segment values must be exact safe positive integers
  • python3 scripts/wjzh_operator_regression.py --preflight-only: preflight the production-style WJZH real-data operator regression against a live API; requires explicit SOURCE_PATH, UPLOAD_ROOT_PREFIX, storage connection selector, target video filenames, route/filter values, real Robotics dataset paths, incremental seed metadata, quality thresholds, video timing, and SAMPLING_RULES_JSON; integer timing values must be exact safe positive integers

Builtin operator source is organized under operator-repos/builtin/<slug>/. Each directory has an operator.json contract and a committed release.json release manifest. The release manifest is the production image truth: it records the digest-pinned CCR image, source commit, source tree hash, environment release evidence, and acceptance evidence. repository.json and internal/api/builtin_operators_catalog.json are generated from operator.json + release.json; changing only the catalog or only the generated repository manifest is blocked by npm run operators:manifests:check and npm run operators:images:check. Reconciliation updates live builtin operators from the current environment entry in release.json and uses registry lookup only to verify that the digest still exists.

The Python operator-runner keeps missing optional hyperparams on catalog defaults, but explicit invalid overrides fail the node instead of silently falling back. JSON hyperparams such as EXPECTED_JSON, FIELD_MAP_JSON, RULES_JSON, and SOURCE_STATE_JSON must parse as valid JSON. Field maps must be JSON objects with non-empty string source/destination fields and cannot collapse or overwrite output fields; source-state checkpoints must contain valid string fingerprints and exact positive integer directory depth; sampling rules must be JSON arrays whose rules include id, field, operator, value, and finite numeric (0, 100] samplePercent values; boolean samplePercent values are rejected instead of being coerced to 1 or 0. Comparison operators must be supported, boolean hyperparams such as REQUIRE_INTRINSICS accept only explicit true/false spellings, source DATA_SOURCE accepts only INLINE, LOCAL, BOS, or UPSTREAM when present, SCAN_MODE accepts only FULL or INCREMENTAL when present, INCREMENTAL_UNIT accepts only OBJECT or DIRECTORY when present, and SEGMENT_DURATION_SECONDS / INCREMENTAL_DIRECTORY_DEPTH must be exact safe positive integers.

Runtime Configuration

Key environment variables:

  • PORT: API port, default 3001
  • WORKER_PORT: worker port, default 3002
  • MONGO_URI: required MongoDB connection string; it must be set explicitly before API or worker startup, and placeholder markers such as <host> are rejected
  • MONGO_DB: Mongo database name
  • MONGO_CONNECT_RETRIES: startup retry count for API and worker Mongo connections
  • MONGO_CONNECT_RETRY_DELAY_MS: delay between Mongo connection retries
  • WORKER_POLL_INTERVAL_MS: worker polling interval
  • WORKER_API_BASE_URL: API-side upstream URL for /worker-api/*, default http://droboticflow-worker:3002
  • BOS_ENDPOINT: optional global BOS endpoint used when scan configs or BOS storage connections omit an endpoint; leave unset only when every BOS workflow or connection provides its own endpoint
  • K8S_NAMESPACE: namespace used for Jobs
  • K8S_IMAGE_PULL_SECRET_NAME: image pull secret injected into operator-runner Jobs
  • WORKSPACE_PVC_NAME: shared PVC used by worker, API, and operator-runner Jobs
  • WORKSPACE_MOUNT_PATH: shared workspace mount path, default /workspace
  • REGISTRY_HOST / REGISTRY_NAMESPACE: explicit registry coordinates used by build/deploy/render scripts to derive image references when the three image variables below are not set
  • IMAGE_TAG: optional shared image tag for build/deploy/render scripts; scripts default to the current git short SHA only after explicit registry coordinates are configured
  • API_IMAGE / WORKER_IMAGE / OPERATOR_RUNNER_IMAGE: container images used for the API, worker, and node execution Jobs; set all three explicitly, or set REGISTRY_HOST and REGISTRY_NAMESPACE so scripts can derive them
  • OPERATOR_GPU_NODE_SELECTOR: optional comma-separated node selector for GPU Jobs, for example accelerator=nvidia
  • OPERATOR_GPU_TOLERATIONS_JSON: optional Kubernetes tolerations JSON array for GPU Jobs
  • OPERATOR_GPU_RUNTIME_CLASS: optional runtimeClassName for GPU Jobs, for example nvidia
  • OPERATOR_JOB_TTL_SECONDS: Kubernetes TTL for finished operator-runner Jobs, default 86400; set 0 to disable automatic TTL injection
  • OPERATOR_ALLOW_SHELL_COMMANDS: controls whether COMMAND / SCRIPT shell execution is allowed, default false
  • OPERATOR_COMMAND_ALLOWLIST_REGEX: optional regular expression that explicit shell commands must match
  • OPERATOR_COMMAND_DENYLIST_REGEX: optional regular expression that rejects matching explicit shell commands
  • DROBOTICFLOW_API_TOKEN: optional shared API token for non-browser API clients. When set, non-public API calls require Authorization: Bearer <token> or X-Droboticflow-API-Token unless a valid session cookie is already present while session auth is enabled.
  • DROBOTICFLOW_REQUIRE_SESSION_AUTH: optional cookie-session auth enforcement for business APIs, default false locally; public LoadBalancer deploys force it to true
  • DROBOTICFLOW_SESSION_COOKIE_SECURE: sets the Secure attribute on session cookies, default false for local development; enable it behind HTTPS
  • DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD: optional credentials used by live acceptance and reconcile scripts to log in, persist cookies, and send CSRF headers against session-authenticated deployments. Workflow acceptance checks can use admin, pipeline-engineer, data-operator, or legacy operator accounts depending on the exercised surface; reconcile still needs an admin account when it updates settings or management resources.
  • DROBOTICFLOW_TRUSTED_PROXY_CIDRS: optional comma/space-separated proxy CIDRs whose X-Forwarded-For or X-Real-IP headers may be trusted for rate limiting; empty means trust no forwarded headers
  • API_SERVICE_TYPE: Kubernetes API service type for rendered manifests, NodePort by default and ClusterIP when exposure is handled by LoadBalancer or ingress
  • AI_API_BASE_URL: OpenAI-compatible endpoint used by server-side AI generation; required when AI_API_KEY is set
  • AI_MODEL: server-side model name; required when AI_API_KEY is set
  • AI_API_KEY: server-side OpenAI-compatible API key used for workflow/operator generation; leave unset to disable AI generation
  • VITE_API_BASE_URL: optional frontend API base URL override; leave empty for same-origin /api, or set an absolute http:// / https:// URL
  • VITE_DEV_API_BASE_URL: optional Vite dev proxy target for /api, default http://127.0.0.1:3001
  • VITE_DEV_WORKER_BASE_URL: optional Vite dev proxy target for /worker-api, default http://127.0.0.1:3002
  • WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT: set to true for deployed-environment workflow acceptance so omitted API endpoints fail instead of falling back to localhost.

Explicit startup values are validated before API/worker startup, before Kubernetes manifest rendering, and before scripts/deploy-k8s.sh performs cluster writes. MONGO_URI is required at runtime and must be a mongodb:// or mongodb+srv:// URI without whitespace or unreplaced placeholder markers. PORT/WORKER_PORT must be valid TCP ports, Mongo retry count and retry delay must be positive, worker poll interval must be positive, replica counts must be non-negative in manifests, NodePort must be in the Kubernetes 30000..32767 range, and OPERATOR_JOB_TTL_SECONDS must be non-negative (0 disables TTL injection). Manifest, build, and deploy wrappers no longer carry a repository-default image registry: callers must set API_IMAGE, WORKER_IMAGE, and OPERATOR_RUNNER_IMAGE explicitly, or set REGISTRY_HOST and REGISTRY_NAMESPACE so those image references can be derived from IMAGE_TAG; all three wrappers validate generated or explicit image references before Docker or Kubernetes calls. Manifest and deploy wrappers also reject invalid Kubernetes namespace/Secret/PVC/Service names, non-absolute workspace mount paths, malformed worker/BOS service URLs, malformed non-empty AI service URLs, missing AI base/model when an AI key is configured, and container image references with whitespace. Multi-line ConfigMap block values such as GPU tolerations JSON and shell command allow/deny regexes are indented before manifest output so readable multi-line input stays valid YAML. Explicit boolean env values accept true/false, 1/0, yes/no, or on/off; other spellings fail fast instead of silently falling back, and accepted true aliases drive deploy-time branches such as public LoadBalancer creation.

Development and acceptance entrypoints resolve API endpoints through shared helpers instead of each script carrying its own localhost default. Node/Vite entrypoints use scripts/lib/service-endpoints.mjs, Python operational scripts use scripts/lib/droboticflow_env.py, and shell wrappers use scripts/lib/service-endpoints.sh. BASE_URL is accepted by browser-oriented acceptance scripts, while deployment/reconcile wrappers preserve API_BASE_URL as their primary environment variable. Explicit endpoint values must be absolute http:// or https:// URLs so malformed values fail before Vite, curl, Playwright, or urllib requests are attempted. Browser-side VITE_API_BASE_URL follows the same rule, with the empty value reserved for same-origin API requests.

The frontend never reads the AI key directly. Workflow generation and operator generation are proxied through the Go API so secrets stay on the server side. AI-generated operators use an explicit publication lifecycle: draft -> built -> verified -> published. Generated drafts can be saved and exercised only in sandbox workflows; production workflows, templates, runs, and AI workflow generation accept only published AI operators with repository, digest, fixture, test, and acceptance evidence. The settings page can update K8s access details, the worker defaults, the global timezone, provider, baseURL, model, and the server-side AI key. GET /api/settings only returns whether secrets exist via k8s.hasKubeconfig / ai.hasAPIKey; it does not echo plaintext kubeconfig or API keys back to the browser. When session auth is required, /api/settings, /api/users, database browser routes, object-storage management routes, Mongo connection tests, and external business-DB token management are organization-admin surfaces; creating a user or setting a password requires a non-blank password value, user names cannot be blank, and account status must be active or disabled. Business roles are modeled around embodied-AI data production: admin owns organization/system governance, pipeline_engineer owns operators/templates/workflow design and AI-assisted generation, data_operator owns project execution, run control, production monitoring, and storage browsing, qa_reviewer owns workflow audit decisions and post-package sampling acceptance, and compliance_auditor is read-only for monitoring, data assets, and audit traceability. Legacy operator and auditor users remain accepted for migration compatibility. Project mutations reject blank names and unknown statuses; project status must be active or archived. Workflow and template mutations reject blank names and unknown metadata enum values before persistence. Operator mutations reject blank metadata, unknown operator/port enum values, duplicate port IDs, and invalid resource defaults before persistence. Reusable storage/Mongo connection list/get routes and workflow storage browse helpers require admin, pipeline engineer, data operator, or legacy operator role so workflow editors and production operators can select real connections without exposing connection or bucket metadata to reviewers. Creating, updating, deleting, or testing connections remains admin-only. Model-backed AI generation requires admin, pipeline engineer, or legacy operator role. Non-admin pages use /api/public/settings, which exposes only timezone, and users can update only their own sidebar name / email through /api/users/me. Worker polling and concurrency values are positive-integer gated in the browser, the shared settings payload builder, and the API, so values such as 0, 1.5, 60s, or blank strings fail instead of being ignored. Default CPU/MEM resource values are trimmed and validated as positive Kubernetes resource quantities when settings are saved, and default GPU must be a positive integer when present. Timestamps in the UI are rendered in the configured IANA timezone with compact minute-level formatting for cards, lists, and summaries, while logs keep finer granularity when needed. OSS-related bucket and path settings belong on OSS operators in workflows, not in the global system settings page. Scan workflows can now reference reusable BOS storage connections via scanConfig.connectionId, so endpoint and AK/SK are stored once in /api/storage/connections and resolved by the API/worker at preview and execution time instead of being duplicated into each workflow; if a scan config or connection omits endpoint, both API preview and worker execution use the validated BOS_ENDPOINT startup value when it is set, otherwise real BOS requests fail with an explicit missing-endpoint error.

The API now includes a cookie-session account foundation for organization-based access. Auth endpoints cover bootstrap, login, logout, current session, and super-admin organization/user management. Existing local deployments remain compatible by default; public LoadBalancer deploys require DROBOTICFLOW_REQUIRE_SESSION_AUTH=true so browser business APIs keep role checks. Session-authenticated unsafe requests, including logout, use a double-submit droboticflow_csrf cookie plus X-Droboticflow-CSRF-Token header. Browser JSON, object-storage preview, and multipart object upload requests share the same credentials and CSRF header path; object-storage preview byte-count headers are accepted only when they are exact safe non-negative integers. Worker control endpoints under /worker-api require an admin, pipeline engineer, data operator, or legacy operator session, or an API token when session auth is not enforced. Bundled acceptance and reconcile scripts can log in with DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD and will send CSRF headers on writes. Do not embed DROBOTICFLOW_API_TOKEN into frontend builds; it is intended for machine-to-machine calls only.

The Kubernetes runtime now assumes a shared workspace PVC. Worker writes per-node input-artifacts.json files into the shared workspace, operator-runner Jobs read those files through INPUT_ARTIFACTS_PATH, then persist result.json and stage-manifest.json back into the same PVC. Large artifact lists no longer need to travel through giant env vars or stdout payloads. Node CPU and memory requests/limits are applied to Kubernetes Job container resources and must be valid positive Kubernetes resource quantities; malformed explicit values fail before a Job is created instead of being omitted. Operator defaults and workflow-node overrides can also set sharedMemorySize for a writable /dev/shm memory volume; the effective order is node override, operator default, then DROBOTICFLOW_SHARED_MEMORY_SIZE (POC default 8Gi). GPU requests use nvidia.com/gpu when the node or default resource config sets a positive GPU count. GPU scheduling requires a cluster with GPU nodes and the NVIDIA device plugin/runtime installed. Optional GPU node selector, tolerations, and runtime class settings are only injected into Jobs that request a positive GPU count. Invalid GPU selector or tolerations configuration fails worker startup instead of being ignored. Operator-runner Jobs also receive ttlSecondsAfterFinished from OPERATOR_JOB_TTL_SECONDS, and the cleanup script can remove old completed Jobs that were created before TTL was enabled. Run npm run acceptance:gpu against a real GPU cluster to validate allocatable GPU resources, scheduling, runtime, and nvidia-smi execution; the acceptance script rejects invalid namespace, image, timeout, GPU count, selector, toleration, resource-name, Job-name, and runtime-class inputs before it touches the cluster, and renders dynamic manifest values as quoted YAML scalars. Runs that execute on NVIDIA hardware report GPU utilization, GPU memory, GPU count, and GPU model in the resource samples shown on the monitoring page.

Explicit COMMAND / SCRIPT execution is disabled by default. Deployments that need the shell operator must set OPERATOR_ALLOW_SHELL_COMMANDS=true and should constrain commands through allowlist and denylist regular expressions. These policy variables are protected system env vars: workflow and node hyperparams cannot override the worker-provided values.

Kubernetes Job failures include operator logs plus Job conditions, Pod phase, container waiting or termination reasons, and Pod events in the run error path. This makes image pull, scheduling, PVC mount, timeout, and GPU allocation failures visible without separately describing the Pod.

The worker exposes lightweight runtime metrics at /worker-api/metrics in Prometheus text format. The endpoint reports run status gauges, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the latest dispatch queue depth. Access through the API requires an admin, pipeline-engineer, data-operator, or legacy operator session or an API token when session auth is not enforced; the in-cluster worker Service keeps Prometheus scrape annotations for internal collection.

Workflow execution concurrency now has two levels:

  • system default: settings.worker.maxConcurrentRuns
  • workflow override: workflow.maxConcurrentRuns

The worker enforces both. A workflow value of 0 means "inherit the global default". Dispatch only starts queued runs while the global active run count is below the system limit and the active count for that workflow is below its effective limit.

Workflow-level execution safeguards are configured alongside the other top-bar workflow settings in the editor:

  • workflow.nodeTimeoutSeconds
    • 0: disable workflow-level per-node timeout
    • 1+: fail an operator execution once a single node exceeds the configured seconds
  • workflow.runTimeoutSeconds
    • 0: disable workflow-level timeout
    • 1+: fail the current run once total execution time exceeds the configured seconds
  • workflow.maxRetryAttempts
    • 0: disable automatic retry
    • 1+: automatically enqueue a new retry run after a failure, up to the configured count
  • workflow.retryBackoffSeconds
    • 0: retry immediately
    • 1+: keep the retry run queued until its backoff window expires

Automatic retries create real follow-up runs with trigger=retry, attempt=2+, parentRunId pointing at the failed attempt, and availableAt marking the first dispatchable time after backoff. Monitoring and history stay auditable because the original run is never mutated into the retry attempt.

Workflow templates now carry the same workflow-level defaults as project workflows. When you save a workflow as a template, duplicate a template, edit a template, or instantiate a template into a project, the workflow type, schedule, and execution-safeguard fields move across with it instead of being dropped or reset.

Artifact previews are persisted with the run artifacts themselves. JSON artifacts keep their structured payload in artifact.data, while non-JSON artifacts can attach a compact artifact.data.preview block for text, CSV/TSV, JSONL, images, audio/video metadata, PDF/document hints, SQLite/rosbag summaries, and LeRobot dataset summaries. Artifacts also carry portId / portName, which lets the worker route multi-output operators by edge handle instead of mixing all outputs together.

When a JSON artifact declares artifact.data.dataRef.payloadPath, the local external payload is treated as the authoritative data source for data-asset APIs. Unreadable, malformed, or non-object local payloads are omitted instead of falling back to stale inline artifact.data. Data-asset counts, byte sizes, frame counts, and similar integer summaries accept only exact integer values, so decimal, NaN, invalid json.Number, or overflowing values are ignored instead of being truncated into plausible totals.

The built-in operator catalog now includes real data-processing, X-platform video pipeline, and robotics helpers on top of the original source/merge/upload/audit flow: BOS 增量扫描, X 平台数据集索引, 采集目录校验, 视频质检, 数据过滤, 视频切片, 视频标注, 成品打包, 抽检采样, 目录扫描, CSV 转 JSON, JSON 转 CSV, JSON 字段提取, JSON 过滤, JSON 映射重命名, 文本模板渲染, 条件分流, LeRobot 数据集概览, ROSBag 数据概览, and Shell 命令执行.

BOS 增量扫描 is a stateful source operator. The workflow trigger still controls whether a workflow runs once, on a timer, or in a streaming loop, but the operator itself persists a per-workflowId/nodeId/sourceFingerprint checkpoint. Each run lists the configured BOS prefix, emits only unseen objects, and the worker commits the updated checkpoint only after the run completes successfully. Failed downstream runs do not advance the checkpoint, so the same increment can be retried safely. The runner and worker reject malformed SOURCE_STATE_JSON, unreadable source-checkpoint.json, lossy checkpoint counts, and invalid directory depth values instead of silently resetting or truncating checkpoint state.

The frontend now exposes a unified 数据资产 module backed by real APIs, while scan-oriented definitions live inside the workflow system project 数据源扫描 instead of a separate first-level page:

  • 源数据: source collections indexed by completed workflows that produced a real dataset-index.json
  • 成品数据: finished clip packages aggregated from packages-manifest.json
  • 抽检任务: persisted human review queues synced from audit-samples.json

数据资产 groups 源数据 and 成品数据 as two tabs instead of two unrelated menu entries:

  • 源数据 shows upstream collected data and its real metadata such as meta.json / video_meta.json
  • 成品数据 shows downstream outputs produced by project workflows, with provenance back to project / workflow / run
  • both tabs accept project/workflow/run scope filters so project detail and workflow editor can deep-link into the relevant assets

抽检任务 and the existing workflow 任务审核 share the same manual-review idea, but they serve different phases:

  • 任务审核: approval gates during workflow execution
  • 抽检任务: post-processing quality review over completed data packages

Deployment

The repository includes three container images:

  • droboticflow-api
  • droboticflow-worker
  • droboticflow-operator-runner

The API image builds the Vite frontend and a Go binary into one container. The worker image contains a Go binary. The operator-runner image is Python-only and is launched by the worker as a Kubernetes Job.

Build and push images:

./scripts/build-and-push-images.sh

Kubernetes deployment requires an explicit external MONGO_URI. The main manifest and deploy script no longer create or manage a cluster-local MongoDB instance.

Render Kubernetes manifests:

./scripts/render-manifests.sh

Validate the rendered Kubernetes YAML before applying it:

npm run k8s:validate

Deploy to Kubernetes:

export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
./scripts/deploy-k8s.sh

When an existing cluster already has droboticflow-ai-secret, deploy-k8s.sh reuses its AI_API_KEY and fills missing AI_API_BASE_URL / AI_MODEL from the current droboticflow-config ConfigMap. Initial AI setup still requires providing AI_API_KEY, AI_API_BASE_URL, and AI_MODEL together.

Reconcile persistent API data after deployment:

export K8S_ENDPOINT=https://<api-server>:6443
# Optional when the API is protected by DROBOTICFLOW_API_TOKEN and session auth is not enforced:
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
# Required for public session-auth deployments; use an organization admin account:
export DROBOTICFLOW_ACCEPTANCE_EMAIL=<admin-email>
export DROBOTICFLOW_ACCEPTANCE_PASSWORD=<admin-password>
./scripts/reconcile-system-data.sh http://<node-ip>:30080

Run live workflow acceptance locally:

WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
npm run acceptance:workflow

Run live workflow acceptance against a deployed cluster:

WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true \
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh http://<node-ip>:30080

WORKFLOW_ACCEPTANCE_SOURCE_PATH can be a filesystem/PVC path or a bos://bucket/prefix path. The real-source operator infers LOCAL or BOS from that path and fails fast when no real source is configured, so an empty source configuration cannot pass by producing an empty manifest. Set WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true for deployed workflow acceptance so the script fails instead of falling back to localhost when BASE_URL / API_BASE_URL is omitted.

Run browser object-storage acceptance against a deployed cluster:

BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
npm run acceptance:object-storage

Set OBJECT_STORAGE_VIDEO_KEY=<existing-video-key> to additionally verify browser video playback from object storage. Public session-auth deployments should also set DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD; token-protected non-session deployments can set DROBOTICFLOW_API_TOKEN.

Run real-environment acceptance from a local shell or external CI by invoking the scripts directly. Workflow acceptance uses ./scripts/live-workflow-acceptance.sh and needs WORKFLOW_ACCEPTANCE_SOURCE_PATH, a real path visible to operator-runner Jobs. GPU acceptance uses npm run acceptance:gpu and needs a kubeconfig with permission to create a temporary Job in K8S_NAMESPACE. Object-storage and X-platform BOS acceptance need existing storage connection selectors and explicit live data paths. Operator preview acceptance accepts PREVIEW_CHECK_SCAN_MIN_FILES only as an exact safe positive integer. X-platform acceptance also needs explicit sampling rules, finite quality thresholds, positive slice/annotation timing, and every XPLATFORM_MIN_* count through environment variables; count and segment values reject signs, leading zeros, decimals, units, and unsafe integers.

See the detailed guides:

Description
No description provided
Readme 80 MiB
Languages
Python 48.8%
Go 29.8%
TypeScript 10.3%
JavaScript 7.9%
Shell 1.3%
Other 1.7%