22 KiB
Deployment Guide
Images
This project deploys three images:
droboticflow-apidroboticflow-workerdroboticflow-operator-runner
Example Baidu registry naming:
ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:<tag>ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:<tag>ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:<tag>
1. Log In to the Registry
Use the public endpoint unless you are deploying from the same VPC as the registry:
docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com --username <registry-user>
2. Build and Push Images
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260408 \
./scripts/build-and-push-images.sh
Image contents:
droboticflow-api: Vite frontend assets + Go API binarydroboticflow-worker: Go worker binarydroboticflow-operator-runner: Python runtime +backend/operator-runner/main.py
The script prints the exact image URLs. Reuse those values for deployment:
API_IMAGE=...
WORKER_IMAGE=...
OPERATOR_RUNNER_IMAGE=...
You can also load a named deployment preset:
DEPLOY_ENV=wjzh \
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260421 \
./scripts/build-and-push-images.sh
On the dev branch, the scripts auto-resolve to DEPLOY_ENV=poc when you leave DEPLOY_ENV unset.
3. Prepare Kubernetes Access
Point kubectl at the target cluster before deploying. The deployment scripts do not store kubeconfig in the repository.
Recommended environment variables:
export K8S_NAMESPACE=droboticflow
export REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_NAMESPACE=dataflow
export IMAGE_TAG=20260408
export API_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:20260408
export WORKER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:20260408
export OPERATOR_RUNNER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:20260408
export OPERATOR_GPU_NODE_SELECTOR=accelerator=nvidia
export OPERATOR_GPU_TOLERATIONS_JSON='[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]'
export OPERATOR_GPU_RUNTIME_CLASS=nvidia
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
export REGISTRY_SERVER=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_USERNAME=<registry-user>
export REGISTRY_PASSWORD=<registry-password>
export K8S_IMAGE_PULL_SECRET_NAME=droboticflow-registry
export DROBOTICFLOW_API_TOKEN=<optional-shared-api-token>
# Optional global BOS endpoint. Leave unset only when each BOS connection or workflow provides its own endpoint.
# export BOS_ENDPOINT=https://<region>.bcebos.com
# Optional AI generation. Leave all three unset to disable AI generation.
# When AI_API_KEY is set, AI_API_BASE_URL and AI_MODEL are required.
# export AI_API_BASE_URL=https://<openai-compatible-endpoint>/v1
# export AI_MODEL=<model-name>
# export AI_API_KEY=<openai-compatible-server-key>
For the wjzh environment, the repo now carries non-secret defaults in scripts/deploy-envs/wjzh.env. Sensitive values stay out of git and can be stored in either:
$HOME/.config/droboticflow/deploy/wjzh.secrets.env.env.wjzh.local
The loader also looks for $HOME/.config/droboticflow/kubeconfigs/wjzh.config and exports KUBECONFIG automatically when DEPLOY_ENV=wjzh is set.
For the poc environment, use scripts/deploy-envs/poc.env. It is separated from wjzh and defaults to:
MONGO_DB=pocAPI_PUBLIC_LOAD_BALANCER=trueBOS_ENDPOINT=https://bj.bcebos.com- the same namespace, AI defaults, and PVC settings as
wjzh
The repository does not hardcode registry coordinates for either preset. Provide either all three image references (API_IMAGE, WORKER_IMAGE, OPERATOR_RUNNER_IMAGE) or REGISTRY_HOST plus REGISTRY_NAMESPACE so build, render, and deploy scripts can derive the image references from IMAGE_TAG.
Store poc secrets outside git in:
$HOME/.config/droboticflow/deploy/poc.secrets.env.env.poc.local
At minimum, the poc secrets file should provide REGISTRY_HOST, REGISTRY_NAMESPACE or the three explicit image references, REGISTRY_USERNAME, REGISTRY_PASSWORD, MONGO_URI, DROBOTICFLOW_API_TOKEN, and for first-time workspace setup WORKSPACE_BOS_BUCKET, BOS_ACCESS_KEY_ID, and BOS_SECRET_ACCESS_KEY. Replace every placeholder in MONGO_URI before running API, worker, render, or deploy commands; runtime startup rejects missing values, whitespace, and unreplaced markers such as <host>.
The loader also checks $HOME/.config/droboticflow/kubeconfigs/poc.config automatically. On the dev branch and poc branches, trying to deploy with another DEPLOY_ENV now fails fast to avoid cross-environment mistakes.
For first-time POC cluster setup, configure the shared workspace before deploying the app:
DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh
This installs the BOS CSI node plugin and creates the droboticflow-workspace PV/PVC. The BOS PV mount options include meta_expires=0, allow_other, and mount_umask=000 so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. WORKSPACE_BOS_BUCKET has no repository default; it must come from the local poc secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
The POC environment also creates a public droboticflow-api-public LoadBalancer service. The standard droboticflow-api service defaults to NodePort for compatibility, but can be rendered as ClusterIP with API_SERVICE_TYPE=ClusterIP when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When API_PUBLIC_LOAD_BALANCER=true, scripts/deploy-k8s.sh requires DROBOTICFLOW_API_TOKEN and requires DROBOTICFLOW_REQUIRE_SESSION_AUTH=true; if the token is omitted locally, the script can reuse an existing droboticflow-auth-secret from the target namespace. The current POC LoadBalancer is HTTP-only, so scripts/deploy-envs/poc.env sets DROBOTICFLOW_SESSION_COOKIE_SECURE=false to keep session login usable for browser preview and acceptance automation. Override it to true only when POC is served through HTTPS. Boolean aliases such as yes, on, and 1 are treated as true by both manifest rendering and deployment flow control.
4. Deploy
./scripts/deploy-k8s.sh
Or, with the reusable preset:
DEPLOY_ENV=wjzh ./scripts/deploy-k8s.sh
Or for the POC environment:
DEPLOY_ENV=poc ./scripts/deploy-k8s.sh
The deploy script performs these actions:
- creates the namespace if it does not exist
- creates or updates the image pull secret
- loads
MONGO_URIfrom the current cluster secret when omitted, then validates that it points to an external MongoDB instance - creates or updates the
droboticflow-mongo-secretSecret withMONGO_URI - creates or updates the
droboticflow-ai-secretSecret for server-side AI generation - renders
k8s/droboticflow.template.yaml - validates the rendered manifest with
kubectl apply --dry-run=client - applies the manifest to the cluster
- waits for API and worker rollouts
The main deployment path no longer creates a platform MongoDB Pod or Service. You must provision the backing MongoDB outside this manifest and provide its URI explicitly.
5. Services Created
droboticflow-api: NodePort service that exposes the frontend and APIdroboticflow-worker: ClusterIP service for worker health/debugging
Default API NodePort:
30080
If the cluster node IP is reachable, the UI and API are exposed at:
http://<node-ip>:30080
6. Manifest Variables
scripts/render-manifests.sh accepts:
K8S_NAMESPACEMONGO_DBMONGO_CONNECT_RETRIESMONGO_CONNECT_RETRY_DELAY_MSWORKER_POLL_INTERVAL_MSWORKER_API_BASE_URLAPI_NODE_PORTAPI_REPLICASWORKER_REPLICASK8S_IMAGE_PULL_SECRET_NAMEDROBOTICFLOW_API_TOKENDROBOTICFLOW_REQUIRE_SESSION_AUTHDROBOTICFLOW_SESSION_COOKIE_SECUREBOS_ENDPOINTAI_API_BASE_URLAI_MODELAI_API_KEY(validation only; the manifest reads the key fromdroboticflow-ai-secret)REGISTRY_HOSTREGISTRY_NAMESPACEIMAGE_TAGAPI_IMAGEWORKER_IMAGEOPERATOR_RUNNER_IMAGEOPERATOR_GPU_NODE_SELECTOROPERATOR_GPU_TOLERATIONS_JSONOPERATOR_GPU_RUNTIME_CLASSOPERATOR_JOB_TTL_SECONDSOPERATOR_ALLOW_SHELL_COMMANDSOPERATOR_COMMAND_ALLOWLIST_REGEXOPERATOR_COMMAND_DENYLIST_REGEXDROBOTICFLOW_TRUSTED_PROXY_CIDRSAPI_SERVICE_TYPE
7. Notes
- The main manifest only deploys the application components. It does not create MongoDB resources in-cluster.
MONGO_URIis required by API and worker startup and must be a concretemongodb://ormongodb+srv://URI without whitespace or unreplaced placeholder markers. Kubernetes deployment must reference an external MongoDB instance; localhost,mongodb:27017, and in-cluster.svcaddresses are rejected byscripts/deploy-k8s.sh.- Startup configuration is validated before manifest output and before
scripts/deploy-k8s.shperforms any cluster writes.MONGO_CONNECT_RETRIES,MONGO_CONNECT_RETRY_DELAY_MS, andWORKER_POLL_INTERVAL_MSmust be positive integers;API_REPLICAS,WORKER_REPLICAS, andOPERATOR_JOB_TTL_SECONDSmust be non-negative integers;API_NODE_PORTmust be in30000..32767whenAPI_SERVICE_TYPE=NodePort; boolean env values must be one oftrue/false,1/0,yes/no, oron/off. Kubernetes namespace, image-pull-secret, workspace PVC, and public LoadBalancer Service names must be lowercase DNS-compatible values; workspace mount paths must be absolute container paths; worker service URLs must be absolutehttp://orhttps://URLs; non-empty BOS and AI endpoint values must also be absolutehttp://orhttps://URLs; ifAI_API_KEYis configured, bothAI_API_BASE_URLandAI_MODELare required and there is no provider/model hardcoded fallback; container image references must be non-empty and whitespace-free.scripts/render-manifests.sh,scripts/deploy-k8s.sh, andscripts/build-and-push-images.shrequire explicitAPI_IMAGE,WORKER_IMAGE, andOPERATOR_RUNNER_IMAGEvalues unlessREGISTRY_HOSTandREGISTRY_NAMESPACEare both set, and all three scripts reject malformed image references before Docker or Kubernetes calls. Multi-line ConfigMap block values, includingOPERATOR_GPU_TOLERATIONS_JSON,OPERATOR_COMMAND_ALLOWLIST_REGEX, andOPERATOR_COMMAND_DENYLIST_REGEX, are indented before manifest output so formatted JSON and multi-line regex lists remain valid YAML. BOS_ENDPOINTis rendered into the API and worker ConfigMap as an empty value when unset. Leave it unset only when every BOS scan config or storage connection provides its own endpoint; if both the workflow/connection endpoint and globalBOS_ENDPOINTare empty, real BOS operations fail with an explicit missing-endpoint error instead of falling back to a hardcoded region.- API and worker entrypoints are Go binaries with Mongo startup retry, so they tolerate the database becoming ready after the Deployments start.
- The API proxies
/worker-api/*toWORKER_API_BASE_URL, defaulting to the in-clusterhttp://droboticflow-worker:3002Service. Override it only when the worker Service name, namespace, or exposure model differs. - The worker runs with Kubernetes RBAC that allows creating and watching Jobs in the namespace.
- The operator-runner Job template inherits the private registry pull secret name from
K8S_IMAGE_PULL_SECRET_NAME. - Operator-runner Jobs apply CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities and fail before Job creation when malformed. A positive GPU value is emitted as
nvidia.com/gpuin both requests and limits; the target cluster must have GPU nodes and the NVIDIA device plugin/runtime configured. - GPU-only scheduling settings are optional:
OPERATOR_GPU_NODE_SELECTORaccepts comma-separatedkey=valuepairs,OPERATOR_GPU_TOLERATIONS_JSONaccepts a Kubernetes tolerations JSON array, andOPERATOR_GPU_RUNTIME_CLASSsetsruntimeClassName. These settings are only injected into Jobs that request a positive GPU count. Invalid selector or tolerations values fail worker startup. - Finished operator-runner Jobs receive
ttlSecondsAfterFinishedfromOPERATOR_JOB_TTL_SECONDS, default86400. Usenpm run k8s:cleanup-jobsto delete old finished Jobs that predate TTL, or setDRY_RUN=truefirst to list candidates. - Explicit
COMMAND/SCRIPTexecution is disabled by default. SetOPERATOR_ALLOW_SHELL_COMMANDS=trueonly for deployments that need shell operators, and constrain it withOPERATOR_COMMAND_ALLOWLIST_REGEX/OPERATOR_COMMAND_DENYLIST_REGEXwhere possible. These values are injected by the worker and are not overridable from workflow hyperparams. - The operator-runner is not a standalone Deployment. It is launched on demand as a Python Job container.
- Failed operator-runner Jobs report Job conditions, Pod phase, container waiting or termination reasons, Pod events, and available logs back through the run error path.
- Worker runtime metrics are available from
GET /worker-api/metricsin Prometheus text format. Through the API, stats/metrics and worker control endpoints require an admin/operator session or an API token when session auth is not enforced. The worker Service includes Prometheus scrape annotations for internal collection, and the endpoint includes duration histograms for dispatch loop processing and node execution. - AI generation runs through the API Deployment.
AI_API_KEYis mounted through thedroboticflow-ai-secretSecret and must not be baked into frontend env vars. DROBOTICFLOW_API_TOKENis mounted throughdroboticflow-auth-secret. When enabled, non-browser clients call APIs withAuthorization: Bearer <token>orX-Droboticflow-API-Token. WhenDROBOTICFLOW_REQUIRE_SESSION_AUTH=true, this shared token can read run-monitoring APIs for automation (GET /api/runs, run logs/resources/artifacts/lineage, and workflow run history) and can call explicitly documented import endpoints, but it cannot create projects, update workflows, stop runs, or read admin/settings resources. Do not bake this shared token into frontend assets.DROBOTICFLOW_REQUIRE_SESSION_AUTH=trueenables HttpOnly cookie-session login for browser business APIs. Public LoadBalancer deploys require it so role checks are not bypassed by a shared machine token. Bootstrap remains available through/api/auth/bootstrap-super-adminbefore the first super admin exists.DROBOTICFLOW_SESSION_COOKIE_SECURE=truesets the session cookieSecureattribute.scripts/deploy-k8s.shdefaults it totruewhenAPI_PUBLIC_LOAD_BALANCER=true; local development can leave itfalse.DROBOTICFLOW_TRUSTED_PROXY_CIDRScontrols which reverse proxies may supplyX-Forwarded-For/X-Real-IPfor rate limiting. Leave it empty unless ingress or load balancer source ranges are known.- Session-authenticated unsafe requests, including logout, require
X-Droboticflow-CSRF-Tokento match the readabledroboticflow_csrfcookie. The bundled frontend sets this header automatically. IMAGE_TAGdefaults to the current git short SHA in build, deploy, and render scripts only when explicit registry coordinates are configured. Set it explicitly for release names or rollback-friendly tags.scripts/live-workflow-acceptance.shrequiresWORKFLOW_ACCEPTANCE_SOURCE_PATH, a real path visible to operator-runner Jobs. The真实数据源runner infers filesystem/PVC paths asLOCALandbos://bucket/prefixpaths asBOS; if no real source is configured, the source node fails instead of emitting an empty manifest. SetWORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=truefor deployed-environment workflow acceptance so missingBASE_URL/API_BASE_URLfails instead of falling back to localhost. The script can exercise session-authenticated deployments whenDROBOTICFLOW_ACCEPTANCE_EMAILandDROBOTICFLOW_ACCEPTANCE_PASSWORDare set; it logs in, persists cookies, and sends the CSRF header on write requests.scripts/reconcile-system-data.shdelegates toscripts/reconcile_system_data.pyand uses the same protected-client behavior. SetDROBOTICFLOW_API_TOKENfor token-protected APIs when session auth is not enforced. For public session-auth deployments, setDROBOTICFLOW_ACCEPTANCE_EMAILandDROBOTICFLOW_ACCEPTANCE_PASSWORDto an organization admin account because the reconcile flow updates settings, users, templates, and operators. Builtin operators are read from the live/api/operatorscatalog and refreshed toOPERATOR_RUNNER_IMAGE; the script does not carry a separate builtin catalog. POC acceptance result import is narrower:/api/acceptance/poc-operator-resultalso accepts the sharedDROBOTICFLOW_API_TOKENunder session auth because it only writes fixed POC acceptance run artifacts, while normal project/workflow/settings writes still require a user session.scripts/object-storage-acceptance.mjsruns browser create, preview, rename, upload, delete, and optional video playback checks against a live storage connection. It never rewrites storage connection metadata; provideOBJECT_STORAGE_CONNECTION_ID,OBJECT_STORAGE_CONNECTION_NAME, orOBJECT_STORAGE_BUCKETexplicitly.OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILEis required and must point to a non-empty real UTF-8 text or JSON payload used for create/upload/preview checks; setOBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPEwhen auto-detection is not enough. SetOBJECT_STORAGE_VIDEO_KEYonly when an existing playable object should be validated.PLAYWRIGHT_IGNORE_HTTPS_ERRORSmust be an explicit boolean spelling when set. The script supportsDROBOTICFLOW_API_TOKENfor token-protected non-session APIs andDROBOTICFLOW_ACCEPTANCE_EMAIL/DROBOTICFLOW_ACCEPTANCE_PASSWORDfor session-authenticated browser access; session-auth object-storage management checks require an organization admin account.scripts/playwright-preview-operators-check.mjsruns browser/operator preview acceptance with caller-supplied payloads and real source paths. ProvidePREVIEW_CHECK_CSV_PAYLOAD_FILE,PREVIEW_CHECK_TEXT_PAYLOAD_FILE,PREVIEW_CHECK_JSON_PAYLOAD_FILE,PREVIEW_CHECK_JSON_EXTRACT_FIELD,PREVIEW_CHECK_JSON_FILTER_FIELD,PREVIEW_CHECK_JSON_FIELD_MAP_JSON,PREVIEW_CHECK_SCAN_SOURCE_PATH,PREVIEW_CHECK_ROSBAG_SOURCE_PATH, andPREVIEW_CHECK_LEROBOT_SOURCE_PATH. The script no longer generates built-in CSV/text/JSON/ROSBag/LeRobot data. OptionalPREVIEW_CHECK_SCAN_MIN_FILESmust be an exact safe positive integer.scripts/xplatform-live-acceptance.pyruns the X-platform chain only when the caller provides the real source path plus explicit acceptance criteria. ProvideXPLATFORM_RULES_JSON, allXPLATFORM_MIN_*counts, finite normalizedXPLATFORM_MIN_CLARITY,XPLATFORM_MAX_SHAKE, andXPLATFORM_MAX_HAND_OCCLUSION, finite positiveXPLATFORM_CLARITY_SCALEandXPLATFORM_FRAME_INTERVAL_SECONDS, and exact safe positive integer values forXPLATFORM_SEGMENT_DURATION_SECONDSand everyXPLATFORM_MIN_*count; the script has no built-in sampling rule or threshold fallback.npm run acceptance:gpuloads the same deploy environment resolver as the POC deploy scripts, sodevresolves to the POC kubeconfig whenDEPLOY_ENVis unset. It then creates a temporary GPU Job in the target namespace and runsnvidia-smi. OverrideGPU_ACCEPTANCE_IMAGE,GPU_ACCEPTANCE_TIMEOUT,GPU_RESOURCE_NAME,GPU_RESOURCE_COUNT, and theOPERATOR_GPU_*scheduling variables when the cluster uses custom GPU images, wait windows, or labels. The script validates namespace, image, timeout, Job name, resource name/count, node selector, tolerations JSON, and runtime class before anykubectlcall so malformed GPU acceptance inputs fail locally instead of mutating the cluster with a bad manifest. It also checks node allocatable resources before creating the Job; if no node reports at least the requested GPU count, the script exits with the checked node/resource evidence and the environment must be treated as GPU-not-accepted. Dynamic manifest strings are rendered as quoted YAML scalars.npm run acceptance:poc-gpu-workflowcreates a real POC project/workflow/run with the source node requestinggpu=1and expects the worker to fail fast withgpu request cannot be scheduledwhile POC has no GPU nodes. Because POC write APIs require a real session, setDROBOTICFLOW_ACCEPTANCE_EMAILandDROBOTICFLOW_ACCEPTANCE_PASSWORDbefore running it. OverridePOC_GPU_WORKFLOW_SOURCE_PATHonly when the default real fixture path is not present.- Deployed-environment checks run from a local shell or external CI by invoking the acceptance scripts directly. Configure
WORKFLOW_ACCEPTANCE_SOURCE_PATHfor live workflow acceptance, a kubeconfig for GPU acceptance, and optionallyDROBOTICFLOW_API_TOKEN,DROBOTICFLOW_ACCEPTANCE_EMAIL,DROBOTICFLOW_ACCEPTANCE_PASSWORD,OBJECT_STORAGE_CONNECTION_ID,OBJECT_STORAGE_CONNECTION_NAME,OBJECT_STORAGE_BUCKET,OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE,OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE,OBJECT_STORAGE_VIDEO_KEY,PREVIEW_CHECK_CSV_PAYLOAD_FILE,PREVIEW_CHECK_TEXT_PAYLOAD_FILE,PREVIEW_CHECK_JSON_PAYLOAD_FILE, the preview JSON field settings, the preview scan/ROSBag/LeRobot source paths, and the X-platform source path plus explicit rules/threshold/count inputs for authenticated acceptance tests. - If you use DashScope-compatible models, prefer a backend-allowed API key. Do not deploy tool-only Coding Plan keys into the cluster.
- Production hardening still depends on the target cluster policy: TLS ingress, network policies, backup strategy, and secret rotation.