droboticflow/docs/deployment.md
2026-06-23 08:38:39 +08:00

20 KiB

Deployment Guide

Images

This project deploys three images:

  • droboticflow-api
  • droboticflow-worker
  • droboticflow-operator-runner

Example Baidu registry naming:

  • ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:<tag>
  • ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:<tag>
  • ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:<tag>

1. Log In to the Registry

Use the public endpoint unless you are deploying from the same VPC as the registry:

docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com --username <registry-user>

2. Build and Push Images

REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260408 \
./scripts/build-and-push-images.sh

Image contents:

  • droboticflow-api: Vite frontend assets + Go API binary
  • droboticflow-worker: Go worker binary
  • droboticflow-operator-runner: Python runtime + backend/operator-runner/main.py

The script prints the exact image URLs. Reuse those values for deployment:

API_IMAGE=...
WORKER_IMAGE=...
OPERATOR_RUNNER_IMAGE=...

You can also load a named deployment preset:

DEPLOY_ENV=wjzh \
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260421 \
./scripts/build-and-push-images.sh

On the dev branch, the scripts auto-resolve to DEPLOY_ENV=poc when you leave DEPLOY_ENV unset.

3. Prepare Kubernetes Access

Point kubectl at the target cluster before deploying. The deployment scripts do not store kubeconfig in the repository.

Recommended environment variables:

export K8S_NAMESPACE=droboticflow
export REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_NAMESPACE=dataflow
export IMAGE_TAG=20260408
export API_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:20260408
export WORKER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:20260408
export OPERATOR_RUNNER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:20260408
export OPERATOR_GPU_NODE_SELECTOR=accelerator=nvidia
export OPERATOR_GPU_TOLERATIONS_JSON='[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]'
export OPERATOR_GPU_RUNTIME_CLASS=nvidia
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
export REGISTRY_SERVER=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_USERNAME=<registry-user>
export REGISTRY_PASSWORD=<registry-password>
export K8S_IMAGE_PULL_SECRET_NAME=droboticflow-registry
export DROBOTICFLOW_API_TOKEN=<optional-shared-api-token>
# Optional global BOS endpoint. Leave unset only when each BOS connection or workflow provides its own endpoint.
# export BOS_ENDPOINT=https://<region>.bcebos.com
# Optional AI generation. Leave all three unset to disable AI generation.
# When AI_API_KEY is set, AI_API_BASE_URL and AI_MODEL are required.
# export AI_API_BASE_URL=https://<openai-compatible-endpoint>/v1
# export AI_MODEL=<model-name>
# export AI_API_KEY=<openai-compatible-server-key>

For the wjzh environment, the repo now carries non-secret defaults in scripts/deploy-envs/wjzh.env. Sensitive values stay out of git and can be stored in either:

  • $HOME/.config/droboticflow/deploy/wjzh.secrets.env
  • .env.wjzh.local

The loader also looks for $HOME/.config/droboticflow/kubeconfigs/wjzh.config and exports KUBECONFIG automatically when DEPLOY_ENV=wjzh is set.

For the poc environment, use scripts/deploy-envs/poc.env. It is separated from wjzh and defaults to:

  • MONGO_DB=poc
  • API_PUBLIC_LOAD_BALANCER=true
  • BOS_ENDPOINT=https://bj.bcebos.com
  • the same namespace, AI defaults, and PVC settings as wjzh

The repository does not hardcode registry coordinates for either preset. Provide either all three image references (API_IMAGE, WORKER_IMAGE, OPERATOR_RUNNER_IMAGE) or REGISTRY_HOST plus REGISTRY_NAMESPACE so build, render, and deploy scripts can derive the image references from IMAGE_TAG.

Store poc secrets outside git in:

  • $HOME/.config/droboticflow/deploy/poc.secrets.env
  • .env.poc.local

At minimum, the poc secrets file should provide REGISTRY_HOST, REGISTRY_NAMESPACE or the three explicit image references, REGISTRY_USERNAME, REGISTRY_PASSWORD, MONGO_URI, DROBOTICFLOW_API_TOKEN, and for first-time workspace setup WORKSPACE_BOS_BUCKET, BOS_ACCESS_KEY_ID, and BOS_SECRET_ACCESS_KEY. Replace every placeholder in MONGO_URI before running API, worker, render, or deploy commands; runtime startup rejects missing values, whitespace, and unreplaced markers such as <host>.

The loader also checks $HOME/.config/droboticflow/kubeconfigs/poc.config automatically. On the dev branch and poc branches, trying to deploy with another DEPLOY_ENV now fails fast to avoid cross-environment mistakes.

For first-time POC cluster setup, configure the shared workspace before deploying the app:

DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh

This installs the BOS CSI node plugin and creates the droboticflow-workspace PV/PVC. The BOS PV mount options include meta_expires=0, allow_other, and mount_umask=000 so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. WORKSPACE_BOS_BUCKET has no repository default; it must come from the local poc secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.

The POC environment also creates a public droboticflow-api-public LoadBalancer service. The standard droboticflow-api service defaults to NodePort for compatibility, but can be rendered as ClusterIP with API_SERVICE_TYPE=ClusterIP when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When API_PUBLIC_LOAD_BALANCER=true, scripts/deploy-k8s.sh requires DROBOTICFLOW_API_TOKEN and requires DROBOTICFLOW_REQUIRE_SESSION_AUTH=true; if the token is omitted locally, the script can reuse an existing droboticflow-auth-secret from the target namespace. The current POC LoadBalancer is HTTP-only, so scripts/deploy-envs/poc.env sets DROBOTICFLOW_SESSION_COOKIE_SECURE=false to keep session login usable for browser preview and acceptance automation. Override it to true only when POC is served through HTTPS. Boolean aliases such as yes, on, and 1 are treated as true by both manifest rendering and deployment flow control.

4. Deploy

./scripts/deploy-k8s.sh

Or, with the reusable preset:

DEPLOY_ENV=wjzh ./scripts/deploy-k8s.sh

Or for the POC environment:

DEPLOY_ENV=poc ./scripts/deploy-k8s.sh

The deploy script performs these actions:

  1. creates the namespace if it does not exist
  2. creates or updates the image pull secret
  3. loads MONGO_URI from the current cluster secret when omitted, then validates that it points to an external MongoDB instance
  4. creates or updates the droboticflow-mongo-secret Secret with MONGO_URI
  5. creates or updates the droboticflow-ai-secret Secret for server-side AI generation
  6. renders k8s/droboticflow.template.yaml
  7. validates the rendered manifest with kubectl apply --dry-run=client
  8. applies the manifest to the cluster
  9. waits for API and worker rollouts

The main deployment path no longer creates a platform MongoDB Pod or Service. You must provision the backing MongoDB outside this manifest and provide its URI explicitly.

5. Services Created

  • droboticflow-api: NodePort service that exposes the frontend and API
  • droboticflow-worker: ClusterIP service for worker health/debugging

Default API NodePort:

  • 30080

If the cluster node IP is reachable, the UI and API are exposed at:

http://<node-ip>:30080

6. Manifest Variables

scripts/render-manifests.sh accepts:

  • K8S_NAMESPACE
  • MONGO_DB
  • MONGO_CONNECT_RETRIES
  • MONGO_CONNECT_RETRY_DELAY_MS
  • WORKER_POLL_INTERVAL_MS
  • WORKER_API_BASE_URL
  • API_NODE_PORT
  • API_REPLICAS
  • WORKER_REPLICAS
  • K8S_IMAGE_PULL_SECRET_NAME
  • DROBOTICFLOW_API_TOKEN
  • DROBOTICFLOW_REQUIRE_SESSION_AUTH
  • DROBOTICFLOW_SESSION_COOKIE_SECURE
  • BOS_ENDPOINT
  • AI_API_BASE_URL
  • AI_MODEL
  • AI_API_KEY (validation only; the manifest reads the key from droboticflow-ai-secret)
  • REGISTRY_HOST
  • REGISTRY_NAMESPACE
  • IMAGE_TAG
  • API_IMAGE
  • WORKER_IMAGE
  • OPERATOR_RUNNER_IMAGE
  • OPERATOR_GPU_NODE_SELECTOR
  • OPERATOR_GPU_TOLERATIONS_JSON
  • OPERATOR_GPU_RUNTIME_CLASS
  • OPERATOR_JOB_TTL_SECONDS
  • OPERATOR_ALLOW_SHELL_COMMANDS
  • OPERATOR_COMMAND_ALLOWLIST_REGEX
  • OPERATOR_COMMAND_DENYLIST_REGEX
  • DROBOTICFLOW_TRUSTED_PROXY_CIDRS
  • API_SERVICE_TYPE

7. Notes

  • The main manifest only deploys the application components. It does not create MongoDB resources in-cluster.
  • MONGO_URI is required by API and worker startup and must be a concrete mongodb:// or mongodb+srv:// URI without whitespace or unreplaced placeholder markers. Kubernetes deployment must reference an external MongoDB instance; localhost, mongodb:27017, and in-cluster .svc addresses are rejected by scripts/deploy-k8s.sh.
  • Startup configuration is validated before manifest output and before scripts/deploy-k8s.sh performs any cluster writes. MONGO_CONNECT_RETRIES, MONGO_CONNECT_RETRY_DELAY_MS, and WORKER_POLL_INTERVAL_MS must be positive integers; API_REPLICAS, WORKER_REPLICAS, and OPERATOR_JOB_TTL_SECONDS must be non-negative integers; API_NODE_PORT must be in 30000..32767 when API_SERVICE_TYPE=NodePort; boolean env values must be one of true/false, 1/0, yes/no, or on/off. Kubernetes namespace, image-pull-secret, workspace PVC, and public LoadBalancer Service names must be lowercase DNS-compatible values; workspace mount paths must be absolute container paths; worker service URLs must be absolute http:// or https:// URLs; non-empty BOS and AI endpoint values must also be absolute http:// or https:// URLs; if AI_API_KEY is configured, both AI_API_BASE_URL and AI_MODEL are required and there is no provider/model hardcoded fallback; container image references must be non-empty and whitespace-free. scripts/render-manifests.sh, scripts/deploy-k8s.sh, and scripts/build-and-push-images.sh require explicit API_IMAGE, WORKER_IMAGE, and OPERATOR_RUNNER_IMAGE values unless REGISTRY_HOST and REGISTRY_NAMESPACE are both set, and all three scripts reject malformed image references before Docker or Kubernetes calls. Multi-line ConfigMap block values, including OPERATOR_GPU_TOLERATIONS_JSON, OPERATOR_COMMAND_ALLOWLIST_REGEX, and OPERATOR_COMMAND_DENYLIST_REGEX, are indented before manifest output so formatted JSON and multi-line regex lists remain valid YAML.
  • BOS_ENDPOINT is rendered into the API and worker ConfigMap as an empty value when unset. Leave it unset only when every BOS scan config or storage connection provides its own endpoint; if both the workflow/connection endpoint and global BOS_ENDPOINT are empty, real BOS operations fail with an explicit missing-endpoint error instead of falling back to a hardcoded region.
  • API and worker entrypoints are Go binaries with Mongo startup retry, so they tolerate the database becoming ready after the Deployments start.
  • The API proxies /worker-api/* to WORKER_API_BASE_URL, defaulting to the in-cluster http://droboticflow-worker:3002 Service. Override it only when the worker Service name, namespace, or exposure model differs.
  • The worker runs with Kubernetes RBAC that allows creating and watching Jobs in the namespace.
  • The operator-runner Job template inherits the private registry pull secret name from K8S_IMAGE_PULL_SECRET_NAME.
  • Operator-runner Jobs apply CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities and fail before Job creation when malformed. A positive GPU value is emitted as nvidia.com/gpu in both requests and limits; the target cluster must have GPU nodes and the NVIDIA device plugin/runtime configured.
  • GPU-only scheduling settings are optional: OPERATOR_GPU_NODE_SELECTOR accepts comma-separated key=value pairs, OPERATOR_GPU_TOLERATIONS_JSON accepts a Kubernetes tolerations JSON array, and OPERATOR_GPU_RUNTIME_CLASS sets runtimeClassName. These settings are only injected into Jobs that request a positive GPU count. Invalid selector or tolerations values fail worker startup.
  • Finished operator-runner Jobs receive ttlSecondsAfterFinished from OPERATOR_JOB_TTL_SECONDS, default 86400. Use npm run k8s:cleanup-jobs to delete old finished Jobs that predate TTL, or set DRY_RUN=true first to list candidates.
  • Explicit COMMAND / SCRIPT execution is disabled by default. Set OPERATOR_ALLOW_SHELL_COMMANDS=true only for deployments that need shell operators, and constrain it with OPERATOR_COMMAND_ALLOWLIST_REGEX / OPERATOR_COMMAND_DENYLIST_REGEX where possible. These values are injected by the worker and are not overridable from workflow hyperparams.
  • The operator-runner is not a standalone Deployment. It is launched on demand as a Python Job container.
  • Failed operator-runner Jobs report Job conditions, Pod phase, container waiting or termination reasons, Pod events, and available logs back through the run error path.
  • Worker runtime metrics are available from GET /worker-api/metrics in Prometheus text format. Through the API, stats/metrics and worker control endpoints require an admin/operator session or an API token when session auth is not enforced. The worker Service includes Prometheus scrape annotations for internal collection, and the endpoint includes duration histograms for dispatch loop processing and node execution.
  • AI generation runs through the API Deployment. AI_API_KEY is mounted through the droboticflow-ai-secret Secret and must not be baked into frontend env vars.
  • DROBOTICFLOW_API_TOKEN is mounted through droboticflow-auth-secret. When enabled, non-browser clients call APIs with Authorization: Bearer <token> or X-Droboticflow-API-Token. Do not bake this shared token into frontend assets.
  • DROBOTICFLOW_REQUIRE_SESSION_AUTH=true enables HttpOnly cookie-session login for browser business APIs. Public LoadBalancer deploys require it so role checks are not bypassed by a shared machine token. Bootstrap remains available through /api/auth/bootstrap-super-admin before the first super admin exists.
  • DROBOTICFLOW_SESSION_COOKIE_SECURE=true sets the session cookie Secure attribute. scripts/deploy-k8s.sh defaults it to true when API_PUBLIC_LOAD_BALANCER=true; local development can leave it false.
  • DROBOTICFLOW_TRUSTED_PROXY_CIDRS controls which reverse proxies may supply X-Forwarded-For / X-Real-IP for rate limiting. Leave it empty unless ingress or load balancer source ranges are known.
  • Session-authenticated unsafe requests, including logout, require X-Droboticflow-CSRF-Token to match the readable droboticflow_csrf cookie. The bundled frontend sets this header automatically.
  • IMAGE_TAG defaults to the current git short SHA in build, deploy, and render scripts only when explicit registry coordinates are configured. Set it explicitly for release names or rollback-friendly tags.
  • scripts/live-workflow-acceptance.sh requires WORKFLOW_ACCEPTANCE_SOURCE_PATH, a real path visible to operator-runner Jobs. The 真实数据源 runner infers filesystem/PVC paths as LOCAL and bos://bucket/prefix paths as BOS; if no real source is configured, the source node fails instead of emitting an empty manifest. Set WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true for deployed-environment workflow acceptance so missing BASE_URL / API_BASE_URL fails instead of falling back to localhost. The script can exercise session-authenticated deployments when DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD are set; it logs in, persists cookies, and sends the CSRF header on write requests.
  • scripts/reconcile-system-data.sh delegates to scripts/reconcile_system_data.py and uses the same protected-client behavior. Set DROBOTICFLOW_API_TOKEN for token-protected APIs when session auth is not enforced. For public session-auth deployments, set DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD to an organization admin account because the reconcile flow updates settings, users, templates, and operators. Builtin operators are read from the live /api/operators catalog and refreshed to OPERATOR_RUNNER_IMAGE; the script does not carry a separate builtin catalog.
  • scripts/object-storage-acceptance.mjs runs browser create, preview, rename, upload, delete, and optional video playback checks against a live storage connection. It never rewrites storage connection metadata; provide OBJECT_STORAGE_CONNECTION_ID, OBJECT_STORAGE_CONNECTION_NAME, or OBJECT_STORAGE_BUCKET explicitly. OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE is required and must point to a non-empty real UTF-8 text or JSON payload used for create/upload/preview checks; set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when auto-detection is not enough. Set OBJECT_STORAGE_VIDEO_KEY only when an existing playable object should be validated. PLAYWRIGHT_IGNORE_HTTPS_ERRORS must be an explicit boolean spelling when set. The script supports DROBOTICFLOW_API_TOKEN for token-protected non-session APIs and DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD for session-authenticated browser access; session-auth object-storage management checks require an organization admin account.
  • scripts/playwright-preview-operators-check.mjs runs browser/operator preview acceptance with caller-supplied payloads and real source paths. Provide PREVIEW_CHECK_CSV_PAYLOAD_FILE, PREVIEW_CHECK_TEXT_PAYLOAD_FILE, PREVIEW_CHECK_JSON_PAYLOAD_FILE, PREVIEW_CHECK_JSON_EXTRACT_FIELD, PREVIEW_CHECK_JSON_FILTER_FIELD, PREVIEW_CHECK_JSON_FIELD_MAP_JSON, PREVIEW_CHECK_SCAN_SOURCE_PATH, PREVIEW_CHECK_ROSBAG_SOURCE_PATH, and PREVIEW_CHECK_LEROBOT_SOURCE_PATH. The script no longer generates built-in CSV/text/JSON/ROSBag/LeRobot data. Optional PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.
  • scripts/xplatform-live-acceptance.py runs the X-platform chain only when the caller provides the real source path plus explicit acceptance criteria. Provide XPLATFORM_RULES_JSON, all XPLATFORM_MIN_* counts, finite normalized XPLATFORM_MIN_CLARITY, XPLATFORM_MAX_SHAKE, and XPLATFORM_MAX_HAND_OCCLUSION, finite positive XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS, and exact safe positive integer values for XPLATFORM_SEGMENT_DURATION_SECONDS and every XPLATFORM_MIN_* count; the script has no built-in sampling rule or threshold fallback.
  • npm run acceptance:gpu creates a temporary GPU Job in the target namespace and runs nvidia-smi. Override GPU_ACCEPTANCE_IMAGE, GPU_ACCEPTANCE_TIMEOUT, GPU_RESOURCE_NAME, GPU_RESOURCE_COUNT, and the OPERATOR_GPU_* scheduling variables when the cluster uses custom GPU images, wait windows, or labels. The script validates namespace, image, timeout, Job name, resource name/count, node selector, tolerations JSON, and runtime class before any kubectl call so malformed GPU acceptance inputs fail locally instead of mutating the cluster with a bad manifest. Dynamic manifest strings are rendered as quoted YAML scalars.
  • Deployed-environment checks run from a local shell or external CI by invoking the acceptance scripts directly. Configure WORKFLOW_ACCEPTANCE_SOURCE_PATH for live workflow acceptance, a kubeconfig for GPU acceptance, and optionally DROBOTICFLOW_API_TOKEN, DROBOTICFLOW_ACCEPTANCE_EMAIL, DROBOTICFLOW_ACCEPTANCE_PASSWORD, OBJECT_STORAGE_CONNECTION_ID, OBJECT_STORAGE_CONNECTION_NAME, OBJECT_STORAGE_BUCKET, OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE, OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE, OBJECT_STORAGE_VIDEO_KEY, PREVIEW_CHECK_CSV_PAYLOAD_FILE, PREVIEW_CHECK_TEXT_PAYLOAD_FILE, PREVIEW_CHECK_JSON_PAYLOAD_FILE, the preview JSON field settings, the preview scan/ROSBag/LeRobot source paths, and the X-platform source path plus explicit rules/threshold/count inputs for authenticated acceptance tests.
  • If you use DashScope-compatible models, prefer a backend-allowed API key. Do not deploy tool-only Coding Plan keys into the cluster.
  • Production hardening still depends on the target cluster policy: TLS ingress, network policies, backup strategy, and secret rotation.