droboticflow/docs/deployment.md
2026-06-23 16:03:38 +08:00

244 lines
21 KiB
Markdown

# Deployment Guide
## Images
This project deploys three images:
- `droboticflow-api`
- `droboticflow-worker`
- `droboticflow-operator-runner`
Example Baidu registry naming:
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:<tag>`
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:<tag>`
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:<tag>`
## 1. Log In to the Registry
Use the public endpoint unless you are deploying from the same VPC as the registry:
```bash
docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com --username <registry-user>
```
## 2. Build and Push Images
```bash
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260408 \
./scripts/build-and-push-images.sh
```
Image contents:
- `droboticflow-api`: Vite frontend assets + Go API binary
- `droboticflow-worker`: Go worker binary
- `droboticflow-operator-runner`: Python runtime + `backend/operator-runner/main.py`
The script prints the exact image URLs. Reuse those values for deployment:
```bash
API_IMAGE=...
WORKER_IMAGE=...
OPERATOR_RUNNER_IMAGE=...
```
You can also load a named deployment preset:
```bash
DEPLOY_ENV=wjzh \
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow \
IMAGE_TAG=20260421 \
./scripts/build-and-push-images.sh
```
On the `dev` branch, the scripts auto-resolve to `DEPLOY_ENV=poc` when you leave `DEPLOY_ENV` unset.
## 3. Prepare Kubernetes Access
Point `kubectl` at the target cluster before deploying. The deployment scripts do not store kubeconfig in the repository.
Recommended environment variables:
```bash
export K8S_NAMESPACE=droboticflow
export REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_NAMESPACE=dataflow
export IMAGE_TAG=20260408
export API_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:20260408
export WORKER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:20260408
export OPERATOR_RUNNER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:20260408
export OPERATOR_GPU_NODE_SELECTOR=accelerator=nvidia
export OPERATOR_GPU_TOLERATIONS_JSON='[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]'
export OPERATOR_GPU_RUNTIME_CLASS=nvidia
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
export REGISTRY_SERVER=ccr-29eug8s3-pub.cnc.bj.baidubce.com
export REGISTRY_USERNAME=<registry-user>
export REGISTRY_PASSWORD=<registry-password>
export K8S_IMAGE_PULL_SECRET_NAME=droboticflow-registry
export DROBOTICFLOW_API_TOKEN=<optional-shared-api-token>
# Optional global BOS endpoint. Leave unset only when each BOS connection or workflow provides its own endpoint.
# export BOS_ENDPOINT=https://<region>.bcebos.com
# Optional AI generation. Leave all three unset to disable AI generation.
# When AI_API_KEY is set, AI_API_BASE_URL and AI_MODEL are required.
# export AI_API_BASE_URL=https://<openai-compatible-endpoint>/v1
# export AI_MODEL=<model-name>
# export AI_API_KEY=<openai-compatible-server-key>
```
For the `wjzh` environment, the repo now carries non-secret defaults in [`scripts/deploy-envs/wjzh.env`](../scripts/deploy-envs/wjzh.env). Sensitive values stay out of git and can be stored in either:
- `$HOME/.config/droboticflow/deploy/wjzh.secrets.env`
- `.env.wjzh.local`
The loader also looks for `$HOME/.config/droboticflow/kubeconfigs/wjzh.config` and exports `KUBECONFIG` automatically when `DEPLOY_ENV=wjzh` is set.
For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env). It is separated from `wjzh` and defaults to:
- `MONGO_DB=poc`
- `API_PUBLIC_LOAD_BALANCER=true`
- `BOS_ENDPOINT=https://bj.bcebos.com`
- the same namespace, AI defaults, and PVC settings as `wjzh`
The repository does not hardcode registry coordinates for either preset. Provide either all three image references (`API_IMAGE`, `WORKER_IMAGE`, `OPERATOR_RUNNER_IMAGE`) or `REGISTRY_HOST` plus `REGISTRY_NAMESPACE` so build, render, and deploy scripts can derive the image references from `IMAGE_TAG`.
Store `poc` secrets outside git in:
- `$HOME/.config/droboticflow/deploy/poc.secrets.env`
- `.env.poc.local`
At minimum, the `poc` secrets file should provide `REGISTRY_HOST`, `REGISTRY_NAMESPACE` or the three explicit image references, `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `MONGO_URI`, `DROBOTICFLOW_API_TOKEN`, and for first-time workspace setup `WORKSPACE_BOS_BUCKET`, `BOS_ACCESS_KEY_ID`, and `BOS_SECRET_ACCESS_KEY`. Replace every placeholder in `MONGO_URI` before running API, worker, render, or deploy commands; runtime startup rejects missing values, whitespace, and unreplaced markers such as `<host>`.
The loader also checks `$HOME/.config/droboticflow/kubeconfigs/poc.config` automatically. On the `dev` branch and `poc` branches, trying to deploy with another `DEPLOY_ENV` now fails fast to avoid cross-environment mistakes.
For first-time POC cluster setup, configure the shared workspace before deploying the app:
```bash
DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh
```
This installs the BOS CSI node plugin and creates the `droboticflow-workspace` PV/PVC. The BOS PV mount options include `meta_expires=0`, `allow_other`, and `mount_umask=000` so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. `WORKSPACE_BOS_BUCKET` has no repository default; it must come from the local `poc` secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
The POC environment also creates a public `droboticflow-api-public` LoadBalancer service. The standard `droboticflow-api` service defaults to NodePort for compatibility, but can be rendered as ClusterIP with `API_SERVICE_TYPE=ClusterIP` when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When `API_PUBLIC_LOAD_BALANCER=true`, `scripts/deploy-k8s.sh` requires `DROBOTICFLOW_API_TOKEN` and requires `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true`; if the token is omitted locally, the script can reuse an existing `droboticflow-auth-secret` from the target namespace. The current POC LoadBalancer is HTTP-only, so [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env) sets `DROBOTICFLOW_SESSION_COOKIE_SECURE=false` to keep session login usable for browser preview and acceptance automation. Override it to `true` only when POC is served through HTTPS. Boolean aliases such as `yes`, `on`, and `1` are treated as true by both manifest rendering and deployment flow control.
## 4. Deploy
```bash
./scripts/deploy-k8s.sh
```
Or, with the reusable preset:
```bash
DEPLOY_ENV=wjzh ./scripts/deploy-k8s.sh
```
Or for the POC environment:
```bash
DEPLOY_ENV=poc ./scripts/deploy-k8s.sh
```
The deploy script performs these actions:
1. creates the namespace if it does not exist
2. creates or updates the image pull secret
3. loads `MONGO_URI` from the current cluster secret when omitted, then validates that it points to an external MongoDB instance
4. creates or updates the `droboticflow-mongo-secret` Secret with `MONGO_URI`
5. creates or updates the `droboticflow-ai-secret` Secret for server-side AI generation
6. renders `k8s/droboticflow.template.yaml`
7. validates the rendered manifest with `kubectl apply --dry-run=client`
8. applies the manifest to the cluster
9. waits for API and worker rollouts
The main deployment path no longer creates a platform MongoDB Pod or Service. You must provision the backing MongoDB outside this manifest and provide its URI explicitly.
## 5. Services Created
- `droboticflow-api`: NodePort service that exposes the frontend and API
- `droboticflow-worker`: ClusterIP service for worker health/debugging
Default API NodePort:
- `30080`
If the cluster node IP is reachable, the UI and API are exposed at:
```text
http://<node-ip>:30080
```
## 6. Manifest Variables
`scripts/render-manifests.sh` accepts:
- `K8S_NAMESPACE`
- `MONGO_DB`
- `MONGO_CONNECT_RETRIES`
- `MONGO_CONNECT_RETRY_DELAY_MS`
- `WORKER_POLL_INTERVAL_MS`
- `WORKER_API_BASE_URL`
- `API_NODE_PORT`
- `API_REPLICAS`
- `WORKER_REPLICAS`
- `K8S_IMAGE_PULL_SECRET_NAME`
- `DROBOTICFLOW_API_TOKEN`
- `DROBOTICFLOW_REQUIRE_SESSION_AUTH`
- `DROBOTICFLOW_SESSION_COOKIE_SECURE`
- `BOS_ENDPOINT`
- `AI_API_BASE_URL`
- `AI_MODEL`
- `AI_API_KEY` (validation only; the manifest reads the key from `droboticflow-ai-secret`)
- `REGISTRY_HOST`
- `REGISTRY_NAMESPACE`
- `IMAGE_TAG`
- `API_IMAGE`
- `WORKER_IMAGE`
- `OPERATOR_RUNNER_IMAGE`
- `OPERATOR_GPU_NODE_SELECTOR`
- `OPERATOR_GPU_TOLERATIONS_JSON`
- `OPERATOR_GPU_RUNTIME_CLASS`
- `OPERATOR_JOB_TTL_SECONDS`
- `OPERATOR_ALLOW_SHELL_COMMANDS`
- `OPERATOR_COMMAND_ALLOWLIST_REGEX`
- `OPERATOR_COMMAND_DENYLIST_REGEX`
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS`
- `API_SERVICE_TYPE`
## 7. Notes
- The main manifest only deploys the application components. It does not create MongoDB resources in-cluster.
- `MONGO_URI` is required by API and worker startup and must be a concrete `mongodb://` or `mongodb+srv://` URI without whitespace or unreplaced placeholder markers. Kubernetes deployment must reference an external MongoDB instance; localhost, `mongodb:27017`, and in-cluster `.svc` addresses are rejected by `scripts/deploy-k8s.sh`.
- Startup configuration is validated before manifest output and before `scripts/deploy-k8s.sh` performs any cluster writes. `MONGO_CONNECT_RETRIES`, `MONGO_CONNECT_RETRY_DELAY_MS`, and `WORKER_POLL_INTERVAL_MS` must be positive integers; `API_REPLICAS`, `WORKER_REPLICAS`, and `OPERATOR_JOB_TTL_SECONDS` must be non-negative integers; `API_NODE_PORT` must be in `30000..32767` when `API_SERVICE_TYPE=NodePort`; boolean env values must be one of `true/false`, `1/0`, `yes/no`, or `on/off`. Kubernetes namespace, image-pull-secret, workspace PVC, and public LoadBalancer Service names must be lowercase DNS-compatible values; workspace mount paths must be absolute container paths; worker service URLs must be absolute `http://` or `https://` URLs; non-empty BOS and AI endpoint values must also be absolute `http://` or `https://` URLs; if `AI_API_KEY` is configured, both `AI_API_BASE_URL` and `AI_MODEL` are required and there is no provider/model hardcoded fallback; container image references must be non-empty and whitespace-free. `scripts/render-manifests.sh`, `scripts/deploy-k8s.sh`, and `scripts/build-and-push-images.sh` require explicit `API_IMAGE`, `WORKER_IMAGE`, and `OPERATOR_RUNNER_IMAGE` values unless `REGISTRY_HOST` and `REGISTRY_NAMESPACE` are both set, and all three scripts reject malformed image references before Docker or Kubernetes calls. Multi-line ConfigMap block values, including `OPERATOR_GPU_TOLERATIONS_JSON`, `OPERATOR_COMMAND_ALLOWLIST_REGEX`, and `OPERATOR_COMMAND_DENYLIST_REGEX`, are indented before manifest output so formatted JSON and multi-line regex lists remain valid YAML.
- `BOS_ENDPOINT` is rendered into the API and worker ConfigMap as an empty value when unset. Leave it unset only when every BOS scan config or storage connection provides its own endpoint; if both the workflow/connection endpoint and global `BOS_ENDPOINT` are empty, real BOS operations fail with an explicit missing-endpoint error instead of falling back to a hardcoded region.
- API and worker entrypoints are Go binaries with Mongo startup retry, so they tolerate the database becoming ready after the Deployments start.
- The API proxies `/worker-api/*` to `WORKER_API_BASE_URL`, defaulting to the in-cluster `http://droboticflow-worker:3002` Service. Override it only when the worker Service name, namespace, or exposure model differs.
- The worker runs with Kubernetes RBAC that allows creating and watching Jobs in the namespace.
- The operator-runner Job template inherits the private registry pull secret name from `K8S_IMAGE_PULL_SECRET_NAME`.
- Operator-runner Jobs apply CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities and fail before Job creation when malformed. A positive GPU value is emitted as `nvidia.com/gpu` in both requests and limits; the target cluster must have GPU nodes and the NVIDIA device plugin/runtime configured.
- GPU-only scheduling settings are optional: `OPERATOR_GPU_NODE_SELECTOR` accepts comma-separated `key=value` pairs, `OPERATOR_GPU_TOLERATIONS_JSON` accepts a Kubernetes tolerations JSON array, and `OPERATOR_GPU_RUNTIME_CLASS` sets `runtimeClassName`. These settings are only injected into Jobs that request a positive GPU count. Invalid selector or tolerations values fail worker startup.
- Finished operator-runner Jobs receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, default `86400`. Use `npm run k8s:cleanup-jobs` to delete old finished Jobs that predate TTL, or set `DRY_RUN=true` first to list candidates.
- Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Set `OPERATOR_ALLOW_SHELL_COMMANDS=true` only for deployments that need shell operators, and constrain it with `OPERATOR_COMMAND_ALLOWLIST_REGEX` / `OPERATOR_COMMAND_DENYLIST_REGEX` where possible. These values are injected by the worker and are not overridable from workflow hyperparams.
- The operator-runner is not a standalone Deployment. It is launched on demand as a Python Job container.
- Failed operator-runner Jobs report Job conditions, Pod phase, container waiting or termination reasons, Pod events, and available logs back through the run error path.
- Worker runtime metrics are available from `GET /worker-api/metrics` in Prometheus text format. Through the API, stats/metrics and worker control endpoints require an admin/operator session or an API token when session auth is not enforced. The worker Service includes Prometheus scrape annotations for internal collection, and the endpoint includes duration histograms for dispatch loop processing and node execution.
- AI generation runs through the API Deployment. `AI_API_KEY` is mounted through the `droboticflow-ai-secret` Secret and must not be baked into frontend env vars.
- `DROBOTICFLOW_API_TOKEN` is mounted through `droboticflow-auth-secret`. When enabled, non-browser clients call APIs with `Authorization: Bearer <token>` or `X-Droboticflow-API-Token`. Do not bake this shared token into frontend assets.
- `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` enables HttpOnly cookie-session login for browser business APIs. Public LoadBalancer deploys require it so role checks are not bypassed by a shared machine token. Bootstrap remains available through `/api/auth/bootstrap-super-admin` before the first super admin exists.
- `DROBOTICFLOW_SESSION_COOKIE_SECURE=true` sets the session cookie `Secure` attribute. `scripts/deploy-k8s.sh` defaults it to `true` when `API_PUBLIC_LOAD_BALANCER=true`; local development can leave it `false`.
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS` controls which reverse proxies may supply `X-Forwarded-For` / `X-Real-IP` for rate limiting. Leave it empty unless ingress or load balancer source ranges are known.
- Session-authenticated unsafe requests, including logout, require `X-Droboticflow-CSRF-Token` to match the readable `droboticflow_csrf` cookie. The bundled frontend sets this header automatically.
- `IMAGE_TAG` defaults to the current git short SHA in build, deploy, and render scripts only when explicit registry coordinates are configured. Set it explicitly for release names or rollback-friendly tags.
- `scripts/live-workflow-acceptance.sh` requires `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. The `真实数据源` runner infers filesystem/PVC paths as `LOCAL` and `bos://bucket/prefix` paths as `BOS`; if no real source is configured, the source node fails instead of emitting an empty manifest. Set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` for deployed-environment workflow acceptance so missing `BASE_URL` / `API_BASE_URL` fails instead of falling back to localhost. The script can exercise session-authenticated deployments when `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` are set; it logs in, persists cookies, and sends the CSRF header on write requests.
- `scripts/reconcile-system-data.sh` delegates to `scripts/reconcile_system_data.py` and uses the same protected-client behavior. Set `DROBOTICFLOW_API_TOKEN` for token-protected APIs when session auth is not enforced. For public session-auth deployments, set `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` to an organization admin account because the reconcile flow updates settings, users, templates, and operators. Builtin operators are read from the live `/api/operators` catalog and refreshed to `OPERATOR_RUNNER_IMAGE`; the script does not carry a separate builtin catalog. POC acceptance result import is narrower: `/api/acceptance/poc-operator-result` also accepts the shared `DROBOTICFLOW_API_TOKEN` under session auth because it only writes fixed POC acceptance run artifacts, while normal project/workflow/settings writes still require a user session.
- `scripts/object-storage-acceptance.mjs` runs browser create, preview, rename, upload, delete, and optional video playback checks against a live storage connection. It never rewrites storage connection metadata; provide `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, or `OBJECT_STORAGE_BUCKET` explicitly. `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` is required and must point to a non-empty real UTF-8 text or JSON payload used for create/upload/preview checks; set `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE` when auto-detection is not enough. Set `OBJECT_STORAGE_VIDEO_KEY` only when an existing playable object should be validated. `PLAYWRIGHT_IGNORE_HTTPS_ERRORS` must be an explicit boolean spelling when set. The script supports `DROBOTICFLOW_API_TOKEN` for token-protected non-session APIs and `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` for session-authenticated browser access; session-auth object-storage management checks require an organization admin account.
- `scripts/playwright-preview-operators-check.mjs` runs browser/operator preview acceptance with caller-supplied payloads and real source paths. Provide `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_EXTRACT_FIELD`, `PREVIEW_CHECK_JSON_FILTER_FIELD`, `PREVIEW_CHECK_JSON_FIELD_MAP_JSON`, `PREVIEW_CHECK_SCAN_SOURCE_PATH`, `PREVIEW_CHECK_ROSBAG_SOURCE_PATH`, and `PREVIEW_CHECK_LEROBOT_SOURCE_PATH`. The script no longer generates built-in CSV/text/JSON/ROSBag/LeRobot data. Optional `PREVIEW_CHECK_SCAN_MIN_FILES` must be an exact safe positive integer.
- `scripts/xplatform-live-acceptance.py` runs the X-platform chain only when the caller provides the real source path plus explicit acceptance criteria. Provide `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, finite normalized `XPLATFORM_MIN_CLARITY`, `XPLATFORM_MAX_SHAKE`, and `XPLATFORM_MAX_HAND_OCCLUSION`, finite positive `XPLATFORM_CLARITY_SCALE` and `XPLATFORM_FRAME_INTERVAL_SECONDS`, and exact safe positive integer values for `XPLATFORM_SEGMENT_DURATION_SECONDS` and every `XPLATFORM_MIN_*` count; the script has no built-in sampling rule or threshold fallback.
- `npm run acceptance:gpu` creates a temporary GPU Job in the target namespace and runs `nvidia-smi`. Override `GPU_ACCEPTANCE_IMAGE`, `GPU_ACCEPTANCE_TIMEOUT`, `GPU_RESOURCE_NAME`, `GPU_RESOURCE_COUNT`, and the `OPERATOR_GPU_*` scheduling variables when the cluster uses custom GPU images, wait windows, or labels. The script validates namespace, image, timeout, Job name, resource name/count, node selector, tolerations JSON, and runtime class before any `kubectl` call so malformed GPU acceptance inputs fail locally instead of mutating the cluster with a bad manifest. Dynamic manifest strings are rendered as quoted YAML scalars.
- Deployed-environment checks run from a local shell or external CI by invoking the acceptance scripts directly. Configure `WORKFLOW_ACCEPTANCE_SOURCE_PATH` for live workflow acceptance, a kubeconfig for GPU acceptance, and optionally `DROBOTICFLOW_API_TOKEN`, `DROBOTICFLOW_ACCEPTANCE_EMAIL`, `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, `OBJECT_STORAGE_BUCKET`, `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE`, `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE`, `OBJECT_STORAGE_VIDEO_KEY`, `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, the preview JSON field settings, the preview scan/ROSBag/LeRobot source paths, and the X-platform source path plus explicit rules/threshold/count inputs for authenticated acceptance tests.
- If you use DashScope-compatible models, prefer a backend-allowed API key. Do not deploy tool-only Coding Plan keys into the cluster.
- Production hardening still depends on the target cluster policy: TLS ingress, network policies, backup strategy, and secret rotation.