256 lines
24 KiB
Markdown
256 lines
24 KiB
Markdown
# Deployment Guide
|
||
|
||
## Images
|
||
|
||
The core deployment plus the泛宇 Headpose service uses these images:
|
||
|
||
- `droboticflow-api`
|
||
- `droboticflow-worker`
|
||
- `droboticflow-operator-runner`
|
||
- `droboticflow-headpose-service`(digest-pinned 6DRepNet service)
|
||
|
||
Example Baidu registry naming:
|
||
|
||
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:<tag>`
|
||
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:<tag>`
|
||
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:<tag>`
|
||
|
||
## 1. Log In to the Registry
|
||
|
||
Use the public endpoint unless you are deploying from the same VPC as the registry:
|
||
|
||
```bash
|
||
docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com --username <registry-user>
|
||
```
|
||
|
||
## 2. Build and Push Images
|
||
|
||
```bash
|
||
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
||
REGISTRY_NAMESPACE=dataflow \
|
||
IMAGE_TAG=20260408 \
|
||
./scripts/build-and-push-images.sh
|
||
```
|
||
|
||
Image contents:
|
||
|
||
- `droboticflow-api`: Vite frontend assets + Go API binary
|
||
- `droboticflow-worker`: Go worker binary
|
||
- `droboticflow-operator-runner`: Python runtime + `backend/operator-runner/main.py`
|
||
|
||
The script prints the exact image URLs. Reuse those values for deployment:
|
||
|
||
```bash
|
||
API_IMAGE=...
|
||
WORKER_IMAGE=...
|
||
OPERATOR_RUNNER_IMAGE=...
|
||
```
|
||
|
||
You can also load a named deployment preset:
|
||
|
||
```bash
|
||
DEPLOY_ENV=wjzh \
|
||
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
||
REGISTRY_NAMESPACE=dataflow \
|
||
IMAGE_TAG=20260421 \
|
||
./scripts/build-and-push-images.sh
|
||
```
|
||
|
||
On the `dev` branch, the scripts auto-resolve to `DEPLOY_ENV=poc` when you leave `DEPLOY_ENV` unset.
|
||
|
||
## 3. Prepare Kubernetes Access
|
||
|
||
Point `kubectl` at the target cluster before deploying. The deployment scripts do not store kubeconfig in the repository.
|
||
|
||
Recommended environment variables:
|
||
|
||
```bash
|
||
export K8S_NAMESPACE=droboticflow
|
||
export REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com
|
||
export REGISTRY_NAMESPACE=dataflow
|
||
export IMAGE_TAG=20260408
|
||
export API_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:20260408
|
||
export WORKER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:20260408
|
||
export OPERATOR_RUNNER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:20260408
|
||
export HEADPOSE_SERVICE_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-headpose-service:20260408
|
||
export HEADPOSE_SERVICE_IMAGE_ID=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-headpose-service@sha256:<64-hex-digest>
|
||
export HEADPOSE_MODEL_SHA256=sha256:9bdcd0e072df0d21e92288b40ea1d1b3cf7173c44728ab9c45c4d673f361e018
|
||
# Supplier routing is bound when a published workflow is installed into a
|
||
# project. It is deliberately not a deployment-wide environment variable.
|
||
export OPERATOR_GPU_NODE_SELECTOR=accelerator=nvidia
|
||
export OPERATOR_GPU_TOLERATIONS_JSON='[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]'
|
||
export OPERATOR_GPU_RUNTIME_CLASS=nvidia
|
||
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
|
||
export REGISTRY_SERVER=ccr-29eug8s3-pub.cnc.bj.baidubce.com
|
||
export REGISTRY_USERNAME=<registry-user>
|
||
export REGISTRY_PASSWORD=<registry-password>
|
||
export K8S_IMAGE_PULL_SECRET_NAME=droboticflow-registry
|
||
export DROBOTICFLOW_API_TOKEN=<optional-shared-api-token>
|
||
# Optional global BOS endpoint. Leave unset only when each BOS connection or workflow provides its own endpoint.
|
||
# export BOS_ENDPOINT=https://<region>.bcebos.com
|
||
# Optional AI generation. Leave all three unset to disable AI generation.
|
||
# When AI_API_KEY is set, AI_API_BASE_URL and AI_MODEL are required.
|
||
# export AI_API_BASE_URL=https://<openai-compatible-endpoint>/v1
|
||
# export AI_MODEL=<model-name>
|
||
# export AI_API_KEY=<openai-compatible-server-key>
|
||
```
|
||
|
||
For the `wjzh` environment, the repo now carries non-secret defaults in [`scripts/deploy-envs/wjzh.env`](../scripts/deploy-envs/wjzh.env). Sensitive values stay out of git and can be stored in either:
|
||
|
||
- `$HOME/.config/droboticflow/deploy/wjzh.secrets.env`
|
||
- `.env.wjzh.local`
|
||
|
||
The loader also looks for `$HOME/.config/droboticflow/kubeconfigs/wjzh.config` and exports `KUBECONFIG` automatically when `DEPLOY_ENV=wjzh` is set.
|
||
|
||
For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env). It is separated from `wjzh` and defaults to:
|
||
|
||
- `MONGO_DB=poc`
|
||
- `API_PUBLIC_LOAD_BALANCER=true`
|
||
- 已验收 Qwen 节点本地缓存:`DROBOTICFLOW_QWEN_NODE_CACHE_ENABLED=true`、
|
||
`droboticflow-qwen-node-cache` PVC、`/qwen-node-cache` 和 GPU node hostname selector;
|
||
这只影响 allowlisted Qwen 粗切 Job,通用模板仍默认关闭。
|
||
- `BOS_ENDPOINT=https://bj.bcebos.com`
|
||
- the same namespace, AI defaults, and PVC settings as `wjzh`
|
||
|
||
The repository does not hardcode registry coordinates for either preset. Provide either all three image references (`API_IMAGE`, `WORKER_IMAGE`, `OPERATOR_RUNNER_IMAGE`) or `REGISTRY_HOST` plus `REGISTRY_NAMESPACE` so build, render, and deploy scripts can derive the image references from `IMAGE_TAG`.
|
||
|
||
Store `poc` secrets outside git in:
|
||
|
||
- `$HOME/.config/droboticflow/deploy/poc.secrets.env`
|
||
- `.env.poc.local`
|
||
|
||
At minimum, the `poc` secrets file should provide `REGISTRY_HOST`, `REGISTRY_NAMESPACE` or the three explicit image references, `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `MONGO_URI`, `DROBOTICFLOW_API_TOKEN`, and for first-time workspace setup `WORKSPACE_BOS_BUCKET`, `BOS_ACCESS_KEY_ID`, and `BOS_SECRET_ACCESS_KEY`. Replace every placeholder in `MONGO_URI` before running API, worker, render, or deploy commands; runtime startup rejects missing values, whitespace, and unreplaced markers such as `<host>`.
|
||
|
||
The loader also checks `$HOME/.config/droboticflow/kubeconfigs/poc.config` automatically. On the `dev` branch and `poc` branches, trying to deploy with another `DEPLOY_ENV` now fails fast to avoid cross-environment mistakes.
|
||
|
||
For first-time POC cluster setup, configure the shared workspace before deploying the app:
|
||
|
||
```bash
|
||
DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh
|
||
```
|
||
|
||
This installs the BOS CSI node plugin and creates the `droboticflow-workspace` PV/PVC. The BOS PV mount options include `meta_expires=0`, `allow_other`, and `mount_umask=000` so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. `WORKSPACE_BOS_BUCKET` has no repository default; it must come from the local `poc` secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
|
||
|
||
The POC environment also creates a public `droboticflow-api-public` LoadBalancer service. The standard `droboticflow-api` service defaults to NodePort for compatibility, but can be rendered as ClusterIP with `API_SERVICE_TYPE=ClusterIP` when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When `API_PUBLIC_LOAD_BALANCER=true`, `scripts/deploy-k8s.sh` requires `DROBOTICFLOW_API_TOKEN` and requires `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true`; if the token is omitted locally, the script can reuse an existing `droboticflow-auth-secret` from the target namespace. The current POC LoadBalancer is HTTP-only, so [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env) sets `DROBOTICFLOW_SESSION_COOKIE_SECURE=false` to keep session login usable for browser preview and acceptance automation. Override it to `true` only when POC is served through HTTPS. Boolean aliases such as `yes`, `on`, and `1` are treated as true by both manifest rendering and deployment flow control.
|
||
|
||
## 4. Deploy
|
||
|
||
```bash
|
||
./scripts/deploy-k8s.sh
|
||
```
|
||
|
||
Or, with the reusable preset:
|
||
|
||
```bash
|
||
DEPLOY_ENV=wjzh ./scripts/deploy-k8s.sh
|
||
```
|
||
|
||
Or for the POC environment:
|
||
|
||
```bash
|
||
DEPLOY_ENV=poc ./scripts/deploy-k8s.sh
|
||
```
|
||
|
||
The deploy script performs these actions:
|
||
|
||
1. creates the namespace if it does not exist
|
||
2. creates or updates the image pull secret
|
||
3. loads `MONGO_URI` from the current cluster secret when omitted, then validates that it points to an external MongoDB instance
|
||
4. creates or updates the `droboticflow-mongo-secret` Secret with `MONGO_URI`
|
||
5. creates or updates the `droboticflow-ai-secret` Secret for server-side AI generation
|
||
6. renders `k8s/droboticflow.template.yaml`
|
||
7. validates the rendered manifest with `kubectl apply --dry-run=client`
|
||
8. applies the manifest to the cluster
|
||
9. waits for API, worker, and Headpose service rollouts
|
||
|
||
The main deployment path no longer creates a platform MongoDB Pod or Service. You must provision the backing MongoDB outside this manifest and provide its URI explicitly.
|
||
|
||
## 5. Services Created
|
||
|
||
- `droboticflow-api`: NodePort service that exposes the frontend and API
|
||
- `droboticflow-worker`: ClusterIP service for worker health/debugging
|
||
- `droboticflow-headpose`: ClusterIP GPU inference service; its `/health` reports model digest, license, CUDA, and GPU identity
|
||
|
||
Default API NodePort:
|
||
|
||
- `30080`
|
||
|
||
If the cluster node IP is reachable, the UI and API are exposed at:
|
||
|
||
```text
|
||
http://<node-ip>:30080
|
||
```
|
||
|
||
## 6. Manifest Variables
|
||
|
||
`scripts/render-manifests.sh` accepts:
|
||
|
||
- `K8S_NAMESPACE`
|
||
- `MONGO_DB`
|
||
- `MONGO_CONNECT_RETRIES`
|
||
- `MONGO_CONNECT_RETRY_DELAY_MS`
|
||
- `WORKER_POLL_INTERVAL_MS`
|
||
- `WORKER_API_BASE_URL`
|
||
- `API_NODE_PORT`
|
||
- `API_REPLICAS`
|
||
- `WORKER_REPLICAS`
|
||
- `K8S_IMAGE_PULL_SECRET_NAME`
|
||
- `DROBOTICFLOW_API_TOKEN`
|
||
- `DROBOTICFLOW_REQUIRE_SESSION_AUTH`
|
||
- `DROBOTICFLOW_SESSION_COOKIE_SECURE`
|
||
- `BOS_ENDPOINT`
|
||
- `AI_API_BASE_URL`
|
||
- `AI_MODEL`
|
||
- `AI_API_KEY` (validation only; the manifest reads the key from `droboticflow-ai-secret`)
|
||
- `REGISTRY_HOST`
|
||
- `REGISTRY_NAMESPACE`
|
||
- `IMAGE_TAG`
|
||
- `API_IMAGE`
|
||
- `WORKER_IMAGE`
|
||
- `OPERATOR_RUNNER_IMAGE`
|
||
- `OPERATOR_GPU_NODE_SELECTOR`
|
||
- `OPERATOR_GPU_TOLERATIONS_JSON`
|
||
- `OPERATOR_GPU_RUNTIME_CLASS`
|
||
- `OPERATOR_JOB_TTL_SECONDS`
|
||
- `OPERATOR_ALLOW_SHELL_COMMANDS`
|
||
- `OPERATOR_COMMAND_ALLOWLIST_REGEX`
|
||
- `OPERATOR_COMMAND_DENYLIST_REGEX`
|
||
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS`
|
||
- `API_SERVICE_TYPE`
|
||
|
||
## 7. Notes
|
||
|
||
- The main manifest only deploys the application components. It does not create MongoDB resources in-cluster.
|
||
- `MONGO_URI` is required by API and worker startup and must be a concrete `mongodb://` or `mongodb+srv://` URI without whitespace or unreplaced placeholder markers. Kubernetes deployment must reference an external MongoDB instance; localhost, `mongodb:27017`, and in-cluster `.svc` addresses are rejected by `scripts/deploy-k8s.sh`.
|
||
- Startup configuration is validated before manifest output and before `scripts/deploy-k8s.sh` performs any cluster writes. `MONGO_CONNECT_RETRIES`, `MONGO_CONNECT_RETRY_DELAY_MS`, and `WORKER_POLL_INTERVAL_MS` must be positive integers; `API_REPLICAS`, `WORKER_REPLICAS`, and `OPERATOR_JOB_TTL_SECONDS` must be non-negative integers; `API_NODE_PORT` must be in `30000..32767` when `API_SERVICE_TYPE=NodePort`; boolean env values must be one of `true/false`, `1/0`, `yes/no`, or `on/off`. Kubernetes namespace, image-pull-secret, workspace PVC, and public LoadBalancer Service names must be lowercase DNS-compatible values; workspace mount paths must be absolute container paths; worker service URLs must be absolute `http://` or `https://` URLs; non-empty BOS and AI endpoint values must also be absolute `http://` or `https://` URLs; if `AI_API_KEY` is configured, both `AI_API_BASE_URL` and `AI_MODEL` are required and there is no provider/model hardcoded fallback; container image references must be non-empty and whitespace-free. `scripts/render-manifests.sh`, `scripts/deploy-k8s.sh`, and `scripts/build-and-push-images.sh` require explicit `API_IMAGE`, `WORKER_IMAGE`, and `OPERATOR_RUNNER_IMAGE` values unless `REGISTRY_HOST` and `REGISTRY_NAMESPACE` are both set, and all three scripts reject malformed image references before Docker or Kubernetes calls. Multi-line ConfigMap block values, including `OPERATOR_GPU_TOLERATIONS_JSON`, `OPERATOR_COMMAND_ALLOWLIST_REGEX`, and `OPERATOR_COMMAND_DENYLIST_REGEX`, are indented before manifest output so formatted JSON and multi-line regex lists remain valid YAML.
|
||
- `BOS_ENDPOINT` is rendered into the API and worker ConfigMap as an empty value when unset. Leave it unset only when every BOS scan config or storage connection provides its own endpoint; if both the workflow/connection endpoint and global `BOS_ENDPOINT` are empty, real BOS operations fail with an explicit missing-endpoint error instead of falling back to a hardcoded region.
|
||
- API and worker entrypoints are Go binaries with Mongo startup retry, so they tolerate the database becoming ready after the Deployments start.
|
||
- The API proxies `/worker-api/*` to `WORKER_API_BASE_URL`, defaulting to the in-cluster `http://droboticflow-worker:3002` Service. Override it only when the worker Service name, namespace, or exposure model differs.
|
||
- The worker runs with Kubernetes RBAC that allows creating and watching Jobs in the namespace.
|
||
- The operator-runner Job template inherits the private registry pull secret name from `K8S_IMAGE_PULL_SECRET_NAME`.
|
||
- Operator-runner Jobs apply CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities and fail before Job creation when malformed. A positive GPU value is emitted as `nvidia.com/gpu` in both requests and limits; the target cluster must have GPU nodes and the NVIDIA device plugin/runtime configured.
|
||
- GPU-only scheduling settings are optional: `OPERATOR_GPU_NODE_SELECTOR` accepts comma-separated `key=value` pairs, `OPERATOR_GPU_TOLERATIONS_JSON` accepts a Kubernetes tolerations JSON array, and `OPERATOR_GPU_RUNTIME_CLASS` sets `runtimeClassName`. These settings are only injected into Jobs that request a positive GPU count. Invalid selector or tolerations values fail worker startup.
|
||
- Finished operator-runner Jobs receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, default `86400`. Use `npm run k8s:cleanup-jobs` to delete old finished Jobs that predate TTL, or set `DRY_RUN=true` first to list candidates.
|
||
- Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Set `OPERATOR_ALLOW_SHELL_COMMANDS=true` only for deployments that need shell operators, and constrain it with `OPERATOR_COMMAND_ALLOWLIST_REGEX` / `OPERATOR_COMMAND_DENYLIST_REGEX` where possible. These values are injected by the worker and are not overridable from workflow hyperparams.
|
||
- The operator-runner is not a standalone Deployment. It is launched on demand as a Python Job container.
|
||
- Failed operator-runner Jobs report Job conditions, Pod phase, container waiting or termination reasons, Pod events, and available logs back through the run error path.
|
||
- Worker runtime metrics are available from `GET /worker-api/metrics` in Prometheus text format. Through the API, stats/metrics and worker control endpoints require an admin/operator session or an API token when session auth is not enforced. The worker Service includes Prometheus scrape annotations for internal collection, and the endpoint includes duration histograms for dispatch loop processing and node execution.
|
||
- AI generation runs through the API Deployment. `AI_API_KEY` is mounted through the `droboticflow-ai-secret` Secret and must not be baked into frontend env vars.
|
||
- `DROBOTICFLOW_API_TOKEN` is mounted through `droboticflow-auth-secret`. When enabled, non-browser clients call APIs with `Authorization: Bearer <token>` or `X-Droboticflow-API-Token`. When `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true`, this shared token can read run-monitoring APIs for automation (`GET /api/runs`, run logs/resources/artifacts/lineage, and workflow run history), read the narrowed POC readiness gate (`GET /api/acceptance/poc-readiness`), and call explicitly documented import endpoints, but it cannot create projects, update workflows, stop runs, read full dashboard aggregates, or read admin/settings resources. Do not bake this shared token into frontend assets.
|
||
- `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` enables HttpOnly cookie-session login for browser business APIs. Public LoadBalancer deploys require it so role checks are not bypassed by a shared machine token. Bootstrap remains available through `/api/auth/bootstrap-super-admin` before the first super admin exists.
|
||
- `DROBOTICFLOW_SESSION_COOKIE_SECURE=true` sets the session cookie `Secure` attribute. `scripts/deploy-k8s.sh` defaults it to `true` when `API_PUBLIC_LOAD_BALANCER=true`; local development can leave it `false`.
|
||
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS` controls which reverse proxies may supply `X-Forwarded-For` / `X-Real-IP` for rate limiting. Leave it empty unless ingress or load balancer source ranges are known.
|
||
- Session-authenticated unsafe requests, including logout, require `X-Droboticflow-CSRF-Token` to match the readable `droboticflow_csrf` cookie. The bundled frontend sets this header automatically.
|
||
- `IMAGE_TAG` defaults to the current git short SHA in build, deploy, and render scripts only when explicit registry coordinates are configured. Set it explicitly for release names or rollback-friendly tags.
|
||
- `scripts/live-workflow-acceptance.sh` requires `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. The `真实数据源` runner infers filesystem/PVC paths as `LOCAL` and `bos://bucket/prefix` paths as `BOS`; if no real source is configured, the source node fails instead of emitting an empty manifest. Set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` for deployed-environment workflow acceptance so missing `BASE_URL` / `API_BASE_URL` fails instead of falling back to localhost. The script can exercise session-authenticated deployments when `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` are set; it logs in, persists cookies, and sends the CSRF header on write requests.
|
||
- `scripts/reconcile-system-data.sh` delegates to `scripts/reconcile_system_data.py` and uses the same protected-client behavior. Set `DROBOTICFLOW_API_TOKEN` for token-protected APIs when session auth is not enforced. For public session-auth deployments, set `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` to an organization admin account because the reconcile flow updates settings, users, templates, and operators. Builtin operators are read from the live `/api/operators` catalog and refreshed to the digest-pinned CCR image declared in `operator-repos/builtin/<slug>/release.json` for the target environment; registry lookup only verifies that the committed digest still exists. POC acceptance routes are narrower: `/api/acceptance/poc-operator-result` accepts the shared `DROBOTICFLOW_API_TOKEN` under session auth because it only writes fixed POC acceptance run artifacts, and `/api/acceptance/poc-gpu-workflow` only creates a bounded GPU-preflight workflow/run under the configured workspace mount. Normal project/workflow/settings writes still require a user session.
|
||
- `scripts/object-storage-acceptance.mjs` runs browser create, preview, rename, upload, delete, and optional video playback checks against a live storage connection. It never rewrites storage connection metadata; provide `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, or `OBJECT_STORAGE_BUCKET` explicitly. `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` is required and must point to a non-empty real UTF-8 text or JSON payload used for create/upload/preview checks; set `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE` when auto-detection is not enough. Set `OBJECT_STORAGE_VIDEO_KEY` only when an existing playable object should be validated. `PLAYWRIGHT_IGNORE_HTTPS_ERRORS` must be an explicit boolean spelling when set. The script supports `DROBOTICFLOW_API_TOKEN` for token-protected non-session APIs and `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` for session-authenticated browser access; session-auth object-storage management checks require an organization admin account.
|
||
- `scripts/playwright-preview-operators-check.mjs` runs browser/operator preview acceptance with caller-supplied payloads and real source paths. Provide `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_EXTRACT_FIELD`, `PREVIEW_CHECK_JSON_FILTER_FIELD`, `PREVIEW_CHECK_JSON_FIELD_MAP_JSON`, `PREVIEW_CHECK_SCAN_SOURCE_PATH`, `PREVIEW_CHECK_ROSBAG_SOURCE_PATH`, and `PREVIEW_CHECK_LEROBOT_SOURCE_PATH`. The script no longer generates built-in CSV/text/JSON/ROSBag/LeRobot data. Optional `PREVIEW_CHECK_SCAN_MIN_FILES` must be an exact safe positive integer.
|
||
- `scripts/xplatform-live-acceptance.py` runs the X-platform chain only when the caller provides the real source path plus explicit acceptance criteria. Provide `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, finite normalized `XPLATFORM_MIN_CLARITY`, `XPLATFORM_MAX_SHAKE`, and `XPLATFORM_MAX_HAND_OCCLUSION`, finite positive `XPLATFORM_CLARITY_SCALE` and `XPLATFORM_FRAME_INTERVAL_SECONDS`, and exact safe positive integer values for `XPLATFORM_SEGMENT_DURATION_SECONDS` and every `XPLATFORM_MIN_*` count; the script has no built-in sampling rule or threshold fallback.
|
||
- `npm run acceptance:gpu` loads the same deploy environment resolver as the POC deploy scripts, so `dev` resolves to the POC kubeconfig when `DEPLOY_ENV` is unset. It then creates a temporary GPU Job in the target namespace and runs `nvidia-smi`. Override `GPU_ACCEPTANCE_IMAGE`, `GPU_ACCEPTANCE_TIMEOUT`, `GPU_RESOURCE_NAME`, `GPU_RESOURCE_COUNT`, and the `OPERATOR_GPU_*` scheduling variables when the cluster uses custom GPU images, wait windows, or labels. The script validates namespace, image, timeout, Job name, resource name/count, node selector, tolerations JSON, and runtime class before any `kubectl` call so malformed GPU acceptance inputs fail locally instead of mutating the cluster with a bad manifest. It also requires a selector-matched, non-unschedulable node with at least the requested GPU count before creating the Job; if none exists, the script exits with checked node/resource/selector/unschedulable evidence and the environment must be treated as GPU-not-accepted. Dynamic manifest strings are rendered as quoted YAML scalars.
|
||
- `npm run acceptance:poc-gpu-capacity` starts a temporary in-cluster Pod with the deployed operator-runner image and calls the deployed worker service at `/worker-api/capacity`. It is the no-login POC capacity acceptance check: the default expectation is CPU-only POC, and `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` should be used after real GPU hardware is installed.
|
||
- `npm run acceptance:poc-gpu-workflow` creates a real POC run with the source node requesting `gpu=1`. With `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, it exercises the normal business project/workflow APIs. Without session credentials, it uses `DROBOTICFLOW_API_TOKEN` from the environment or reads it from `droboticflow-auth-secret`, calls the bounded `/api/acceptance/poc-gpu-workflow` route, port-forwards `service/droboticflow-worker`, triggers `/worker-api/process-once`, and then reads the run/logs through token-allowed run read APIs. CPU-only mode expects `gpu request cannot be scheduled`; GPU-available mode is enabled with `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true` or `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` and expects a completed run with artifacts. The bounded route reuses one fixed POC GPU acceptance project/workflow and creates a fresh run for each invocation, so repeated post-deploy checks preserve audit evidence without creating duplicate projects. Override `POC_GPU_WORKFLOW_SOURCE_PATH` only when the default real fixture path is not present; the token route rejects source paths outside the configured workspace mount. The latest result is surfaced in POC readiness as `gpuWorkflowPreflight`, while the separate `gpu` gate continues to represent real hardware capacity. `npm run acceptance:poc-readiness` now consumes the same bounded readiness API by default, so run the GPU workflow check once after deployment before treating readiness output as current.
|
||
- Deployed-environment checks run from a local shell or external CI by invoking the acceptance scripts directly. Configure `WORKFLOW_ACCEPTANCE_SOURCE_PATH` for live workflow acceptance, a kubeconfig for GPU acceptance, and optionally `DROBOTICFLOW_API_TOKEN`, `DROBOTICFLOW_ACCEPTANCE_EMAIL`, `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, `OBJECT_STORAGE_BUCKET`, `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE`, `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE`, `OBJECT_STORAGE_VIDEO_KEY`, `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, the preview JSON field settings, the preview scan/ROSBag/LeRobot source paths, and the X-platform source path plus explicit rules/threshold/count inputs for authenticated acceptance tests.
|
||
- If you use DashScope-compatible models, prefer a backend-allowed API key. Do not deploy tool-only Coding Plan keys into the cluster.
|
||
- Production hardening still depends on the target cluster policy: TLS ingress, network policies, backup strategy, and secret rotation.
|