forked from longtao.wu/droboticflow
246 lines
24 KiB
Markdown
246 lines
24 KiB
Markdown
# Deployment Guide
|
|
|
|
Current POC contract and rollout prerequisites: [cutover record](../doc/兼容代码清理记录.md). Historical compatibility paths listed there are retired.
|
|
## Images
|
|
|
|
This project deploys three images:
|
|
|
|
- `droboticflow-api`
|
|
- `droboticflow-worker`
|
|
- `droboticflow-operator-runner`
|
|
|
|
Example Baidu registry naming:
|
|
|
|
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:<tag>`
|
|
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:<tag>`
|
|
- `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:<tag>`
|
|
|
|
## 1. Log In to the Registry
|
|
|
|
Use the public endpoint unless you are deploying from the same VPC as the registry:
|
|
|
|
```bash
|
|
docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com --username <registry-user>
|
|
```
|
|
|
|
## 2. Build and Push Images
|
|
|
|
```bash
|
|
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
|
REGISTRY_NAMESPACE=dataflow \
|
|
IMAGE_TAG=20260408 \
|
|
./scripts/build-and-push-images.sh
|
|
```
|
|
|
|
Image contents:
|
|
|
|
- `droboticflow-api`: Vite frontend assets + Go API binary
|
|
- `droboticflow-worker`: Go worker binary
|
|
- `droboticflow-operator-runner`: Python runtime + `backend/operator-runner/main.py`
|
|
|
|
The script prints the exact image URLs. Reuse those values for deployment:
|
|
|
|
```bash
|
|
API_IMAGE=...
|
|
WORKER_IMAGE=...
|
|
OPERATOR_RUNNER_IMAGE=...
|
|
```
|
|
|
|
You can also load a named deployment preset:
|
|
|
|
```bash
|
|
DEPLOY_ENV=wjzh \
|
|
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
|
REGISTRY_NAMESPACE=dataflow \
|
|
IMAGE_TAG=20260421 \
|
|
./scripts/build-and-push-images.sh
|
|
```
|
|
|
|
On the `dev` branch, the scripts auto-resolve to `DEPLOY_ENV=poc` when you leave `DEPLOY_ENV` unset.
|
|
|
|
## 3. Prepare Kubernetes Access
|
|
|
|
Point `kubectl` at the target cluster before deploying. The deployment scripts do not store kubeconfig in the repository.
|
|
|
|
Recommended environment variables:
|
|
|
|
```bash
|
|
export K8S_NAMESPACE=droboticflow
|
|
export REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com
|
|
export REGISTRY_NAMESPACE=dataflow
|
|
export IMAGE_TAG=20260408
|
|
export API_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-api:20260408
|
|
export WORKER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-worker:20260408
|
|
export OPERATOR_RUNNER_IMAGE=ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow/droboticflow-operator-runner:20260408
|
|
export OPERATOR_GPU_NODE_SELECTOR=accelerator=nvidia
|
|
export OPERATOR_GPU_TOLERATIONS_JSON='[{"key":"nvidia.com/gpu","operator":"Exists","effect":"NoSchedule"}]'
|
|
export OPERATOR_GPU_RUNTIME_CLASS=nvidia
|
|
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
|
|
export REGISTRY_SERVER=ccr-29eug8s3-pub.cnc.bj.baidubce.com
|
|
export REGISTRY_USERNAME=<registry-user>
|
|
export REGISTRY_PASSWORD=<registry-password>
|
|
export K8S_IMAGE_PULL_SECRET_NAME=droboticflow-registry
|
|
export DROBOTICFLOW_API_TOKEN=<optional-shared-api-token>
|
|
# Optional global BOS endpoint. Leave unset only when each BOS connection or workflow provides its own endpoint.
|
|
# export BOS_ENDPOINT=https://<region>.bcebos.com
|
|
# Optional AI generation. Leave all three unset to disable AI generation.
|
|
# When AI_API_KEY is set, AI_API_BASE_URL and AI_MODEL are required.
|
|
# export AI_API_BASE_URL=https://<openai-compatible-endpoint>/v1
|
|
# export AI_MODEL=<model-name>
|
|
# export AI_API_KEY=<openai-compatible-server-key>
|
|
```
|
|
|
|
For the `wjzh` environment, the repo now carries non-secret defaults in [`scripts/deploy-envs/wjzh.env`](../scripts/deploy-envs/wjzh.env). Sensitive values stay out of git and can be stored in either:
|
|
|
|
- `$HOME/.config/droboticflow/deploy/wjzh.secrets.env`
|
|
- `.env.wjzh.local`
|
|
|
|
The loader also looks for `$HOME/.config/droboticflow/kubeconfigs/wjzh.config` and exports `KUBECONFIG` automatically when `DEPLOY_ENV=wjzh` is set.
|
|
|
|
For the `poc` environment, use [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env). It is separated from `wjzh` and defaults to:
|
|
|
|
- `MONGO_DB=poc`
|
|
- `API_PUBLIC_LOAD_BALANCER=true`
|
|
- `BOS_ENDPOINT=https://bj.bcebos.com`
|
|
- the same namespace, AI defaults, and PVC settings as `wjzh`
|
|
|
|
The repository does not hardcode registry coordinates for either preset. Provide either all three image references (`API_IMAGE`, `WORKER_IMAGE`, `OPERATOR_RUNNER_IMAGE`) or `REGISTRY_HOST` plus `REGISTRY_NAMESPACE` so build, render, and deploy scripts can derive the image references from `IMAGE_TAG`.
|
|
|
|
Store `poc` secrets outside git in:
|
|
|
|
- `$HOME/.config/droboticflow/deploy/poc.secrets.env`
|
|
- `.env.poc.local`
|
|
|
|
At minimum, the `poc` secrets file should provide `REGISTRY_HOST`, `REGISTRY_NAMESPACE` or the three explicit image references, `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `MONGO_URI`, `DROBOTICFLOW_API_TOKEN`, and for first-time workspace setup `WORKSPACE_BOS_BUCKET`, `BOS_ACCESS_KEY_ID`, and `BOS_SECRET_ACCESS_KEY`. Replace every placeholder in `MONGO_URI` before running API, worker, render, or deploy commands; runtime startup rejects missing values, whitespace, and unreplaced markers such as `<host>`.
|
|
|
|
The loader also checks `$HOME/.config/droboticflow/kubeconfigs/poc.config` automatically. On the `dev` branch and `poc` branches, trying to deploy with another `DEPLOY_ENV` now fails fast to avoid cross-environment mistakes.
|
|
|
|
For first-time POC cluster setup, configure the shared workspace before deploying the app:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh
|
|
```
|
|
|
|
This installs the BOS CSI node plugin and creates the `droboticflow-workspace` PV/PVC. The BOS PV mount options include `meta_expires=0`, `allow_other`, and `mount_umask=000` so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. `WORKSPACE_BOS_BUCKET` has no repository default; it must come from the local `poc` secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
|
|
|
|
The POC environment also creates a public `droboticflow-api-public` LoadBalancer service. The standard `droboticflow-api` service defaults to NodePort for compatibility, but can be rendered as ClusterIP with `API_SERVICE_TYPE=ClusterIP` when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When `API_PUBLIC_LOAD_BALANCER=true`, `scripts/deploy-k8s.sh` requires `DROBOTICFLOW_API_TOKEN` and always uses session authentication for browser business APIs; if the token is omitted locally, the script can reuse an existing `droboticflow-auth-secret` from the target namespace. The current POC LoadBalancer is HTTP-only, so [`scripts/deploy-envs/poc.env`](../scripts/deploy-envs/poc.env) sets `DROBOTICFLOW_SESSION_COOKIE_SECURE=false` to keep session login usable for browser preview and acceptance automation. Override it to `true` only when POC is served through HTTPS. Boolean aliases such as `yes`, `on`, and `1` are treated as true by both manifest rendering and deployment flow control.
|
|
|
|
## 4. Deploy
|
|
|
|
```bash
|
|
./scripts/deploy-k8s.sh
|
|
```
|
|
|
|
Or, with the reusable preset:
|
|
|
|
```bash
|
|
DEPLOY_ENV=wjzh ./scripts/deploy-k8s.sh
|
|
```
|
|
|
|
Or for the POC environment:
|
|
|
|
```bash
|
|
DEPLOY_ENV=poc ./scripts/deploy-k8s.sh
|
|
```
|
|
|
|
The deploy script performs these actions:
|
|
|
|
1. creates the namespace if it does not exist
|
|
2. creates or updates the image pull secret
|
|
3. loads `MONGO_URI` from the current cluster secret when omitted, then validates that it points to an external MongoDB instance
|
|
4. creates or updates the `droboticflow-mongo-secret` Secret with `MONGO_URI`
|
|
5. creates or updates the `droboticflow-ai-secret` Secret for server-side AI generation
|
|
6. renders `k8s/droboticflow.template.yaml`
|
|
7. validates the rendered manifest with `kubectl apply --dry-run=client`
|
|
8. applies the manifest to the cluster
|
|
9. waits for API and worker rollouts
|
|
|
|
The main deployment path no longer creates a platform MongoDB Pod or Service. You must provision the backing MongoDB outside this manifest and provide its URI explicitly.
|
|
|
|
## 5. Services Created
|
|
|
|
- `droboticflow-api`: NodePort service that exposes the frontend and API
|
|
- `droboticflow-worker`: ClusterIP service for worker health/debugging
|
|
|
|
Default API NodePort:
|
|
|
|
- `30080`
|
|
|
|
If the cluster node IP is reachable, the UI and API are exposed at:
|
|
|
|
```text
|
|
http://<node-ip>:30080
|
|
```
|
|
|
|
## 6. Manifest Variables
|
|
|
|
`scripts/render-manifests.sh` accepts:
|
|
|
|
- `K8S_NAMESPACE`
|
|
- `MONGO_DB`
|
|
- `MONGO_CONNECT_RETRIES`
|
|
- `MONGO_CONNECT_RETRY_DELAY_MS`
|
|
- `WORKER_POLL_INTERVAL_MS`
|
|
- `WORKER_API_BASE_URL`
|
|
- `API_NODE_PORT`
|
|
- `API_REPLICAS`
|
|
- `WORKER_REPLICAS`
|
|
- `K8S_IMAGE_PULL_SECRET_NAME`
|
|
- `DROBOTICFLOW_API_TOKEN`
|
|
- `DROBOTICFLOW_SESSION_COOKIE_SECURE`
|
|
- `BOS_ENDPOINT`
|
|
- `AI_API_BASE_URL`
|
|
- `AI_MODEL`
|
|
- `AI_API_KEY` (validation only; the manifest reads the key from `droboticflow-ai-secret`)
|
|
- `REGISTRY_HOST`
|
|
- `REGISTRY_NAMESPACE`
|
|
- `IMAGE_TAG`
|
|
- `API_IMAGE`
|
|
- `WORKER_IMAGE`
|
|
- `OPERATOR_RUNNER_IMAGE`
|
|
- `OPERATOR_GPU_NODE_SELECTOR`
|
|
- `OPERATOR_GPU_TOLERATIONS_JSON`
|
|
- `OPERATOR_GPU_RUNTIME_CLASS`
|
|
- `OPERATOR_JOB_TTL_SECONDS`
|
|
- `OPERATOR_ALLOW_SHELL_COMMANDS`
|
|
- `OPERATOR_COMMAND_ALLOWLIST_REGEX`
|
|
- `OPERATOR_COMMAND_DENYLIST_REGEX`
|
|
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS`
|
|
- `API_SERVICE_TYPE`
|
|
|
|
## 7. Notes
|
|
|
|
- The main manifest only deploys the application components. It does not create MongoDB resources in-cluster.
|
|
- `MONGO_URI` is required by API and worker startup and must be a concrete `mongodb://` or `mongodb+srv://` URI without whitespace or unreplaced placeholder markers. Kubernetes deployment must reference an external MongoDB instance; localhost, `mongodb:27017`, and in-cluster `.svc` addresses are rejected by `scripts/deploy-k8s.sh`.
|
|
- Startup configuration is validated before manifest output and before `scripts/deploy-k8s.sh` performs any cluster writes. `MONGO_CONNECT_RETRIES`, `MONGO_CONNECT_RETRY_DELAY_MS`, and `WORKER_POLL_INTERVAL_MS` must be positive integers; `API_REPLICAS`, `WORKER_REPLICAS`, and `OPERATOR_JOB_TTL_SECONDS` must be non-negative integers; `API_NODE_PORT` must be in `30000..32767` when `API_SERVICE_TYPE=NodePort`; boolean env values must be one of `true/false`, `1/0`, `yes/no`, or `on/off`. Kubernetes namespace, image-pull-secret, workspace PVC, and public LoadBalancer Service names must be lowercase DNS-compatible values; workspace mount paths must be absolute container paths; worker service URLs must be absolute `http://` or `https://` URLs; non-empty BOS and AI endpoint values must also be absolute `http://` or `https://` URLs; if `AI_API_KEY` is configured, both `AI_API_BASE_URL` and `AI_MODEL` are required and there is no provider/model hardcoded fallback; container image references must be non-empty and whitespace-free. `scripts/render-manifests.sh`, `scripts/deploy-k8s.sh`, and `scripts/build-and-push-images.sh` require explicit `API_IMAGE`, `WORKER_IMAGE`, and `OPERATOR_RUNNER_IMAGE` values unless `REGISTRY_HOST` and `REGISTRY_NAMESPACE` are both set, and all three scripts reject malformed image references before Docker or Kubernetes calls. Multi-line ConfigMap block values, including `OPERATOR_GPU_TOLERATIONS_JSON`, `OPERATOR_COMMAND_ALLOWLIST_REGEX`, and `OPERATOR_COMMAND_DENYLIST_REGEX`, are indented before manifest output so formatted JSON and multi-line regex lists remain valid YAML.
|
|
- `BOS_ENDPOINT` is rendered into the API and worker ConfigMap as an empty value when unset. Leave it unset only when every BOS scan config or storage connection provides its own endpoint; if both the workflow/connection endpoint and global `BOS_ENDPOINT` are empty, real BOS operations fail with an explicit missing-endpoint error instead of falling back to a hardcoded region.
|
|
- API and worker entrypoints are Go binaries with Mongo startup retry, so they tolerate the database becoming ready after the Deployments start.
|
|
- The API proxies `/worker-api/*` to `WORKER_API_BASE_URL`, defaulting to the in-cluster `http://droboticflow-worker:3002` Service. Override it only when the worker Service name, namespace, or exposure model differs.
|
|
- The worker runs with Kubernetes RBAC that allows creating and watching Jobs in the namespace.
|
|
- The operator-runner Job template inherits the private registry pull secret name from `K8S_IMAGE_PULL_SECRET_NAME`.
|
|
- Operator-runner Jobs apply CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities and fail before Job creation when malformed. A positive GPU value is emitted as `nvidia.com/gpu` in both requests and limits; the target cluster must have GPU nodes and the NVIDIA device plugin/runtime configured.
|
|
- GPU-only scheduling settings are optional: `OPERATOR_GPU_NODE_SELECTOR` accepts comma-separated `key=value` pairs, `OPERATOR_GPU_TOLERATIONS_JSON` accepts a Kubernetes tolerations JSON array, and `OPERATOR_GPU_RUNTIME_CLASS` sets `runtimeClassName`. These settings are only injected into Jobs that request a positive GPU count. Invalid selector or tolerations values fail worker startup.
|
|
- Finished operator-runner Jobs receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, default `86400`. Use `npm run k8s:cleanup-jobs` to delete old finished Jobs that predate TTL, or set `DRY_RUN=true` first to list candidates.
|
|
- Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Set `OPERATOR_ALLOW_SHELL_COMMANDS=true` only for deployments that need shell operators, and constrain it with `OPERATOR_COMMAND_ALLOWLIST_REGEX` / `OPERATOR_COMMAND_DENYLIST_REGEX` where possible. These values are injected by the worker and are not overridable from workflow hyperparams.
|
|
- The operator-runner is not a standalone Deployment. It is launched on demand as a Python Job container.
|
|
- Failed operator-runner Jobs report Job conditions, Pod phase, container waiting or termination reasons, Pod events, and available logs back through the run error path.
|
|
- Worker runtime metrics are available from `GET /worker-api/metrics` in Prometheus text format. Through the API, stats/metrics and worker control endpoints require an admin, pipeline_engineer, or data_operator session. The worker Service includes Prometheus scrape annotations for internal collection, and the endpoint includes duration histograms for dispatch loop processing and node execution.
|
|
- AI generation runs through the API Deployment. `AI_API_KEY` is mounted through the `droboticflow-ai-secret` Secret and must not be baked into frontend env vars.
|
|
- `DROBOTICFLOW_API_TOKEN` is mounted through `droboticflow-auth-secret`. When enabled, non-browser clients call APIs with `Authorization: Bearer <token>` or `X-Droboticflow-API-Token`. Browser session authentication is mandatory; this shared token can read run-monitoring APIs for automation (`GET /api/runs`, run logs/resources/artifacts/lineage, and workflow run history), read the narrowed POC readiness gate (`GET /api/acceptance/poc-readiness`), and call explicitly documented import endpoints, but it cannot create projects, update workflows, stop runs, read full dashboard aggregates, or read admin/settings resources. Do not bake this shared token into frontend assets.
|
|
- HttpOnly cookie-session login is mandatory for browser business APIs. Shared machine tokens cannot bypass business role checks. Bootstrap remains available through `/api/auth/bootstrap-super-admin` before the first super admin exists.
|
|
- `DROBOTICFLOW_SESSION_COOKIE_SECURE=true` sets the session cookie `Secure` attribute. `scripts/deploy-k8s.sh` defaults it to `true` when `API_PUBLIC_LOAD_BALANCER=true`; local development can leave it `false`.
|
|
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS` controls which reverse proxies may supply `X-Forwarded-For` / `X-Real-IP` for rate limiting. Leave it empty unless ingress or load balancer source ranges are known.
|
|
- Session-authenticated unsafe requests, including logout, require `X-Droboticflow-CSRF-Token` to match the readable `droboticflow_csrf` cookie. The bundled frontend sets this header automatically.
|
|
- `IMAGE_TAG` defaults to the current git short SHA in build, deploy, and render scripts only when explicit registry coordinates are configured. Set it explicitly for release names or rollback-friendly tags.
|
|
- `scripts/live-workflow-acceptance.sh` requires `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. The `真实数据源` runner infers filesystem/PVC paths as `LOCAL` and `bos://bucket/prefix` paths as `BOS`; if no real source is configured, the source node fails instead of emitting an empty manifest. Set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` for deployed-environment workflow acceptance so missing `BASE_URL` / `API_BASE_URL` fails instead of falling back to localhost. The script can exercise session-authenticated deployments when `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` are set; it logs in, persists cookies, and sends the CSRF header on write requests.
|
|
- `scripts/reconcile-system-data.sh` delegates to `scripts/reconcile_system_data.py` and uses the same protected-client behavior. Set `DROBOTICFLOW_API_TOKEN` for token-protected APIs for explicitly authorized internal machine APIs. For public session-auth deployments, set `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD` to an organization admin account because the reconcile flow updates settings, users, templates, and operators. Builtin operators are read from the live `/api/operators` catalog and refreshed to the digest-pinned CCR image declared in `operator-repos/builtin/<slug>/release.json` for the target environment; registry lookup only verifies that the committed digest still exists. POC acceptance routes are narrower: `/api/acceptance/poc-operator-result` accepts the shared `DROBOTICFLOW_API_TOKEN` under session auth because it only writes fixed POC acceptance run artifacts, and `/api/acceptance/poc-gpu-workflow` only creates a bounded GPU-preflight workflow/run under the configured workspace mount. Normal project/workflow/settings writes still require a user session.
|
|
- `scripts/object-storage-acceptance.mjs` runs browser create, preview, rename, upload, delete, and optional video playback checks against a live storage connection. It never rewrites storage connection metadata; provide `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, or `OBJECT_STORAGE_BUCKET` explicitly. `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` is required and must point to a non-empty real UTF-8 text or JSON payload used for create/upload/preview checks; set `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE` when auto-detection is not enough. Set `OBJECT_STORAGE_VIDEO_KEY` only when an existing playable object should be validated. `PLAYWRIGHT_IGNORE_HTTPS_ERRORS` must be an explicit boolean spelling when set. The script supports `DROBOTICFLOW_API_TOKEN` for token-protected non-session APIs and `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` for session-authenticated browser access; session-auth object-storage management checks require an organization admin account.
|
|
- `scripts/playwright-preview-operators-check.mjs` runs browser/operator preview acceptance with caller-supplied payloads and real source paths. Provide `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_EXTRACT_FIELD`, `PREVIEW_CHECK_JSON_FILTER_FIELD`, `PREVIEW_CHECK_JSON_FIELD_MAP_JSON`, `PREVIEW_CHECK_SCAN_SOURCE_PATH`, `PREVIEW_CHECK_ROSBAG_SOURCE_PATH`, and `PREVIEW_CHECK_LEROBOT_SOURCE_PATH`. The script no longer generates built-in CSV/text/JSON/ROSBag/LeRobot data. Optional `PREVIEW_CHECK_SCAN_MIN_FILES` must be an exact safe positive integer.
|
|
- `scripts/xplatform-live-acceptance.py` runs the X-platform chain only when the caller provides the real source path plus explicit acceptance criteria. Provide `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, finite normalized `XPLATFORM_MIN_CLARITY`, `XPLATFORM_MAX_SHAKE`, and `XPLATFORM_MAX_HAND_OCCLUSION`, finite positive `XPLATFORM_CLARITY_SCALE` and `XPLATFORM_FRAME_INTERVAL_SECONDS`, and exact safe positive integer values for `XPLATFORM_SEGMENT_DURATION_SECONDS` and every `XPLATFORM_MIN_*` count; the script has no built-in sampling rule or threshold fallback.
|
|
- `npm run acceptance:gpu` loads the same deploy environment resolver as the POC deploy scripts, so `dev` resolves to the POC kubeconfig when `DEPLOY_ENV` is unset. It then creates a temporary GPU Job in the target namespace and runs `nvidia-smi`. Override `GPU_ACCEPTANCE_IMAGE`, `GPU_ACCEPTANCE_TIMEOUT`, `GPU_RESOURCE_NAME`, `GPU_RESOURCE_COUNT`, and the `OPERATOR_GPU_*` scheduling variables when the cluster uses custom GPU images, wait windows, or labels. The script validates namespace, image, timeout, Job name, resource name/count, node selector, tolerations JSON, and runtime class before any `kubectl` call so malformed GPU acceptance inputs fail locally instead of mutating the cluster with a bad manifest. It also requires a selector-matched, non-unschedulable node with at least the requested GPU count before creating the Job; if none exists, the script exits with checked node/resource/selector/unschedulable evidence and the environment must be treated as GPU-not-accepted. Dynamic manifest strings are rendered as quoted YAML scalars.
|
|
- `npm run acceptance:poc-gpu-capacity` starts a temporary in-cluster Pod with the deployed operator-runner image and calls the deployed worker service at `/worker-api/capacity`. It is the no-login POC capacity acceptance check: the default expectation is CPU-only POC, and `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` should be used after real GPU hardware is installed.
|
|
- `npm run acceptance:poc-gpu-workflow` creates a real POC run with the source node requesting `gpu=1`. With `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, it exercises the normal business project/workflow APIs. Without session credentials, it uses `DROBOTICFLOW_API_TOKEN` from the environment or reads it from `droboticflow-auth-secret`, calls the bounded `/api/acceptance/poc-gpu-workflow` route, port-forwards `service/droboticflow-worker`, triggers `/worker-api/process-once`, and then reads the run/logs through token-allowed run read APIs. CPU-only mode expects `gpu request cannot be scheduled`; GPU-available mode is enabled with `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true` or `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` and expects a completed run with artifacts. The bounded route reuses one fixed POC GPU acceptance project/workflow and creates a fresh run for each invocation, so repeated post-deploy checks preserve audit evidence without creating duplicate projects. Override `POC_GPU_WORKFLOW_SOURCE_PATH` only when the default real fixture path is not present; the token route rejects source paths outside the configured workspace mount. The latest result is surfaced in POC readiness as `gpuWorkflowPreflight`, while the separate `gpu` gate continues to represent real hardware capacity. `npm run acceptance:poc-readiness` now consumes the same bounded readiness API by default, so run the GPU workflow check once after deployment before treating readiness output as current.
|
|
- Deployed-environment checks run from a local shell or external CI by invoking the acceptance scripts directly. Configure `WORKFLOW_ACCEPTANCE_SOURCE_PATH` for live workflow acceptance, a kubeconfig for GPU acceptance, and optionally `DROBOTICFLOW_API_TOKEN`, `DROBOTICFLOW_ACCEPTANCE_EMAIL`, `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, `OBJECT_STORAGE_BUCKET`, `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE`, `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE`, `OBJECT_STORAGE_VIDEO_KEY`, `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, the preview JSON field settings, the preview scan/ROSBag/LeRobot source paths, and the X-platform source path plus explicit rules/threshold/count inputs for authenticated acceptance tests.
|
|
- If you use DashScope-compatible models, prefer a backend-allowed API key. Do not deploy tool-only Coding Plan keys into the cluster.
|
|
- Production hardening still depends on the target cluster policy: TLS ingress, network policies, backup strategy, and secret rotation.
|