# Operations Guide ## Service Responsibilities ### API - validates incoming payloads - persists projects, workflows, operators, templates, settings, runs, and audit tasks - serves the built SPA in production - runs as a single Go binary ### Worker - polls MongoDB for queued runs - auto-enqueues streaming workflows based on workflow-level schedule modes: - `schedule.interval` - `schedule.intervalHours` - `schedule.dayInterval + schedule.timeOfDay` - executes workflow nodes in topological order - pauses runs on human-review nodes - resumes approved or rejected audit branches - creates Kubernetes Jobs for every operator execution - writes node input manifests into the shared workspace PVC before launching Jobs - runs as a single Go binary ### Operator Runner - receives `RUN_ID`, `WORKFLOW_ID`, `NODE_ID`, and `NODE_LABEL` - executes a generic Python operator contract - reads input manifests from the shared workspace PVC - writes `result.json` and `stage-manifest.json` back to the shared workspace PVC - preserves catalog defaults only when optional hyperparams are omitted; explicit malformed JSON, boolean, comparison-operator, sampling-rule, or exact-positive-integer hyperparams fail the node instead of falling back to defaults ## MongoDB Collections - `projects` - `operators` - `workflows` - `templates` - `runs` - `auditTasks` - `users` - `settings` API and worker startup both call `EnsureIndexes` on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups. Incremental source-state checkpoints are treated as authoritative runtime state, not best-effort metadata. Scan-workflow `DATA_SOURCE` accepts only `BOS`, `SCAN_MODE` accepts only `FULL` or `INCREMENTAL` when explicitly configured, and `INCREMENTAL_UNIT` accepts only `OBJECT` or `DIRECTORY`. The shared source runner accepts explicit `DATA_SOURCE` only as `INLINE`, `LOCAL`, `BOS`, or `UPSTREAM`; unknown values fail instead of producing an empty manifest. `SOURCE_STATE_JSON` must be a JSON object whose `seenFingerprints` entries are strings, `incrementalUnit` is `OBJECT` or `DIRECTORY`, and `directoryDepth` is a positive integer. When a completed run exposes `source-checkpoint.json`, the worker reads that file as the source of truth; unreadable, malformed, empty, missing-fingerprint, or lossy-count checkpoints fail checkpoint commit instead of falling back to stale inline artifact data. Data-asset APIs apply the same stale-data rule for local external payloads. If `artifact.data.dataRef.payloadPath` points to a local file, that payload must be readable JSON object data; unreadable, malformed, or non-object payloads are skipped instead of using older inline `artifact.data`. Collection/package integer summaries only accept exact integer values, so lossy numeric fields are ignored rather than truncated. ## Run Lifecycle 1. frontend calls `POST /api/workflows/:id/run` 2. API inserts a `queued` run document 3. worker claims the run and marks it `running` 4. worker executes start nodes and records logs, resource samples, and artifacts 5. if a node requires review, the run moves to `waiting_for_audit` 6. audit action updates the task to `approved` or `rejected` 7. worker resumes the graph from the appropriate edge 8. run ends in `completed` or `failed` ## Local Verification Run local verification and the security audit as separate gates: ```bash npm run verify:local npm run verify:security ``` `verify:local` includes TypeScript/Go/script linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, and Kubernetes manifest dry-run validation. `verify:security` runs `npm audit --audit-level=moderate` through a retry wrapper so transient registry failures do not prevent local validation from running first. `npm run verify` runs both in that order. ## Runtime Metrics The worker exposes Prometheus text metrics at `/worker-api/metrics`. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the most recent dispatch queue depth. Kubernetes deployments annotate the worker Service for Prometheus scraping at this path. Runs that execute on NVIDIA hardware also persist GPU utilization, GPU memory used/total in MiB, GPU count, and GPU model into `run.resourceSeries`. The monitoring page shows these fields in the resource modal. A missing GPU memory value means the operator-runner could not execute `nvidia-smi`, not that the workflow requested no GPU. When a node requests GPU resources, the worker checks Kubernetes Node allocatable `nvidia.com/gpu` before creating the operator-runner Job. If no schedulable node can satisfy the requested GPU count and configured GPU node selector, the node fails immediately with a `gpu request cannot be scheduled` message instead of leaving a Pending Job to time out. This is a real cluster-capacity check; it does not emulate GPU execution. The workflow editor also reads the same `GET /worker-api/capacity` snapshot. If the current workflow contains GPU-requesting nodes and the loaded capacity proves that no schedulable GPU exists, or that a single node requests more GPUs than any schedulable node can provide, the run button is disabled before a run is created. This prevents predictable POC failures while keeping the worker-side Kubernetes preflight as the authoritative runtime guard. ## Job Retention New operator-runner Jobs use `OPERATOR_JOB_TTL_SECONDS` as `ttlSecondsAfterFinished`, defaulting to one day. For clusters upgraded from older manifests, clean pre-existing finished Jobs with: ```bash DRY_RUN=true npm run k8s:cleanup-jobs npm run k8s:cleanup-jobs ``` Set `OPERATOR_JOB_CLEANUP_MIN_AGE_SECONDS` and `OPERATOR_JOB_CLEANUP_STATUSES=succeeded,failed` to adjust the retention window and terminal states. ## Account Security The API supports HttpOnly cookie-session auth through `/api/auth/*` and super-admin organization/user management under `/api/admin/*`. Existing deployments keep their current behavior unless `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` is set. Before enabling it, bootstrap the first super admin and verify login through `/api/auth/login`. ## Shell Command Policy Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Set `OPERATOR_ALLOW_SHELL_COMMANDS=true` only for deployments that need shell operators, and constrain commands with `OPERATOR_COMMAND_ALLOWLIST_REGEX` and `OPERATOR_COMMAND_DENYLIST_REGEX`. These are platform-controlled variables passed from the worker to operator-runner Jobs, and workflow hyperparams cannot override them. ## Acceptance Checks Run acceptance checks after a deployment from a local shell or external CI. Live workflow acceptance requires `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. GPU acceptance loads `scripts/load-deploy-env.sh`, so on `dev` it uses the POC kubeconfig when `DEPLOY_ENV` is unset; other environments still require a kubeconfig with permission to create a temporary Job in `K8S_NAMESPACE`. The GPU acceptance wrapper validates namespace, image, timeout, GPU count, selector, tolerations JSON, resource name, Job name, and runtime class before any `kubectl` call; invalid acceptance inputs fail locally instead of creating a malformed Job. It also checks for a selector-matched, non-unschedulable Node with enough allocatable GPU before creating its probe Job, so a CPU-only cluster or selector mismatch reports a hardware-capacity blocker explicitly. The deployed worker exposes the same allocatable-node view at `GET /worker-api/capacity`; API callers reach it through the normal worker API proxy and should expect `gpu.available=true` only after the target cluster has schedulable `nvidia.com/gpu` resources. `npm run acceptance:poc-gpu-capacity` verifies the deployed POC worker capacity endpoint from inside the cluster. It creates a short-lived Pod with the deployed `OPERATOR_RUNNER_IMAGE`, calls `http://droboticflow-worker:3002/worker-api/capacity`, and requires node-level capacity evidence. The default expectation is CPU-only POC (`gpu.available=false`, zero schedulable GPU, zero max single-node GPU); set `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` only after a real GPU node pool and NVIDIA runtime/device plugin are installed. `npm run acceptance:poc-gpu-workflow` is the POC workflow-level GPU check. With `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, it logs in and exercises the normal project/workflow/run APIs. Without session credentials, it uses `DROBOTICFLOW_API_TOKEN` from the environment or reads it from `droboticflow-auth-secret`, calls the bounded `/api/acceptance/poc-gpu-workflow` route to create or reuse one fixed GPU acceptance workflow, port-forwards the deployed worker service, and triggers `/worker-api/process-once`. The default CPU-only mode requires a failed run log containing `gpu request cannot be scheduled`. After real GPU hardware is installed, run with `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true` or `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true`; the same script then requires a completed run with artifacts. Each invocation creates a fresh run for evidence but reuses the fixed POC GPU acceptance project/workflow to avoid cluttering the platform with duplicate acceptance containers. This verifies the worker path, not only the standalone `nvidia-smi` probe, while regular business writes still require a user session. `GET /api/acceptance/poc-readiness` and the dashboard expose this latest workflow-level proof as `gpuWorkflowPreflight`, separate from the hardware-capacity `gpu` gate. `npm run acceptance:poc-readiness` also calls that bounded platform readiness endpoint by default and requires matching CPU-only blocked evidence or GPU-available completed evidence unless `POC_PLATFORM_READINESS_ENABLED=false` is explicitly set for isolated diagnostics. ## Video Preview Performance Monitoring artifact previews, data-asset artifact previews, workflow edge previews, sampling previews, and object-storage video previews lazy-load browser media. The preview card initially renders a play action without a `