droboticflow/docs/operations.md
2026-06-26 05:28:50 +08:00

50 KiB

Operations Guide

Service Responsibilities

API

  • validates incoming payloads
  • persists projects, workflows, operators, templates, settings, runs, and audit tasks
  • serves the built SPA in production
  • runs as a single Go binary

Worker

  • polls MongoDB for queued runs
  • auto-enqueues streaming workflows based on workflow-level schedule modes:
    • schedule.interval
    • schedule.intervalHours
    • schedule.dayInterval + schedule.timeOfDay
  • executes workflow nodes in topological order
  • pauses runs on human-review nodes
  • resumes approved or rejected audit branches
  • creates Kubernetes Jobs for every operator execution
  • writes node input manifests into the shared workspace PVC before launching Jobs
  • runs as a single Go binary

Operator Runner

  • receives RUN_ID, WORKFLOW_ID, NODE_ID, and NODE_LABEL
  • executes a generic Python operator contract
  • reads input manifests from the shared workspace PVC
  • writes result.json and stage-manifest.json back to the shared workspace PVC
  • preserves catalog defaults only when optional hyperparams are omitted; explicit malformed JSON, boolean, comparison-operator, sampling-rule, or exact-positive-integer hyperparams fail the node instead of falling back to defaults

MongoDB Collections

  • projects
  • operators
  • workflows
  • templates
  • runs
  • auditTasks
  • users
  • settings

API and worker startup both call EnsureIndexes on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups.

Incremental source-state checkpoints are treated as authoritative runtime state, not best-effort metadata. Scan-workflow DATA_SOURCE accepts only BOS, SCAN_MODE accepts only FULL or INCREMENTAL when explicitly configured, and INCREMENTAL_UNIT accepts only OBJECT or DIRECTORY. The shared source runner accepts explicit DATA_SOURCE only as INLINE, LOCAL, BOS, or UPSTREAM; unknown values fail instead of producing an empty manifest. SOURCE_STATE_JSON must be a JSON object whose seenFingerprints entries are strings, incrementalUnit is OBJECT or DIRECTORY, and directoryDepth is a positive integer. When a completed run exposes source-checkpoint.json, the worker reads that file as the source of truth; unreadable, malformed, empty, missing-fingerprint, or lossy-count checkpoints fail checkpoint commit instead of falling back to stale inline artifact data.

Data-asset APIs apply the same stale-data rule for local external payloads. If artifact.data.dataRef.payloadPath points to a local file, that payload must be readable JSON object data; unreadable, malformed, or non-object payloads are skipped instead of using older inline artifact.data. Collection/package integer summaries only accept exact integer values, so lossy numeric fields are ignored rather than truncated.

Run Lifecycle

  1. frontend calls POST /api/workflows/:id/run
  2. API inserts a queued run document
  3. worker claims the run and marks it running
  4. worker executes start nodes and records logs, resource samples, and artifacts
  5. if a node requires review, the run moves to waiting_for_audit
  6. audit action updates the task to approved or rejected
  7. worker resumes the graph from the appropriate edge
  8. run ends in completed or failed

Local Verification

Run local verification and the security audit as separate gates:

npm run verify:local
npm run verify:security

verify:local includes TypeScript/Go/script linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, and Kubernetes manifest dry-run validation. verify:security runs npm audit --audit-level=moderate through a retry wrapper so transient registry failures do not prevent local validation from running first. npm run verify runs both in that order.

Runtime Metrics

The worker exposes Prometheus text metrics at /worker-api/metrics. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the most recent dispatch queue depth. Kubernetes deployments annotate the worker Service for Prometheus scraping at this path.

Runs that execute on NVIDIA hardware also persist GPU utilization, GPU memory used/total in MiB, GPU count, and GPU model into run.resourceSeries. The monitoring page shows these fields in the resource modal. A missing GPU memory value means the operator-runner could not execute nvidia-smi, not that the workflow requested no GPU.

When a node requests GPU resources, the worker checks Kubernetes Node allocatable nvidia.com/gpu before creating the operator-runner Job. If no schedulable node can satisfy the requested GPU count and configured GPU node selector, the node fails immediately with a gpu request cannot be scheduled message instead of leaving a Pending Job to time out. This is a real cluster-capacity check; it does not emulate GPU execution.

The workflow editor also reads the same GET /worker-api/capacity snapshot. If the current workflow contains GPU-requesting nodes and the loaded capacity proves that no schedulable GPU exists, or that a single node requests more GPUs than any schedulable node can provide, the run button is disabled before a run is created. This prevents predictable POC failures while keeping the worker-side Kubernetes preflight as the authoritative runtime guard.

Job Retention

New operator-runner Jobs use OPERATOR_JOB_TTL_SECONDS as ttlSecondsAfterFinished, defaulting to one day. For clusters upgraded from older manifests, clean pre-existing finished Jobs with:

DRY_RUN=true npm run k8s:cleanup-jobs
npm run k8s:cleanup-jobs

Set OPERATOR_JOB_CLEANUP_MIN_AGE_SECONDS and OPERATOR_JOB_CLEANUP_STATUSES=succeeded,failed to adjust the retention window and terminal states.

Account Security

The API supports HttpOnly cookie-session auth through /api/auth/* and super-admin organization/user management under /api/admin/*. Existing deployments keep their current behavior unless DROBOTICFLOW_REQUIRE_SESSION_AUTH=true is set. Before enabling it, bootstrap the first super admin and verify login through /api/auth/login.

Shell Command Policy

Explicit COMMAND / SCRIPT execution is disabled by default. Set OPERATOR_ALLOW_SHELL_COMMANDS=true only for deployments that need shell operators, and constrain commands with OPERATOR_COMMAND_ALLOWLIST_REGEX and OPERATOR_COMMAND_DENYLIST_REGEX. These are platform-controlled variables passed from the worker to operator-runner Jobs, and workflow hyperparams cannot override them.

Acceptance Checks

Run acceptance checks after a deployment from a local shell or external CI. Live workflow acceptance requires WORKFLOW_ACCEPTANCE_SOURCE_PATH, a real path visible to operator-runner Jobs. GPU acceptance loads scripts/load-deploy-env.sh, so on dev it uses the POC kubeconfig when DEPLOY_ENV is unset; other environments still require a kubeconfig with permission to create a temporary Job in K8S_NAMESPACE. The GPU acceptance wrapper validates namespace, image, timeout, GPU count, selector, tolerations JSON, resource name, Job name, and runtime class before any kubectl call; invalid acceptance inputs fail locally instead of creating a malformed Job. It also checks for a selector-matched, non-unschedulable Node with enough allocatable GPU before creating its probe Job, so a CPU-only cluster or selector mismatch reports a hardware-capacity blocker explicitly. The deployed worker exposes the same allocatable-node view at GET /worker-api/capacity; API callers reach it through the normal worker API proxy and should expect gpu.available=false on POC until the cluster has schedulable nvidia.com/gpu resources.

npm run acceptance:poc-gpu-capacity verifies the deployed POC worker capacity endpoint from inside the cluster. It creates a short-lived Pod with the deployed OPERATOR_RUNNER_IMAGE, calls http://droboticflow-worker:3002/worker-api/capacity, and requires node-level capacity evidence. The default expectation is CPU-only POC (gpu.available=false, zero schedulable GPU, zero max single-node GPU); set POC_GPU_CAPACITY_EXPECT_AVAILABLE=true only after a real GPU node pool and NVIDIA runtime/device plugin are installed.

npm run acceptance:poc-gpu-workflow is the POC workflow-level GPU preflight check. With DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD, it logs in and exercises the normal project/workflow/run APIs. Without session credentials, it requires DROBOTICFLOW_API_TOKEN, calls the bounded /api/acceptance/poc-gpu-workflow route to create only a fixed GPU-preflight workflow/run, port-forwards the deployed worker service, triggers /worker-api/process-once, and requires a failed run log containing gpu request cannot be scheduled. This verifies the worker path, not only the standalone nvidia-smi probe, while regular business writes still require a user session. GET /api/acceptance/poc-readiness and the dashboard expose this latest workflow-level proof as gpuWorkflowPreflight, separate from the hardware-capacity gpu gate, so CPU-only POC can prove the blocked path without pretending GPU runtime acceptance has passed.

Video Preview Performance

Monitoring artifact previews, data-asset artifact previews, workflow edge previews, sampling previews, and object-storage video previews lazy-load browser media. The preview card initially renders a play action without a <video src>; the real Range-backed, transcode=browser URL is attached only after the user starts the preview. This keeps artifact lists and run detail pages from opening multiple large MP4 streams while the operator is only browsing metadata.

Generated Artifacts

Build output, local Playwright screenshots/videos, pytest caches, and zip bundles should stay out of git. Regenerate validation artifacts locally or publish them as CI/release artifacts when they need to be shared.

Start the stack locally:

npm run dev:full

Run live workflow acceptance against the API and worker:

WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh

WORKFLOW_ACCEPTANCE_SOURCE_PATH may be a filesystem/PVC path or a bos://bucket/prefix path. The 真实数据源 runner infers LOCAL or BOS from the path and rejects a run that has no SOURCE_TEXT, SOURCE_JSON, SOURCE_PATH, configured BOS source, local STORAGE_PREFIX, or upstream artifact input. This prevents an unconfigured source node from completing with an empty manifest.

For deployed workflow acceptance, pass the deployed URL or set BASE_URL / API_BASE_URL, and set WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true so localhost defaults are disabled.

Run the production-style WJZH real-data regression:

API_BASE_URL=http://<node-ip>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only

The WJZH regression script does not provide built-in quality thresholds, video timing, or sampling thresholds; QUALITY_MIN_CLARITY, QUALITY_MAX_SHAKE, QUALITY_MAX_HAND_OCCLUSION, the *_SECONDS timing values, and SAMPLING_RULES_JSON must match the target dataset before the run is accepted as evidence. Quality thresholds are normalized acceptance gates and must be in [0, 1]; integer timing values must be exact safe positive integers; out-of-range values fail preflight instead of weakening the run evidence. See the full runbook in docs/wjzh-operator-regression.md.

Run browser object-storage acceptance against a live storage connection:

BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage

OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE must point to a non-empty UTF-8 text or JSON file that represents real acceptance content. The script uses that same payload for browser create, preview, rename, upload, and delete checks; it no longer has a built-in object body fallback. Set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when the file extension is ambiguous, and set OBJECT_STORAGE_VIDEO_KEY=<existing-video-key> when video playback should be included. PLAYWRIGHT_IGNORE_HTTPS_ERRORS accepts only true/false, 1/0, yes/no, or on/off when explicitly set. The script creates temporary objects under an object-storage-acceptance-* prefix and removes them at the end. For session-authenticated public APIs, provide an organization admin through DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD; for token-protected non-session APIs, provide DROBOTICFLOW_API_TOKEN.

Node/Playwright acceptance scripts share scripts/lib/acceptance-client.mjs for API token headers, Playwright session-cookie CSRF propagation, JSON requests, retries, and optional session login. Keep new browser acceptance scripts on that helper instead of carrying local request wrappers. scripts/playwright-functional-check.mjs and scripts/playwright-preview-operators-check.mjs also forward DROBOTICFLOW_API_TOKEN into browser contexts so UI fetches and preflight API setup use the same token-authenticated path.

Run operator preview acceptance with explicit real payloads and source paths:

BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview

The CSV, text, and JSON payloads must be supplied by the caller; the script materializes those payloads inside operator-runner jobs and no longer creates built-in payload data. PREVIEW_CHECK_JSON_PAYLOAD_FILE must contain a non-empty object array or an object with items[]; the extract/filter/map variables define how the JSON data-operator chain is validated. PREVIEW_CHECK_JSON_FIELD_MAP_JSON must use non-empty string source/destination fields and must not map multiple input fields to the same output field. The scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs, not just from the machine launching the script. Optional checks: PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON, PREVIEW_CHECK_SCAN_MIN_FILES, PREVIEW_CHECK_ROSBAG_TOPIC, and PREVIEW_CHECK_LEROBOT_ROBOT_TYPE. PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.

Run the deployed POC operator-runner full builtin-operator acceptance Job:

npm run acceptance:poc-operators

Build and push all three POC images from the cluster when local Docker or Docker Hub pull limits would make a workstation build unreliable:

DEPLOY_ENV=poc \
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow-poc \
npm run images:poc-kaniko

The Kaniko wrapper archives the current git HEAD into the shared /workspace PVC, creates one Kubernetes Job each for API, worker, and operator-runner, and mounts the existing droboticflow-registry Secret as /kaniko/.docker/config.json. IMAGE_TAG defaults to <short-sha>-poc-<timestamp> for rollback-friendly POC pushes. POC_KANIKO_REGISTRY_MIRRORS defaults to mirror.gcr.io so base image pulls do not rely on unauthenticated Docker Hub quota; set it to a comma-separated mirror list or an empty value only when the cluster registry path is already reliable. POC also sets POC_KANIKO_GOPROXY=https://goproxy.cn,direct by default for Go module downloads inside the cluster; override it when the cluster has a different approved module proxy.

Before launching the Job, run the read-only readiness gate:

DEPLOY_ENV=poc npm run acceptance:poc-readiness

The readiness gate emits JSON and exits non-zero when the deployed API/worker images are not ready, the configured operator-runner image is missing, the deployed worker GPU capacity endpoint cannot be verified, droboticflow-dcloud-secret is absent or incomplete, generated fixtures are enabled, or the real fixture manifest is not usable on the shared workspace PVC. GPU capacity readiness defaults to a local kubectl port-forward probe against service/droboticflow-worker, disables local HTTP proxies for that loopback request, and expects CPU-only POC capacity; set POC_GPU_CAPACITY_EXPECT_AVAILABLE=true only after real GPU nodes are installed. Use POC_GPU_CAPACITY_PROBE_MODE=pod only when port-forward is unavailable and a short-lived in-cluster Pod probe is acceptable. Its default manifest path is ${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json, and every manifest or fixture path must stay under the same WORKSPACE_MOUNT_PATH used by the deployed Pods. Manifest readiness means the JSON is parseable, declares xplatform, csv, json, lerobot, rosbag, model, and shellCommand, and every declared fixture path is readable with the expected file or directory type from the deployed Pod. POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true is diagnostic-only and blocks strict POC readiness instead of producing a ready result. The gate decodes secret keys only to check that required values are non-empty and never prints token, cookie, password, AK, SK, or kubeconfig data. A ready result is only a precondition for final acceptance; the strict pass signal still comes from npm run acceptance:poc-operators returning fullExternalAcceptance=true.

For the final no-mock POC gate, prefer the strict orchestrator:

DEPLOY_ENV=poc npm run acceptance:poc-strict

acceptance:poc-strict refuses dependency-isolation scopes by requiring POC_ACCEPTANCE_REQUIRED_EXTERNALS to include bos,dcloud,mongo. It runs acceptance:poc-readiness, probes the two D-Cloud FAT contracts from droboticflow-dcloud-secret, and only then launches acceptance:poc-operators. Readiness or D-Cloud probe failures emit a standard DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> preflight-failure result and stop before the Kubernetes acceptance Job is created; the D-Cloud probe reports status, response shape, configured-header booleans, and redacted diagnostic fields such as upstream status/message/location, never secret values or business payloads. When POC_ACCEPTANCE_IMPORT_RESULT=true and POC_ACCEPTANCE_IMPORT_API_BASE_URL, API_BASE_URL, or BASE_URL is configured, those blocked preflights are imported into the POC 算子验收 platform history exactly like Job results. Use narrower POC_ACCEPTANCE_REQUIRED_EXTERNALS only when intentionally running acceptance:poc-operators as dependency-isolation diagnostics, not as final strict evidence.

scripts/poc-operator-full-acceptance.sh loads the poc deployment environment on dev, reads the current OPERATOR_RUNNER_IMAGE, workspace PVC, Mongo secret, and BOS workspace secret references from Kubernetes, mounts scripts/poc_operator_full_acceptance_job.py into a temporary Job, and prints one DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> result. The Job executes builtin modules from the deployed runner image and writes evidence under /workspace/acceptance/<run-id>/; <run-id> is set from POC_OPERATOR_ACCEPTANCE_JOB_NAME so the raw result, Kubernetes Job, imported platform run source, and workspace evidence directory stay traceable to the same acceptance execution. If the PVC is misconfigured and the runner cannot write there, the run fails by default instead of silently accepting temporary evidence. Set POC_ACCEPTANCE_ALLOW_TMP_FALLBACK=true only for diagnostic runs that intentionally write to /tmp/droboticflow-acceptance/<run-id>/ and report workspaceFallbackReason.

Final POC operator acceptance requires real input fixtures that already exist inside the Job container, usually on the shared workspace PVC mounted at ${WORKSPACE_MOUNT_PATH:-/workspace}. Prefer the default POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH=/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json file so the platform run records one reproducible fixture contract:

{
  "fixtures": {
    "xplatform": "/workspace/fixtures/xplatform",
    "csv": "/workspace/fixtures/records.csv",
    "json": "/workspace/fixtures/records.json",
    "lerobot": "/workspace/fixtures/lerobot",
    "rosbag": "/workspace/fixtures/rosbag2",
    "model": "/workspace/fixtures/model.json"
  },
  "json": {
    "filterField": "label",
    "filterExpected": "keep",
    "extractField": "items.0.decision",
    "fieldMap": {"label": "decision", "score": "confidence"},
    "templateText": "accepted={items.0.id}; decision={items.0.decision}"
  },
  "video": {
    "minClarity": 0,
    "maxShake": 1,
    "maxHandOcclusion": 1,
    "clarityScale": 35,
    "segmentDurationSeconds": 1,
    "frameIntervalSeconds": 0.5
  },
  "xplatform": {
    "dateStart": "",
    "dateEnd": "",
    "place": "",
    "type": ""
  },
  "audit": {
    "rules": [{"id": "all-items", "field": "collectionId", "operator": "exists", "samplePercent": 100}]
  },
  "shellCommand": "python3 /workspace/fixtures/shell_acceptance.py"
}

The legacy explicit variables remain supported and take precedence over the manifest: POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH, POC_ACCEPTANCE_CSV_INPUT_PATH, POC_ACCEPTANCE_JSON_INPUT_PATH, POC_ACCEPTANCE_LEROBOT_SOURCE_PATH, POC_ACCEPTANCE_ROSBAG_SOURCE_PATH, POC_ACCEPTANCE_MODEL_JSON_PATH, and POC_ACCEPTANCE_SHELL_COMMAND. The script fails by default when any required real fixture path is missing or unreadable. POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true exists only for explicitly labeled diagnostics; generated video, CSV, JSON, LeRobot, ROSBag, model, or shell-command fixtures are not valid final POC acceptance evidence. Tune real fixture semantics with either the manifest fields above or the equivalent environment variables: POC_ACCEPTANCE_JSON_FILTER_FIELD, POC_ACCEPTANCE_JSON_FILTER_EXPECTED, POC_ACCEPTANCE_JSON_EXTRACT_FIELD, POC_ACCEPTANCE_JSON_FIELD_MAP_JSON, POC_ACCEPTANCE_TEMPLATE_TEXT, POC_ACCEPTANCE_MIN_CLARITY, POC_ACCEPTANCE_MAX_SHAKE, POC_ACCEPTANCE_MAX_HAND_OCCLUSION, POC_ACCEPTANCE_CLARITY_SCALE, POC_ACCEPTANCE_XPLATFORM_DATE_START, POC_ACCEPTANCE_XPLATFORM_DATE_END, POC_ACCEPTANCE_XPLATFORM_PLACE, POC_ACCEPTANCE_XPLATFORM_TYPE, POC_ACCEPTANCE_SEGMENT_DURATION_SECONDS, POC_ACCEPTANCE_FRAME_INTERVAL_SECONDS, and POC_ACCEPTANCE_AUDIT_RULES_JSON.

By default POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo, so missing BOS_ENDPOINT, BOS bucket/AK/SK, MONGO_URI, or accepted D-Cloud FAT responses is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with POC_ACCEPTANCE_BOS_ENDPOINT only when validating a different bucket region. When dcloud is required, the wrapper checks that POC_ACCEPTANCE_DCLOUD_SECRET_NAME exists and contains DCLOUD_FAT_BASE_URL, a usable auth path, DCLOUD_ORG_SYMBOL, and DCLOUD_SOURCEAPP before creating the Kubernetes Job; missing or incomplete D-Cloud context fails fast without printing secret values. For dependency isolation runs, explicitly narrow that list, for example POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators; such a run is useful diagnostics, but it is not full external acceptance evidence. Use POC_ACCEPTANCE_BOS_BUCKET when the cluster defaults do not expose the bucket name. D-Cloud FAT context is read from POC_ACCEPTANCE_DCLOUD_SECRET_NAME (default droboticflow-dcloud-secret): provide DCLOUD_FAT_BASE_URL, plus one auth path: DCLOUD_FAT_AUTH_TOKEN or DCLOUD_API_TOKEN, DCLOUD_FAT_COOKIE or DCLOUD_COOKIE, DCLOUD_FAT_USERNAME/DCLOUD_FAT_PASSWORD with optional DCLOUD_FAT_SIGNIN_URL/DCLOUD_FAT_SIGNIN_ENDPOINT, or direct service headers through DCLOUD_D_USER_ID/D_USER_ID; also provide DCLOUD_ORG_SYMBOL and DCLOUD_SOURCEAPP. Do not pass tokens, cookies, passwords, d-user-id values, or other secret values through dry-run output.

The result JSON exposes acceptanceScope, fullExternalAcceptance, fixtureSources, generatedFixtureCount, realFixtureCount, allFixturesReal, operatorPassedCount, coveredOperatorCount, allOperatorCoverageComplete, blockingOperatorCoverageComplete, externalVerificationPassed, externalVerificationMissing, externalVerificationPassedCount, externalVerificationRequiredCount, operatorRepositoryCount, operatorRepositoryRequiredCount, operatorRepositoryCoverageComplete, operatorRepositoryPolicyClean, operatorRepositoryPolicyViolations, statusCounts, and skipped-count fields. The Job sets fullExternalAcceptance=true only when the same strict evidence set is already present in the raw result: scope is full, required externals include bos,dcloud,mongo, every required external system is verified, every operator is covered and passed, every builtin operator has a matching local repository manifest, repository policy violations are empty, no failed/required-skipped/missing-coverage entries exist, allFixturesReal=true, generatedFixtureCount=0, and realFixtureCount>0. Treat fullExternalAcceptance=true plus status=passed as the strict full-acceptance signal; any dependency-isolation result can prove the configured dependencies but not the omitted external operators. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, Mongo requires a write followed by an independent count/drop verification, and D-Cloud requires both FAT API operators to pass. The Monitoring summary card applies the same strict display rules.

When the Job emits a DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> marker, the wrapper extracts it from the Job logs even if the Job exits failed. If POC_ACCEPTANCE_IMPORT_API_BASE_URL, API_BASE_URL, or BASE_URL is set, it posts that result to /api/acceptance/poc-operator-result and records it as a normal platform run under the POC 算子验收 project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD, or the shared DROBOTICFLOW_API_TOKEN for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a poc-operator-full-acceptance-result.json artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Automation can read the narrowed POC gate summary with GET /api/acceptance/poc-readiness and Authorization: Bearer ${DROBOTICFLOW_API_TOKEN}; this endpoint returns only GPU, D-Cloud, strict-acceptance, fixture, coverage, and next-action evidence, while the full /api/dashboard remains session-only in public POC deployments. For fullExternalAcceptance=true, the import must also carry Kubernetes source evidence (source.namespace, source.jobName, source.image, and source.imageId); for every imported result, result.runId should equal source.jobName, any provided source.imageId must be a full @sha256:<64 hex> image digest, must identify the same repository as source.image, and any provided source.image must not contain whitespace. The Monitoring card displays these fields so a final pass is tied to the exact POC Job and operator-runner image digest that produced it. Set POC_ACCEPTANCE_IMPORT_RESULT=false for one-off diagnostics that should not create or update platform acceptance history.

After importing a POC operator result, verify the platform record with the same Kubernetes/Mongo evidence path instead of relying on the import response alone:

DEPLOY_ENV=poc \
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
OPERATOR_RUNNER_IMAGE="${OPERATOR_RUNNER_IMAGE}" \
npm run acceptance:poc-import-verify -- \
  --run-id "run-poc-operator-acceptance-<imported-run-id>" \
  --expected-runner-image "${OPERATOR_RUNNER_IMAGE}" \
  --expected-scope dependency-isolation \
  --expected-external mongo \
  --expected-human-review-task-count 10

acceptance:poc-import-verify creates a short-lived mongo:7 Pod in the target namespace, injects MONGO_URI from droboticflow-mongo-secret, reads MONGO_DB from droboticflow-config, asserts that the imported run contains poc-operator-full-acceptance-result.json, checks the runner image, requires a sha256 image digest, requires Kubernetes source evidence (namespace, jobName, image, and imageId), requires real fixture evidence by default (allFixturesReal=true, generatedFixtureCount=0, positive realFixtureCount, absolute fixture paths, and no generated fixture sources), verifies every --expected-external system is in externalVerificationPassed, enforces clean repository-policy evidence, and optionally checks human_review task evidence. It deletes the probe Pod by default and prints only the bounded verification summary, never the Mongo URI or secret values. --result-file <file> can be used instead of --run-id; the script derives the platform run id from the raw DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> marker.

If strict acceptance cannot even create the Kubernetes Job because required readiness inputs are missing, the wrapper still emits a standard failed DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> preflight result. It evaluates D-Cloud FAT secret readiness and real fixture readiness in the same pass instead of stopping at the first locally detectable blocker. A single blocker keeps its specific preflightStage such as dcloud-secret-readiness or fixture-readiness; multiple blockers use preflightStage=preflight-readiness, list each specific stage in preflightStages, and include one failed records entry per blocker. If fixture readiness passed while another preflight blocker failed, the result still carries the passed real-fixture evidence as fixtureSources, realFixtureCount, generatedFixtureCount, and allFixturesReal, so the platform does not lose positive fixture readiness evidence just because D-Cloud or another external gate is blocked. Fixture-readiness failures also include a bounded fixtureDiscovery evidence summary from the deployed API Pod's shared workspace mount: scanned roots, missing fixture categories, selected paths if any, candidate counts, and next actions. The preflight result also includes the static builtin-operator repository evidence fields (static, operatorRepositoryCount, operatorRepositoryRequiredCount, operatorRepositoryCoverageComplete, operatorRepositoryPolicyClean, and operatorRepositoryPolicyViolations) so a blocked platform run still proves whether local independent operator repositories and no-mock/no-simulation/no-hardcoded policy metadata are complete. When platform import is enabled, the failure is imported as a failed POC acceptance run with preflightFailure=true, acceptanceScope=full, fullExternalAcceptance=false, and blocking coverage derived from the affected repository manifests. This keeps the platform history honest: missing D-Cloud credentials and missing real fixture contracts are visible together instead of being hidden behind the first failing dependency.

Verify an imported strict preflight blocker with the same Mongo probe by naming the expected stage. This switches the verifier to failed-run semantics and checks preflightFailure, preflightStages, failed preflight records, blocking coverage, and failure messages instead of requiring passed-run fixture/image-digest evidence:

DEPLOY_ENV=poc \
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
npm run acceptance:poc-import-verify -- \
  --run-id "run-poc-operator-acceptance-<strict-preflight-run-id>" \
  --expected-scope full \
  --expected-preflight-stage dcloud-secret-readiness

Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH or all explicit real fixture inputs, including POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH, POC_ACCEPTANCE_CSV_INPUT_PATH, POC_ACCEPTANCE_JSON_INPUT_PATH, POC_ACCEPTANCE_LEROBOT_SOURCE_PATH, POC_ACCEPTANCE_ROSBAG_SOURCE_PATH, POC_ACCEPTANCE_MODEL_JSON_PATH, and POC_ACCEPTANCE_SHELL_COMMAND. Manifest and explicit fixture paths must point under WORKSPACE_MOUNT_PATH; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with fixture-readiness coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later.

The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with fullExternalAcceptance=true must also have status=passed, acceptanceScope=full, requiredExternals containing bos,dcloud,mongo, externalVerificationPassed containing all three systems, empty externalVerificationMissing, externalVerificationPassedCount>=3, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, allFixturesReal=true, generatedFixtureCount=0, realFixtureCount>0, structured fixtureSources.items covering csv, json, lerobot, model, rosbag, and xplatform with absolute fixture paths and generated=false for every item, fixture counts that match the fixtureSources.items evidence, static.operatorFileCount>0, static.repositoryManifestCount==static.operatorFileCount, empty static manifest/duplicate/legacy/helper/hardcoded-source violation lists, operatorRepositoryCoverageComplete=true, and operatorRepositoryPolicyClean=true. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.

Configure or refresh the POC real fixture manifest from local fixture files and directories:

DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover

acceptance:poc-fixtures:discover scans existing files and directories on the deployed POC shared /workspace PVC through the API Pod mount. It reports candidate xplatform, csv, json, lerobot, rosbag, and model paths, but it does not generate data, infer a shell validation command, or write a manifest unless --write-manifest is explicitly passed and POC_ACCEPTANCE_SHELL_COMMAND is configured. Historical /workspace/acceptance/... and /workspace/droboticflow-acceptance/... output, build contexts, image-build scratch directories, workflow run output under /workspace/runs, and poc-workflow-acceptance-* scratch inputs are excluded by default so generated artifacts and deployment scratch trees cannot be selected as final fixture input or slow down a broad /workspace scan. The discovery result records excludedRoots next to roots for auditability. Set POC_ACCEPTANCE_DISCOVERY_INCLUDE_ACCEPTANCE_OUTPUT=true only when auditing old runs; override the full prune list with POC_ACCEPTANCE_DISCOVERY_EXCLUDE_ROOTS=/workspace/archive,/workspace/tmp when a POC cluster has different large non-fixture trees. Narrow scanning with POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets when the PVC contains large unrelated trees.

When discovery finds every real fixture and the shell validation command is known, write the manifest in place:

POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets \
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover -- --write-manifest

If the real fixtures are still local to the machine running the command, copy them to the POC PVC and write the manifest with explicit paths:

POC_ACCEPTANCE_LOCAL_XPLATFORM_SOURCE_PATH=/path/to/xplatform-root \
POC_ACCEPTANCE_LOCAL_CSV_INPUT_PATH=/path/to/records.csv \
POC_ACCEPTANCE_LOCAL_JSON_INPUT_PATH=/path/to/records.json \
POC_ACCEPTANCE_LOCAL_LEROBOT_SOURCE_PATH=/path/to/lerobot-root \
POC_ACCEPTANCE_LOCAL_ROSBAG_SOURCE_PATH=/path/to/rosbag-or-dir \
POC_ACCEPTANCE_LOCAL_MODEL_JSON_PATH=/path/to/model.json \
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
npm run acceptance:poc-fixtures

acceptance:poc-fixtures copies only caller-provided non-empty local files/directories, or validates existing paths under ${WORKSPACE_MOUNT_PATH:-/workspace} when those are already present on the shared PVC. It writes the manifest to ${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json by default and prints the corresponding POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH export line. It does not create mock, simulated, generated, or hardcoded acceptance data; dry-run mode (POC_ACCEPTANCE_FIXTURE_DRY_RUN=true) validates local inputs and prints the resolved workspace paths without touching Kubernetes.

Configure or refresh the POC D-Cloud FAT secret from the current shell environment:

DCLOUD_FAT_USERNAME=... \
DCLOUD_FAT_PASSWORD=... \
DCLOUD_FAT_SIGNIN_URL=https://sso-fat.d-robotics.cc/api/signin \
DCLOUD_FAT_BASE_URL=https://cloud-fat.d-robotics.cc \
DCLOUD_ORG_SYMBOL=... \
DCLOUD_SOURCEAPP=... \
scripts/configure-poc-dcloud-secret.sh

The script also accepts DCLOUD_FAT_AUTH_TOKEN/DCLOUD_API_TOKEN or DCLOUD_FAT_COOKIE/DCLOUD_COOKIE instead of username/password. When username/password auth is used, set DCLOUD_FAT_SIGNIN_URL for SSO-hosted FAT login endpoints such as https://sso-fat.d-robotics.cc/api/signin; otherwise the runtime falls back to ${DCLOUD_FAT_BASE_URL}/${DCLOUD_FAT_SIGNIN_ENDPOINT:-/api/signin}. JWT-style auth tokens are decoded locally and rejected when the exp claim is missing, invalid, or already expired; refresh the token or use cookie / username-password auth before treating POC full acceptance as final evidence. DCLOUD_FAT_BASE_URL is required and is stored in the Kubernetes secret; D-Cloud operators and POC full acceptance do not fall back to a hardcoded FAT domain. The script writes canonical secret keys through kubectl apply -f - without placing secret values in kubectl command-line arguments. Use POC_DCLOUD_SECRET_DRY_RUN=true scripts/configure-poc-dcloud-secret.sh to validate the configured key set without printing values or touching the cluster.

Every builtin operator directory also carries a local repository manifest at backend/operators/builtin/<slug>/repository.json. Regenerate the manifests with npm run operators:manifests after catalog or runtime metadata changes, and enforce drift checks with npm run operators:manifests:check. Each manifest records the operator entrypoint, runtime dependencies, strict POC external requirements, optional external API source/route metadata, and the no-mock/no-simulation/no-hardcoded-output acceptance policy. The POC acceptance Job promotes this static check into top-level operatorRepository* evidence fields, and the platform import gate refuses a fullExternalAcceptance=true payload when repository coverage is incomplete or policy violations are present. D-Cloud source repos, route roots, and POC acceptance probe paths live in the D-Cloud operators' operator.json / repository.json metadata, while the FAT base URL is runtime configuration from DCLOUD_FAT_BASE_URL. A strict full POC acceptance run must include bos,dcloud,mongo; runs that omit dcloud cannot be used as final evidence for the two D-Cloud FAT API operators.

The POC static operator-layout check derives the builtin operator count from the deployed runner image's backend/operators/builtin/*/operator.json files. It fails when no builtin metadata exists, when any operator lacks repository.json, when repository manifests do not match their paired metadata, or when builtin operator IDs/names are duplicated. Do not add fixed operator-count assertions to the acceptance Job; adding or removing a real builtin operator should update the catalog, metadata, generated repository manifest, docs, and tests together.

Run X-platform operator-chain acceptance against an explicit real dataset:

BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=/workspace/datasets/xplatform/<collection-root> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='[{"id":"...", "field":"quality.clarity", "operator":"gte", "value":0.3, "samplePercent":100}]' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only

For BOS-backed datasets, provide a storage connection selector and use a full bos://bucket/prefix source path:

BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only

The X-platform acceptance script does not generate synthetic videos, fallback data, sampling rules, quality thresholds, slice/annotation parameters, or minimum-count assertions. It creates a temporary project/workflow that runs BOS 增量扫描 when the source is BOS, then X 平台数据集索引, collection validation, video quality, filtering, slicing, annotation, packaging, sampling, and result upload. XPLATFORM_RULES_JSON, XPLATFORM_MIN_*, XPLATFORM_MIN_CLARITY, XPLATFORM_MAX_SHAKE, XPLATFORM_MAX_HAND_OCCLUSION, XPLATFORM_CLARITY_SCALE, XPLATFORM_SEGMENT_DURATION_SECONDS, and XPLATFORM_FRAME_INTERVAL_SECONDS must match the target real dataset before the run is accepted as evidence. X-platform quality thresholds are finite normalized [0, 1] gates, matching the runtime clarity, shake, and handOcclusion metrics; out-of-range or non-finite values fail before any live workflow is created. XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS must be finite positive numbers, while XPLATFORM_SEGMENT_DURATION_SECONDS and every XPLATFORM_MIN_* count must be exact safe positive integers to match operator-runner runtime validation and acceptance assertions. XPLATFORM_RULES_JSON is validated with the same rule-shape gate as the WJZH regression: every rule needs id, field, operator, value, and samplePercent, ids must be unique, operators must be supported, and samplePercent must be finite numeric data in (0, 100]; booleans are rejected instead of being coerced. Optional filters such as XPLATFORM_DATE_START, XPLATFORM_DATE_END, XPLATFORM_PLACE, and XPLATFORM_TYPE further narrow the target dataset. For protected APIs, use DROBOTICFLOW_API_TOKEN or DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD.

Python live-acceptance and reconciliation scripts share scripts/lib/droboticflow_acceptance.py for API token headers, session login cookies, CSRF headers, JSON requests, optional transient-error retries, and binary object-storage requests. Keep new Python acceptance, regression, or operational API scripts on that shared client so protected deployments are tested through the same real auth path as scripts/xplatform-live-acceptance.py, scripts/wjzh_operator_regression.py, and scripts/reconcile_system_data.py.

Reconcile persistent system data and remove legacy demo records:

export K8S_ENDPOINT=https://<api-server>:6443
# Optional for token-protected APIs when DROBOTICFLOW_REQUIRE_SESSION_AUTH=false:
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
# Required for session-authenticated public APIs; use an organization admin:
export DROBOTICFLOW_ACCEPTANCE_EMAIL=<admin-email>
export DROBOTICFLOW_ACCEPTANCE_PASSWORD=<admin-password>
API_BASE_URL=http://<node-ip>:30080 ./scripts/reconcile-system-data.sh

scripts/reconcile-system-data.sh is only the environment wrapper. The Python implementation reads the current builtin operator catalog from /api/operators and updates those live records to the requested OPERATOR_RUNNER_IMAGE; it does not carry a second static builtin-operator list. The cleanup step skips builtin operators even when their image tag is :latest, so old custom/demo operators can be removed without tombstoning platform-provided builtins.

The source catalog for platform-provided builtin operators is internal/api/builtin_operators_catalog.json. The API embeds this file, validates required fields and duplicate IDs/names at startup, and injects only runtime deployment fields such as the current operator-runner image and timestamps. scripts/generate_operator_docs.py defaults to the same local file and contacts a live API only when --api-base-url is provided, so adding or changing a builtin operator should update the catalog first instead of editing separate Go and Python maps.

The Python operator-runner keeps the video quality and data-filter fallback thresholds aligned with that catalog. If an old workflow node is missing MIN_CLARITY, MAX_SHAKE, or MAX_HAND_OCCLUSION, runtime falls back to 0.8, 0.1, and 0.1; it should not silently accept every video with 0.0/1.0/1.0. If a workflow node explicitly sets any of those thresholds, the value must be finite, numeric, and in [0, 1]; invalid values fail the node instead of being silently replaced with defaults.

The same fail-fast rule applies to video processing timing and normalization parameters. Missing CLARITY_SCALE, SEGMENT_DURATION_SECONDS, or FRAME_INTERVAL_SECONDS use catalog defaults 35, 300, and 1.0; explicit overrides must be finite positive numbers, and SEGMENT_DURATION_SECONDS must be an exact safe positive integer. Invalid overrides fail the node instead of being clamped or replaced with defaults.

Kubernetes Verification

Check workloads:

kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc

Inspect logs:

kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>

Check the shared workspace:

kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200

Common Failure Modes

API starts but returns Mongo errors

  • confirm mongodb pod is healthy
  • confirm the app secret and database name match the Mongo init script
  • inspect kubectl logs deploy/mongodb

Worker cannot launch Jobs

  • confirm the worker pod is running with service account droboticflow-worker
  • confirm Role and RoleBinding exist in the namespace
  • inspect worker logs for Kubernetes API errors
  • confirm the operator-runner image pull secret name matches K8S_IMAGE_PULL_SECRET_NAME
  • confirm the shared PVC exists and is bound
  • confirm worker and API pods both mount the same workspace path
  • for GPU nodes, confirm GET /worker-api/capacity returns matching schedulable nodes before submitting GPU workloads

Job starts but downstream nodes cannot see outputs

  • confirm the worker pod and operator-runner Job mount the same PVC and mount path
  • confirm the node directory contains input-artifacts.json, result.json, stage-manifest.json, and outputs/
  • inspect worker logs for DROBOTICFLOW_RESULT_PATH parse failures

Image pull failure

  • recreate droboticflow-registry secret with valid registry credentials
  • confirm the image tags pushed to the registry match the deployment values
  • confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest

UI cannot load data

  • confirm the API pod is healthy at /api/health
  • confirm the NodePort is reachable from the target network
  • inspect browser network traffic for failing endpoints

Old demo data still appears after an upgrade

  • run K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080
  • include DROBOTICFLOW_API_TOKEN when token auth is enabled without required session auth
  • include DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD for public session-auth deployments, using an organization admin account
  • confirm /api/users, /api/projects, and /api/templates no longer return legacy seeded records
  • confirm /api/operators only returns the current built-in operator catalog with the active operator-runner image tag