Replace plain-text replies with react-markdown and GFM tables, lists, code and safe links. Recognize complete topology fences, JSON fences and standalone graph data in source order without promoting ordinary JSON. Render escaped SVG elements with cycle-safe layouts, zoom, node details and accessible data tables; show incomplete graphs as recoverable errors. Preserve finish_reason=length, prevent truncated tool calls from executing, and give concise topology replies a 4096-token budget. Tighten the prompt to return bounded, closed diagrams rather than an exhaustive node dump. Add parser, model-completion and isolated browser regression coverage. Fix existing frontend type declarations exposed by the React types now provided with react-markdown, without changing workflow execution. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
55 KiB
Operations Guide
Service Responsibilities
API
- validates incoming payloads
- persists projects, workflows, operators, templates, settings, runs, and audit tasks
- serves the built SPA in production
- runs as a single Go binary
Worker
- polls MongoDB for queued runs
- auto-enqueues streaming workflows based on workflow-level schedule modes:
schedule.intervalschedule.intervalHoursschedule.dayInterval + schedule.timeOfDay
- executes workflow nodes in topological order
- pauses runs on human-review nodes
- resumes approved or rejected audit branches
- creates Kubernetes Jobs for every operator execution
- writes node input manifests into the shared workspace PVC before launching Jobs
- runs as a single Go binary
Operator Runner
- receives
RUN_ID,WORKFLOW_ID,NODE_ID, andNODE_LABEL - executes a generic Python operator contract
- reads input manifests from the shared workspace PVC
- writes
result.jsonandstage-manifest.jsonback to the shared workspace PVC - preserves catalog defaults only when optional hyperparams are omitted; explicit malformed JSON, boolean, comparison-operator, sampling-rule, or exact-positive-integer hyperparams fail the node instead of falling back to defaults
MongoDB Collections
projectsoperatorsworkflowstemplatesrunsauditTasksuserssettings
API and worker startup both call EnsureIndexes on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups.
Incremental source-state checkpoints are treated as authoritative runtime state, not best-effort metadata. Scan-workflow DATA_SOURCE accepts only BOS, SCAN_MODE accepts only FULL or INCREMENTAL when explicitly configured, and INCREMENTAL_UNIT accepts only OBJECT or DIRECTORY. The shared source runner accepts explicit DATA_SOURCE only as INLINE, LOCAL, BOS, or UPSTREAM; unknown values fail instead of producing an empty manifest. SOURCE_STATE_JSON must be a JSON object whose seenFingerprints entries are strings, incrementalUnit is OBJECT or DIRECTORY, and directoryDepth is a positive integer. When a completed run exposes source-checkpoint.json, the worker reads that file as the source of truth; unreadable, malformed, empty, missing-fingerprint, or lossy-count checkpoints fail checkpoint commit instead of falling back to stale inline artifact data.
Data-asset APIs apply the same stale-data rule for local external payloads. If artifact.data.dataRef.payloadPath points to a local file, that payload must be readable JSON object data; unreadable, malformed, or non-object payloads are skipped instead of using older inline artifact.data. Collection/package integer summaries only accept exact integer values, so lossy numeric fields are ignored rather than truncated.
Run Lifecycle
- frontend calls
POST /api/workflows/:id/run - API inserts a
queuedrun document - worker claims the run and marks it
running - worker executes start nodes and records logs, resource samples, and artifacts
- if a node requires review, the run moves to
waiting_for_audit - audit action updates the task to
approvedorrejected - worker resumes the graph from the appropriate edge
- run ends in
completedorfailed
Local Verification
Run local verification and the security audit as separate gates:
npm run verify:local
npm run verify:security
verify:local includes TypeScript/Go/script linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, and Kubernetes manifest dry-run validation. verify:security runs npm audit --audit-level=moderate through a retry wrapper so transient registry failures do not prevent local validation from running first. npm run verify runs both in that order.
Runtime Metrics
The worker exposes Prometheus text metrics at /worker-api/metrics. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the most recent dispatch queue depth. Kubernetes deployments annotate the worker Service for Prometheus scraping at this path.
Runs that execute on NVIDIA hardware also persist GPU utilization, GPU memory used/total in MiB, GPU count, and GPU model into run.resourceSeries. The monitoring page shows these fields in the resource modal. A missing GPU memory value means the operator-runner could not execute nvidia-smi, not that the workflow requested no GPU.
When a node requests GPU resources, the worker checks Kubernetes Node allocatable nvidia.com/gpu before creating the operator-runner Job. If no schedulable node can satisfy the requested GPU count and configured GPU node selector, the node fails immediately with a gpu request cannot be scheduled message instead of leaving a Pending Job to time out. This is a real cluster-capacity check; it does not emulate GPU execution.
The workflow editor also reads the same GET /worker-api/capacity snapshot. If the current workflow contains GPU-requesting nodes and the loaded capacity proves that no schedulable GPU exists, or that a single node requests more GPUs than any schedulable node can provide, the run button is disabled before a run is created. This prevents predictable POC failures while keeping the worker-side Kubernetes preflight as the authoritative runtime guard.
Job Retention
New operator-runner Jobs use OPERATOR_JOB_TTL_SECONDS as ttlSecondsAfterFinished, defaulting to one day. For clusters upgraded from older manifests, clean pre-existing finished Jobs with:
DRY_RUN=true npm run k8s:cleanup-jobs
npm run k8s:cleanup-jobs
Set OPERATOR_JOB_CLEANUP_MIN_AGE_SECONDS and OPERATOR_JOB_CLEANUP_STATUSES=succeeded,failed to adjust the retention window and terminal states.
Ops Entry Points
Every deploy, acceptance, release, diagnose and platform action is declared once in ops/catalog/*.yaml
(field reference: ops/README.md, generated index: ops-catalog.md) and reached through
three entry points that share one validation, preflight and execution path:
- CLI:
go run ./cmd/droctl list | describe <action> | run <action> --arg k=v [--dry-run] [--yes], plusdroctl statusanddroctl diagnose <symptom>. Confirm-required actions need--yes;tier: adminactions also needDROCTL_ALLOW_ADMIN=1. HTTP actions log in withDROBOTICFLOW_ACCEPTANCE_EMAIL/DROBOTICFLOW_ACCEPTANCE_PASSWORD(or useDROBOTICFLOW_API_TOKEN) againstDROBOTICFLOW_API_BASE_URL. - Claude Code: the
droflow-opsskill and/droflow-*slash commands drive droctl. - Product: the 运维助手 page (
/assistant)./runs an action directly; free text goes to the model when AI is configured. Actions are filtered by the user's role tier, mutating actions dry-run by default where the script supports it, confirm-required actions (and every real mutation the model proposes) wait for the user's confirmation, and the user is re-loaded before each execution. HTTP actions are replayed in-process through the normal router as that user. Script actions run as an ops-runner Job (Dockerfile.ops-runner, ServiceAccountdroboticflow-ops-runner) and are admin-only; they stay listed but not runnable untilOPS_RUNNER_IMAGEis set. Each execution is stored as an audit record inassistantSteps(actor, role, action, args, dry-run, Job, exit code, output tail).
The API ServiceAccount can create Jobs, so anyone who controls the API pod can start ops-runner Jobs;
the ops-runner Role is therefore kept to what the product-surface scripts call (no pods/exec, secrets
only by name). Actions that need the developer's worktree, keychain, Docker or local credentials
(image builds, deploys from a fresh build, drosync releases, poc.configure-fixtures) are CLI/Claude only.
Assistant answer rendering
Assistant answers render GFM Markdown (headings, lists, tables, code and links). Complete
ops-topology / rca-topology / diagnostic-topology / topology fences, JSON fences containing a
graph, and standalone graph JSON are displayed as SVG diagrams. Ordinary JSON remains code;
node-only arrays must include diagram fields and never gain inferred edges. Diagrams support
zoom, keyboard-accessible node details and a tabular data view. Raw HTML is inert, unsafe links
are removed, and model-selected image URLs are links rather than automatic image requests.
Replies that hit the model output cap persist finishReason=length and show a truncation notice.
Incomplete graphs show an explanation with their source collapsed, not a fabricated partial
relationship graph. Tool calls from truncated replies are not executed. Existing incomplete
answers must be regenerated; missing nodes and edges cannot be recovered by the renderer.
Local browser regression: start the Vite dev server and run
node tests/frontend/assistant-rich-text-browser.mjs http://127.0.0.1:5191 (or your local port).
It uses synthetic API fixtures and isolated Chrome, never production accounts or a real model.
Screenshots are written under ignored output/assistant-rendering/.
Account Security
The API supports HttpOnly cookie-session auth through /api/auth/* and super-admin organization/user management under /api/admin/*. Existing deployments keep their current behavior unless DROBOTICFLOW_REQUIRE_SESSION_AUTH=true is set. Before enabling it, bootstrap the first super admin and verify login through /api/auth/login.
Shell Command Policy
Explicit COMMAND / SCRIPT execution is disabled by default. Set OPERATOR_ALLOW_SHELL_COMMANDS=true only for deployments that need shell operators, and constrain commands with OPERATOR_COMMAND_ALLOWLIST_REGEX and OPERATOR_COMMAND_DENYLIST_REGEX. These are platform-controlled variables passed from the worker to operator-runner Jobs, and workflow hyperparams cannot override them.
Acceptance Checks
Run acceptance checks after a deployment from a local shell or external CI. Live workflow acceptance requires WORKFLOW_ACCEPTANCE_SOURCE_PATH, a real path visible to operator-runner Jobs. GPU acceptance loads scripts/load-deploy-env.sh, so on dev it uses the POC kubeconfig when DEPLOY_ENV is unset; other environments still require a kubeconfig with permission to create a temporary Job in K8S_NAMESPACE. The GPU acceptance wrapper validates namespace, image, timeout, GPU count, selector, tolerations JSON, resource name, Job name, and runtime class before any kubectl call; invalid acceptance inputs fail locally instead of creating a malformed Job. It also checks for a selector-matched, non-unschedulable Node with enough allocatable GPU before creating its probe Job, so a CPU-only cluster or selector mismatch reports a hardware-capacity blocker explicitly. The deployed worker exposes the same allocatable-node view at GET /worker-api/capacity; API callers reach it through the normal worker API proxy and should expect gpu.available=true only after the target cluster has schedulable nvidia.com/gpu resources.
npm run acceptance:poc-gpu-capacity verifies the deployed POC worker capacity endpoint from inside the cluster. It creates a short-lived Pod with the deployed OPERATOR_RUNNER_IMAGE, calls http://droboticflow-worker:3002/worker-api/capacity, and requires node-level capacity evidence. The default expectation is CPU-only POC (gpu.available=false, zero schedulable GPU, zero max single-node GPU); set POC_GPU_CAPACITY_EXPECT_AVAILABLE=true only after a real GPU node pool and NVIDIA runtime/device plugin are installed.
npm run acceptance:poc-gpu-workflow is the POC workflow-level GPU check. With DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD, it logs in and exercises the normal project/workflow/run APIs. Without session credentials, it uses DROBOTICFLOW_API_TOKEN from the environment or reads it from droboticflow-auth-secret, calls the bounded /api/acceptance/poc-gpu-workflow route to create or reuse one fixed GPU acceptance workflow, port-forwards the deployed worker service, and triggers /worker-api/process-once. The default CPU-only mode requires a failed run log containing gpu request cannot be scheduled. After real GPU hardware is installed, run with POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true or POC_GPU_CAPACITY_EXPECT_AVAILABLE=true; the same script then requires a completed run with artifacts. Each invocation creates a fresh run for evidence but reuses the fixed POC GPU acceptance project/workflow to avoid cluttering the platform with duplicate acceptance containers. This verifies the worker path, not only the standalone nvidia-smi probe, while regular business writes still require a user session. GET /api/acceptance/poc-readiness and the dashboard expose this latest workflow-level proof as gpuWorkflowPreflight, separate from the hardware-capacity gpu gate. npm run acceptance:poc-readiness also calls that bounded platform readiness endpoint by default and requires matching CPU-only blocked evidence or GPU-available completed evidence unless POC_PLATFORM_READINESS_ENABLED=false is explicitly set for isolated diagnostics.
Video Preview Performance
Monitoring artifact previews, data-asset artifact previews, workflow edge previews, sampling previews, and object-storage video previews lazy-load browser media. The preview card initially renders a play action without a <video src>; the real Range-backed, transcode=browser URL is attached only after the user starts the preview. This keeps artifact lists and run detail pages from opening multiple large MP4 streams while the operator is only browsing metadata.
Generated Artifacts
Build output, local Playwright screenshots/videos, pytest caches, and zip bundles should stay out of git. Regenerate validation artifacts locally or publish them as CI/release artifacts when they need to be shared.
Start the stack locally:
npm run dev:full
Run live workflow acceptance against the API and worker:
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh
WORKFLOW_ACCEPTANCE_SOURCE_PATH may be a filesystem/PVC path or a bos://bucket/prefix path. The 真实数据源 runner infers LOCAL or BOS from the path and rejects a run that has no SOURCE_TEXT, SOURCE_JSON, SOURCE_PATH, configured BOS source, local STORAGE_PREFIX, or upstream artifact input. This prevents an unconfigured source node from completing with an empty manifest.
For deployed workflow acceptance, pass the deployed URL or set BASE_URL / API_BASE_URL, and set WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true so localhost defaults are disabled.
Run the production-style WJZH real-data regression:
API_BASE_URL=http://<node-ip>:30080 \
SOURCE_PATH=bos://<bucket>/<input-prefix> \
UPLOAD_ROOT_PREFIX=bos://<bucket>/<output-prefix> \
STORAGE_CONNECTION_ID=<storage-connection-id> \
MEDIUM_VIDEO=<medium-video-filename> \
HUGE_VIDEO=<huge-video-filename> \
ROUTE_DEVICE_ID=<device-id-present-in-source> \
FILTER_DATE_START=<yyyy-mm-dd> \
FILTER_DATE_END=<yyyy-mm-dd> \
LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
INCREMENTAL_PLACE=<real-place-label> \
INCREMENTAL_CAMERA_MODEL=<camera-model> \
INCREMENTAL_VIDEO_SUBDIR=<video-subdir> \
QUALITY_MIN_CLARITY=<threshold> \
QUALITY_MAX_SHAKE=<threshold> \
QUALITY_MAX_HAND_OCCLUSION=<threshold> \
HUGE_WINDOW_SECONDS=<seconds> \
MEDIUM_WINDOW_SECONDS=<seconds> \
MEDIUM_SEGMENT_DURATION_SECONDS=<seconds> \
HUGE_SEGMENT_DURATION_SECONDS=<seconds> \
BATCH_WINDOW_SECONDS=<seconds> \
BATCH_QUALITY_WINDOW_SECONDS=<seconds> \
BATCH_SEGMENT_DURATION_SECONDS=<seconds> \
INCREMENTAL_WINDOW_SECONDS=<seconds> \
INCREMENTAL_SEGMENT_DURATION_SECONDS=<seconds> \
ANNOTATION_FRAME_INTERVAL_SECONDS=<seconds> \
SAMPLING_RULES_JSON='<sampling-rules-json-array>' \
python3 scripts/wjzh_operator_regression.py --preflight-only
The WJZH regression script does not provide built-in quality thresholds, video timing, or sampling thresholds; QUALITY_MIN_CLARITY, QUALITY_MAX_SHAKE, QUALITY_MAX_HAND_OCCLUSION, the *_SECONDS timing values, and SAMPLING_RULES_JSON must match the target dataset before the run is accepted as evidence. Quality thresholds are normalized acceptance gates and must be in [0, 1]; integer timing values must be exact safe positive integers; out-of-range values fail preflight instead of weakening the run evidence. See the full runbook in docs/wjzh-operator-regression.md.
Run browser object-storage acceptance against a live storage connection:
BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE must point to a non-empty UTF-8 text or JSON file that represents real acceptance content. The script uses that same payload for browser create, preview, rename, upload, and delete checks; it no longer has a built-in object body fallback. Set OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE when the file extension is ambiguous, and set OBJECT_STORAGE_VIDEO_KEY=<existing-video-key> when video playback should be included. PLAYWRIGHT_IGNORE_HTTPS_ERRORS accepts only true/false, 1/0, yes/no, or on/off when explicitly set. The script creates temporary objects under an object-storage-acceptance-* prefix and removes them at the end. For session-authenticated public APIs, provide an organization admin through DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD; for token-protected non-session APIs, provide DROBOTICFLOW_API_TOKEN.
Node/Playwright acceptance scripts share scripts/lib/acceptance-client.mjs for API token headers, Playwright session-cookie CSRF propagation, JSON requests, retries, and optional session login. Keep new browser acceptance scripts on that helper instead of carrying local request wrappers. scripts/playwright-functional-check.mjs and scripts/playwright-preview-operators-check.mjs also forward DROBOTICFLOW_API_TOKEN into browser contexts so UI fetches and preflight API setup use the same token-authenticated path.
Run operator preview acceptance with explicit real payloads and source paths:
BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview
The CSV, text, and JSON payloads must be supplied by the caller; the script materializes those payloads inside operator-runner jobs and no longer creates built-in payload data. PREVIEW_CHECK_JSON_PAYLOAD_FILE must contain a non-empty object array or an object with items[]; the extract/filter/map variables define how the JSON data-operator chain is validated. PREVIEW_CHECK_JSON_FIELD_MAP_JSON must use non-empty string source/destination fields and must not map multiple input fields to the same output field. The scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs, not just from the machine launching the script. Optional checks: PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON, PREVIEW_CHECK_SCAN_MIN_FILES, PREVIEW_CHECK_ROSBAG_TOPIC, and PREVIEW_CHECK_LEROBOT_ROBOT_TYPE. PREVIEW_CHECK_SCAN_MIN_FILES must be an exact safe positive integer.
Run the deployed POC operator-runner full builtin-operator acceptance Job:
npm run acceptance:poc-operators
Build and push all three POC images from the cluster when local Docker or Docker Hub pull limits would make a workstation build unreliable:
DEPLOY_ENV=poc \
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
REGISTRY_NAMESPACE=dataflow-poc \
npm run images:poc-kaniko
The Kaniko wrapper uploads the current tracked worktree files into the shared /workspace PVC by default, creates one Kubernetes Job each for API, worker, and operator-runner, and mounts the existing droboticflow-registry Secret as /kaniko/.docker/config.json. Set POC_KANIKO_CONTEXT_SOURCE=head when a reproducible build from committed HEAD is required. IMAGE_TAG defaults to <short-sha>-poc-<timestamp> for rollback-friendly POC pushes. The uploaded build context is removed on success, failure, or timeout by default; set POC_KANIKO_CLEAN_CONTEXT=false only when retaining the context for cluster-side diagnostics. POC_KANIKO_REGISTRY_MIRRORS defaults to mirror.gcr.io so base image pulls do not rely on unauthenticated Docker Hub quota; set it to a comma-separated mirror list or an empty value only when the cluster registry path is already reliable. POC also sets POC_KANIKO_GOPROXY=https://goproxy.cn,direct by default for Go module downloads inside the cluster; override it when the cluster has a different approved module proxy.
Before launching the Job, run the read-only readiness gate:
DEPLOY_ENV=poc npm run acceptance:poc-readiness
The readiness gate emits JSON and exits non-zero when the deployed API/worker images are not ready, the configured operator-runner image is missing, the deployed worker GPU capacity endpoint cannot be verified, the platform gpuWorkflowPreflight gate is missing or stale, generated fixtures are enabled, or the real fixture manifest is not usable on the shared workspace PVC. GPU capacity readiness defaults to a local kubectl port-forward probe against service/droboticflow-worker, disables local HTTP proxies for that loopback request, and expects CPU-only POC capacity unless POC_GPU_CAPACITY_EXPECT_AVAILABLE=true is set. Platform GPU workflow readiness resolves API_BASE_URL / BASE_URL first, otherwise reads service/droboticflow-api-public, and authenticates with DROBOTICFLOW_API_TOKEN or the token in droboticflow-auth-secret; run npm run acceptance:poc-gpu-workflow before final CPU-only readiness, or POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true npm run acceptance:poc-gpu-workflow before final GPU-available readiness. Use POC_GPU_CAPACITY_PROBE_MODE=pod only when port-forward is unavailable and a short-lived in-cluster Pod probe is acceptable. Its default manifest path is ${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json, and every manifest or fixture path must stay under the same WORKSPACE_MOUNT_PATH used by the deployed Pods. Manifest readiness means the JSON is parseable, declares xplatform, csv, json, lerobot, rosbag, model, and shellCommand, and every declared fixture path is readable with the expected file or directory type from the deployed Pod. POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true is diagnostic-only and blocks strict POC readiness instead of producing a ready result. The gate decodes secret keys only to check that required values are non-empty and never prints token, cookie, password, AK, SK, or kubeconfig data. A ready result is only a precondition for final acceptance; the strict pass signal still comes from npm run acceptance:poc-operators returning fullExternalAcceptance=true.
For the final no-mock POC gate, prefer the strict orchestrator:
DEPLOY_ENV=poc npm run acceptance:poc-strict
acceptance:poc-strict refuses dependency-isolation scopes by requiring POC_ACCEPTANCE_REQUIRED_EXTERNALS to include bos,mongo. It runs acceptance:poc-readiness and only then launches acceptance:poc-operators. Readiness failures emit a standard DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> preflight-failure result and stop before the Kubernetes acceptance Job is created. With POC_ACCEPTANCE_IMPORT_RESULT=true, POC strict acceptance imports those blocked preflights into the POC 算子验收 platform history exactly like Job results; it uses POC_ACCEPTANCE_IMPORT_API_BASE_URL, API_BASE_URL, or BASE_URL when provided, otherwise it discovers the droboticflow-api-public LoadBalancer URL and loads DROBOTICFLOW_API_TOKEN from droboticflow-auth-secret. Use narrower POC_ACCEPTANCE_REQUIRED_EXTERNALS only when intentionally running acceptance:poc-operators as dependency-isolation diagnostics, not as final strict evidence.
scripts/poc-operator-full-acceptance.sh loads the poc deployment environment on dev, reads the current OPERATOR_RUNNER_IMAGE, workspace PVC, Mongo secret, and BOS workspace secret references from Kubernetes, mounts scripts/poc_operator_full_acceptance_job.py into a temporary Job, and prints one DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> result. The Job executes builtin modules from the deployed runner image and writes evidence under /workspace/acceptance/<run-id>/; <run-id> is set from POC_OPERATOR_ACCEPTANCE_JOB_NAME so the raw result, Kubernetes Job, imported platform run source, and workspace evidence directory stay traceable to the same acceptance execution. If the PVC is misconfigured and the runner cannot write there, the run fails by default instead of silently accepting temporary evidence. Set POC_ACCEPTANCE_ALLOW_TMP_FALLBACK=true only for diagnostic runs that intentionally write to /tmp/droboticflow-acceptance/<run-id>/ and report workspaceFallbackReason.
Final POC operator acceptance requires real input fixtures that already exist inside the Job container, usually on the shared workspace PVC mounted at ${WORKSPACE_MOUNT_PATH:-/workspace}. Prefer the default POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH=/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json file so the platform run records one reproducible fixture contract:
{
"fixtures": {
"xplatform": "/workspace/fixtures/xplatform",
"csv": "/workspace/fixtures/records.csv",
"json": "/workspace/fixtures/records.json",
"lerobot": "/workspace/fixtures/lerobot",
"rosbag": "/workspace/fixtures/rosbag2",
"model": "/workspace/fixtures/model.json"
},
"json": {
"filterField": "label",
"filterExpected": "keep",
"extractField": "items.0.decision",
"fieldMap": {"label": "decision", "score": "confidence"},
"templateText": "accepted={items.0.id}; decision={items.0.decision}"
},
"video": {
"minClarity": 0,
"maxShake": 1,
"maxHandOcclusion": 1,
"clarityScale": 35,
"segmentDurationSeconds": 1,
"frameIntervalSeconds": 0.5
},
"xplatform": {
"dateStart": "",
"dateEnd": "",
"place": "",
"type": ""
},
"audit": {
"rules": [{"id": "all-items", "field": "collectionId", "operator": "exists", "samplePercent": 100}]
},
"shellCommand": "python3 /workspace/fixtures/shell_acceptance.py"
}
The legacy explicit variables remain supported and take precedence over the manifest: POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH, POC_ACCEPTANCE_CSV_INPUT_PATH, POC_ACCEPTANCE_JSON_INPUT_PATH, POC_ACCEPTANCE_LEROBOT_SOURCE_PATH, POC_ACCEPTANCE_ROSBAG_SOURCE_PATH, POC_ACCEPTANCE_MODEL_JSON_PATH, and POC_ACCEPTANCE_SHELL_COMMAND. The script fails by default when any required real fixture path is missing or unreadable. POC_ACCEPTANCE_ALLOW_GENERATED_FIXTURES=true exists only for explicitly labeled diagnostics; generated video, CSV, JSON, LeRobot, ROSBag, model, or shell-command fixtures are not valid final POC acceptance evidence. Tune real fixture semantics with either the manifest fields above or the equivalent environment variables: POC_ACCEPTANCE_JSON_FILTER_FIELD, POC_ACCEPTANCE_JSON_FILTER_EXPECTED, POC_ACCEPTANCE_JSON_EXTRACT_FIELD, POC_ACCEPTANCE_JSON_FIELD_MAP_JSON, POC_ACCEPTANCE_TEMPLATE_TEXT, POC_ACCEPTANCE_MIN_CLARITY, POC_ACCEPTANCE_MAX_SHAKE, POC_ACCEPTANCE_MAX_HAND_OCCLUSION, POC_ACCEPTANCE_CLARITY_SCALE, POC_ACCEPTANCE_XPLATFORM_DATE_START, POC_ACCEPTANCE_XPLATFORM_DATE_END, POC_ACCEPTANCE_XPLATFORM_PLACE, POC_ACCEPTANCE_XPLATFORM_TYPE, POC_ACCEPTANCE_SEGMENT_DURATION_SECONDS, POC_ACCEPTANCE_FRAME_INTERVAL_SECONDS, and POC_ACCEPTANCE_AUDIT_RULES_JSON.
By default POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,mongo, so missing BOS_ENDPOINT, BOS bucket/AK/SK, or MONGO_URI is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with POC_ACCEPTANCE_BOS_ENDPOINT only when validating a different bucket region. For dependency isolation runs, explicitly narrow that list, for example POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators; such a run is useful diagnostics, but it is not full external acceptance evidence. Use POC_ACCEPTANCE_BOS_BUCKET when the cluster defaults do not expose the bucket name. D-Cloud FAT operators were removed from the builtin catalog; historical D-Cloud result artifacts may still render in Monitoring, but current strict POC evidence is based on BOS, Mongo, real fixtures, operator coverage, and repository policy.
The result JSON exposes acceptanceScope, fullExternalAcceptance, fixtureSources, generatedFixtureCount, realFixtureCount, allFixturesReal, operatorPassedCount, coveredOperatorCount, allOperatorCoverageComplete, blockingOperatorCoverageComplete, externalVerificationPassed, externalVerificationMissing, externalVerificationPassedCount, externalVerificationRequiredCount, operatorRepositoryCount, operatorRepositoryRequiredCount, operatorRepositoryCoverageComplete, operatorRepositoryPolicyClean, operatorRepositoryPolicyViolations, statusCounts, and skipped-count fields. The Job sets fullExternalAcceptance=true only when the same strict evidence set is already present in the raw result: scope is full, required externals include bos,mongo, every required external system is verified, every operator is covered and passed, every builtin operator has a matching local repository manifest, repository policy violations are empty, no failed/required-skipped/missing-coverage entries exist, allFixturesReal=true, generatedFixtureCount=0, and realFixtureCount>0. Treat fullExternalAcceptance=true plus status=passed as the strict full-acceptance signal; any dependency-isolation result can prove the configured dependencies but not the omitted external systems. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, and Mongo requires a write followed by an independent count/drop verification. The Monitoring summary card applies the same strict display rules.
When the Job emits a DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> marker, the wrapper extracts it from the Job logs even if the Job exits failed. If POC_ACCEPTANCE_IMPORT_API_BASE_URL, API_BASE_URL, or BASE_URL is set, it posts that result to /api/acceptance/poc-operator-result and records it as a normal platform run under the POC 算子验收 project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through DROBOTICFLOW_ACCEPTANCE_EMAIL and DROBOTICFLOW_ACCEPTANCE_PASSWORD, or the shared DROBOTICFLOW_API_TOKEN for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a poc-operator-full-acceptance-result.json artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Automation can read the narrowed POC gate summary with GET /api/acceptance/poc-readiness and Authorization: Bearer ${DROBOTICFLOW_API_TOKEN}; this endpoint returns only GPU, D-Cloud, strict-acceptance, fixture, coverage, and next-action evidence, while the full /api/dashboard remains session-only in public POC deployments. For fullExternalAcceptance=true, the import must also carry Kubernetes source evidence (source.namespace, source.jobName, source.image, and source.imageId); for every imported result, result.runId should equal source.jobName, any provided source.imageId must be a full @sha256:<64 hex> image digest, must identify the same repository as source.image, and any provided source.image must not contain whitespace. The Monitoring card displays these fields so a final pass is tied to the exact POC Job and operator-runner image digest that produced it. Set POC_ACCEPTANCE_IMPORT_RESULT=false for one-off diagnostics that should not create or update platform acceptance history.
After importing a POC operator result, verify the platform record with the same Kubernetes/Mongo evidence path instead of relying on the import response alone:
DEPLOY_ENV=poc \
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
OPERATOR_RUNNER_IMAGE="${OPERATOR_RUNNER_IMAGE}" \
npm run acceptance:poc-import-verify -- \
--run-id "run-poc-operator-acceptance-<imported-run-id>" \
--expected-runner-image "${OPERATOR_RUNNER_IMAGE}" \
--expected-scope dependency-isolation \
--expected-external mongo \
--expected-human-review-task-count 10
acceptance:poc-import-verify creates a short-lived mongo:7 Pod in the target namespace, injects MONGO_URI from droboticflow-mongo-secret, reads MONGO_DB from droboticflow-config, asserts that the imported run contains poc-operator-full-acceptance-result.json, checks the runner image, requires a sha256 image digest, requires Kubernetes source evidence (namespace, jobName, image, and imageId), requires real fixture evidence by default (allFixturesReal=true, generatedFixtureCount=0, positive realFixtureCount, absolute fixture paths, and no generated fixture sources), verifies every --expected-external system is in externalVerificationPassed, enforces clean repository-policy evidence, and optionally checks human_review task evidence. It deletes the probe Pod by default and prints only the bounded verification summary, never the Mongo URI or secret values. --result-file <file> can be used instead of --run-id; the script derives the platform run id from the raw DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> marker.
If strict acceptance cannot even create the Kubernetes Job because required readiness inputs are missing, the wrapper still emits a standard failed DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json> preflight result. A single blocker keeps its specific preflightStage such as fixture-readiness; multiple blockers use preflightStage=preflight-readiness, list each specific stage in preflightStages, and include one failed records entry per blocker. Fixture-readiness failures also include a bounded fixtureDiscovery evidence summary from the deployed API Pod's shared workspace mount: scanned roots, missing fixture categories, selected paths if any, candidate counts, and next actions. The preflight result also includes the static builtin-operator repository evidence fields (static, operatorRepositoryCount, operatorRepositoryRequiredCount, operatorRepositoryCoverageComplete, operatorRepositoryPolicyClean, and operatorRepositoryPolicyViolations) so a blocked platform run still proves whether local independent operator repositories and no-mock/no-simulation/no-hardcoded policy metadata are complete. When platform import is enabled, the failure is imported as a failed POC acceptance run with preflightFailure=true, acceptanceScope=full, fullExternalAcceptance=false, and blocking coverage derived from the affected repository manifests. This keeps the platform history honest: missing real fixture contracts are visible before a Job is created.
Verify an imported strict preflight blocker with the same Mongo probe by naming the expected stage. This switches the verifier to failed-run semantics and checks preflightFailure, preflightStages, failed preflight records, blocking coverage, and failure messages instead of requiring passed-run fixture/image-digest evidence:
DEPLOY_ENV=poc \
KUBECONFIG=/Users/d-robotics/.config/droboticflow/kubeconfigs/poc.config \
npm run acceptance:poc-import-verify -- \
--run-id "run-poc-operator-acceptance-<strict-preflight-run-id>" \
--expected-scope full \
--expected-preflight-stage fixture-readiness
Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH or all explicit real fixture inputs, including POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH, POC_ACCEPTANCE_CSV_INPUT_PATH, POC_ACCEPTANCE_JSON_INPUT_PATH, POC_ACCEPTANCE_LEROBOT_SOURCE_PATH, POC_ACCEPTANCE_ROSBAG_SOURCE_PATH, POC_ACCEPTANCE_MODEL_JSON_PATH, and POC_ACCEPTANCE_SHELL_COMMAND. Manifest and explicit fixture paths must point under WORKSPACE_MOUNT_PATH; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with fixture-readiness coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later.
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with fullExternalAcceptance=true must also have status=passed, acceptanceScope=full, requiredExternals containing bos,mongo, externalVerificationPassed containing both systems, empty externalVerificationMissing, externalVerificationPassedCount>=2, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, allFixturesReal=true, generatedFixtureCount=0, realFixtureCount>0, structured fixtureSources.items covering csv, json, lerobot, model, rosbag, and xplatform with absolute fixture paths and generated=false for every item, fixture counts that match the fixtureSources.items evidence, static.operatorFileCount>0, static.repositoryManifestCount==static.operatorFileCount, empty static manifest/duplicate/legacy/helper/hardcoded-source violation lists, operatorRepositoryCoverageComplete=true, and operatorRepositoryPolicyClean=true. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
Configure or refresh the POC real fixture manifest from local fixture files and directories:
DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover
acceptance:poc-fixtures:discover scans existing files and directories on the deployed POC shared /workspace PVC through the API Pod mount. It reports candidate xplatform, csv, json, lerobot, rosbag, and model paths, but it does not generate data, infer a shell validation command, or write a manifest unless --write-manifest is explicitly passed and POC_ACCEPTANCE_SHELL_COMMAND is configured. Historical /workspace/acceptance/... and /workspace/droboticflow-acceptance/... output, build contexts, image-build scratch directories, workflow run output under /workspace/runs, and poc-workflow-acceptance-* scratch inputs are excluded by default so generated artifacts and deployment scratch trees cannot be selected as final fixture input or slow down a broad /workspace scan. The discovery result records excludedRoots next to roots for auditability. Set POC_ACCEPTANCE_DISCOVERY_INCLUDE_ACCEPTANCE_OUTPUT=true only when auditing old runs; override the full prune list with POC_ACCEPTANCE_DISCOVERY_EXCLUDE_ROOTS=/workspace/archive,/workspace/tmp when a POC cluster has different large non-fixture trees. Narrow scanning with POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets when the PVC contains large unrelated trees.
When discovery finds every real fixture and the shell validation command is known, write the manifest in place:
POC_ACCEPTANCE_DISCOVERY_ROOTS=/workspace/fixtures,/workspace/datasets \
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
DEPLOY_ENV=poc npm run acceptance:poc-fixtures:discover -- --write-manifest
If the real fixtures are still local to the machine running the command, copy them to the POC PVC and write the manifest with explicit paths:
POC_ACCEPTANCE_LOCAL_XPLATFORM_SOURCE_PATH=/path/to/xplatform-root \
POC_ACCEPTANCE_LOCAL_CSV_INPUT_PATH=/path/to/records.csv \
POC_ACCEPTANCE_LOCAL_JSON_INPUT_PATH=/path/to/records.json \
POC_ACCEPTANCE_LOCAL_LEROBOT_SOURCE_PATH=/path/to/lerobot-root \
POC_ACCEPTANCE_LOCAL_ROSBAG_SOURCE_PATH=/path/to/rosbag-or-dir \
POC_ACCEPTANCE_LOCAL_MODEL_JSON_PATH=/path/to/model.json \
POC_ACCEPTANCE_SHELL_COMMAND='python3 /workspace/fixtures/poc-operator-acceptance/shell_acceptance.py' \
npm run acceptance:poc-fixtures
acceptance:poc-fixtures copies only caller-provided non-empty local files/directories, or validates existing paths under ${WORKSPACE_MOUNT_PATH:-/workspace} when those are already present on the shared PVC. It writes the manifest to ${WORKSPACE_MOUNT_PATH:-/workspace}/fixtures/poc-operator-acceptance/poc-operator-acceptance.json by default and prints the corresponding POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH export line. It does not create mock, simulated, generated, or hardcoded acceptance data; dry-run mode (POC_ACCEPTANCE_FIXTURE_DRY_RUN=true) validates local inputs and prints the resolved workspace paths without touching Kubernetes.
The historical D-Cloud FAT secret helper is retained only for old deployments that still run external D-Cloud operators. The current builtin catalog does not register those operators, and strict POC acceptance does not require this secret:
DCLOUD_FAT_USERNAME=... \
DCLOUD_FAT_PASSWORD=... \
DCLOUD_FAT_SIGNIN_URL=https://sso-fat.d-robotics.cc/api/signin \
DCLOUD_FAT_BASE_URL=https://cloud-fat.d-robotics.cc \
DCLOUD_ORG_SYMBOL=... \
DCLOUD_SOURCEAPP=... \
scripts/configure-poc-dcloud-secret.sh
The script also accepts DCLOUD_FAT_AUTH_TOKEN/DCLOUD_API_TOKEN or DCLOUD_FAT_COOKIE/DCLOUD_COOKIE instead of username/password. When username/password auth is used, set DCLOUD_FAT_SIGNIN_URL for SSO-hosted FAT login endpoints such as https://sso-fat.d-robotics.cc/api/signin; otherwise the runtime falls back to ${DCLOUD_FAT_BASE_URL}/${DCLOUD_FAT_SIGNIN_ENDPOINT:-/api/signin}. JWT-style auth tokens are decoded locally and rejected when the exp claim is missing, invalid, or already expired. The script writes canonical secret keys through kubectl apply -f - without placing secret values in kubectl command-line arguments. Use POC_DCLOUD_SECRET_DRY_RUN=true scripts/configure-poc-dcloud-secret.sh to validate the configured key set without printing values or touching the cluster.
Every builtin operator directory carries a local release manifest at operator-repos/builtin/<slug>/release.json and a generated repository manifest at operator-repos/builtin/<slug>/repository.json. The release manifest is the production image truth: it records the digest-pinned CCR image, source commit, source tree hash, environment release evidence, and acceptance evidence. Regenerate repository manifests with npm run operators:manifests after contract changes, and enforce drift checks with npm run operators:manifests:check plus npm run operators:images:check. These checks require every builtin operator to have a matching release.json, require release.json.imageId == repository.json.imageId == catalog.imageId, and fail when the operator source tree changes without a refreshed release manifest. The POC acceptance Job promotes this static check into top-level operatorRepository* evidence fields, and the platform import gate refuses a fullExternalAcceptance=true payload when repository coverage is incomplete or policy violations are present. A strict full POC acceptance run must include bos,mongo; D-Cloud FAT API operators are intentionally absent from the current builtin catalog.
The POC static operator-layout check derives the builtin operator count from the deployed runner image's backend/operators/builtin/*/operator.json files. It fails when no builtin metadata exists, when any operator lacks repository.json, when repository manifests do not match their paired metadata, or when builtin operator IDs/names are duplicated. Do not add fixed operator-count assertions to the acceptance Job; adding or removing a real builtin operator should update the catalog, metadata, generated repository manifest, docs, and tests together.
Run X-platform operator-chain acceptance against an explicit real dataset:
BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=/workspace/datasets/xplatform/<collection-root> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='[{"id":"...", "field":"quality.clarity", "operator":"gte", "value":0.3, "samplePercent":100}]' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only
For BOS-backed datasets, provide a storage connection selector and use a full bos://bucket/prefix source path:
BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform -- --preflight-only
The X-platform acceptance script does not generate synthetic videos, fallback data, sampling rules, quality thresholds, slice/annotation parameters, or minimum-count assertions. It creates a temporary project/workflow that runs BOS 增量扫描 when the source is BOS, then X 平台数据集索引, collection validation, video quality, filtering, slicing, annotation, packaging, sampling, and result upload. XPLATFORM_RULES_JSON, XPLATFORM_MIN_*, XPLATFORM_MIN_CLARITY, XPLATFORM_MAX_SHAKE, XPLATFORM_MAX_HAND_OCCLUSION, XPLATFORM_CLARITY_SCALE, XPLATFORM_SEGMENT_DURATION_SECONDS, and XPLATFORM_FRAME_INTERVAL_SECONDS must match the target real dataset before the run is accepted as evidence. X-platform quality thresholds are finite normalized [0, 1] gates, matching the runtime clarity, shake, and handOcclusion metrics; out-of-range or non-finite values fail before any live workflow is created. XPLATFORM_CLARITY_SCALE and XPLATFORM_FRAME_INTERVAL_SECONDS must be finite positive numbers, while XPLATFORM_SEGMENT_DURATION_SECONDS and every XPLATFORM_MIN_* count must be exact safe positive integers to match operator-runner runtime validation and acceptance assertions. XPLATFORM_RULES_JSON is validated with the same rule-shape gate as the WJZH regression: every rule needs id, field, operator, value, and samplePercent, ids must be unique, operators must be supported, and samplePercent must be finite numeric data in (0, 100]; booleans are rejected instead of being coerced. Optional filters such as XPLATFORM_DATE_START, XPLATFORM_DATE_END, XPLATFORM_PLACE, and XPLATFORM_TYPE further narrow the target dataset. For protected APIs, use DROBOTICFLOW_API_TOKEN or DROBOTICFLOW_ACCEPTANCE_EMAIL / DROBOTICFLOW_ACCEPTANCE_PASSWORD.
Python live-acceptance and reconciliation scripts share scripts/lib/droboticflow_acceptance.py for API token headers, session login cookies, CSRF headers, JSON requests, optional transient-error retries, and binary object-storage requests. Keep new Python acceptance, regression, or operational API scripts on that shared client so protected deployments are tested through the same real auth path as scripts/xplatform-live-acceptance.py, scripts/wjzh_operator_regression.py, and scripts/reconcile_system_data.py.
Reconcile persistent system data and remove legacy demo records:
export K8S_ENDPOINT=https://<api-server>:6443
# Optional for token-protected APIs when DROBOTICFLOW_REQUIRE_SESSION_AUTH=false:
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
# Required for session-authenticated public APIs; use an organization admin:
export DROBOTICFLOW_ACCEPTANCE_EMAIL=<admin-email>
export DROBOTICFLOW_ACCEPTANCE_PASSWORD=<admin-password>
API_BASE_URL=http://<node-ip>:30080 ./scripts/reconcile-system-data.sh
scripts/reconcile-system-data.sh is only the environment wrapper. The Python implementation reads the current builtin operator catalog from /api/operators and updates those live records to the digest-pinned image declared by operator-repos/builtin/<slug>/release.json for the target environment. Registry lookup is a validation step only: if the registry digest differs from the committed release manifest, reconcile fails instead of silently replacing the source of truth. The cleanup step skips builtin operators even when their image tag is :latest, so old custom/demo operators can be removed without tombstoning platform-provided builtins.
The source of truth for platform-provided builtin operators is operator-repos/builtin/<slug>/operator.json plus operator-repos/builtin/<slug>/release.json. internal/api/builtin_operators_catalog.json is a generated API seed artifact and must not be hand-edited as the only change. The API embeds the generated catalog and still validates required fields and duplicate IDs/names at startup. scripts/generate_operator_docs.py defaults to the local generated catalog and contacts a live API only when --api-base-url is provided, so adding or changing a builtin operator should update the operator repo directory, refresh the release manifest after image publication, and regenerate the catalog/manifests.
The Python operator-runner keeps the video quality and data-filter fallback thresholds aligned with that catalog. If an old workflow node is missing MIN_CLARITY, MAX_SHAKE, or MAX_HAND_OCCLUSION, runtime falls back to 0.8, 0.1, and 0.1; it should not silently accept every video with 0.0/1.0/1.0. If a workflow node explicitly sets any of those thresholds, the value must be finite, numeric, and in [0, 1]; invalid values fail the node instead of being silently replaced with defaults.
The same fail-fast rule applies to video processing timing and normalization parameters. Missing CLARITY_SCALE, SEGMENT_DURATION_SECONDS, or FRAME_INTERVAL_SECONDS use catalog defaults 35, 300, and 1.0; explicit overrides must be finite positive numbers, and SEGMENT_DURATION_SECONDS must be an exact safe positive integer. Invalid overrides fail the node instead of being clamped or replaced with defaults.
Kubernetes Verification
Check workloads:
kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc
Inspect logs:
kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>
Check the shared workspace:
kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200
Common Failure Modes
API starts but returns Mongo errors
- confirm
mongodbpod is healthy - confirm the app secret and database name match the Mongo init script
- inspect
kubectl logs deploy/mongodb
Worker cannot launch Jobs
- confirm the worker pod is running with service account
droboticflow-worker - confirm Role and RoleBinding exist in the namespace
- inspect worker logs for Kubernetes API errors
- confirm the operator-runner image pull secret name matches
K8S_IMAGE_PULL_SECRET_NAME - confirm the shared PVC exists and is bound
- confirm worker and API pods both mount the same workspace path
- for GPU nodes, confirm
GET /worker-api/capacityreturns matching schedulable nodes before submitting GPU workloads
Job starts but downstream nodes cannot see outputs
- confirm the worker pod and operator-runner Job mount the same PVC and mount path
- confirm the node directory contains
input-artifacts.json,result.json,stage-manifest.json, andoutputs/ - inspect worker logs for
DROBOTICFLOW_RESULT_PATHparse failures
Training fails with [Errno 107] Transport endpoint is not connected
- the BOS FUSE mount died under the pod; check
kubectl get events -A | grep -i evictfor an evictedkube-system/bosfs-<hash>pod andDiskPressureon the node - CCE csi-bosplugin runs bosfs with
-o tmpdir=/tmp(host/tmp/bosfs/logs/bosfs-<hash>on the root disk) and leavesbosfs.tmp.*files behind when a mount pod ends; on the A800 node each full training run leaked ~100G until the root disk reached the kubelet eviction threshold and the BestEffort bosfs pod was evicted mid-run - keep
k8s/bosfs-tmp-reaper.yamldeployed (kubectl apply -f k8s/bosfs-tmp-reaper.yaml); it removes temp files of dead mounts that no process holds open, and logsroot_availper node every 5 minutes (kubectl -n droboticflow logs -l app=bosfs-tmp-reaper --prefix --tail=3) - the reaper pins the worker image by digest; update the digest if that image is ever garbage-collected from the registry
- a climbing
RESTARTScount on a reaper pod means its heartbeat went stale; aWARN skip cyclelog means bosfs args no longer carrycredentials/<hash>(likely a csi-bosplugin upgrade) and nothing is reaped on that node until the hash parsing is updated
Image pull failure
- recreate
droboticflow-registrysecret with valid registry credentials - confirm the image tags pushed to the registry match the deployment values
- confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest
UI cannot load data
- confirm the API pod is healthy at
/api/health - confirm the NodePort is reachable from the target network
- inspect browser network traffic for failing endpoints
Old demo data still appears after an upgrade
- run
K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080 - include
DROBOTICFLOW_API_TOKENwhen token auth is enabled without required session auth - include
DROBOTICFLOW_ACCEPTANCE_EMAILandDROBOTICFLOW_ACCEPTANCE_PASSWORDfor public session-auth deployments, using an organization admin account - confirm
/api/users,/api/projects, and/api/templatesno longer return legacy seeded records - confirm
/api/operatorsonly returns the current built-in operator catalog with the active operator-runner image tag