2026-07-13 19:21:12 +08:00

294 lines
32 KiB
Markdown

# Drobodata
Drobodata is a workflow orchestration platform for embodied AI and vision pipelines. This repository now contains:
- a React/Vite frontend
- a Go-based MongoDB-backed REST API
- a Go-based worker service that executes workflow DAGs
- a Python-based Kubernetes Job operator runner
- deployment assets for Kubernetes and Baidu Cloud Container Registry
## Architecture
- `cmd/api`: Go API entrypoint and SPA host
- `cmd/worker`: Go worker entrypoint and worker control API
- `internal/api`: REST handlers and request validation
- `internal/worker`: run scheduler, DAG executor, audit pause/resume, worker control API
- `internal/store`: Mongo-backed repositories plus in-memory test store
- `internal/k8s`: Kubernetes Job launcher for operator nodes
- `backend/operator-runner`: Python entrypoint used by Kubernetes Jobs
- `src`: existing frontend, now wired to real APIs
- `k8s`: Kubernetes manifest template
- `scripts`: build, render, deploy, and live acceptance scripts
- `docs`: API, deployment, operations, and design documents
## Local Development
### Prerequisites
- Node.js 22.12+ or 24+
- Go 1.24+
- Python 3.9+
- MongoDB 7+ reachable at `MONGO_URI`
### Setup
1. Install dependencies:
```bash
npm install
```
2. Copy the example environment file and replace the MongoDB placeholder with a real URI:
```bash
cp .env.example .env.local
```
3. Start the Go API and frontend:
```bash
npm run dev
```
4. Start the full stack including Go worker:
```bash
npm run dev:full
```
### Useful Scripts
- `npm run lint`: type-check frontend and verify Go formatting
- `npm run format`: format frontend source and frontend tests with Prettier
- `npm run format:check`: check frontend formatting without writing files
- `npm test`: Go unit and integration tests
- `npm run build`: build frontend assets and Go binaries
- `npm run verify:local`: web/Go/script lint, Go tests, Python operator-runner tests, frontend node tests, build, and Kubernetes manifest dry-run validation
- `npm run verify:security`: dependency audit with retry
- `npm run verify`: run local verification first, then dependency audit
- `npm run images:poc-kaniko`: build and push API, worker, and operator-runner images inside the POC cluster with Kaniko, using the shared workspace PVC and registry secret
- `npm run acceptance:workflow`: live workflow acceptance against the running API and worker; requires an explicit real source path
- `npm run acceptance:gpu`: real Kubernetes GPU scheduling and `nvidia-smi` acceptance test
- `npm run k8s:cleanup-jobs`: delete old finished operator-runner Jobs that predate TTL cleanup
- `npm run reconcile:data -- <api-base-url>`: remove legacy demo data, reset settings, and bootstrap built-in operators through the real API; honors `DROBOTICFLOW_API_TOKEN` and optional session login env vars for protected deployments
- `npm run acceptance:object-storage`: browser-driven object-storage acceptance against a configured live storage connection; requires `OBJECT_STORAGE_CONNECTION_ID`, `OBJECT_STORAGE_CONNECTION_NAME`, or `OBJECT_STORAGE_BUCKET`
- `npm run acceptance:poc-strict`: run the strict POC gate: readiness, D-Cloud FAT probe, then full operator acceptance with `bos,dcloud,mongo` required; blocked preflights emit the same standard platform-importable result marker as the Kubernetes acceptance Job
- `npm run acceptance:poc-operators`: run the POC Kubernetes Job that executes the deployed builtin operator modules with real files, video, MongoDB, BOS, and D-Cloud FAT calls when the required POC external credentials are configured
- `npm run acceptance:poc-readiness`: preflight POC deployment image, D-Cloud FAT secret, and real fixture manifest readiness as JSON without launching an acceptance Job or printing secret values
- `npm run acceptance:dcloud-fat-probe`: non-destructively probe the two D-Cloud FAT API contracts from `droboticflow-dcloud-secret` before enabling strict full POC acceptance; it reports status, response shape, redacted diagnostic fields, and header names, never secret values or user/task payloads
- `npm run acceptance:poc-fixtures:discover`: scan existing real fixture candidates on the shared POC `/workspace` PVC and optionally write a manifest when every required fixture and shell validation command is present
- `npm run acceptance:poc-fixtures`: copy caller-provided real POC acceptance fixture files/directories into the shared workspace PVC and write the fixture manifest consumed by `acceptance:poc-operators`; it does not generate mock, simulated, or hardcoded fixtures
- `npm run acceptance:xplatform`: X-platform operator-chain acceptance against an explicit real dataset; requires `XPLATFORM_SOURCE_PATH`, explicit `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, quality/slicing thresholds, and a storage connection selector when the source or upload path is `bos://...`; count and segment values must be exact safe positive integers
- `python3 scripts/wjzh_operator_regression.py --preflight-only`: preflight the production-style WJZH real-data operator regression against a live API; requires explicit `SOURCE_PATH`, `UPLOAD_ROOT_PREFIX`, storage connection selector, target video filenames, route/filter values, real Robotics dataset paths, incremental seed metadata, quality thresholds, video timing, and `SAMPLING_RULES_JSON`; integer timing values must be exact safe positive integers
Builtin operator source is organized under `operator-repos/builtin/<slug>/`. Each directory has an `operator.json` contract and a committed `release.json` release manifest. The release manifest is the production image truth: it records the digest-pinned CCR image, source commit, source tree hash, environment release evidence, and acceptance evidence. `repository.json` and `internal/api/builtin_operators_catalog.json` are generated from `operator.json + release.json`; changing only the catalog or only the generated repository manifest is blocked by `npm run operators:manifests:check` and `npm run operators:images:check`. Reconciliation updates live builtin operators from the current environment entry in `release.json` and uses registry lookup only to verify that the digest still exists.
The Python operator-runner keeps missing optional hyperparams on catalog defaults, but explicit invalid overrides fail the node instead of silently falling back. JSON hyperparams such as `EXPECTED_JSON`, `FIELD_MAP_JSON`, `RULES_JSON`, and `SOURCE_STATE_JSON` must parse as valid JSON. Field maps must be JSON objects with non-empty string source/destination fields and cannot collapse or overwrite output fields; source-state checkpoints must contain valid string fingerprints and exact positive integer directory depth; sampling rules must be JSON arrays whose rules include `id`, `field`, `operator`, `value`, and finite numeric `(0, 100]` `samplePercent` values; boolean `samplePercent` values are rejected instead of being coerced to `1` or `0`. Comparison operators must be supported, boolean hyperparams such as `REQUIRE_INTRINSICS` accept only explicit true/false spellings, source `DATA_SOURCE` accepts only `INLINE`, `LOCAL`, `BOS`, or `UPSTREAM` when present, `SCAN_MODE` accepts only `FULL` or `INCREMENTAL` when present, `INCREMENTAL_UNIT` accepts only `OBJECT` or `DIRECTORY` when present, and `SEGMENT_DURATION_SECONDS` / `INCREMENTAL_DIRECTORY_DEPTH` must be exact safe positive integers.
## Runtime Configuration
Key environment variables:
- `PORT`: API port, default `3001`
- `WORKER_PORT`: worker port, default `3002`
- `MONGO_URI`: required MongoDB connection string; it must be set explicitly before API or worker startup, and placeholder markers such as `<host>` are rejected
- `MONGO_DB`: Mongo database name
- `MONGO_CONNECT_RETRIES`: startup retry count for API and worker Mongo connections
- `MONGO_CONNECT_RETRY_DELAY_MS`: delay between Mongo connection retries
- `WORKER_POLL_INTERVAL_MS`: worker polling interval
- `WORKER_API_BASE_URL`: API-side upstream URL for `/worker-api/*`, default `http://droboticflow-worker:3002`
- `BOS_ENDPOINT`: optional global BOS endpoint used when scan configs or BOS storage connections omit an endpoint; leave unset only when every BOS workflow or connection provides its own endpoint
- `K8S_NAMESPACE`: namespace used for Jobs
- `K8S_IMAGE_PULL_SECRET_NAME`: image pull secret injected into operator-runner Jobs
- `WORKSPACE_PVC_NAME`: shared PVC used by worker, API, and operator-runner Jobs
- `WORKSPACE_MOUNT_PATH`: shared workspace mount path, default `/workspace`
- `REGISTRY_HOST` / `REGISTRY_NAMESPACE`: explicit registry coordinates used by build/deploy/render scripts to derive image references when the three image variables below are not set
- `IMAGE_TAG`: optional shared image tag for build/deploy/render scripts; scripts default to the current git short SHA only after explicit registry coordinates are configured
- `API_IMAGE` / `WORKER_IMAGE` / `OPERATOR_RUNNER_IMAGE`: container images used for the API, worker, and node execution Jobs; set all three explicitly, or set `REGISTRY_HOST` and `REGISTRY_NAMESPACE` so scripts can derive them
- `OPERATOR_GPU_NODE_SELECTOR`: optional comma-separated node selector for GPU Jobs, for example `accelerator=nvidia`
- `OPERATOR_GPU_TOLERATIONS_JSON`: optional Kubernetes tolerations JSON array for GPU Jobs
- `OPERATOR_GPU_RUNTIME_CLASS`: optional runtimeClassName for GPU Jobs, for example `nvidia`
- `OPERATOR_JOB_TTL_SECONDS`: Kubernetes TTL for finished operator-runner Jobs, default `86400`; set `0` to disable automatic TTL injection
- `OPERATOR_ALLOW_SHELL_COMMANDS`: controls whether `COMMAND` / `SCRIPT` shell execution is allowed, default `false`
- `OPERATOR_COMMAND_ALLOWLIST_REGEX`: optional regular expression that explicit shell commands must match
- `OPERATOR_COMMAND_DENYLIST_REGEX`: optional regular expression that rejects matching explicit shell commands
- `DROBOTICFLOW_API_TOKEN`: optional shared API token for non-browser API clients. When set, non-public API calls require `Authorization: Bearer <token>` or `X-Droboticflow-API-Token` unless a valid session cookie is already present while session auth is enabled.
- `DROBOTICFLOW_REQUIRE_SESSION_AUTH`: optional cookie-session auth enforcement for business APIs, default `false` locally; public LoadBalancer deploys force it to `true`
- `DROBOTICFLOW_SESSION_COOKIE_SECURE`: sets the `Secure` attribute on session cookies, default `false` for local development; enable it behind HTTPS
- `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD`: optional credentials used by live acceptance and reconcile scripts to log in, persist cookies, and send CSRF headers against session-authenticated deployments. Workflow acceptance checks can use admin, pipeline-engineer, data-operator, or legacy operator accounts depending on the exercised surface; reconcile still needs an admin account when it updates settings or management resources.
- `DROBOTICFLOW_TRUSTED_PROXY_CIDRS`: optional comma/space-separated proxy CIDRs whose `X-Forwarded-For` or `X-Real-IP` headers may be trusted for rate limiting; empty means trust no forwarded headers
- `API_SERVICE_TYPE`: Kubernetes API service type for rendered manifests, `NodePort` by default and `ClusterIP` when exposure is handled by LoadBalancer or ingress
- `AI_API_BASE_URL`: OpenAI-compatible endpoint used by server-side AI generation; required when `AI_API_KEY` is set
- `AI_MODEL`: server-side model name; required when `AI_API_KEY` is set
- `AI_API_KEY`: server-side OpenAI-compatible API key used for workflow/operator generation; leave unset to disable AI generation
- `VITE_API_BASE_URL`: optional frontend API base URL override; leave empty for same-origin `/api`, or set an absolute `http://` / `https://` URL
- `VITE_DEV_API_BASE_URL`: optional Vite dev proxy target for `/api`, default `http://127.0.0.1:3001`
- `VITE_DEV_WORKER_BASE_URL`: optional Vite dev proxy target for `/worker-api`, default `http://127.0.0.1:3002`
- `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT`: set to `true` for deployed-environment workflow acceptance so omitted API endpoints fail instead of falling back to localhost.
Explicit startup values are validated before API/worker startup, before Kubernetes manifest rendering, and before `scripts/deploy-k8s.sh` performs cluster writes. `MONGO_URI` is required at runtime and must be a `mongodb://` or `mongodb+srv://` URI without whitespace or unreplaced placeholder markers. `PORT`/`WORKER_PORT` must be valid TCP ports, Mongo retry count and retry delay must be positive, worker poll interval must be positive, replica counts must be non-negative in manifests, NodePort must be in the Kubernetes `30000..32767` range, and `OPERATOR_JOB_TTL_SECONDS` must be non-negative (`0` disables TTL injection). Manifest, build, and deploy wrappers no longer carry a repository-default image registry: callers must set `API_IMAGE`, `WORKER_IMAGE`, and `OPERATOR_RUNNER_IMAGE` explicitly, or set `REGISTRY_HOST` and `REGISTRY_NAMESPACE` so those image references can be derived from `IMAGE_TAG`; all three wrappers validate generated or explicit image references before Docker or Kubernetes calls. Manifest and deploy wrappers also reject invalid Kubernetes namespace/Secret/PVC/Service names, non-absolute workspace mount paths, malformed worker/BOS service URLs, malformed non-empty AI service URLs, missing AI base/model when an AI key is configured, and container image references with whitespace. Multi-line ConfigMap block values such as GPU tolerations JSON and shell command allow/deny regexes are indented before manifest output so readable multi-line input stays valid YAML. Explicit boolean env values accept `true/false`, `1/0`, `yes/no`, or `on/off`; other spellings fail fast instead of silently falling back, and accepted true aliases drive deploy-time branches such as public LoadBalancer creation.
Development and acceptance entrypoints resolve API endpoints through shared helpers instead of each script carrying its own localhost default. Node/Vite entrypoints use `scripts/lib/service-endpoints.mjs`, Python operational scripts use `scripts/lib/droboticflow_env.py`, and shell wrappers use `scripts/lib/service-endpoints.sh`. `BASE_URL` is accepted by browser-oriented acceptance scripts, while deployment/reconcile wrappers preserve `API_BASE_URL` as their primary environment variable. Explicit endpoint values must be absolute `http://` or `https://` URLs so malformed values fail before Vite, curl, Playwright, or urllib requests are attempted. Browser-side `VITE_API_BASE_URL` follows the same rule, with the empty value reserved for same-origin API requests.
The frontend never reads the AI key directly. Workflow generation and operator generation are proxied through the Go API so secrets stay on the server side. AI-generated operators use an explicit publication lifecycle: `draft -> built -> verified -> published`. Generated drafts can be saved and exercised only in sandbox workflows; production workflows, templates, runs, and AI workflow generation accept only published AI operators with repository, digest, fixture, test, and acceptance evidence.
The settings page can update K8s access details, the worker defaults, the global `timezone`, `provider`, `baseURL`, `model`, and the server-side AI key. `GET /api/settings` only returns whether secrets exist via `k8s.hasKubeconfig` / `ai.hasAPIKey`; it does not echo plaintext kubeconfig or API keys back to the browser. When session auth is required, `/api/settings`, `/api/users`, database browser routes, object-storage management routes, Mongo connection tests, and external business-DB token management are organization-admin surfaces; creating a user or setting a password requires a non-blank password value, user names cannot be blank, and account status must be `active` or `disabled`. Business roles are modeled around embodied-AI data production: `admin` owns organization/system governance, `pipeline_engineer` owns operators/templates/workflow design and AI-assisted generation, `data_operator` owns project execution, run control, production monitoring, and storage browsing, `qa_reviewer` owns workflow audit decisions and post-package sampling acceptance, and `compliance_auditor` is read-only for monitoring, data assets, and audit traceability. Legacy `operator` and `auditor` users remain accepted for migration compatibility. Project mutations reject blank names and unknown statuses; project status must be `active` or `archived`. Workflow and template mutations reject blank names and unknown metadata enum values before persistence. Operator mutations reject blank metadata, unknown operator/port enum values, duplicate port IDs, and invalid resource defaults before persistence. Reusable storage/Mongo connection list/get routes and workflow storage browse helpers require admin, pipeline engineer, data operator, or legacy operator role so workflow editors and production operators can select real connections without exposing connection or bucket metadata to reviewers. Creating, updating, deleting, or testing connections remains admin-only. Model-backed AI generation requires admin, pipeline engineer, or legacy operator role. Non-admin pages use `/api/public/settings`, which exposes only `timezone`, and users can update only their own sidebar `name` / `email` through `/api/users/me`. Worker polling and concurrency values are positive-integer gated in the browser, the shared settings payload builder, and the API, so values such as `0`, `1.5`, `60s`, or blank strings fail instead of being ignored. Default CPU/MEM resource values are trimmed and validated as positive Kubernetes resource quantities when settings are saved, and default GPU must be a positive integer when present. Timestamps in the UI are rendered in the configured IANA timezone with compact minute-level formatting for cards, lists, and summaries, while logs keep finer granularity when needed. OSS-related bucket and path settings belong on OSS operators in workflows, not in the global system settings page. Scan workflows can now reference reusable BOS storage connections via `scanConfig.connectionId`, so endpoint and AK/SK are stored once in `/api/storage/connections` and resolved by the API/worker at preview and execution time instead of being duplicated into each workflow; if a scan config or connection omits endpoint, both API preview and worker execution use the validated `BOS_ENDPOINT` startup value when it is set, otherwise real BOS requests fail with an explicit missing-endpoint error.
The API now includes a cookie-session account foundation for organization-based access. Auth endpoints cover bootstrap, login, logout, current session, and super-admin organization/user management. Existing local deployments remain compatible by default; public LoadBalancer deploys require `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` so browser business APIs keep role checks. Session-authenticated unsafe requests, including logout, use a double-submit `droboticflow_csrf` cookie plus `X-Droboticflow-CSRF-Token` header. Browser JSON, object-storage preview, and multipart object upload requests share the same credentials and CSRF header path; object-storage preview byte-count headers are accepted only when they are exact safe non-negative integers. Worker control endpoints under `/worker-api` require an admin, pipeline engineer, data operator, or legacy operator session, or an API token when session auth is not enforced. Bundled acceptance and reconcile scripts can log in with `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` and will send CSRF headers on writes. Do not embed `DROBOTICFLOW_API_TOKEN` into frontend builds; it is intended for machine-to-machine calls only.
The Kubernetes runtime now assumes a shared workspace PVC. Worker writes per-node `input-artifacts.json` files into the shared workspace, operator-runner Jobs read those files through `INPUT_ARTIFACTS_PATH`, then persist `result.json` and `stage-manifest.json` back into the same PVC. Large artifact lists no longer need to travel through giant env vars or stdout payloads. Node CPU and memory requests/limits are applied to Kubernetes Job container resources and must be valid positive Kubernetes resource quantities; malformed explicit values fail before a Job is created instead of being omitted. Operator defaults and workflow-node overrides can also set `sharedMemorySize` for a writable `/dev/shm` memory volume; the effective order is node override, operator default, then `DROBOTICFLOW_SHARED_MEMORY_SIZE` (POC default `8Gi`). GPU requests use `nvidia.com/gpu` when the node or default resource config sets a positive GPU count. GPU scheduling requires a cluster with GPU nodes and the NVIDIA device plugin/runtime installed. Optional GPU node selector, tolerations, and runtime class settings are only injected into Jobs that request a positive GPU count. Invalid GPU selector or tolerations configuration fails worker startup instead of being ignored. Operator-runner Jobs also receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, and the cleanup script can remove old completed Jobs that were created before TTL was enabled. Run `npm run acceptance:gpu` against a real GPU cluster to validate allocatable GPU resources, scheduling, runtime, and `nvidia-smi` execution; the acceptance script rejects invalid namespace, image, timeout, GPU count, selector, toleration, resource-name, Job-name, and runtime-class inputs before it touches the cluster, and renders dynamic manifest values as quoted YAML scalars. Runs that execute on NVIDIA hardware report GPU utilization, GPU memory, GPU count, and GPU model in the resource samples shown on the monitoring page.
Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Deployments that need the shell operator must set `OPERATOR_ALLOW_SHELL_COMMANDS=true` and should constrain commands through allowlist and denylist regular expressions. These policy variables are protected system env vars: workflow and node hyperparams cannot override the worker-provided values.
Kubernetes Job failures include operator logs plus Job conditions, Pod phase, container waiting or termination reasons, and Pod events in the run error path. This makes image pull, scheduling, PVC mount, timeout, and GPU allocation failures visible without separately describing the Pod.
The worker exposes lightweight runtime metrics at `/worker-api/metrics` in Prometheus text format. The endpoint reports run status gauges, dispatch counters, node execution counters, cumulative duration counters, duration histograms, and the latest dispatch queue depth. Access through the API requires an admin, pipeline-engineer, data-operator, or legacy operator session or an API token when session auth is not enforced; the in-cluster worker Service keeps Prometheus scrape annotations for internal collection.
Workflow execution concurrency now has two levels:
- system default: `settings.worker.maxConcurrentRuns`
- workflow override: `workflow.maxConcurrentRuns`
The worker enforces both. A workflow value of `0` means "inherit the global default". Dispatch only starts queued runs while the global active run count is below the system limit and the active count for that workflow is below its effective limit.
Workflow-level execution safeguards are configured alongside the other top-bar workflow settings in the editor:
- `workflow.nodeTimeoutSeconds`
- `0`: disable workflow-level per-node timeout
- `1+`: fail an operator execution once a single node exceeds the configured seconds
- `workflow.runTimeoutSeconds`
- `0`: disable workflow-level timeout
- `1+`: fail the current run once total execution time exceeds the configured seconds
- `workflow.maxRetryAttempts`
- `0`: disable automatic retry
- `1+`: automatically enqueue a new retry run after a failure, up to the configured count
- `workflow.retryBackoffSeconds`
- `0`: retry immediately
- `1+`: keep the retry run queued until its backoff window expires
Automatic retries create real follow-up runs with `trigger=retry`, `attempt=2+`, `parentRunId` pointing at the failed attempt, and `availableAt` marking the first dispatchable time after backoff. Monitoring and history stay auditable because the original run is never mutated into the retry attempt.
Workflow templates now carry the same workflow-level defaults as project workflows. When you save a workflow as a template, duplicate a template, edit a template, or instantiate a template into a project, the workflow `type`, `schedule`, and execution-safeguard fields move across with it instead of being dropped or reset.
Artifact previews are persisted with the run artifacts themselves. JSON artifacts keep their structured payload in `artifact.data`, while non-JSON artifacts can attach a compact `artifact.data.preview` block for text, CSV/TSV, JSONL, images, audio/video metadata, PDF/document hints, SQLite/rosbag summaries, and LeRobot dataset summaries. Artifacts also carry `portId` / `portName`, which lets the worker route multi-output operators by edge handle instead of mixing all outputs together.
When a JSON artifact declares `artifact.data.dataRef.payloadPath`, the local external payload is treated as the authoritative data source for data-asset APIs. Unreadable, malformed, or non-object local payloads are omitted instead of falling back to stale inline `artifact.data`. Data-asset counts, byte sizes, frame counts, and similar integer summaries accept only exact integer values, so decimal, `NaN`, invalid `json.Number`, or overflowing values are ignored instead of being truncated into plausible totals.
The built-in operator catalog now includes real data-processing, X-platform video pipeline, and robotics helpers on top of the original source/merge/upload/audit flow: `BOS 增量扫描`, `X 平台数据集索引`, `采集目录校验`, `视频质检`, `数据过滤`, `视频切片`, `视频标注`, `成品打包`, `抽检采样`, `目录扫描`, `CSV 转 JSON`, `JSON 转 CSV`, `JSON 字段提取`, `JSON 过滤`, `JSON 映射重命名`, `文本模板渲染`, `条件分流`, `LeRobot 数据集概览`, `ROSBag 数据概览`, and `Shell 命令执行`.
`BOS 增量扫描` is a stateful source operator. The workflow trigger still controls whether a workflow runs once, on a timer, or in a streaming loop, but the operator itself persists a per-`workflowId/nodeId/sourceFingerprint` checkpoint. Each run lists the configured BOS prefix, emits only unseen objects, and the worker commits the updated checkpoint only after the run completes successfully. Failed downstream runs do not advance the checkpoint, so the same increment can be retried safely. The runner and worker reject malformed `SOURCE_STATE_JSON`, unreadable `source-checkpoint.json`, lossy checkpoint counts, and invalid directory depth values instead of silently resetting or truncating checkpoint state.
The frontend now exposes a unified `数据资产` module backed by real APIs, while scan-oriented definitions live inside the workflow system project `数据源扫描` instead of a separate first-level page:
- `源数据`: source collections indexed by completed workflows that produced a real `dataset-index.json`
- `成品数据`: finished clip packages aggregated from `packages-manifest.json`
- `抽检任务`: persisted human review queues synced from `audit-samples.json`
`数据资产` groups `源数据` and `成品数据` as two tabs instead of two unrelated menu entries:
- `源数据` shows upstream collected data and its real metadata such as `meta.json` / `video_meta.json`
- `成品数据` shows downstream outputs produced by project workflows, with provenance back to `project / workflow / run`
- both tabs accept project/workflow/run scope filters so project detail and workflow editor can deep-link into the relevant assets
`抽检任务` and the existing workflow `任务审核` share the same manual-review idea, but they serve different phases:
- `任务审核`: approval gates during workflow execution
- `抽检任务`: post-processing quality review over completed data packages
## Deployment
The repository includes three container images:
- `droboticflow-api`
- `droboticflow-worker`
- `droboticflow-operator-runner`
The API image builds the Vite frontend and a Go binary into one container. The worker image contains a Go binary. The operator-runner image is Python-only and is launched by the worker as a Kubernetes Job.
Build and push images:
```bash
./scripts/build-and-push-images.sh
```
Kubernetes deployment requires an explicit external `MONGO_URI`. The main manifest and deploy script no longer create or manage a cluster-local MongoDB instance.
Render Kubernetes manifests:
```bash
./scripts/render-manifests.sh
```
Validate the rendered Kubernetes YAML before applying it:
```bash
npm run k8s:validate
```
Deploy to Kubernetes:
```bash
export MONGO_URI=mongodb://<user>:<password>@<external-host>:27017/
./scripts/deploy-k8s.sh
```
When an existing cluster already has `droboticflow-ai-secret`, `deploy-k8s.sh` reuses its `AI_API_KEY` and fills missing `AI_API_BASE_URL` / `AI_MODEL` from the current `droboticflow-config` ConfigMap. Initial AI setup still requires providing `AI_API_KEY`, `AI_API_BASE_URL`, and `AI_MODEL` together.
Reconcile persistent API data after deployment:
```bash
export K8S_ENDPOINT=https://<api-server>:6443
# Optional when the API is protected by DROBOTICFLOW_API_TOKEN and session auth is not enforced:
export DROBOTICFLOW_API_TOKEN=<shared-api-token>
# Required for public session-auth deployments; use an organization admin account:
export DROBOTICFLOW_ACCEPTANCE_EMAIL=<admin-email>
export DROBOTICFLOW_ACCEPTANCE_PASSWORD=<admin-password>
./scripts/reconcile-system-data.sh http://<node-ip>:30080
```
Run live workflow acceptance locally:
```bash
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
npm run acceptance:workflow
```
Run live workflow acceptance against a deployed cluster:
```bash
WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true \
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh http://<node-ip>:30080
```
`WORKFLOW_ACCEPTANCE_SOURCE_PATH` can be a filesystem/PVC path or a `bos://bucket/prefix` path. The real-source operator infers `LOCAL` or `BOS` from that path and fails fast when no real source is configured, so an empty source configuration cannot pass by producing an empty manifest. Set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` for deployed workflow acceptance so the script fails instead of falling back to localhost when `BASE_URL` / `API_BASE_URL` is omitted.
Run browser object-storage acceptance against a deployed cluster:
```bash
BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
npm run acceptance:object-storage
```
Set `OBJECT_STORAGE_VIDEO_KEY=<existing-video-key>` to additionally verify browser video playback from object storage. Public session-auth deployments should also set `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`; token-protected non-session deployments can set `DROBOTICFLOW_API_TOKEN`.
Run real-environment acceptance from a local shell or external CI by invoking the scripts directly. Workflow acceptance uses `./scripts/live-workflow-acceptance.sh` and needs `WORKFLOW_ACCEPTANCE_SOURCE_PATH`, a real path visible to operator-runner Jobs. GPU acceptance uses `npm run acceptance:gpu` and needs a kubeconfig with permission to create a temporary Job in `K8S_NAMESPACE`. Object-storage and X-platform BOS acceptance need existing storage connection selectors and explicit live data paths. Operator preview acceptance accepts `PREVIEW_CHECK_SCAN_MIN_FILES` only as an exact safe positive integer. X-platform acceptance also needs explicit sampling rules, finite quality thresholds, positive slice/annotation timing, and every `XPLATFORM_MIN_*` count through environment variables; count and segment values reject signs, leading zeros, decimals, units, and unsafe integers.
See the detailed guides:
- [`docs/api.md`](docs/api.md)
- [`docs/deployment.md`](docs/deployment.md)
- [`docs/platform-usage-manual.zh-CN.md`](docs/platform-usage-manual.zh-CN.md)
- [`docs/custom-operator-manual.zh-CN.md`](docs/custom-operator-manual.zh-CN.md)
- [`docs/custom-operators/README.md`](docs/custom-operators/README.md)
- [`docs/builtin-operators-manual.zh-CN.md`](docs/builtin-operators-manual.zh-CN.md)
- [`docs/operators/README.md`](docs/operators/README.md)
- [`docs/operator-authoring.md`](docs/operator-authoring.md)
- [`docs/operations.md`](docs/operations.md)
- [`docs/production-readiness.md`](docs/production-readiness.md)
- [`docs/wjzh-operator-regression.md`](docs/wjzh-operator-regression.md)
- [`docs/feature-overview-playwright-validation-2026-04-15.md`](docs/feature-overview-playwright-validation-2026-04-15.md)
- [`docs/feature-overview-playwright-validation-2026-04-15.html`](docs/feature-overview-playwright-validation-2026-04-15.html)