droboticflow-new/docs/operations.md
eust-w 43326d6a5e feat(platform): land workflow, data tooling, and docs updates
Include the accumulated workflow scheduling, run history, subtree rerun, operator UI, deployment, data asset, object storage, database browser, and documentation changes currently in the worktree.
2026-04-24 12:29:46 +08:00

4.4 KiB

Operations Guide

Service Responsibilities

API

  • validates incoming payloads
  • persists projects, workflows, operators, templates, settings, runs, and audit tasks
  • serves the built SPA in production
  • runs as a single Go binary

Worker

  • polls MongoDB for queued runs
  • auto-enqueues streaming workflows based on workflow-level schedule modes:
    • schedule.interval
    • schedule.intervalHours
    • schedule.dayInterval + schedule.timeOfDay
  • executes workflow nodes in topological order
  • pauses runs on human-review nodes
  • resumes approved or rejected audit branches
  • creates Kubernetes Jobs for every operator execution
  • writes node input manifests into the shared workspace PVC before launching Jobs
  • runs as a single Go binary

Operator Runner

  • receives RUN_ID, WORKFLOW_ID, NODE_ID, and NODE_LABEL
  • executes a generic Python operator contract
  • reads input manifests from the shared workspace PVC
  • writes result.json and stage-manifest.json back to the shared workspace PVC

MongoDB Collections

  • projects
  • operators
  • workflows
  • templates
  • runs
  • auditTasks
  • users
  • settings

Run Lifecycle

  1. frontend calls POST /api/workflows/:id/run
  2. API inserts a queued run document
  3. worker claims the run and marks it running
  4. worker executes start nodes and records logs, resource samples, and artifacts
  5. if a node requires review, the run moves to waiting_for_audit
  6. audit action updates the task to approved or rejected
  7. worker resumes the graph from the appropriate edge
  8. run ends in completed or failed

Local Verification

Run the full verification loop:

npm run verify

Start the stack locally:

npm run dev:full

Smoke test the API and worker:

./scripts/smoke-test.sh

Run the production-style WJZH real-data regression:

API_BASE_URL=http://<node-ip>:30080 \
STORAGE_CONNECTION_ID=<storage-connection-id> \
python3 scripts/wjzh_operator_regression.py --preflight-only

See the full runbook in docs/wjzh-operator-regression.md.

Reconcile persistent system data and remove legacy demo records:

./scripts/reconcile-system-data.sh http://127.0.0.1:3001

Kubernetes Verification

Check workloads:

kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc

Inspect logs:

kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>

Check the shared workspace:

kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200

Common Failure Modes

API starts but returns Mongo errors

  • confirm mongodb pod is healthy
  • confirm the app secret and database name match the Mongo init script
  • inspect kubectl logs deploy/mongodb

Worker cannot launch Jobs

  • confirm the worker pod is running with service account droboticflow-worker
  • confirm Role and RoleBinding exist in the namespace
  • inspect worker logs for Kubernetes API errors
  • confirm the operator-runner image pull secret name matches K8S_IMAGE_PULL_SECRET_NAME
  • confirm the shared PVC exists and is bound
  • confirm worker and API pods both mount the same workspace path

Job starts but downstream nodes cannot see outputs

  • confirm the worker pod and operator-runner Job mount the same PVC and mount path
  • confirm the node directory contains input-artifacts.json, result.json, stage-manifest.json, and outputs/
  • inspect worker logs for DROBOTICFLOW_RESULT_PATH parse failures

Image pull failure

  • recreate droboticflow-registry secret with valid registry credentials
  • confirm the image tags pushed to the registry match the deployment values
  • confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest

UI cannot load data

  • confirm the API pod is healthy at /api/health
  • confirm the NodePort is reachable from the target network
  • inspect browser network traffic for failing endpoints

Old demo data still appears after an upgrade

  • run ./scripts/reconcile-system-data.sh http://<node-ip>:30080
  • confirm /api/users, /api/projects, and /api/templates no longer return legacy seeded records
  • confirm /api/operators only returns the current built-in operator catalog with the active operator-runner image tag