forked from longtao.wu/droboticflow
Include the accumulated workflow scheduling, run history, subtree rerun, operator UI, deployment, data asset, object storage, database browser, and documentation changes currently in the worktree.
4.4 KiB
4.4 KiB
Operations Guide
Service Responsibilities
API
- validates incoming payloads
- persists projects, workflows, operators, templates, settings, runs, and audit tasks
- serves the built SPA in production
- runs as a single Go binary
Worker
- polls MongoDB for queued runs
- auto-enqueues streaming workflows based on workflow-level schedule modes:
schedule.intervalschedule.intervalHoursschedule.dayInterval + schedule.timeOfDay
- executes workflow nodes in topological order
- pauses runs on human-review nodes
- resumes approved or rejected audit branches
- creates Kubernetes Jobs for every operator execution
- writes node input manifests into the shared workspace PVC before launching Jobs
- runs as a single Go binary
Operator Runner
- receives
RUN_ID,WORKFLOW_ID,NODE_ID, andNODE_LABEL - executes a generic Python operator contract
- reads input manifests from the shared workspace PVC
- writes
result.jsonandstage-manifest.jsonback to the shared workspace PVC
MongoDB Collections
projectsoperatorsworkflowstemplatesrunsauditTasksuserssettings
Run Lifecycle
- frontend calls
POST /api/workflows/:id/run - API inserts a
queuedrun document - worker claims the run and marks it
running - worker executes start nodes and records logs, resource samples, and artifacts
- if a node requires review, the run moves to
waiting_for_audit - audit action updates the task to
approvedorrejected - worker resumes the graph from the appropriate edge
- run ends in
completedorfailed
Local Verification
Run the full verification loop:
npm run verify
Start the stack locally:
npm run dev:full
Smoke test the API and worker:
./scripts/smoke-test.sh
Run the production-style WJZH real-data regression:
API_BASE_URL=http://<node-ip>:30080 \
STORAGE_CONNECTION_ID=<storage-connection-id> \
python3 scripts/wjzh_operator_regression.py --preflight-only
See the full runbook in docs/wjzh-operator-regression.md.
Reconcile persistent system data and remove legacy demo records:
./scripts/reconcile-system-data.sh http://127.0.0.1:3001
Kubernetes Verification
Check workloads:
kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc
Inspect logs:
kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>
Check the shared workspace:
kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200
Common Failure Modes
API starts but returns Mongo errors
- confirm
mongodbpod is healthy - confirm the app secret and database name match the Mongo init script
- inspect
kubectl logs deploy/mongodb
Worker cannot launch Jobs
- confirm the worker pod is running with service account
droboticflow-worker - confirm Role and RoleBinding exist in the namespace
- inspect worker logs for Kubernetes API errors
- confirm the operator-runner image pull secret name matches
K8S_IMAGE_PULL_SECRET_NAME - confirm the shared PVC exists and is bound
- confirm worker and API pods both mount the same workspace path
Job starts but downstream nodes cannot see outputs
- confirm the worker pod and operator-runner Job mount the same PVC and mount path
- confirm the node directory contains
input-artifacts.json,result.json,stage-manifest.json, andoutputs/ - inspect worker logs for
DROBOTICFLOW_RESULT_PATHparse failures
Image pull failure
- recreate
droboticflow-registrysecret with valid registry credentials - confirm the image tags pushed to the registry match the deployment values
- confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest
UI cannot load data
- confirm the API pod is healthy at
/api/health - confirm the NodePort is reachable from the target network
- inspect browser network traffic for failing endpoints
Old demo data still appears after an upgrade
- run
./scripts/reconcile-system-data.sh http://<node-ip>:30080 - confirm
/api/users,/api/projects, and/api/templatesno longer return legacy seeded records - confirm
/api/operatorsonly returns the current built-in operator catalog with the active operator-runner image tag