5.2 KiB
Operations Guide
Service Responsibilities
API
- validates incoming payloads
- persists projects, workflows, operators, templates, settings, runs, and audit tasks
- serves the built SPA in production
- runs as a single Go binary
Worker
- polls MongoDB for queued runs
- auto-enqueues streaming workflows based on workflow-level schedule modes:
schedule.intervalschedule.intervalHoursschedule.dayInterval + schedule.timeOfDay
- executes workflow nodes in topological order
- pauses runs on human-review nodes
- resumes approved or rejected audit branches
- creates Kubernetes Jobs for every operator execution
- writes node input manifests into the shared workspace PVC before launching Jobs
- runs as a single Go binary
Operator Runner
- receives
RUN_ID,WORKFLOW_ID,NODE_ID, andNODE_LABEL - executes a generic Python operator contract
- reads input manifests from the shared workspace PVC
- writes
result.jsonandstage-manifest.jsonback to the shared workspace PVC
MongoDB Collections
projectsoperatorsworkflowstemplatesrunsauditTasksuserssettings
API and worker startup both call EnsureIndexes on the platform MongoDB. Index creation is idempotent and covers the main lookup and dispatch paths such as run status queues, workflow/project run history, source-state checkpoints, audit queues, and singleton/settings lookups.
Run Lifecycle
- frontend calls
POST /api/workflows/:id/run - API inserts a
queuedrun document - worker claims the run and marks it
running - worker executes start nodes and records logs, resource samples, and artifacts
- if a node requires review, the run moves to
waiting_for_audit - audit action updates the task to
approvedorrejected - worker resumes the graph from the appropriate edge
- run ends in
completedorfailed
Local Verification
Run the full verification loop:
npm run verify
This includes TypeScript/Go linting, frontend Prettier formatting checks, Go tests, Python operator-runner tests, frontend node tests, production build, npm audit --audit-level=moderate, and Kubernetes manifest dry-run validation.
Runtime Metrics
The worker exposes Prometheus text metrics at /worker-api/metrics. The endpoint includes run status gauges, ProcessAvailable counters, dispatch counters, node execution counters, cumulative node execution duration, and the most recent dispatch queue depth.
Start the stack locally:
npm run dev:full
Smoke test the API and worker:
./scripts/smoke-test.sh
Run the production-style WJZH real-data regression:
API_BASE_URL=http://<node-ip>:30080 \
STORAGE_CONNECTION_ID=<storage-connection-id> \
python3 scripts/wjzh_operator_regression.py --preflight-only
See the full runbook in docs/wjzh-operator-regression.md.
Reconcile persistent system data and remove legacy demo records:
./scripts/reconcile-system-data.sh http://127.0.0.1:3001
Kubernetes Verification
Check workloads:
kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc
Inspect logs:
kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>
Check the shared workspace:
kubectl -n droboticflow exec deploy/droboticflow-worker -- ls -R /workspace/runs | head -200
Common Failure Modes
API starts but returns Mongo errors
- confirm
mongodbpod is healthy - confirm the app secret and database name match the Mongo init script
- inspect
kubectl logs deploy/mongodb
Worker cannot launch Jobs
- confirm the worker pod is running with service account
droboticflow-worker - confirm Role and RoleBinding exist in the namespace
- inspect worker logs for Kubernetes API errors
- confirm the operator-runner image pull secret name matches
K8S_IMAGE_PULL_SECRET_NAME - confirm the shared PVC exists and is bound
- confirm worker and API pods both mount the same workspace path
Job starts but downstream nodes cannot see outputs
- confirm the worker pod and operator-runner Job mount the same PVC and mount path
- confirm the node directory contains
input-artifacts.json,result.json,stage-manifest.json, andoutputs/ - inspect worker logs for
DROBOTICFLOW_RESULT_PATHparse failures
Image pull failure
- recreate
droboticflow-registrysecret with valid registry credentials - confirm the image tags pushed to the registry match the deployment values
- confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest
UI cannot load data
- confirm the API pod is healthy at
/api/health - confirm the NodePort is reachable from the target network
- inspect browser network traffic for failing endpoints
Old demo data still appears after an upgrade
- run
./scripts/reconcile-system-data.sh http://<node-ip>:30080 - confirm
/api/users,/api/projects, and/api/templatesno longer return legacy seeded records - confirm
/api/operatorsonly returns the current built-in operator catalog with the active operator-runner image tag