3.2 KiB
3.2 KiB
Operations Guide
Service Responsibilities
API
- validates incoming payloads
- persists projects, workflows, operators, templates, settings, runs, and audit tasks
- serves the built SPA in production
- runs as a single Go binary
Worker
- polls MongoDB for queued runs
- auto-enqueues streaming workflows based on
schedule.interval - executes workflow nodes in topological order
- pauses runs on human-review nodes
- resumes approved or rejected audit branches
- creates Kubernetes Jobs when
K8S_ENABLED=true - runs as a single Go binary
Operator Runner
- receives
RUN_ID,WORKFLOW_ID,NODE_ID, andNODE_LABEL - executes a generic Python operator contract
- emits logs for the worker to associate with the run
MongoDB Collections
projectsoperatorsworkflowstemplatesrunsauditTasksuserssettings
Run Lifecycle
- frontend calls
POST /api/workflows/:id/run - API inserts a
queuedrun document - worker claims the run and marks it
running - worker executes start nodes and records logs, resource samples, and artifacts
- if a node requires review, the run moves to
waiting_for_audit - audit action updates the task to
approvedorrejected - worker resumes the graph from the appropriate edge
- run ends in
completedorfailed
Local Verification
Run the full verification loop:
npm run verify
Start the stack locally:
npm run dev:full
Smoke test the API and worker:
./scripts/smoke-test.sh
Reconcile persistent system data and remove legacy demo records:
./scripts/reconcile-system-data.sh http://127.0.0.1:3001
Kubernetes Verification
Check workloads:
kubectl -n droboticflow get pods
kubectl -n droboticflow get jobs
kubectl -n droboticflow get svc
Inspect logs:
kubectl -n droboticflow logs deploy/droboticflow-api
kubectl -n droboticflow logs deploy/droboticflow-worker
kubectl -n droboticflow logs job/<job-name>
Common Failure Modes
API starts but returns Mongo errors
- confirm
mongodbpod is healthy - confirm the app secret and database name match the Mongo init script
- inspect
kubectl logs deploy/mongodb
Worker cannot launch Jobs
- confirm the worker pod is running with service account
droboticflow-worker - confirm Role and RoleBinding exist in the namespace
- inspect worker logs for Kubernetes API errors
- confirm the operator-runner image pull secret name matches
K8S_IMAGE_PULL_SECRET_NAME
Image pull failure
- recreate
droboticflow-registrysecret with valid registry credentials - confirm the image tags pushed to the registry match the deployment values
- confirm operator-runner Jobs inherit the same pull secret via the rendered Job manifest
UI cannot load data
- confirm the API pod is healthy at
/api/health - confirm the NodePort is reachable from the target network
- inspect browser network traffic for failing endpoints
Old demo data still appears after an upgrade
- run
./scripts/reconcile-system-data.sh http://<node-ip>:30080 - confirm
/api/users,/api/projects, and/api/templatesno longer return legacy seeded records - confirm
/api/operatorsonly returns the current built-in operator catalog with the active operator-runner image tag