droboticflow/docs/drosync/operations-runbook.md
2026-07-28 20:43:46 +08:00

2.5 KiB

Operations and SLO runbook

Deployment

On dev, leave DEPLOY_ENV unset; scripts resolve to poc. The feature flag defaults off:

export DROBOTICFLOW_OBJECT_SYNC_ENABLED=false
./scripts/render-manifests.sh > /tmp/droboticflow-poc.yaml
kubectl apply -f /tmp/droboticflow-poc.yaml

Deploy API, Web, and coordinator before enabling client access. The coordinator uses Mongo atomic claims and may run multiple replicas. Its read-only root filesystem has a size-limited /tmp volume for streaming release manifests.

Before setting the feature flag to true, create drosync-signing-secret with DROSYNC_AUDIT_SIGNING_PRIVATE_KEY and DROSYNC_AUDIT_SIGNING_KEY_ID. A missing or malformed key intentionally prevents the enabled coordinator from starting.

If the cluster has Prometheus Operator and a Grafana dashboard sidecar, install the optional SLO rules and dashboard:

./scripts/render-drosync-observability.sh | kubectl apply -f -

Health and metrics

  • API: /api/health, /api/ready, existing API metrics.
  • Coordinator: /health, /ready, /metrics.
  • Key counters: cycles, errors, sessions planned, items committed, staging cleaned, reconciles, releases, and daily audit seals. Cycle timestamps and duration make stalled replicas visible.

Alert when coordinator errors increase for 5 minutes, no cycle occurs for 2 minutes while enabled, oldest planning/committing item exceeds 15 minutes, staging exceeds 24 hours, audit chain continuity fails, or Mongo/BOS error rate threatens 99.9%.

Backup check

Every shift:

  1. Confirm the Mongo backup job completed within the 15-minute RPO and includes all objectSync* collections.
  2. Restore the latest backup into an isolated database and run index creation plus audit continuity verification.
  3. HEAD a sample immutable release manifest and daily audit seal in BOS.
  4. Record Mongo snapshot ID, BOS version/ETag/size, test restore duration, and verifier result.

Cleanup

Coordinator aborts expired multipart uploads and deletes staging after 24 hours. Mongo TTL indexes remove session/item after 90 days and version/audit after one year. Lease documents and latest object heads are not TTL-deleted. Business objects and conflict objects are never automatic cleanup targets.

Rollback

Disable the feature flag, revoke device tokens, and stop new sessions. Do not delete committed objects, sessions, versions, releases, or audit records. Roll server components back independently; protocol v1 remains the compatibility boundary.