# Operations and SLO runbook ## Deployment On `dev`, leave `DEPLOY_ENV` unset; scripts resolve to `poc`. The feature flag defaults off: ```bash export DROBOTICFLOW_OBJECT_SYNC_ENABLED=false ./scripts/render-manifests.sh > /tmp/droboticflow-poc.yaml kubectl apply -f /tmp/droboticflow-poc.yaml ``` Deploy API, Web, and coordinator before enabling client access. The coordinator uses Mongo atomic claims and may run multiple replicas. Its read-only root filesystem has a size-limited `/tmp` volume for streaming release manifests. Before setting the feature flag to `true`, create `drosync-signing-secret` with `DROSYNC_AUDIT_SIGNING_PRIVATE_KEY` and `DROSYNC_AUDIT_SIGNING_KEY_ID`. A missing or malformed key intentionally prevents the enabled coordinator from starting. If the cluster has Prometheus Operator and a Grafana dashboard sidecar, install the optional SLO rules and dashboard: ```bash ./scripts/render-drosync-observability.sh | kubectl apply -f - ``` ## Health and metrics - API: `/api/health`, `/api/ready`, existing API metrics. - Coordinator: `/health`, `/ready`, `/metrics`. - Key counters: cycles, errors, sessions planned, items committed, staging cleaned, reconciles, releases, and daily audit seals. Cycle timestamps and duration make stalled replicas visible. Alert when coordinator errors increase for 5 minutes, no cycle occurs for 2 minutes while enabled, oldest planning/committing item exceeds 15 minutes, staging exceeds 24 hours, audit chain continuity fails, or Mongo/BOS error rate threatens 99.9%. ## Backup check Every shift: 1. Confirm the Mongo backup job completed within the 15-minute RPO and includes all `objectSync*` collections. 2. Restore the latest backup into an isolated database and run index creation plus audit continuity verification. 3. HEAD a sample immutable release manifest and daily audit seal in BOS. 4. Record Mongo snapshot ID, BOS version/ETag/size, test restore duration, and verifier result. ## Cleanup Coordinator aborts expired multipart uploads and deletes staging after 24 hours. Mongo TTL indexes remove session/item after 90 days and version/audit after one year. Lease documents and latest object heads are not TTL-deleted. Business objects and conflict objects are never automatic cleanup targets. ## Rollback Disable the feature flag, revoke device tokens, and stop new sessions. Do not delete committed objects, sessions, versions, releases, or audit records. Roll server components back independently; protocol `v1` remains the compatibility boundary.