droboticflow/docs/drosync/operations-runbook.md
2026-07-28 20:43:46 +08:00

47 lines
2.5 KiB
Markdown

# Operations and SLO runbook
## Deployment
On `dev`, leave `DEPLOY_ENV` unset; scripts resolve to `poc`. The feature flag defaults off:
```bash
export DROBOTICFLOW_OBJECT_SYNC_ENABLED=false
./scripts/render-manifests.sh > /tmp/droboticflow-poc.yaml
kubectl apply -f /tmp/droboticflow-poc.yaml
```
Deploy API, Web, and coordinator before enabling client access. The coordinator uses Mongo atomic claims and may run multiple replicas. Its read-only root filesystem has a size-limited `/tmp` volume for streaming release manifests.
Before setting the feature flag to `true`, create `drosync-signing-secret` with `DROSYNC_AUDIT_SIGNING_PRIVATE_KEY` and `DROSYNC_AUDIT_SIGNING_KEY_ID`. A missing or malformed key intentionally prevents the enabled coordinator from starting.
If the cluster has Prometheus Operator and a Grafana dashboard sidecar, install the optional SLO rules and dashboard:
```bash
./scripts/render-drosync-observability.sh | kubectl apply -f -
```
## Health and metrics
- API: `/api/health`, `/api/ready`, existing API metrics.
- Coordinator: `/health`, `/ready`, `/metrics`.
- Key counters: cycles, errors, sessions planned, items committed, staging cleaned, reconciles, releases, and daily audit seals. Cycle timestamps and duration make stalled replicas visible.
Alert when coordinator errors increase for 5 minutes, no cycle occurs for 2 minutes while enabled, oldest planning/committing item exceeds 15 minutes, staging exceeds 24 hours, audit chain continuity fails, or Mongo/BOS error rate threatens 99.9%.
## Backup check
Every shift:
1. Confirm the Mongo backup job completed within the 15-minute RPO and includes all `objectSync*` collections.
2. Restore the latest backup into an isolated database and run index creation plus audit continuity verification.
3. HEAD a sample immutable release manifest and daily audit seal in BOS.
4. Record Mongo snapshot ID, BOS version/ETag/size, test restore duration, and verifier result.
## Cleanup
Coordinator aborts expired multipart uploads and deletes staging after 24 hours. Mongo TTL indexes remove session/item after 90 days and version/audit after one year. Lease documents and latest object heads are not TTL-deleted. Business objects and conflict objects are never automatic cleanup targets.
## Rollback
Disable the feature flag, revoke device tokens, and stop new sessions. Do not delete committed objects, sessions, versions, releases, or audit records. Roll server components back independently; protocol `v1` remains the compatibility boundary.