forked from longtao.wu/droboticflow
47 lines
2.5 KiB
Markdown
47 lines
2.5 KiB
Markdown
# Operations and SLO runbook
|
|
|
|
## Deployment
|
|
|
|
On `dev`, leave `DEPLOY_ENV` unset; scripts resolve to `poc`. The feature flag defaults off:
|
|
|
|
```bash
|
|
export DROBOTICFLOW_OBJECT_SYNC_ENABLED=false
|
|
./scripts/render-manifests.sh > /tmp/droboticflow-poc.yaml
|
|
kubectl apply -f /tmp/droboticflow-poc.yaml
|
|
```
|
|
|
|
Deploy API, Web, and coordinator before enabling client access. The coordinator uses Mongo atomic claims and may run multiple replicas. Its read-only root filesystem has a size-limited `/tmp` volume for streaming release manifests.
|
|
|
|
Before setting the feature flag to `true`, create `drosync-signing-secret` with `DROSYNC_AUDIT_SIGNING_PRIVATE_KEY` and `DROSYNC_AUDIT_SIGNING_KEY_ID`. A missing or malformed key intentionally prevents the enabled coordinator from starting.
|
|
|
|
If the cluster has Prometheus Operator and a Grafana dashboard sidecar, install the optional SLO rules and dashboard:
|
|
|
|
```bash
|
|
./scripts/render-drosync-observability.sh | kubectl apply -f -
|
|
```
|
|
|
|
## Health and metrics
|
|
|
|
- API: `/api/health`, `/api/ready`, existing API metrics.
|
|
- Coordinator: `/health`, `/ready`, `/metrics`.
|
|
- Key counters: cycles, errors, sessions planned, items committed, staging cleaned, reconciles, releases, and daily audit seals. Cycle timestamps and duration make stalled replicas visible.
|
|
|
|
Alert when coordinator errors increase for 5 minutes, no cycle occurs for 2 minutes while enabled, oldest planning/committing item exceeds 15 minutes, staging exceeds 24 hours, audit chain continuity fails, or Mongo/BOS error rate threatens 99.9%.
|
|
|
|
## Backup check
|
|
|
|
Every shift:
|
|
|
|
1. Confirm the Mongo backup job completed within the 15-minute RPO and includes all `objectSync*` collections.
|
|
2. Restore the latest backup into an isolated database and run index creation plus audit continuity verification.
|
|
3. HEAD a sample immutable release manifest and daily audit seal in BOS.
|
|
4. Record Mongo snapshot ID, BOS version/ETag/size, test restore duration, and verifier result.
|
|
|
|
## Cleanup
|
|
|
|
Coordinator aborts expired multipart uploads and deletes staging after 24 hours. Mongo TTL indexes remove session/item after 90 days and version/audit after one year. Lease documents and latest object heads are not TTL-deleted. Business objects and conflict objects are never automatic cleanup targets.
|
|
|
|
## Rollback
|
|
|
|
Disable the feature flag, revoke device tokens, and stop new sessions. Do not delete committed objects, sessions, versions, releases, or audit records. Roll server components back independently; protocol `v1` remains the compatibility boundary.
|