droboticflow/docs/drosync/disaster-recovery.md
2026-07-28 20:43:46 +08:00

1.5 KiB

Disaster recovery

Target RTO is 4 hours and RPO is 15 minutes.

Mongo loss

  1. Disable the object-sync feature flag and revoke exposed credentials if compromise is suspected.
  2. Restore Mongo into an isolated instance; verify objectSync* indexes and TTL definitions.
  3. Verify the organization audit chains and compare last release/audit BOS records with restored Mongo.
  4. Mark all nonterminal sessions expired. Do not infer successful commit from an uploading/committing state alone.
  5. Reconcile every affected space against BOS object HEAD metadata.
  6. Enable internal create-only canary before general traffic.

Coordinator crash

Atomic claim leases expire and another replica resumes. Item commit is protected by the permanent fencing token and conditional target completion. Staging cleanup is safe to retry. A response-loss client queries item/session status instead of assuming failure.

Client crash

The daemon reopens WAL SQLite, runs migration/integrity checks, loads the active session journal, reuses deterministic idempotency keys, resumes known multipart parts, obtains fresh presigned URLs when needed, and queries commit state. If the local file changed, start a new manifest; never continue using the old SHA-256.

BOS outage

Pause new client sessions. Preserve staging and journals. After recovery, run capability probe only on the dedicated temporary key, then reconcile tracked heads. No automatic business-object deletion is part of recovery.