8.5 KiB
Operations and SLO runbook — 1.0.7
POC deployment boundary and order
This checkout deploys the poc branch to namespace droboticflow, public entry 120.48.164.19. Build POC images from poc under dataflow-poc; do not merge/build from dev or deploy to Fanyu. Select the POC kubeconfig explicitly rather than relying on the current default context.
The following are deployment instructions, not evidence that this change has been deployed. With the required POC image/environment variables configured:
export DEPLOY_ENV=poc
export DROBOTICFLOW_OBJECT_SYNC_ENABLED=false
./scripts/render-manifests.sh > /tmp/droboticflow-poc.yaml
kubectl --kubeconfig '<POC_KUBECONFIG>' -n droboticflow apply -f /tmp/droboticflow-poc.yaml
Deploy API/Web/coordinator and ensure indexes first. Before enabling object sync, configure drosync-signing-secret with DROSYNC_AUDIT_SIGNING_PRIVATE_KEY and DROSYNC_AUDIT_SIGNING_KEY_ID; missing/malformed audit signing keys prevent an enabled coordinator from starting. Confirm /capabilities advertises features.sessionHeartbeat=true, features.previewSessions=true and features.credentialActivation=true, features.metadataRecovery=true, features.missingFileSkip=true, features.safeClean=true, Session heartbeat works, and status returns progress.totalItems/terminalItems before upgrading devices. Missing capabilities cause an explicit upgrade-required error, not old per-item heartbeats or repeated item-list polling.
The coordinator can run multiple replicas using atomic claims. Planning stays planning until work finishes; unique claim tokens, renewal and completion CAS reject stale owners. Planning, commit, cleanup, reconcile, release and audit loops are independent. Per replica, planning reads use up to 16 concurrent items, commit has 4 continuous workers, and staging cleanup has 2. This is bounded work concurrency, not a measured throughput guarantee or a transaction across BOS and Mongo.
If Prometheus Operator and a Grafana sidecar are present, render optional observability resources and apply them only to the selected POC cluster:
DEPLOY_ENV=poc ./scripts/render-drosync-observability.sh > /tmp/drosync-observability-poc.yaml
kubectl --kubeconfig '<POC_KUBECONFIG>' -n droboticflow apply -f /tmp/drosync-observability-poc.yaml
Health, progress and queues
- API:
/api/health,/api/ready; coordinator:/health,/ready,/metrics. - Ready means component/store readiness, not successful uploads or that the installed images match this source revision.
drosync_coordinator_cycles_total,last_cycle_timestamp_seconds,last_success_timestamp_secondsandcycle_duration_secondsnow describe commit work cycles. Continuous workers process one item per cycle; explicit ProcessOnce can process a bounded batch. Do not compare this rate directly with old monolithic-cycle counts or divide it by 16 to infer file latency.- Per-phase metrics use
phase="planning|commit|cleanup|reconcile|release|audit":phase_cycles_total,phase_errors_total,phase_item_errors_total,phase_completed_total,phase_in_flight,phase_duration_secondsandphase_duration_seconds_total. In-flight means executing work, not queued item count. Errors include item failures and relevant post-commit cleanup failures instead of silently continuing; phase duration is the last completed work cycle, not a percentile. pending_commitscounts all committing items, including claimed/in-flight work.pending_planning_sessionscounts planning sessions. They are sampled independently every 30 seconds with a 10-second query deadline; scraping metrics reads the retained in-memory snapshot.queue_sample_timestamp_seconds=0means no successful observation. A failed sample retains the prior values/time and records an error; check freshness before trusting a zero or interpreting an age.oldest_pending_age_seconds{queue="commit|planning"}is available only whenqueue_age_available{queue=...}=1. Commit age uses the recorded enqueue time, untouched by claims/renewals. Older records without it still contribute to counts but not to age. Do not interpret unavailable age as zero waiting.
All metric names above have the drosync_coordinator_ prefix. Use error increases, stale queue samples, growing queue depth/oldest age, stuck phase in-flight work, and object-store/Mongo latency together. Suggested investigation thresholds are a fresh oldest queue age over 15 minutes, error growth for 5 minutes or no commit work cycles for 2 minutes while enabled; tune alerts to load and sampling intervals. Alert rules existing in a dashboard are not proof every new metric is already monitored.
Client status is paginated local diagnostics: drosync status --limit 100, then --cursor <nextCursor>. Platform Session and selected-session conflict views paginate cloud data. They are management observations, not additional local controls. The old platform daily report/export and admin-stop routes are retired; use local web/report and abort respectively.
Recovery and source retention
On a device, preserve its state namespace and start the current daemon to migrate SQLite schema 5. Reports are read-only and reject a schema they cannot read. Current upload journals are version 3; incompatible in-flight records require confirmed remote abort and rescan. If cancellation cannot be confirmed, keep the mapping/journal and retry rather than deleting the database. A pending create whose response was lost is recovered using its exact request/key before abort.
Pause stops new dispatch; abort cancels local work and requests cloud cancellation. Mapping removal uses that cancellation path and only removes the mapping after confirmation. Neither action deletes committed cloud objects or source files. CLI sync returns nonzero for partial/failure/aborted/expired/paused (50), and conflict (30). A local report's completeness is based on verified receipts/current source identity, not a live bucket scan; missing historical identity is 待校验. Do not use a green badge or process exit alone as permission to delete sources.
Backup and cleanup
Each shift, confirm the Mongo backup met the 15-minute RPO and includes all objectSync* collections; restore an isolated copy, ensure indexes and verify audit continuity. HEAD a sample immutable release manifest and daily audit seal. Record snapshot IDs, object ETag/version/size, restore duration and verification results. Back up device state consistently; never remove live SQLite WAL/SHM to resolve an apparent lock.
The cleanup worker aborts old multipart uploads and removes eligible staging older than 24 hours. Session/item TTL is 90 days; version/audit retention is one year. Lease documents and latest object heads are not TTL-deleted. Business and keep-both conflict objects are not automatic cleanup targets. Cleanup errors remain observable, and claim loss prevents stale terminal writes.
Rollback
Stop new client work and choose server/client rollback deliberately. Do not delete committed objects, sessions, versions, releases or audit data. Restoring an older server behind current clients requires its actual heartbeat/progress contract to remain available; the v1 string alone is not sufficient and clients will not fall back. Native client rollback uses the recorded verified previous package through CLI OS authorization and must respect schema compatibility. Linux portable uses its canonical installer and does not cross into native self-update. Windows user-startup and Linux user-service modes are retired. Missing/unknown installation markers and historical partial signed-release matrices are rejected; do not restore old aliases or change markers to bypass this contract.
Current end-to-end user workflows and remaining real-environment gates: 1.0.6 closure manual.
Signed client releases additionally require API DROSYNC_UPDATE_SIGNING_PUBLIC_KEY matching the client trust root. Configure it before signed manifest registration. New task history/detail/retry endpoints and verified release/audit downloads expose asynchronous outcomes; inspect terminal status rather than treating HTTP202 as completion.
1.0.6 metadata, timeout and BOS recovery changes: repair and verification.
1.0.7 当前新增能力和配置迁移以 V1.1 需求逐项实现 和 运行配置完整清单 为准;本页引用的 1.0.6 验证数量仍仅代表历史版本。