✨ Verify legacy SHA256-only archives and backfill their MD5 server-side

Archived heads from protocol v1 carry only a SHA-256 receipt, so every sync
that still holds those files ends partial and they can never be cleaned.

Add an administrator-approved digest backfill job run by the coordinator.
It reads each legacy object with conditional (If-Match) ranged GETs, hashes
MD5 and SHA-256 in one pass, cross-checks every surviving receipt (retired
head sha256, drosync object metadata, BOS stored digests), and records the
MD5 on the head with a compare-and-set write, a bound attestation and an
audit event. It never writes to BOS and never accepts a client digest.

The cleanup proof now accepts an object without drosync-md5sum metadata
only through an attestation bound to its live ETag, version and size. The
next ordinary sync plans those files unchanged, completes, and they become
cleanable; closed partial sessions are left untouched.

Includes the design doc comparing remediation options, store/coordinator/
API unit tests and an end-to-end recovery test on MemoryStore and fake
object storage. Nothing is deployed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
eust-w 2026-10-30 15:44:14 +08:00
parent 143a24b1e9
commit a56882bedc
21 changed files with 2733 additions and 25 deletions

View File

@ -104,6 +104,7 @@ func main() {
"drosync_coordinator_reconciles_total %d\n"+
"drosync_coordinator_releases_total %d\n"+
"drosync_coordinator_audit_seals_total %d\n"+
"drosync_coordinator_digest_backfills_total %d\n"+
"drosync_coordinator_last_cycle_timestamp_seconds %d\n"+
"drosync_coordinator_last_success_timestamp_seconds %d\n"+
"drosync_coordinator_cycle_duration_seconds %.3f\n",
@ -116,6 +117,7 @@ func main() {
metrics.Reconciled,
metrics.Released,
metrics.Sealed,
metrics.Backfilled,
metrics.LastCycle,
metrics.LastOK,
float64(metrics.DurationMS)/1000,
@ -133,7 +135,7 @@ func main() {
_, _ = fmt.Fprintf(writer, "drosync_coordinator_queue_age_available{queue=%q} 0\n", name)
}
}
for _, name := range []string{"planning", "commit", "cleanup", "reconcile", "release", "audit"} {
for _, name := range []string{"planning", "commit", "cleanup", "reconcile", "release", "audit", "digest_backfill"} {
phase := metrics.Phases[name]
_, _ = fmt.Fprintf(writer,
"drosync_coordinator_phase_cycles_total{phase=%q} %d\n"+

View File

@ -0,0 +1,156 @@
# drosync 旧 SHA256 归档的 MD5 核验补录设计
日期:2026-10-30(北京时间)。基线:`poc` 分支 `143a24b1`。本次只有设计与原型,未部署,未读写 POC Mongo、真实 BOS 或采集机数据,也未构建发布包。
## 结论
推荐方案:**由管理员批准、在协调器内执行的流式核验补录(digest backfill)**。
对缺少可信 MD5 回执的归档对象,协调器先做 HEAD,再按 ETag 发起分段条件 GET(`If-Match`),读取一遍同时算出 MD5 和 SHA-256。随后与仍然存在的旧回执交叉核验:Mongo head 的 `sha256`、对象元数据 `drosync-sha256`/`drosync-md5sum`、BOS 自身记录的摘要。全部一致才用 CAS 写入 MD5,并附上绑定该对象 ETag、版本和大小的 attestation,同时写入审计链。
整个过程不写 BOS,也不接受任何客户端提交的摘要。补录后,设备下一次普通同步会把这些文件规划为 `unchanged`,Session 进入 `completed`,文件即可逐个安全清理。
## 1. 现状与根因
- 内容协议 v2 只用 MD5(见[仅 MD5 校验](drosync仅MD5校验-2026-10-12.md))。`9a5f9ed5` 把回执字段从 BSON `sha256` 改为 `md5sum`;`poc-schema-cutover` 不处理 `objectSyncHeads`。按代码推断,POC 中 v1 head 仍是“有 `sha256`、无 `md5sum`”:Go 结构体不再解码前者,读出来的 `MD5Sum` 为空。这一点未在 POC 数据上直接核对。
- 规划逻辑遇到 `len(head.MD5Sum) != 32` 就返回 `deny_archived_hash_change`,并保留云端原文件。回归测试 `TestArchivePlanningRetainsLegacyObjectsWithoutMD5` 覆盖了缺失、格式错误和只有 SHA256 三种形态。
- 现场证据(见 [1.1.2 partial 诊断](drosync-1.1.2-partial修复与验证-2026-10-22.md)):mapping `a75fa7d2…` 共有 188 个 partial Session,每轮 denied=4198(MCAP、episode_meta.json)。后果如下:
- 这些文件永远无法清理,采集机磁盘持续增长;
- 每轮同步都是 partial,而 confirmed skip 只认 completed 回执,所以每轮都要重新规划全部文件;
- 新的同步会取代旧回执,在旧 Session 上执行 clean 会因“newer source receipt”保留文件。
- 补充发现:即使 head 有了 MD5,清理证明仍要求实时 HEAD 返回的 `drosync-md5sum` 元数据与条目 MD5 相等,而 v1 对象只有 `drosync-sha256`。因此补救方案还必须规定:在不改写对象的前提下,清理证明如何认可服务端核验出的证据。
## 2. 不可打破的约束
1. 本地计算的 MD5 永远不能成为云端证据,只能拿来和云端证据比较。
2. 不覆盖、不改写已归档对象,数据面只读。
3. 每条补录都绑定唯一的对象版本,并且可审计、幂等、可中断续跑。
4. 任何不确定都按失败关闭处理:保留本地文件,保留云端对象,条目维持 denied。
5. 不改写已关闭的 Session 和条目,历史保持原样。
## 3. 方案比较
| 方案 | 安全性 | 大桶成本 | 幂等 | 受影响文件如何恢复 | 结论 |
|---|---|---|---|---|---|
| A 协调器流式核验补录 | MD5 由服务端从云端字节算出;用旧 SHA-256 回执交叉核验;If-Match、HEAD 和 CAS 绑定对象版本;对 BOS 只读 | 每个对象成功读取一次;可按 Session 定向;有预算上限 | CAS 加确定性审计键,可重跑 | 下一次普通同步即 completed,可以清理 | **推荐** |
| B 管理员批准的重新归档/覆盖(设备重传加 `force_overwrite`) | 用设备当前文件覆盖正式对象:本地若已被改写(同尺寸改写、截断)就会替换原归档;要求 `update_enabled` 和独占前缀,POC 为 `create_only`;已发布清单引用的 ETag 会失效 | 采集机上传全部旧文件,正好消耗最紧缺的设备带宽 | 每次都生成新版本,难以幂等 | 需逐文件人工批准,无法扩展 | 不采用;仅留作真实冲突的个案处理 |
| C 另存副本(keep-both 重传) | 原 key 仍缺 MD5;命名空间被冲突副本污染 | 存储翻倍,设备重传 | 一般 | 只有新 key 可清理,原问题仍在 | 不采用 |
| D 客户端双哈希(设备补算 SHA-256 与旧回执比较) | 推翻“SHA-256 退出内容比较与清理”的既定决策;证据仍来自设备 | 云端零读取,但每台设备每轮都要多读一遍文件 | 无法补录:写入设备 MD5 被禁止,head 永远缺 MD5 | 需要发版和回滚到全部采集机,且每轮都要双哈希 | 不采用 |
| E 直接把 BOS ETag 当 MD5 | 只有简单 PutObject 的 ETag 等于 MD5;v1 正式对象大多由分段复制发布,ETag 不是内容 MD5,无法可靠区分 | 零读取 | — | — | 不采用;至多作为后续优化的预筛选 |
| F 放宽策略:旧对象 denied 不计 partial | 掩盖真实状态;文件仍不可清理 | 零 | — | Session 变 completed,但磁盘问题仍在 | 不采用 |
## 4. 推荐方案设计
### 4.1 触发与范围
- `POST /api/object-sync/spaces/{id}/digest-backfills`:仅管理员可调用,必须带幂等键和 `reason`(作为批准理由写入审计 `space.digest_backfill`)。三种范围恰好选一种:
- `sessionIds`:至多 100 个,**推荐**。传入每个受影响 mapping 最新的 partial Session,只核验其中 `deny_archived_hash_change` 条目指向的 key;
- `keyPrefix`:按空间内相对目录前缀核验;
- `wholeSpace`:核验整个空间。
- 预算:`maxObjects` 默认 1 万、上限 100 万;`maxBytes` 默认 1 TiB、上限 1 PiB。`allowUnanchored` 默认 false。
- `GET /api/object-sync/digest-backfills/{id}` 返回状态、计数、`budgetExhausted`、断点和至多 200 条逐 key 例外。
### 4.2 单个对象的核验步骤
1. head 已有合法 MD5:如果是上传回执,跳过(`notLegacy`);如果是已核验的,只补写审计(`alreadyAttested`)。
2. head 被 reconcile 标记为 drifted,或回执没有 ETag:记为例外,不读取。
3. 取得对象键租约。这与上传和清理证明用的是同一把锁(`ObjectLeaseResourceHash`);被占用则记为 `busy`,下次重跑再处理。
4. HEAD 检查对象存在,且 ETag、大小、版本都与回执一致,否则记为 `drifted`。`ARCHIVE` 存储类型需要先解冻,直接跳过。
5. 收集信任锚:head 的旧 `sha256`(或误存进 `md5sum` 的 64 位摘要)、元数据 `drosync-sha256`/`drosync-md5sum`、BOS 的 `x-bce-content-sha256`/`Content-MD5`。所有锚必须一致;drosync 自己写的锚格式错误时按失败关闭处理。没有任何 SHA-256 锚时记为 `unanchored`,只有管理员显式设置 `allowUnanchored` 才继续,此时只靠 ETag/版本/大小绑定,attestation 的锚记为 `none_admin_waived`。
6. 检查预算。
7. 按 64 MiB 分段 GET,每段都带 `If-Match: <回执 ETag>`,一遍同时计算 MD5 和 SHA-256。遇到 412 或段内 ETag 变化记为 `drifted`,短读或网络错误记为 `read_failed`。
8. 计算结果与锚比对,不一致记为 `content_mismatch`,写审计 `head.md5.rejected`,head 不动。
9. 再做一次 HEAD,确认对象身份没有变化。
10. CAS 写 head:ETag、版本、大小、`platformVersion`、原摘要、旧 `sha256`、session/item 都没变,且未 drifted、未核验过,才写入 `md5sum` 和 `md5Attestation`。attestation 记录来源、作业、批准人、协调器、ETag/版本/大小、MD5/SHA-256、锚和原摘要。`platformVersion` 不变,因为内容没有变。
11. 写审计 `head.md5.attested`,其 eventKey 由对象身份和 MD5 派生。
### 4.3 清理证明如何认可
对象元数据里有 `drosync-md5sum` 时,仍必须与条目 MD5 相等。没有时,只有同时满足以下条件才放行:head 的 attestation 来源为 `coordinator_stream`、带有锚,并且其 MD5/ETag/版本/大小与 head 一致;head 本身又已经与实时 HEAD 逐项一致。
伪造设备来源、ETag/版本/大小过期、MD5 不一致、没有锚,都会被拒绝。证明响应新增 `md5Evidence`(`upload_receipt` 或 `coordinator_attestation`),客户端会忽略未知字段,无需升级。
### 4.4 安全性
- **证据来源**:MD5 由协调器从云端字节算出。SHA-256 锚把它绑定到 v1 当时的上传回执;If-Match、前后两次 HEAD 和 CAS 把它绑定到唯一的对象版本。API 请求体没有任何摘要字段。测试中设备报告的 MD5 与归档不同时,head 记录的是归档字节的 MD5。
- **只读**:只发 HEAD 和 GET;单元测试和端到端测试都断言对 BOS 零写入。
- **并发**:靠对象租约加 CAS。新的 commit 会替换回执:Mongo 的 head upsert 现在会 `$unset` 新回执里没有的 `sha256`/`md5Attestation`;即使有残留,attestation 绑定的是旧 ETag,也不会被误用。
- **已知交互**:reconcile 读出整页 head 后整文档回写,可能把刚补录的 MD5 改回旧值。这是失败关闭的:条目重新 denied,重跑补录即可。这是 reconcile 原有的问题,它同样可能覆盖并发 commit 的新回执,列为后续修复。
### 4.5 大桶成本
- 读取量等于被核验对象的总字节数,每个对象只成功读取一次。Session 范围只读设备仍持有、且被拒绝的对象。每个对象的请求数为 2 次 HEAD 加 ⌈大小 / 64 MiB⌉ 次 GET。
- 估算:4198 个对象 × 平均 200 MiB ≈ 820 GiB。按 100 MiB/s 约 2.3 小时,按 30 MiB/s 约 7.8 小时,实际以网络为准。每个协调器副本同一时间只读一个对象,有独立的阶段循环,不占用 commit 和 planning 线程。
- 费用:协调器若经同地域内网访问 BOS,读取不产生公网流出流量费;若走公网 endpoint,则按流出流量计费,大批量运行前必须确认。低频和冷存储有取回费;归档存储会被跳过。MD5 和 SHA-256 在同一遍读取中计算,瓶颈在网络。
- 预算用尽时作业以 `completed` 结束并标记 `budgetExhausted=true`。用同一范围新建作业即可续跑,已核验的对象会被快速跳过。
### 4.6 幂等与故障恢复
- 已核验的 head 不会再次读取;CAS 防止重复写入;审计事件按确定性 eventKey 去重。
- 作业在每读完一个对象和每翻一页后保存断点。协调器崩溃时其他副本按 claim 接管,从断点继续;中断的那个对象从头重读。计数是至少一次。
- 审计写入失败时作业进入 `failed`。重跑时,已核验路径会用同一 eventKey 补写事件。
### 4.7 受影响的 Session 和文件如何恢复
- 历史不改写:旧 partial Session 和其中的 denied 条目保持原样。
- 下一次普通同步(daemon 定时触发或 `drosync sync`):本地 MD5 和大小与核验结果一致的文件会规划为 `unchanged`,`expectedBase` 指向同一对象版本;没有其他问题时 Session 即 `completed`。
- Session completed 之后:confirmed skip 生效,后续同步在本地直接跳过这些文件;`drosync clean` 使用新的 completed Session,逐文件证明通过。旧 Session 仍会因“newer source receipt”保留文件,这符合预期。
- 本地内容与归档不同的文件(例如重新生成的日报)会变成真实的“同名文件内容冲突”,需要人工决定是另存、排除还是覆盖。作业的 `itemsConflicting` 会提前给出数量。
- 云端字节与旧回执不一致的(`content_mismatch`)保持 denied,需要数据负责人介入。
## 5. 原型(本分支)
| 位置 | 内容 |
|---|---|
| `internal/domain/object_sync.go` | head 新增 `LegacySHA256`(只读解码 BSON `sha256`)和 `MD5Attestation`;新增作业记录 `ObjectSyncDigestBackfillRecord` |
| `internal/store/object_sync_digest_backfill.go` | 内存与 Mongo 实现:作业的 claim/续租/受控保存,head 的 CAS `AttestObjectSyncHeadMD5`;Mongo 新集合 `objectSyncDigestBackfills` 及索引;head upsert 会清除过期锚 |
| `internal/objectsync/digest_backfill.go` | 新协调器阶段 `digest_backfill`、单对象核验、锚交叉核验、审计;`bos.go` 新增条件分段 GET,HEAD 带回存储摘要和存储类型 |
| `internal/api/object_sync_digest_backfill.go` | 管理员创建/查询路由;清理证明认可已绑定的 attestation |
| `docs/drosync/openapi.yaml`、`audit-dictionary.md` | 新路由、`md5Evidence` 字段和新审计动作 |
客户端(Rust)无需改动:被拒绝的文件本来每轮都会重新进入清单,补录后自然规划为 `unchanged`。
测试只用 MemoryStore 和假对象存储:
- `internal/objectsync/digest_backfill_test.go`:
- 四种旧回执形态;
- 分段 If-Match 读取;
- 内容与旧回执不符;
- 无锚与管理员豁免;
- 锚互相矛盾;
- ETag/大小/版本变化、对象丢失、已 drift、无 ETag、ARCHIVE、租约占用、读取中被替换、短读、写入前回执变化;
- 幂等与审计补写;
- 预算;
- Session 范围(每个 key 只读一次,记录的是云端字节而非设备摘要);
- 前缀范围;
- claim 丢失;
- 补录后的规划结果。
- `internal/store/object_sync_digest_backfill_test.go`:
- CAS 的每个比较字段;
- 旧 BSON `sha256` 解码;
- Mongo 过滤器对缺失字段的匹配;
- upsert 时 `$unset`;
- 作业的 claim 与写入保护。
- `internal/api/object_sync_digest_backfill_test.go`:
- 路由校验与审计;
- 清理证明的 7 种拒绝情形;
- 端到端 `TestLegacyArchiveSyncCompletesAndCleansAfterDigestBackfill`:通过真实 API 路由,补录前同步为 partial 且不可清理;管理员批准补录后,下一次普通同步为 completed,每个文件都拿到清理证明(`md5Evidence=coordinator_attestation`),旧 Session 保持不变,对 BOS 零写入。
- 变异检查:分别移除清理证明对 attestation 的认可、SHA-256 锚比较、ETag 绑定、CAS 的 platformVersion 比较、无锚拒绝、读取前身份检查、锚冲突检查、审计补写,均有测试失败。
## 6. 未包含与后续
- 真实 BOS 上仍需验证:条件分段 GET 的 412 行为,HEAD 返回的 `Content-MD5`、`x-bce-content-sha256` 和存储类型的格式,内网 endpoint 的计费;还要在 POC 数据上只读确认 v1 head 的实际形态。Mongo 侧的 CAS 过滤器、`$unset` 和新集合目前只有纯函数测试,需要 Mongo 集成测试。
- reconcile 改为按 ETag 条件的字段级 drift 写入,同时修复它覆盖并发 commit 的问题。
- 作业列表/重试 API、管理界面、可选的双人审批;观察成本后,可考虑“规划时按需自动排队核验”。
- 没有 head 的 `deny_existing_untracked` 对象需要“认领”,风险更高,不在本方案范围内。
## 7. 部署后的操作步骤
1. 只读找出受影响 mapping 最新的 partial Session(拒绝原因为“已归档对象缺少 MD5 回执”)。
2. 用 `POST /api/object-sync/spaces/{spaceId}/digest-backfills` 提交 `{"reason": "...", "sessionIds": ["<最新 partial>"]}`,必要时调大 `maxBytes`。
3. 用 `GET /api/object-sync/digest-backfills/{id}` 等待作业 `completed`,逐条处理 `mismatched`、`drifted`、`unanchored`、`busy` 和 `budgetExhausted`。
4. 设备正常同步,得到新的 completed Session,先用该 Session 做 `drosync clean` 预览,再确认执行。
## 8. 验证
`go test ./internal/api ./internal/objectsync ./internal/store -count=1` 全部通过;`go vet` 通过;`scripts/generate_drosync_contract.py --check` 无差异。

View File

@ -17,6 +17,8 @@ Each audit event contains:
| `previousHash`, `eventHash` | Organization hash-chain link and SHA-256 hash of the canonical event payload. |
| `occurredAt` | UTC RFC3339 timestamp. |
Legacy digest backfill records four actions. `space.digest_backfill` (user, `result=accepted`) is the administrator approval with scope, budgets and `reason`. `head.md5.attested` (system) records each archived head whose MD5 the coordinator computed by reading the object: `before` holds the previous digest and the retired v1 SHA-256, `after` the MD5, SHA-256, ETag, version, size, platform version, the receipts it was checked against and the approver. Its `eventKey` is derived from the attested object identity, so a rerun that repairs a missing event cannot duplicate it. `head.md5.rejected` (system, `result=denied`) records archived bytes or receipts that disagree; the head is left unchanged. `space.digest_backfill.completed` (system) carries the job counters.
An organization-scoped fencing lease serializes hash-chain append across API replicas. Events retain for one year. Export endpoints are read-only and available as CSV or NDJSON.
The coordinator verifies every event hash and intra-day chain link, then conditionally writes the previous UTC day's count, first/last event hash, aggregate digest, Ed25519 signature, and verifier key ID to `.drosync-internal/audit/{organizationId}/{YYYY-MM-DD}.json`. Mongo stores the BOS ETag/version ID and seal index for one year. `GET /api/object-sync/audit-seals/{day}` returns the index record.

View File

@ -121,6 +121,59 @@ paths:
drifted: { type: integer, format: int64 }
default:
$ref: "#/components/responses/Error"
/api/object-sync/spaces/{id}/digest-backfills:
post:
tags: [management]
operationId: createDigestBackfill
description: |
Administrator-approved verification of archived heads that lack a trusted
MD5 receipt (v1 SHA-256-only, missing or malformed). The coordinator reads
each archived object with conditional ranged GETs, checks the computed
SHA-256/MD5 against every surviving receipt, and records the MD5 with a
bound attestation and audit event. The request never carries a digest and
BOS objects are never written. Exactly one scope is required.
parameters:
- $ref: "#/components/parameters/ResourceID"
- $ref: "#/components/parameters/IdempotencyKey"
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [reason]
properties:
reason: { type: string, maxLength: 1000 }
sessionIds: { type: array, maxItems: 100, items: { type: string }, description: "Verify keys denied with deny_archived_hash_change in these sessions." }
keyPrefix: { type: string, description: "Relative directory prefix inside the sync space." }
wholeSpace: { type: boolean }
allowUnanchored: { type: boolean, default: false, description: "Also attest heads with no surviving SHA-256 receipt, bound only to ETag/version/size." }
maxObjects: { type: integer, format: int64, default: 10000, maximum: 1000000 }
maxBytes: { type: integer, format: int64, default: 1099511627776 }
responses:
"202":
description: Verification accepted.
content:
application/json:
schema:
$ref: "#/components/schemas/DigestBackfill"
default:
$ref: "#/components/responses/Error"
/api/object-sync/digest-backfills/{id}:
get:
tags: [management]
operationId: getDigestBackfill
parameters:
- $ref: "#/components/parameters/ResourceID"
responses:
"200":
description: Durable progress, counters and per-key exceptions.
content:
application/json:
schema:
$ref: "#/components/schemas/DigestBackfill"
default:
$ref: "#/components/responses/Error"
/api/object-sync/devices:
get:
tags: [management]
@ -808,7 +861,9 @@ paths:
Device-scoped, rate-limited and never idempotency-cached. Requires an owned
completed or partial non-preview Session, a committed/unchanged item with a
complete MD5 receipt, matching current platform head and live BOS HEAD
(MD5 metadata, size, ETag and version). Other items remain ineligible;
(size, ETag, version, and MD5 metadata; a v1 archive without MD5 metadata
qualifies only through a coordinator digest attestation bound to that same
ETag, version and size). Other items remain ineligible;
issuing a proof never changes the Session or item status.
Acquires a short object lease. The local daemon must independently prove
the current source identity/content before deleting anything.
@ -834,6 +889,7 @@ paths:
relativePath: { type: string }
targetKey: { type: string }
md5sum: { type: string }
md5Evidence: { type: string, enum: [upload_receipt, coordinator_attestation] }
sizeBytes: { type: integer, format: int64 }
eTag: { type: string }
versionId: { type: string }
@ -1943,6 +1999,53 @@ components:
drifted:
type: integer
format: int64
DigestBackfill:
type: object
properties:
id: { type: string }
operationId: { type: string }
syncSpaceId: { type: string }
status: { type: string, enum: [pending, running, completed, failed] }
reason: { type: string }
sessionIds: { type: array, items: { type: string } }
keyPrefix: { type: string }
wholeSpace: { type: boolean }
allowUnanchored: { type: boolean }
maxObjects: { type: integer, format: int64 }
maxBytes: { type: integer, format: int64 }
budgetExhausted: { type: boolean }
counts:
type: object
description: At-least-once counters; a resumed job may revisit its last page.
properties:
scanned: { type: integer, format: int64 }
attested: { type: integer, format: int64 }
alreadyAttested: { type: integer, format: int64 }
notLegacy: { type: integer, format: int64 }
unanchored: { type: integer, format: int64 }
mismatched: { type: integer, format: int64 }
drifted: { type: integer, format: int64 }
busy: { type: integer, format: int64 }
readFailed: { type: integer, format: int64 }
objectsRead: { type: integer, format: int64 }
bytesRead: { type: integer, format: int64 }
itemsMatched: { type: integer, format: int64 }
itemsConflicting: { type: integer, format: int64 }
exceptions:
type: array
maxItems: 200
items:
type: object
properties:
key: { type: string }
outcome: { type: string, enum: [unanchored, anchor_conflict, content_mismatch, drifted, unbound_receipt, busy, storage_class_unreadable, read_failed, budget_exhausted, session_unavailable] }
detail: { type: string }
at: { type: string, format: date-time }
error: { type: string }
createdBy: { type: string }
createdAt: { type: string }
startedAt: { type: string }
completedAt: { type: string }
DataRelease:
type: object
properties:

View File

@ -2073,8 +2073,7 @@ func abortObjectSyncSession(ctx context.Context, target store.ObjectSyncStore, s
}
func objectSyncLeaseResourceHash(space domain.ObjectSyncSpaceRecord, key string) string {
sum := sha256.Sum256([]byte(space.ConnectionID + "\x00" + space.Bucket + "\x00" + key))
return hex.EncodeToString(sum[:])
return objectsync.ObjectLeaseResourceHash(space, key)
}
func (server *server) handleExternalObjectSyncItemReserve(writer http.ResponseWriter, request *http.Request) {

View File

@ -119,7 +119,7 @@ func (server *server) handleExternalObjectSyncCleanupProof(writer http.ResponseW
renderObjectSyncError(writer, request, 409, "cleanup_cloud_unverified", "Cloud object is missing or HEAD verification failed; local file retained", true, nil)
return
}
if remote.SizeBytes != item.SizeBytes || remote.ETag != head.RemoteETag || remote.VersionID != head.VersionID || objectSyncMetadataValue(remote.Metadata, "drosync-md5sum") != item.MD5Sum {
if remote.SizeBytes != item.SizeBytes || remote.ETag != head.RemoteETag || remote.VersionID != head.VersionID || !objectSyncCloudMD5Matches(remote, *head, item.MD5Sum) {
renderObjectSyncError(writer, request, 409, "cleanup_cloud_changed", "Cloud object content or identity changed; local file retained", false, nil)
return
}
@ -129,5 +129,5 @@ func (server *server) handleExternalObjectSyncCleanupProof(writer http.ResponseW
return
}
success = true
renderJSON(writer, 200, map[string]any{"sessionId": session.ID, "mappingId": session.MappingID, "syncSpaceId": session.SyncSpaceID, "credentialId": session.CredentialID, "deviceId": session.DeviceID, "itemId": item.ID, "relativePath": item.RelativePath, "targetKey": item.TargetKey, "md5sum": item.MD5Sum, "sizeBytes": item.SizeBytes, "eTag": remote.ETag, "versionId": remote.VersionID, "verifiedAt": nowISO(), "expiresAt": lease.ExpiresAt, "leaseId": lease.ID, "leaseOwner": lease.OwnerItemID, "fencingToken": lease.FencingToken})
renderJSON(writer, 200, map[string]any{"sessionId": session.ID, "mappingId": session.MappingID, "syncSpaceId": session.SyncSpaceID, "credentialId": session.CredentialID, "deviceId": session.DeviceID, "itemId": item.ID, "relativePath": item.RelativePath, "targetKey": item.TargetKey, "md5sum": item.MD5Sum, "md5Evidence": objectSyncMD5Evidence(*head), "sizeBytes": item.SizeBytes, "eTag": remote.ETag, "versionId": remote.VersionID, "verifiedAt": nowISO(), "expiresAt": lease.ExpiresAt, "leaseId": lease.ID, "leaseOwner": lease.OwnerItemID, "fencingToken": lease.FencingToken})
}

View File

@ -0,0 +1,179 @@
package api
import (
"net/http"
"slices"
"strings"
"droboticflow/internal/domain"
"droboticflow/internal/objectsync"
"github.com/go-chi/chi/v5"
)
const (
objectSyncBackfillDefaultMaxObjects = int64(10_000)
objectSyncBackfillDefaultMaxBytes = int64(1) << 40
objectSyncBackfillMaxObjects = int64(1_000_000)
objectSyncBackfillMaxBytes = int64(1) << 50
objectSyncBackfillMaxSessions = 100
objectSyncBackfillMaxReasonBytes = 1000
)
// A digest backfill is the only way an archived legacy head gains an MD5. The
// request names what to verify, never a digest: the coordinator streams the
// archived object and computes it, so device-reported MD5 cannot become cloud
// evidence through this route.
func (server *server) handleObjectSyncDigestBackfillCreate(writer http.ResponseWriter, request *http.Request) {
target, ok := server.objectSyncStore(writer, request)
if !ok {
return
}
principal := objectSyncManagementPrincipal(request)
space, err := target.GetObjectSyncSpace(request.Context(), chi.URLParam(request, "id"))
if err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "space_lookup_failed", err.Error(), true, nil)
return
}
if space == nil || space.OrganizationID != principal.User.OrganizationID {
renderObjectSyncError(writer, request, http.StatusNotFound, "space_not_found", "Object sync space not found", false, nil)
return
}
var input struct {
Reason string `json:"reason"`
SessionIDs []string `json:"sessionIds"`
KeyPrefix string `json:"keyPrefix"`
WholeSpace bool `json:"wholeSpace"`
AllowUnanchored bool `json:"allowUnanchored"`
MaxObjects int64 `json:"maxObjects"`
MaxBytes int64 `json:"maxBytes"`
}
if !decodeJSON(writer, request, &input) {
return
}
reason := strings.TrimSpace(input.Reason)
if reason == "" || len(reason) > objectSyncBackfillMaxReasonBytes {
renderObjectSyncError(writer, request, http.StatusBadRequest, "backfill_reason_required", "An approval reason of at most 1000 bytes is required", false, nil)
return
}
sessionIDs := make([]string, 0, len(input.SessionIDs))
for _, id := range input.SessionIDs {
if id = strings.TrimSpace(id); id != "" && !slices.Contains(sessionIDs, id) {
sessionIDs = append(sessionIDs, id)
}
}
keyPrefix := strings.TrimSpace(input.KeyPrefix)
scopes := 0
for _, selected := range []bool{len(sessionIDs) > 0, keyPrefix != "", input.WholeSpace} {
if selected {
scopes++
}
}
if scopes != 1 || len(sessionIDs) > objectSyncBackfillMaxSessions {
renderObjectSyncError(writer, request, http.StatusBadRequest, "backfill_scope_invalid", "Choose exactly one scope: 1-100 sessionIds, a keyPrefix, or wholeSpace", false, nil)
return
}
if keyPrefix != "" {
normalized, pathErr := normalizeObjectSyncRelativePath(strings.TrimSuffix(keyPrefix, "/"))
if pathErr != nil {
renderObjectSyncError(writer, request, http.StatusBadRequest, "backfill_prefix_invalid", pathErr.Error(), false, nil)
return
}
keyPrefix = normalized + "/"
}
for _, id := range sessionIDs {
session, err := target.GetObjectSyncSession(request.Context(), id)
if err != nil {
renderObjectSyncError(writer, request, http.StatusServiceUnavailable, "session_lookup_failed", "Could not read session state", true, nil)
return
}
if session == nil || session.OrganizationID != space.OrganizationID || session.SyncSpaceID != space.ID {
renderObjectSyncError(writer, request, http.StatusBadRequest, "backfill_session_invalid", "Every session must belong to this sync space", false, map[string]any{"sessionId": id})
return
}
}
maxObjects, maxBytes := input.MaxObjects, input.MaxBytes
if maxObjects == 0 {
maxObjects = objectSyncBackfillDefaultMaxObjects
}
if maxBytes == 0 {
maxBytes = objectSyncBackfillDefaultMaxBytes
}
if maxObjects < 0 || maxObjects > objectSyncBackfillMaxObjects || maxBytes < 0 || maxBytes > objectSyncBackfillMaxBytes {
renderObjectSyncError(writer, request, http.StatusBadRequest, "backfill_budget_invalid", "maxObjects must be 1-1000000 and maxBytes 1 byte to 1 PiB", false, nil)
return
}
now := nowISO()
job := domain.ObjectSyncDigestBackfillRecord{
ID: objectSyncNewID("osdb"),
OperationID: objectSyncNewID("op"),
OrganizationID: space.OrganizationID,
SyncSpaceID: space.ID,
Status: "pending",
Reason: reason,
SessionIDs: sessionIDs,
KeyPrefix: keyPrefix,
WholeSpace: input.WholeSpace,
AllowUnanchored: input.AllowUnanchored,
MaxObjects: maxObjects,
MaxBytes: maxBytes,
CreatedBy: principal.User.ID,
CreatedAt: now,
UpdatedAt: now,
}
if err := target.UpsertObjectSyncDigestBackfill(request.Context(), job); err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "backfill_enqueue_failed", err.Error(), true, nil)
return
}
if !server.requireObjectSyncAudit(writer, request, target, domain.ObjectSyncAuditEvent{
OrganizationID: space.OrganizationID,
ActorType: "user",
ActorID: principal.User.ID,
RequestID: objectSyncRequestID(request),
Action: "space.digest_backfill",
Target: map[string]any{"spaceId": space.ID, "backfillId": job.ID, "operationId": job.OperationID},
After: map[string]any{
"sessionIds": job.SessionIDs, "keyPrefix": job.KeyPrefix, "wholeSpace": job.WholeSpace,
"allowUnanchored": job.AllowUnanchored, "maxObjects": job.MaxObjects, "maxBytes": job.MaxBytes,
},
Result: "accepted",
Reason: reason,
}) {
return
}
renderJSON(writer, http.StatusAccepted, job)
}
func (server *server) handleObjectSyncDigestBackfillGet(writer http.ResponseWriter, request *http.Request) {
target, ok := server.objectSyncStore(writer, request)
if !ok {
return
}
job, err := target.GetObjectSyncDigestBackfill(request.Context(), chi.URLParam(request, "id"))
if err != nil {
renderObjectSyncError(writer, request, http.StatusServiceUnavailable, "backfill_lookup_failed", "Could not read digest backfill", true, nil)
return
}
if job == nil || job.OrganizationID != objectSyncManagementPrincipal(request).User.OrganizationID {
renderObjectSyncError(writer, request, http.StatusNotFound, "backfill_not_found", "Digest backfill not found", false, nil)
return
}
renderJSON(writer, http.StatusOK, job)
}
// Upload receipts carry their MD5 in object metadata. A legacy archive has an
// MD5 only through a coordinator attestation bound to this exact object version;
// the caller has already matched the live object's ETag, version and size.
func objectSyncCloudMD5Matches(remote domain.ObjectStorageObjectSummary, head domain.ObjectSyncObjectHeadRecord, md5sum string) bool {
if declared := objectSyncMetadataValue(remote.Metadata, "drosync-md5sum"); declared != "" {
return declared == md5sum
}
return objectsync.AttestationBindsHead(head) && head.MD5Sum == md5sum
}
func objectSyncMD5Evidence(head domain.ObjectSyncObjectHeadRecord) string {
if objectsync.AttestationBindsHead(head) {
return "coordinator_attestation"
}
return "upload_receipt"
}

View File

@ -0,0 +1,361 @@
package api
import (
"bytes"
"context"
"crypto/md5"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"droboticflow/internal/domain"
"droboticflow/internal/objectsync"
"github.com/baidubce/bce-sdk-go/bce"
)
func TestDigestBackfillCreateIsAdminScopedAndAudited(t *testing.T) {
handler, data := objectSyncTestHandler(t, true)
org := domain.DefaultOrganizationID
for _, space := range []domain.ObjectSyncSpaceRecord{{ID: "space", OrganizationID: org, ConnectionID: "bos-connection", Bucket: "bucket"}, {ID: "other-space", OrganizationID: org, ConnectionID: "bos-connection", Bucket: "bucket"}} {
if err := data.UpsertObjectSyncSpace(t.Context(), space); err != nil {
t.Fatal(err)
}
}
for _, session := range []domain.ObjectSyncSessionRecord{{ID: "partial", OrganizationID: org, SyncSpaceID: "space", Status: domain.ObjectSyncSessionPartial}, {ID: "elsewhere", OrganizationID: org, SyncSpaceID: "other-space", Status: domain.ObjectSyncSessionPartial}} {
if err := data.UpsertObjectSyncSession(t.Context(), session); err != nil {
t.Fatal(err)
}
}
create := func(space string, body map[string]any, key string) *httptest.ResponseRecorder {
return objectSyncRequest(t, handler, http.MethodPost, "/api/object-sync/spaces/"+space+"/digest-backfills", body, key)
}
for name, test := range map[string]struct {
space string
body map[string]any
code string
}{
"missing-reason": {"space", map[string]any{"wholeSpace": true}, "backfill_reason_required"},
"no-scope": {"space", map[string]any{"reason": "verify"}, "backfill_scope_invalid"},
"two-scopes": {"space", map[string]any{"reason": "verify", "wholeSpace": true, "keyPrefix": "node-04"}, "backfill_scope_invalid"},
"foreign-session": {"space", map[string]any{"reason": "verify", "sessionIds": []string{"partial", "elsewhere"}}, "backfill_session_invalid"},
"bad-prefix": {"space", map[string]any{"reason": "verify", "keyPrefix": "../node-04"}, "backfill_prefix_invalid"},
"bad-budget": {"space", map[string]any{"reason": "verify", "wholeSpace": true, "maxBytes": -1}, "backfill_budget_invalid"},
"unknown-space": {"missing", map[string]any{"reason": "verify", "wholeSpace": true}, "space_not_found"},
} {
if response := create(test.space, test.body, "invalid-"+name); response.Code/100 != 4 || !strings.Contains(response.Body.String(), test.code) {
t.Fatalf("%s: %d %s", name, response.Code, response.Body.String())
}
}
response := create("space", map[string]any{"reason": " verify node-04 v1 archives ", "keyPrefix": "node-04", "md5sum": strings.Repeat("a", 32)}, "valid")
if response.Code != http.StatusAccepted {
t.Fatalf("valid request: %d %s", response.Code, response.Body.String())
}
var job domain.ObjectSyncDigestBackfillRecord
if err := json.Unmarshal(response.Body.Bytes(), &job); err != nil {
t.Fatal(err)
}
if job.Status != "pending" || job.KeyPrefix != "node-04/" || job.Reason != "verify node-04 v1 archives" || job.MaxObjects != 10_000 || job.MaxBytes != 1<<40 || job.CreatedBy != "business-test-admin" || job.AllowUnanchored {
t.Fatalf("job=%#v", job)
}
events, _ := data.ListObjectSyncAudit(t.Context(), domain.ObjectSyncListQuery{OrganizationID: org, Limit: 100})
if len(events) != 1 || events[0].Action != "space.digest_backfill" || events[0].Reason != job.Reason || events[0].Target["backfillId"] != job.ID || events[0].ActorID != "business-test-admin" {
t.Fatalf("approval audit=%#v", events)
}
got := objectSyncRequest(t, handler, http.MethodGet, "/api/object-sync/digest-backfills/"+job.ID, nil, "")
if got.Code != http.StatusOK || !strings.Contains(got.Body.String(), `"keyPrefix":"node-04/"`) {
t.Fatalf("get: %d %s", got.Code, got.Body.String())
}
foreign := job
foreign.ID = "foreign-job"
foreign.OrganizationID = "other-org"
data.UpsertObjectSyncDigestBackfill(t.Context(), foreign)
for _, id := range []string{"foreign-job", "missing-job"} {
if response := objectSyncRequest(t, handler, http.MethodGet, "/api/object-sync/digest-backfills/"+id, nil, ""); response.Code != http.StatusNotFound {
t.Fatalf("%s visible: %d", id, response.Code)
}
}
}
// attestedCleanupFixture turns the standard cleanup fixture into a v1 archive:
// the object carries only drosync-sha256 and the head an attested MD5.
func attestedCleanupFixture(t *testing.T, mutate func(*domain.ObjectSyncDigestAttestation, domain.ObjectStorageObjectSummary)) (*httptest.ResponseRecorder, map[string]any) {
t.Helper()
_, data, fake, call := cleanupFixture(t)
head, _ := data.GetObjectSyncHead(t.Context(), "org", "space", "prefix/capture.bin")
attestation := domain.ObjectSyncDigestAttestation{
Source: domain.ObjectSyncDigestSourceCoordinatorStream, BackfillID: "backfill", ETag: "etag", VersionID: "v1", SizeBytes: 4,
MD5Sum: head.MD5Sum, SHA256: strings.Repeat("f", 64), Anchors: []string{domain.ObjectSyncDigestAnchorHeadSHA256}, AttestedAt: nowISO(),
}
remote := fake.objects["bucket|prefix/capture.bin"]
remote.Metadata = map[string]any{"drosync-sha256": strings.Repeat("f", 64)}
mutate(&attestation, remote)
head.MD5Attestation = &attestation
head.LegacySHA256 = strings.Repeat("f", 64)
if err := data.UpsertObjectSyncHead(t.Context(), *head); err != nil {
t.Fatal(err)
}
fake.objects["bucket|prefix/capture.bin"] = remote
response := call(http.MethodPost, "device", "")
var proof map[string]any
_ = json.Unmarshal(response.Body.Bytes(), &proof)
return response, proof
}
func TestCleanupProofAcceptsOnlyBoundCoordinatorAttestations(t *testing.T) {
response, proof := attestedCleanupFixture(t, func(*domain.ObjectSyncDigestAttestation, domain.ObjectStorageObjectSummary) {})
if response.Code != http.StatusOK || proof["md5Evidence"] != "coordinator_attestation" || proof["md5sum"] != strings.Repeat("a", 32) {
t.Fatalf("bound attestation rejected: %d %s", response.Code, response.Body.String())
}
for name, mutate := range map[string]func(*domain.ObjectSyncDigestAttestation, domain.ObjectStorageObjectSummary){
"attested-other-etag": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) {
a.ETag = "older-etag"
},
"attested-other-version": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) { a.VersionID = "" },
"attested-other-size": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) { a.SizeBytes = 5 },
"attested-other-md5": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) {
a.MD5Sum = strings.Repeat("c", 32)
},
"device-source": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) {
a.Source = "device_report"
},
"no-anchor": func(a *domain.ObjectSyncDigestAttestation, _ domain.ObjectStorageObjectSummary) { a.Anchors = nil },
"object-declares-other-md5": func(_ *domain.ObjectSyncDigestAttestation, remote domain.ObjectStorageObjectSummary) {
remote.Metadata["drosync-md5sum"] = strings.Repeat("c", 32)
},
} {
if response, _ := attestedCleanupFixture(t, mutate); response.Code != http.StatusConflict || !strings.Contains(response.Body.String(), "cleanup_cloud_changed") {
t.Fatalf("%s: unsafe proof %d %s", name, response.Code, response.Body.String())
}
}
}
// legacyArchiveStorage stands in for BOS on the coordinator side. It serves
// conditional reads of the archived bytes and refuses every write.
type legacyArchiveStorage struct {
mu sync.Mutex
objects map[string]objectsync.ObjectMetadata
content map[string][]byte
reads int
writes int
}
func (storage *legacyArchiveStorage) Head(_ context.Context, _ domain.StorageConnectionRecord, bucket, key string) (objectsync.ObjectMetadata, error) {
storage.mu.Lock()
defer storage.mu.Unlock()
return storage.objects[bucket+"/"+key], nil
}
func (storage *legacyArchiveStorage) GetObjectRange(_ context.Context, _ domain.StorageConnectionRecord, bucket, key, ifMatch string, offset, length int64) (io.ReadCloser, objectsync.ObjectMetadata, error) {
storage.mu.Lock()
defer storage.mu.Unlock()
storage.reads++
meta, ok := storage.objects[bucket+"/"+key]
if !ok || meta.ETag != ifMatch {
return nil, objectsync.ObjectMetadata{}, &bce.BceServiceError{StatusCode: http.StatusPreconditionFailed}
}
return io.NopCloser(bytes.NewReader(storage.content[bucket+"/"+key][offset : offset+length])), meta, nil
}
func (storage *legacyArchiveStorage) refuse() error {
storage.mu.Lock()
defer storage.mu.Unlock()
storage.writes++
return errors.New("the v1 archive is read-only")
}
func (storage *legacyArchiveStorage) PublishConditional(context.Context, domain.StorageConnectionRecord, string, string, string, int64, string, string, string, map[string]string) (objectsync.ObjectMetadata, error) {
return objectsync.ObjectMetadata{}, storage.refuse()
}
func (storage *legacyArchiveStorage) PutInternal(context.Context, domain.StorageConnectionRecord, string, string, io.Reader, int64, string, map[string]string) (objectsync.ObjectMetadata, error) {
return objectsync.ObjectMetadata{}, storage.refuse()
}
func (storage *legacyArchiveStorage) AbortMultipart(context.Context, domain.StorageConnectionRecord, string, string, string) error {
return storage.refuse()
}
func (storage *legacyArchiveStorage) Delete(context.Context, domain.StorageConnectionRecord, string, string) error {
return storage.refuse()
}
// A device keeps v1-archived files whose receipts only have SHA-256. Its syncs
// stay partial and nothing is cleanable until an administrator approves a
// digest backfill; then the next ordinary sync completes and every file gets a
// cleanup proof. Only the BOS adapter and Mongo are in-memory fakes.
func TestLegacyArchiveSyncCompletesAndCleansAfterDigestBackfill(t *testing.T) {
handler, data := objectSyncTestHandler(t, true)
ctx := t.Context()
org := domain.DefaultOrganizationID
space := domain.ObjectSyncSpaceRecord{
ID: "legacy-space", OrganizationID: org, Name: "v1 archive", ConnectionID: "bos-connection", Bucket: "archive", RootPrefix: "robots",
Enabled: true, Mode: "create_only", Policy: domain.DefaultObjectSyncSpacePolicy(),
Capability: domain.ObjectSyncCapabilityRecord{State: domain.ObjectSyncCapabilityCreateOnly, ConditionalCreate: true},
}
token := "legacy-device-token"
credential := domain.ObjectSyncCredentialRecord{
ID: "legacy-credential", OrganizationID: org, Enabled: true,
TokenBindings: []domain.ObjectSyncCredentialTokenBinding{{TokenHash: sessionTokenHash(token), DeviceID: "legacy-device"}},
Scopes: []domain.ObjectSyncCredentialScope{{SyncSpaceID: space.ID, AllowedRelativePrefixes: []string{""}, Actions: []string{"plan", "create"}}},
}
for _, err := range []error{
data.UpsertObjectSyncSpace(ctx, space),
data.UpsertObjectSyncCredential(ctx, credential),
data.UpsertObjectSyncDevice(ctx, domain.ObjectSyncDeviceRecord{ID: "legacy-device", OrganizationID: org, Status: "active"}),
} {
if err != nil {
t.Fatal(err)
}
}
files := []struct {
path string
payload []byte
}{
{"node-04/episode_0001/capture.mcap", bytes.Repeat([]byte("mcap-frame;"), 3000)},
{"node-04/episode_0001/episode_meta.json", []byte(`{"episode":1,"robot":"node-04"}`)},
}
archive := &legacyArchiveStorage{objects: map[string]objectsync.ObjectMetadata{}, content: map[string][]byte{}}
browser := newFakeObjectStorageBrowser()
browser.objects = map[string]domain.ObjectStorageObjectSummary{}
localMD5 := map[string]string{}
var archivedBytes int64
for index, file := range files {
key := "robots/" + file.path
md5Sum, sha256Sum := md5.Sum(file.payload), sha256.Sum256(file.payload)
localMD5[file.path] = hex.EncodeToString(md5Sum[:])
legacySHA := hex.EncodeToString(sha256Sum[:])
etag := fmt.Sprintf("v1-multipart-etag-%d", index)
size := int64(len(file.payload))
archivedBytes += size
if err := data.UpsertObjectSyncHead(ctx, domain.ObjectSyncObjectHeadRecord{OrganizationID: org, SyncSpaceID: space.ID, Key: key, PlatformVersion: 1, RemoteETag: etag, SizeBytes: size, LegacySHA256: legacySHA, SessionID: "oses_v1", ItemID: fmt.Sprintf("osi_v1_%d", index)}); err != nil {
t.Fatal(err)
}
archive.objects["archive/"+key] = objectsync.ObjectMetadata{Exists: true, ETag: etag, SizeBytes: size, UserMeta: map[string]string{"drosync-sha256": legacySHA}}
archive.content["archive/"+key] = file.payload
browser.objects["archive|"+key] = domain.ObjectStorageObjectSummary{Bucket: "archive", Key: key, ETag: etag, SizeBytes: size, Metadata: map[string]any{"drosync-sha256": legacySHA}}
}
previousResolver := storageConnectionBrowserResolver
storageConnectionBrowserResolver = func(context.Context, *domain.StorageConnectionRecord) (objectStorageBrowser, func(), error) {
return &resumableObjectSyncBrowser{fakeObjectStorageBrowser: browser}, func() {}, nil
}
t.Cleanup(func() { storageConnectionBrowserResolver = previousResolver })
coordinator, err := objectsync.NewCoordinator(data, archive, "e2e-coordinator")
if err != nil {
t.Fatal(err)
}
device := func(method, path string, body any, key string) *httptest.ResponseRecorder {
t.Helper()
var payload bytes.Buffer
if body != nil {
if err := json.NewEncoder(&payload).Encode(body); err != nil {
t.Fatal(err)
}
}
request := httptest.NewRequest(method, "/api/external/object-sync/"+path, &payload)
request.Header.Set("Content-Type", "application/json")
request.Header.Set(objectSyncCredentialHeader, token)
if key != "" {
request.Header.Set("Idempotency-Key", key)
}
writer := httptest.NewRecorder()
handler.ServeHTTP(writer, request)
return writer
}
expect := func(response *httptest.ResponseRecorder, status int, target any) {
t.Helper()
if response.Code != status {
t.Fatalf("status %d, want %d: %s", response.Code, status, response.Body.String())
}
if target != nil {
if err := json.Unmarshal(response.Body.Bytes(), target); err != nil {
t.Fatal(err)
}
}
}
// One ordinary daemon sync: manifest of local MD5s, server planning, finish.
sync := func(label string) (domain.ObjectSyncSessionRecord, []domain.ObjectSyncItemRecord) {
t.Helper()
var session domain.ObjectSyncSessionRecord
expect(device(http.MethodPost, "sessions", map[string]any{"syncSpaceId": space.ID, "mappingId": "mapping-node-04", "clientVersion": "1.1.1", "protocolVersion": domain.ObjectSyncProtocolVersion}, label+"-create"), http.StatusCreated, &session)
manifest := make([]map[string]any, 0, len(files))
for index, file := range files {
manifest = append(manifest, map[string]any{"clientItemId": fmt.Sprintf("file-%d", index), "relativePath": file.path, "kind": "file", "sizeBytes": len(file.payload), "md5sum": localMD5[file.path], "modifiedAt": "2026-09-20T08:00:00Z"})
}
expect(device(http.MethodPost, "sessions/"+session.ID+"/manifest-pages", map[string]any{"pageNumber": 1, "final": true, "items": manifest}, label+"-manifest"), http.StatusAccepted, nil)
expect(device(http.MethodPost, "sessions/"+session.ID+"/plan", map[string]any{}, label+"-plan"), http.StatusAccepted, nil)
if _, err := coordinator.ProcessOnce(ctx); err != nil {
t.Fatal(err)
}
var final domain.ObjectSyncSessionRecord
expect(device(http.MethodPost, "sessions/"+session.ID+"/complete", map[string]any{}, label+"-complete"), http.StatusOK, &final)
page, err := data.ListObjectSyncItems(ctx, domain.ObjectSyncListQuery{SessionID: session.ID, Limit: 100})
if err != nil || len(page.Items) != len(files) {
t.Fatalf("items=%#v err=%v", page.Items, err)
}
return final, page.Items
}
before, deniedItems := sync("before")
if before.Status != domain.ObjectSyncSessionPartial || before.Counts.Denied != int64(len(files)) {
t.Fatalf("legacy archives did not keep the sync partial: %#v", before)
}
for _, item := range deniedItems {
if item.Status != domain.ObjectSyncItemDenied || item.Action != "deny_archived_hash_change" {
t.Fatalf("item=%#v", item)
}
if response := device(http.MethodPost, "items/"+item.ID+"/cleanup-proof", nil, ""); response.Code != http.StatusConflict || !strings.Contains(response.Body.String(), "cleanup_item_not_committed") {
t.Fatalf("denied legacy file became cleanable: %d %s", response.Code, response.Body.String())
}
}
var job domain.ObjectSyncDigestBackfillRecord
expect(objectSyncRequest(t, handler, http.MethodPost, "/api/object-sync/spaces/"+space.ID+"/digest-backfills", map[string]any{"reason": "node-04 v1 archives keep every sync partial", "sessionIds": []string{before.ID}}, "approve-node-04"), http.StatusAccepted, &job)
if _, err := coordinator.ProcessOnce(ctx); err != nil {
t.Fatal(err)
}
expect(objectSyncRequest(t, handler, http.MethodGet, "/api/object-sync/digest-backfills/"+job.ID, nil, ""), http.StatusOK, &job)
if job.Status != "completed" || job.Counts.Attested != int64(len(files)) || job.Counts.ItemsMatched != int64(len(files)) || job.Counts.ItemsConflicting != 0 || job.Counts.BytesRead != archivedBytes || len(job.Exceptions) != 0 {
t.Fatalf("backfill=%#v", job)
}
if archive.writes != 0 || archive.reads == 0 {
t.Fatalf("archive reads=%d writes=%d", archive.reads, archive.writes)
}
after, items := sync("after")
if after.Status != domain.ObjectSyncSessionCompleted || after.Counts.Unchanged != int64(len(files)) || after.Counts.Denied != 0 {
t.Fatalf("sync after remediation did not complete: %#v", after)
}
for _, item := range items {
head, _ := data.GetObjectSyncHead(ctx, org, space.ID, item.TargetKey)
if item.Status != domain.ObjectSyncItemUnchanged || item.ExpectedBase.ETag != head.RemoteETag || head.MD5Sum != localMD5[item.RelativePath] || head.MD5Attestation == nil || head.MD5Attestation.BackfillID != job.ID {
t.Fatalf("item=%#v head=%#v", item, head)
}
var proof map[string]any
expect(device(http.MethodPost, "items/"+item.ID+"/cleanup-proof", nil, ""), http.StatusOK, &proof)
if proof["sessionId"] != after.ID || proof["md5sum"] != localMD5[item.RelativePath] || proof["md5Evidence"] != "coordinator_attestation" || proof["eTag"] != head.RemoteETag {
t.Fatalf("proof=%#v", proof)
}
release := map[string]any{"leaseId": proof["leaseId"], "leaseOwner": proof["leaseOwner"], "fencingToken": proof["fencingToken"]}
expect(device(http.MethodDelete, "items/"+item.ID+"/cleanup-proof", release, ""), http.StatusOK, nil)
}
history, _ := data.GetObjectSyncSession(ctx, before.ID)
if history.Status != domain.ObjectSyncSessionPartial {
t.Fatal("remediation rewrote the closed partial session")
}
for _, item := range deniedItems {
stored, _ := data.GetObjectSyncItem(ctx, item.ID)
if stored.Status != domain.ObjectSyncItemDenied {
t.Fatal("remediation rewrote a closed denied item")
}
}
}

View File

@ -1529,6 +1529,8 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
router.Get("/reconciles", instance.handleObjectSyncReconcilesList)
router.Get("/reconciles/{id}", instance.handleObjectSyncReconcileGet)
router.Post("/reconciles/{id}/retry", instance.handleObjectSyncReconcileRetry)
router.Post("/spaces/{id}/digest-backfills", instance.handleObjectSyncDigestBackfillCreate)
router.Get("/digest-backfills/{id}", instance.handleObjectSyncDigestBackfillGet)
router.Get("/devices", instance.handleObjectSyncDevicesList)
router.Get("/credentials", instance.handleObjectSyncCredentialsList)
router.Post("/credentials", instance.handleObjectSyncCredentialCreate)

View File

@ -252,10 +252,97 @@ type ObjectSyncObjectHeadRecord struct {
VersionID string `json:"versionId,omitempty" bson:"versionId,omitempty"`
SizeBytes int64 `json:"sizeBytes" bson:"sizeBytes"`
MD5Sum string `json:"md5sum" bson:"md5sum"`
SessionID string `json:"sessionId" bson:"sessionId"`
ItemID string `json:"itemId" bson:"itemId"`
Drifted bool `json:"drifted" bson:"drifted"`
UpdatedAt string `json:"updatedAt" bson:"updatedAt"`
// LegacySHA256 is the retired v1 receipt still stored under "sha256". It is
// only an anchor for coordinator re-verification, never a content comparison.
LegacySHA256 string `json:"legacySha256,omitempty" bson:"sha256,omitempty"`
// MD5Attestation is set when the coordinator, not a device, derived MD5Sum by
// streaming this exact object version. Upload receipts leave it empty.
MD5Attestation *ObjectSyncDigestAttestation `json:"md5Attestation,omitempty" bson:"md5Attestation,omitempty"`
SessionID string `json:"sessionId" bson:"sessionId"`
ItemID string `json:"itemId" bson:"itemId"`
Drifted bool `json:"drifted" bson:"drifted"`
UpdatedAt string `json:"updatedAt" bson:"updatedAt"`
}
const (
ObjectSyncDigestSourceCoordinatorStream = "coordinator_stream"
ObjectSyncDigestAnchorHeadSHA256 = "head_sha256"
ObjectSyncDigestAnchorObjectSHA256 = "object_metadata_sha256"
ObjectSyncDigestAnchorStorageSHA256 = "storage_content_sha256"
ObjectSyncDigestAnchorObjectMD5 = "object_metadata_md5"
ObjectSyncDigestAnchorStorageMD5 = "storage_content_md5"
ObjectSyncDigestAnchorNoneAdminWaive = "none_admin_waived"
)
// ObjectSyncDigestAttestation binds a coordinator-computed MD5 to one archived
// object version and records which independent receipts it was checked against.
type ObjectSyncDigestAttestation struct {
Source string `json:"source" bson:"source"`
BackfillID string `json:"backfillId" bson:"backfillId"`
ApprovedBy string `json:"approvedBy" bson:"approvedBy"`
CoordinatorID string `json:"coordinatorId" bson:"coordinatorId"`
ETag string `json:"etag" bson:"etag"`
VersionID string `json:"versionId,omitempty" bson:"versionId,omitempty"`
SizeBytes int64 `json:"sizeBytes" bson:"sizeBytes"`
MD5Sum string `json:"md5sum" bson:"md5sum"`
SHA256 string `json:"sha256" bson:"sha256"`
Anchors []string `json:"anchors" bson:"anchors"`
PreviousMD5Sum string `json:"previousMd5sum,omitempty" bson:"previousMd5sum,omitempty"`
AttestedAt string `json:"attestedAt" bson:"attestedAt"`
}
// ObjectSyncDigestBackfillRecord is an administrator-approved coordinator job
// that re-verifies archived heads lacking a trusted MD5 receipt.
type ObjectSyncDigestBackfillRecord struct {
ID string `json:"id" bson:"id"`
OperationID string `json:"operationId" bson:"operationId"`
OrganizationID string `json:"organizationId" bson:"organizationId"`
SyncSpaceID string `json:"syncSpaceId" bson:"syncSpaceId"`
Status string `json:"status" bson:"status"`
Reason string `json:"reason" bson:"reason"`
SessionIDs []string `json:"sessionIds,omitempty" bson:"sessionIds,omitempty"`
KeyPrefix string `json:"keyPrefix,omitempty" bson:"keyPrefix,omitempty"`
WholeSpace bool `json:"wholeSpace" bson:"wholeSpace"`
AllowUnanchored bool `json:"allowUnanchored" bson:"allowUnanchored"`
MaxObjects int64 `json:"maxObjects" bson:"maxObjects"`
MaxBytes int64 `json:"maxBytes" bson:"maxBytes"`
Cursor string `json:"-" bson:"cursor,omitempty"`
Counts ObjectSyncDigestBackfillCounts `json:"counts" bson:"counts"`
BudgetExhausted bool `json:"budgetExhausted" bson:"budgetExhausted"`
Exceptions []ObjectSyncDigestBackfillException `json:"exceptions,omitempty" bson:"exceptions,omitempty"`
Error string `json:"error,omitempty" bson:"error,omitempty"`
CreatedBy string `json:"createdBy" bson:"createdBy"`
CreatedAt string `json:"createdAt" bson:"createdAt"`
UpdatedAt string `json:"updatedAt" bson:"updatedAt"`
StartedAt string `json:"startedAt,omitempty" bson:"startedAt,omitempty"`
CompletedAt string `json:"completedAt,omitempty" bson:"completedAt,omitempty"`
CoordinatorOwner string `json:"-" bson:"coordinatorOwner,omitempty"`
CoordinatorLeaseUntil time.Time `json:"-" bson:"coordinatorLeaseUntil,omitempty"`
}
// Counts are at-least-once: a resumed job may revisit keys from its last page.
type ObjectSyncDigestBackfillCounts struct {
Scanned int64 `json:"scanned" bson:"scanned"`
Attested int64 `json:"attested" bson:"attested"`
AlreadyAttested int64 `json:"alreadyAttested" bson:"alreadyAttested"`
NotLegacy int64 `json:"notLegacy" bson:"notLegacy"`
Unanchored int64 `json:"unanchored" bson:"unanchored"`
Mismatched int64 `json:"mismatched" bson:"mismatched"`
Drifted int64 `json:"drifted" bson:"drifted"`
Busy int64 `json:"busy" bson:"busy"`
ReadFailed int64 `json:"readFailed" bson:"readFailed"`
ObjectsRead int64 `json:"objectsRead" bson:"objectsRead"`
BytesRead int64 `json:"bytesRead" bson:"bytesRead"`
ItemsMatched int64 `json:"itemsMatched" bson:"itemsMatched"`
ItemsConflicting int64 `json:"itemsConflicting" bson:"itemsConflicting"`
}
type ObjectSyncDigestBackfillException struct {
Key string `json:"key" bson:"key"`
Outcome string `json:"outcome" bson:"outcome"`
Detail string `json:"detail,omitempty" bson:"detail,omitempty"`
At string `json:"at" bson:"at"`
}
type ObjectSyncVersionRecord struct {

View File

@ -24,6 +24,10 @@ type ObjectMetadata struct {
SizeBytes int64
LastModified string
UserMeta map[string]string
// Storage-side digests recorded at upload time, when the provider kept them.
ContentMD5 string
ContentSHA256 string
StorageClass string
}
type Publisher interface {
@ -34,6 +38,13 @@ type Publisher interface {
Delete(context.Context, domain.StorageConnectionRecord, string, string) error
}
// ObjectReader streams byte ranges of one archived object for coordinator-side
// digest verification. Every range is conditional on the archived ETag, so a
// replacement makes the read fail instead of mixing object versions.
type ObjectReader interface {
GetObjectRange(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key, ifMatchETag string, offset, length int64) (io.ReadCloser, ObjectMetadata, error)
}
type BOSPublisher struct {
GlobalEndpoint string
}
@ -77,20 +88,51 @@ func (publisher BOSPublisher) Head(ctx context.Context, connection domain.Storag
if err != nil {
return ObjectMetadata{}, bosclient.Error(ctx, "head", err)
}
return objectMetadataFromBOS(result.ObjectMeta), nil
}
func objectMetadataFromBOS(meta bosapi.ObjectMeta) ObjectMetadata {
// HTTP header names are case-insensitive; the SDK returns canonicalized
// metadata suffixes while publication identities use lowercase keys.
userMeta := make(map[string]string, len(result.UserMeta))
for key, value := range result.UserMeta {
userMeta := make(map[string]string, len(meta.UserMeta))
for key, value := range meta.UserMeta {
userMeta[strings.ToLower(strings.TrimSpace(key))] = value
}
return ObjectMetadata{
Exists: true,
ETag: strings.Trim(result.ETag, `"`),
VersionID: strings.TrimSpace(result.VersionId),
SizeBytes: result.ContentLength,
LastModified: strings.TrimSpace(result.LastModified),
UserMeta: userMeta,
}, nil
Exists: true,
ETag: strings.Trim(meta.ETag, `"`),
VersionID: strings.TrimSpace(meta.VersionId),
SizeBytes: meta.ContentLength,
LastModified: strings.TrimSpace(meta.LastModified),
UserMeta: userMeta,
ContentMD5: strings.TrimSpace(meta.ContentMD5),
ContentSHA256: strings.TrimSpace(meta.ContentSha256),
StorageClass: strings.TrimSpace(meta.StorageClass),
}
}
// GetObjectRange issues a conditional ranged GET. The SDK client bounds each
// request, so callers read large objects as a sequence of bounded ranges.
func (publisher BOSPublisher) GetObjectRange(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key, ifMatchETag string, offset, length int64) (io.ReadCloser, ObjectMetadata, error) {
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
return nil, ObjectMetadata{}, err
}
ifMatchETag = strings.Trim(strings.TrimSpace(ifMatchETag), `"`)
if ifMatchETag == "" || offset < 0 || length <= 0 {
return nil, ObjectMetadata{}, fmt.Errorf("digest verification requires the archived ETag and a non-empty range")
}
client, err := publisher.client(connection)
if err != nil {
return nil, ObjectMetadata{}, err
}
result, err := client.GetObjectWithArgs(ctx, strings.TrimSpace(bucket), strings.Trim(strings.TrimSpace(key), "/"), &bosapi.GetObjectArgs{
Ranges: []int64{offset, offset + length - 1},
IfMatch: ifMatchETag,
})
if err != nil {
return nil, ObjectMetadata{}, bosclient.Error(ctx, "get range", err)
}
return result.Body, objectMetadataFromBOS(result.ObjectMeta), nil
}
func multipartCopyPartSize(sizeBytes int64) int64 {

View File

@ -46,6 +46,7 @@ type Coordinator struct {
reconciled atomic.Int64
released atomic.Int64
sealed atomic.Int64
backfilled atomic.Int64
lastCycle atomic.Int64
lastOK atomic.Int64
durationMS atomic.Int64
@ -157,6 +158,7 @@ type Metrics struct {
Reconciled int64
Released int64
Sealed int64
Backfilled int64
LastCycle int64
LastOK int64
DurationMS int64
@ -184,6 +186,7 @@ func (coordinator *Coordinator) Metrics() Metrics {
Reconciled: coordinator.reconciled.Load(),
Released: coordinator.released.Load(),
Sealed: coordinator.sealed.Load(),
Backfilled: coordinator.backfilled.Load(),
LastCycle: coordinator.lastCycle.Load(),
LastOK: coordinator.lastOK.Load(),
DurationMS: coordinator.durationMS.Load(),
@ -204,6 +207,7 @@ type ProcessResult struct {
Reconciled int `json:"reconciled"`
Released int `json:"released"`
Sealed int `json:"sealed"`
Backfilled int `json:"backfilled"`
}
func (coordinator *Coordinator) ProcessOnce(ctx context.Context) (ProcessResult, error) {
@ -225,6 +229,7 @@ func (coordinator *Coordinator) ProcessOnce(ctx context.Context) (ProcessResult,
result.Reconciled += value.result.Reconciled
result.Released += value.result.Released
result.Sealed += value.result.Sealed
result.Backfilled += value.result.Backfilled
if value.err != nil {
failures = append(failures, value.err)
}

View File

@ -18,13 +18,14 @@ const (
phaseReconcile
phaseRelease
phaseAudit
phaseDigestBackfill
phaseCount
commitConcurrency = 4
cleanupConcurrency = 2
planConcurrency = 16
)
var phaseNames = [phaseCount]string{"planning", "commit", "cleanup", "reconcile", "release", "audit"}
var phaseNames = [phaseCount]string{"planning", "commit", "cleanup", "reconcile", "release", "audit", "digest_backfill"}
var errClaimLost = errors.New("coordinator claim or object lease lost")
type phaseCounters struct{ cycles, errors, completed, itemErrors, inFlight, durationMS, totalDurationMS atomic.Int64 }
@ -57,7 +58,7 @@ func (coordinator *Coordinator) measureStage(phase int, work func() (ProcessResu
metrics.cycles.Add(1)
metrics.durationMS.Store(elapsed)
metrics.totalDurationMS.Add(elapsed)
completed := result.Planned + result.Committed + result.Cleaned + result.Reconciled + result.Released + result.Sealed
completed := result.Planned + result.Committed + result.Cleaned + result.Reconciled + result.Released + result.Sealed + result.Backfilled
metrics.completed.Add(int64(completed))
coordinator.planned.Add(int64(result.Planned))
coordinator.committed.Add(int64(result.Committed))
@ -65,6 +66,7 @@ func (coordinator *Coordinator) measureStage(phase int, work func() (ProcessResu
coordinator.reconciled.Add(int64(result.Reconciled))
coordinator.released.Add(int64(result.Released))
coordinator.sealed.Add(int64(result.Sealed))
coordinator.backfilled.Add(int64(result.Backfilled))
if err != nil && !errors.Is(err, context.Canceled) {
metrics.errors.Add(1)
coordinator.errors.Add(1)
@ -174,6 +176,32 @@ func (coordinator *Coordinator) processStage(ctx context.Context, phase int) (Pr
result.Sealed = count
return result, err
}
case phaseDigestBackfill:
job, err := coordinator.syncStore.ClaimObjectSyncDigestBackfill(ctx, coordinator.owner, claimTTL)
if err != nil || job == nil {
return result, err
}
token, id := job.CoordinatorOwner, job.ID
err = coordinator.withRenewingClaim(ctx, func(callCtx context.Context) (bool, error) {
return coordinator.syncStore.RenewObjectSyncDigestBackfillClaim(callCtx, id, token, claimTTL)
}, func(callCtx context.Context) error { return coordinator.runDigestBackfill(callCtx, job) })
if err != nil {
if !errors.Is(err, errClaimLost) && !errors.Is(err, context.Canceled) {
job.Status = "failed"
job.Error = err.Error()
job.UpdatedAt = nowISO()
job.CompletedAt = job.UpdatedAt
job.CoordinatorOwner = ""
job.CoordinatorLeaseUntil = time.Time{}
saved, saveErr := coordinator.syncStore.SaveClaimedObjectSyncDigestBackfill(ctx, *job, token)
if !saved && saveErr == nil {
saveErr = errClaimLost
}
err = errors.Join(err, saveErr)
}
return result, err
}
result.Backfilled = 1
}
return result, nil
}

View File

@ -0,0 +1,704 @@
package objectsync
import (
"context"
"crypto/md5"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"io"
"slices"
"strconv"
"strings"
"time"
"droboticflow/internal/bosclient"
"droboticflow/internal/domain"
"droboticflow/internal/store"
)
// Legacy v1 archives carry SHA-256 receipts that protocol v2 deliberately does
// not compare. A digest backfill reads the archived object itself, so the MD5 it
// records is cloud evidence; a device's local MD5 is never written to a head.
const (
digestBackfillPageSize = 1000
digestBackfillMaxExceptions = 200
digestBackfillLeaseTTL = 5 * time.Minute
legacyDeniedAction = "deny_archived_hash_change"
)
// Each ranged GET must finish inside the SDK client's request bound even on a
// slow link, so large objects are read as many conditional ranges.
var digestBackfillRangeBytes = int64(64 << 20)
const (
backfillAttested = "attested"
backfillAlreadyAttested = "already_attested"
backfillNotLegacy = "not_legacy"
backfillUnanchored = "unanchored"
backfillAnchorConflict = "anchor_conflict"
backfillContentMismatch = "content_mismatch"
backfillDrifted = "drifted"
backfillUnbound = "unbound_receipt"
backfillBusy = "busy"
backfillUnreadableClass = "storage_class_unreadable"
backfillReadFailed = "read_failed"
backfillBudget = "budget_exhausted"
)
var errBackfillBudget = errors.New("digest backfill budget exhausted")
// ValidMD5Hex accepts only the lowercase form stored by protocol v2 receipts.
func ValidMD5Hex(value string) bool {
return len(value) == 2*md5.Size && isLowerHex(value)
}
func validSHA256Hex(value string) bool {
return len(value) == 2*sha256.Size && isLowerHex(value)
}
func isLowerHex(value string) bool {
for _, character := range value {
if (character < '0' || character > '9') && (character < 'a' || character > 'f') {
return false
}
}
return true
}
// ObjectLeaseResourceHash names the per-object lease shared by uploads,
// cleanup proofs and digest verification.
func ObjectLeaseResourceHash(space domain.ObjectSyncSpaceRecord, key string) string {
sum := sha256.Sum256([]byte(space.ConnectionID + "\x00" + space.Bucket + "\x00" + key))
return hex.EncodeToString(sum[:])
}
// AttestationBindsHead reports whether head.MD5Sum was streamed by the
// coordinator from exactly the object identity the head still names.
func AttestationBindsHead(head domain.ObjectSyncObjectHeadRecord) bool {
attestation := head.MD5Attestation
return attestation != nil &&
attestation.Source == domain.ObjectSyncDigestSourceCoordinatorStream &&
ValidMD5Hex(attestation.MD5Sum) && attestation.MD5Sum == head.MD5Sum &&
attestation.ETag != "" && strings.Trim(attestation.ETag, `"`) == strings.Trim(head.RemoteETag, `"`) &&
attestation.VersionID == head.VersionID && attestation.SizeBytes == head.SizeBytes &&
len(attestation.Anchors) > 0
}
type backfillVerdict struct {
outcome string
detail string
read bool
bytes int64
head *domain.ObjectSyncObjectHeadRecord
}
type digestBackfillRun struct {
coordinator *Coordinator
job *domain.ObjectSyncDigestBackfillRecord
token string
space *domain.ObjectSyncSpaceRecord
connection *domain.StorageConnectionRecord
reader ObjectReader
}
func (coordinator *Coordinator) runDigestBackfill(ctx context.Context, job *domain.ObjectSyncDigestBackfillRecord) error {
token := job.CoordinatorOwner
if token == "" {
return errClaimLost
}
reader, ok := coordinator.publisher.(ObjectReader)
if !ok {
return fmt.Errorf("object storage cannot stream archived objects for digest verification")
}
space, err := coordinator.syncStore.GetObjectSyncSpace(ctx, job.SyncSpaceID)
if err != nil || space == nil || space.OrganizationID != job.OrganizationID {
return fmt.Errorf("sync space is unavailable")
}
connection, err := coordinator.store.GetStorageConnection(ctx, space.ConnectionID)
if err != nil || connection == nil {
return fmt.Errorf("storage connection is unavailable")
}
if job.StartedAt == "" {
job.StartedAt = nowISO()
}
run := &digestBackfillRun{coordinator: coordinator, job: job, token: token, space: space, connection: connection, reader: reader}
if len(job.SessionIDs) > 0 {
err = run.sessions(ctx)
} else {
err = run.heads(ctx)
}
if err != nil && !errors.Is(err, errBackfillBudget) {
return err
}
job.Status = "completed"
job.CompletedAt = nowISO()
job.UpdatedAt = job.CompletedAt
job.CoordinatorOwner = ""
job.CoordinatorLeaseUntil = time.Time{}
saved, err := coordinator.syncStore.SaveClaimedObjectSyncDigestBackfill(ctx, *job, token)
if err != nil {
return err
}
if !saved {
return errClaimLost
}
coordinator.recordSystemAudit(ctx, domain.ObjectSyncAuditEvent{
OrganizationID: job.OrganizationID,
ActorType: "system",
ActorID: coordinator.owner,
Action: "space.digest_backfill.completed",
Target: map[string]any{"backfillId": job.ID, "operationId": job.OperationID, "spaceId": job.SyncSpaceID},
After: map[string]any{
"scanned": job.Counts.Scanned, "attested": job.Counts.Attested, "alreadyAttested": job.Counts.AlreadyAttested,
"unanchored": job.Counts.Unanchored, "mismatched": job.Counts.Mismatched, "drifted": job.Counts.Drifted,
"busy": job.Counts.Busy, "readFailed": job.Counts.ReadFailed, "bytesRead": job.Counts.BytesRead,
"itemsMatched": job.Counts.ItemsMatched, "itemsConflicting": job.Counts.ItemsConflicting,
"budgetExhausted": job.BudgetExhausted,
},
Result: "success",
})
return nil
}
func (run *digestBackfillRun) checkpoint(ctx context.Context) error {
run.job.UpdatedAt = nowISO()
run.job.CoordinatorLeaseUntil = time.Now().UTC().Add(claimTTL)
saved, err := run.coordinator.syncStore.SaveClaimedObjectSyncDigestBackfill(ctx, *run.job, run.token)
if err != nil {
return err
}
if !saved {
return errClaimLost
}
return nil
}
// Session scope only visits archives that devices still hold and were denied
// for, which keeps the read volume proportional to what cleanup can recover.
func (run *digestBackfillRun) sessions(ctx context.Context) error {
start, cursor := parseSessionBackfillCursor(run.job.Cursor)
seen := map[string]*domain.ObjectSyncObjectHeadRecord{}
for index := start; index < len(run.job.SessionIDs); index++ {
sessionID := run.job.SessionIDs[index]
session, err := run.coordinator.syncStore.GetObjectSyncSession(ctx, sessionID)
if err != nil {
return err
}
if session == nil || session.OrganizationID != run.job.OrganizationID || session.SyncSpaceID != run.job.SyncSpaceID {
run.addException("", "session_unavailable", "session "+sessionID+" is missing or belongs to another space")
} else {
for {
page, err := run.coordinator.syncStore.ListObjectSyncItems(ctx, domain.ObjectSyncListQuery{
SessionID: session.ID, Status: string(domain.ObjectSyncItemDenied), Action: legacyDeniedAction,
Cursor: cursor, Limit: digestBackfillPageSize,
})
if err != nil {
return err
}
for _, item := range page.Items {
head, visited := seen[item.TargetKey]
if !visited {
verdict, err := run.process(ctx, item.TargetKey)
if err != nil {
return err
}
head = verdict.head
seen[item.TargetKey] = head
}
run.compareDeniedItem(item, head)
}
if page.NextCursor == "" {
break
}
cursor = page.NextCursor
run.job.Cursor = formatSessionBackfillCursor(index, cursor)
if err := run.checkpoint(ctx); err != nil {
return err
}
}
}
cursor = ""
run.job.Cursor = formatSessionBackfillCursor(index+1, "")
if err := run.checkpoint(ctx); err != nil {
return err
}
}
return nil
}
func (run *digestBackfillRun) heads(ctx context.Context) error {
prefix := ""
if !run.job.WholeSpace {
prefix = spaceKeyPrefix(*run.space, run.job.KeyPrefix)
}
cursor := run.job.Cursor
for {
page, err := run.coordinator.syncStore.ListObjectSyncHeads(ctx, run.job.OrganizationID, run.space.ID, cursor, digestBackfillPageSize)
if err != nil {
return err
}
for _, head := range page.Items {
if prefix != "" && !strings.HasPrefix(head.Key, prefix) {
if head.Key > prefix {
// Heads are key-ordered, so nothing later can match.
return nil
}
continue
}
if ValidMD5Hex(head.MD5Sum) && head.MD5Attestation == nil {
run.job.Counts.Scanned++
run.job.Counts.NotLegacy++
continue
}
if _, err := run.process(ctx, head.Key); err != nil {
return err
}
}
if page.NextCursor == "" {
return nil
}
cursor = page.NextCursor
run.job.Cursor = cursor
if err := run.checkpoint(ctx); err != nil {
return err
}
}
}
func (run *digestBackfillRun) process(ctx context.Context, key string) (backfillVerdict, error) {
verdict, err := run.coordinator.backfillKey(ctx, run, key)
if err != nil {
return verdict, err
}
counts := &run.job.Counts
counts.BytesRead += verdict.bytes
if verdict.read {
counts.ObjectsRead++
}
if verdict.outcome == backfillBudget {
run.job.BudgetExhausted = true
run.addException(key, verdict.outcome, verdict.detail)
return verdict, errBackfillBudget
}
counts.Scanned++
switch verdict.outcome {
case backfillAttested:
counts.Attested++
case backfillAlreadyAttested:
counts.AlreadyAttested++
case backfillNotLegacy:
counts.NotLegacy++
case backfillUnanchored:
counts.Unanchored++
case backfillAnchorConflict, backfillContentMismatch:
counts.Mismatched++
case backfillDrifted, backfillUnbound:
counts.Drifted++
case backfillBusy:
counts.Busy++
default:
counts.ReadFailed++
}
switch verdict.outcome {
case backfillAttested, backfillAlreadyAttested, backfillNotLegacy:
default:
run.addException(key, verdict.outcome, verdict.detail)
}
if verdict.read {
if err := run.checkpoint(ctx); err != nil {
return verdict, err
}
}
return verdict, nil
}
// compareDeniedItem previews recovery without touching the closed session:
// the device's next sync plans the same comparison against the trusted head.
func (run *digestBackfillRun) compareDeniedItem(item domain.ObjectSyncItemRecord, head *domain.ObjectSyncObjectHeadRecord) {
if head == nil || !ValidMD5Hex(head.MD5Sum) {
return
}
if item.MD5Sum == head.MD5Sum && item.SizeBytes == head.SizeBytes {
run.job.Counts.ItemsMatched++
} else {
run.job.Counts.ItemsConflicting++
}
}
func (run *digestBackfillRun) addException(key, outcome, detail string) {
if len(run.job.Exceptions) >= digestBackfillMaxExceptions {
return
}
run.job.Exceptions = append(run.job.Exceptions, domain.ObjectSyncDigestBackfillException{Key: key, Outcome: outcome, Detail: detail, At: nowISO()})
}
func (coordinator *Coordinator) backfillKey(ctx context.Context, run *digestBackfillRun, key string) (backfillVerdict, error) {
head, err := coordinator.syncStore.GetObjectSyncHead(ctx, run.job.OrganizationID, run.space.ID, key)
if err != nil {
return backfillVerdict{}, err
}
if head == nil {
return backfillVerdict{outcome: backfillNotLegacy, detail: "no archived receipt"}, nil
}
if ValidMD5Hex(head.MD5Sum) {
if !AttestationBindsHead(*head) {
return backfillVerdict{outcome: backfillNotLegacy, head: head}, nil
}
// A crash between the head write and its audit event is repaired here;
// the event key is derived from the attestation, so replays are no-ops.
if err := coordinator.appendAttestationAudit(ctx, *head); err != nil {
return backfillVerdict{}, err
}
return backfillVerdict{outcome: backfillAlreadyAttested, head: head}, nil
}
if head.MD5Attestation != nil {
return backfillVerdict{outcome: backfillDrifted, detail: "archived receipt has an attestation that no longer matches it"}, nil
}
if head.Drifted {
return backfillVerdict{outcome: backfillDrifted, detail: "reconciliation marked the archived object as drifted"}, nil
}
if strings.Trim(head.RemoteETag, `"`) == "" || head.SizeBytes < 0 {
return backfillVerdict{outcome: backfillUnbound, detail: "archived receipt has no object ETag to bind a digest to"}, nil
}
ownerSession := "digest:" + run.job.ID
ownerItem := ownerSession + ":" + domain.NewID("verify")
lease, acquired, err := coordinator.syncStore.AcquireObjectSyncLease(ctx, domain.ObjectSyncLeaseRecord{
OrganizationID: run.job.OrganizationID,
ResourceHash: ObjectLeaseResourceHash(*run.space, key),
ResourceType: "object",
ConnectionID: run.space.ConnectionID,
Bucket: run.space.Bucket,
KeyOrPrefix: key,
OwnerSessionID: ownerSession,
OwnerItemID: ownerItem,
}, digestBackfillLeaseTTL)
if errors.Is(err, store.ErrObjectSyncLeaseHeld) || (err == nil && !acquired) {
return backfillVerdict{outcome: backfillBusy, detail: "another operation holds the object key"}, nil
}
if err != nil {
return backfillVerdict{}, err
}
defer func() {
releaseCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
if err := coordinator.syncStore.ReleaseObjectSyncLease(releaseCtx, lease.ID, ownerSession, ownerItem, lease.FencingToken); err != nil {
coordinator.recordAuxiliaryFailure(phaseDigestBackfill, "release verification lease", err)
}
}()
var verdict backfillVerdict
err = coordinator.withRenewingClaim(ctx, func(callCtx context.Context) (bool, error) {
return coordinator.syncStore.RenewObjectSyncLease(callCtx, lease.ID, ownerSession, ownerItem, lease.FencingToken, digestBackfillLeaseTTL)
}, func(callCtx context.Context) error {
var verifyErr error
verdict, verifyErr = coordinator.verifyAndAttest(callCtx, run, *head)
return verifyErr
})
if err != nil && errors.Is(err, errClaimLost) && ctx.Err() == nil {
verdict.outcome = backfillBusy
verdict.detail = "object lease was lost during verification"
verdict.head = nil
return verdict, nil
}
return verdict, err
}
func (coordinator *Coordinator) verifyAndAttest(ctx context.Context, run *digestBackfillRun, head domain.ObjectSyncObjectHeadRecord) (backfillVerdict, error) {
bucket := run.space.Bucket
remote, err := coordinator.publisher.Head(ctx, *run.connection, bucket, head.Key)
if err != nil {
if ctx.Err() != nil {
return backfillVerdict{}, ctx.Err()
}
return backfillVerdict{outcome: backfillReadFailed, detail: "head: " + err.Error()}, nil
}
if detail := archivedIdentityChange(head, remote); detail != "" {
return backfillVerdict{outcome: backfillDrifted, detail: detail}, nil
}
if strings.EqualFold(remote.StorageClass, "ARCHIVE") {
return backfillVerdict{outcome: backfillUnreadableClass, detail: "archive storage class must be restored before verification"}, nil
}
anchors, conflict := collectDigestAnchors(head, remote)
if conflict != "" {
coordinator.recordDigestRejection(ctx, run, head, backfillAnchorConflict, conflict)
return backfillVerdict{outcome: backfillAnchorConflict, detail: conflict}, nil
}
if anchors.sha256 == "" && !run.job.AllowUnanchored {
return backfillVerdict{outcome: backfillUnanchored, detail: "no independent SHA-256 receipt; an administrator must explicitly allow unanchored verification"}, nil
}
if run.job.MaxObjects > 0 && run.job.Counts.ObjectsRead >= run.job.MaxObjects {
return backfillVerdict{outcome: backfillBudget, detail: "maxObjects reached before this object"}, nil
}
if run.job.MaxBytes > 0 && run.job.Counts.BytesRead+head.SizeBytes > run.job.MaxBytes {
return backfillVerdict{outcome: backfillBudget, detail: fmt.Sprintf("reading %d bytes would exceed maxBytes", head.SizeBytes)}, nil
}
md5sum, sha256sum, read, err := coordinator.streamArchivedDigests(ctx, run, head)
verdict := backfillVerdict{read: true, bytes: read}
if err != nil {
if ctx.Err() != nil {
return verdict, err
}
verdict.outcome, verdict.detail = backfillReadFailed, err.Error()
if IsPreconditionFailure(err) || errors.Is(err, errArchivedIdentityChanged) {
verdict.outcome = backfillDrifted
}
return verdict, nil
}
if anchors.sha256 != "" && sha256sum != anchors.sha256 {
verdict.outcome = backfillContentMismatch
verdict.detail = fmt.Sprintf("archived bytes have SHA-256 %s but the legacy receipt records %s", sha256sum, anchors.sha256)
coordinator.recordDigestRejection(ctx, run, head, verdict.outcome, verdict.detail)
return verdict, nil
}
if anchors.md5 != "" && md5sum != anchors.md5 {
verdict.outcome = backfillContentMismatch
verdict.detail = fmt.Sprintf("archived bytes have MD5 %s but the object records %s", md5sum, anchors.md5)
coordinator.recordDigestRejection(ctx, run, head, verdict.outcome, verdict.detail)
return verdict, nil
}
after, err := coordinator.publisher.Head(ctx, *run.connection, bucket, head.Key)
if err != nil {
if ctx.Err() != nil {
return verdict, ctx.Err()
}
verdict.outcome, verdict.detail = backfillReadFailed, "head after read: "+err.Error()
return verdict, nil
}
if detail := archivedIdentityChange(head, after); detail != "" {
verdict.outcome, verdict.detail = backfillDrifted, detail
return verdict, nil
}
names := anchors.names
if len(names) == 0 {
names = []string{domain.ObjectSyncDigestAnchorNoneAdminWaive}
}
attestation := domain.ObjectSyncDigestAttestation{
Source: domain.ObjectSyncDigestSourceCoordinatorStream,
BackfillID: run.job.ID,
ApprovedBy: run.job.CreatedBy,
CoordinatorID: coordinator.owner,
ETag: strings.Trim(head.RemoteETag, `"`),
VersionID: head.VersionID,
SizeBytes: head.SizeBytes,
MD5Sum: md5sum,
SHA256: sha256sum,
Anchors: names,
PreviousMD5Sum: head.MD5Sum,
AttestedAt: nowISO(),
}
// The shape that stored SHA-256 in md5sum keeps it as the legacy anchor.
preservedLegacy := ""
if head.LegacySHA256 == "" && validSHA256Hex(strings.ToLower(head.MD5Sum)) {
preservedLegacy = strings.ToLower(head.MD5Sum)
}
if err := ctx.Err(); err != nil {
return verdict, err
}
written, err := coordinator.syncStore.AttestObjectSyncHeadMD5(ctx, head, attestation, preservedLegacy)
if err != nil {
return verdict, err
}
if !written {
verdict.outcome, verdict.detail = backfillBusy, "archived receipt changed during verification"
return verdict, nil
}
attested := head
attested.MD5Sum = md5sum
attested.MD5Attestation = &attestation
if preservedLegacy != "" {
attested.LegacySHA256 = preservedLegacy
}
if err := coordinator.appendAttestationAudit(ctx, attested); err != nil {
return verdict, err
}
verdict.outcome = backfillAttested
verdict.head = &attested
return verdict, nil
}
var errArchivedIdentityChanged = errors.New("archived object identity changed during verification")
func archivedIdentityChange(head domain.ObjectSyncObjectHeadRecord, remote ObjectMetadata) string {
switch {
case !remote.Exists:
return "archived object is missing"
case strings.Trim(remote.ETag, `"`) != strings.Trim(head.RemoteETag, `"`):
return fmt.Sprintf("archived ETag is %q, receipt records %q", remote.ETag, head.RemoteETag)
case remote.SizeBytes != head.SizeBytes:
return fmt.Sprintf("archived size is %d, receipt records %d", remote.SizeBytes, head.SizeBytes)
case remote.VersionID != head.VersionID:
return fmt.Sprintf("archived version is %q, receipt records %q", remote.VersionID, head.VersionID)
}
return ""
}
// The stream reads bounded ranges, each conditional on the archived ETag, and
// hashes MD5 and SHA-256 in one pass so the legacy receipt can vouch for it.
func (coordinator *Coordinator) streamArchivedDigests(ctx context.Context, run *digestBackfillRun, head domain.ObjectSyncObjectHeadRecord) (string, string, int64, error) {
md5Hash, sha256Hash := md5.New(), sha256.New()
writer := io.MultiWriter(md5Hash, sha256Hash)
var offset int64
for offset < head.SizeBytes {
length := min(digestBackfillRangeBytes, head.SizeBytes-offset)
read, err := coordinator.readArchivedRange(ctx, run, head, offset, length, writer)
offset += read
if err != nil {
return "", "", offset, err
}
}
return hex.EncodeToString(md5Hash.Sum(nil)), hex.EncodeToString(sha256Hash.Sum(nil)), offset, nil
}
func (coordinator *Coordinator) readArchivedRange(ctx context.Context, run *digestBackfillRun, head domain.ObjectSyncObjectHeadRecord, offset, length int64, writer io.Writer) (int64, error) {
rangeCtx, cancel := context.WithTimeout(ctx, bosclient.DataTimeout)
defer cancel()
body, meta, err := run.reader.GetObjectRange(rangeCtx, *run.connection, run.space.Bucket, head.Key, head.RemoteETag, offset, length)
if err != nil {
return 0, err
}
defer body.Close()
if strings.Trim(meta.ETag, `"`) != strings.Trim(head.RemoteETag, `"`) || (meta.VersionID != "" && meta.VersionID != head.VersionID) {
return 0, errArchivedIdentityChanged
}
read, err := io.Copy(writer, io.LimitReader(body, length+1))
if err != nil {
return read, fmt.Errorf("read range at %d: %w", offset, err)
}
if read != length {
return read, fmt.Errorf("range at %d returned %d bytes, expected %d", offset, read, length)
}
return read, nil
}
type digestAnchors struct {
sha256 string
md5 string
names []string
}
// Every independent receipt that is present must agree. Receipts written by
// drosync fail closed when malformed; provider headers in unknown form are
// ignored because they are optional extras, not drosync evidence.
func collectDigestAnchors(head domain.ObjectSyncObjectHeadRecord, remote ObjectMetadata) (digestAnchors, string) {
var anchors digestAnchors
var conflicts []string
add := func(slot *string, name, value string) {
if *slot != "" && *slot != value {
conflicts = append(conflicts, fmt.Sprintf("%s=%s disagrees with %s", name, value, *slot))
return
}
*slot = value
if !slices.Contains(anchors.names, name) {
anchors.names = append(anchors.names, name)
}
}
addSHA256 := func(name, raw string, strict bool) {
value := strings.ToLower(strings.TrimSpace(raw))
if value == "" {
return
}
if !validSHA256Hex(value) {
if strict {
conflicts = append(conflicts, fmt.Sprintf("%s is malformed", name))
}
return
}
add(&anchors.sha256, name, value)
}
addMD5 := func(name, raw string, strict bool) {
value := strings.TrimSpace(raw)
if value == "" {
return
}
if decoded, err := base64.StdEncoding.DecodeString(value); !strict && err == nil && len(decoded) == md5.Size {
value = hex.EncodeToString(decoded)
}
value = strings.ToLower(value)
if !ValidMD5Hex(value) {
if strict {
conflicts = append(conflicts, fmt.Sprintf("%s is malformed", name))
}
return
}
add(&anchors.md5, name, value)
}
addSHA256(domain.ObjectSyncDigestAnchorHeadSHA256, head.LegacySHA256, true)
if len(head.MD5Sum) == 2*sha256.Size {
addSHA256(domain.ObjectSyncDigestAnchorHeadSHA256, head.MD5Sum, true)
}
addSHA256(domain.ObjectSyncDigestAnchorObjectSHA256, remote.UserMeta["drosync-sha256"], true)
addSHA256(domain.ObjectSyncDigestAnchorStorageSHA256, remote.ContentSHA256, false)
addMD5(domain.ObjectSyncDigestAnchorObjectMD5, remote.UserMeta["drosync-md5sum"], true)
addMD5(domain.ObjectSyncDigestAnchorStorageMD5, remote.ContentMD5, false)
slices.Sort(anchors.names)
return anchors, strings.Join(conflicts, "; ")
}
func attestationEventKey(head domain.ObjectSyncObjectHeadRecord) string {
attestation := head.MD5Attestation
sum := sha256.Sum256([]byte(strings.Join([]string{head.OrganizationID, head.SyncSpaceID, head.Key, attestation.ETag, attestation.VersionID, strconv.FormatInt(attestation.SizeBytes, 10), attestation.MD5Sum}, "\x00")))
return "head.md5.attested:" + hex.EncodeToString(sum[:16])
}
// The event is built only from the persisted attestation, so whichever job
// repairs a missing event writes exactly the same, deduplicated record.
func (coordinator *Coordinator) appendAttestationAudit(ctx context.Context, head domain.ObjectSyncObjectHeadRecord) error {
attestation := head.MD5Attestation
return AppendAudit(ctx, coordinator.syncStore, domain.ObjectSyncAuditEvent{
EventKey: attestationEventKey(head),
OrganizationID: head.OrganizationID,
ActorType: "system",
ActorID: attestation.CoordinatorID,
SessionID: head.SessionID,
ItemID: head.ItemID,
Action: "head.md5.attested",
Target: map[string]any{"spaceId": head.SyncSpaceID, "key": head.Key, "backfillId": attestation.BackfillID},
Before: map[string]any{"md5sum": attestation.PreviousMD5Sum, "legacySha256": head.LegacySHA256},
After: map[string]any{
"md5sum": attestation.MD5Sum, "sha256": attestation.SHA256, "etag": attestation.ETag,
"versionId": attestation.VersionID, "sizeBytes": attestation.SizeBytes, "platformVersion": head.PlatformVersion,
"anchors": attestation.Anchors, "approvedBy": attestation.ApprovedBy, "source": attestation.Source,
},
Result: "success",
})
}
func (coordinator *Coordinator) recordDigestRejection(ctx context.Context, run *digestBackfillRun, head domain.ObjectSyncObjectHeadRecord, outcome, detail string) {
sum := sha256.Sum256([]byte(strings.Join([]string{run.job.ID, head.Key, head.RemoteETag, outcome}, "\x00")))
coordinator.recordSystemAudit(ctx, domain.ObjectSyncAuditEvent{
EventKey: "head.md5.rejected:" + hex.EncodeToString(sum[:16]),
OrganizationID: head.OrganizationID,
ActorType: "system",
ActorID: coordinator.owner,
SessionID: head.SessionID,
ItemID: head.ItemID,
Action: "head.md5.rejected",
Target: map[string]any{"spaceId": head.SyncSpaceID, "key": head.Key, "backfillId": run.job.ID},
Before: map[string]any{"md5sum": head.MD5Sum, "legacySha256": head.LegacySHA256, "etag": head.RemoteETag},
Result: "denied",
Reason: outcome + ": " + detail,
})
}
func spaceKeyPrefix(space domain.ObjectSyncSpaceRecord, relativePrefix string) string {
root := strings.Trim(strings.TrimSpace(space.RootPrefix), "/")
if root == "" {
return relativePrefix
}
return root + "/" + relativePrefix
}
func formatSessionBackfillCursor(index int, cursor string) string {
return strconv.Itoa(index) + "\x00" + cursor
}
func parseSessionBackfillCursor(value string) (int, string) {
indexText, cursor, found := strings.Cut(value, "\x00")
index, err := strconv.Atoi(indexText)
if !found || err != nil || index < 0 {
return 0, ""
}
return index, cursor
}

View File

@ -0,0 +1,614 @@
package objectsync
import (
"bytes"
"context"
"crypto/md5"
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"slices"
"strings"
"sync"
"testing"
"time"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"github.com/baidubce/bce-sdk-go/bce"
)
type rangeRead struct {
key, ifMatch string
offset, length int64
}
// streamingPublisher serves archived bytes and records every read and write so
// tests can prove verification is read-only and conditional on the ETag.
type streamingPublisher struct {
*fakePublisher
mu sync.Mutex
content map[string][]byte
reads []rangeRead
writes int
beforeGet func(key string, offset int64)
shortRead bool
}
func (publisher *streamingPublisher) GetObjectRange(_ context.Context, _ domain.StorageConnectionRecord, bucket, key, ifMatch string, offset, length int64) (io.ReadCloser, ObjectMetadata, error) {
if publisher.beforeGet != nil {
publisher.beforeGet(key, offset)
}
publisher.mu.Lock()
publisher.reads = append(publisher.reads, rangeRead{key, ifMatch, offset, length})
publisher.mu.Unlock()
meta, err := publisher.Head(context.Background(), domain.StorageConnectionRecord{}, bucket, key)
if err != nil || !meta.Exists || meta.ETag != ifMatch {
return nil, ObjectMetadata{}, &bce.BceServiceError{StatusCode: 412}
}
payload := publisher.content[bucket+"/"+key]
end := min(offset+length, int64(len(payload)))
if publisher.shortRead {
end--
}
return io.NopCloser(bytes.NewReader(payload[offset:end])), meta, nil
}
func (publisher *streamingPublisher) PublishConditional(ctx context.Context, connection domain.StorageConnectionRecord, bucket, source, target string, size int64, sourceETag, ifMatch, ifNoneMatch string, metadata map[string]string) (ObjectMetadata, error) {
publisher.writes++
return publisher.fakePublisher.PublishConditional(ctx, connection, bucket, source, target, size, sourceETag, ifMatch, ifNoneMatch, metadata)
}
func (publisher *streamingPublisher) PutInternal(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string, reader io.Reader, size int64, contentType string, metadata map[string]string) (ObjectMetadata, error) {
publisher.writes++
return publisher.fakePublisher.PutInternal(ctx, connection, bucket, key, reader, size, contentType, metadata)
}
func (publisher *streamingPublisher) Delete(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string) error {
publisher.writes++
return publisher.fakePublisher.Delete(ctx, connection, bucket, key)
}
func (publisher *streamingPublisher) readsOf(key string) int {
publisher.mu.Lock()
defer publisher.mu.Unlock()
count := 0
for _, read := range publisher.reads {
if read.key == key {
count++
}
}
return count
}
func digestHex(payload []byte) (string, string) {
md5Sum, sha256Sum := md5.Sum(payload), sha256.Sum256(payload)
return hex.EncodeToString(md5Sum[:]), hex.EncodeToString(sha256Sum[:])
}
func backfillFixture(t *testing.T) (*Coordinator, *store.MemoryStore, *streamingPublisher) {
t.Helper()
coordinator, data, fake := coordinatorFixture(t)
publisher := &streamingPublisher{fakePublisher: fake, content: map[string][]byte{}}
coordinator.publisher = publisher
return coordinator, data, publisher
}
type legacyShape struct {
md5sum, legacySHA256, metadataSHA256 string
}
// seedLegacyArchive stores a v1-era head whose receipt has no MD5 together
// with the archived object bytes the coordinator will verify.
func seedLegacyArchive(t *testing.T, data *store.MemoryStore, publisher *streamingPublisher, key string, payload []byte, shape legacyShape) domain.ObjectSyncObjectHeadRecord {
t.Helper()
head := domain.ObjectSyncObjectHeadRecord{
OrganizationID: domain.DefaultOrganizationID, SyncSpaceID: "space", Key: key, PlatformVersion: 1,
RemoteETag: "legacy-" + key, SizeBytes: int64(len(payload)), MD5Sum: shape.md5sum, LegacySHA256: shape.legacySHA256,
SessionID: "v1-session", ItemID: "v1-item-" + key, UpdatedAt: "2026-09-01T00:00:00Z",
}
if err := data.UpsertObjectSyncHead(t.Context(), head); err != nil {
t.Fatal(err)
}
meta := map[string]string{}
if shape.metadataSHA256 != "" {
meta["drosync-sha256"] = shape.metadataSHA256
}
publisher.objects["bucket/"+key] = ObjectMetadata{Exists: true, ETag: head.RemoteETag, SizeBytes: head.SizeBytes, UserMeta: meta}
publisher.content["bucket/"+key] = payload
return head
}
func seedBackfillJob(t *testing.T, data *store.MemoryStore, job domain.ObjectSyncDigestBackfillRecord) {
t.Helper()
job.OrganizationID = domain.DefaultOrganizationID
job.SyncSpaceID = "space"
job.Status = "pending"
job.CreatedBy = "admin-user"
job.Reason = "verify v1 archives"
if job.ID == "" {
job.ID = "backfill"
}
job.OperationID = "op-" + job.ID
job.CreatedAt = nowISO()
if err := data.UpsertObjectSyncDigestBackfill(t.Context(), job); err != nil {
t.Fatal(err)
}
}
func runBackfill(t *testing.T, coordinator *Coordinator, data *store.MemoryStore, id string) *domain.ObjectSyncDigestBackfillRecord {
t.Helper()
if _, err := coordinator.executeStage(t.Context(), phaseDigestBackfill); err != nil {
t.Fatal(err)
}
job, err := data.GetObjectSyncDigestBackfill(t.Context(), id)
if err != nil || job == nil {
t.Fatalf("job %s: %v", id, err)
}
return job
}
func auditActions(t *testing.T, data *store.MemoryStore, action string) []domain.ObjectSyncAuditEvent {
t.Helper()
events, err := data.ListObjectSyncAudit(t.Context(), domain.ObjectSyncListQuery{OrganizationID: domain.DefaultOrganizationID, Limit: 1000})
if err != nil {
t.Fatal(err)
}
var matched []domain.ObjectSyncAuditEvent
for _, event := range events {
if event.Action == action {
matched = append(matched, event)
}
}
return matched
}
func TestDigestBackfillAttestsLegacyShapesFromArchivedBytes(t *testing.T) {
payload := []byte("legacy mcap payload archived by drosync v1")
md5sum, sha := digestHex(payload)
for name, test := range map[string]struct {
shape legacyShape
anchors []string
wantLegacy string
}{
"retired-sha256-field": {legacyShape{legacySHA256: sha, metadataSHA256: sha}, []string{domain.ObjectSyncDigestAnchorHeadSHA256, domain.ObjectSyncDigestAnchorObjectSHA256}, sha},
"sha256-in-md5-field": {legacyShape{md5sum: sha}, []string{domain.ObjectSyncDigestAnchorHeadSHA256}, sha},
"missing-digest": {legacyShape{metadataSHA256: sha}, []string{domain.ObjectSyncDigestAnchorObjectSHA256}, ""},
"malformed-digest": {legacyShape{md5sum: "bad", metadataSHA256: strings.ToUpper(sha)}, []string{domain.ObjectSyncDigestAnchorObjectSHA256}, ""},
} {
t.Run(name, func(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
key := "incoming/node-04/episode.mcap"
before := seedLegacyArchive(t, data, publisher, key, payload, test.shape)
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
job := runBackfill(t, coordinator, data, "backfill")
if job.Status != "completed" || job.Counts.Attested != 1 || job.Counts.BytesRead != int64(len(payload)) || job.Counts.ObjectsRead != 1 || len(job.Exceptions) != 0 || job.CoordinatorOwner != "" {
t.Fatalf("job=%#v", job)
}
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
attestation := head.MD5Attestation
if head.MD5Sum != md5sum || attestation == nil || !AttestationBindsHead(*head) {
t.Fatalf("head not attested from archived bytes: %#v", head)
}
if attestation.SHA256 != sha || attestation.BackfillID != "backfill" || attestation.ApprovedBy != "admin-user" || attestation.CoordinatorID != "test-owner" ||
attestation.PreviousMD5Sum != before.MD5Sum || !slices.Equal(attestation.Anchors, test.anchors) {
t.Fatalf("attestation=%#v", attestation)
}
if head.RemoteETag != before.RemoteETag || head.PlatformVersion != before.PlatformVersion || head.SessionID != before.SessionID || head.ItemID != before.ItemID || head.LegacySHA256 != test.wantLegacy {
t.Fatalf("receipt identity changed: %#v", head)
}
if publisher.writes != 0 || len(publisher.publishedKeys) != 0 {
t.Fatal("verification wrote to object storage")
}
for _, read := range publisher.reads {
if read.ifMatch != before.RemoteETag {
t.Fatalf("unconditional read: %#v", read)
}
}
events := auditActions(t, data, "head.md5.attested")
if len(events) != 1 || events[0].After["md5sum"] != md5sum || events[0].After["sha256"] != sha || events[0].Target["backfillId"] != "backfill" || events[0].ActorType != "system" {
t.Fatalf("attestation audit=%#v", events)
}
if len(auditActions(t, data, "space.digest_backfill.completed")) != 1 {
t.Fatal("job completion not audited")
}
space, _ := data.GetObjectSyncSpace(t.Context(), "space")
if _, acquired, err := data.AcquireObjectSyncLease(t.Context(), domain.ObjectSyncLeaseRecord{ResourceHash: ObjectLeaseResourceHash(*space, key), OwnerSessionID: "next", OwnerItemID: "next"}, time.Minute); err != nil || !acquired {
t.Fatal("verification lease was not released")
}
})
}
}
func TestDigestBackfillReadsLargeObjectsInConditionalRanges(t *testing.T) {
previous := digestBackfillRangeBytes
digestBackfillRangeBytes = 7
t.Cleanup(func() { digestBackfillRangeBytes = previous })
coordinator, data, publisher := backfillFixture(t)
payload := []byte("twenty bytes of data")
md5sum, sha := digestHex(payload)
seedLegacyArchive(t, data, publisher, "incoming/big.bin", payload, legacyShape{legacySHA256: sha})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
if job := runBackfill(t, coordinator, data, "backfill"); job.Counts.Attested != 1 {
t.Fatalf("job=%#v", job)
}
want := []rangeRead{{"incoming/big.bin", "legacy-incoming/big.bin", 0, 7}, {"incoming/big.bin", "legacy-incoming/big.bin", 7, 7}, {"incoming/big.bin", "legacy-incoming/big.bin", 14, 6}}
if !slices.Equal(publisher.reads, want) {
t.Fatalf("reads=%#v", publisher.reads)
}
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", "incoming/big.bin")
if head.MD5Sum != md5sum {
t.Fatal("ranged digest differs from whole-object MD5")
}
}
func TestDigestBackfillRefusesBytesThatDifferFromLegacyReceipt(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
_, otherSHA := digestHex([]byte("what v1 originally uploaded"))
key := "incoming/corrupted.mcap"
before := seedLegacyArchive(t, data, publisher, key, []byte("bytes found in the archive today"), legacyShape{legacySHA256: otherSHA})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
job := runBackfill(t, coordinator, data, "backfill")
if job.Status != "completed" || job.Counts.Mismatched != 1 || job.Counts.Attested != 0 || len(job.Exceptions) != 1 || job.Exceptions[0].Outcome != backfillContentMismatch || job.Exceptions[0].Key != key {
t.Fatalf("job=%#v", job)
}
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if head.MD5Sum != before.MD5Sum || head.MD5Attestation != nil {
t.Fatalf("mismatched archive was attested: %#v", head)
}
rejected := auditActions(t, data, "head.md5.rejected")
if len(rejected) != 1 || rejected[0].Result != "denied" || !strings.Contains(rejected[0].Reason, backfillContentMismatch) {
t.Fatalf("rejection audit=%#v", rejected)
}
}
func TestDigestBackfillNeedsAnIndependentAnchorUnlessWaived(t *testing.T) {
for _, waived := range []bool{false, true} {
t.Run(map[bool]string{false: "refused", true: "waived"}[waived], func(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
payload := []byte("no receipt survived for this archive")
md5sum, _ := digestHex(payload)
seedLegacyArchive(t, data, publisher, "incoming/orphan.bin", payload, legacyShape{})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true, AllowUnanchored: waived})
job := runBackfill(t, coordinator, data, "backfill")
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", "incoming/orphan.bin")
if !waived {
if job.Counts.Unanchored != 1 || len(publisher.reads) != 0 || head.MD5Attestation != nil || job.Exceptions[0].Outcome != backfillUnanchored {
t.Fatalf("unanchored archive was read or attested: job=%#v head=%#v", job, head)
}
return
}
if job.Counts.Attested != 1 || head.MD5Sum != md5sum || !slices.Equal(head.MD5Attestation.Anchors, []string{domain.ObjectSyncDigestAnchorNoneAdminWaive}) {
t.Fatalf("waived verification: job=%#v head=%#v", job, head)
}
})
}
}
func TestDigestBackfillFailsClosedOnDisagreeingReceipts(t *testing.T) {
payload := []byte("archived bytes")
md5sum, sha := digestHex(payload)
_, otherSHA := digestHex([]byte("other bytes"))
for name, mutate := range map[string]func(*domain.ObjectSyncObjectHeadRecord, *ObjectMetadata){
"head-vs-metadata-sha256": func(head *domain.ObjectSyncObjectHeadRecord, remote *ObjectMetadata) {
head.LegacySHA256 = sha
remote.UserMeta["drosync-sha256"] = otherSHA
},
"malformed-metadata-sha256": func(head *domain.ObjectSyncObjectHeadRecord, remote *ObjectMetadata) {
head.LegacySHA256 = sha
remote.UserMeta["drosync-sha256"] = "not-a-digest"
},
"storage-md5-disagrees": func(head *domain.ObjectSyncObjectHeadRecord, remote *ObjectMetadata) {
head.LegacySHA256 = sha
remote.UserMeta["drosync-md5sum"] = strings.Repeat("0", 32)
remote.ContentMD5 = md5sum
},
} {
t.Run(name, func(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
key := "incoming/conflicting.bin"
head := seedLegacyArchive(t, data, publisher, key, payload, legacyShape{})
remote := publisher.objects["bucket/"+key]
mutate(&head, &remote)
publisher.objects["bucket/"+key] = remote
data.UpsertObjectSyncHead(t.Context(), head)
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
job := runBackfill(t, coordinator, data, "backfill")
stored, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if job.Counts.Mismatched != 1 || stored.MD5Attestation != nil || len(publisher.reads) != 0 {
t.Fatalf("conflicting receipts were trusted: job=%#v reads=%d", job, len(publisher.reads))
}
})
}
}
type headCASStore struct {
*store.MemoryStore
beforeAttest func()
}
func (data *headCASStore) AttestObjectSyncHeadMD5(ctx context.Context, expected domain.ObjectSyncObjectHeadRecord, attestation domain.ObjectSyncDigestAttestation, legacy string) (bool, error) {
if data.beforeAttest != nil {
data.beforeAttest()
}
return data.MemoryStore.AttestObjectSyncHeadMD5(ctx, expected, attestation, legacy)
}
func TestDigestBackfillLeavesChangedOrBusyArchivesUntouched(t *testing.T) {
payload := []byte("archived bytes for identity checks")
_, sha := digestHex(payload)
key := "incoming/identity.bin"
for name, test := range map[string]struct {
outcome string
reads bool
arrange func(*testing.T, *Coordinator, *store.MemoryStore, *streamingPublisher)
}{
"etag-changed": {backfillDrifted, false, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
remote := p.objects["bucket/"+key]
remote.ETag = "rewritten"
p.objects["bucket/"+key] = remote
}},
"size-changed": {backfillDrifted, false, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
remote := p.objects["bucket/"+key]
remote.SizeBytes++
p.objects["bucket/"+key] = remote
}},
"version-changed": {backfillDrifted, false, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
remote := p.objects["bucket/"+key]
remote.VersionID = "v9"
p.objects["bucket/"+key] = remote
}},
"object-missing": {backfillDrifted, false, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
delete(p.objects, "bucket/"+key)
}},
"reconcile-marked-drift": {backfillDrifted, false, func(t *testing.T, _ *Coordinator, data *store.MemoryStore, _ *streamingPublisher) {
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
head.Drifted = true
data.UpsertObjectSyncHead(t.Context(), *head)
}},
"receipt-without-etag": {backfillUnbound, false, func(t *testing.T, _ *Coordinator, data *store.MemoryStore, _ *streamingPublisher) {
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
head.RemoteETag = ""
data.UpsertObjectSyncHead(t.Context(), *head)
}},
"archive-storage-class": {backfillUnreadableClass, false, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
remote := p.objects["bucket/"+key]
remote.StorageClass = "ARCHIVE"
p.objects["bucket/"+key] = remote
}},
"key-leased-by-upload": {backfillBusy, false, func(t *testing.T, _ *Coordinator, data *store.MemoryStore, _ *streamingPublisher) {
space, _ := data.GetObjectSyncSpace(t.Context(), "space")
if _, ok, err := data.AcquireObjectSyncLease(t.Context(), domain.ObjectSyncLeaseRecord{ResourceHash: ObjectLeaseResourceHash(*space, key), OwnerSessionID: "upload", OwnerItemID: "upload-item"}, time.Minute); err != nil || !ok {
t.Fatal(err)
}
}},
"replaced-during-read": {backfillDrifted, true, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
p.beforeGet = func(string, int64) {
p.fakePublisher.mu.Lock()
remote := p.objects["bucket/"+key]
remote.ETag = "replaced"
p.objects["bucket/"+key] = remote
p.fakePublisher.mu.Unlock()
}
}},
"short-read": {backfillReadFailed, true, func(_ *testing.T, _ *Coordinator, _ *store.MemoryStore, p *streamingPublisher) {
p.shortRead = true
}},
"receipt-changed-before-write": {backfillBusy, true, func(t *testing.T, coordinator *Coordinator, data *store.MemoryStore, _ *streamingPublisher) {
wrapped := &headCASStore{MemoryStore: data}
wrapped.beforeAttest = func() {
head, _ := data.GetObjectSyncHead(context.Background(), domain.DefaultOrganizationID, "space", key)
head.PlatformVersion++
data.UpsertObjectSyncHead(context.Background(), *head)
}
coordinator.syncStore = wrapped
}},
} {
t.Run(name, func(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
seedLegacyArchive(t, data, publisher, key, payload, legacyShape{legacySHA256: sha})
test.arrange(t, coordinator, data, publisher)
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
job := runBackfill(t, coordinator, data, "backfill")
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if job.Status != "completed" || job.Counts.Attested != 0 || len(job.Exceptions) != 1 || job.Exceptions[0].Outcome != test.outcome {
t.Fatalf("job=%#v", job)
}
if head.MD5Attestation != nil || ValidMD5Hex(head.MD5Sum) {
t.Fatalf("unsafe attestation: %#v", head)
}
if (len(publisher.reads) > 0) != test.reads || publisher.writes != 0 {
t.Fatalf("reads=%d writes=%d", len(publisher.reads), publisher.writes)
}
})
}
}
type flakyAuditStore struct {
*store.MemoryStore
failAttestationAudit bool
}
func (data *flakyAuditStore) AppendObjectSyncAudit(ctx context.Context, event domain.ObjectSyncAuditEvent) error {
if data.failAttestationAudit && event.Action == "head.md5.attested" {
return errors.New("audit store unavailable")
}
return data.MemoryStore.AppendObjectSyncAudit(ctx, event)
}
func TestDigestBackfillIsIdempotentAndRepairsMissingAudit(t *testing.T) {
_, data, publisher := backfillFixture(t)
wrapped := &flakyAuditStore{MemoryStore: data, failAttestationAudit: true}
coordinator, err := NewCoordinator(wrapped, publisher, "test-owner")
if err != nil {
t.Fatal(err)
}
payload := []byte("idempotent archive")
md5sum, sha := digestHex(payload)
key := "incoming/once.bin"
seedLegacyArchive(t, data, publisher, key, payload, legacyShape{legacySHA256: sha})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{ID: "first", WholeSpace: true})
if _, err := coordinator.executeStage(t.Context(), phaseDigestBackfill); err == nil {
t.Fatal("missing audit was not surfaced")
}
first, _ := data.GetObjectSyncDigestBackfill(t.Context(), "first")
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if first.Status != "failed" || head.MD5Sum != md5sum || len(auditActions(t, data, "head.md5.attested")) != 0 {
t.Fatalf("first=%#v head=%#v", first, head)
}
wrapped.failAttestationAudit = false
for _, id := range []string{"second", "third"} {
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{ID: id, WholeSpace: true})
job := runBackfill(t, coordinator, data, id)
if job.Status != "completed" || job.Counts.AlreadyAttested != 1 || job.Counts.Attested != 0 || job.Counts.BytesRead != 0 {
t.Fatalf("%s=%#v", id, job)
}
}
events := auditActions(t, data, "head.md5.attested")
if len(events) != 1 || events[0].Target["backfillId"] != "first" || publisher.readsOf(key) != 1 {
t.Fatalf("events=%#v reads=%d", events, publisher.readsOf(key))
}
after, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if after.MD5Attestation.AttestedAt != head.MD5Attestation.AttestedAt {
t.Fatal("rerun rewrote the attestation")
}
}
func TestDigestBackfillStopsAtItsBudget(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
payload := []byte("0123456789")
_, sha := digestHex(payload)
for _, key := range []string{"incoming/a.bin", "incoming/b.bin"} {
seedLegacyArchive(t, data, publisher, key, payload, legacyShape{legacySHA256: sha})
}
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{ID: "objects", WholeSpace: true, MaxObjects: 1})
job := runBackfill(t, coordinator, data, "objects")
if job.Status != "completed" || !job.BudgetExhausted || job.Counts.Attested != 1 || job.Counts.ObjectsRead != 1 || publisher.readsOf("incoming/b.bin") != 0 {
t.Fatalf("object budget: %#v", job)
}
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{ID: "bytes", WholeSpace: true, MaxBytes: 9})
job = runBackfill(t, coordinator, data, "bytes")
if !job.BudgetExhausted || job.Counts.AlreadyAttested != 1 || job.Counts.Attested != 0 || publisher.readsOf("incoming/b.bin") != 0 {
t.Fatalf("byte budget: %#v", job)
}
}
func TestDigestBackfillSessionScopeUsesCloudBytesNotDeviceDigests(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
same, changed := []byte("episode bytes kept on the device"), []byte("daily report archived in september")
sameMD5, sameSHA := digestHex(same)
changedMD5, changedSHA := digestHex(changed)
deviceChangedMD5, _ := digestHex([]byte("daily report regenerated in october"))
seedLegacyArchive(t, data, publisher, "incoming/episode.mcap", same, legacyShape{legacySHA256: sameSHA})
seedLegacyArchive(t, data, publisher, "incoming/report.xlsx", changed, legacyShape{legacySHA256: changedSHA})
for _, sessionID := range []string{"partial-1", "partial-2"} {
if err := data.UpsertObjectSyncSession(t.Context(), domain.ObjectSyncSessionRecord{ID: sessionID, OrganizationID: domain.DefaultOrganizationID, SyncSpaceID: "space", Status: domain.ObjectSyncSessionPartial}); err != nil {
t.Fatal(err)
}
if err := data.UpsertObjectSyncItems(t.Context(), []domain.ObjectSyncItemRecord{
{ID: sessionID + "-episode", OrganizationID: domain.DefaultOrganizationID, SessionID: sessionID, TargetKey: "incoming/episode.mcap", SizeBytes: int64(len(same)), MD5Sum: sameMD5, Status: domain.ObjectSyncItemDenied, Action: legacyDeniedAction},
{ID: sessionID + "-report", OrganizationID: domain.DefaultOrganizationID, SessionID: sessionID, TargetKey: "incoming/report.xlsx", SizeBytes: int64(len(changed)), MD5Sum: deviceChangedMD5, Status: domain.ObjectSyncItemDenied, Action: legacyDeniedAction},
{ID: sessionID + "-uploaded", OrganizationID: domain.DefaultOrganizationID, SessionID: sessionID, TargetKey: "incoming/new.bin", Status: domain.ObjectSyncItemCommitted, Action: "create"},
}); err != nil {
t.Fatal(err)
}
}
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{SessionIDs: []string{"partial-1", "partial-2"}})
job := runBackfill(t, coordinator, data, "backfill")
if job.Counts.Attested != 2 || job.Counts.Scanned != 2 || job.Counts.ItemsMatched != 2 || job.Counts.ItemsConflicting != 2 || job.Cursor != formatSessionBackfillCursor(2, "") {
t.Fatalf("job=%#v", job)
}
if publisher.readsOf("incoming/episode.mcap") != 1 || publisher.readsOf("incoming/report.xlsx") != 1 || publisher.readsOf("incoming/new.bin") != 0 {
t.Fatalf("keys were not read exactly once: %#v", publisher.reads)
}
report, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", "incoming/report.xlsx")
if report.MD5Sum != changedMD5 || report.MD5Sum == deviceChangedMD5 {
t.Fatalf("head recorded a device digest instead of the archived bytes: %#v", report)
}
for _, id := range []string{"partial-1-episode", "partial-2-report"} {
item, _ := data.GetObjectSyncItem(t.Context(), id)
if item.Status != domain.ObjectSyncItemDenied {
t.Fatal("closed session items were rewritten")
}
}
episode, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", "incoming/episode.mcap")
if episode.MD5Sum != sameMD5 {
t.Fatal("matching archive not attested")
}
}
func TestDigestBackfillPrefixScopeOnlyVisitsItsRange(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
payload := []byte("prefix payload")
_, sha := digestHex(payload)
keys := []string{"incoming/a/1.bin", "incoming/b/1.bin", "incoming/b/2.bin", "incoming/ba/1.bin", "incoming/c/1.bin"}
for _, key := range keys {
seedLegacyArchive(t, data, publisher, key, payload, legacyShape{legacySHA256: sha})
}
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{KeyPrefix: "b/"})
job := runBackfill(t, coordinator, data, "backfill")
if job.Counts.Attested != 2 || job.Counts.Scanned != 2 {
t.Fatalf("job=%#v", job)
}
for _, key := range keys {
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if inPrefix := strings.HasPrefix(key, "incoming/b/"); (head.MD5Attestation != nil) != inPrefix || (publisher.readsOf(key) > 0) != inPrefix {
t.Fatalf("%s attested=%v reads=%d", key, head.MD5Attestation != nil, publisher.readsOf(key))
}
}
}
func TestDigestBackfillClaimLossStopsBeforeCompletion(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
payload := []byte("claim loss payload")
_, sha := digestHex(payload)
seedLegacyArchive(t, data, publisher, "incoming/claimed.bin", payload, legacyShape{legacySHA256: sha})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
publisher.beforeGet = func(string, int64) {
job, _ := data.GetObjectSyncDigestBackfill(context.Background(), "backfill")
job.CoordinatorOwner = "replacement-coordinator"
data.UpsertObjectSyncDigestBackfill(context.Background(), *job)
}
if _, err := coordinator.executeStage(t.Context(), phaseDigestBackfill); !errors.Is(err, errClaimLost) {
t.Fatalf("claim loss not detected: %v", err)
}
job, _ := data.GetObjectSyncDigestBackfill(t.Context(), "backfill")
if job.Status != "running" || job.CoordinatorOwner != "replacement-coordinator" || job.CompletedAt != "" {
t.Fatalf("stale coordinator completed the job: %#v", job)
}
}
func TestPlanningComparesLaterSyncsWithAttestedMD5(t *testing.T) {
coordinator, data, publisher := backfillFixture(t)
payload := []byte("planned against an attested receipt")
md5sum, sha := digestHex(payload)
key := "planned/000"
seedLegacyArchive(t, data, publisher, key, payload, legacyShape{legacySHA256: sha})
seedBackfillJob(t, data, domain.ObjectSyncDigestBackfillRecord{WholeSpace: true})
runBackfill(t, coordinator, data, "backfill")
for name, local := range map[string]string{"same": md5sum, "rewritten": strings.Repeat("c", 32)} {
t.Run(name, func(t *testing.T) {
seedPlanning(t, data, 1)
item, _ := data.GetObjectSyncItem(t.Context(), "plan-000")
item.MD5Sum = local
item.SizeBytes = int64(len(payload))
data.UpsertObjectSyncItems(t.Context(), []domain.ObjectSyncItemRecord{*item})
if _, err := coordinator.executeStage(t.Context(), phasePlanning); err != nil {
t.Fatal(err)
}
planned, _ := data.GetObjectSyncItem(t.Context(), "plan-000")
head, _ := data.GetObjectSyncHead(t.Context(), domain.DefaultOrganizationID, "space", key)
if local == md5sum {
if planned.Status != domain.ObjectSyncItemUnchanged || planned.ExpectedBase.ETag != head.RemoteETag || planned.ExpectedBase.PlatformVersion != head.PlatformVersion {
t.Fatalf("matching file not unchanged: %#v", planned)
}
return
}
if planned.Status != domain.ObjectSyncItemDenied || !strings.Contains(planned.LastError, "同名文件内容冲突") {
t.Fatalf("rewritten file not reported as a content conflict: %#v", planned)
}
})
}
}

View File

@ -46,6 +46,7 @@ type MemoryStore struct {
objectSyncReleases map[string]domain.ObjectSyncReleaseRecord
objectSyncClientReleases map[string]domain.ObjectSyncClientReleaseRecord
objectSyncReconciles map[string]domain.ObjectSyncReconcileRecord
objectSyncBackfills map[string]domain.ObjectSyncDigestBackfillRecord
objectSyncIdempotency map[string]domain.ObjectSyncIdempotencyRecord
objectSyncRateLimits map[string]domain.ObjectSyncRateLimitRecord
databaseCollections map[string][]map[string]any
@ -86,6 +87,7 @@ func NewMemoryStore() *MemoryStore {
objectSyncReleases: map[string]domain.ObjectSyncReleaseRecord{},
objectSyncClientReleases: map[string]domain.ObjectSyncClientReleaseRecord{},
objectSyncReconciles: map[string]domain.ObjectSyncReconcileRecord{},
objectSyncBackfills: map[string]domain.ObjectSyncDigestBackfillRecord{},
objectSyncIdempotency: map[string]domain.ObjectSyncIdempotencyRecord{},
objectSyncRateLimits: map[string]domain.ObjectSyncRateLimitRecord{},
databaseCollections: map[string][]map[string]any{},

View File

@ -51,6 +51,7 @@ type MongoStore struct {
objectSyncReleases *mongo.Collection
objectSyncClientReleases *mongo.Collection
objectSyncReconciles *mongo.Collection
objectSyncBackfills *mongo.Collection
objectSyncIdempotency *mongo.Collection
objectSyncRateLimits *mongo.Collection
settings *mongo.Collection
@ -91,6 +92,7 @@ func NewMongoStore(db *mongo.Database) *MongoStore {
objectSyncReleases: db.Collection("objectSyncReleases"),
objectSyncClientReleases: db.Collection("objectSyncClientReleases"),
objectSyncReconciles: db.Collection("objectSyncReconciles"),
objectSyncBackfills: db.Collection("objectSyncDigestBackfills"),
objectSyncIdempotency: db.Collection("objectSyncIdempotency"),
objectSyncRateLimits: db.Collection("objectSyncRateLimits"),
settings: db.Collection("settings"),
@ -236,6 +238,7 @@ func (store *MongoStore) DeleteOrganizationCascade(ctx context.Context, id strin
store.objectSyncVersions,
store.objectSyncAuditEvents,
store.objectSyncReleases,
store.objectSyncBackfills,
store.objectSyncClientReleases,
store.settings,
} {

View File

@ -226,6 +226,12 @@ func (store *MongoStore) EnsureIndexes(ctx context.Context) error {
uniqueIndex("operationId_unique", bson.D{{Key: "operationId", Value: 1}}),
index("organizationId_status_createdAt", bson.D{{Key: "organizationId", Value: 1}, {Key: "status", Value: 1}, {Key: "createdAt", Value: -1}}),
},
store.objectSyncBackfills: {
uniqueIndex("id_unique", bson.D{{Key: "id", Value: 1}}),
uniqueIndex("operationId_unique", bson.D{{Key: "operationId", Value: 1}}),
index("status_createdAt", bson.D{{Key: "status", Value: 1}, {Key: "createdAt", Value: 1}}),
index("organizationId_syncSpaceId_createdAt", bson.D{{Key: "organizationId", Value: 1}, {Key: "syncSpaceId", Value: 1}, {Key: "createdAt", Value: -1}}),
},
store.objectSyncIdempotency: {
uniqueIndex("id_unique", bson.D{{Key: "id", Value: 1}}),
ttlIndex("expiresAt_ttl", bson.D{{Key: "expiresAt", Value: 1}}, 0),

View File

@ -81,8 +81,15 @@ type ObjectSyncStore interface {
GetObjectSyncHead(context.Context, string, string, string) (*domain.ObjectSyncObjectHeadRecord, error)
UpsertObjectSyncHead(context.Context, domain.ObjectSyncObjectHeadRecord) error
ListObjectSyncHeads(context.Context, string, string, string, int) (domain.ObjectSyncHeadPage, error)
AttestObjectSyncHeadMD5(context.Context, domain.ObjectSyncObjectHeadRecord, domain.ObjectSyncDigestAttestation, string) (bool, error)
InsertObjectSyncVersion(context.Context, domain.ObjectSyncVersionRecord) error
UpsertObjectSyncDigestBackfill(context.Context, domain.ObjectSyncDigestBackfillRecord) error
GetObjectSyncDigestBackfill(context.Context, string) (*domain.ObjectSyncDigestBackfillRecord, error)
ClaimObjectSyncDigestBackfill(context.Context, string, time.Duration) (*domain.ObjectSyncDigestBackfillRecord, error)
RenewObjectSyncDigestBackfillClaim(context.Context, string, string, time.Duration) (bool, error)
SaveClaimedObjectSyncDigestBackfill(context.Context, domain.ObjectSyncDigestBackfillRecord, string) (bool, error)
AppendObjectSyncAudit(context.Context, domain.ObjectSyncAuditEvent) error
ListObjectSyncAudit(context.Context, domain.ObjectSyncListQuery) ([]domain.ObjectSyncAuditEvent, error)
ListObjectSyncAuditRange(context.Context, string, string, string, domain.ObjectSyncListQuery) (domain.ObjectSyncAuditPage, error)
@ -611,14 +618,14 @@ func (store *MemoryStore) GetObjectSyncHead(_ context.Context, organizationID, s
if !ok {
return nil, nil
}
copy := item
copy := cloneObjectSyncHead(item)
return &copy, nil
}
func (store *MemoryStore) UpsertObjectSyncHead(_ context.Context, item domain.ObjectSyncObjectHeadRecord) error {
store.mu.Lock()
defer store.mu.Unlock()
store.objectSyncHeads[objectSyncHeadKey(item.OrganizationID, item.SyncSpaceID, item.Key)] = item
store.objectSyncHeads[objectSyncHeadKey(item.OrganizationID, item.SyncSpaceID, item.Key)] = cloneObjectSyncHead(item)
return nil
}
@ -631,7 +638,7 @@ func (store *MemoryStore) ListObjectSyncHeads(_ context.Context, organizationID,
if item.OrganizationID != normalizedOrganizationID(organizationID) || item.SyncSpaceID != syncSpaceID || (decodedCursor != "" && item.Key <= decodedCursor) {
continue
}
items = append(items, item)
items = append(items, cloneObjectSyncHead(item))
}
sort.Slice(items, func(i, j int) bool { return items[i].Key < items[j].Key })
page := domain.ObjectSyncHeadPage{Items: items}
@ -1186,10 +1193,27 @@ func (store *MongoStore) UpsertObjectSyncHead(ctx context.Context, item domain.O
"organizationId": item.OrganizationID,
"syncSpaceId": item.SyncSpaceID,
"key": item.Key,
}, bson.M{"$set": item}, options.Update().SetUpsert(true))
}, objectSyncHeadReplacement(item), options.Update().SetUpsert(true))
return err
}
// A head write replaces the logical receipt. Optional legacy anchors and
// attestations that the new receipt does not carry must not survive it.
func objectSyncHeadReplacement(item domain.ObjectSyncObjectHeadRecord) bson.M {
update := bson.M{"$set": item}
unset := bson.M{}
if item.LegacySHA256 == "" {
unset["sha256"] = ""
}
if item.MD5Attestation == nil {
unset["md5Attestation"] = ""
}
if len(unset) > 0 {
update["$unset"] = unset
}
return update
}
func (store *MongoStore) ListObjectSyncHeads(ctx context.Context, organizationID, syncSpaceID, cursor string, limit int) (domain.ObjectSyncHeadPage, error) {
filter := bson.M{
"organizationId": normalizedOrganizationID(organizationID),

View File

@ -0,0 +1,220 @@
package store
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"slices"
"time"
"droboticflow/internal/domain"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func cloneObjectSyncHead(item domain.ObjectSyncObjectHeadRecord) domain.ObjectSyncObjectHeadRecord {
if item.MD5Attestation != nil {
attestation := *item.MD5Attestation
attestation.Anchors = slices.Clone(attestation.Anchors)
item.MD5Attestation = &attestation
}
return item
}
func cloneObjectSyncDigestBackfill(item domain.ObjectSyncDigestBackfillRecord) domain.ObjectSyncDigestBackfillRecord {
item.SessionIDs = slices.Clone(item.SessionIDs)
item.Exceptions = slices.Clone(item.Exceptions)
return item
}
func validObjectSyncAttestation(attestation domain.ObjectSyncDigestAttestation) error {
digest, err := hex.DecodeString(attestation.MD5Sum)
if err != nil || len(digest) != 16 || hex.EncodeToString(digest) != attestation.MD5Sum {
return fmt.Errorf("attested MD5 must be a lowercase 32-character hex digest")
}
if attestation.Source == "" || attestation.BackfillID == "" || attestation.ETag == "" || len(attestation.Anchors) == 0 {
return fmt.Errorf("attestation must name its source, job, object ETag and anchors")
}
return nil
}
// AttestObjectSyncHeadMD5 is a compare-and-set: the head is updated only if it
// still has exactly the identity and previous digest the coordinator verified.
func (store *MemoryStore) AttestObjectSyncHeadMD5(_ context.Context, expected domain.ObjectSyncObjectHeadRecord, attestation domain.ObjectSyncDigestAttestation, legacySHA256 string) (bool, error) {
if err := validObjectSyncAttestation(attestation); err != nil {
return false, err
}
store.mu.Lock()
defer store.mu.Unlock()
key := objectSyncHeadKey(expected.OrganizationID, expected.SyncSpaceID, expected.Key)
current, ok := store.objectSyncHeads[key]
if !ok || current.Drifted || current.MD5Attestation != nil ||
current.RemoteETag != expected.RemoteETag || current.VersionID != expected.VersionID ||
current.SizeBytes != expected.SizeBytes || current.PlatformVersion != expected.PlatformVersion ||
current.MD5Sum != expected.MD5Sum || current.LegacySHA256 != expected.LegacySHA256 ||
current.SessionID != expected.SessionID || current.ItemID != expected.ItemID {
return false, nil
}
attestation.Anchors = slices.Clone(attestation.Anchors)
current.MD5Sum = attestation.MD5Sum
current.MD5Attestation = &attestation
if legacySHA256 != "" {
current.LegacySHA256 = legacySHA256
}
current.UpdatedAt = attestation.AttestedAt
store.objectSyncHeads[key] = current
return true, nil
}
// Legacy documents may lack fields that the current struct always writes.
func mongoOptionalString(value string) any {
if value == "" {
return bson.M{"$in": bson.A{nil, ""}}
}
return value
}
func objectSyncHeadAttestationFilter(expected domain.ObjectSyncObjectHeadRecord) bson.M {
return bson.M{
"organizationId": normalizedOrganizationID(expected.OrganizationID),
"syncSpaceId": expected.SyncSpaceID,
"key": expected.Key,
"remoteETag": expected.RemoteETag,
"sizeBytes": expected.SizeBytes,
"platformVersion": expected.PlatformVersion,
"versionId": mongoOptionalString(expected.VersionID),
"md5sum": mongoOptionalString(expected.MD5Sum),
"sha256": mongoOptionalString(expected.LegacySHA256),
"sessionId": mongoOptionalString(expected.SessionID),
"itemId": mongoOptionalString(expected.ItemID),
"drifted": bson.M{"$ne": true},
"md5Attestation": bson.M{"$exists": false},
}
}
func (store *MongoStore) AttestObjectSyncHeadMD5(ctx context.Context, expected domain.ObjectSyncObjectHeadRecord, attestation domain.ObjectSyncDigestAttestation, legacySHA256 string) (bool, error) {
if err := validObjectSyncAttestation(attestation); err != nil {
return false, err
}
set := bson.M{"md5sum": attestation.MD5Sum, "md5Attestation": attestation, "updatedAt": attestation.AttestedAt}
if legacySHA256 != "" {
set["sha256"] = legacySHA256
}
result, err := store.objectSyncHeads.UpdateOne(ctx, objectSyncHeadAttestationFilter(expected), bson.M{"$set": set})
return err == nil && result.MatchedCount == 1, err
}
func (store *MemoryStore) UpsertObjectSyncDigestBackfill(_ context.Context, item domain.ObjectSyncDigestBackfillRecord) error {
store.mu.Lock()
defer store.mu.Unlock()
store.objectSyncBackfills[item.ID] = cloneObjectSyncDigestBackfill(item)
return nil
}
func (store *MemoryStore) GetObjectSyncDigestBackfill(_ context.Context, id string) (*domain.ObjectSyncDigestBackfillRecord, error) {
store.mu.Lock()
defer store.mu.Unlock()
item, ok := store.objectSyncBackfills[id]
if !ok {
return nil, nil
}
copy := cloneObjectSyncDigestBackfill(item)
return &copy, nil
}
func (store *MemoryStore) ClaimObjectSyncDigestBackfill(_ context.Context, owner string, ttl time.Duration) (*domain.ObjectSyncDigestBackfillRecord, error) {
store.mu.Lock()
defer store.mu.Unlock()
now := time.Now().UTC()
var claimed *domain.ObjectSyncDigestBackfillRecord
for _, item := range store.objectSyncBackfills {
if (item.Status != "pending" && item.Status != "running") || (item.CoordinatorOwner != "" && item.CoordinatorLeaseUntil.After(now)) {
continue
}
if claimed == nil || item.CreatedAt < claimed.CreatedAt || (item.CreatedAt == claimed.CreatedAt && item.ID < claimed.ID) {
candidate := item
claimed = &candidate
}
}
if claimed == nil {
return nil, nil
}
claimed.Status = "running"
claimed.CoordinatorOwner = owner + ":" + rand.Text()
claimed.CoordinatorLeaseUntil = now.Add(ttl)
claimed.UpdatedAt = now.Format(time.RFC3339Nano)
store.objectSyncBackfills[claimed.ID] = cloneObjectSyncDigestBackfill(*claimed)
copy := cloneObjectSyncDigestBackfill(*claimed)
return &copy, nil
}
func (store *MemoryStore) RenewObjectSyncDigestBackfillClaim(_ context.Context, id, owner string, ttl time.Duration) (bool, error) {
store.mu.Lock()
defer store.mu.Unlock()
item, ok := store.objectSyncBackfills[id]
if !ok || owner == "" || item.CoordinatorOwner != owner || !item.CoordinatorLeaseUntil.After(time.Now()) || (item.Status != "pending" && item.Status != "running") {
return false, nil
}
item.CoordinatorLeaseUntil = time.Now().UTC().Add(ttl)
store.objectSyncBackfills[id] = item
return true, nil
}
func (store *MemoryStore) SaveClaimedObjectSyncDigestBackfill(_ context.Context, item domain.ObjectSyncDigestBackfillRecord, owner string) (bool, error) {
store.mu.Lock()
defer store.mu.Unlock()
current, ok := store.objectSyncBackfills[item.ID]
if !ok || owner == "" || current.CoordinatorOwner != owner || !current.CoordinatorLeaseUntil.After(time.Now()) || (current.Status != "pending" && current.Status != "running") {
return false, nil
}
store.objectSyncBackfills[item.ID] = cloneObjectSyncDigestBackfill(item)
return true, nil
}
func (store *MongoStore) UpsertObjectSyncDigestBackfill(ctx context.Context, item domain.ObjectSyncDigestBackfillRecord) error {
return mongoObjectSyncUpsert(ctx, store.objectSyncBackfills, item.ID, item)
}
func (store *MongoStore) GetObjectSyncDigestBackfill(ctx context.Context, id string) (*domain.ObjectSyncDigestBackfillRecord, error) {
return mongoObjectSyncGet[domain.ObjectSyncDigestBackfillRecord](ctx, store.objectSyncBackfills, bson.M{"id": id})
}
func (store *MongoStore) ClaimObjectSyncDigestBackfill(ctx context.Context, owner string, ttl time.Duration) (*domain.ObjectSyncDigestBackfillRecord, error) {
now := time.Now().UTC()
result := store.objectSyncBackfills.FindOneAndUpdate(
ctx,
bson.M{
"status": bson.M{"$in": bson.A{"pending", "running"}},
"$or": bson.A{
bson.M{"coordinatorOwner": ""},
bson.M{"coordinatorOwner": bson.M{"$exists": false}},
bson.M{"coordinatorLeaseUntil": bson.M{"$lte": now}},
},
},
bson.M{"$set": bson.M{
"status": "running",
"updatedAt": now.Format(time.RFC3339Nano),
"coordinatorOwner": owner + ":" + rand.Text(),
"coordinatorLeaseUntil": now.Add(ttl),
}},
options.FindOneAndUpdate().SetSort(bson.D{{Key: "createdAt", Value: 1}}).SetReturnDocument(options.After),
)
var item domain.ObjectSyncDigestBackfillRecord
if err := result.Decode(&item); err == mongo.ErrNoDocuments {
return nil, nil
} else if err != nil {
return nil, err
}
return &item, nil
}
func (store *MongoStore) RenewObjectSyncDigestBackfillClaim(ctx context.Context, id, owner string, ttl time.Duration) (bool, error) {
return renewObjectSyncBackgroundClaim(ctx, store.objectSyncBackfills, id, owner, ttl)
}
func (store *MongoStore) SaveClaimedObjectSyncDigestBackfill(ctx context.Context, item domain.ObjectSyncDigestBackfillRecord, owner string) (bool, error) {
return saveObjectSyncBackgroundClaim(ctx, store.objectSyncBackfills, item.ID, owner, item)
}

View File

@ -0,0 +1,169 @@
package store
import (
"reflect"
"strings"
"testing"
"time"
"droboticflow/internal/domain"
"go.mongodb.org/mongo-driver/bson"
)
func legacyHeadFixture() domain.ObjectSyncObjectHeadRecord {
return domain.ObjectSyncObjectHeadRecord{
OrganizationID: domain.DefaultOrganizationID, SyncSpaceID: "space", Key: "root/a.mcap", PlatformVersion: 1,
RemoteETag: "legacy-etag", SizeBytes: 4, LegacySHA256: strings.Repeat("a", 64), SessionID: "v1-session", ItemID: "v1-item",
}
}
func attestationFixture() domain.ObjectSyncDigestAttestation {
return domain.ObjectSyncDigestAttestation{
Source: domain.ObjectSyncDigestSourceCoordinatorStream, BackfillID: "backfill", ETag: "legacy-etag", SizeBytes: 4,
MD5Sum: strings.Repeat("b", 32), SHA256: strings.Repeat("a", 64), Anchors: []string{domain.ObjectSyncDigestAnchorHeadSHA256}, AttestedAt: "2026-10-30T00:00:00Z",
}
}
func TestAttestObjectSyncHeadMD5IsCompareAndSet(t *testing.T) {
data := NewMemoryStore()
head := legacyHeadFixture()
if err := data.UpsertObjectSyncHead(t.Context(), head); err != nil {
t.Fatal(err)
}
for name, change := range map[string]func(*domain.ObjectSyncObjectHeadRecord){
"etag": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.RemoteETag = "other" },
"version": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.VersionID = "v2" },
"size": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.SizeBytes = 5 },
"platform": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.PlatformVersion = 2 },
"digest": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.MD5Sum = "bad" },
"legacy": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.LegacySHA256 = strings.Repeat("c", 64) },
"session": func(expected *domain.ObjectSyncObjectHeadRecord) { expected.SessionID = "other" },
} {
expected := head
change(&expected)
if written, err := data.AttestObjectSyncHeadMD5(t.Context(), expected, attestationFixture(), ""); err != nil || written {
t.Fatalf("%s: stale expectation attested: %v %v", name, written, err)
}
}
invalid := attestationFixture()
invalid.MD5Sum = strings.Repeat("B", 32)
if _, err := data.AttestObjectSyncHeadMD5(t.Context(), head, invalid, ""); err == nil {
t.Fatal("non-canonical MD5 accepted")
}
if written, err := data.AttestObjectSyncHeadMD5(t.Context(), head, attestationFixture(), ""); err != nil || !written {
t.Fatalf("matching expectation rejected: %v %v", written, err)
}
stored, _ := data.GetObjectSyncHead(t.Context(), head.OrganizationID, head.SyncSpaceID, head.Key)
if stored.MD5Sum != strings.Repeat("b", 32) || stored.MD5Attestation == nil || stored.LegacySHA256 != head.LegacySHA256 || stored.RemoteETag != head.RemoteETag || stored.UpdatedAt != "2026-10-30T00:00:00Z" {
t.Fatalf("stored=%#v", stored)
}
if written, _ := data.AttestObjectSyncHeadMD5(t.Context(), head, attestationFixture(), ""); written {
t.Fatal("head was attested twice")
}
stored.MD5Attestation.Anchors[0] = "mutated"
again, _ := data.GetObjectSyncHead(t.Context(), head.OrganizationID, head.SyncSpaceID, head.Key)
if again.MD5Attestation.Anchors[0] != domain.ObjectSyncDigestAnchorHeadSHA256 {
t.Fatal("memory store leaked a mutable attestation")
}
drifted := legacyHeadFixture()
drifted.Key = "root/drifted.mcap"
drifted.Drifted = true
data.UpsertObjectSyncHead(t.Context(), drifted)
if written, _ := data.AttestObjectSyncHeadMD5(t.Context(), drifted, attestationFixture(), ""); written {
t.Fatal("drifted head attested")
}
moved := legacyHeadFixture()
moved.Key = "root/moved.mcap"
moved.LegacySHA256 = ""
moved.MD5Sum = strings.Repeat("d", 64)
data.UpsertObjectSyncHead(t.Context(), moved)
if written, _ := data.AttestObjectSyncHeadMD5(t.Context(), moved, attestationFixture(), moved.MD5Sum); !written {
t.Fatal("legacy digest stored in md5sum was not attestable")
}
stored, _ = data.GetObjectSyncHead(t.Context(), moved.OrganizationID, moved.SyncSpaceID, moved.Key)
if stored.LegacySHA256 != moved.MD5Sum || stored.MD5Attestation.PreviousMD5Sum != "" || stored.MD5Sum != strings.Repeat("b", 32) {
t.Fatalf("legacy anchor was not preserved: %#v", stored)
}
}
func TestLegacySHA256DecodesFromRetiredBSONField(t *testing.T) {
raw, err := bson.Marshal(bson.M{"organizationId": "org", "syncSpaceId": "space", "key": "k", "remoteETag": "e", "sizeBytes": int64(4), "sha256": strings.Repeat("a", 64)})
if err != nil {
t.Fatal(err)
}
var head domain.ObjectSyncObjectHeadRecord
if err := bson.Unmarshal(raw, &head); err != nil {
t.Fatal(err)
}
if head.LegacySHA256 != strings.Repeat("a", 64) || head.MD5Sum != "" || head.MD5Attestation != nil {
t.Fatalf("v1 head decoded as %#v", head)
}
}
func TestMongoHeadWritesDropAbsentLegacyAnchorsAndAttestations(t *testing.T) {
fresh := legacyHeadFixture()
fresh.LegacySHA256 = ""
if update := objectSyncHeadReplacement(fresh); !reflect.DeepEqual(update["$unset"], bson.M{"sha256": "", "md5Attestation": ""}) {
t.Fatalf("new receipt keeps stale fields: %#v", update)
}
attested := legacyHeadFixture()
attestation := attestationFixture()
attested.MD5Attestation = &attestation
if update := objectSyncHeadReplacement(attested); update["$unset"] != nil {
t.Fatalf("receipt write dropped its own anchor: %#v", update)
}
}
func TestMongoAttestationFilterMatchesLegacyDocumentsExactly(t *testing.T) {
expected := legacyHeadFixture()
filter := objectSyncHeadAttestationFilter(expected)
missing := bson.M{"$in": bson.A{nil, ""}}
for field, want := range map[string]any{
"remoteETag": "legacy-etag", "sizeBytes": int64(4), "platformVersion": int64(1), "sha256": strings.Repeat("a", 64),
"sessionId": "v1-session", "itemId": "v1-item", "md5sum": missing, "versionId": missing,
"drifted": bson.M{"$ne": true}, "md5Attestation": bson.M{"$exists": false},
} {
if !reflect.DeepEqual(filter[field], want) {
t.Fatalf("%s filter = %#v, want %#v", field, filter[field], want)
}
}
expected.MD5Sum = "bad"
if objectSyncHeadAttestationFilter(expected)["md5sum"] != "bad" {
t.Fatal("malformed digest must match exactly")
}
}
func TestDigestBackfillClaimsOldestJobAndFencesWrites(t *testing.T) {
data := NewMemoryStore()
for _, job := range []domain.ObjectSyncDigestBackfillRecord{
{ID: "newer", Status: "pending", CreatedAt: "2026-10-30T02:00:00Z"},
{ID: "older", Status: "pending", CreatedAt: "2026-10-30T01:00:00Z"},
{ID: "done", Status: "completed", CreatedAt: "2026-10-30T00:00:00Z"},
} {
if err := data.UpsertObjectSyncDigestBackfill(t.Context(), job); err != nil {
t.Fatal(err)
}
}
claimed, err := data.ClaimObjectSyncDigestBackfill(t.Context(), "coordinator", time.Minute)
if err != nil || claimed == nil || claimed.ID != "older" || claimed.Status != "running" || !strings.HasPrefix(claimed.CoordinatorOwner, "coordinator:") {
t.Fatalf("claimed=%#v err=%v", claimed, err)
}
if renewed, _ := data.RenewObjectSyncDigestBackfillClaim(t.Context(), "older", "someone-else", time.Minute); renewed {
t.Fatal("foreign owner renewed the claim")
}
claimed.Status = "completed"
if saved, _ := data.SaveClaimedObjectSyncDigestBackfill(t.Context(), *claimed, "someone-else"); saved {
t.Fatal("foreign owner saved the job")
}
if saved, _ := data.SaveClaimedObjectSyncDigestBackfill(t.Context(), *claimed, claimed.CoordinatorOwner); !saved {
t.Fatal("owner could not complete the job")
}
next, _ := data.ClaimObjectSyncDigestBackfill(t.Context(), "coordinator", time.Minute)
if next == nil || next.ID != "newer" {
t.Fatalf("next=%#v", next)
}
if again, _ := data.ClaimObjectSyncDigestBackfill(t.Context(), "coordinator", time.Minute); again != nil {
t.Fatalf("live claim or completed job was claimed: %#v", again)
}
}