Compare commits

...

17 Commits

Author SHA1 Message Date
eust-w
111332f409 📝 Record live POC directory provisioning acceptance 2026-10-10 18:08:33 +08:00
eust-w
07d8566b4d 🔄 Reconcile new organization IAM provisioning automatically 2026-10-10 17:49:59 +08:00
eust-w
cfbb451879 🛠️ Preserve inline outputs for scoped organizations 2026-10-10 17:39:48 +08:00
eust-w
88e6d08ef0 🔒 Provision isolated BOS directories for new POC organizations 2026-10-10 17:34:58 +08:00
eust-w
109290eeb1 ✨ Add drobotic-pull Rust tool and directory manifest API 2026-09-17 08:16:31 +08:00
eust-w
02c654c0ea 🐛 Resolve storage connection credentials from input bindings in worker 2026-09-07 19:52:43 +08:00
eust-w
eb1f49c146 🐛 Fix sessions respawning after mapping removal
- Add database guard in scheduler sync path to re-check mapping
  existence before calling sync_mapping, closing the race window
  between the 60s in-memory cache refresh and mapping.remove
- Abort orphaned non-terminal sessions when a mapping is removed
- Clear pending journal entry on mapping removal to prevent
  restart-triggered session creation
2026-09-07 17:47:42 +08:00
eust-w
fe10b9c048 🧪 Add live drosync manifest boundary regression 2026-09-06 04:32:10 +08:00
eust-w
6a2056f67b 🧪 Add reproducible drosync POC abort regression and evidence 2026-09-05 23:10:58 +08:00
eust-w
5b80cb68e0 🐛 Fix drosync aborted session lease cleanup and heartbeat guards 2026-09-05 22:39:34 +08:00
eust-w
1f994927c0 🔒 Preserve POC rollback baseline and drosync script permissions 2026-09-05 14:29:14 +08:00
eust-w
0d5578421e 🐛 Stabilize drosync automatic scheduling 2026-09-05 14:22:21 +08:00
eust-w
76cc571c49 📚 Add drosync Chinese user manual 2026-09-05 14:22:20 +08:00
eust-w
685671f390 🔒 Support headless drosync system services 2026-09-05 14:22:20 +08:00
eust-w
927b1f7163 📦 Harden drosync Linux packages 2026-09-05 14:22:20 +08:00
eust-w
0c02227082 ✨ Add bucket-based drosync mappings 2026-09-05 14:22:20 +08:00
eust-w
6ff9e55405 📦 Preserve drosync ARM64 packaging and signed updates 2026-09-05 14:22:20 +08:00
73 changed files with 4237 additions and 186 deletions

View File

@ -2,9 +2,10 @@
## Deployment Environment
- The `dev` branch uses the `poc` deployment and test environment.
- Leave `DEPLOY_ENV` unset on `dev`; the deploy scripts should resolve it to `poc`.
- Do not deploy `dev` to `wjzh` unless the user explicitly overrides this rule.
- The `poc` branch uses the POC environment (namespace `droboticflow`, public entry `120.48.164.19`).
- Keep the pre-Fanyu baseline `d947a20b` plus drosync changes. Do not merge `dev` or Fanyu commits into `poc`.
- Build POC images from `poc` and publish them under `dataflow-poc`, never from the Fanyu development branch.
- Fanyu is a separate environment and is not a target of POC deployment commands.
## Git Workflow

View File

@ -13,6 +13,7 @@
## Document map
- [中文用户使用手册](用户使用手册.md)
- [Product requirements](product-requirements.md)
- [Architecture and state machines](architecture.md)
- [OpenAPI 3.1 contract](openapi.yaml)

View File

@ -0,0 +1,56 @@
# POC drosync repair acceptance — 2026-09-05
Scope: POC `120.48.164.19`, API revision `5b80cb68`; Linux test host
`120.48.123.215`. No Fanyu deployment or merge.
## Reproducible abort regression
From the repository, copy `scripts/drosync-poc-abort-regression.py` to the test
host. The token file must contain an existing authorized device token and must
not be printed or added to source control. Use a prefix permitted by that token.
```sh
python3 /tmp/drosync-poc-abort-regression.py \
--server http://120.48.164.19 \
--token-file /var/lib/drosync/token \
--space oss_KoX8zMpkW8bybqDj \
--prefix payload/repair-20260905 \
--output /srv/drosync-fulltest-20260905-r2/abort-retest-5b80cb68.json
```
The script creates a fresh random object key and two sessions, uploads no object
bytes, and aborts its sessions. Output contains no bearer or signed upload URLs.
Exit code 0 means all assertions passed; nonzero is a failure, not a success marker.
Local regression:
```sh
go test ./internal/api -run TestObjectSyncAbort -count=1
go test ./internal/api ./internal/store -count=1
```
## Actual results
- Before repair: abort did not release the object lease; the aborted session could
still renew it. Keep the earlier failure receipts, rather than replacing them.
- After repair: 8/8 real API assertions passed, including immediate takeover with
fencing 1 → 2, closed-session heartbeat/reserve/commit rejection, repeated abort
preserving the new owner, and rejection of complete after abort.
- Sessions: `oses_q9GHF-0wDHN8uZ6v`, `oses_xb9D0k8R_sdiqiYq`.
- API image: `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:305ea9fb9dfecef081b80ee55f961f19de7a889a3dd9b3769afba43e46d522ac`.
- Large object retest: 5,369,757,696 bytes (5 GiB + 1 MiB), 21 parts,
84.073 seconds, committed Session `oses_7oSRbW34E_vrRLdl`.
- Local SHA-256 equals the server receipt:
`a81fcc42cd9781340440e780df1bb20b1b783d06b1d3bddcd84f4e1ae3e038b4`.
## Boundaries
The large test uploaded real file bytes; it was not just a sparse file or plan.
Independent BOS download/re-hashing remains unverified. The 24-hour low-load soak
is ongoing and spans this API rollout, so it is not a single-version 24-hour
certification. This does not certify all 132 checklist points, peak throughput,
1-TiB files, 20-TiB sessions, disaster recovery, or monthly SLA. The helper is a
sequential abort regression, not a proof of every possible concurrent race.
Detailed command-by-command Feishu manual:
https://horizonrobotics.feishu.cn/wiki/C3JswTXyUiwiX8kMgq8cxyjbnse

View File

@ -12,7 +12,7 @@ Commands:
| `configure --server URL` | Persist the control-plane URL through daemon IPC. |
| `auth --server URL [--profile NAME] --access-key-id ID (--secret-file PATH \| --secret-stdin)` | Validate a scoped drosync Key/Secret, automatically register the device, and store the resulting bearer token in the OS credential store. `DROSYNC_ACCESS_KEY_ID` and `DROSYNC_SECRET` are supported for non-interactive injection. |
| `mapping list` | List local mappings. |
| `mapping add ROOT --space ID [--remote-prefix P] [--include-hidden] [--require-done]` | Add a mapping. |
| `mapping add ROOT (--bucket NAME \| --space ID) [--remote-prefix P] [--include-hidden] [--require-done]` | Add a mapping. `--bucket` resolves only within spaces visible to the current scoped credential; `--space` selects an exact administrative ID. |
| `mapping remove ID` | Remove a mapping without deleting local or remote objects. |
| `plan ID` | Scan, upload a paged manifest, wait for the asynchronous plan, and retain the resumable session without uploading bytes. |
| `sync ID [--dry-run]` | Resume a planned/crashed session or execute scan through final commit. |

View File

@ -23,6 +23,26 @@ sudo systemctl enable --now drosync.service
The portable tarball contains all three binaries and a user systemd unit. Validate the detached GPG signature and SHA-256 before extraction.
## Ubuntu 22.04 ARM64 / RDK S100
Use only the `linux-arm64` DEB or portable tar. These packages contain the CLI
and daemon, omit the desktop GUI and identify themselves as
`releasePlatform=linux-arm64`.
The headless build deliberately does not depend on the desktop D-Bus keyring.
Configure a writable token file in the daemon environment before authenticating:
```bash
sudo install -d -o drosync -g drosync -m 0700 /var/lib/drosync
sudo install -o drosync -g drosync -m 0600 /dev/null /var/lib/drosync/token
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' | sudo tee /etc/drosync/environment >/dev/null
```
`drosync auth` then writes the returned bearer token through a same-directory
0600 temporary file, `fsync` and atomic rename. Missing `DROSYNC_SECRET_FILE`,
unsafe parent permissions, symlinks, special files or non-0600 files fail
closed. The package and environment file never contain the token itself.
## macOS 13+ arm64
Install the notarized `.pkg`. It installs `/Applications/drosync.app`, `/usr/local/bin/drosync`, a LaunchAgent, and a LaunchDaemon definition. Enable only the per-user agent or the administrator-managed daemon:
@ -55,6 +75,19 @@ drosync plan <mapping-id>
drosync sync <mapping-id>
```
普通用户也可以直接填写凭据允许访问的 bucket。守护进程会通过控制面把
bucket 解析为唯一的受限 Sync Space;不会在客户端保存 BOS AK/SK,也不会
绕过 Space 的 root prefix:
```bash
drosync mapping add /data/capture \
--bucket bucket-a \
--remote-prefix ubt/incoming/
```
`--bucket` 与管理员使用的 `--space` 互斥。若同一凭据可见多个指向同一
bucket 的 Space,命令会拒绝猜测,并要求管理员明确提供 `--space`。
For a strict profile, configure and authenticate with the same policy:
```bash

View File

@ -0,0 +1,254 @@
# drosync 用户使用手册
本文面向需要把 Ubuntu 本地目录持续、单向同步到 Drobodata POC 对象存储的用户和管理员。
## 1. 能做什么
`drosync` 监听一个本地目录,在文件稳定后自动上传到指定 bucket 的相对 prefix。它支持断点续传、失败重试、冲突保留、SQLite 状态恢复和 systemd 守护运行。
它不是网络盘:不会下载远端文件、删除远端对象、合并文件内容或跟随软链接。
## 2. 安全模型
客户端不保存完整 BOS AK/SK。管理员先在 POC 创建 Sync Space,再创建受限 drosync Key:
- Access Key ID 用于标识客户端;
- Secret 只在创建时显示一次;
- Scope 限制可用 Space、相对 prefix 和 `plan/create/update` 操作;
- 首次认证换取的设备 Token 保存在 `/var/lib/drosync/token`,权限必须为 `0600`。
已有 Key 的 Secret 无法再次读取。新机器应创建独立 Key,不应多人共用一个 Secret。
## 3. 安装
### 3.1 DEB 安装(推荐)
先校验:
```bash
sha256sum -c drosync_1.0.0-poc.20260901-bucket-map_amd64.deb.sha256
```
安装并启动系统服务:
```bash
sudo apt-get install -y ./drosync_1.0.0-poc.20260901-bucket-map_amd64.deb
sudo systemctl enable --now drosync.service
systemctl status drosync.service --no-pager
```
### 3.2 portable 安装
```bash
sha256sum -c drosync-1.0.0-poc.20260901-bucket-map-linux-x86_64.tar.gz.sha256
tar -xzf drosync-1.0.0-poc.20260901-bucket-map-linux-x86_64.tar.gz
cd drosync-1.0.0-poc.20260901-bucket-map
sudo apt-get update
sudo apt-get install -y ca-certificates libdbus-1-3
sudo ./install-systemd-portable.sh --enable
```
## 4. POC 管理员准备
管理员在“对象存储 / 对象同步”中完成:
1. 创建或选择 Storage Connection;
2. 创建 Sync Space,填写 bucket 和管理员限定的 root prefix;
3. 执行 capability probe;
4. 创建 drosync Key;
5. 将 Scope 限制到该 Space 和需要的相对 prefix;
6. 首次测试建议仅授权 `plan/create`,验证后再决定是否增加 `update`。
## 5. 系统服务模式配置
以下命令显式设置两个 XDG 路径,让 CLI 连接系统 daemon 的 Unix socket。
配置 POC 地址:
```bash
sudo env \
XDG_DATA_HOME=/var/lib/drosync \
XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync configure --server http://120.48.164.19
```
POC 当前为 HTTP,认证元数据不加密;只应在认可的隔离网络中使用。正式环境应使用 HTTPS 并增加 `--require-https`。
将一次性 Secret 写入 root 私有文件:
```bash
sudo install -m 0600 /dev/null /root/drosync-onetime-secret
sudo nano /root/drosync-onetime-secret
```
认证:
```bash
sudo env \
XDG_DATA_HOME=/var/lib/drosync \
XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync auth \
--server http://120.48.164.19 \
--access-key-id '<ACCESS_KEY_ID>' \
--secret-file /root/drosync-onetime-secret
```
成功后立即删除一次性 Secret:
```bash
sudo rm /root/drosync-onetime-secret
sudo stat -c '%a %U:%G %n' /var/lib/drosync/token
```
期望 Token 文件权限为 `600 drosync:drosync`。
## 6. 添加同步目录
用户可以直接填写本地目录、bucket 和相对 prefix:
```bash
sudo env \
XDG_DATA_HOME=/var/lib/drosync \
XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync mapping add /data/capture \
--bucket bucket-a \
--remote-prefix ubt/incoming/
```
系统服务启用了 `PrivateTmp=true`。不要把同步源目录配置在 `/tmp` 或
`/var/tmp`;systemd 私有临时目录会让 daemon 看不到用户 shell 创建的文件。
应使用 `/data`、`/srv`、挂载的数据盘或实际采集目录。
`--bucket` 只会在当前 Key 可见的 Space 中解析。没有匹配或存在多个匹配时命令会拒绝猜测。管理员可改用 `--space <SPACE_ID>` 精确指定。
可选参数:
- `--include-hidden`:包含隐藏文件;默认不包含;
- `--require-done`:只有存在 `{file}.done` 时才上传;
- 临时后缀 `.tmp/.part/.partial/.crdownload/.swp/.download` 始终忽略;
- 软链接、FIFO、socket 和设备文件始终忽略。
## 7. 检查和首次同步
先执行健康检查:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync doctor
```
查看映射并取得 mapping ID:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync mapping list
```
只生成计划,不上传文件:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync plan '<MAPPING_ID>'
```
确认计划后执行同步:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync sync '<MAPPING_ID>'
```
daemon 会持续监听映射目录。文件必须至少 60 秒保持大小和 mtime 不变,哈希前后 stat 也必须一致,才会进入上传。
## 8. 状态、日志和恢复
查看状态:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync --json status
```
查看日志:
```bash
sudo journalctl -u drosync.service -f
```
失败后重试:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync retry '<MAPPING_ID>'
```
SQLite 位于 `/var/lib/drosync/drosync/state.db`。不要在服务运行时复制或手工编辑数据库;备份前先停止服务,或使用 SQLite 在线备份机制。
## 9. 冲突处理
客户端不会静默覆盖并发写入:
- create 使用条件写入;
- update 使用 ETag/version 前置条件;
- 条件失败时保留原对象,并将新内容发布为 `.__conflict__<UTC>_<item>`;
- multipart ETag 只用于传输校验,不当作文件 SHA-256。
查看冲突:
```bash
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
drosync conflicts
```
## 10. 常见问题
### `server is not configured`
先运行 `drosync configure --server ...`,并确保 CLI 使用系统服务的两个 XDG 路径。
### `bucket ... is not available`
当前 Key 没有该 bucket 对应 Space 的 Scope,或管理员尚未创建/启用 Space。
### `bucket ... maps to multiple sync spaces`
让管理员提供准确 Space ID,然后使用 `--space <ID>`。
### 文件没有上传
检查:文件是否稳定满 60 秒、是否为临时后缀、是否为隐藏文件、是否要求 `.done`、Key 是否包含目标 prefix。
### 服务循环重启
```bash
sudo systemctl status drosync.service --no-pager
sudo journalctl -u drosync.service -n 100 --no-pager
sudo stat -c '%a %U:%G %n' /var/lib/drosync /var/lib/drosync/config /etc/drosync
```
## 11. 停止和卸载
停止自动同步:
```bash
sudo systemctl disable --now drosync.service
```
卸载 DEB:
```bash
sudo apt-get remove drosync
```
卸载不会自动删除 `/var/lib/drosync` 和 `/etc/drosync`,避免误删状态和凭据。确认不再需要后再由管理员单独清理。
## 12. 验收清单
- `drosync --json version` 显示 `linux-x86_64`;
- `systemctl is-active drosync.service` 返回 `active`;
- `drosync doctor` 通过 SQLite、IPC、凭据、控制面和 Scope 检查;
- `plan` 的对象路径全部位于授权 prefix;
- 上传后比对本地 SHA-256、远端 size/ETag/version;
- 中断上传后可续传;
- 并发覆盖测试生成冲突对象而不是覆盖原对象;
- 审计中可追溯设备、session、item、操作和目标 key。

17
docs/poc-baseline.md Normal file
View File

@ -0,0 +1,17 @@
# POC baseline
The POC branch is based on `d947a20b` (2026-08-19), immediately before the Fanyu delivery merge. It keeps the original dark interface, role navigation, projects, workflows, operators, storage and drosync pages.
The original drosync API/coordinator and access-key implementation are already in that baseline. The branch additionally preserves the drosync-only ARM64 packaging and signed-update changes from `f93654df`, plus these five changes from the former POC branch:
- `4e5166fe`: bucket-based mappings.
- `eaec4c2c`: Linux package hardening.
- `e19995ce`: headless systemd services.
- `f7ca2755`: Chinese user manual.
- `921fc372`: automatic scheduler reliability.
The resulting `tools/drosync`, `docs/drosync`, and `internal/api/object_sync.go` match the former `921fc372` POC versions, including executable script permissions. Fanyu business pages, global theme changes, role migrations and delivery bundles are excluded.
Deploy this branch to namespace `droboticflow`, public entry `http://120.48.164.19`, using the POC kubeconfig and the `dataflow-poc` image repository. Verify actual Pod image digests and API/Worker build metadata; changing the local branch does not change a running deployment.
Before this rollback, the branch was backed up as `backup/poc-before-fanyu-rollback-20260905`. Existing projects, users, runs and object-sync records are retained; reverting application code does not authorize deleting business data.

View File

@ -0,0 +1,64 @@
# 新组织目录隔离部署验收(2026-10-10)
## 结果
已部署到 POC(命名空间 `droboticflow`,入口 `http://120.48.164.19`)。
UBT 和 POC 主验收组织仍使用 `testxyz` 整桶连接。后续新建组织按组织标识
自动创建 `testxyz/<slug>/`,自动配置独立的受限 IAM 用户、AK/SK 和产物存储。
现有组织不自动加入新规则;组织改名不改变其已分配目录。
创建接口立即返回 202 / pending,后台自动配置和重试,管理页面自动刷新状态。
存储就绪后才允许添加业务账号。本次真实验收:创建请求约 0.2 秒返回,约
100 秒后后台自动转为 ready,未调用手动重试接口。密钥传播等待期间不回退
到共享凭证或整桶授权。
## 已部署版本
维护发布分支 `poc-org-storage`,代码提交 `07d8566b`,构建版本 `1.0.19`。
该分支基于已部署 POC API 镜像标签证实的 `109290ee`,保留线上数据契约。
主分支 `poc` 已同步功能,提交 `3fb11571`。未执行全库 schema 切换。
| 服务 | 就绪副本 | 实际镜像 |
|---|---|---|
| droboticflow-api | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:faa2fe5335298a3206321397d538774b9e100f5b1c00fe3f6a52c1720c2a08f2` |
| droboticflow-worker | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-worker@sha256:c556567d98627bab0f734a5fc0b8cfff406c92c6b726cf23aabf7e0c8040d695` |
| drosync-coordinator | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/drosync-coordinator@sha256:155b46cd5ee7c82e12fa67efc8f8203d67035c68ff3461e000f7d5450c8fec2e` |
API 和 Worker 的公开 health 接口均返回 `buildCommit: 07d8566b`、`status: ok`。
Coordinator 的 `/health` 和 `/ready` 分别返回健康、ready=true,enabled=true。
清除了部署中覆盖镜像版本信息的旧环境变量,运行时版本与镜像摘要已核对。
## 实测证据
| 检查 | 结果 |
|---|---|
| 新组织创建与后台 IAM 配置 | 202 后自动转 ready,账号创建返回 201 |
| 新组织默认对象列表 | 定位到自己的顶层目录 |
| 平台访问其他目录、读取/写入其他目录 | HTTP 403 |
| 组织管理员扩大 rootPrefix 或创建替代的无目录限制连接 | HTTP 403 |
| 授权目录内上传、读取、覆盖、删除 | 通过 |
| 专用凭证直接访问 BOS:自己目录列举 | HTTP 200 |
| 专用凭证直接访问 BOS:桶根、其他目录列举、其他目录读取/写入 | HTTP 403 |
| 专用凭证直接访问 BOS:目录内上传、读取、覆盖、删除 | 通过 |
| 现有 UBT 登录及整桶根目录列举 | 通过 |
| 现有 POC 主验收组织登录及整桶根目录列举 | 通过 |
| 现有 UBT / POC 存储连接 metadata 对比 | 未改变 |
| 部署前记录的 13 个未结束同步 Session | 全部保留,状态未改变 |
| 本地后端回归、路径/符号链接/挂载隔离、异步创建、并发状态保留 | 通过 |
| 前端类型检查、255 项维护分支前端测试、构建 | 通过 |
| 部署模板测试 | 9 项通过 |
| 本地管理页面创建组织交互 | 自动显示 testxyz/company-a/ 和存储已就绪 |
后台为新组织配置专属目录的默认产物存储。其工作流容器仅挂载本次运行目录
和只读的算子代码目录,拒绝共享 PVC/注册表凭据挂载;不能通过符号链接把其他
运行目录的文件作为输出发布。旧组织保留原有行为。
## 清理与交付
两次真实 API 验收创建的临时组织、业务账号和存储连接已删除(回读均为 0)。
其 IAM 密钥先由组织删除流程停用,再清理测试身份和策略;独立 SDK 探针资源
也已清理。仅移除了本次测试的空目录标记和测试对象,目录保留登记仍用于防止
名字复用造成数据继承。旁路 Deployment 已删除。
本地交互预览:`http://127.0.0.1:18090/`(独立测试数据,不是线上账号)。
实现契约见 [新组织存储配置](new-organization-storage.md)。

View File

@ -0,0 +1,60 @@
# New-organization storage provisioning
## Contract
Existing organizations, including UBT and POC Real Acceptance Organization, keep
their current `testxyz` whole-bucket connections. Only organizations created after
this deployment are enrolled. Their normalized organization slug becomes their
immutable top-level root: e.g. `company-a` maps to `testxyz/company-a/`. Renaming
an organization does not change its root.
The POC preset sets `ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID` to its existing
POC connection. That credential remains server-side and is used for IAM management.
New organizations receive a separate IAM user and AK/SK with only LIST/READ/WRITE
on their root, plus READ of bucket metadata. No bucket-wide LIST or FULL_CONTROL
is granted. Policy syntax follows the official [BOS IAM reference](https://intl.cloud.baidu.com/zh/doc/BOS/s/5k1rlj1lk-intl).
Creation reserves the directory permanently, rejects existing data, provisions
cloud identity, saves the one-time secret, and verifies directory listing succeeds
while root listing returns AccessDenied. A conditional empty directory marker
materializes the folder. Artifacts default to that same root. Organization creation returns 202 with a pending state. A background reconciler
retries pending provisioning every 15 seconds, and the console polls its status.
Pending provisioning cannot add business users; the console also exposes a retry button. The
lease prevents concurrent API replicas from issuing competing credentials.
IAM propagation can take time; a failed verification retains the scoped connection
for retry, never substitutes a shared key. An interrupted key issue is recovered
only for an owned identity with no persisted tenant connection. Organization
removal disables its IAM keys and retains directory reservations and BOS objects.
Scope is enforced for object operations and signed URLs, scans, sync staging,
publication, cleanup and artifacts. New scoped-organization jobs only mount their
run workspace and read-only operator catalog code. Shared PVC mounts and registry
credential mounts are rejected. They cannot opt into raw shared filesystem scans.
Files and symlinks outside the run directory cannot be externalized as artifacts.
Old organizations keep the existing mount and storage behavior.
## Maintenance release
The live API image label identifies POC commit `109290ee`. The current `poc` head
requires a separate global schema cutover which would interrupt existing sync
sessions. This targeted maintenance release branches from the verified live POC
ancestor as `poc-org-storage`. It carries the storage feature without the unrelated
schema retirement. It never merges or deploys Fanyu/dev code. Build images only
under `dataflow-poc`; client packages are not installed by this rollout.
Release build version is `1.0.19`. Earlier build candidates were refined for inline
artifacts and automatic recovery from real IAM key-propagation delays.
The main `poc` branch should receive the additive provisioning changes as well, so
a future full cutover keeps the same new-organization policy.
## Acceptance
- Preserve old UBT/POC connection metadata and whole-bucket listing.
- Create a disposable new organization through the real super-admin API.
- Verify its storage root, unique IAM key, default artifact policy and directory.
- Login as its admin; default listing must start at its own root.
- Verify inside upload/read/overwrite/delete; deny root/sibling lists and sibling
reads/writes through both platform and direct BOS SDK calls.
- Deny connection scope/credential widening and alternative unscoped connections.
- Retain active POC sessions, and verify API/Worker/Coordinator rollouts and images.
- Disable the disposable organization's keys and clean only its acceptance objects.

View File

@ -707,10 +707,34 @@ func (server *server) handleAdminOrganizationsCreate(writer http.ResponseWriter,
renderValidationError(writer, validationIssue{Field: "status", Message: "status must be active or disabled"})
return
}
if server.organizationStorageSourceConnectionID != "" {
source, err := server.store.GetStorageConnection(request.Context(), server.organizationStorageSourceConnectionID)
if err != nil || source == nil || source.Bucket == "" || source.RootPrefix != "" {
renderJSON(writer, 503, map[string]string{"error": "default organization storage is unavailable"})
return
}
item.StorageRootPrefix = slug + "/"
item.StorageBucket = source.Bucket
item.StorageConnectionID = "storage-" + item.ID
item.StorageState = "pending"
}
if err := server.store.UpsertOrganization(request.Context(), item); err != nil {
renderOrganizationMutationError(writer, err)
return
}
if item.StorageRootPrefix != "" && server.organizationStorageAsync {
go func(org domain.OrganizationRecord) {
_ = server.provisionOrganizationStorage(context.Background(), &org)
}(item)
renderJSON(writer, http.StatusAccepted, item)
return
}
if item.StorageRootPrefix != "" {
if err := server.provisionOrganizationStorage(request.Context(), &item); err != nil {
renderJSON(writer, http.StatusServiceUnavailable, map[string]any{"error": err.Error(), "organization": item})
return
}
}
renderJSON(writer, http.StatusCreated, item)
}
@ -762,6 +786,39 @@ func (server *server) handleAdminOrganizationsUpdate(writer http.ResponseWriter,
}
func (server *server) handleAdminOrganizationsDelete(writer http.ResponseWriter, request *http.Request) {
org, lookupErr := server.store.GetOrganization(request.Context(), chi.URLParam(request, "id"))
if lookupErr != nil {
renderError(writer, 500, lookupErr)
return
}
if org != nil && org.StorageRootPrefix != "" {
owner := domain.NewID("storage-delete")
acquired, err := server.store.AcquireOrganizationStorageLease(request.Context(), org.ID, owner, time.Now().Add(3*time.Minute))
if err != nil || !acquired {
renderJSON(writer, 409, map[string]string{"error": "organization storage provisioning is in progress; retry deletion later"})
return
}
defer func() {
ctx, cancel := context.WithTimeout(context.WithoutCancel(request.Context()), 5*time.Second)
defer cancel()
_ = server.store.ReleaseOrganizationStorageLease(ctx, org.ID, owner)
}()
source, err := server.store.GetStorageConnection(request.Context(), server.organizationStorageSourceConnectionID)
if err != nil || source == nil {
renderJSON(writer, 503, map[string]string{"error": "cannot revoke organization storage"})
return
}
org.Status = domain.OrganizationStatusDisabled
if err := server.store.UpsertOrganization(request.Context(), *org); err != nil {
renderError(writer, 500, err)
return
}
if err := organizationStorageRevoke(request.Context(), *source, *org); err != nil {
renderError(writer, 503, err)
return
}
}
deleted, err := server.store.DeleteOrganizationCascade(request.Context(), chi.URLParam(request, "id"))
if err != nil {
renderError(writer, http.StatusBadRequest, err)
@ -801,6 +858,10 @@ func (server *server) handleAdminOrganizationUsersCreate(writer http.ResponseWri
renderJSON(writer, http.StatusNotFound, map[string]string{"error": "Organization not found"})
return
}
if organization.StorageRootPrefix != "" && organization.StorageState != "ready" {
renderJSON(writer, http.StatusConflict, map[string]string{"error": "organization storage provisioning must finish before adding users"})
return
}
var payload adminUserMutationRequest
if !decodeJSON(writer, request, &payload) {
return

View File

@ -299,6 +299,11 @@ func (server *server) resolveScanConfig(ctx context.Context, scanConfig *domain.
if scanConfig == nil {
return nil, nil
}
if principal, ok := ctx.Value(authContextKey{}).(*authenticatedPrincipal); ok && principal != nil && principal.Organization.StorageRootPrefix != "" {
if scanConfig.ConnectionID != principal.Organization.StorageConnectionID || chooseDataSource(scanConfig.DataSource) != "BOS" {
return nil, domain.ErrStorageScope
}
}
copy := *scanConfig
connection, err := server.getStorageConnection(ctx, copy.ConnectionID)
if err != nil {
@ -310,6 +315,17 @@ func (server *server) resolveScanConfig(ctx context.Context, scanConfig *domain.
}
return &copy, nil
}
if err := connection.CheckStorageSource(copy.Bucket, copy.Prefix, copy.SourcePath); err != nil {
return nil, err
}
copy.StorageRootPrefix = connection.RootPrefix
if connection.RootPrefix != "" && copy.Bucket == "" {
copy.Bucket = connection.Bucket
}
if connection.RootPrefix != "" {
copy.AccessKeyID = connection.AccessKeyID
copy.SecretAccessKey = connection.SecretAccessKey
}
copy.Endpoint = server.storageEndpointForProvider(connection.Provider, connection.Endpoint)
if strings.TrimSpace(copy.AccessKeyID) == "" {
copy.AccessKeyID = connection.AccessKeyID
@ -2132,6 +2148,10 @@ func buildSourceCollectionSignedPreviewURL(scanConfig *domain.ScanConfig, source
if !ok {
return "", fmt.Errorf("invalid BOS source path: %s", sourcePath)
}
scope := &domain.StorageConnectionRecord{Bucket: scanConfig.Bucket, RootPrefix: scanConfig.StorageRootPrefix}
if err := scope.CheckStoragePath(bucket, object, false); err != nil {
return "", err
}
accessKeyID := strings.TrimSpace(scanConfig.AccessKeyID)
secretAccessKey := strings.TrimSpace(scanConfig.SecretAccessKey)
if accessKeyID == "" || secretAccessKey == "" {

View File

@ -27,6 +27,7 @@ import (
type objectStorageBrowser interface {
ListBuckets(context.Context) ([]domain.ObjectStorageBucketSummary, error)
ListObjects(context.Context, string, string) ([]domain.ObjectStorageObjectSummary, error)
ListObjectsRecursive(context.Context, string, string) ([]domain.ObjectStorageObjectSummary, error)
CreateBucket(context.Context, string) (domain.ObjectStorageBucketSummary, error)
DeleteBucket(context.Context, string) error
PutObject(context.Context, string, string, []byte, string) (domain.ObjectStorageObjectSummary, error)
@ -203,6 +204,10 @@ func objectStorageSummaryFromMeta(bucket string, key string, meta *bosapi.GetObj
}
func (browser *bosObjectStorageBrowser) ListBuckets(ctx context.Context) ([]domain.ObjectStorageBucketSummary, error) {
if browser.connection != nil && browser.connection.RootPrefix != "" {
return []domain.ObjectStorageBucketSummary{{Name: browser.connection.Bucket}}, nil
}
result, err := browser.client.ListBucketsWithContext(ctx)
if err != nil {
return nil, err
@ -219,6 +224,11 @@ func (browser *bosObjectStorageBrowser) ListBuckets(ctx context.Context) ([]doma
}
func (browser *bosObjectStorageBrowser) ListObjects(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
prefix = browser.connection.StorageBrowsePrefix(prefix)
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
return nil, err
}
items := make([]domain.ObjectStorageObjectSummary, 0)
seen := make(map[string]struct{})
marker := ""
@ -243,6 +253,48 @@ func (browser *bosObjectStorageBrowser) ListObjects(ctx context.Context, bucket
return items, nil
}
func (browser *bosObjectStorageBrowser) ListObjectsRecursive(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
prefix = browser.connection.StorageBrowsePrefix(prefix)
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
return nil, err
}
items := make([]domain.ObjectStorageObjectSummary, 0)
seen := make(map[string]struct{})
marker := ""
normalizedBucket := strings.TrimSpace(bucket)
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
for {
result, err := browser.client.ListObjectsWithContext(ctx, normalizedBucket, &bosapi.ListObjectsArgs{
Prefix: storageObjectRequestPrefix(prefix),
MaxKeys: 1000,
Marker: strings.TrimSpace(marker),
})
if err != nil {
return nil, err
}
if result == nil {
break
}
for _, item := range result.Contents {
keyValue := strings.Trim(strings.TrimSpace(item.Key), "/")
if keyValue == "" || keyValue == normalizedPrefix {
continue
}
if _, exists := seen[keyValue]; exists {
continue
}
seen[keyValue] = struct{}{}
items = append(items, objectStorageSummaryFromListObject(normalizedBucket, item))
}
if !result.IsTruncated || strings.TrimSpace(result.NextMarker) == "" || strings.TrimSpace(result.NextMarker) == marker {
break
}
marker = strings.TrimSpace(result.NextMarker)
}
return items, nil
}
func (browser *bosObjectStorageBrowser) ListObjectsPage(
ctx context.Context,
bucket string,
@ -250,6 +302,11 @@ func (browser *bosObjectStorageBrowser) ListObjectsPage(
marker string,
limit int,
) (objectStoragePage, error) {
prefix = browser.connection.StorageBrowsePrefix(prefix)
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
return objectStoragePage{}, err
}
normalizedBucket := strings.TrimSpace(bucket)
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
if limit <= 0 {
@ -304,6 +361,10 @@ func (browser *bosObjectStorageBrowser) ListObjectsPage(
}
func (browser *bosObjectStorageBrowser) CreateBucket(_ context.Context, bucket string) (domain.ObjectStorageBucketSummary, error) {
if browser.connection != nil && browser.connection.RootPrefix != "" {
return domain.ObjectStorageBucketSummary{}, domain.ErrStorageScope
}
if _, err := browser.client.PutBucket(strings.TrimSpace(bucket)); err != nil {
return domain.ObjectStorageBucketSummary{}, err
}
@ -311,6 +372,10 @@ func (browser *bosObjectStorageBrowser) CreateBucket(_ context.Context, bucket s
}
func (browser *bosObjectStorageBrowser) DeleteBucket(_ context.Context, bucket string) error {
if browser.connection != nil && browser.connection.RootPrefix != "" {
return domain.ErrStorageScope
}
return browser.client.DeleteBucket(strings.TrimSpace(bucket))
}
@ -439,6 +504,10 @@ func writeObjectStorageRangeHeaders(writer http.ResponseWriter, summary domain.O
}
func (browser *bosObjectStorageBrowser) PutObject(ctx context.Context, bucket string, key string, payload []byte, contentType string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
args := &bosapi.PutObjectArgs{ContentType: detectObjectStorageContentType(objectKey, payload, contentType)}
if _, err := browser.client.PutObjectFromBytesWithContext(ctx, strings.TrimSpace(bucket), objectKey, payload, args); err != nil {
@ -457,6 +526,10 @@ func (browser *bosObjectStorageBrowser) PutObjectConditional(
ifNoneMatch string,
userMeta map[string]string,
) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
args := &bosapi.PutObjectArgs{
ContentType: detectObjectStorageContentType(objectKey, payload, contentType),
@ -471,6 +544,10 @@ func (browser *bosObjectStorageBrowser) PutObjectConditional(
}
func (browser *bosObjectStorageBrowser) PutObjectStream(ctx context.Context, bucket string, key string, reader io.Reader, contentType string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
args := &bosapi.PutObjectArgs{ContentType: detectObjectStorageContentType(objectKey, nil, contentType)}
if _, err := browser.client.PutObjectFromStreamWithContext(ctx, strings.TrimSpace(bucket), objectKey, reader, args); err != nil {
@ -502,6 +579,10 @@ func objectStorageUploadHeaders(contentType string) map[string]string {
}
func (browser *bosObjectStorageBrowser) PresignPutObject(_ context.Context, bucket string, key string, contentType string, expiresInSeconds int) (domain.ObjectStoragePresignedUploadResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStoragePresignedUploadResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
headers := objectStorageUploadHeaders(contentType)
@ -526,6 +607,10 @@ func (browser *bosObjectStorageBrowser) PresignPutObjectBound(
metadata map[string]string,
expiresInSeconds int,
) (domain.ObjectStoragePresignedUploadResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStoragePresignedUploadResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
headers := objectStorageUploadHeaders(contentType)
@ -546,6 +631,10 @@ func (browser *bosObjectStorageBrowser) PresignPutObjectBound(
}
func (browser *bosObjectStorageBrowser) InitiateMultipartUpload(_ context.Context, bucket string, key string, contentType string, expiresInSeconds int) (domain.ObjectStorageMultipartUploadCreateResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageMultipartUploadCreateResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
result, err := browser.client.InitiateMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(contentType), nil)
if err != nil {
@ -568,6 +657,10 @@ func (browser *bosObjectStorageBrowser) InitiateMultipartUploadBound(
metadata map[string]string,
expiresInSeconds int,
) (domain.ObjectStorageMultipartUploadCreateResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageMultipartUploadCreateResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
result, err := browser.client.InitiateMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(contentType), nil)
if err != nil {
@ -583,6 +676,10 @@ func (browser *bosObjectStorageBrowser) InitiateMultipartUploadBound(
}
func (browser *bosObjectStorageBrowser) PresignMultipartUploadPart(_ context.Context, bucket string, key string, uploadID string, partNumber int, expiresInSeconds int) (domain.ObjectStorageMultipartUploadPartURLResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageMultipartUploadPartURLResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
params := map[string]string{
@ -612,6 +709,10 @@ func (browser *bosObjectStorageBrowser) PresignMultipartUploadPartBound(
sizeBytes int64,
expiresInSeconds int,
) (domain.ObjectStorageMultipartUploadPartURLResult, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageMultipartUploadPartURLResult{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
params := map[string]string{
@ -636,6 +737,10 @@ func (browser *bosObjectStorageBrowser) PresignMultipartUploadPartBound(
}
func (browser *bosObjectStorageBrowser) CompleteMultipartUpload(ctx context.Context, bucket string, key string, uploadID string, parts []domain.ObjectStorageMultipartUploadPart) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
return browser.CompleteMultipartUploadBound(ctx, bucket, key, uploadID, parts, nil)
}
@ -647,6 +752,10 @@ func (browser *bosObjectStorageBrowser) CompleteMultipartUploadBound(
parts []domain.ObjectStorageMultipartUploadPart,
metadata map[string]string,
) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
uploadParts := make([]bosapi.UploadInfoType, 0, len(parts))
for _, part := range parts {
@ -659,6 +768,10 @@ func (browser *bosObjectStorageBrowser) CompleteMultipartUploadBound(
}
func (browser *bosObjectStorageBrowser) AbortMultipartUpload(_ context.Context, bucket string, key string, uploadID string) error {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return err
}
objectKey := strings.Trim(strings.TrimSpace(key), "/")
return browser.client.AbortMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(uploadID))
}
@ -691,6 +804,13 @@ func (browser *bosObjectStorageBrowser) PublishMultipartCopyConditional(
ifNoneMatch string,
userMeta map[string]string,
) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, sourceKey, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
if err := browser.connection.CheckStoragePath(bucket, targetKey, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
normalizedBucket := strings.TrimSpace(bucket)
sourceKey = exactObjectStorageKey(sourceKey)
targetKey = exactObjectStorageKey(targetKey)
@ -769,6 +889,10 @@ func (browser *bosObjectStorageBrowser) PublishMultipartCopyConditional(
}
func (browser *bosObjectStorageBrowser) GetBucketVersioning(_ context.Context, bucket string) (string, error) {
if browser.connection != nil && browser.connection.RootPrefix != "" && bucket != browser.connection.Bucket {
return "", domain.ErrStorageScope
}
result, err := browser.client.GetBucketVersioning(strings.TrimSpace(bucket))
if err != nil {
return "", err
@ -780,6 +904,13 @@ func (browser *bosObjectStorageBrowser) GetBucketVersioning(_ context.Context, b
}
func (browser *bosObjectStorageBrowser) RenameObject(ctx context.Context, bucket string, key string, newKey string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
if err := browser.connection.CheckStoragePath(bucket, newKey, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := exactObjectStorageKey(key)
destinationKey := exactObjectStorageKey(newKey)
if objectKey == "" || destinationKey == "" {
@ -798,6 +929,13 @@ func (browser *bosObjectStorageBrowser) RenameObject(ctx context.Context, bucket
}
func (browser *bosObjectStorageBrowser) MovePrefix(ctx context.Context, bucket string, prefix string, newPrefix string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
if err := browser.connection.CheckStoragePath(bucket, newPrefix, true); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
normalizedBucket := strings.TrimSpace(bucket)
sourcePrefix := strings.Trim(strings.TrimSpace(prefix), "/")
destinationPrefix := strings.Trim(strings.TrimSpace(newPrefix), "/")
@ -900,10 +1038,18 @@ func objectStorageMoveDestination(sourceKey string, destinationPrefix string) st
}
func (browser *bosObjectStorageBrowser) DeleteObject(_ context.Context, bucket string, key string) error {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return err
}
return browser.client.DeleteObject(strings.TrimSpace(bucket), exactObjectStorageKey(key))
}
func (browser *bosObjectStorageBrowser) DeletePrefix(ctx context.Context, bucket string, prefix string) error {
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
return err
}
normalizedBucket := strings.TrimSpace(bucket)
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
if normalizedBucket == "" || normalizedPrefix == "" {
@ -947,6 +1093,10 @@ func (browser *bosObjectStorageBrowser) DeletePrefix(ctx context.Context, bucket
}
func (browser *bosObjectStorageBrowser) GetObjectSummary(ctx context.Context, bucket string, key string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
objectKey := exactObjectStorageKey(key)
meta, err := browser.client.GetObjectMetaWithContext(ctx, strings.TrimSpace(bucket), objectKey)
if err != nil {
@ -956,6 +1106,10 @@ func (browser *bosObjectStorageBrowser) GetObjectSummary(ctx context.Context, bu
}
func (browser *bosObjectStorageBrowser) getObjectSummaryAfterWrite(ctx context.Context, bucket string, key string) (domain.ObjectStorageObjectSummary, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return domain.ObjectStorageObjectSummary{}, err
}
return retryObjectStorageSummary(ctx, objectStorageWriteMetadataRetryAttempts, objectStorageWriteMetadataRetryDelay, func() (domain.ObjectStorageObjectSummary, error) {
return browser.GetObjectSummary(ctx, bucket, key)
})
@ -989,6 +1143,10 @@ func retryObjectStorageSummary(
}
func (browser *bosObjectStorageBrowser) ReadObject(ctx context.Context, bucket string, key string) ([]byte, string, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return nil, "", err
}
objectKey := exactObjectStorageKey(key)
result, err := browser.client.GetObjectWithContext(ctx, strings.TrimSpace(bucket), objectKey, nil)
if err != nil {
@ -1003,6 +1161,10 @@ func (browser *bosObjectStorageBrowser) ReadObject(ctx context.Context, bucket s
}
func (browser *bosObjectStorageBrowser) ReadObjectRange(ctx context.Context, bucket string, key string, start int64, end int64) (io.ReadCloser, string, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return nil, "", err
}
objectKey := exactObjectStorageKey(key)
result, err := browser.client.GetObjectWithContext(ctx, strings.TrimSpace(bucket), objectKey, nil, start, end)
if err != nil {
@ -1012,6 +1174,10 @@ func (browser *bosObjectStorageBrowser) ReadObjectRange(ctx context.Context, buc
}
func (browser *bosObjectStorageBrowser) ReadObjectPreview(ctx context.Context, bucket string, key string, maxBytes int64) ([]byte, string, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return nil, "", err
}
if maxBytes <= 0 {
return browser.ReadObject(ctx, bucket, key)
}
@ -1036,6 +1202,10 @@ func (browser *bosObjectStorageBrowser) ReadObjectPreview(ctx context.Context, b
}
func (browser *bosObjectStorageBrowser) BuildPreviewURL(_ context.Context, bucket string, key string) (string, error) {
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
return "", err
}
previewURL := strings.TrimSpace(browser.client.BasicGeneratePresignedUrl(strings.TrimSpace(bucket), strings.Trim(strings.TrimSpace(key), "/"), 3600))
if previewURL == "" {
return "", fmt.Errorf("failed to generate BOS preview url")
@ -1232,7 +1402,7 @@ func (server *server) handleObjectStorageObjects(writer http.ResponseWriter, req
renderValidationError(writer, validationIssue{Field: "bucket", Message: err.Error()})
return
}
prefix := strings.Trim(strings.TrimSpace(query.Get("prefix")), "/")
prefix := connection.StorageBrowsePrefix(strings.Trim(strings.TrimSpace(query.Get("prefix")), "/"))
marker := strings.TrimSpace(query.Get("marker"))
paginationRequested := query.Has("limit") || query.Has("marker")
limit := 100
@ -2055,3 +2225,93 @@ func (server *server) handleObjectStorageContent(writer http.ResponseWriter, req
}
writeSourceContent(writer, request, path.Base(key), contentType, payload)
}
type objectStorageDirectoryManifestItem struct {
Key string `json:"key"`
RelativePath string `json:"relativePath"`
SizeBytes int64 `json:"sizeBytes"`
DownloadURL string `json:"downloadUrl"`
}
type objectStorageDirectoryManifestResponse struct {
Bucket string `json:"bucket"`
BasePrefix string `json:"basePrefix"`
TotalFiles int `json:"totalFiles"`
TotalBytes int64 `json:"totalBytes"`
Files []objectStorageDirectoryManifestItem `json:"files"`
}
func (server *server) handleObjectStorageDirectoryManifest(writer http.ResponseWriter, request *http.Request) {
connectionID := strings.TrimSpace(request.URL.Query().Get("connectionId"))
browser, connection, cleanup, err := server.resolveObjectStorageBrowser(request.Context(), connectionID)
if err != nil {
renderError(writer, http.StatusInternalServerError, err)
return
}
if cleanup != nil {
defer cleanup()
}
if browser == nil || connection == nil {
renderValidationError(writer, validationIssue{Field: "connectionId", Message: "connectionId is required"})
return
}
bucket, err := resolveObjectStorageBucket(connection, request.URL.Query().Get("bucket"))
if err != nil {
renderValidationError(writer, validationIssue{Field: "bucket", Message: err.Error()})
return
}
prefix := strings.Trim(strings.TrimSpace(request.URL.Query().Get("prefix")), "/")
if bucket == "" {
renderValidationError(writer, validationIssue{Field: "bucket", Message: "bucket is required"})
return
}
objects, err := browser.ListObjectsRecursive(request.Context(), bucket, prefix)
if err != nil {
renderObjectStorageWriteError(writer, err)
return
}
items := make([]objectStorageDirectoryManifestItem, 0, len(objects))
var totalBytes int64
basePrefixWithSlash := prefix
if basePrefixWithSlash != "" && !strings.HasSuffix(basePrefixWithSlash, "/") {
basePrefixWithSlash += "/"
}
for _, obj := range objects {
if obj.Kind == "directory" {
continue
}
relPath := obj.Key
if basePrefixWithSlash != "" && strings.HasPrefix(obj.Key, basePrefixWithSlash) {
relPath = strings.TrimPrefix(obj.Key, basePrefixWithSlash)
} else if prefix != "" && strings.HasPrefix(obj.Key, prefix) {
relPath = strings.TrimPrefix(obj.Key, prefix)
relPath = strings.TrimPrefix(relPath, "/")
}
if relPath == "" {
relPath = path.Base(obj.Key)
}
downloadURL, urlErr := browser.BuildPreviewURL(request.Context(), bucket, obj.Key)
if urlErr != nil {
renderError(writer, http.StatusInternalServerError, fmt.Errorf("failed to presign %s: %w", obj.Key, urlErr))
return
}
items = append(items, objectStorageDirectoryManifestItem{
Key: obj.Key,
RelativePath: relPath,
SizeBytes: obj.SizeBytes,
DownloadURL: downloadURL,
})
totalBytes += obj.SizeBytes
}
renderJSON(writer, http.StatusOK, objectStorageDirectoryManifestResponse{
Bucket: bucket,
BasePrefix: prefix,
TotalFiles: len(items),
TotalBytes: totalBytes,
Files: items,
})
}

View File

@ -44,7 +44,10 @@ const (
objectSyncReleasePreview = "preview"
)
var objectSyncClientPlatforms = []string{"macos-arm64", "windows-x86_64", "linux-x86_64"}
// Keep this list in lockstep with drosync_platform::release_platform_key_for.
// S100 devices register as linux/arm64, so their preview artifact must be
// required and exposed alongside the existing desktop targets.
var objectSyncClientPlatforms = []string{"macos-arm64", "windows-x86_64", "linux-x86_64", "linux-arm64"}
func normalizedOrganizationID(value string) string {
if trimmed := strings.TrimSpace(value); trimmed != "" {
@ -527,6 +530,9 @@ func (server *server) objectSyncBrowser(ctx context.Context, space *domain.Objec
if connection == nil || normalizedOrganizationID(connection.OrganizationID) != normalizedOrganizationID(space.OrganizationID) {
return nil, nil, fmt.Errorf("storage connection not found in the sync space organization")
}
if err := connection.CheckStoragePath(space.Bucket, space.RootPrefix, true); err != nil {
return nil, nil, err
}
browser, closeBrowser, err := storageConnectionBrowserResolver(ctx, server.storageConnectionWithResolvedEndpoint(connection))
if err != nil {
return nil, nil, err
@ -643,6 +649,10 @@ func (server *server) handleObjectSyncSpaceCreate(writer http.ResponseWriter, re
renderObjectSyncError(writer, request, http.StatusBadRequest, "invalid_storage_connection", "The storage connection is not available in this organization", false, nil)
return
}
if err := connection.CheckStoragePath(input.Bucket, input.RootPrefix, true); err != nil {
renderError(writer, http.StatusForbidden, err)
return
}
policy := domain.DefaultObjectSyncSpacePolicy()
if input.Policy != nil {
policy = *input.Policy
@ -1314,10 +1324,7 @@ func (server *server) handleObjectSyncSessionAdminAbort(writer http.ResponseWrit
renderObjectSyncError(writer, request, http.StatusNotFound, "session_not_found", "Object sync session not found", false, nil)
return
}
item.Status = domain.ObjectSyncSessionAborted
item.UpdatedAt = nowISO()
item.CompletedAt = item.UpdatedAt
if err := target.UpsertObjectSyncSession(request.Context(), *item); err != nil {
if err := abortObjectSyncSession(request.Context(), target, item); err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
return
}
@ -2353,9 +2360,56 @@ func (server *server) externalObjectSyncItem(writer http.ResponseWriter, request
renderObjectSyncError(writer, request, http.StatusNotFound, "item_not_found", "Object sync item not found for this device credential", false, nil)
return nil, nil, principal, false
}
if session.Status == domain.ObjectSyncSessionAborted || session.Status == domain.ObjectSyncSessionExpired || session.Status == domain.ObjectSyncSessionFailed {
renderObjectSyncError(writer, request, http.StatusConflict, "session_closed", "Session is no longer active", false, nil)
return nil, nil, principal, false
}
return item, session, principal, true
}
// Persist the terminal session before releasing leases so new client operations
// cannot extend an aborted session. Retrying abort also retries lease cleanup.
func abortObjectSyncSession(ctx context.Context, target store.ObjectSyncStore, session *domain.ObjectSyncSessionRecord) error {
if session.Status == domain.ObjectSyncSessionCompleted || session.Status == domain.ObjectSyncSessionPartial || session.Status == domain.ObjectSyncSessionConflicted {
return nil
}
session.Status = domain.ObjectSyncSessionAborted
session.UpdatedAt = nowISO()
session.CompletedAt = session.UpdatedAt
if err := target.UpsertObjectSyncSession(ctx, *session); err != nil {
return err
}
cursor := ""
for {
page, err := target.ListObjectSyncItems(ctx, domain.ObjectSyncListQuery{SessionID: session.ID, Cursor: cursor, Limit: 1000})
if err != nil {
return err
}
updates := make([]domain.ObjectSyncItemRecord, 0, len(page.Items))
for _, item := range page.Items {
if item.LeaseID != "" {
if err := target.ReleaseObjectSyncLease(ctx, item.LeaseID, session.ID, item.ID, item.FencingToken); err != nil {
return err
}
}
switch item.Status {
case domain.ObjectSyncItemCommitted, domain.ObjectSyncItemKeptBoth, domain.ObjectSyncItemUnchanged, domain.ObjectSyncItemSkipped, domain.ObjectSyncItemDenied, domain.ObjectSyncItemFailed, domain.ObjectSyncItemAborted, domain.ObjectSyncItemExpired:
continue
}
item.Status = domain.ObjectSyncItemAborted
item.UpdatedAt = nowISO()
updates = append(updates, item)
}
if err := target.UpsertObjectSyncItems(ctx, updates); err != nil {
return err
}
if page.NextCursor == "" {
return nil
}
cursor = page.NextCursor
}
}
func objectSyncLeaseResourceHash(space domain.ObjectSyncSpaceRecord, key string) string {
sum := sha256.Sum256([]byte(space.ConnectionID + "\x00" + space.Bucket + "\x00" + key))
return hex.EncodeToString(sum[:])
@ -2383,6 +2437,15 @@ func (server *server) handleExternalObjectSyncItemReserve(writer http.ResponseWr
renderObjectSyncError(writer, request, http.StatusConflict, "space_not_found", "Session sync space is unavailable", false, nil)
return
}
connection, err := server.store.GetStorageConnection(request.Context(), space.ConnectionID)
if err != nil || connection == nil {
renderObjectSyncError(writer, request, http.StatusConflict, "storage_unavailable", "Storage connection is unavailable", false, nil)
return
}
if err := connection.CheckStoragePath(space.Bucket, item.TargetKey, false); err != nil {
renderError(writer, http.StatusForbidden, err)
return
}
leaseTemplate := domain.ObjectSyncLeaseRecord{
ID: "osl_" + objectSyncLeaseResourceHash(*space, item.TargetKey),
OrganizationID: session.OrganizationID,
@ -2412,7 +2475,7 @@ func (server *server) handleExternalObjectSyncItemReserve(writer http.ResponseWr
}
item.LeaseID = lease.ID
item.FencingToken = lease.FencingToken
item.StagingKey = path.Join(".drosync-internal/staging", session.OrganizationID, session.ID, item.ID)
item.StagingKey = connection.InternalStorageKey(path.Join(".drosync-internal/staging", session.OrganizationID, session.ID, item.ID))
item.Status = domain.ObjectSyncItemReserved
item.UpdatedAt = nowISO()
session.Status = domain.ObjectSyncSessionRunning
@ -2768,6 +2831,12 @@ func (server *server) handleExternalObjectSyncHeartbeat(writer http.ResponseWrit
if !ok {
return
}
switch item.Status {
case domain.ObjectSyncItemReserved, domain.ObjectSyncItemUploading, domain.ObjectSyncItemUploaded, domain.ObjectSyncItemCommitting:
default:
renderObjectSyncError(writer, request, http.StatusConflict, "item_not_active", "Only active leased items may send heartbeats", false, nil)
return
}
if item.LeaseID == "" {
renderObjectSyncError(writer, request, http.StatusConflict, "lease_missing", "Item has no active lease", false, nil)
return
@ -2881,35 +2950,17 @@ func (server *server) handleExternalObjectSyncSessionFinish(writer http.Response
}
action := chi.URLParam(request, "action")
if action == "abort" {
session.Status = domain.ObjectSyncSessionAborted
cursor := ""
for {
page, err := target.ListObjectSyncItems(request.Context(), domain.ObjectSyncListQuery{SessionID: session.ID, Cursor: cursor, Limit: 1000})
if err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
return
}
updates := make([]domain.ObjectSyncItemRecord, 0, len(page.Items))
for _, item := range page.Items {
switch item.Status {
case domain.ObjectSyncItemCommitted, domain.ObjectSyncItemKeptBoth, domain.ObjectSyncItemUnchanged, domain.ObjectSyncItemSkipped, domain.ObjectSyncItemDenied, domain.ObjectSyncItemFailed, domain.ObjectSyncItemAborted, domain.ObjectSyncItemExpired:
continue
default:
item.Status = domain.ObjectSyncItemAborted
item.UpdatedAt = nowISO()
updates = append(updates, item)
}
}
if err := target.UpsertObjectSyncItems(request.Context(), updates); err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
return
}
if page.NextCursor == "" {
break
}
cursor = page.NextCursor
if err := abortObjectSyncSession(request.Context(), target, session); err != nil {
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
return
}
renderJSON(writer, http.StatusOK, session)
return
} else if action == "complete" {
if session.Status == domain.ObjectSyncSessionAborted || session.Status == domain.ObjectSyncSessionExpired || session.Status == domain.ObjectSyncSessionFailed {
renderObjectSyncError(writer, request, http.StatusConflict, "session_closed", "Session is no longer active", false, nil)
return
}
counts := domain.ObjectSyncSessionCounts{Manifest: session.Counts.Manifest}
cursor := ""
for {

View File

@ -0,0 +1,99 @@
package api
import (
"bytes"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"net/http"
"net/http/httptest"
"testing"
"time"
)
func TestObjectSyncAbortReleasesLeaseAndRejectsClosedOperations(t *testing.T) {
for _, admin := range []bool{false, true} {
name := "device"
if admin {
name = "admin"
}
t.Run(name, func(t *testing.T) {
handler, data := objectSyncTestHandler(t, true)
target, _ := store.AsObjectSyncStore(data)
ctx := t.Context()
org := domain.DefaultOrganizationID
token := "abort-test-token"
space := domain.ObjectSyncSpaceRecord{ID: "abort-space", OrganizationID: org, ConnectionID: "bos-connection", Bucket: "test", Enabled: true, Mode: "create_only", Policy: domain.DefaultObjectSyncSpacePolicy(), Capability: domain.ObjectSyncCapabilityRecord{State: domain.ObjectSyncCapabilityCreateOnly, ConditionalCreate: true}}
cred := domain.ObjectSyncCredentialRecord{ID: "abort-credential", OrganizationID: org, DeviceID: "abort-device", Enabled: true, TokenHash: sessionTokenHash(token), Scopes: []domain.ObjectSyncCredentialScope{{SyncSpaceID: space.ID, AllowedRelativePrefixes: []string{""}, Actions: []string{"plan", "create"}}}}
must := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
must(target.UpsertObjectSyncSpace(ctx, space))
must(target.UpsertObjectSyncCredential(ctx, cred))
must(target.UpsertObjectSyncDevice(ctx, domain.ObjectSyncDeviceRecord{ID: cred.DeviceID, OrganizationID: org, Status: "active"}))
session := domain.ObjectSyncSessionRecord{ID: "abort-session", OrganizationID: org, SyncSpaceID: space.ID, CredentialID: cred.ID, DeviceID: cred.DeviceID, Status: domain.ObjectSyncSessionRunning}
must(target.UpsertObjectSyncSession(ctx, session))
lease, ok, err := target.AcquireObjectSyncLease(ctx, domain.ObjectSyncLeaseRecord{ID: "abort-lease", OrganizationID: org, ResourceHash: "abort-resource", OwnerSessionID: session.ID, OwnerItemID: "abort-item"}, time.Minute)
must(err)
if !ok {
t.Fatal("lease not acquired")
}
item := domain.ObjectSyncItemRecord{ID: "abort-item", OrganizationID: org, SessionID: session.ID, Status: domain.ObjectSyncItemUploading, LeaseID: lease.ID, FencingToken: lease.FencingToken}
committed := domain.ObjectSyncItemRecord{ID: "committed-item", OrganizationID: org, SessionID: session.ID, Status: domain.ObjectSyncItemCommitted}
must(target.UpsertObjectSyncItems(ctx, []domain.ObjectSyncItemRecord{item, committed}))
request := func(path, key string, management bool) *httptest.ResponseRecorder {
r := httptest.NewRequest(http.MethodPost, path, bytes.NewBufferString("{}"))
r.Header.Set("Content-Type", "application/json")
r.Header.Set("Idempotency-Key", key)
if !management {
r.Header.Set(objectSyncCredentialHeader, token)
}
w := httptest.NewRecorder()
handler.ServeHTTP(w, r)
return w
}
abortPath := "/api/external/object-sync/sessions/" + session.ID + "/abort"
if admin {
abortPath = "/api/object-sync/sessions/" + session.ID + "/admin-abort"
}
w := request(abortPath, "abort-first", admin)
if w.Code != 200 {
t.Fatalf("abort %d %s", w.Code, w.Body.String())
}
got, _ := target.GetObjectSyncItem(ctx, item.ID)
if got.Status != domain.ObjectSyncItemAborted {
t.Fatalf("item status %s", got.Status)
}
saved, _ := target.GetObjectSyncItem(ctx, committed.ID)
if saved.Status != domain.ObjectSyncItemCommitted {
t.Fatal("committed item changed")
}
next, acquired, err := target.AcquireObjectSyncLease(ctx, domain.ObjectSyncLeaseRecord{ID: lease.ID, OrganizationID: org, ResourceHash: lease.ResourceHash, OwnerSessionID: "next-session", OwnerItemID: "next-item"}, time.Minute)
must(err)
if !acquired || next.FencingToken <= lease.FencingToken {
t.Fatal("lease not released / fencing not increased")
}
for _, action := range []string{"heartbeat", "reserve", "commit"} {
w = request("/api/external/object-sync/items/"+item.ID+"/"+action, "closed-"+action, false)
if w.Code != 409 {
t.Fatalf("%s status %d %s", action, w.Code, w.Body.String())
}
}
w = request(abortPath, "abort-repeat", admin)
if w.Code != 200 {
t.Fatalf("repeat abort %d", w.Code)
}
renewed, err := target.RenewObjectSyncLease(ctx, next.ID, "next-session", "next-item", next.FencingToken, time.Minute)
must(err)
if !renewed {
t.Fatal("repeat abort released new owner's lease")
}
w = request("/api/external/object-sync/sessions/"+session.ID+"/complete", "closed-complete", false)
if w.Code != 409 {
t.Fatalf("closed complete %d", w.Code)
}
})
}
}

View File

@ -0,0 +1,357 @@
package api
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"time"
"droboticflow/internal/domain"
"github.com/baidubce/bce-sdk-go/auth"
"github.com/baidubce/bce-sdk-go/bce"
"github.com/baidubce/bce-sdk-go/services/bos"
bosapi "github.com/baidubce/bce-sdk-go/services/bos/api"
"github.com/baidubce/bce-sdk-go/services/iam"
iamapi "github.com/baidubce/bce-sdk-go/services/iam/api"
"github.com/go-chi/chi/v5"
)
type organizationStorageTransport struct{ ctx context.Context }
func (t organizationStorageTransport) RoundTrip(r *http.Request) (*http.Response, error) {
return http.DefaultTransport.RoundTrip(r.WithContext(t.ctx))
}
func organizationStoragePolicy(bucket, root string) string {
raw, _ := json.Marshal(map[string]any{"accessControlList": []any{
map[string]any{"service": "bce:bos", "region": "*", "effect": "Allow", "permission": []string{"LIST", "READ", "WRITE"}, "resource": []string{bucket + "/" + root, bucket + "/" + root + "*"}},
map[string]any{"service": "bce:bos", "region": "*", "effect": "Allow", "permission": []string{"READ"}, "resource": []string{bucket}},
}})
return string(raw)
}
func cloudStatus(err error, code int) bool {
var e *bce.BceServiceError
return errors.As(err, &e) && e.StatusCode == code
}
func samePolicy(a, b string) bool {
var x, y any
if json.Unmarshal([]byte(a), &x) != nil || json.Unmarshal([]byte(b), &y) != nil {
return false
}
if m, ok := x.(map[string]any); ok {
delete(m, "id")
}
if m, ok := y.(map[string]any); ok {
delete(m, "id")
}
l, _ := json.Marshal(x)
r, _ := json.Marshal(y)
return string(l) == string(r)
}
// Replaced only in tests. No master credential is copied to a tenant connection.
var organizationStorageProvision = provisionBOSOrganizationStorage
var organizationStorageVerify = verifyOrganizationBOSScope
var organizationDirectoryEmpty = bosOrganizationDirectoryEmpty
func bosOrganizationDirectoryEmpty(ctx context.Context, c domain.StorageConnectionRecord, root string) (bool, error) {
client, err := bos.NewClient(c.AccessKeyID, c.SecretAccessKey, c.Endpoint)
if err != nil {
return false, err
}
client.Config.Retry = bce.NewNoRetryPolicy()
page, err := client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{Prefix: root, MaxKeys: 1})
if err != nil {
return false, err
}
return len(page.Contents) == 0 && len(page.CommonPrefixes) == 0, nil
}
func provisionBOSOrganizationStorage(ctx context.Context, source domain.StorageConnectionRecord, org domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
out := domain.StorageConnectionRecord{ID: org.StorageConnectionID, OrganizationID: org.ID, Name: "BOS Workspace", Provider: domain.StorageProviderBOS, Endpoint: source.Endpoint, Bucket: org.StorageBucket, RootPrefix: org.StorageRootPrefix, CreatedAt: nowISO(), UpdatedAt: nowISO()}
root, err := domain.NormalizeStorageRoot(org.StorageRootPrefix)
if err != nil || root == "" || source.Bucket != org.StorageBucket || source.RootPrefix != "" {
return out, fmt.Errorf("invalid organization storage scope")
}
client, err := organizationIAMClient(ctx, source)
if err != nil {
return out, err
}
name := "poc-org-" + strings.TrimPrefix(org.ID, "org-")
description := "Droboticflow organization " + org.ID
user, err := client.GetUser(name)
if cloudStatus(err, 404) {
_, err = client.CreateUser(&iamapi.CreateUserArgs{Name: name, Description: description})
} else if err == nil && user.Description != description {
return out, fmt.Errorf("IAM identity ownership mismatch")
}
if err != nil {
return out, iamOperationError("user provisioning", err)
}
policyName := strings.ReplaceAll(name, "-", "_")
policy := organizationStoragePolicy(org.StorageBucket, root)
existing, err := client.GetPolicy(policyName, "")
if cloudStatus(err, 404) {
_, err = client.CreatePolicy(&iamapi.CreatePolicyArgs{Name: policyName, Description: description, Document: policy})
} else if err == nil && !samePolicy(existing.Document, policy) {
return out, fmt.Errorf("IAM policy differs from the organization directory")
}
if err != nil {
return out, iamOperationError("policy provisioning", err)
}
if err = client.AttachPolicyToUser(&iamapi.AttachPolicyToUserArgs{UserName: name, PolicyName: policyName, PolicyType: ""}); err != nil {
return out, iamOperationError("policy attachment", err)
}
policies, err := client.ListUserAttachedPolicies(name)
if err != nil {
return out, fmt.Errorf("cannot verify IAM policies")
}
for _, p := range policies.Policies {
if p.Name != policyName {
return out, fmt.Errorf("IAM user has unexpected policies")
}
}
groups, err := client.ListGroupsForUser(name)
if err != nil || len(groups.Groups) != 0 {
return out, fmt.Errorf("IAM user group isolation could not be verified")
}
keys, err := client.ListAccessKey(name)
if err != nil {
return out, fmt.Errorf("IAM key lookup failed")
}
for _, key := range keys.AccessKeys {
if key.Enabled {
if key.Secret == "" {
// No persisted tenant connection exists when this provisioner runs.
// Recover an interrupted key-creation response by revoking only
// this owned, unissued key, then creating its replacement.
if err := client.DeleteAccessKey(name, key.Id); err != nil {
return out, fmt.Errorf("cannot recover unissued IAM key")
}
continue
}
out.AccessKeyID = key.Id
out.SecretAccessKey = key.Secret
break
}
}
if out.AccessKeyID == "" {
key, e := client.CreateAccessKey(name)
if e != nil {
return out, iamOperationError("key creation", e)
}
out.AccessKeyID = key.Id
out.SecretAccessKey = key.Secret
}
return out, nil
}
func verifyOrganizationBOSScope(ctx context.Context, c domain.StorageConnectionRecord) error {
client, err := bos.NewClient(c.AccessKeyID, c.SecretAccessKey, c.Endpoint)
if err != nil {
return fmt.Errorf("BOS client initialization failed")
}
client.Config.Retry = bce.NewNoRetryPolicy()
// IAM propagation is checked before enabling a tenant. Retry is bounded by the
// caller's deadline; failures retain a pending organization for explicit retry.
for attempt := 0; attempt < 45; attempt++ {
_, err = client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{Prefix: c.RootPrefix, MaxKeys: 1})
if err == nil {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(2 * time.Second):
}
}
if err != nil {
return iamOperationError("BOS authorized directory", err)
}
if _, err = client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{MaxKeys: 1}); !bosScopeDenied(err) {
return fmt.Errorf("BOS root listing was not denied")
}
_, err = client.PutObjectFromStringWithContext(ctx, c.Bucket, c.RootPrefix, "", &bosapi.PutObjectArgs{ContentType: "application/x-directory", IfNoneMatch: "*"})
if err != nil && !cloudStatus(err, 412) {
return fmt.Errorf("cannot create organization directory")
}
return nil
}
func bosScopeDenied(err error) bool {
var e *bce.BceServiceError
return errors.As(err, &e) && e.StatusCode == 403 && e.Code == "AccessDenied"
}
func (s *server) provisionOrganizationStorage(ctx context.Context, org *domain.OrganizationRecord) error {
if org.StorageRootPrefix == "" {
return fmt.Errorf("existing organizations are not enrolled automatically")
}
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
owner := domain.NewID("storage-claim")
ok, err := s.store.AcquireOrganizationStorageLease(ctx, org.ID, owner, time.Now().Add(3*time.Minute))
if err != nil || !ok {
return fmt.Errorf("organization storage provisioning is already in progress")
}
defer func() {
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
defer cancel()
_ = s.store.ReleaseOrganizationStorageLease(cleanup, org.ID, owner)
}()
current, err := s.store.GetOrganization(ctx, org.ID)
if err != nil || current == nil {
return fmt.Errorf("organization is unavailable")
}
*org = *current
if org.StorageState == "ready" {
return nil
}
source, err := s.store.GetStorageConnection(ctx, s.organizationStorageSourceConnectionID)
if err != nil || source == nil || source.Bucket != org.StorageBucket || source.RootPrefix != "" {
return fmt.Errorf("organization storage source is unavailable")
}
if err = s.store.ReserveOrganizationStorageRoot(ctx, org.StorageBucket, org.StorageRootPrefix, org.ID); err != nil {
return err
}
existing, err := s.store.GetStorageConnection(ctx, org.StorageConnectionID)
if err != nil {
return err
}
if existing == nil {
// A claimed name must never inherit pre-existing data from the shared bucket.
empty, e := organizationDirectoryEmpty(ctx, *s.storageConnectionWithResolvedEndpoint(source), org.StorageRootPrefix)
if e != nil {
return fmt.Errorf("cannot check organization directory")
}
if !empty {
return fmt.Errorf("directory already contains data; choose another organization slug")
}
connection, e := organizationStorageProvision(ctx, *s.storageConnectionWithResolvedEndpoint(source), *org)
if e != nil {
return e
}
if e = s.store.UpsertStorageConnection(ctx, connection); e != nil {
return fmt.Errorf("cannot persist organization storage; retry provisioning")
}
} else if existing.RootPrefix != org.StorageRootPrefix || existing.Bucket != org.StorageBucket || existing.OrganizationID != org.ID {
return fmt.Errorf("organization connection scope mismatch")
}
saved, err := s.store.GetStorageConnection(ctx, org.StorageConnectionID)
if err != nil || saved == nil {
return fmt.Errorf("organization connection is unavailable")
}
if err := organizationStorageVerify(ctx, *saved); err != nil {
return err
}
if err := s.store.UpsertArtifactStorageSettings(ctx, org.ID, domain.ArtifactStorageSettings{ConnectionID: org.StorageConnectionID, Bucket: org.StorageBucket, PrefixTemplate: org.StorageRootPrefix + "artifacts/{projectId}/{runId}/{nodeId}"}); err != nil {
return fmt.Errorf("cannot configure organization artifact storage")
}
if err := s.store.MarkOrganizationStorageReady(ctx, org.ID); err != nil {
return err
}
updated, err := s.store.GetOrganization(ctx, org.ID)
if err != nil || updated == nil {
return fmt.Errorf("organization is unavailable")
}
*org = *updated
return nil
}
func (s *server) handleAdminOrganizationStorageProvision(w http.ResponseWriter, r *http.Request) {
org, err := s.store.GetOrganization(r.Context(), chi.URLParam(r, "id"))
if err != nil || org == nil {
renderJSON(w, 404, map[string]string{"error": "organization not found"})
return
}
if err = s.provisionOrganizationStorage(r.Context(), org); err != nil {
renderJSON(w, 503, map[string]any{"error": err.Error(), "organization": org})
return
}
renderJSON(w, 200, org)
}
func iamOperationError(stage string, err error) error {
var e *bce.BceServiceError
if errors.As(err, &e) {
return fmt.Errorf("IAM %s failed (HTTP %d, %s)", stage, e.StatusCode, e.Code)
}
return fmt.Errorf("IAM %s failed", stage)
}
func organizationIAMClient(ctx context.Context, source domain.StorageConnectionRecord) (*iam.Client, error) {
client, err := iam.NewClientWithEndpoint(source.AccessKeyID, source.SecretAccessKey, "https://iam.bj.baidubce.com")
if err != nil {
return nil, fmt.Errorf("IAM client initialization failed")
}
client.Config.Retry = bce.NewNoRetryPolicy()
timeout := 15 * time.Second
client.Config.HTTPClient = &http.Client{Transport: organizationStorageTransport{ctx}, Timeout: timeout}
client.Config.HTTPClientTimeout = &timeout
client.BceClient, err = bce.NewBceClientWithExclusiveHTTPClient(client.Config, &auth.BceV1Signer{})
if err != nil {
return nil, fmt.Errorf("IAM client initialization failed")
}
return client, nil
}
var organizationStorageRevoke = revokeBOSOrganizationStorage
func revokeBOSOrganizationStorage(ctx context.Context, source domain.StorageConnectionRecord, org domain.OrganizationRecord) error {
client, err := organizationIAMClient(ctx, source)
if err != nil {
return err
}
name := "poc-org-" + strings.TrimPrefix(org.ID, "org-")
user, err := client.GetUser(name)
if cloudStatus(err, 404) {
return nil
}
if err != nil {
return iamOperationError("identity lookup", err)
}
if user.Description != "Droboticflow organization "+org.ID {
return fmt.Errorf("IAM identity ownership mismatch")
}
keys, err := client.ListAccessKey(name)
if err != nil {
return iamOperationError("key lookup", err)
}
for _, key := range keys.AccessKeys {
if key.Enabled {
if _, err := client.DisableAccessKey(name, key.Id); err != nil {
return iamOperationError("key revocation", err)
}
}
}
return nil
}
// Pending organizations are a durable queue. Leases serialize replicas and
// the per-organization deadline bounds every IAM/BOS attempt.
func (s *server) startOrganizationStorageReconciler() {
go func() {
ticker := time.NewTicker(15 * time.Second)
defer ticker.Stop()
for range ticker.C {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
organizations, err := s.store.ListOrganizations(ctx)
cancel()
if err != nil {
continue
}
processed := 0
for _, org := range organizations {
if org.StorageRootPrefix != "" && org.StorageState == "pending" && org.Status == domain.OrganizationStatusActive {
_ = s.provisionOrganizationStorage(context.Background(), &org)
processed++
if processed >= 20 {
break
}
}
}
}
}()
}

View File

@ -0,0 +1,180 @@
package api
import (
"context"
"droboticflow/internal/config"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"encoding/json"
"errors"
"net/http"
"testing"
"time"
)
func TestNewOrganizationsGetScopedStorageAndOldOnesStayWholeBucket(t *testing.T) {
data := store.NewMemoryStore()
source := domain.StorageConnectionRecord{ID: "source", OrganizationID: domain.DefaultOrganizationID, Name: "BOS", Provider: "BOS", Bucket: "testxyz", Endpoint: "https://bj.bcebos.com", AccessKeyID: "master-ak", SecretAccessKey: "master-sk"}
if err := data.UpsertStorageConnection(t.Context(), source); err != nil {
t.Fatal(err)
}
old := domain.OrganizationRecord{ID: "old-ubt", Slug: "old-ubt", Name: "UBT", Status: "active"}
data.UpsertOrganization(t.Context(), old)
browser := newFakeObjectStorageBrowser()
browser.objects = map[string]domain.ObjectStorageObjectSummary{}
browser.content = map[string][]byte{}
oldResolver, oldProvision := storageConnectionBrowserResolver, organizationStorageProvision
oldVerify := organizationStorageVerify
oldEmpty := organizationDirectoryEmpty
organizationDirectoryEmpty = func(context.Context, domain.StorageConnectionRecord, string) (bool, error) { return true, nil }
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
t.Cleanup(func() {
organizationDirectoryEmpty = oldEmpty
organizationStorageVerify = oldVerify
storageConnectionBrowserResolver = oldResolver
organizationStorageProvision = oldProvision
})
storageConnectionBrowserResolver = func(context.Context, *domain.StorageConnectionRecord) (objectStorageBrowser, func(), error) {
return browser, func() {}, nil
}
calls := 0
fail := false
organizationStorageProvision = func(_ context.Context, s domain.StorageConnectionRecord, o domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
calls++
if fail {
return domain.StorageConnectionRecord{}, errors.New("IAM unavailable")
}
return domain.StorageConnectionRecord{ID: o.StorageConnectionID, OrganizationID: o.ID, Name: "BOS Workspace", Provider: "BOS", Bucket: o.StorageBucket, RootPrefix: o.StorageRootPrefix, AccessKeyID: "tenant-ak", SecretAccessKey: "tenant-sk"}, nil
}
h := NewHandlerWithConfig(data, nil, config.AppConfig{RequireSessionAuth: true, OrganizationStorageSourceConnectionID: "source"})
root := bootstrapSuperAdminSession(t, h)
created := adminCreateOrganization(t, h, root, "Company A", "company-a")
if created.StorageRootPrefix != "company-a/" || created.StorageState != "ready" {
t.Fatalf("organization %+v", created)
}
c, _ := data.GetStorageConnection(t.Context(), created.StorageConnectionID)
if c == nil || c.RootPrefix != "company-a/" || c.AccessKeyID == source.AccessKeyID {
t.Fatal("tenant inherited master credentials")
}
unchanged, _ := data.GetOrganization(t.Context(), old.ID)
if unchanged.StorageRootPrefix != "" {
t.Fatal("old organization changed")
}
savedSource, _ := data.GetStorageConnection(t.Context(), "source")
if savedSource.RootPrefix != "" {
t.Fatal("source was restricted")
}
r := serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+created.ID+"/storage/provision", []byte(`{}`), root)
if r.Code != 200 || calls != 1 {
t.Fatalf("idempotency: %d calls=%d", r.Code, calls)
}
// Organization admins cannot add a new unscoped connection.
u := adminCreateUser(t, h, root, created.ID, "Admin", "scoped@example.com", "admin", "fixture-password")
cookie := loginSession(t, h, u.Email, "fixture-password")
r = serveJSONWithCookie(h, "POST", "/api/storage/connections", []byte(`{"name":"escape","provider":"BOS","bucket":"testxyz","endpoint":"https://bj.bcebos.com","accessKeyId":"ak","secretAccessKey":"sk"}`), cookie)
if r.Code != 403 {
t.Fatalf("unscoped create = %d", r.Code)
}
fail = true
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Pending","slug":"pending-org"}`), root)
if r.Code != 503 {
t.Fatal(r.Code)
}
var failure struct {
Organization domain.OrganizationRecord `json:"organization"`
}
json.Unmarshal(r.Body.Bytes(), &failure)
pending := failure.Organization
if pending.StorageState != "pending" {
t.Fatal("failed open")
}
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+pending.ID+"/users", []byte(`{"name":"No","email":"no@example.com","role":"admin","password":"password"}`), root)
if r.Code != http.StatusConflict {
t.Fatalf("pending user create %d", r.Code)
}
fail = false
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+pending.ID+"/storage/provision", []byte(`{}`), root)
if r.Code != 200 {
t.Fatalf("retry: %d %s", r.Code, r.Body.String())
}
// A one-time secret is persisted before IAM propagation verification, so
// retries do not mint or discard another tenant key.
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return errors.New("IAM propagating") }
beforeCalls := calls
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Propagation","slug":"propagation-test"}`), root)
if r.Code != 503 {
t.Fatal(r.Code)
}
json.Unmarshal(r.Body.Bytes(), &failure)
propagating := failure.Organization
stored, _ := data.GetStorageConnection(t.Context(), propagating.StorageConnectionID)
if stored == nil || stored.SecretAccessKey != "tenant-sk" {
t.Fatal("one-time secret was lost")
}
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+propagating.ID+"/storage/provision", []byte(`{}`), root)
if r.Code != 200 || calls != beforeCalls+1 {
t.Fatalf("propagation retry: %d, calls=%d", r.Code, calls)
}
// A reservation survives org deletion; its old directory cannot be reassigned.
if _, err := data.DeleteOrganizationCascade(t.Context(), created.ID); err != nil {
t.Fatal(err)
}
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Replacement","slug":"company-a"}`), root)
if r.Code != 503 {
t.Fatalf("directory reassigned: %d", r.Code)
}
}
func TestOrganizationStoragePolicyScope(t *testing.T) {
var p struct {
ACL []struct {
Permission []string `json:"permission"`
Resource []string `json:"resource"`
} `json:"accessControlList"`
}
json.Unmarshal([]byte(organizationStoragePolicy("testxyz", "company-a/")), &p)
if len(p.ACL) != 2 || len(p.ACL[0].Resource) != 2 || p.ACL[0].Resource[1] != "testxyz/company-a/*" || p.ACL[1].Permission[0] != "READ" {
t.Fatalf("bad policy %s", organizationStoragePolicy("testxyz", "company-a/"))
}
}
func TestAsyncOrganizationCreationReturnsPendingThenBecomesReady(t *testing.T) {
data := store.NewMemoryStore()
data.UpsertStorageConnection(t.Context(), domain.StorageConnectionRecord{ID: "source", Bucket: "testxyz", Endpoint: "https://bj.bcebos.com", Provider: "BOS"})
oldP, oldV, oldE := organizationStorageProvision, organizationStorageVerify, organizationDirectoryEmpty
defer func() {
organizationStorageProvision = oldP
organizationStorageVerify = oldV
organizationDirectoryEmpty = oldE
}()
proceed := make(chan struct{})
organizationDirectoryEmpty = func(context.Context, domain.StorageConnectionRecord, string) (bool, error) { return true, nil }
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
organizationStorageProvision = func(_ context.Context, _ domain.StorageConnectionRecord, o domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
<-proceed
return domain.StorageConnectionRecord{ID: o.StorageConnectionID, OrganizationID: o.ID, Bucket: o.StorageBucket, RootPrefix: o.StorageRootPrefix, Provider: "BOS", AccessKeyID: "scoped", SecretAccessKey: "secret"}, nil
}
h := NewHandlerWithConfig(data, nil, config.AppConfig{RequireSessionAuth: true, OrganizationStorageSourceConnectionID: "source", OrganizationStorageReconcileEnabled: true})
root := bootstrapSuperAdminSession(t, h)
r := serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Async","slug":"async-scope"}`), root)
if r.Code != 202 {
close(proceed)
t.Fatal(r.Code)
}
var org domain.OrganizationRecord
json.Unmarshal(r.Body.Bytes(), &org)
if org.StorageState != "pending" {
close(proceed)
t.Fatal("expected pending response")
}
close(proceed)
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) {
current, _ := data.GetOrganization(t.Context(), org.ID)
if current.StorageState == "ready" {
return
}
time.Sleep(time.Millisecond)
}
t.Fatal("background provisioning did not finish")
}

View File

@ -46,6 +46,8 @@ type validationIssue struct {
}
type server struct {
organizationStorageAsync bool
organizationStorageSourceConnectionID string
store store.Store
ai ai.Service
operatorRunnerImage string
@ -232,6 +234,9 @@ func listStorageBucketsFromConnection(ctx context.Context, connection *domain.St
if connection == nil {
return []string{}, nil
}
if connection.RootPrefix != "" {
return []string{connection.Bucket}, nil
}
endpoint := storageEndpointForProvider(connection.Provider, connection.Endpoint)
if normalizeStorageProvider(connection.Provider) == domain.StorageProviderBOS && endpoint == "" {
return nil, fmt.Errorf("BOS endpoint is required for storage connection %s", strings.TrimSpace(connection.Name))
@ -291,6 +296,10 @@ func listStorageObjectsFromConnection(ctx context.Context, connection *domain.St
if connection == nil {
return []string{}, nil
}
prefix = connection.StorageBrowsePrefix(prefix)
if err := connection.CheckStoragePath(bucket, prefix, true); err != nil {
return nil, err
}
normalizedBucket := strings.TrimSpace(bucket)
if normalizedBucket == "" {
return []string{}, nil
@ -1251,6 +1260,8 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
operatorRunnerImage: appConfig.OperatorRunnerImage,
workerAPIBaseURL: appConfig.WorkerAPIBaseURL,
globalBOSEndpoint: appConfig.GlobalBOSEndpoint,
organizationStorageSourceConnectionID: appConfig.OrganizationStorageSourceConnectionID,
organizationStorageAsync: appConfig.OrganizationStorageReconcileEnabled,
apiToken: appConfig.APIToken,
requireSessionAuth: appConfig.RequireSessionAuth,
sessionCookieSecure: appConfig.SessionCookieSecure,
@ -1271,6 +1282,9 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
buildTime: appConfig.BuildTime,
image: appConfig.Image,
}
if instance.organizationStorageAsync {
instance.startOrganizationStorageReconciler()
}
router := chi.NewRouter()
router.Use(middleware.Compress(5))
router.Use(instance.apiTokenAuth)
@ -1289,6 +1303,7 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
router.Use(instance.requireSuperAdmin)
router.Get("/organizations", instance.handleAdminOrganizationsList)
router.Post("/organizations", instance.handleAdminOrganizationsCreate)
router.Post("/organizations/{id}/storage/provision", instance.handleAdminOrganizationStorageProvision)
router.Put("/organizations/{id}", instance.handleAdminOrganizationsUpdate)
router.Delete("/organizations/{id}", instance.handleAdminOrganizationsDelete)
router.Get("/organizations/{id}/users", instance.handleAdminOrganizationUsersList)
@ -1510,6 +1525,7 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
router.Get("/api/object-storage/objects", instance.handleObjectStorageObjects)
router.Get("/api/object-storage/content", instance.handleObjectStorageContent)
router.Head("/api/object-storage/content", instance.handleObjectStorageContent)
router.Get("/api/object-storage/directory/manifest", instance.handleObjectStorageDirectoryManifest)
router.Post("/api/object-storage/buckets", instance.handleObjectStorageBucketCreate)
router.Delete("/api/object-storage/buckets/{name}", instance.handleObjectStorageBucketDelete)
router.Post("/api/object-storage/objects", instance.handleObjectStorageObjectCreate)
@ -3594,9 +3610,13 @@ func (server *server) validateScanConnectionReference(ctx context.Context, scanC
if scanConfig == nil || strings.TrimSpace(scanConfig.ConnectionID) == "" {
return nil
}
if _, err := server.getStorageConnection(ctx, scanConfig.ConnectionID); err != nil {
connection, err := server.getStorageConnection(ctx, scanConfig.ConnectionID)
if err != nil || connection == nil {
return []validationIssue{{Field: "scanConfig.connectionId", Message: "connectionId must reference an existing storage connection"}}
}
if err := connection.CheckStorageSource(scanConfig.Bucket, scanConfig.Prefix, scanConfig.SourcePath); err != nil {
return []validationIssue{{Field: "scanConfig.prefix", Message: err.Error()}}
}
return nil
}
@ -7421,6 +7441,7 @@ func (server *server) handleUserStoragePathFavoritesUpdate(writer http.ResponseW
}
type storageConnectionPatch struct {
RootPrefix *string `json:"rootPrefix"`
Name *string `json:"name"`
Description *string `json:"description"`
Provider *domain.StorageProvider `json:"provider"`
@ -7539,10 +7560,28 @@ func (server *server) handleStorageConnectionsList(writer http.ResponseWriter, r
}
func (server *server) handleStorageConnectionsCreate(writer http.ResponseWriter, request *http.Request) {
principal, authenticated := currentPrincipal(request)
if authenticated && !principal.User.IsSuperAdmin && principal.Organization.StorageRootPrefix != "" {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "organization storage is managed by the platform"})
return
}
var payload domain.CreateStorageConnectionInput
if !decodeJSON(writer, request, &payload) {
return
}
if payload.RootPrefix != "" {
principal, ok := currentPrincipal(request)
if !ok || !principal.User.IsSuperAdmin {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "only super admin can configure storage root"})
return
}
root, err := domain.NormalizeStorageRoot(payload.RootPrefix)
if err != nil || root == "" || strings.TrimSpace(payload.Bucket) == "" {
renderValidationError(writer, validationIssue{Field: "rootPrefix", Message: "valid root prefix and bucket are required"})
return
}
payload.RootPrefix = root
}
payload.Name = strings.TrimSpace(payload.Name)
payload.Description = strings.TrimSpace(payload.Description)
payload.Provider = normalizeStorageProvider(payload.Provider)
@ -7613,6 +7652,20 @@ func (server *server) handleStorageConnectionsUpdate(writer http.ResponseWriter,
if !decodeJSON(writer, request, &payload) {
return
}
principal, isAuthenticated := currentPrincipal(request)
superAdmin := isAuthenticated && principal.User.IsSuperAdmin
if !superAdmin && (payload.RootPrefix != nil || (item.RootPrefix != "" && (payload.Bucket != nil || payload.Endpoint != nil || payload.Provider != nil || (payload.AccessKeyID != nil && strings.TrimSpace(*payload.AccessKeyID) != "") || (payload.SecretAccessKey != nil && strings.TrimSpace(*payload.SecretAccessKey) != "")))) {
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "only super admin can change a scoped storage connection"})
return
}
if payload.RootPrefix != nil {
root, err := domain.NormalizeStorageRoot(*payload.RootPrefix)
if err != nil {
renderValidationError(writer, validationIssue{Field: "rootPrefix", Message: err.Error()})
return
}
item.RootPrefix = root
}
if payload.Name != nil {
item.Name = strings.TrimSpace(*payload.Name)
}
@ -7651,6 +7704,10 @@ func (server *server) handleStorageConnectionsUpdate(writer http.ResponseWriter,
renderValidationError(writer, *issue)
return
}
if item.RootPrefix != "" && item.Bucket == "" {
renderValidationError(writer, validationIssue{Field: "bucket", Message: "scoped connections require a bucket"})
return
}
item.Endpoint = server.storageEndpointForProvider(item.Provider, item.Endpoint)
item.UpdatedAt = nowISO()
if err := server.store.UpsertStorageConnection(request.Context(), *item); err != nil {
@ -8518,6 +8575,9 @@ func renderValidationError(writer http.ResponseWriter, issues ...validationIssue
}
func renderError(writer http.ResponseWriter, status int, err error) {
if errors.Is(err, domain.ErrStorageScope) {
status = http.StatusForbidden
}
renderJSON(writer, status, map[string]string{"error": err.Error()})
}

View File

@ -9295,6 +9295,10 @@ func (browser *fakeObjectStorageBrowser) ListObjects(_ context.Context, bucket s
return items, nil
}
func (browser *fakeObjectStorageBrowser) ListObjectsRecursive(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
return browser.ListObjects(ctx, bucket, prefix)
}
func (browser *fakeObjectStorageBrowser) CreateBucket(_ context.Context, bucket string) (domain.ObjectStorageBucketSummary, error) {
browser.buckets[bucket] = struct{}{}
return domain.ObjectStorageBucketSummary{Name: bucket}, nil
@ -9612,6 +9616,25 @@ func TestObjectStorageBrowserSupportsConnectionCRUDAndPreview(t *testing.T) {
t.Fatalf("unexpected content type: %s", got)
}
manifestRecorder := httptest.NewRecorder()
handler.ServeHTTP(
manifestRecorder,
httptest.NewRequest(http.MethodGet, "/api/object-storage/directory/manifest?connectionId="+connection.ID+"&prefix=algo", nil),
)
if manifestRecorder.Code != http.StatusOK {
t.Fatalf("directory manifest status: %d body=%s", manifestRecorder.Code, manifestRecorder.Body.String())
}
var manifestResponse objectStorageDirectoryManifestResponse
if err := json.NewDecoder(manifestRecorder.Body).Decode(&manifestResponse); err != nil {
t.Fatalf("decode manifest response: %v", err)
}
if manifestResponse.TotalFiles == 0 || len(manifestResponse.Files) == 0 {
t.Fatalf("expected manifest files, got %d", manifestResponse.TotalFiles)
}
if manifestResponse.Files[0].DownloadURL == "" {
t.Fatalf("expected download url in manifest item: %+v", manifestResponse.Files[0])
}
var multipartBody bytes.Buffer
multipartWriter := multipart.NewWriter(&multipartBody)
fileWriter, err := multipartWriter.CreateFormFile("file", "hello.txt")

View File

@ -0,0 +1,110 @@
package api
import (
"context"
"droboticflow/internal/domain"
"droboticflow/internal/store"
"errors"
"github.com/baidubce/bce-sdk-go/services/bos"
"net/http"
"net/http/httptest"
"testing"
)
func TestScopedBOSRejectsBeforeNetworkOrSigning(t *testing.T) {
// A nil client makes any missed guard fail immediately; no remote service or
// credential policy can accidentally hide a missing platform check.
b := &bosObjectStorageBrowser{connection: &domain.StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}}
ctx := context.Background()
checks := map[string]func() error{
"list": func() error { _, e := b.ListObjects(ctx, "testxyz", "other"); return e },
"recursive": func() error { _, e := b.ListObjectsRecursive(ctx, "testxyz", "other"); return e },
"page": func() error { _, e := b.ListObjectsPage(ctx, "testxyz", "other", "", 10); return e },
"get": func() error { _, _, e := b.ReadObject(ctx, "testxyz", "other/x"); return e },
"range": func() error { _, _, e := b.ReadObjectRange(ctx, "testxyz", "other/x", 0, 1); return e },
"preview": func() error { _, _, e := b.ReadObjectPreview(ctx, "testxyz", "other/x", 10); return e },
"head": func() error { _, e := b.GetObjectSummary(ctx, "testxyz", "other/x"); return e },
"put": func() error { _, e := b.PutObject(ctx, "testxyz", "other/x", nil, ""); return e },
"presign put": func() error { _, e := b.PresignPutObject(ctx, "testxyz", "other/x", "", 60); return e },
"presign get": func() error { _, e := b.BuildPreviewURL(ctx, "testxyz", "other/x"); return e },
"initiate": func() error { _, e := b.InitiateMultipartUpload(ctx, "testxyz", "other/x", "", 60); return e },
"part": func() error { _, e := b.PresignMultipartUploadPart(ctx, "testxyz", "other/x", "id", 1, 60); return e },
"complete": func() error { _, e := b.CompleteMultipartUpload(ctx, "testxyz", "other/x", "id", nil); return e },
"abort": func() error { return b.AbortMultipartUpload(ctx, "testxyz", "other/x", "id") },
"delete": func() error { return b.DeleteObject(ctx, "testxyz", "other/x") },
"delete prefix": func() error { return b.DeletePrefix(ctx, "testxyz", "other") },
"rename source": func() error { _, e := b.RenameObject(ctx, "testxyz", "other/x", "ubt/x"); return e },
"rename target": func() error { _, e := b.RenameObject(ctx, "testxyz", "ubt/x", "other/x"); return e },
"move": func() error { _, e := b.MovePrefix(ctx, "testxyz", "ubt/x", "other"); return e },
}
for name, fn := range checks {
t.Run(name, func(t *testing.T) {
if e := fn(); !errors.Is(e, domain.ErrStorageScope) {
t.Fatalf("expected scope denial, got %v", e)
}
})
}
w := httptest.NewRecorder()
renderError(w, 500, domain.ErrStorageScope)
if w.Code != http.StatusForbidden {
t.Fatal(w.Code)
}
}
func TestScopedConnectionCannotBeWidenedByOrganizationAdmin(t *testing.T) {
t.Setenv("DROBOTICFLOW_REQUIRE_SESSION_AUTH", "true")
data := store.NewMemoryStore()
h := NewHandler(data, nil)
root := bootstrapSuperAdminSession(t, h)
org := adminCreateOrganization(t, h, root, "UBT", "ubt-scope")
user := adminCreateUser(t, h, root, org.ID, "UBT Admin", "ubt-scope@example.com", "admin", "fixture-password")
cookie := loginSession(t, h, user.Email, "fixture-password")
c := domain.StorageConnectionRecord{ID: "scoped", OrganizationID: org.ID, Name: "UBT", Provider: domain.StorageProviderBOS, Bucket: "testxyz", RootPrefix: "ubt/", Endpoint: "https://bj.bcebos.com", AccessKeyID: "fixture", SecretAccessKey: "fixture"}
if err := data.UpsertStorageConnection(t.Context(), c); err != nil {
t.Fatal(err)
}
for _, body := range []string{`{"rootPrefix":""}`, `{"rootPrefix":"other/"}`, `{"bucket":"other"}`, `{"endpoint":"https://example.com"}`, `{"accessKeyId":"broad"}`, `{"secretAccessKey":"broad"}`} {
r := serveJSONWithCookie(h, http.MethodPut, "/api/storage/connections/scoped", []byte(body), cookie)
if r.Code != 403 {
t.Fatalf("mutation %s: %d %s", body, r.Code, r.Body.String())
}
}
for _, target := range []string{
"/api/object-storage/objects?connectionId=scoped&bucket=testxyz&prefix=other",
"/api/object-storage/content?connectionId=scoped&bucket=testxyz&key=other/x",
} {
r := serveJSONWithCookie(h, http.MethodGet, target, nil, cookie)
if r.Code != 403 {
t.Fatalf("scope escape: %d %s", r.Code, r.Body.String())
}
}
r := serveJSONWithCookie(h, http.MethodPut, "/api/storage/connections/scoped", []byte(`{"name":"UBT Storage"}`), cookie)
if r.Code != 200 {
t.Fatalf("rename: %d %s", r.Code, r.Body.String())
}
got, _ := data.GetStorageConnection(t.Context(), "scoped")
if got.RootPrefix != "ubt/" {
t.Fatal("scope lost")
}
}
func TestScopedListingUsesAuthorizedRoot(t *testing.T) {
endpoint := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("prefix") != "ubt/" {
t.Errorf("unexpected prefix: %s", r.URL.String())
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"name":"testxyz","prefix":"ubt/","contents":[{"key":"ubt/file.txt","size":3}],"isTruncated":false}`))
}))
defer endpoint.Close()
client, err := bos.NewClient("fixture-key", "fixture-secret", endpoint.URL)
if err != nil {
t.Fatal(err)
}
b := &bosObjectStorageBrowser{client: client}
b.connection = &domain.StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}
page, err := b.ListObjectsPage(t.Context(), "testxyz", "", "", 10)
if err != nil || len(page.Items) != 1 || page.Items[0].Key != "ubt/file.txt" {
t.Fatalf("page=%+v err=%v", page, err)
}
}

View File

@ -10,6 +10,8 @@ import (
)
type AppConfig struct {
OrganizationStorageReconcileEnabled bool
OrganizationStorageSourceConnectionID string
Port int
WorkerPort int
CoordinatorPort int
@ -129,7 +131,7 @@ func LoadValidated() (AppConfig, error) {
return AppConfig{}, err
}
return AppConfig{
Port: port,
OrganizationStorageReconcileEnabled: strings.TrimSpace(os.Getenv("ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID")) != "", OrganizationStorageSourceConnectionID: strings.TrimSpace(os.Getenv("ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID")), Port: port,
WorkerPort: workerPort,
CoordinatorPort: coordinatorPort,
MongoURI: mongoURI,

View File

@ -8,6 +8,7 @@ type ScanNotificationSettings struct {
}
type ScanConfig struct {
StorageRootPrefix string `json:"-" bson:"-"`
DataSource string `json:"dataSource,omitempty" bson:"dataSource,omitempty"`
SourcePath string `json:"sourcePath,omitempty" bson:"sourcePath,omitempty"`
ConnectionID string `json:"connectionId,omitempty" bson:"connectionId,omitempty"`

View File

@ -0,0 +1,93 @@
package domain
import (
"errors"
"fmt"
"strings"
)
var ErrStorageScope = errors.New("storage path is outside the authorized directory")
// NormalizeStorageRoot accepts a literal directory, never a wildcard or a path
// that another consumer could interpret differently. Empty means legacy scope.
func NormalizeStorageRoot(root string) (string, error) {
root = strings.Trim(root, "/")
if root == "" {
return "", nil
}
if !safeStorageKey(root) {
return "", fmt.Errorf("invalid storage root prefix")
}
return root + "/", nil
}
func safeStorageKey(key string) bool {
if strings.ContainsAny(key, "\\%*?\x00\r\n") || strings.HasPrefix(key, "/") {
return false
}
for _, part := range strings.Split(strings.TrimSuffix(key, "/"), "/") {
if part == "" || part == "." || part == ".." {
return false
}
}
return strings.TrimSpace(key) == key
}
// CheckStoragePath is shared by the API, coordinator and worker. Directory
// checks allow the root itself; object checks cannot address its sibling object.
func (c *StorageConnectionRecord) CheckStoragePath(bucket, key string, directory bool) error {
if c == nil || c.RootPrefix == "" {
return nil
}
root, err := NormalizeStorageRoot(c.RootPrefix)
if err != nil || root == "" || c.Bucket == "" || bucket != c.Bucket || !safeStorageKey(key) {
return ErrStorageScope
}
if directory && strings.TrimSuffix(key, "/") == strings.TrimSuffix(root, "/") {
return nil
}
if !strings.HasPrefix(key, root) {
return ErrStorageScope
}
return nil
}
func (c *StorageConnectionRecord) StorageBrowsePrefix(prefix string) string {
if c != nil && c.RootPrefix != "" && prefix == "" {
return strings.TrimSuffix(c.RootPrefix, "/")
}
return prefix
}
func (c *StorageConnectionRecord) InternalStorageKey(key string) string {
if c == nil || c.RootPrefix == "" {
return key
}
return strings.TrimSuffix(c.RootPrefix, "/") + "/" + key
}
// CheckStorageSource validates the effective source URI as well as form fields.
func (c *StorageConnectionRecord) CheckStorageSource(bucket, prefix, sourcePath string) error {
if c == nil || c.RootPrefix == "" {
return nil
}
if sourcePath != "" {
if !strings.HasPrefix(sourcePath, "bos://") {
return ErrStorageScope
}
parts := strings.SplitN(strings.TrimPrefix(sourcePath, "bos://"), "/", 2)
if len(parts) != 2 {
return ErrStorageScope
}
if err := c.CheckStoragePath(parts[0], parts[1], true); err != nil {
return err
}
}
if bucket != "" || prefix != "" {
return c.CheckStoragePath(bucket, prefix, true)
}
if sourcePath == "" {
return ErrStorageScope
}
return nil
}

View File

@ -0,0 +1,33 @@
package domain
import "testing"
func TestStorageScopeBoundaries(t *testing.T) {
c := &StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}
for _, key := range []string{"other/x", "ubt2/x", "ubt", "", "ubt/../other", "ubt/%2e%2e/other", "ubt//x", "/ubt/x", "ubt/./x", "ubt/a\\b"} {
if c.CheckStoragePath("testxyz", key, false) == nil {
t.Errorf("allowed %q", key)
}
}
for _, key := range []string{"ubt/x", "ubt/dataset/capture.mcap", "ubt/.drosync-internal/staging/x"} {
if err := c.CheckStoragePath("testxyz", key, false); err != nil {
t.Errorf("denied %q: %v", key, err)
}
}
if c.CheckStoragePath("other", "ubt/x", false) == nil {
t.Fatal("cross bucket allowed")
}
if c.CheckStoragePath("testxyz", "ubt", true) != nil {
t.Fatal("root directory denied")
}
if c.CheckStorageSource("testxyz", "ubt", "bos://testxyz/other") == nil {
t.Fatal("source URI bypass")
}
if c.InternalStorageKey(".drosync-internal/audit/x") != "ubt/.drosync-internal/audit/x" {
t.Fatal("internal path escaped")
}
legacy := &StorageConnectionRecord{}
if legacy.CheckStoragePath("testxyz", "anything", false) != nil {
t.Fatal("legacy changed")
}
}

View File

@ -863,6 +863,11 @@ type UserRecord struct {
}
type OrganizationRecord struct {
StorageRootPrefix string `json:"storageRootPrefix,omitempty" bson:"storageRootPrefix,omitempty"`
StorageBucket string `json:"storageBucket,omitempty" bson:"storageBucket,omitempty"`
StorageState string `json:"storageState,omitempty" bson:"storageState,omitempty"`
StorageConnectionID string `json:"storageConnectionId,omitempty" bson:"storageConnectionId,omitempty"`
ID string `json:"id" bson:"id"`
Name string `json:"name" bson:"name"`
Slug string `json:"slug" bson:"slug"`
@ -1086,6 +1091,7 @@ type DashboardPOCReadiness struct {
}
type StorageConnectionRecord struct {
RootPrefix string `json:"rootPrefix,omitempty" bson:"rootPrefix,omitempty"`
ID string `json:"id" bson:"id"`
OrganizationID string `json:"organizationId,omitempty" bson:"organizationId,omitempty"`
Name string `json:"name" bson:"name"`
@ -1123,6 +1129,7 @@ type CreateProjectInput struct {
}
type CreateStorageConnectionInput struct {
RootPrefix string `json:"rootPrefix,omitempty" bson:"rootPrefix,omitempty"`
OrganizationID string `json:"organizationId,omitempty"`
Name string `json:"name"`
Description string `json:"description,omitempty"`

View File

@ -38,6 +38,7 @@ const (
)
type LaunchInput struct {
ScopedWorkspace bool
Image string
Command []string
Args []string
@ -581,11 +582,17 @@ func CreateJobManifest(namespace string, imagePullSecretName string, workspacePV
}
sharedMemorySize = quantity.String()
}
if input.ScopedWorkspace && (len(input.PVCMounts) > 0 || input.RegistryAuth) {
return nil, fmt.Errorf("scoped organizations cannot mount shared PVCs or registry credentials")
}
codeSubPath, err := cleanVolumeSubPath(input.CodeSubPath)
if err != nil {
return nil, err
}
codeMountPath := ""
if input.ScopedWorkspace && codeSubPath != "" && !strings.HasPrefix(codeSubPath, "lightweight-operators/") && !strings.HasPrefix(codeSubPath, "operator-repos/") {
return nil, fmt.Errorf("scoped organization code mount must use the operator catalog")
}
if codeSubPath != "" {
codeMountPath, err = cleanMountPath(input.CodeMountPath)
if err != nil {
@ -725,26 +732,38 @@ func CreateJobManifest(namespace string, imagePullSecretName string, workspacePV
},
},
})
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
Name: "workspace",
MountPath: workspaceMountPath,
ReadOnly: input.CodeReadOnly,
})
if codeSubPath != "" {
runSubPath, err := cleanVolumeSubPath(pathClean("system/runs/" + input.RunID))
if !input.ScopedWorkspace {
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
Name: "workspace",
MountPath: workspaceMountPath,
ReadOnly: input.CodeReadOnly,
})
} else {
automount := false
job.Spec.Template.Spec.AutomountServiceAccountToken = &automount
runSubPath, err := cleanVolumeSubPath("system/runs/" + input.RunID)
if err != nil {
return nil, err
}
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
Name: "workspace",
MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID,
SubPath: runSubPath,
})
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{Name: "workspace", MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID, SubPath: runSubPath})
}
if codeSubPath != "" {
if !input.ScopedWorkspace {
runSubPath, err := cleanVolumeSubPath(pathClean("system/runs/" + input.RunID))
if err != nil {
return nil, err
}
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
Name: "workspace",
MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID,
SubPath: runSubPath,
})
}
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
Name: "workspace",
MountPath: codeMountPath,
SubPath: codeSubPath,
ReadOnly: input.CodeReadOnly,
ReadOnly: input.CodeReadOnly || input.ScopedWorkspace,
})
}
}

View File

@ -930,3 +930,30 @@ func TestReadJobDiagnosticsIncludesPodStatusAndEvents(t *testing.T) {
}
}
}
func TestScopedOrganizationJobsCannotMountSharedWorkspace(t *testing.T) {
input := LaunchInput{Image: "example/operator:1", RunID: "run-scoped", NodeID: "node", WorkflowID: "workflow", ScopedWorkspace: true}
job, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input)
if err != nil {
t.Fatal(err)
}
mounts := job.Spec.Template.Spec.Containers[0].VolumeMounts
for _, m := range mounts {
if m.Name == "workspace" && (m.MountPath == "/workspace" || m.SubPath != "system/runs/run-scoped") {
t.Fatalf("shared workspace exposed: %+v", m)
}
}
if job.Spec.Template.Spec.AutomountServiceAccountToken == nil || *job.Spec.Template.Spec.AutomountServiceAccountToken {
t.Fatal("service account exposed")
}
input.PVCMounts = []PVCMount{{Name: "escape", PVCName: "workspace", MountPath: "/other"}}
if _, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input); err == nil {
t.Fatal("shared PVC accepted")
}
input.PVCMounts = nil
input.CodeSubPath = "private/other-org"
input.CodeMountPath = "/app/code"
if _, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input); err == nil {
t.Fatal("private code path accepted")
}
}

View File

@ -59,6 +59,10 @@ func isNotFound(err error) bool {
}
func (publisher BOSPublisher) Head(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string) (ObjectMetadata, error) {
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
return ObjectMetadata{}, err
}
client, err := publisher.client(connection)
if err != nil {
return ObjectMetadata{}, err
@ -108,6 +112,13 @@ func (publisher BOSPublisher) PublishConditional(
ifNoneMatch string,
userMeta map[string]string,
) (ObjectMetadata, error) {
if err := connection.CheckStoragePath(bucket, sourceKey, false); err != nil {
return ObjectMetadata{}, err
}
if err := connection.CheckStoragePath(bucket, targetKey, false); err != nil {
return ObjectMetadata{}, err
}
client, err := publisher.client(connection)
if err != nil {
return ObjectMetadata{}, err
@ -200,6 +211,10 @@ func (publisher BOSPublisher) PutInternal(
contentType string,
userMeta map[string]string,
) (ObjectMetadata, error) {
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
return ObjectMetadata{}, err
}
client, err := publisher.client(connection)
if err != nil {
return ObjectMetadata{}, err
@ -220,6 +235,10 @@ func (publisher BOSPublisher) PutInternal(
}
func (publisher BOSPublisher) Delete(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string) error {
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
return err
}
client, err := publisher.client(connection)
if err != nil {
return err
@ -233,6 +252,10 @@ func (publisher BOSPublisher) Delete(ctx context.Context, connection domain.Stor
}
func (publisher BOSPublisher) AbortMultipart(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key, uploadID string) error {
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
return err
}
client, err := publisher.client(connection)
if err != nil {
return err

View File

@ -431,7 +431,7 @@ func (coordinator *Coordinator) sealAuditDay(ctx context.Context, organizationID
if err != nil || connection == nil {
return false, fmt.Errorf("audit seal storage connection is unavailable")
}
objectKey := path.Join(".drosync-internal/audit", organizationID, dayUTC+".json")
objectKey := connection.InternalStorageKey(path.Join(".drosync-internal/audit", organizationID, dayUTC+".json"))
object, err := coordinator.publisher.PutInternal(
ctx,
*connection,
@ -1033,7 +1033,7 @@ func (coordinator *Coordinator) publishRelease(ctx context.Context, release *dom
return err
}
defer reader.Close()
manifestKey := path.Join(".drosync-internal/releases", release.OrganizationID, release.ID+".ndjson")
manifestKey := connection.InternalStorageKey(path.Join(".drosync-internal/releases", release.OrganizationID, release.ID+".ndjson"))
digest := hex.EncodeToString(hasher.Sum(nil))
if _, err := coordinator.publisher.PutInternal(ctx, *connection, space.Bucket, manifestKey, reader, info.Size(), "application/x-ndjson", map[string]string{
"drosync-release": release.ID,

View File

@ -12,6 +12,9 @@ import (
)
type MemoryStore struct {
organizationStorageRoots map[string]string
organizationStorageLeases map[string]organizationStorageLease
mu sync.Mutex
organizations map[string]domain.OrganizationRecord
projects map[string]domain.ProjectRecord
@ -1824,6 +1827,7 @@ func (store *MemoryStore) CreateStorageConnection(_ context.Context, input domai
defer store.mu.Unlock()
now := nowISO()
item := domain.StorageConnectionRecord{
RootPrefix: input.RootPrefix,
ID: domain.NewID("storage"),
OrganizationID: input.OrganizationID,
Name: input.Name,

View File

@ -1798,6 +1798,7 @@ func (store *MongoStore) GetStorageConnection(ctx context.Context, id string) (*
func (store *MongoStore) CreateStorageConnection(ctx context.Context, input domain.CreateStorageConnectionInput) (*domain.StorageConnectionRecord, error) {
now := nowISO()
item := domain.StorageConnectionRecord{
RootPrefix: input.RootPrefix,
ID: domain.NewID("storage"),
OrganizationID: input.OrganizationID,
Name: input.Name,

View File

@ -0,0 +1,90 @@
package store
import (
"context"
"fmt"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"time"
)
type organizationStorageLease struct {
owner string
until time.Time
}
func (s *MemoryStore) ReserveOrganizationStorageRoot(_ context.Context, bucket, root, org string) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.organizationStorageRoots == nil {
s.organizationStorageRoots = map[string]string{}
}
key := bucket + "/" + root
if owner, ok := s.organizationStorageRoots[key]; ok && owner != org {
return fmt.Errorf("directory is reserved by another organization")
}
s.organizationStorageRoots[key] = org
return nil
}
func (s *MongoStore) ReserveOrganizationStorageRoot(ctx context.Context, bucket, root, org string) error {
_, err := s.db.Collection("organizationStorageRoots").UpdateOne(ctx, bson.M{"_id": bucket + "/" + root, "organizationId": org}, bson.M{"$setOnInsert": bson.M{"organizationId": org, "createdAt": time.Now().UTC()}}, options.Update().SetUpsert(true))
if mongo.IsDuplicateKeyError(err) {
return fmt.Errorf("directory is reserved by another organization")
}
return err
}
func (s *MemoryStore) AcquireOrganizationStorageLease(_ context.Context, id, owner string, until time.Time) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.organizationStorageLeases == nil {
s.organizationStorageLeases = map[string]organizationStorageLease{}
}
if l, ok := s.organizationStorageLeases[id]; ok && l.until.After(time.Now()) {
return false, nil
}
s.organizationStorageLeases[id] = organizationStorageLease{owner, until}
return true, nil
}
func (s *MemoryStore) ReleaseOrganizationStorageLease(_ context.Context, id, owner string) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.organizationStorageLeases[id].owner == owner {
delete(s.organizationStorageLeases, id)
}
return nil
}
func (s *MongoStore) AcquireOrganizationStorageLease(ctx context.Context, id, owner string, until time.Time) (bool, error) {
_, err := s.db.Collection("organizationStorageLeases").UpdateOne(ctx, bson.M{"_id": id, "until": bson.M{"$lte": time.Now()}}, bson.M{"$set": bson.M{"owner": owner, "until": until}}, options.Update().SetUpsert(true))
if mongo.IsDuplicateKeyError(err) {
return false, nil
}
return err == nil, err
}
func (s *MongoStore) ReleaseOrganizationStorageLease(ctx context.Context, id, owner string) error {
_, err := s.db.Collection("organizationStorageLeases").DeleteOne(ctx, bson.M{"_id": id, "owner": owner})
return err
}
func (s *MemoryStore) MarkOrganizationStorageReady(_ context.Context, id string) error {
s.mu.Lock()
defer s.mu.Unlock()
org, ok := s.organizations[id]
if !ok {
return fmt.Errorf("organization is unavailable")
}
org.StorageState = "ready"
org.UpdatedAt = nowISO()
s.organizations[id] = org
return nil
}
func (s *MongoStore) MarkOrganizationStorageReady(ctx context.Context, id string) error {
result, err := s.organizations.UpdateOne(ctx, bson.M{"id": id}, bson.M{"$set": bson.M{"storageState": "ready", "updatedAt": nowISO()}})
if err != nil {
return err
}
if result.MatchedCount == 0 {
return fmt.Errorf("organization is unavailable")
}
return nil
}

View File

@ -0,0 +1,23 @@
package store
import (
"droboticflow/internal/domain"
"testing"
)
func TestReadyTransitionPreservesConcurrentOrganizationChanges(t *testing.T) {
s := NewMemoryStore()
org := domain.OrganizationRecord{ID: "scoped-org", Name: "Renamed", Status: domain.OrganizationStatusDisabled, StorageState: "pending"}
s.UpsertOrganization(t.Context(), org)
if err := s.MarkOrganizationStorageReady(t.Context(), org.ID); err != nil {
t.Fatal(err)
}
got, _ := s.GetOrganization(t.Context(), org.ID)
if got.Status != domain.OrganizationStatusDisabled || got.Name != "Renamed" || got.StorageState != "ready" {
t.Fatalf("lost concurrent changes: %+v", got)
}
s.DeleteOrganizationCascade(t.Context(), org.ID)
if s.MarkOrganizationStorageReady(t.Context(), org.ID) == nil {
t.Fatal("deleted organization revived")
}
}

View File

@ -21,6 +21,11 @@ type WorkerRunCounts struct {
}
type Store interface {
MarkOrganizationStorageReady(context.Context, string) error
ReserveOrganizationStorageRoot(context.Context, string, string, string) error
AcquireOrganizationStorageLease(context.Context, string, string, time.Time) (bool, error)
ReleaseOrganizationStorageLease(context.Context, string, string) error
Ready(context.Context) error
EnsureDefaultOrganization(context.Context) (*domain.OrganizationRecord, error)
ListOrganizations(context.Context) ([]domain.OrganizationRecord, error)

View File

@ -105,6 +105,16 @@ func artifactPrefix(policy domain.ArtifactStoragePolicy, run *domain.RunRecord,
}
func (runtime *Runtime) artifactStoragePolicy(ctx context.Context, workflow *domain.WorkflowRecord) (*domain.ArtifactStoragePolicy, error) {
if workflow != nil {
org, err := runtime.store.GetOrganization(ctx, workflow.OrganizationID)
if err != nil {
return nil, err
}
if org != nil && org.StorageRootPrefix != "" {
return &domain.ArtifactStoragePolicy{ConnectionID: org.StorageConnectionID, Bucket: org.StorageBucket, PrefixTemplate: org.StorageRootPrefix + "artifacts/{projectId}/{runId}/{nodeId}"}, nil
}
}
if workflow != nil && workflow.ArtifactStorage != nil && strings.TrimSpace(workflow.ArtifactStorage.ConnectionID) != "" {
copy := *workflow.ArtifactStorage
return &copy, nil
@ -195,6 +205,11 @@ func (runtime *Runtime) externalizeDirectory(ctx context.Context, publisher obje
if err != nil {
return err
}
if connection.RootPrefix != "" {
if err := validateScopedArtifactPath(filepath.Join(runtime.workspaceRoot, "system", "runs", run.ID), path); err != nil {
return err
}
}
key := basePrefix + "/files/" + normalizeArtifactKey(filepath.ToSlash(relative))
meta, sha, err := putArtifactFile(ctx, publisher, connection, policy.Bucket, key, path)
if err != nil {
@ -243,6 +258,20 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
result := make([]domain.ExecutionArtifact, len(artifacts))
for index, artifact := range artifacts {
result[index] = artifact
if artifact.Payload != nil && artifact.Payload.Mode == domain.ArtifactTransferModeInline {
if err := domain.ValidateArtifactPayload(*artifact.Payload, artifact.Type); err != nil {
return nil, err
}
continue
}
if artifact.Type == domain.PortTypeJSON && artifact.Path == "" && artifact.Data != nil {
inline := domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: artifact.Data}
if err := domain.ValidateArtifactPayload(inline, artifact.Type); err != nil {
return nil, err
}
result[index].Payload = &inline
continue
}
if artifact.Payload != nil && artifact.Payload.Mode == domain.ArtifactTransferModeRef {
if err := domain.ValidateArtifactPayload(*artifact.Payload, artifact.Type); err != nil {
return nil, fmt.Errorf("artifact_ref_invalid: %w", err)
@ -268,6 +297,11 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
}
continue
}
if connection != nil && connection.RootPrefix != "" && !strings.HasPrefix(artifact.Path, "bos://") {
if err := validateScopedArtifactPath(filepath.Join(runtime.workspaceRoot, "system", "runs", run.ID), artifact.Path); err != nil {
return nil, err
}
}
info, statErr := os.Stat(artifact.Path)
if statErr == nil && info.Mode().IsRegular() && artifact.Type == domain.PortTypeJSON && info.Size() <= domain.MaxInlinePayloadBytes {
payloadBytes, readErr := os.ReadFile(artifact.Path)
@ -294,6 +328,11 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
connectionID = policy.ConnectionID
}
if connectionID != "" {
if connection != nil {
if err := connection.CheckStoragePath(parts[0], parts[1], false); err != nil {
return nil, err
}
}
ref := &domain.ArtifactReference{Provider: domain.StorageProviderBOS, ConnectionID: connectionID, Bucket: parts[0], Key: parts[1], SizeBytes: artifact.SizeBytes, Kind: domain.ArtifactRefKindObject}
result[index].Payload = &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Type: artifact.Type, Ref: ref}
result[index].Data = compactArtifactData(artifact.Data)
@ -371,6 +410,9 @@ func (runtime *Runtime) resolveDirectoryRunInputs(ctx context.Context, run *doma
resolvedByInputID[binding.WorkflowInputID] = &refCopy
continue
}
if err := connection.CheckStoragePath(ref.Bucket, ref.Prefix, true); err != nil {
return fmt.Errorf("input_resolution_failed: %w", err)
}
client, err := bos.NewClient(connection.AccessKeyID, connection.SecretAccessKey, connection.Endpoint)
if err != nil {
return fmt.Errorf("input_resolution_failed: %w", err)
@ -429,3 +471,19 @@ func (runtime *Runtime) resolveDirectoryRunInputs(ctx context.Context, run *doma
}
return nil
}
func validateScopedArtifactPath(root, candidate string) error {
base, err := filepath.EvalSymlinks(root)
if err != nil {
return domain.ErrStorageScope
}
target, err := filepath.EvalSymlinks(candidate)
if err != nil {
return domain.ErrStorageScope
}
rel, err := filepath.Rel(base, target)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return domain.ErrStorageScope
}
return nil
}

View File

@ -841,8 +841,14 @@ func (runtime *Runtime) executeNode(ctx context.Context, run *domain.RunRecord,
if workflow != nil {
nodeTimeoutSeconds = workflow.NodeTimeoutSeconds
}
organization, orgErr := runtime.store.GetOrganization(ctx, run.OrganizationID)
if orgErr != nil {
return executionResult{}, nil, orgErr
}
scopedWorkspace := organization != nil && organization.StorageRootPrefix != ""
launched, err := runtime.launcher.Launch(ctx, k8s.LaunchInput{
Image: chooseImage(node, runtime.operatorRunnerImage),
ScopedWorkspace: scopedWorkspace,
Command: cloneStringSlice(node.Data.ContainerCommand),
Args: cloneStringSlice(node.Data.ContainerArgs),
RunID: run.ID,

View File

@ -1739,7 +1739,40 @@ func stringList(value any) []string {
return result
}
func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *domain.WorkflowRecord, node domain.WorkflowNode) (domain.WorkflowNode, error) {
func inputStorageConnectionID(bindings []domain.NodeInputBinding) (string, error) {
connectionID := ""
consider := func(payload *domain.ArtifactPayload) error {
if payload == nil || payload.Mode != domain.ArtifactTransferModeRef || payload.Ref == nil {
return nil
}
candidate := strings.TrimSpace(payload.Ref.ConnectionID)
if candidate == "" {
return nil
}
if connectionID != "" && connectionID != candidate {
return fmt.Errorf("input_storage_ambiguous: node inputs reference multiple storage connections")
}
connectionID = candidate
return nil
}
for index := range bindings {
if err := consider(bindings[index].Payload); err != nil {
return "", err
}
for artifactIndex := range bindings[index].Artifacts {
if err := consider(bindings[index].Artifacts[artifactIndex].Payload); err != nil {
return "", err
}
}
}
return connectionID, nil
}
func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *domain.WorkflowRecord, node domain.WorkflowNode, bindingSets ...[]domain.NodeInputBinding) (domain.WorkflowNode, error) {
var inputBindings []domain.NodeInputBinding
if len(bindingSets) > 0 {
inputBindings = bindingSets[0]
}
copy, err := runtime.resolveBuiltinNodeRuntime(ctx, node)
if err != nil {
return node, err
@ -1752,6 +1785,13 @@ func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *doma
if err != nil {
return node, err
}
if strings.TrimSpace(copy.Data.ConnectionID) == "" {
connectionID, inputErr := inputStorageConnectionID(inputBindings)
if inputErr != nil {
return node, inputErr
}
copy.Data.ConnectionID = connectionID
}
copy, err = runtime.resolveNodeStorageConnection(ctx, copy)
if err != nil {
return node, err
@ -1856,6 +1896,16 @@ func (runtime *Runtime) resolveBuiltinNodeRuntime(ctx context.Context, node doma
}
func (runtime *Runtime) resolveScanWorkflow(ctx context.Context, workflow *domain.WorkflowRecord) (*domain.WorkflowRecord, error) {
if workflow != nil && workflow.ScanConfig != nil {
organization, err := runtime.store.GetOrganization(ctx, workflow.OrganizationID)
if err != nil {
return nil, err
}
if organization != nil && organization.StorageRootPrefix != "" && (workflow.ScanConfig.ConnectionID != organization.StorageConnectionID || (strings.TrimSpace(workflow.ScanConfig.DataSource) != "" && !strings.EqualFold(workflow.ScanConfig.DataSource, "BOS"))) {
return nil, domain.ErrStorageScope
}
}
if workflow == nil || workflow.ScanConfig == nil || strings.TrimSpace(workflow.ScanConfig.ConnectionID) == "" {
return workflow, nil
}
@ -1868,6 +1918,13 @@ func (runtime *Runtime) resolveScanWorkflow(ctx context.Context, workflow *domai
}
copy := *workflow
scanConfig := *workflow.ScanConfig
if err := connection.CheckStorageSource(scanConfig.Bucket, scanConfig.Prefix, scanConfig.SourcePath); err != nil {
return nil, err
}
if connection.RootPrefix != "" {
scanConfig.AccessKeyID = connection.AccessKeyID
scanConfig.SecretAccessKey = connection.SecretAccessKey
}
scanConfig.Endpoint = runtime.storageConnectionEndpoint(connection)
if strings.TrimSpace(scanConfig.AccessKeyID) == "" {
scanConfig.AccessKeyID = connection.AccessKeyID
@ -1906,6 +1963,9 @@ func (runtime *Runtime) resolveNodeStorageConnection(ctx context.Context, node d
if sourcePath == "" {
sourcePath = bosSourcePath(copy.Data.Bucket, copy.Data.Prefix)
}
if err := connection.CheckStorageSource(copy.Data.Bucket, copy.Data.Prefix, sourcePath); err != nil {
return node, err
}
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "DATA_SOURCE", "BOS")
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "SOURCE_PATH", sourcePath)
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "BOS_ENDPOINT", runtime.storageConnectionEndpoint(connection))
@ -2763,7 +2823,7 @@ func (runtime *Runtime) processRun(ctx context.Context, claimed *domain.RunRecor
// Use the node attempt for the Kubernetes Job suffix so a recovered
// execution cannot collide with the vanished attempt name.
runSnapshot.Attempt = executionAttempt
nodeSnapshot, err := runtime.prepareExecutionNode(ctx, workflow, *node)
nodeSnapshot, err := runtime.prepareExecutionNode(ctx, workflow, *node, inputBindings)
if err != nil {
results <- nodeExecutionOutcome{
nodeID: node.ID,

View File

@ -2834,6 +2834,75 @@ func TestRuntimeCommitsIncrementalSourceStateOnlyOnSuccessfulRuns(t *testing.T)
})
}
func TestInputStorageConnectionIDUsesDirectoryReferenceAndRejectsAmbiguity(t *testing.T) {
payload := domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Type: domain.PortTypeFile, Ref: &domain.ArtifactReference{ConnectionID: "storage-source", Bucket: "bucket", Prefix: "capture", Kind: domain.ArtifactRefKindDirectory}}
connectionID, err := inputStorageConnectionID([]domain.NodeInputBinding{{TargetPortID: "in-1", Payload: &payload}})
if err != nil || connectionID != "storage-source" {
t.Fatalf("connectionID=%q err=%v", connectionID, err)
}
other := payload
refCopy := *payload.Ref
refCopy.ConnectionID = "storage-other"
other.Ref = &refCopy
if _, err := inputStorageConnectionID([]domain.NodeInputBinding{{Payload: &payload}, {Payload: &other}}); err == nil {
t.Fatal("multiple input storage connections must fail closed")
}
}
func TestPrepareExecutionNodeResolvesStorageConnectionFromInputBindings(t *testing.T) {
ctx := context.Background()
dataStore := store.NewMemoryStore()
connection, err := dataStore.CreateStorageConnection(ctx, domain.CreateStorageConnectionInput{
Name: "BOS 动态绑定连接",
Provider: domain.StorageProviderBOS,
Endpoint: "https://bj.bcebos.com",
AccessKeyID: "ak-input-binding-1234",
SecretAccessKey: "sk-input-binding-5678",
})
if err != nil {
t.Fatalf("create storage connection: %v", err)
}
runtime, err := newTestRuntime(t, dataStore)
if err != nil {
t.Fatalf("create runtime: %v", err)
}
node := domain.WorkflowNode{
ID: "fanyu-v56-directory",
Type: "operator",
Data: domain.WorkflowNodeData{
Label: "目录读取(多端口)",
},
}
bindings := []domain.NodeInputBinding{
{
TargetPortID: "sourceDirectory",
Payload: &domain.ArtifactPayload{
Mode: domain.ArtifactTransferModeRef,
Type: domain.PortTypeFile,
Ref: &domain.ArtifactReference{
ConnectionID: connection.ID,
Bucket: "testxyz",
Prefix: "datasets/capture1",
Kind: domain.ArtifactRefKindDirectory,
},
},
},
}
prepared, err := runtime.prepareExecutionNode(ctx, nil, node, bindings)
if err != nil {
t.Fatalf("prepare execution node: %v", err)
}
params := nodeHyperparamsMap(prepared)
if params["BOS_ENDPOINT"] != "https://bj.bcebos.com" {
t.Fatalf("expected resolved endpoint, got %q", params["BOS_ENDPOINT"])
}
if params["BOS_ACCESS_KEY_ID"] != "ak-input-binding-1234" || params["BOS_SECRET_ACCESS_KEY"] != "sk-input-binding-5678" {
t.Fatalf("expected resolved bos secrets, got %#v", params)
}
}
func TestPrepareExecutionNodeResolvesStorageConnectionSecrets(t *testing.T) {
ctx := context.Background()
dataStore := store.NewMemoryStore()

View File

@ -0,0 +1,50 @@
package worker
import (
"droboticflow/internal/domain"
"droboticflow/internal/store"
"os"
"path/filepath"
"testing"
)
func TestScopedArtifactPathsRejectSymlinkEscape(t *testing.T) {
root := t.TempDir()
run := filepath.Join(root, "run")
os.Mkdir(run, 0700)
outside := filepath.Join(root, "secret")
os.WriteFile(outside, []byte("fixture"), 0600)
inside := filepath.Join(run, "result")
os.WriteFile(inside, []byte("data"), 0600)
if e := validateScopedArtifactPath(run, inside); e != nil {
t.Fatal(e)
}
if e := validateScopedArtifactPath(run, outside); e == nil {
t.Fatal("outside file accepted")
}
link := filepath.Join(run, "link")
os.Symlink(outside, link)
if e := validateScopedArtifactPath(run, link); e == nil {
t.Fatal("symlink escape accepted")
}
}
func TestScopedOrganizationKeepsInlineArtifacts(t *testing.T) {
data := store.NewMemoryStore()
org := domain.OrganizationRecord{ID: "scoped-org", StorageRootPrefix: "company-a/", StorageBucket: "testxyz", StorageConnectionID: "scoped-connection", StorageState: "ready"}
data.UpsertOrganization(t.Context(), org)
data.UpsertStorageConnection(t.Context(), domain.StorageConnectionRecord{ID: org.StorageConnectionID, OrganizationID: org.ID, RootPrefix: org.StorageRootPrefix, Bucket: org.StorageBucket, Provider: "BOS", Endpoint: "https://bj.bcebos.com"})
rt := &Runtime{store: data}
wf := &domain.WorkflowRecord{OrganizationID: org.ID}
run := &domain.RunRecord{ID: "run-inline", OrganizationID: org.ID}
artifacts := []domain.ExecutionArtifact{{ID: "inline", Type: domain.PortTypeJSON, Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: map[string]any{"ok": true}}}, {ID: "json-data", Type: domain.PortTypeJSON, Data: map[string]any{"value": 1}}}
out, err := rt.normalizeOutputArtifacts(t.Context(), wf, run, "node", artifacts)
if err != nil {
t.Fatal(err)
}
for _, a := range out {
if a.Payload == nil || a.Payload.Mode != domain.ArtifactTransferModeInline {
t.Fatalf("inline artifact lost: %+v", a)
}
}
}

View File

@ -140,6 +140,7 @@ data:
WORKER_MISSING_JOB_MAX_RECOVERIES: "__WORKER_MISSING_JOB_MAX_RECOVERIES__"
WORKER_API_BASE_URL: "__WORKER_API_BASE_URL__"
BOS_ENDPOINT: "__BOS_ENDPOINT__"
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID: "__ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID__"
K8S_NAMESPACE: "__NAMESPACE__"
K8S_IMAGE_PULL_SECRET_NAME: "__K8S_IMAGE_PULL_SECRET_NAME__"
WORKSPACE_PVC_NAME: "__WORKSPACE_PVC_NAME__"
@ -324,6 +325,11 @@ spec:
configMapKeyRef:
name: droboticflow-config
key: WORKER_API_BASE_URL
- name: ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID
valueFrom:
configMapKeyRef:
name: droboticflow-config
key: ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID
- name: BOS_ENDPOINT
valueFrom:
configMapKeyRef:

View File

@ -49,3 +49,6 @@ export CUSTOM_OPERATOR_IMAGE_ARCHIVE_MAX_MIB="${CUSTOM_OPERATOR_IMAGE_ARCHIVE_MA
export AI_API_BASE_URL="${AI_API_BASE_URL:-}"
export AI_MODEL="${AI_MODEL:-}"
# New organizations inherit a scoped child IAM identity, never this source key.
export ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID="${ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID:-storage-485ec27477c334ada4372c3fca91154a}"

View File

@ -0,0 +1,42 @@
#!/usr/bin/env python3
"""Real POC abort/lease regression; creates only isolated sessions and no objects."""
import argparse,json,pathlib,time,uuid,urllib.request,urllib.error
p=argparse.ArgumentParser();p.add_argument('--server',required=True);p.add_argument('--token-file',required=True);p.add_argument('--space',required=True);p.add_argument('--prefix',required=True);p.add_argument('--output',required=True);a=p.parse_args()
token=pathlib.Path(a.token_file).read_text().strip();results=[];sessions=[]
def api(method,path,body=None):
req=urllib.request.Request(a.server.rstrip('/')+path,data=json.dumps(body).encode() if body is not None else None,headers={'Authorization':'Bearer '+token,'Content-Type':'application/json','Idempotency-Key':str(uuid.uuid4())},method=method)
try:
with urllib.request.urlopen(req,timeout=30)as r:return r.status,json.load(r)
except urllib.error.HTTPError as e:return e.code,json.load(e)
def check(name,ok,detail):
results.append({'name':name,'status':'PASS' if ok else 'FAIL','detail':detail});print(json.dumps(results[-1],ensure_ascii=False),flush=True)
if not ok:raise AssertionError(name)
def create(path):
code,s=api('POST','/api/external/object-sync/sessions',{'syncSpaceId':a.space,'mappingId':str(uuid.uuid4()),'clientVersion':'1.0.0','protocolVersion':'v1'});assert code==201,s
sid=s['id'];sessions.append(sid)
code,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':True,'items':[{'clientItemId':str(uuid.uuid4()),'relativePath':path,'kind':'file','sizeBytes':0,'sha256':'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855','contentType':'application/octet-stream'}]});assert code==202,r
code,r=api('POST',f'/api/external/object-sync/sessions/{sid}/plan',{});assert code==202,r
for _ in range(120):
code,r=api('GET',f'/api/external/object-sync/sessions/{sid}/items');items=r.get('items',[])
if items and items[0]['status']=='planned':return sid,items[0]
time.sleep(.5)
raise RuntimeError('plan did not finish')
try:
path=a.prefix.rstrip('/')+'/abort-regression-'+uuid.uuid4().hex+'/empty.bin'
sa,ia=create(path);sb,ib=create(path)
c,r=api('POST',f"/api/external/object-sync/items/{ia['id']}/reserve",{});assert c==200,r;old=r.get('item',r)['fencingToken']
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/reserve",{});check('competing reserve denied',c==409 and r.get('code')=='lease_held',{'http':c})
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/abort',{});check('abort acknowledged',c==200,{'sessionId':sa,'http':c})
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/reserve",{});new=r.get('item',r).get('fencingToken');check('immediate takeover with increased fencing',c==200 and new is not None and new>old,{'http':c,'oldFencing':old,'newFencing':new,'sessionId':sb})
for action in ['heartbeat','reserve','commit']:
c,r=api('POST',f"/api/external/object-sync/items/{ia['id']}/{action}",{});check('aborted '+action+' denied',c==409,{'http':c,'code':r.get('code')})
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/abort',{});assert c==200,r
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/heartbeat",{});check('repeat abort preserves new owner',c==200 and r.get('renewed') is True,{'http':c})
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/complete',{});check('aborted session cannot complete',c==409,{'http':c})
except Exception as e:
if not results or results[-1]['status']!='FAIL':results.append({'name':'execution','status':'FAIL','detail':str(e)})
finally:
for sid in sessions:api('POST',f'/api/external/object-sync/sessions/{sid}/abort',{})
summary={'timestamp':time.time(),'scope':'two sessions, same existing device; no object bytes uploaded','results':results,'passed':sum(r['status']=='PASS' for r in results),'failed':sum(r['status']=='FAIL' for r in results)}
pathlib.Path(a.output).write_text(json.dumps(summary,ensure_ascii=False,indent=2))
raise SystemExit(1 if summary['failed'] else 0)

View File

@ -0,0 +1,30 @@
#!/usr/bin/env python3
"""Manifest validation tests: metadata only, no file upload or capacity claim."""
import argparse,json,pathlib,time,uuid,urllib.request,urllib.error
p=argparse.ArgumentParser();p.add_argument('--server',required=True);p.add_argument('--token-file',required=True);p.add_argument('--space',required=True);p.add_argument('--prefix',required=True);p.add_argument('--output',required=True);a=p.parse_args()
token=pathlib.Path(a.token_file).read_text().strip();results=[];sid=None
def api(method,path,body=None):
req=urllib.request.Request(a.server.rstrip('/')+path,data=json.dumps(body).encode() if body is not None else None,headers={'Authorization':'Bearer '+token,'Content-Type':'application/json','Idempotency-Key':str(uuid.uuid4())},method=method)
try:
with urllib.request.urlopen(req,timeout=30)as r:return r.status,json.load(r)
except urllib.error.HTTPError as e:return e.code,json.load(e)
def check(name,expected,actual,detail=None):
row={'name':name,'expected':expected,'actual':actual,'status':'PASS' if expected==actual else 'FAIL','detail':detail};results.append(row);print(json.dumps(row,ensure_ascii=False),flush=True)
def item(**changes):
v={'clientItemId':str(uuid.uuid4()),'relativePath':a.prefix.rstrip('/')+'/'+uuid.uuid4().hex+'.txt','sizeBytes':0,'kind':'file','sha256':'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'};v.update(changes);return v
try:
c,s=api('POST','/api/external/object-sync/sessions',{'syncSpaceId':a.space,'mappingId':str(uuid.uuid4()),'clientVersion':'1.0.0','protocolVersion':'v1'});assert c==201,s;sid=s['id'];limit=s['policySnapshot']['maxObjectBytes'];quota=s['policySnapshot']['maxSessionBytes']
d=item()
cases=[('zero page number',{'pageNumber':0,'items':[item()]},400),('empty page',{'items':[]},400),('1001 entries in one page',{'items':[item() for _ in range(1001)]},400),('duplicate client ID',{'items':[d,d]},400),('missing client ID',{'items':[item(clientItemId='')]},400),('nonhex SHA',{'items':[item(sha256='g'*64)]},400),('short SHA',{'items':[item(sha256='a'*63)]},400),('negative bytes',{'items':[item(sizeBytes=-1)]},413),('above object policy limit',{'items':[item(sizeBytes=limit+1)]},413),('non-file kind',{'items':[item(kind='directory')]},400),('parent traversal',{'items':[item(relativePath=a.prefix+'/../escape')]},400),('absolute path',{'items':[item(relativePath='/absolute')]},400),('empty path segment',{'items':[item(relativePath=a.prefix+'//file')]},400),('control character',{'items':[item(relativePath=a.prefix+'/line\nfeed')]},400),('above session byte quota',{'items':[item(sizeBytes=limit) for _ in range(quota//limit+1)]},413)]
for name,body,expected in cases:
c,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':False,**body});check(name,expected,c,{'code':r.get('code')})
c,r=api('GET',f'/api/external/object-sync/sessions/{sid}/status');check('rejected pages preserve manifest count',0,r['counts'].get('manifest',0));check('rejected pages preserve total bytes',0,r.get('totalBytes',0))
valid=[item() for _ in range(999)]+[item(relativePath=a.prefix+'/中文 空格/特殊#%.txt')]
c,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':True,'items':valid});check('exactly 1000 entries with Unicode accepted',202,c)
c,r=api('GET',f'/api/external/object-sync/sessions/{sid}/status');check('exact accepted count',1000,r['counts'].get('manifest'))
except Exception as e:results.append({'name':'execution','status':'FAIL','detail':str(e)})
finally:
if sid:api('POST',f'/api/external/object-sync/sessions/{sid}/abort',{})
summary={'scope':'metadata validation only; no 1TiB or 20TiB actual upload','sessionId':sid,'timestamp':time.time(),'results':results,'passed':sum(x['status']=='PASS'for x in results),'failed':sum(x['status']=='FAIL'for x in results)}
pathlib.Path(a.output).write_text(json.dumps(summary,ensure_ascii=False,indent=2))
raise SystemExit(1 if summary['failed'] else 0)

View File

@ -25,6 +25,7 @@ WORKER_POLL_INTERVAL_MS="${WORKER_POLL_INTERVAL_MS:-2000}"
DROBOTICFLOW_SHARED_MEMORY_SIZE="${DROBOTICFLOW_SHARED_MEMORY_SIZE:-8Gi}"
WORKER_API_BASE_URL="${WORKER_API_BASE_URL:-http://droboticflow-worker:3002}"
BOS_ENDPOINT="${BOS_ENDPOINT:-}"
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID="${ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID:-}"
K8S_IMAGE_PULL_SECRET_NAME="${K8S_IMAGE_PULL_SECRET_NAME:-droboticflow-registry}"
WORKSPACE_PVC_NAME="${WORKSPACE_PVC_NAME:-droboticflow-workspace}"
WORKSPACE_MOUNT_PATH="${WORKSPACE_MOUNT_PATH:-/workspace}"
@ -113,6 +114,7 @@ export \
DROBOTICFLOW_SHARED_MEMORY_SIZE \
WORKER_API_BASE_URL \
BOS_ENDPOINT \
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID \
K8S_IMAGE_PULL_SECRET_NAME \
WORKSPACE_PVC_NAME \
WORKSPACE_MOUNT_PATH \
@ -168,6 +170,7 @@ replacements = {
"__DROBOTICFLOW_SHARED_MEMORY_SIZE__": os.environ["DROBOTICFLOW_SHARED_MEMORY_SIZE"],
"__WORKER_API_BASE_URL__": os.environ["WORKER_API_BASE_URL"],
"__BOS_ENDPOINT__": os.environ["BOS_ENDPOINT"],
"__ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID__": os.environ["ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID"],
"__K8S_IMAGE_PULL_SECRET_NAME__": os.environ["K8S_IMAGE_PULL_SECRET_NAME"],
"__WORKSPACE_PVC_NAME__": os.environ["WORKSPACE_PVC_NAME"],
"__WORKSPACE_MOUNT_PATH__": os.environ["WORKSPACE_MOUNT_PATH"],

View File

@ -97,6 +97,18 @@ export function SuperAdminConsole() {
});
}, [loadUsers, selectedOrganizationId]);
const hasPendingStorage = organizations.some((item) => item.storageState === "pending");
useEffect(() => {
if (!hasPendingStorage) return;
const timer = window.setInterval(() => {
void adminApi
.listOrganizations()
.then((result) => setOrganizations(result.items))
.catch(() => {});
}, 5000);
return () => window.clearInterval(timer);
}, [hasPendingStorage]);
const createOrganization = async (event: React.FormEvent) => {
event.preventDefault();
const name = organizationDraft.name.trim();
@ -112,7 +124,26 @@ export function SuperAdminConsole() {
setSelectedOrganizationId(created.id);
setOrganizationDraft(emptyOrganizationDraft);
} catch (err) {
setError(getActionErrorMessage(err, "创建组织失败"));
const message = getActionErrorMessage(err, "创建组织失败");
try {
await loadOrganizations();
} catch {
/* Preserve the original provisioning failure. */
}
setError(message);
} finally {
setSaving(false);
}
};
const retryStorageProvisioning = async (organization: Organization) => {
setSaving(true);
setError("");
try {
const updated = await adminApi.provisionOrganizationStorage(organization.id);
setOrganizations((items) => items.map((item) => (item.id === updated.id ? updated : item)));
} catch (err) {
setError(getActionErrorMessage(err, "配置组织存储失败"));
} finally {
setSaving(false);
}
@ -270,9 +301,12 @@ export function SuperAdminConsole() {
<input
value={organizationDraft.slug}
onChange={(event) => setOrganizationDraft((item) => ({ ...item, slug: event.target.value }))}
placeholder="slug"
placeholder="组织标识,例如 company-a"
className="w-full rounded-lg border border-[#2a2b2f] bg-[#0a0a0a] px-3 py-2 text-sm text-white outline-none focus:border-blue-500"
/>
<p className="text-xs text-gray-400">
新组织将自动配置独立的顶层存储目录。组织标识确定后,目录不会随组织改名而改变。
</p>
<button
type="submit"
disabled={saving || !organizationDraft.name.trim() || !organizationDraft.slug.trim()}
@ -303,6 +337,12 @@ export function SuperAdminConsole() {
<span>
<span className="block text-sm font-semibold text-white">{organization.name}</span>
<span className="block text-xs text-gray-500">{organization.slug}</span>
{organization.storageRootPrefix && (
<span className="block text-xs text-blue-300">
{organization.storageBucket}/{organization.storageRootPrefix} ·{" "}
{organization.storageState === "ready" ? "存储已就绪" : "存储配置中,自动重试"}
</span>
)}
</span>
<span className="text-xs text-gray-400">{statusLabel(organization.status)}</span>
</button>
@ -317,6 +357,21 @@ export function SuperAdminConsole() {
<div>
<h2 className="text-xl font-bold text-white">{selectedOrganization.name}</h2>
<div className="mt-1 text-xs text-gray-500">{selectedOrganization.id}</div>
{selectedOrganization.storageRootPrefix && (
<div className="mt-2 text-sm text-blue-300">
授权目录:{selectedOrganization.storageBucket}/{selectedOrganization.storageRootPrefix}
</div>
)}
{selectedOrganization.storageRootPrefix && selectedOrganization.storageState !== "ready" && (
<button
type="button"
disabled={saving}
onClick={() => void retryStorageProvisioning(selectedOrganization)}
className="mt-2 rounded-lg bg-blue-600 px-3 py-2 text-sm text-white disabled:opacity-50"
>
重试配置存储
</button>
)}
</div>
<div className="flex items-center gap-2">
<button
@ -380,6 +435,7 @@ export function SuperAdminConsole() {
disabled={
saving ||
!selectedOrganizationId ||
(Boolean(selectedOrganization?.storageRootPrefix) && selectedOrganization?.storageState !== "ready") ||
!userDraft.name.trim() ||
!userDraft.email.trim() ||
!userDraft.password.trim()

View File

@ -210,6 +210,7 @@ export default function ObjectStorageBrowser() {
const initialSelectionAppliedRef = useRef(false);
const selectedConnection = connections.find((item) => item.id === selectedConnectionId) || null;
const connectionRoot = normalizePrefix(selectedConnection?.rootPrefix);
const previewURL = selectedItem
? objectStorageApi.buildContentURL({
connectionId: selectedConnectionId,
@ -334,7 +335,7 @@ export default function ObjectStorageBrowser() {
Boolean(initialBucket || initialPrefix || initialConnectionId) &&
(!initialConnectionId || selectedConnectionId === initialConnectionId);
initialSelectionAppliedRef.current = true;
setCurrentPrefix(shouldApplyInitialSelection ? initialPrefix : "");
setCurrentPrefix(shouldApplyInitialSelection && initialPrefix ? initialPrefix : connectionRoot);
setItems([]);
setObjectMarker("");
setNextObjectMarker("");
@ -349,7 +350,7 @@ export default function ObjectStorageBrowser() {
return;
}
void reloadOverview(shouldApplyInitialSelection ? initialBucket : "");
}, [selectedConnectionId]);
}, [selectedConnectionId, connectionRoot]);
const reloadObjects = async (preferredKey = "", prefixOverride?: string, markerOverride = objectMarker) => {
const requestId = objectsRequestIdRef.current + 1;
@ -372,6 +373,7 @@ export default function ObjectStorageBrowser() {
if (objectsRequestIdRef.current !== requestId) {
return result;
}
setCurrentPrefix(result.prefix || connectionRoot);
setItems(result.items || []);
setNextObjectMarker(result.nextMarker || "");
setHasMoreObjects(Boolean(result.hasMore));
@ -586,15 +588,20 @@ export default function ObjectStorageBrowser() {
});
await loadConnections(created.id);
} else if (selectedConnectionId) {
await storageConnectionsApi.update(selectedConnectionId, {
name,
description: connectionDraft.description.trim(),
provider: "BOS",
endpoint,
bucket,
accessKeyId: connectionDraft.accessKeyId.trim(),
secretAccessKey: connectionDraft.secretAccessKey.trim(),
});
await storageConnectionsApi.update(
selectedConnectionId,
selectedConnection?.rootPrefix
? { name, description: connectionDraft.description.trim() }
: {
name,
description: connectionDraft.description.trim(),
provider: "BOS",
endpoint,
bucket,
accessKeyId: connectionDraft.accessKeyId.trim(),
secretAccessKey: connectionDraft.secretAccessKey.trim(),
},
);
await loadConnections(selectedConnectionId);
}
setIsConnectionFormOpen(false);
@ -791,7 +798,7 @@ export default function ObjectStorageBrowser() {
setObjectActionError("请输入目标目录前缀。");
return;
}
const destinationPrefix = moveDestinationInput === "/" ? "" : normalizePrefix(moveDestinationInput);
const destinationPrefix = moveDestinationInput === "/" ? connectionRoot : normalizePrefix(moveDestinationInput);
if (moveDestinationInput !== "/" && !destinationPrefix) {
setObjectActionError("请输入目标目录前缀。");
return;
@ -1122,6 +1129,7 @@ export default function ObjectStorageBrowser() {
<div className="space-y-2">
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Endpoint</label>
<input
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
value={connectionDraft.endpoint}
onChange={(event) => setConnectionDraft((previous) => ({ ...previous, endpoint: event.target.value }))}
className="w-full rounded-xl border border-[#2a2b2f] bg-[#0b0c10] px-4 py-3 text-sm text-white focus:border-blue-500 focus:outline-none"
@ -1132,6 +1140,7 @@ export default function ObjectStorageBrowser() {
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">绑定 Bucket</label>
<input
data-testid="connection-bucket-input"
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
value={connectionDraft.bucket}
onChange={(event) => setConnectionDraft((previous) => ({ ...previous, bucket: event.target.value }))}
className="w-full rounded-xl border border-[#2a2b2f] bg-[#0b0c10] px-4 py-3 text-sm text-white focus:border-blue-500 focus:outline-none"
@ -1155,6 +1164,7 @@ export default function ObjectStorageBrowser() {
<div className="space-y-2">
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Access Key ID</label>
<input
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
value={connectionDraft.accessKeyId}
onChange={(event) =>
setConnectionDraft((previous) => ({
@ -1174,6 +1184,7 @@ export default function ObjectStorageBrowser() {
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Secret Access Key</label>
<input
type="password"
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
value={connectionDraft.secretAccessKey}
onChange={(event) =>
setConnectionDraft((previous) => ({
@ -1250,6 +1261,11 @@ export default function ObjectStorageBrowser() {
>
{selectedConnection ? (
<div className="space-y-4">
{connectionRoot && (
<div className="text-sm text-blue-200">
授权目录:{selectedBucket}/{connectionRoot}/(由平台管理员配置)
</div>
)}
<div>
<div className="text-xs uppercase tracking-[0.2em] text-gray-500">连接名称</div>
<div className="mt-2 text-sm font-medium text-white">{selectedConnection.name}</div>
@ -1272,7 +1288,7 @@ export default function ObjectStorageBrowser() {
</div>
{selectedBucket ? (
<div className="rounded-xl border border-emerald-500/20 bg-emerald-500/10 px-3 py-3 text-sm text-emerald-100">
当前对象浏览和操作都限定在 bucket `{selectedBucket}` 下。
当前对象浏览和操作都限定在 `{selectedBucket}/{connectionRoot ? `${connectionRoot}/` : ""}` 下。
</div>
) : (
<div className="rounded-xl border border-amber-500/20 bg-amber-500/10 px-3 py-3 text-sm text-amber-100">
@ -1318,32 +1334,34 @@ export default function ObjectStorageBrowser() {
<div className="mt-4 flex flex-wrap items-center gap-2 text-xs text-gray-500">
<button
type="button"
onClick={() => setCurrentPrefix("")}
onClick={() => setCurrentPrefix(connectionRoot)}
className={cn(
"rounded-full border px-3 py-1.5",
currentPrefix === ""
currentPrefix === connectionRoot
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
)}
>
根目录
{connectionRoot ? "授权根目录" : "根目录"}
</button>
{prefixSegments.map((segment) => (
<button
key={segment.value}
type="button"
onClick={() => setCurrentPrefix(segment.value)}
className={cn(
"rounded-full border px-3 py-1.5",
currentPrefix === segment.value
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
)}
>
{segment.name}
</button>
))}
{currentPrefix && (
{prefixSegments
.filter((segment) => !connectionRoot || segment.value.startsWith(connectionRoot + "/"))
.map((segment) => (
<button
key={segment.value}
type="button"
onClick={() => setCurrentPrefix(segment.value)}
className={cn(
"rounded-full border px-3 py-1.5",
currentPrefix === segment.value
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
)}
>
{segment.name}
</button>
))}
{currentPrefix && currentPrefix !== connectionRoot && (
<button
type="button"
onClick={() => setCurrentPrefix(parentPrefix(currentPrefix))}
@ -1634,7 +1652,8 @@ export default function ObjectStorageBrowser() {
{selectedObjectKeys.length > 0
? `将移动已勾选的 ${selectedObjectKeys.length} 项。`
: "未勾选时移动当前选中的 1 项。"}
输入 / 会移动到 bucket 根目录。目录会作为目录本身移动到目标目录下,并递归保留内部所有内容。
输入 / 会移动到{connectionRoot ? "授权根目录" : "bucket 根目录"}
。目录会作为目录本身移动到目标目录下,并递归保留内部所有内容。
</div>
</div>
<button

View File

@ -2,6 +2,8 @@ import { apiRequest } from "./client";
import type { Organization, User, UserRole } from "../../types";
export const adminApi = {
provisionOrganizationStorage: (id: string) =>
apiRequest<Organization>(`/api/admin/organizations/${id}/storage/provision`, { method: "POST" }, {}),
listOrganizations: () => apiRequest<{ items: Organization[] }>("/api/admin/organizations"),
createOrganization: (payload: { name: string; slug: string; status?: Organization["status"] }) =>
apiRequest<Organization>("/api/admin/organizations", { method: "POST" }, payload),

View File

@ -2,6 +2,7 @@ import type { StorageConnection } from "../../types";
import { apiRequest } from "./client";
type StorageConnectionPayload = {
rootPrefix?: string;
name: string;
description?: string;
provider: "BOS";

View File

@ -22,6 +22,10 @@ export interface User {
}
export interface Organization {
storageRootPrefix?: string;
storageBucket?: string;
storageState?: "pending" | "ready";
storageConnectionId?: string;
id: string;
name: string;
slug: string;
@ -743,6 +747,7 @@ export interface ScanConfig {
}
export interface StorageConnection {
rootPrefix?: string;
id: string;
name: string;
description?: string;

View File

@ -0,0 +1,16 @@
[package]
name = "drobotic-pull"
version = "1.0.0"
edition = "2021"
description = "Fast, secure direct-from-BOS directory and file downloader"
[dependencies]
anyhow = "1.0"
clap = { version = "4.5", features = ["derive", "env"] }
futures-util = "0.3"
indicatif = { version = "0.17", features = ["tokio"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream", "cookies"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
tokio = { version = "1.40", features = ["fs", "io-util", "macros", "rt-multi-thread", "sync", "time"] }
urlencoding = "2.1"

View File

@ -0,0 +1,443 @@
use anyhow::{anyhow, Context, Result};
use clap::Parser;
use futures_util::StreamExt;
use indicatif::{MultiProgress, ProgressBar, ProgressStyle};
use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::io::AsyncWriteExt;
use tokio::sync::Semaphore;
#[derive(Parser, Debug)]
#[command(
name = "drobotic-pull",
about = "High-speed, direct-from-BOS directory and file downloader with zero AK/SK exposure",
version = "1.0.0"
)]
struct Cli {
/// Droboticflow API base URL (e.g., http://120.48.164.19)
#[arg(long, env = "DROBOTICFLOW_API_URL")]
api: Option<String>,
/// Login email for platform authentication
#[arg(long, env = "DROBOTICFLOW_EMAIL")]
email: Option<String>,
/// Login password for platform authentication
#[arg(long, env = "DROBOTICFLOW_PASSWORD")]
password: Option<String>,
/// Storage connection ID (optional if only 1 connection exists)
#[arg(long, env = "DROBOTICFLOW_CONNECTION_ID")]
connection: Option<String>,
/// Storage bucket name (defaults to connection default bucket)
#[arg(long)]
bucket: Option<String>,
/// BOS remote directory prefix to download (e.g., ubbdata/2026_09_09-09_21_24)
#[arg(long, short = 'p')]
prefix: Option<String>,
/// Direct Manifest API URL (alternative to email/password)
#[arg(long)]
manifest_url: Option<String>,
/// Direct local manifest JSON file path
#[arg(long)]
manifest_file: Option<PathBuf>,
/// Session cookie string (e.g. droboticflow_session=...)
#[arg(long, env = "DROBOTICFLOW_COOKIE")]
cookie: Option<String>,
/// Local destination directory
#[arg(long, short = 'o', default_value = "./downloads")]
output: PathBuf,
/// Number of concurrent file downloads
#[arg(long, short = 'j', default_value_t = 8)]
threads: usize,
}
#[derive(Debug, Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
struct ManifestResponse {
bucket: String,
base_prefix: String,
total_files: usize,
total_bytes: u64,
files: Vec<ManifestFileItem>,
}
#[derive(Debug, Deserialize, Serialize)]
#[serde(rename_all = "camelCase")]
struct ManifestFileItem {
key: String,
relative_path: String,
size_bytes: u64,
download_url: String,
}
#[derive(Debug, Deserialize)]
struct StorageConnectionSummary {
id: String,
name: String,
#[serde(default)]
bucket: String,
}
#[tokio::main]
async fn main() -> Result<()> {
let args = Cli::parse();
println!("============================================================");
println!(" drobotic-pull: Direct BOS Directory Downloader v1.0.0");
println!("============================================================");
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(300))
.cookie_store(true)
.build()
.context("create HTTP client")?;
let manifest = resolve_manifest(&client, &args).await?;
if manifest.files.is_empty() {
println!("ℹ No files found under the specified prefix.");
return Ok(());
}
println!("📦 Remote Bucket : {}", manifest.bucket);
println!("📁 Remote Prefix : {}", manifest.base_prefix);
println!(
"📄 Total Files : {} (Total Size: {})",
manifest.total_files,
format_bytes(manifest.total_bytes)
);
println!("💾 Local Output : {}", args.output.display());
println!("⚡ Concurrency : {} workers", args.threads);
println!("------------------------------------------------------------");
tokio::fs::create_dir_all(&args.output)
.await
.context("create local destination output directory")?;
let multi_progress = MultiProgress::new();
let total_style = ProgressStyle::default_bar()
.template("{spinner:.green} [{elapsed_precise}] [{bar:40.cyan/blue}] {bytes}/{total_bytes} ({percent}%) | {msg} | {eta_precise} ETA")?
.progress_chars("█▓▒░ ");
let total_progress = multi_progress.add(ProgressBar::new(manifest.total_bytes));
total_progress.set_style(total_style);
let completed_files = Arc::new(AtomicU64::new(0));
let skipped_files = Arc::new(AtomicU64::new(0));
let failed_files = Arc::new(AtomicU64::new(0));
let total_file_count = manifest.files.len() as u64;
let semaphore = Arc::new(Semaphore::new(args.threads));
let mut tasks = Vec::with_capacity(manifest.files.len());
for item in manifest.files {
let permit = semaphore.clone().acquire_owned().await.unwrap();
let client_clone = client.clone();
let output_root = args.output.clone();
let total_bar = total_progress.clone();
let completed_counter = completed_files.clone();
let skipped_counter = skipped_files.clone();
let failed_counter = failed_files.clone();
let task = tokio::spawn(async move {
let _permit = permit;
let result = download_file(&client_clone, &output_root, &item, &total_bar).await;
match result {
Ok(DownloadOutcome::Downloaded) => {
let completed = completed_counter.fetch_add(1, Ordering::Relaxed) + 1;
let skipped = skipped_counter.load(Ordering::Relaxed);
total_bar.set_message(format!("{}/{} files", completed + skipped, total_file_count));
}
Ok(DownloadOutcome::Skipped) => {
let skipped = skipped_counter.fetch_add(1, Ordering::Relaxed) + 1;
let completed = completed_counter.load(Ordering::Relaxed);
total_bar.set_message(format!("{}/{} files", completed + skipped, total_file_count));
}
Err(err) => {
failed_counter.fetch_add(1, Ordering::Relaxed);
eprintln!("\n❌ Failed to download {}: {:?}", item.relative_path, err);
}
}
});
tasks.push(task);
}
for task in tasks {
let _ = task.await;
}
total_progress.finish_with_message("Download Complete");
let final_completed = completed_files.load(Ordering::Relaxed);
let final_skipped = skipped_files.load(Ordering::Relaxed);
let final_failed = failed_files.load(Ordering::Relaxed);
println!("------------------------------------------------------------");
println!("✅ All operations finished!");
println!(" - Downloaded : {} files", final_completed);
println!(" - Skipped : {} files (already present & size matched)", final_skipped);
if final_failed > 0 {
println!(" - Failed : {} files (please retry)", final_failed);
}
println!("📂 Files saved to: {}", args.output.canonicalize().unwrap_or(args.output).display());
println!("============================================================");
if final_failed > 0 {
Err(anyhow!("{} files failed to download", final_failed))
} else {
Ok(())
}
}
enum DownloadOutcome {
Downloaded,
Skipped,
}
async fn download_file(
client: &reqwest::Client,
output_root: &Path,
item: &ManifestFileItem,
total_bar: &ProgressBar,
) -> Result<DownloadOutcome> {
let clean_relative = item
.relative_path
.trim_start_matches('/')
.replace('\\', "/");
let target_path = output_root.join(clean_relative);
if let Some(parent) = target_path.parent() {
tokio::fs::create_dir_all(parent)
.await
.context("create parent directory")?;
}
// Check if file already exists with identical size
if let Ok(metadata) = tokio::fs::metadata(&target_path).await {
if metadata.len() == item.size_bytes {
total_bar.inc(item.size_bytes);
return Ok(DownloadOutcome::Skipped);
}
}
let temp_path = target_path.with_extension("drobotic-tmp");
// Retry up to 3 times on network failure
let mut attempts = 0;
let max_attempts = 3;
loop {
attempts += 1;
match stream_download(client, &item.download_url, &temp_path, item.size_bytes, total_bar).await {
Ok(_) => {
tokio::fs::rename(&temp_path, &target_path)
.await
.context("rename temp file to target")?;
return Ok(DownloadOutcome::Downloaded);
}
Err(err) if attempts < max_attempts => {
eprintln!("\n⚠️ Attempt {}/{} failed for {}: {}. Retrying...", attempts, max_attempts, item.relative_path, err);
tokio::time::sleep(Duration::from_millis(500 * attempts as u64)).await;
}
Err(err) => {
let _ = tokio::fs::remove_file(&temp_path).await;
return Err(err);
}
}
}
}
async fn stream_download(
client: &reqwest::Client,
download_url: &str,
temp_path: &Path,
expected_size: u64,
total_bar: &ProgressBar,
) -> Result<()> {
let response = client
.get(download_url)
.send()
.await
.context("send GET request to BOS")?;
if !response.status().is_success() {
return Err(anyhow!("BOS returned HTTP error status: {}", response.status()));
}
let mut file = tokio::fs::File::create(temp_path)
.await
.context("create temp file")?;
let mut stream = response.bytes_stream();
let mut downloaded_bytes = 0u64;
while let Some(chunk_result) = stream.next().await {
let chunk = chunk_result.context("read chunk from BOS stream")?;
file.write_all(&chunk).await.context("write chunk to disk")?;
let len = chunk.len() as u64;
downloaded_bytes += len;
total_bar.inc(len);
}
file.flush().await.context("flush temp file")?;
if expected_size > 0 && downloaded_bytes != expected_size {
return Err(anyhow!(
"Incomplete download: expected {} bytes but got {}",
expected_size,
downloaded_bytes
));
}
Ok(())
}
async fn resolve_manifest(client: &reqwest::Client, args: &Cli) -> Result<ManifestResponse> {
if let Some(file_path) = &args.manifest_file {
println!("📖 Reading manifest from local file: {}", file_path.display());
let content = tokio::fs::read_to_string(file_path)
.await
.context("read manifest file")?;
let manifest: ManifestResponse = serde_json::from_str(&content)
.context("parse manifest JSON")?;
return Ok(manifest);
}
if let Some(manifest_url) = &args.manifest_url {
println!("🌐 Fetching manifest from URL: {}", manifest_url);
let mut request = client.get(manifest_url);
if let Some(cookie) = &args.cookie {
request = request.header("Cookie", cookie);
}
let response = request.send().await.context("fetch manifest URL")?;
if !response.status().is_success() {
let status = response.status();
let text = response.text().await.unwrap_or_default();
return Err(anyhow!("Manifest request failed with status {}: {}", status, text));
}
let manifest: ManifestResponse = response.json().await.context("decode manifest JSON")?;
return Ok(manifest);
}
let api_base = args
.api
.as_deref()
.ok_or_else(|| anyhow!("Please specify --api or --manifest-url or --manifest-file"))?;
let api_base = api_base.trim_end_matches('/');
let prefix = args
.prefix
.as_deref()
.ok_or_else(|| anyhow!("Please specify --prefix (e.g. ubbdata/2026_09_09-09_21_24)"))?;
let mut session_cookie = args.cookie.clone();
// If email and password provided, perform login
if let (Some(email), Some(password)) = (&args.email, &args.password) {
println!("🔑 Authenticating as {} on {}...", email, api_base);
let login_payload = serde_json::json!({
"email": email,
"password": password,
});
let login_resp = client
.post(format!("{}/api/auth/login", api_base))
.json(&login_payload)
.send()
.await
.context("send login request")?;
if !login_resp.status().is_success() {
let text = login_resp.text().await.unwrap_or_default();
return Err(anyhow!("Login failed: {}", text));
}
// Extract cookie if returned
if let Some(cookie_header) = login_resp.headers().get("set-cookie") {
if let Ok(cookie_str) = cookie_header.to_str() {
let first_cookie = cookie_str.split(';').next().unwrap_or(cookie_str);
session_cookie = Some(first_cookie.to_string());
}
}
println!("🔓 Authentication successful!");
}
// Determine connection ID
let connection_id = match &args.connection {
Some(id) => id.clone(),
None => {
println!("🔍 Querying available storage connections...");
let mut req = client.get(format!("{}/api/object-storage", api_base));
if let Some(cookie) = &session_cookie {
req = req.header("Cookie", cookie);
}
let resp = req.send().await.context("query storage connections")?;
if !resp.status().is_success() {
return Err(anyhow!("Failed to query storage connections: HTTP {}", resp.status()));
}
let connections: Vec<StorageConnectionSummary> = resp.json().await.context("parse connections")?;
if connections.is_empty() {
return Err(anyhow!("No storage connections found in this organization"));
}
let chosen = &connections[0];
println!("💡 Using default storage connection: {} ({})", chosen.name, chosen.id);
chosen.id.clone()
}
};
let mut manifest_url = format!(
"{}/api/object-storage/directory/manifest?connectionId={}&prefix={}",
api_base,
urlencoding::encode(&connection_id),
urlencoding::encode(prefix)
);
if let Some(bucket) = &args.bucket {
manifest_url.push_str(&format!("&bucket={}", urlencoding::encode(bucket)));
}
println!("📋 Requesting directory manifest for '{}'...", prefix);
let mut req = client.get(&manifest_url);
if let Some(cookie) = &session_cookie {
req = req.header("Cookie", cookie);
}
let resp = req.send().await.context("request directory manifest")?;
if !resp.status().is_success() {
let status = resp.status();
let text = resp.text().await.unwrap_or_default();
return Err(anyhow!("Directory manifest request failed ({}): {}", status, text));
}
let manifest: ManifestResponse = resp.json().await.context("parse manifest response")?;
Ok(manifest)
}
fn format_bytes(bytes: u64) -> String {
const KB: u64 = 1024;
const MB: u64 = KB * 1024;
const GB: u64 = MB * 1024;
if bytes >= GB {
format!("{:.2} GB", bytes as f64 / GB as f64)
} else if bytes >= MB {
format!("{:.2} MB", bytes as f64 / MB as f64)
} else if bytes >= KB {
format!("{:.2} KB", bytes as f64 / KB as f64)
} else {
format!("{} B", bytes)
}
}

View File

@ -1194,7 +1194,7 @@ checksum = "d8b14ccef22fc6f5a8f4d7d768562a182c04ce9a3b3157b91390b52ddfdf1a76"
[[package]]
name = "drosync-cli"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"anyhow",
"clap",
@ -1210,7 +1210,7 @@ dependencies = [
[[package]]
name = "drosync-core"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"chrono",
"drosync-protocol",
@ -1226,7 +1226,7 @@ dependencies = [
[[package]]
name = "drosync-daemon"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"anyhow",
"drosync-core",
@ -1247,7 +1247,7 @@ dependencies = [
[[package]]
name = "drosync-gui"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"drosync-daemon",
"drosync-platform",
@ -1259,7 +1259,7 @@ dependencies = [
[[package]]
name = "drosync-platform"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"directories",
"keyring",
@ -1271,7 +1271,7 @@ dependencies = [
[[package]]
name = "drosync-protocol"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"chrono",
"serde",
@ -1280,7 +1280,7 @@ dependencies = [
[[package]]
name = "drosync-state"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"drosync-core",
"drosync-protocol",
@ -1293,7 +1293,7 @@ dependencies = [
[[package]]
name = "drosync-transfer"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"base64",
"drosync-protocol",
@ -1309,7 +1309,7 @@ dependencies = [
[[package]]
name = "drosync-updater"
version = "1.0.0"
version = "1.0.19"
dependencies = [
"base64",
"drosync-protocol",

View File

@ -13,7 +13,7 @@ members = [
]
[workspace.package]
version = "1.0.0"
version = "1.0.19"
edition = "2024"
license = "Apache-2.0"
rust-version = "1.89"

View File

@ -79,8 +79,10 @@ enum MappingSubcommand {
List,
Add {
local_root: PathBuf,
#[arg(long)]
space: String,
#[arg(long, required_unless_present = "bucket", conflicts_with = "bucket")]
space: Option<String>,
#[arg(long, required_unless_present = "space", conflicts_with = "space")]
bucket: Option<String>,
#[arg(long)]
remote_prefix: Option<String>,
#[arg(long)]
@ -291,6 +293,7 @@ async fn execute(app: &App) -> Result<Value> {
MappingSubcommand::Add {
local_root,
space,
bucket,
remote_prefix,
include_hidden,
require_done,
@ -298,7 +301,13 @@ async fn execute(app: &App) -> Result<Value> {
let canonical = local_root
.canonicalize()
.with_context(|| format!("invalid mapping root {}", local_root.display()))?;
let mut mapping = Mapping::new(canonical, space.clone());
let mut mapping = match (space.as_ref(), bucket.as_ref()) {
(Some(space), None) => Mapping::new(canonical, space.trim().to_owned()),
(None, Some(bucket)) => {
Mapping::for_bucket(canonical, bucket.trim().to_owned())
}
_ => return Err(anyhow!("exactly one of --space or --bucket is required")),
};
mapping.remote_prefix = remote_prefix.clone().unwrap_or_default();
mapping.include_hidden = *include_hidden;
mapping.require_done_marker = *require_done;
@ -354,6 +363,7 @@ async fn execute(app: &App) -> Result<Value> {
"protocolVersion": PROTOCOL_VERSION,
"platform": drosync_platform::platform_name(),
"architecture": drosync_platform::architecture(),
"releasePlatform": drosync_platform::release_platform_key(),
})),
}
}

View File

@ -40,6 +40,8 @@ pub struct Mapping {
pub local_root: PathBuf,
pub sync_space_id: String,
#[serde(default)]
pub bucket: String,
#[serde(default)]
pub remote_prefix: String,
#[serde(default)]
pub include_hidden: bool,
@ -54,11 +56,19 @@ impl Mapping {
id: Uuid::new_v4().to_string(),
local_root,
sync_space_id,
bucket: String::new(),
remote_prefix: String::new(),
include_hidden: false,
require_done_marker: false,
}
}
#[must_use]
pub fn for_bucket(local_root: PathBuf, bucket: String) -> Self {
let mut mapping = Self::new(local_root, String::new());
mapping.bucket = bucket;
mapping
}
}
#[derive(Debug, Clone)]
@ -369,4 +379,12 @@ mod tests {
"nested/good.bin"
);
}
#[test]
fn bucket_mapping_defers_space_resolution_to_the_authenticated_daemon() {
let mapping =
Mapping::for_bucket(Path::new("/mapping").to_path_buf(), "bucket-a".to_owned());
assert_eq!(mapping.bucket, "bucket-a");
assert!(mapping.sync_space_id.is_empty());
}
}

View File

@ -126,6 +126,43 @@ fn apply_remote_prefix(mapping: &Mapping, manifest: &mut ManifestItem) {
}
}
fn resolve_bucket_space(capabilities: &Value, bucket: &str) -> Result<(String, String)> {
let requested = bucket.trim();
if requested.is_empty() {
return Err(anyhow!("bucket must not be empty"));
}
let spaces = capabilities
.get("spaces")
.and_then(Value::as_array)
.ok_or_else(|| anyhow!("control plane did not return scoped sync spaces"))?;
let matches = spaces
.iter()
.filter(|space| space.get("bucket").and_then(Value::as_str) == Some(requested))
.filter_map(|space| {
let id = space.get("id")?.as_str()?.trim();
(!id.is_empty()).then(|| {
(
id.to_owned(),
space
.get("rootPrefix")
.and_then(Value::as_str)
.unwrap_or_default()
.to_owned(),
)
})
})
.collect::<Vec<_>>();
match matches.as_slice() {
[item] => Ok(item.clone()),
[] => Err(anyhow!(
"bucket {requested} is not available to this credential; ask an administrator to create or grant a sync space"
)),
_ => Err(anyhow!(
"bucket {requested} maps to multiple sync spaces; use --space with the intended space id"
)),
}
}
fn normalize_journal(mut journal: SyncJournal) -> SyncJournal {
if journal.file_count == 0 && !journal.files.is_empty() {
journal.file_count = u64::try_from(journal.files.len()).unwrap_or(u64::MAX);
@ -813,12 +850,31 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
.mappings()
.map(|items| serde_json::to_value(items).expect("serializable mappings"))
.map_err(Into::into),
"mapping.add" => serde_json::from_value::<Mapping>(request.arguments)
.map_err(Into::into)
.and_then(|mapping| {
state.put_mapping(&mapping)?;
Ok(serde_json::to_value(mapping).expect("serializable mapping"))
}),
"mapping.add" => match serde_json::from_value::<Mapping>(request.arguments) {
Ok(mut mapping) => {
let resolved = if mapping.sync_space_id.trim().is_empty() {
match load_transfer_client(state) {
Ok(client) => match client.capabilities().await {
Ok(capabilities) => {
resolve_bucket_space(&capabilities, &mapping.bucket)
}
Err(error) => Err(error.into()),
},
Err(error) => Err(error),
}
} else {
Ok((mapping.sync_space_id.clone(), String::new()))
};
resolved.and_then(|(space_id, space_root_prefix)| {
mapping.sync_space_id = space_id;
state.put_mapping(&mapping)?;
let mut value = serde_json::to_value(mapping).expect("serializable mapping");
value["spaceRootPrefix"] = Value::String(space_root_prefix);
Ok(value)
})
}
Err(error) => Err(error.into()),
},
"mapping.remove" => {
let id = request
.arguments
@ -826,8 +882,17 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("id is required"));
id.and_then(|value| {
// Mark any non-terminal sessions for this mapping as aborted
// before removing the mapping itself. This prevents the
// scheduler from picking up stale "active" sessions that would
// otherwise keep spawning new sync attempts.
let aborted_count = state.mark_sessions_aborted_for_mapping(value)?;
let removed = state.remove_mapping(value)?;
Ok(serde_json::json!({"removed": removed}))
// Clear pending scan journal so a restart won't resume work
// for a mapping that no longer exists.
let pending_key = format!("pending_journal:{value}");
let _ = state.set_meta(&pending_key, "");
Ok(serde_json::json!({"removed": removed, "abortedSessions": aborted_count}))
})
}
"plan" | "sync" => {
@ -948,6 +1013,9 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
"name": "drosync-daemon",
"version": env!("CARGO_PKG_VERSION"),
"protocolVersion": drosync_protocol_version(),
"platform": drosync_platform::platform_name(),
"architecture": drosync_platform::architecture(),
"releasePlatform": drosync_platform::release_platform_key(),
})),
_ => Err(anyhow!("unknown daemon command {}", request.command)),
};
@ -1006,14 +1074,25 @@ async fn handle_update(state: &mut StateStore, arguments: &Value) -> Result<Valu
"offeredVersion": release.version,
}));
}
let encoded_key = option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64")
.ok_or_else(|| anyhow!("client was built without DROSYNC_UPDATE_PUBLIC_KEY_BASE64"))?;
let expected_platform_key = drosync_platform::release_platform_key()
.ok_or_else(|| anyhow!("this client target has no supported signed-update platform key"))?;
let key_environment = drosync_platform::update_public_key_environment(expected_platform_key)
.unwrap_or("DROSYNC_UPDATE_PUBLIC_KEY_BASE64");
let encoded_key = drosync_platform::embedded_update_public_key().ok_or_else(|| {
anyhow!("client was built without {key_environment} or DROSYNC_UPDATE_PUBLIC_KEY_BASE64")
})?;
let public_key = drosync_updater::decode_public_key(encoded_key)?;
drosync_updater::verify_release(&release, &public_key)?;
let platform_key = response
.get("platformKey")
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("update response omitted platformKey"))?;
if platform_key != expected_platform_key {
return Err(anyhow!(
"update response platformKey {platform_key} does not match this client target {expected_platform_key}"
));
}
drosync_updater::validate_release_platform(&release, platform_key)?;
let url = release
.artifacts
.get(platform_key)
@ -1073,7 +1152,9 @@ async fn handle_update(state: &mut StateStore, arguments: &Value) -> Result<Valu
async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphore>) -> Result<()> {
let (event_sender, mut event_receiver) = tokio::sync::mpsc::unbounded_channel();
let mut watcher = notify::recommended_watcher(move |event: notify::Result<notify::Event>| {
if let Ok(event) = event {
if let Ok(event) = event
&& event_triggers_sync(event.kind)
{
let _ = event_sender.send(event.paths);
}
})?;
@ -1093,8 +1174,19 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
let current = state.mappings()?.into_iter().map(|mapping| (mapping.id.clone(), mapping)).collect::<BTreeMap<_, _>>();
for (id, mapping) in &current {
if !mappings.contains_key(id) {
watcher.watch(&mapping.local_root, RecursiveMode::Recursive)?;
dirty.insert(id.clone(), std::time::Instant::now());
match watcher.watch(&mapping.local_root, RecursiveMode::Recursive) {
Ok(()) => {
dirty.insert(id.clone(), std::time::Instant::now());
}
Err(error) => {
tracing::error!(
mapping_id = %id,
local_root = %mapping.local_root.display(),
error = %error,
"failed to watch mapping; scheduler will retry on refresh"
);
}
}
}
}
for (id, mapping) in &mappings {
@ -1120,12 +1212,28 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
.map(|(id, _)| id.clone())
.collect::<Vec<_>>();
for id in ready {
let Some(mapping) = mappings.get(&id).cloned() else {
if !mappings.contains_key(&id) {
dirty.remove(&id);
continue;
}
let _permit = transfer_gate.acquire().await?;
let mut state = StateStore::open(&state_db)?;
// Re-check the mapping against the database before syncing.
// The in-memory cache is only refreshed every 60 s, so a
// `mapping.remove` that arrived in the meantime would leave
// a stale entry that keeps spawning sessions.
let Some(mapping) = state.mapping(&id)? else {
// Mapping was removed — tear down the watcher and
// purge it from the in-memory caches immediately
// instead of waiting for the next refresh tick.
if let Some(old) = mappings.remove(&id) {
let _ = watcher.unwatch(&old.local_root);
}
dirty.remove(&id);
continue;
};
let _permit = transfer_gate.acquire().await?;
let mut state = StateStore::open(&state_db)?;
// Keep in-memory cache in sync with the database.
mappings.insert(id.clone(), mapping.clone());
match sync_mapping(&mut state, &mapping, false).await {
Ok(value) if value.get("paused").and_then(Value::as_bool) == Some(true) => {
dirty.insert(id, std::time::Instant::now());
@ -1151,6 +1259,24 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
}
}
fn event_triggers_sync(kind: notify::EventKind) -> bool {
matches!(
kind,
notify::EventKind::Create(_) | notify::EventKind::Modify(_) | notify::EventKind::Remove(_)
)
}
fn spawn_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphore>) {
tokio::spawn(async move {
loop {
if let Err(error) = run_scheduler(state_db.clone(), transfer_gate.clone()).await {
tracing::error!(error = %error, "drosync scheduler stopped; restarting");
tokio::time::sleep(Duration::from_secs(5)).await;
}
}
});
}
const fn drosync_protocol_version() -> &'static str {
"v1"
}
@ -1172,7 +1298,7 @@ pub async fn run(paths: &Paths) -> Result<()> {
let state = StateStore::open(&paths.state_db)?;
state.set_meta("data_dir", &paths.data_dir.to_string_lossy())?;
let transfer_gate = std::sync::Arc::new(Semaphore::new(1));
tokio::spawn(run_scheduler(paths.state_db.clone(), transfer_gate.clone()));
spawn_scheduler(paths.state_db.clone(), transfer_gate.clone());
loop {
let (stream, _) = listener.accept().await?;
let state_db = paths.state_db.clone();
@ -1230,7 +1356,7 @@ async fn run_windows_pipe(paths: &Paths, pipe_name: &str) -> Result<()> {
let state = StateStore::open(&paths.state_db)?;
state.set_meta("data_dir", &paths.data_dir.to_string_lossy())?;
let transfer_gate = std::sync::Arc::new(Semaphore::new(1));
tokio::spawn(run_scheduler(paths.state_db.clone(), transfer_gate.clone()));
spawn_scheduler(paths.state_db.clone(), transfer_gate.clone());
let mut first_instance = true;
loop {
let mut options = ServerOptions::new();
@ -1341,7 +1467,12 @@ pub fn default_log_path(data_dir: &Path) -> PathBuf {
#[cfg(test)]
mod tests {
use super::{MULTIPART_MAX_PART, multipart_part_size};
use super::{
MULTIPART_MAX_PART, event_triggers_sync, multipart_part_size, resolve_bucket_space,
};
use notify::EventKind;
use notify::event::{AccessKind, CreateKind, ModifyKind, RemoveKind};
use serde_json::json;
#[test]
fn one_tib_object_fits_bos_multipart_limits() {
@ -1363,4 +1494,36 @@ mod tests {
assert!(multipart_part_size(size) <= MULTIPART_MAX_PART);
}
}
#[test]
fn bucket_resolves_only_when_scoped_space_is_unique() {
let capabilities = json!({"spaces": [
{"id": "space-a", "bucket": "bucket-a", "rootPrefix": "tenant/ubt"},
{"id": "space-b", "bucket": "bucket-b", "rootPrefix": ""}
]});
assert_eq!(
resolve_bucket_space(&capabilities, " bucket-a ").unwrap(),
("space-a".to_owned(), "tenant/ubt".to_owned())
);
}
#[test]
fn bucket_resolution_fails_closed_for_missing_or_ambiguous_space() {
let capabilities = json!({"spaces": [
{"id": "space-a", "bucket": "bucket-a"},
{"id": "space-b", "bucket": "bucket-a"}
]});
assert!(resolve_bucket_space(&capabilities, "bucket-a").is_err());
assert!(resolve_bucket_space(&capabilities, "bucket-missing").is_err());
assert!(resolve_bucket_space(&capabilities, " ").is_err());
}
#[test]
fn watcher_ignores_reads_but_tracks_content_changes() {
assert!(!event_triggers_sync(EventKind::Access(AccessKind::Any)));
assert!(!event_triggers_sync(EventKind::Other));
assert!(event_triggers_sync(EventKind::Create(CreateKind::Any)));
assert!(event_triggers_sync(EventKind::Modify(ModifyKind::Any)));
assert!(event_triggers_sync(EventKind::Remove(RemoveKind::Any)));
}
}

View File

@ -7,11 +7,16 @@ rust-version.workspace = true
[dependencies]
directories.workspace = true
keyring.workspace = true
serde.workspace = true
thiserror.workspace = true
uuid.workspace = true
# RDK S100 is a headless Linux ARM64 target. Its systemd service receives a
# token through a restricted environment file or `DROSYNC_SECRET_FILE`, rather
# than pulling in a desktop D-Bus keyring implementation.
[target.'cfg(not(all(target_os = "linux", target_arch = "aarch64")))'.dependencies]
keyring.workspace = true
[dev-dependencies]
tempfile.workspace = true

View File

@ -3,7 +3,7 @@
use directories::ProjectDirs;
use serde::{Deserialize, Serialize};
use std::fs;
use std::io;
use std::io::{self, Write};
use std::path::{Path, PathBuf};
use thiserror::Error;
use uuid::Uuid;
@ -12,12 +12,19 @@ use uuid::Uuid;
pub enum PlatformError {
#[error("operating-system project directories are unavailable")]
DirectoriesUnavailable,
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
#[error("credential storage failed: {0}")]
Credential(#[from] keyring::Error),
#[error("DROSYNC_SECRET_FILE is required for headless Linux ARM64 credentials")]
HeadlessSecretFileRequired,
#[error("I/O error: {0}")]
Io(#[from] io::Error),
#[error("secret file permissions must be 0600")]
InsecureSecretPermissions,
#[error("secret file must be a regular file, not a symlink or special file")]
InvalidSecretFile,
#[error("secret-file parent directory must not be group- or world-writable")]
InsecureSecretDirectoryPermissions,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@ -42,6 +49,97 @@ pub fn architecture() -> &'static str {
}
}
/// Return the release-manifest key for an operating-system/architecture pair.
///
/// The control plane uses this value as the key in a signed client release
/// manifest. Keep it deliberately narrower than `std::env::consts`: an
/// unsupported target must not accidentally select a package built for a
/// different ABI.
#[must_use]
pub fn release_platform_key_for(
operating_system: &str,
target_architecture: &str,
) -> Option<&'static str> {
match (operating_system, target_architecture) {
("linux", "x86_64") => Some("linux-x86_64"),
("linux", "aarch64" | "arm64") => Some("linux-arm64"),
("macos", "aarch64" | "arm64") => Some("macos-arm64"),
("windows", "x86_64") => Some("windows-x86_64"),
_ => None,
}
}
/// Return the release-manifest key compiled into this client.
///
/// This is separate from the two device-registration fields (`platform` and
/// `architecture`), which remain wire-compatible with existing control-plane
/// versions and are combined there into the same key.
#[must_use]
pub fn release_platform_key() -> Option<&'static str> {
release_platform_key_for(std::env::consts::OS, std::env::consts::ARCH)
}
/// Return the environment variable that may pin a public update key for a
/// release platform. A generic `DROSYNC_UPDATE_PUBLIC_KEY_BASE64` remains a
/// backwards-compatible fallback.
#[must_use]
pub fn update_public_key_environment(platform_key: &str) -> Option<&'static str> {
match platform_key {
"linux-arm64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64"),
"linux-x86_64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64"),
"macos-arm64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64"),
"windows-x86_64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64"),
_ => None,
}
}
/// Choose the target-specific update key when one was embedded, otherwise use
/// the legacy generic key. This pure helper keeps the trust-key mapping
/// testable without exposing compile-time configuration as a runtime setting.
#[must_use]
pub fn select_update_public_key(
target_specific: Option<&'static str>,
legacy: Option<&'static str>,
) -> Option<&'static str> {
match target_specific {
Some(value) if !value.is_empty() => Some(value),
_ => legacy,
}
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64");
#[cfg(all(target_os = "linux", target_arch = "x86_64"))]
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64");
#[cfg(all(target_os = "macos", target_arch = "aarch64"))]
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64");
#[cfg(all(target_os = "windows", target_arch = "x86_64"))]
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64");
#[cfg(not(any(
all(target_os = "linux", target_arch = "aarch64"),
all(target_os = "linux", target_arch = "x86_64"),
all(target_os = "macos", target_arch = "aarch64"),
all(target_os = "windows", target_arch = "x86_64"),
)))]
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> = None;
/// Return the Ed25519 public key embedded for this exact release target.
///
/// Build scripts populate the target-specific environment variable and also
/// preserve the legacy generic variable so previously configured CI continues
/// to produce compatible clients.
#[must_use]
pub fn embedded_update_public_key() -> Option<&'static str> {
select_update_public_key(
TARGET_UPDATE_PUBLIC_KEY,
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64"),
)
}
pub fn paths() -> Result<Paths, PlatformError> {
let dirs = ProjectDirs::from("com", "drobotics", "drosync")
.ok_or(PlatformError::DirectoriesUnavailable)?;
@ -57,15 +155,58 @@ pub fn paths() -> Result<Paths, PlatformError> {
})
}
/// Parse an explicitly configured secret-file path without reading process
/// state. Keeping this small helper pure lets all target test suites exercise
/// the headless credential contract.
pub fn headless_secret_file_path(value: Option<&str>) -> Result<PathBuf, PlatformError> {
value
.map(str::trim)
.filter(|value| !value.is_empty())
.map(PathBuf::from)
.ok_or(PlatformError::HeadlessSecretFileRequired)
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
fn configured_headless_secret_file() -> Result<PathBuf, PlatformError> {
let value = std::env::var("DROSYNC_SECRET_FILE").ok();
headless_secret_file_path(value.as_deref())
}
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
fn configured_runtime_secret_file() -> Option<PathBuf> {
std::env::var("DROSYNC_SECRET_FILE").ok().and_then(|value| {
let trimmed = value.trim();
(!trimmed.is_empty()).then(|| PathBuf::from(trimmed))
})
}
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
pub fn save_credential(profile: &str, token: &str) -> Result<(), PlatformError> {
if let Some(path) = configured_runtime_secret_file() {
return write_headless_secret(&path, token);
}
keyring::Entry::new("drosync", profile)?.set_password(token)?;
Ok(())
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
pub fn save_credential(_profile: &str, token: &str) -> Result<(), PlatformError> {
write_headless_secret(&configured_headless_secret_file()?, token)
}
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
pub fn load_credential(profile: &str) -> Result<String, PlatformError> {
if let Some(path) = configured_runtime_secret_file() {
return read_headless_secret(&path);
}
Ok(keyring::Entry::new("drosync", profile)?.get_password()?)
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
pub fn load_credential(_profile: &str) -> Result<String, PlatformError> {
read_headless_secret(&configured_headless_secret_file()?)
}
pub fn load_runtime_credential(profile: &str) -> Result<String, PlatformError> {
if let Ok(value) = std::env::var("DROSYNC_TOKEN")
&& !value.trim().is_empty()
@ -80,11 +221,20 @@ pub fn load_runtime_credential(profile: &str) -> Result<String, PlatformError> {
load_credential(profile)
}
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
pub fn delete_credential(profile: &str) -> Result<(), PlatformError> {
if let Some(path) = configured_runtime_secret_file() {
return delete_headless_secret(&path);
}
keyring::Entry::new("drosync", profile)?.delete_credential()?;
Ok(())
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
pub fn delete_credential(_profile: &str) -> Result<(), PlatformError> {
delete_headless_secret(&configured_headless_secret_file()?)
}
pub fn load_or_create_device_id(profile: &str) -> Result<String, PlatformError> {
let paths = paths()?;
let safe_profile = profile
@ -115,14 +265,98 @@ pub fn load_or_create_device_id(profile: &str) -> Result<String, PlatformError>
pub fn read_headless_secret(path: &Path) -> Result<String, PlatformError> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
if fs::metadata(path)?.permissions().mode() & 0o077 != 0 {
return Err(PlatformError::InsecureSecretPermissions);
}
ensure_secure_secret_file(path)?;
}
Ok(fs::read_to_string(path)?.trim().to_owned())
}
#[cfg(unix)]
fn ensure_secure_secret_file(path: &Path) -> Result<(), PlatformError> {
use std::os::unix::fs::PermissionsExt;
let metadata = fs::symlink_metadata(path)?;
if !metadata.file_type().is_file() {
return Err(PlatformError::InvalidSecretFile);
}
if metadata.permissions().mode() & 0o777 != 0o600 {
return Err(PlatformError::InsecureSecretPermissions);
}
Ok(())
}
#[cfg(unix)]
fn ensure_secure_secret_parent(path: &Path) -> Result<&Path, PlatformError> {
use std::os::unix::fs::PermissionsExt;
let parent = path.parent().ok_or_else(|| {
io::Error::new(
io::ErrorKind::InvalidInput,
"secret file must have a parent directory",
)
})?;
let metadata = fs::metadata(parent)?;
if !metadata.is_dir() {
return Err(PlatformError::Io(io::Error::new(
io::ErrorKind::NotADirectory,
"secret-file parent is not a directory",
)));
}
if metadata.permissions().mode() & 0o022 != 0 {
return Err(PlatformError::InsecureSecretDirectoryPermissions);
}
Ok(parent)
}
/// Write a headless credential by atomically replacing a 0600 file in an
/// already-secure directory. The temporary file lives beside the destination,
/// so the final rename is atomic on the same filesystem.
#[cfg(unix)]
pub fn write_headless_secret(path: &Path, token: &str) -> Result<(), PlatformError> {
use std::fs::OpenOptions;
use std::os::unix::fs::OpenOptionsExt;
let parent = ensure_secure_secret_parent(path)?;
match fs::symlink_metadata(path) {
Ok(metadata) => {
if !metadata.file_type().is_file() {
return Err(PlatformError::InvalidSecretFile);
}
}
Err(error) if error.kind() == io::ErrorKind::NotFound => {}
Err(error) => return Err(error.into()),
}
let temporary = parent.join(format!(".drosync-credential-{}.tmp", Uuid::new_v4()));
let write_result = (|| -> Result<(), PlatformError> {
let mut file = OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&temporary)?;
file.write_all(token.trim().as_bytes())?;
file.write_all(b"\n")?;
file.sync_all()?;
drop(file);
fs::rename(&temporary, path)?;
secure_runtime_file(path)?;
fs::File::open(parent)?.sync_all()?;
Ok(())
})();
if write_result.is_err() {
let _ = fs::remove_file(&temporary);
}
write_result
}
#[cfg(unix)]
pub fn delete_headless_secret(path: &Path) -> Result<(), PlatformError> {
ensure_secure_secret_file(path)?;
fs::remove_file(path)?;
if let Some(parent) = path.parent() {
fs::File::open(parent)?.sync_all()?;
}
Ok(())
}
#[cfg(unix)]
pub fn secure_runtime_file(path: &Path) -> Result<(), PlatformError> {
use std::os::unix::fs::PermissionsExt;
@ -139,12 +373,18 @@ pub fn secure_runtime_file(_path: &Path) -> Result<(), PlatformError> {
#[cfg(all(test, unix))]
mod tests {
use super::{PlatformError, read_headless_secret, secure_runtime_file};
use super::{
PlatformError, delete_headless_secret, headless_secret_file_path, read_headless_secret,
release_platform_key_for, secure_runtime_file, select_update_public_key,
update_public_key_environment, write_headless_secret,
};
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
use keyring::credential::CredentialPersistence;
use std::fs;
use std::os::unix::fs::PermissionsExt;
use std::os::unix::net::UnixListener;
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
#[test]
fn operating_system_credential_store_is_persistent() {
assert!(matches!(
@ -163,9 +403,91 @@ mod tests {
read_headless_secret(&secret),
Err(PlatformError::InsecureSecretPermissions)
));
assert!(matches!(
delete_headless_secret(&secret),
Err(PlatformError::InsecureSecretPermissions)
));
fs::set_permissions(&secret, fs::Permissions::from_mode(0o600)).unwrap();
assert_eq!(read_headless_secret(&secret).unwrap(), "device-token");
fs::set_permissions(&secret, fs::Permissions::from_mode(0o400)).unwrap();
assert!(matches!(
read_headless_secret(&secret),
Err(PlatformError::InsecureSecretPermissions)
));
}
#[test]
fn headless_secret_path_requires_an_explicit_nonempty_value() {
assert!(matches!(
headless_secret_file_path(None),
Err(PlatformError::HeadlessSecretFileRequired)
));
assert!(matches!(
headless_secret_file_path(Some(" ")),
Err(PlatformError::HeadlessSecretFileRequired)
));
assert_eq!(
headless_secret_file_path(Some(" /var/lib/drosync/token ")).unwrap(),
std::path::PathBuf::from("/var/lib/drosync/token")
);
}
#[test]
fn headless_secret_atomic_write_read_and_delete_round_trip() {
let directory = tempfile::tempdir().unwrap();
fs::set_permissions(directory.path(), fs::Permissions::from_mode(0o700)).unwrap();
let secret = directory.path().join("token");
write_headless_secret(&secret, "first-token\n").unwrap();
assert_eq!(read_headless_secret(&secret).unwrap(), "first-token");
assert_eq!(
fs::metadata(&secret).unwrap().permissions().mode() & 0o777,
0o600
);
write_headless_secret(&secret, "second-token").unwrap();
assert_eq!(read_headless_secret(&secret).unwrap(), "second-token");
delete_headless_secret(&secret).unwrap();
assert!(!secret.exists());
}
#[test]
fn headless_secret_rejects_an_unsafe_parent_directory() {
let directory = tempfile::tempdir().unwrap();
fs::set_permissions(directory.path(), fs::Permissions::from_mode(0o777)).unwrap();
let secret = directory.path().join("token");
assert!(matches!(
write_headless_secret(&secret, "device-token"),
Err(PlatformError::InsecureSecretDirectoryPermissions)
));
}
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
#[test]
fn arm64_credential_store_round_trips_through_configured_secret_file() {
use super::{delete_credential, load_credential, save_credential};
let configured = std::env::var("DROSYNC_SECRET_FILE")
.expect("target test runner must provide DROSYNC_SECRET_FILE");
let secret = std::path::PathBuf::from(configured);
assert!(
!secret.exists(),
"target test secret file must start absent"
);
save_credential("default", "arm64-device-token").expect("atomically persist token");
assert_eq!(
load_credential("default").expect("load persisted token"),
"arm64-device-token"
);
assert_eq!(
fs::metadata(&secret).unwrap().permissions().mode() & 0o777,
0o600
);
delete_credential("default").expect("securely remove token");
assert!(!secret.exists());
}
#[test]
@ -179,4 +501,60 @@ mod tests {
0o600
);
}
#[test]
fn release_platform_keys_cover_supported_client_targets() {
assert_eq!(
release_platform_key_for("linux", "x86_64"),
Some("linux-x86_64")
);
assert_eq!(
release_platform_key_for("linux", "aarch64"),
Some("linux-arm64")
);
assert_eq!(
release_platform_key_for("linux", "arm64"),
Some("linux-arm64")
);
assert_eq!(
release_platform_key_for("macos", "aarch64"),
Some("macos-arm64")
);
assert_eq!(
release_platform_key_for("windows", "x86_64"),
Some("windows-x86_64")
);
assert_eq!(release_platform_key_for("linux", "i686"), None);
assert_eq!(release_platform_key_for("windows", "aarch64"), None);
}
#[test]
fn update_key_environment_is_platform_specific_with_legacy_fallback() {
assert_eq!(
update_public_key_environment("linux-arm64"),
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64")
);
assert_eq!(
update_public_key_environment("linux-x86_64"),
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64")
);
assert_eq!(
update_public_key_environment("macos-arm64"),
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64")
);
assert_eq!(
update_public_key_environment("windows-x86_64"),
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64")
);
assert_eq!(update_public_key_environment("linux-riscv64"), None);
assert_eq!(
select_update_public_key(Some("target-key"), Some("legacy-key")),
Some("target-key")
);
assert_eq!(
select_update_public_key(None, Some("legacy-key")),
Some("legacy-key")
);
assert_eq!(select_update_public_key(Some(""), None), None);
}
}

View File

@ -340,6 +340,20 @@ impl StateStore {
.collect()
}
/// Marks all non-terminal sessions for the given mapping as `"aborted"`.
/// Returns the number of sessions updated.
pub fn mark_sessions_aborted_for_mapping(
&self,
mapping_id: &str,
) -> Result<usize, StateError> {
Ok(self.connection.execute(
"UPDATE sessions SET status = 'aborted', updated_at = CURRENT_TIMESTAMP
WHERE mapping_id = ?1
AND status NOT IN ('completed', 'partial', 'conflicted', 'failed', 'aborted', 'expired')",
[mapping_id],
)?)
}
pub fn put_part(&self, part: &StoredPart) -> Result<(), StateError> {
self.connection.execute(
"INSERT INTO transfer_parts(item_id, part_number, etag, size_bytes)

View File

@ -2,6 +2,7 @@
use base64::Engine;
use drosync_protocol::ClientRelease;
use drosync_updater::validate_signed_release_matrix;
use ed25519_dalek::{Signer, SigningKey};
use std::error::Error;
use std::fs;
@ -23,38 +24,13 @@ fn signing_key() -> Result<SigningKey, Box<dyn Error>> {
}
}
fn validate(release: &ClientRelease) -> Result<(), Box<dyn Error>> {
if release.kind != "signed" {
return Err("automatic update manifests must use kind=signed".into());
}
let expected_percent = match release.channel.as_str() {
"canary" => 5,
"pilot" => 20,
"stable" => 100,
_ => return Err("channel must be canary, pilot or stable".into()),
};
if !release.paused && release.rollout_percent != expected_percent {
return Err("active rollout percentage does not match its channel".into());
}
for platform in ["windows-x86_64", "linux-x86_64", "macos-arm64"] {
if !release.artifacts.contains_key(platform) || !release.sha256.contains_key(platform) {
return Err(format!("manifest is missing the {platform} artifact or digest").into());
}
let digest = &release.sha256[platform];
if digest.len() != 64 || !digest.bytes().all(|value| value.is_ascii_hexdigit()) {
return Err(format!("invalid SHA-256 for {platform}").into());
}
}
Ok(())
}
fn main() -> Result<(), Box<dyn Error>> {
let manifest_path = std::env::args_os()
.nth(1)
.map(PathBuf::from)
.ok_or("usage: drosync-sign-manifest MANIFEST.json")?;
let mut release: ClientRelease = serde_json::from_slice(&fs::read(&manifest_path)?)?;
validate(&release)?;
validate_signed_release_matrix(&release)?;
release.signature.clear();
let signature = signing_key()?.sign(&serde_json::to_vec(&release)?);
release.signature = base64::engine::general_purpose::STANDARD.encode(signature.to_bytes());

View File

@ -7,6 +7,15 @@ use sha2::{Digest, Sha256};
use std::path::Path;
use thiserror::Error;
/// Every newly signed manifest carries this complete supported-client matrix so
/// its one Ed25519 signature cannot be replayed as a partial platform release.
pub const REQUIRED_RELEASE_PLATFORMS: [&str; 4] = [
"windows-x86_64",
"linux-x86_64",
"linux-arm64",
"macos-arm64",
];
#[derive(Debug, Error)]
pub enum UpdateError {
#[error("invalid Ed25519 public key")]
@ -27,6 +36,8 @@ pub enum UpdateError {
Install,
#[error("HTTPS is required for update downloads except on loopback")]
InsecureURL,
#[error("invalid signed update manifest: {0}")]
InvalidManifest(String),
}
pub fn decode_public_key(value: &str) -> Result<[u8; 32], UpdateError> {
@ -36,6 +47,124 @@ pub fn decode_public_key(value: &str) -> Result<[u8; 32], UpdateError> {
decoded.try_into().map_err(|_| UpdateError::InvalidKey)
}
fn valid_sha256(value: &str) -> bool {
value.len() == 64 && value.bytes().all(|character| character.is_ascii_hexdigit())
}
/// Validate the signed-update fields that every client must understand.
///
/// The updater accepts already-signed historical manifests that predate
/// `linux-arm64`, provided they contain a valid artifact for the running
/// target. New release publication uses [`validate_signed_release_matrix`] to
/// require the complete matrix.
pub fn validate_signed_release(release: &ClientRelease) -> Result<(), UpdateError> {
if release.kind != "signed" {
return Err(UpdateError::InvalidManifest(
"automatic updates require kind=signed".to_owned(),
));
}
let expected_percent = match release.channel.as_str() {
"canary" => 5,
"pilot" => 20,
"stable" => 100,
_ => {
return Err(UpdateError::InvalidManifest(
"channel must be canary, pilot or stable".to_owned(),
));
}
};
if !release.paused && release.rollout_percent != expected_percent {
return Err(UpdateError::InvalidManifest(
"active rollout percentage does not match its channel".to_owned(),
));
}
if release.artifacts.is_empty() {
return Err(UpdateError::InvalidManifest(
"manifest has no release artifacts".to_owned(),
));
}
for (platform, artifact) in &release.artifacts {
if artifact.is_empty() {
return Err(UpdateError::InvalidManifest(format!(
"manifest is missing the {platform} artifact"
)));
}
let digest = release
.sha256
.get(platform)
.map(String::as_str)
.unwrap_or_default();
if !valid_sha256(digest) {
return Err(UpdateError::InvalidManifest(format!(
"manifest has an invalid {platform} SHA-256"
)));
}
}
Ok(())
}
/// Validate the complete release matrix required for new signed releases.
pub fn validate_signed_release_matrix(release: &ClientRelease) -> Result<(), UpdateError> {
validate_signed_release(release)?;
for platform in REQUIRED_RELEASE_PLATFORMS {
let artifact = release
.artifacts
.get(platform)
.map(String::as_str)
.unwrap_or_default();
if artifact.is_empty() {
return Err(UpdateError::InvalidManifest(format!(
"manifest is missing the {platform} artifact"
)));
}
let digest = release
.sha256
.get(platform)
.map(String::as_str)
.unwrap_or_default();
if !valid_sha256(digest) {
return Err(UpdateError::InvalidManifest(format!(
"manifest has an invalid {platform} SHA-256"
)));
}
}
Ok(())
}
/// Validate that a signed manifest contains the artifact for this exact ABI.
pub fn validate_release_platform(
release: &ClientRelease,
platform_key: &str,
) -> Result<(), UpdateError> {
validate_signed_release(release)?;
if !REQUIRED_RELEASE_PLATFORMS.contains(&platform_key) {
return Err(UpdateError::InvalidManifest(format!(
"unsupported release platform {platform_key}"
)));
}
let artifact = release
.artifacts
.get(platform_key)
.map(String::as_str)
.unwrap_or_default();
if artifact.is_empty() {
return Err(UpdateError::InvalidManifest(format!(
"manifest is missing the {platform_key} artifact"
)));
}
let digest = release
.sha256
.get(platform_key)
.map(String::as_str)
.unwrap_or_default();
if !valid_sha256(digest) {
return Err(UpdateError::InvalidManifest(format!(
"manifest has an invalid {platform_key} SHA-256"
)));
}
Ok(())
}
pub async fn download_verified(
url: &str,
destination: &Path,
@ -145,6 +274,7 @@ pub async fn install_native_package(package: &Path) -> Result<(), UpdateError> {
}
pub fn verify_release(release: &ClientRelease, public_key: &[u8; 32]) -> Result<(), UpdateError> {
validate_signed_release(release)?;
let mut unsigned = release.clone();
unsigned.signature.clear();
let payload = serde_json::to_vec(&unsigned)?;
@ -169,7 +299,10 @@ pub fn verify_artifact(bytes: &[u8], expected_sha256: &str) -> Result<(), Update
#[cfg(test)]
mod tests {
use super::{verify_artifact, verify_release};
use super::{
REQUIRED_RELEASE_PLATFORMS, validate_release_platform, validate_signed_release,
validate_signed_release_matrix, verify_artifact, verify_release,
};
use base64::Engine;
use drosync_protocol::ClientRelease;
use ed25519_dalek::{Signer, SigningKey};
@ -185,14 +318,24 @@ mod tests {
channel: "canary".to_owned(),
protocol_version: "v1".to_owned(),
minimum_server_version: "1.0.0".to_owned(),
artifacts: BTreeMap::from([(
"linux-x86_64".to_owned(),
"https://downloads.example/drosync.deb".to_owned(),
)]),
sha256: BTreeMap::from([(
"linux-x86_64".to_owned(),
hex::encode(sha2::Sha256::digest(b"package")),
)]),
artifacts: REQUIRED_RELEASE_PLATFORMS
.into_iter()
.map(|platform| {
(
platform.to_owned(),
format!("https://downloads.example/{platform}.package"),
)
})
.collect::<BTreeMap<_, _>>(),
sha256: REQUIRED_RELEASE_PLATFORMS
.into_iter()
.map(|platform| {
(
platform.to_owned(),
hex::encode(sha2::Sha256::digest(b"package")),
)
})
.collect::<BTreeMap<_, _>>(),
signature: String::new(),
rollout_percent: 5,
paused: false,
@ -221,4 +364,17 @@ mod tests {
assert!(verify_release(&release, &public_key).is_err());
assert!(verify_artifact(b"tampered", &release.sha256["linux-x86_64"]).is_err());
}
#[test]
fn signed_manifest_requires_linux_arm64_and_exact_platform_key() {
let (mut release, _) = signed_release();
release.artifacts.remove("linux-arm64");
assert!(validate_signed_release(&release).is_ok());
assert!(validate_signed_release_matrix(&release).is_err());
let (release, _) = signed_release();
validate_release_platform(&release, "linux-arm64")
.expect("ARM64 package belongs to the signed release matrix");
assert!(validate_release_platform(&release, "linux-riscv64").is_err());
}
}

View File

@ -19,11 +19,26 @@ build_workspace() {
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
}
prepare_update_key() {
local platform_key="$1"
local specific_variable
case "$platform_key" in
linux-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64" ;;
linux-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64" ;;
macos-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64" ;;
*) echo "unsupported preview update-key platform: $platform_key" >&2; exit 2 ;;
esac
local selected="${!specific_variable:-$DROSYNC_UPDATE_PUBLIC_KEY_BASE64}"
export DROSYNC_UPDATE_PUBLIC_KEY_BASE64="$selected"
export "$specific_variable=$selected"
}
build_macos_preview() {
[[ "$(uname -s)-$(uname -m)" == "Darwin-arm64" ]] || {
echo "macos-arm64 preview must be built on an Apple Silicon runner" >&2
exit 1
}
prepare_update_key macos-arm64
build_workspace
local app
local package_version
@ -58,9 +73,44 @@ build_linux_preview() {
echo "linux-x86_64 preview must be built on a Linux x86_64 runner" >&2
exit 1
}
build_workspace
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" "$ROOT_DIR/packaging/linux/build-deb.sh"
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" "$ROOT_DIR/packaging/linux/build-portable.sh"
prepare_update_key linux-x86_64
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release \
-p drosync-cli -p drosync-daemon
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-deb.sh"
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-portable.sh"
}
build_linux_arm64_preview() {
[[ "$(uname -s)-$(uname -m)" == "Linux-x86_64" ]] || {
echo "linux-arm64 preview must be cross-built on a Linux x86_64 runner" >&2
exit 1
}
prepare_update_key linux-arm64
local target="aarch64-unknown-linux-gnu"
local linker="${DROSYNC_AARCH64_LINKER:-aarch64-linux-gnu-gcc}"
local archiver="${DROSYNC_AARCH64_AR:-aarch64-linux-gnu-ar}"
command -v "$linker" >/dev/null || {
echo "missing ARM64 cross linker $linker (install gcc-aarch64-linux-gnu)" >&2
exit 1
}
command -v "$archiver" >/dev/null || {
echo "missing ARM64 archiver $archiver (install binutils-aarch64-linux-gnu)" >&2
exit 1
}
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="$linker"
export CC_aarch64_unknown_linux_gnu="$linker"
export AR_aarch64_unknown_linux_gnu="$archiver"
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release --target "$target" \
-p drosync-cli -p drosync-daemon
local binary_dir="$ROOT_DIR/target/$target/release"
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_PLATFORM_KEY=linux-arm64 \
DROSYNC_DEB_ARCH=arm64 DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-deb.sh"
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_PLATFORM_KEY=linux-arm64 \
DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-portable.sh"
}
case "$TARGET" in
@ -70,8 +120,11 @@ case "$TARGET" in
linux-x86_64)
build_linux_preview
;;
linux-arm64)
build_linux_arm64_preview
;;
*)
echo "usage: $0 macos-arm64|linux-x86_64" >&2
echo "usage: $0 macos-arm64|linux-x86_64|linux-arm64" >&2
exit 2
;;
esac

View File

@ -2,16 +2,75 @@
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
: "${DROSYNC_UPDATE_PUBLIC_KEY_BASE64:?set DROSYNC_UPDATE_PUBLIC_KEY_BASE64 from CI secrets}"
TARGET="${1:-}"
prepare_update_key() {
local platform_key="$1"
local specific_variable
case "$platform_key" in
linux-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64" ;;
linux-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64" ;;
macos-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64" ;;
windows-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64" ;;
*) echo "unsupported update-key platform: $platform_key" >&2; exit 2 ;;
esac
local selected="${!specific_variable:-${DROSYNC_UPDATE_PUBLIC_KEY_BASE64:-}}"
if [[ -z "$selected" ]]; then
echo "set $specific_variable or DROSYNC_UPDATE_PUBLIC_KEY_BASE64 from CI secrets" >&2
exit 1
fi
# Keep the generic name populated for older targets while the platform crate
# embeds the more specific variable for newly built clients.
export DROSYNC_UPDATE_PUBLIC_KEY_BASE64="$selected"
export "$specific_variable=$selected"
}
require_linux_x86_64_builder() {
[[ "$(uname -s)-$(uname -m)" == "Linux-x86_64" ]] || {
echo "linux-arm64 cross builds require a Linux x86_64 builder" >&2
exit 1
}
}
build_linux_arm64_headless() {
require_linux_x86_64_builder
prepare_update_key linux-arm64
local target="aarch64-unknown-linux-gnu"
local linker="${DROSYNC_AARCH64_LINKER:-aarch64-linux-gnu-gcc}"
local archiver="${DROSYNC_AARCH64_AR:-aarch64-linux-gnu-ar}"
command -v "$linker" >/dev/null || {
echo "missing ARM64 cross linker $linker (install gcc-aarch64-linux-gnu)" >&2
exit 1
}
command -v "$archiver" >/dev/null || {
echo "missing ARM64 archiver $archiver (install binutils-aarch64-linux-gnu)" >&2
exit 1
}
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="$linker"
export CC_aarch64_unknown_linux_gnu="$linker"
export AR_aarch64_unknown_linux_gnu="$archiver"
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release --target "$target" \
-p drosync-cli -p drosync-daemon
local binary_dir="$ROOT_DIR/target/$target/release"
DROSYNC_PLATFORM_KEY=linux-arm64 DROSYNC_DEB_ARCH=arm64 DROSYNC_BINARY_DIR="$binary_dir" \
DROSYNC_INCLUDE_GUI=0 "$ROOT_DIR/packaging/linux/build-deb.sh"
DROSYNC_PLATFORM_KEY=linux-arm64 DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-portable.sh"
}
case "$TARGET" in
linux-x86_64)
prepare_update_key linux-x86_64
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
"$ROOT_DIR/packaging/linux/build-deb.sh"
"$ROOT_DIR/packaging/linux/build-portable.sh"
;;
linux-arm64)
build_linux_arm64_headless
;;
macos-arm64)
[[ "$(uname -s)-$(uname -m)" == "Darwin-arm64" ]] || { echo "macos-arm64 must be built on an Apple Silicon runner" >&2; exit 1; }
prepare_update_key macos-arm64
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
"$ROOT_DIR/packaging/macos/build-pkg.sh"
;;
@ -20,7 +79,7 @@ case "$TARGET" in
exit 2
;;
*)
echo "usage: $0 linux-x86_64|macos-arm64|windows-x86_64" >&2
echo "usage: $0 linux-x86_64|linux-arm64|macos-arm64|windows-x86_64" >&2
exit 2
;;
esac

View File

@ -0,0 +1,29 @@
# drosync portable systemd bundle
This archive is an ABI-specific, no-container package for Linux systems with
systemd. It contains the CLI and daemon; `linux-arm64` builds deliberately omit
the desktop tray GUI so they can run on a headless S100.
Verify the adjacent SHA-256 file before extracting the archive. Then install it
as root:
```bash
sudo ./install-systemd-portable.sh --enable
```
The portable binaries require Ubuntu's `libdbus-1-3` runtime package for the
OS credential-store integration. Install it before running the installer:
```bash
sudo apt-get update
sudo apt-get install -y ca-certificates libdbus-1-3
```
The installer uses `/opt/drosync` by default, creates the restricted `drosync`
system account and `/var/lib/drosync`, writes `/etc/systemd/system/drosync.service`,
and does not store access credentials. Configure credentials through the CLI or
the existing `/etc/drosync/environment` / `DROSYNC_SECRET_FILE` mechanism.
Use `sudo ./uninstall-systemd-portable.sh` to stop and remove the service. The
uninstaller leaves `/var/lib/drosync` and `/etc/drosync` intact so sync state and
credentials are not deleted implicitly.

View File

@ -3,15 +3,56 @@ set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
VERSION="${DROSYNC_VERSION:-$(sed -n 's/^version = "\(.*\)"/\1/p' "$ROOT_DIR/Cargo.toml" | head -1)}"
ARCH="${DROSYNC_DEB_ARCH:-amd64}"
PLATFORM_KEY="${DROSYNC_PLATFORM_KEY:-}"
BINARY_DIR="${DROSYNC_BINARY_DIR:-$ROOT_DIR/target/release}"
if [[ -z "$PLATFORM_KEY" ]]; then
case "$(uname -m)" in
x86_64) PLATFORM_KEY="linux-x86_64" ;;
aarch64|arm64) PLATFORM_KEY="linux-arm64" ;;
*) echo "set DROSYNC_PLATFORM_KEY for this Linux architecture" >&2; exit 2 ;;
esac
fi
case "$PLATFORM_KEY" in
linux-x86_64)
DEFAULT_ARCH="amd64"
DEFAULT_INCLUDE_GUI=1
;;
linux-arm64)
DEFAULT_ARCH="arm64"
DEFAULT_INCLUDE_GUI=0
;;
*)
echo "unsupported Linux release platform: $PLATFORM_KEY" >&2
exit 2
;;
esac
ARCH="${DROSYNC_DEB_ARCH:-$DEFAULT_ARCH}"
INCLUDE_GUI="${DROSYNC_INCLUDE_GUI:-$DEFAULT_INCLUDE_GUI}"
OUTPUT_DIR="${DROSYNC_OUTPUT_DIR:-$ROOT_DIR/dist}"
PACKAGE_ROOT="$(mktemp -d)"
trap 'rm -rf "$PACKAGE_ROOT"' EXIT
for binary in drosync drosync-daemon; do
if [[ ! -x "$BINARY_DIR/$binary" ]]; then
echo "missing executable $BINARY_DIR/$binary; build CLI and daemon for $PLATFORM_KEY first" >&2
exit 1
fi
done
case "$INCLUDE_GUI" in
0|1) ;;
*) echo "DROSYNC_INCLUDE_GUI must be 0 or 1" >&2; exit 2 ;;
esac
if [[ "$INCLUDE_GUI" == 1 && ! -x "$BINARY_DIR/drosync-gui" ]]; then
echo "missing executable $BINARY_DIR/drosync-gui while DROSYNC_INCLUDE_GUI=1" >&2
exit 1
fi
mkdir -p "$PACKAGE_ROOT/DEBIAN" "$PACKAGE_ROOT/usr/bin" "$PACKAGE_ROOT/lib/systemd/system" "$PACKAGE_ROOT/usr/lib/systemd/user"
cp "$ROOT_DIR/target/release/drosync" "$PACKAGE_ROOT/usr/bin/drosync"
cp "$ROOT_DIR/target/release/drosync-daemon" "$PACKAGE_ROOT/usr/bin/drosync-daemon"
cp "$ROOT_DIR/target/release/drosync-gui" "$PACKAGE_ROOT/usr/bin/drosync-gui"
cp "$BINARY_DIR/drosync" "$PACKAGE_ROOT/usr/bin/drosync"
cp "$BINARY_DIR/drosync-daemon" "$PACKAGE_ROOT/usr/bin/drosync-daemon"
if [[ "$INCLUDE_GUI" == 1 ]]; then
cp "$BINARY_DIR/drosync-gui" "$PACKAGE_ROOT/usr/bin/drosync-gui"
fi
sed -e "s/^Version:.*/Version: $VERSION/" -e "s/^Architecture:.*/Architecture: $ARCH/" \
"$ROOT_DIR/packaging/linux/debian/control" >"$PACKAGE_ROOT/DEBIAN/control"
cp "$ROOT_DIR/packaging/linux/debian/postinst" "$PACKAGE_ROOT/DEBIAN/postinst"
@ -21,6 +62,10 @@ cp "$ROOT_DIR/packaging/linux/drosync.service" "$PACKAGE_ROOT/lib/systemd/system
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$PACKAGE_ROOT/usr/lib/systemd/user/drosync.service"
mkdir -p "$OUTPUT_DIR"
dpkg-deb --root-owner-group --build "$PACKAGE_ROOT" "$OUTPUT_DIR/drosync_${VERSION}_${ARCH}.deb"
(
cd "$OUTPUT_DIR"
sha256sum "drosync_${VERSION}_${ARCH}.deb" >"drosync_${VERSION}_${ARCH}.deb.sha256"
)
if [[ -n "${DROSYNC_DEB_SIGN_KEY:-}" ]]; then
dpkg-sig --sign builder -k "$DROSYNC_DEB_SIGN_KEY" "$OUTPUT_DIR/drosync_${VERSION}_${ARCH}.deb"
fi

View File

@ -4,14 +4,62 @@ set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
VERSION="${DROSYNC_VERSION:-1.0.0}"
OUTPUT_DIR="${DROSYNC_OUTPUT_DIR:-$ROOT_DIR/dist}"
PLATFORM_KEY="${DROSYNC_PLATFORM_KEY:-}"
BINARY_DIR="${DROSYNC_BINARY_DIR:-$ROOT_DIR/target/release}"
if [[ -z "$PLATFORM_KEY" ]]; then
case "$(uname -m)" in
x86_64) PLATFORM_KEY="linux-x86_64" ;;
aarch64|arm64) PLATFORM_KEY="linux-arm64" ;;
*) echo "set DROSYNC_PLATFORM_KEY for this Linux architecture" >&2; exit 2 ;;
esac
fi
case "$PLATFORM_KEY" in
linux-x86_64) DEFAULT_INCLUDE_GUI=1 ;;
linux-arm64) DEFAULT_INCLUDE_GUI=0 ;;
*) echo "unsupported Linux release platform: $PLATFORM_KEY" >&2; exit 2 ;;
esac
INCLUDE_GUI="${DROSYNC_INCLUDE_GUI:-$DEFAULT_INCLUDE_GUI}"
STAGE="$(mktemp -d)"
trap 'rm -rf "$STAGE"' EXIT
mkdir -p "$STAGE/drosync-$VERSION" "$OUTPUT_DIR"
cp "$ROOT_DIR/target/release/drosync" "$ROOT_DIR/target/release/drosync-daemon" "$ROOT_DIR/target/release/drosync-gui" "$STAGE/drosync-$VERSION/"
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$STAGE/drosync-$VERSION/"
tar -C "$STAGE" -czf "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz" "drosync-$VERSION"
sha256sum "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz" >"$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz.sha256"
if [[ -n "${DROSYNC_GPG_SIGN_KEY:-}" ]]; then
gpg --batch --yes --local-user "$DROSYNC_GPG_SIGN_KEY" --detach-sign --armor "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz"
for binary in drosync drosync-daemon; do
if [[ ! -x "$BINARY_DIR/$binary" ]]; then
echo "missing executable $BINARY_DIR/$binary; build CLI and daemon for $PLATFORM_KEY first" >&2
exit 1
fi
done
case "$INCLUDE_GUI" in
0|1) ;;
*) echo "DROSYNC_INCLUDE_GUI must be 0 or 1" >&2; exit 2 ;;
esac
if [[ "$INCLUDE_GUI" == 1 && ! -x "$BINARY_DIR/drosync-gui" ]]; then
echo "missing executable $BINARY_DIR/drosync-gui while DROSYNC_INCLUDE_GUI=1" >&2
exit 1
fi
PACKAGE_NAME="drosync-$VERSION"
PACKAGE_ROOT="$STAGE/$PACKAGE_NAME"
mkdir -p "$PACKAGE_ROOT" "$OUTPUT_DIR"
cp "$BINARY_DIR/drosync" "$PACKAGE_ROOT/drosync"
cp "$BINARY_DIR/drosync-daemon" "$PACKAGE_ROOT/drosync-daemon"
if [[ "$INCLUDE_GUI" == 1 ]]; then
cp "$BINARY_DIR/drosync-gui" "$PACKAGE_ROOT/drosync-gui"
fi
cp "$ROOT_DIR/packaging/linux/drosync.service" "$PACKAGE_ROOT/drosync.service"
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$PACKAGE_ROOT/drosync-user.service"
cp "$ROOT_DIR/packaging/linux/drosync-portable.service" "$PACKAGE_ROOT/drosync-portable.service"
cp "$ROOT_DIR/packaging/linux/install-systemd-portable.sh" "$PACKAGE_ROOT/install-systemd-portable.sh"
cp "$ROOT_DIR/packaging/linux/uninstall-systemd-portable.sh" "$PACKAGE_ROOT/uninstall-systemd-portable.sh"
cp "$ROOT_DIR/packaging/linux/PORTABLE-SYSTEMD-README.md" "$PACKAGE_ROOT/README-systemd-portable.md"
chmod 0755 "$PACKAGE_ROOT/install-systemd-portable.sh" "$PACKAGE_ROOT/uninstall-systemd-portable.sh"
ARCHIVE="$OUTPUT_DIR/drosync-${VERSION}-${PLATFORM_KEY}.tar.gz"
tar --sort=name --mtime='@0' --owner=0 --group=0 --numeric-owner \
-C "$STAGE" -czf "$ARCHIVE" "$PACKAGE_NAME"
(
cd "$OUTPUT_DIR"
sha256sum "$(basename "$ARCHIVE")" >"$(basename "$ARCHIVE").sha256"
)
if [[ -n "${DROSYNC_GPG_SIGN_KEY:-}" ]]; then
gpg --batch --yes --local-user "$DROSYNC_GPG_SIGN_KEY" --detach-sign --armor "$ARCHIVE"
fi

View File

@ -4,6 +4,7 @@ Section: utils
Priority: optional
Architecture: amd64
Maintainer: D-Robotics <support@example.invalid>
Depends: ca-certificates, debsig-verify, libsecret-1-0
Depends: ca-certificates, debsig-verify, libdbus-1-3, libsecret-1-0
Description: Secure one-way local-to-BOS synchronization client
drosync provides a CLI, daemon, tray GUI, crash recovery, and signed updates.
drosync provides a CLI, daemon, crash recovery, and signed updates. Some
packages additionally include an optional tray GUI.

View File

@ -8,4 +8,10 @@ if ! getent passwd drosync >/dev/null 2>&1; then
adduser --system --ingroup drosync --home /var/lib/drosync --no-create-home drosync
fi
install -d -o drosync -g drosync -m 0700 /var/lib/drosync
install -d -o drosync -g drosync -m 0700 /var/lib/drosync/config
install -d -o root -g root -m 0755 /etc/drosync
if [ ! -e /etc/drosync/environment ]; then
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' >/etc/drosync/environment
chmod 0644 /etc/drosync/environment
fi
systemctl daemon-reload

View File

@ -0,0 +1,24 @@
[Unit]
Description=drosync portable system synchronization daemon
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=@DROSYNC_INSTALL_ROOT@/drosync-daemon
Restart=on-failure
RestartSec=5
User=drosync
Group=drosync
Environment=HOME=/var/lib/drosync
Environment=XDG_DATA_HOME=/var/lib/drosync
Environment=XDG_CONFIG_HOME=/var/lib/drosync/config
EnvironmentFile=-/etc/drosync/environment
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/lib/drosync
[Install]
WantedBy=multi-user.target

View File

@ -12,7 +12,7 @@ User=drosync
Group=drosync
Environment=HOME=/var/lib/drosync
Environment=XDG_DATA_HOME=/var/lib/drosync
Environment=XDG_CONFIG_HOME=/etc/drosync
Environment=XDG_CONFIG_HOME=/var/lib/drosync/config
EnvironmentFile=-/etc/drosync/environment
NoNewPrivileges=true
PrivateTmp=true

View File

@ -0,0 +1,81 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
echo "run this installer as root (for example: sudo ./install-systemd-portable.sh)" >&2
exit 1
fi
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALL_ROOT="/opt/drosync"
ENABLE=0
usage() {
echo "usage: $0 [--prefix /opt/drosync] [--enable]" >&2
}
while [[ $# -gt 0 ]]; do
case "$1" in
--prefix)
[[ $# -ge 2 ]] || { usage; exit 2; }
INSTALL_ROOT="$2"
shift 2
;;
--enable)
ENABLE=1
shift
;;
-h|--help)
usage
exit 0
;;
*)
usage
exit 2
;;
esac
done
case "$INSTALL_ROOT" in
/*) ;;
*) echo "--prefix must be an absolute path" >&2; exit 2 ;;
esac
for binary in drosync drosync-daemon; do
[[ -x "$SOURCE_DIR/$binary" ]] || {
echo "portable bundle is missing executable $binary" >&2
exit 1
}
done
if ! ldconfig -p 2>/dev/null | grep -Fq 'libdbus-1.so.3'; then
echo "missing runtime dependency libdbus-1.so.3; install it with: apt-get install libdbus-1-3" >&2
exit 1
fi
if ! getent group drosync >/dev/null 2>&1; then
groupadd --system drosync
fi
if ! getent passwd drosync >/dev/null 2>&1; then
useradd --system --gid drosync --home-dir /var/lib/drosync --no-create-home --shell /usr/sbin/nologin drosync
fi
install -d -o drosync -g drosync -m 0700 /var/lib/drosync
install -d -o drosync -g drosync -m 0700 /var/lib/drosync/config
install -d -m 0755 /etc/drosync "$INSTALL_ROOT"
if [[ ! -e /etc/drosync/environment ]]; then
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' >/etc/drosync/environment
chmod 0644 /etc/drosync/environment
fi
install -m 0755 "$SOURCE_DIR/drosync" "$INSTALL_ROOT/drosync"
install -m 0755 "$SOURCE_DIR/drosync-daemon" "$INSTALL_ROOT/drosync-daemon"
if [[ -f "$SOURCE_DIR/drosync-gui" ]]; then
install -m 0755 "$SOURCE_DIR/drosync-gui" "$INSTALL_ROOT/drosync-gui"
fi
sed "s|@DROSYNC_INSTALL_ROOT@|$INSTALL_ROOT|g" "$SOURCE_DIR/drosync-portable.service" \
>/etc/systemd/system/drosync.service
chmod 0644 /etc/systemd/system/drosync.service
systemctl daemon-reload
if [[ "$ENABLE" == 1 ]]; then
systemctl enable --now drosync.service
fi
echo "installed drosync portable systemd bundle to $INSTALL_ROOT"

View File

@ -0,0 +1,27 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
echo "run this uninstaller as root" >&2
exit 1
fi
INSTALL_ROOT="/opt/drosync"
if [[ "${1:-}" == "--prefix" ]]; then
[[ $# -eq 2 ]] || { echo "usage: $0 [--prefix /opt/drosync]" >&2; exit 2; }
INSTALL_ROOT="$2"
elif [[ $# -ne 0 ]]; then
echo "usage: $0 [--prefix /opt/drosync]" >&2
exit 2
fi
case "$INSTALL_ROOT" in
/*) ;;
*) echo "--prefix must be an absolute path" >&2; exit 2 ;;
esac
systemctl disable --now drosync.service >/dev/null 2>&1 || true
rm -f /etc/systemd/system/drosync.service
systemctl daemon-reload
rm -f "$INSTALL_ROOT/drosync" "$INSTALL_ROOT/drosync-daemon" "$INSTALL_ROOT/drosync-gui"
rmdir "$INSTALL_ROOT" 2>/dev/null || true
echo "uninstalled drosync portable systemd bundle from $INSTALL_ROOT"

View File

@ -0,0 +1,53 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
WORK_DIR="$(mktemp -d)"
trap 'rm -rf "$WORK_DIR"' EXIT
BIN_DIR="$WORK_DIR/bin"
OUTPUT_DIR="$WORK_DIR/dist"
VERSION="0.0.0-arm64-package-check"
mkdir -p "$BIN_DIR"
for binary in drosync drosync-daemon; do
printf '#!/bin/sh\nexit 0\n' >"$BIN_DIR/$binary"
chmod 0755 "$BIN_DIR/$binary"
done
DROSYNC_VERSION="$VERSION" \
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" \
DROSYNC_PLATFORM_KEY=linux-arm64 \
DROSYNC_BINARY_DIR="$BIN_DIR" \
DROSYNC_INCLUDE_GUI=0 \
"$ROOT_DIR/packaging/linux/build-portable.sh"
ARCHIVE="$OUTPUT_DIR/drosync-${VERSION}-linux-arm64.tar.gz"
CHECKSUM="$ARCHIVE.sha256"
[[ -f "$ARCHIVE" && -f "$CHECKSUM" ]]
sha256sum --check "$CHECKSUM"
CONTENTS="$WORK_DIR/contents.txt"
tar -tzf "$ARCHIVE" >"$CONTENTS"
PACKAGE_ROOT="drosync-$VERSION"
for path in \
"$PACKAGE_ROOT/drosync" \
"$PACKAGE_ROOT/drosync-daemon" \
"$PACKAGE_ROOT/drosync-portable.service" \
"$PACKAGE_ROOT/install-systemd-portable.sh" \
"$PACKAGE_ROOT/uninstall-systemd-portable.sh" \
"$PACKAGE_ROOT/README-systemd-portable.md"; do
grep -Fqx "$path" "$CONTENTS"
done
if grep -Fq "drosync-gui" "$CONTENTS"; then
echo "ARM64 portable package must not require or include drosync-gui" >&2
exit 1
fi
tar -xOf "$ARCHIVE" "$PACKAGE_ROOT/drosync-portable.service" \
| grep -Fqx 'ExecStart=@DROSYNC_INSTALL_ROOT@/drosync-daemon'
tar -xOf "$ARCHIVE" "$PACKAGE_ROOT/install-systemd-portable.sh" \
| grep -Fqx ' systemctl enable --now drosync.service'
bash -n "$ROOT_DIR/packaging/linux/build-portable.sh" \
"$ROOT_DIR/packaging/linux/install-systemd-portable.sh" \
"$ROOT_DIR/packaging/linux/uninstall-systemd-portable.sh"
echo "validated ARM64 no-GUI systemd portable package: $ARCHIVE"

View File

@ -26,11 +26,12 @@
"paused": { "type": "boolean" },
"artifacts": {
"type": "object",
"required": ["windows-x86_64", "linux-x86_64", "macos-arm64"],
"required": ["windows-x86_64", "linux-x86_64", "linux-arm64", "macos-arm64"],
"additionalProperties": { "type": "string", "format": "uri" }
},
"sha256": {
"type": "object",
"required": ["windows-x86_64", "linux-x86_64", "linux-arm64", "macos-arm64"],
"additionalProperties": { "type": "string", "pattern": "^[a-f0-9]{64}$" }
},
"signature": { "type": "string", "contentEncoding": "base64" }