forked from longtao.wu/droboticflow
Compare commits
17 Commits
dev
...
poc-org-st
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
111332f409 | ||
|
|
07d8566b4d | ||
|
|
cfbb451879 | ||
|
|
88e6d08ef0 | ||
|
|
109290eeb1 | ||
|
|
02c654c0ea | ||
|
|
eb1f49c146 | ||
|
|
fe10b9c048 | ||
|
|
6a2056f67b | ||
|
|
5b80cb68e0 | ||
|
|
1f994927c0 | ||
|
|
0d5578421e | ||
|
|
76cc571c49 | ||
|
|
685671f390 | ||
|
|
927b1f7163 | ||
|
|
0c02227082 | ||
|
|
6ff9e55405 |
@ -2,9 +2,10 @@
|
||||
|
||||
## Deployment Environment
|
||||
|
||||
- The `dev` branch uses the `poc` deployment and test environment.
|
||||
- Leave `DEPLOY_ENV` unset on `dev`; the deploy scripts should resolve it to `poc`.
|
||||
- Do not deploy `dev` to `wjzh` unless the user explicitly overrides this rule.
|
||||
- The `poc` branch uses the POC environment (namespace `droboticflow`, public entry `120.48.164.19`).
|
||||
- Keep the pre-Fanyu baseline `d947a20b` plus drosync changes. Do not merge `dev` or Fanyu commits into `poc`.
|
||||
- Build POC images from `poc` and publish them under `dataflow-poc`, never from the Fanyu development branch.
|
||||
- Fanyu is a separate environment and is not a target of POC deployment commands.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
|
||||
@ -13,6 +13,7 @@
|
||||
|
||||
## Document map
|
||||
|
||||
- [中文用户使用手册](用户使用手册.md)
|
||||
- [Product requirements](product-requirements.md)
|
||||
- [Architecture and state machines](architecture.md)
|
||||
- [OpenAPI 3.1 contract](openapi.yaml)
|
||||
|
||||
56
docs/drosync/acceptance-20260905.md
Normal file
56
docs/drosync/acceptance-20260905.md
Normal file
@ -0,0 +1,56 @@
|
||||
# POC drosync repair acceptance — 2026-09-05
|
||||
|
||||
Scope: POC `120.48.164.19`, API revision `5b80cb68`; Linux test host
|
||||
`120.48.123.215`. No Fanyu deployment or merge.
|
||||
|
||||
## Reproducible abort regression
|
||||
|
||||
From the repository, copy `scripts/drosync-poc-abort-regression.py` to the test
|
||||
host. The token file must contain an existing authorized device token and must
|
||||
not be printed or added to source control. Use a prefix permitted by that token.
|
||||
|
||||
```sh
|
||||
python3 /tmp/drosync-poc-abort-regression.py \
|
||||
--server http://120.48.164.19 \
|
||||
--token-file /var/lib/drosync/token \
|
||||
--space oss_KoX8zMpkW8bybqDj \
|
||||
--prefix payload/repair-20260905 \
|
||||
--output /srv/drosync-fulltest-20260905-r2/abort-retest-5b80cb68.json
|
||||
```
|
||||
|
||||
The script creates a fresh random object key and two sessions, uploads no object
|
||||
bytes, and aborts its sessions. Output contains no bearer or signed upload URLs.
|
||||
Exit code 0 means all assertions passed; nonzero is a failure, not a success marker.
|
||||
|
||||
Local regression:
|
||||
|
||||
```sh
|
||||
go test ./internal/api -run TestObjectSyncAbort -count=1
|
||||
go test ./internal/api ./internal/store -count=1
|
||||
```
|
||||
|
||||
## Actual results
|
||||
|
||||
- Before repair: abort did not release the object lease; the aborted session could
|
||||
still renew it. Keep the earlier failure receipts, rather than replacing them.
|
||||
- After repair: 8/8 real API assertions passed, including immediate takeover with
|
||||
fencing 1 → 2, closed-session heartbeat/reserve/commit rejection, repeated abort
|
||||
preserving the new owner, and rejection of complete after abort.
|
||||
- Sessions: `oses_q9GHF-0wDHN8uZ6v`, `oses_xb9D0k8R_sdiqiYq`.
|
||||
- API image: `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:305ea9fb9dfecef081b80ee55f961f19de7a889a3dd9b3769afba43e46d522ac`.
|
||||
- Large object retest: 5,369,757,696 bytes (5 GiB + 1 MiB), 21 parts,
|
||||
84.073 seconds, committed Session `oses_7oSRbW34E_vrRLdl`.
|
||||
- Local SHA-256 equals the server receipt:
|
||||
`a81fcc42cd9781340440e780df1bb20b1b783d06b1d3bddcd84f4e1ae3e038b4`.
|
||||
|
||||
## Boundaries
|
||||
|
||||
The large test uploaded real file bytes; it was not just a sparse file or plan.
|
||||
Independent BOS download/re-hashing remains unverified. The 24-hour low-load soak
|
||||
is ongoing and spans this API rollout, so it is not a single-version 24-hour
|
||||
certification. This does not certify all 132 checklist points, peak throughput,
|
||||
1-TiB files, 20-TiB sessions, disaster recovery, or monthly SLA. The helper is a
|
||||
sequential abort regression, not a proof of every possible concurrent race.
|
||||
|
||||
Detailed command-by-command Feishu manual:
|
||||
https://horizonrobotics.feishu.cn/wiki/C3JswTXyUiwiX8kMgq8cxyjbnse
|
||||
@ -12,7 +12,7 @@ Commands:
|
||||
| `configure --server URL` | Persist the control-plane URL through daemon IPC. |
|
||||
| `auth --server URL [--profile NAME] --access-key-id ID (--secret-file PATH \| --secret-stdin)` | Validate a scoped drosync Key/Secret, automatically register the device, and store the resulting bearer token in the OS credential store. `DROSYNC_ACCESS_KEY_ID` and `DROSYNC_SECRET` are supported for non-interactive injection. |
|
||||
| `mapping list` | List local mappings. |
|
||||
| `mapping add ROOT --space ID [--remote-prefix P] [--include-hidden] [--require-done]` | Add a mapping. |
|
||||
| `mapping add ROOT (--bucket NAME \| --space ID) [--remote-prefix P] [--include-hidden] [--require-done]` | Add a mapping. `--bucket` resolves only within spaces visible to the current scoped credential; `--space` selects an exact administrative ID. |
|
||||
| `mapping remove ID` | Remove a mapping without deleting local or remote objects. |
|
||||
| `plan ID` | Scan, upload a paged manifest, wait for the asynchronous plan, and retain the resumable session without uploading bytes. |
|
||||
| `sync ID [--dry-run]` | Resume a planned/crashed session or execute scan through final commit. |
|
||||
|
||||
@ -23,6 +23,26 @@ sudo systemctl enable --now drosync.service
|
||||
|
||||
The portable tarball contains all three binaries and a user systemd unit. Validate the detached GPG signature and SHA-256 before extraction.
|
||||
|
||||
## Ubuntu 22.04 ARM64 / RDK S100
|
||||
|
||||
Use only the `linux-arm64` DEB or portable tar. These packages contain the CLI
|
||||
and daemon, omit the desktop GUI and identify themselves as
|
||||
`releasePlatform=linux-arm64`.
|
||||
|
||||
The headless build deliberately does not depend on the desktop D-Bus keyring.
|
||||
Configure a writable token file in the daemon environment before authenticating:
|
||||
|
||||
```bash
|
||||
sudo install -d -o drosync -g drosync -m 0700 /var/lib/drosync
|
||||
sudo install -o drosync -g drosync -m 0600 /dev/null /var/lib/drosync/token
|
||||
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' | sudo tee /etc/drosync/environment >/dev/null
|
||||
```
|
||||
|
||||
`drosync auth` then writes the returned bearer token through a same-directory
|
||||
0600 temporary file, `fsync` and atomic rename. Missing `DROSYNC_SECRET_FILE`,
|
||||
unsafe parent permissions, symlinks, special files or non-0600 files fail
|
||||
closed. The package and environment file never contain the token itself.
|
||||
|
||||
## macOS 13+ arm64
|
||||
|
||||
Install the notarized `.pkg`. It installs `/Applications/drosync.app`, `/usr/local/bin/drosync`, a LaunchAgent, and a LaunchDaemon definition. Enable only the per-user agent or the administrator-managed daemon:
|
||||
@ -55,6 +75,19 @@ drosync plan <mapping-id>
|
||||
drosync sync <mapping-id>
|
||||
```
|
||||
|
||||
普通用户也可以直接填写凭据允许访问的 bucket。守护进程会通过控制面把
|
||||
bucket 解析为唯一的受限 Sync Space;不会在客户端保存 BOS AK/SK,也不会
|
||||
绕过 Space 的 root prefix:
|
||||
|
||||
```bash
|
||||
drosync mapping add /data/capture \
|
||||
--bucket bucket-a \
|
||||
--remote-prefix ubt/incoming/
|
||||
```
|
||||
|
||||
`--bucket` 与管理员使用的 `--space` 互斥。若同一凭据可见多个指向同一
|
||||
bucket 的 Space,命令会拒绝猜测,并要求管理员明确提供 `--space`。
|
||||
|
||||
For a strict profile, configure and authenticate with the same policy:
|
||||
|
||||
```bash
|
||||
|
||||
254
docs/drosync/用户使用手册.md
Normal file
254
docs/drosync/用户使用手册.md
Normal file
@ -0,0 +1,254 @@
|
||||
# drosync 用户使用手册
|
||||
|
||||
本文面向需要把 Ubuntu 本地目录持续、单向同步到 Drobodata POC 对象存储的用户和管理员。
|
||||
|
||||
## 1. 能做什么
|
||||
|
||||
`drosync` 监听一个本地目录,在文件稳定后自动上传到指定 bucket 的相对 prefix。它支持断点续传、失败重试、冲突保留、SQLite 状态恢复和 systemd 守护运行。
|
||||
|
||||
它不是网络盘:不会下载远端文件、删除远端对象、合并文件内容或跟随软链接。
|
||||
|
||||
## 2. 安全模型
|
||||
|
||||
客户端不保存完整 BOS AK/SK。管理员先在 POC 创建 Sync Space,再创建受限 drosync Key:
|
||||
|
||||
- Access Key ID 用于标识客户端;
|
||||
- Secret 只在创建时显示一次;
|
||||
- Scope 限制可用 Space、相对 prefix 和 `plan/create/update` 操作;
|
||||
- 首次认证换取的设备 Token 保存在 `/var/lib/drosync/token`,权限必须为 `0600`。
|
||||
|
||||
已有 Key 的 Secret 无法再次读取。新机器应创建独立 Key,不应多人共用一个 Secret。
|
||||
|
||||
## 3. 安装
|
||||
|
||||
### 3.1 DEB 安装(推荐)
|
||||
|
||||
先校验:
|
||||
|
||||
```bash
|
||||
sha256sum -c drosync_1.0.0-poc.20260901-bucket-map_amd64.deb.sha256
|
||||
```
|
||||
|
||||
安装并启动系统服务:
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y ./drosync_1.0.0-poc.20260901-bucket-map_amd64.deb
|
||||
sudo systemctl enable --now drosync.service
|
||||
systemctl status drosync.service --no-pager
|
||||
```
|
||||
|
||||
### 3.2 portable 安装
|
||||
|
||||
```bash
|
||||
sha256sum -c drosync-1.0.0-poc.20260901-bucket-map-linux-x86_64.tar.gz.sha256
|
||||
tar -xzf drosync-1.0.0-poc.20260901-bucket-map-linux-x86_64.tar.gz
|
||||
cd drosync-1.0.0-poc.20260901-bucket-map
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ca-certificates libdbus-1-3
|
||||
sudo ./install-systemd-portable.sh --enable
|
||||
```
|
||||
|
||||
## 4. POC 管理员准备
|
||||
|
||||
管理员在“对象存储 / 对象同步”中完成:
|
||||
|
||||
1. 创建或选择 Storage Connection;
|
||||
2. 创建 Sync Space,填写 bucket 和管理员限定的 root prefix;
|
||||
3. 执行 capability probe;
|
||||
4. 创建 drosync Key;
|
||||
5. 将 Scope 限制到该 Space 和需要的相对 prefix;
|
||||
6. 首次测试建议仅授权 `plan/create`,验证后再决定是否增加 `update`。
|
||||
|
||||
## 5. 系统服务模式配置
|
||||
|
||||
以下命令显式设置两个 XDG 路径,让 CLI 连接系统 daemon 的 Unix socket。
|
||||
|
||||
配置 POC 地址:
|
||||
|
||||
```bash
|
||||
sudo env \
|
||||
XDG_DATA_HOME=/var/lib/drosync \
|
||||
XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync configure --server http://120.48.164.19
|
||||
```
|
||||
|
||||
POC 当前为 HTTP,认证元数据不加密;只应在认可的隔离网络中使用。正式环境应使用 HTTPS 并增加 `--require-https`。
|
||||
|
||||
将一次性 Secret 写入 root 私有文件:
|
||||
|
||||
```bash
|
||||
sudo install -m 0600 /dev/null /root/drosync-onetime-secret
|
||||
sudo nano /root/drosync-onetime-secret
|
||||
```
|
||||
|
||||
认证:
|
||||
|
||||
```bash
|
||||
sudo env \
|
||||
XDG_DATA_HOME=/var/lib/drosync \
|
||||
XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync auth \
|
||||
--server http://120.48.164.19 \
|
||||
--access-key-id '<ACCESS_KEY_ID>' \
|
||||
--secret-file /root/drosync-onetime-secret
|
||||
```
|
||||
|
||||
成功后立即删除一次性 Secret:
|
||||
|
||||
```bash
|
||||
sudo rm /root/drosync-onetime-secret
|
||||
sudo stat -c '%a %U:%G %n' /var/lib/drosync/token
|
||||
```
|
||||
|
||||
期望 Token 文件权限为 `600 drosync:drosync`。
|
||||
|
||||
## 6. 添加同步目录
|
||||
|
||||
用户可以直接填写本地目录、bucket 和相对 prefix:
|
||||
|
||||
```bash
|
||||
sudo env \
|
||||
XDG_DATA_HOME=/var/lib/drosync \
|
||||
XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync mapping add /data/capture \
|
||||
--bucket bucket-a \
|
||||
--remote-prefix ubt/incoming/
|
||||
```
|
||||
|
||||
系统服务启用了 `PrivateTmp=true`。不要把同步源目录配置在 `/tmp` 或
|
||||
`/var/tmp`;systemd 私有临时目录会让 daemon 看不到用户 shell 创建的文件。
|
||||
应使用 `/data`、`/srv`、挂载的数据盘或实际采集目录。
|
||||
|
||||
`--bucket` 只会在当前 Key 可见的 Space 中解析。没有匹配或存在多个匹配时命令会拒绝猜测。管理员可改用 `--space <SPACE_ID>` 精确指定。
|
||||
|
||||
可选参数:
|
||||
|
||||
- `--include-hidden`:包含隐藏文件;默认不包含;
|
||||
- `--require-done`:只有存在 `{file}.done` 时才上传;
|
||||
- 临时后缀 `.tmp/.part/.partial/.crdownload/.swp/.download` 始终忽略;
|
||||
- 软链接、FIFO、socket 和设备文件始终忽略。
|
||||
|
||||
## 7. 检查和首次同步
|
||||
|
||||
先执行健康检查:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync doctor
|
||||
```
|
||||
|
||||
查看映射并取得 mapping ID:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync mapping list
|
||||
```
|
||||
|
||||
只生成计划,不上传文件:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync plan '<MAPPING_ID>'
|
||||
```
|
||||
|
||||
确认计划后执行同步:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync sync '<MAPPING_ID>'
|
||||
```
|
||||
|
||||
daemon 会持续监听映射目录。文件必须至少 60 秒保持大小和 mtime 不变,哈希前后 stat 也必须一致,才会进入上传。
|
||||
|
||||
## 8. 状态、日志和恢复
|
||||
|
||||
查看状态:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync --json status
|
||||
```
|
||||
|
||||
查看日志:
|
||||
|
||||
```bash
|
||||
sudo journalctl -u drosync.service -f
|
||||
```
|
||||
|
||||
失败后重试:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync retry '<MAPPING_ID>'
|
||||
```
|
||||
|
||||
SQLite 位于 `/var/lib/drosync/drosync/state.db`。不要在服务运行时复制或手工编辑数据库;备份前先停止服务,或使用 SQLite 在线备份机制。
|
||||
|
||||
## 9. 冲突处理
|
||||
|
||||
客户端不会静默覆盖并发写入:
|
||||
|
||||
- create 使用条件写入;
|
||||
- update 使用 ETag/version 前置条件;
|
||||
- 条件失败时保留原对象,并将新内容发布为 `.__conflict__<UTC>_<item>`;
|
||||
- multipart ETag 只用于传输校验,不当作文件 SHA-256。
|
||||
|
||||
查看冲突:
|
||||
|
||||
```bash
|
||||
sudo env XDG_DATA_HOME=/var/lib/drosync XDG_CONFIG_HOME=/var/lib/drosync/config \
|
||||
drosync conflicts
|
||||
```
|
||||
|
||||
## 10. 常见问题
|
||||
|
||||
### `server is not configured`
|
||||
|
||||
先运行 `drosync configure --server ...`,并确保 CLI 使用系统服务的两个 XDG 路径。
|
||||
|
||||
### `bucket ... is not available`
|
||||
|
||||
当前 Key 没有该 bucket 对应 Space 的 Scope,或管理员尚未创建/启用 Space。
|
||||
|
||||
### `bucket ... maps to multiple sync spaces`
|
||||
|
||||
让管理员提供准确 Space ID,然后使用 `--space <ID>`。
|
||||
|
||||
### 文件没有上传
|
||||
|
||||
检查:文件是否稳定满 60 秒、是否为临时后缀、是否为隐藏文件、是否要求 `.done`、Key 是否包含目标 prefix。
|
||||
|
||||
### 服务循环重启
|
||||
|
||||
```bash
|
||||
sudo systemctl status drosync.service --no-pager
|
||||
sudo journalctl -u drosync.service -n 100 --no-pager
|
||||
sudo stat -c '%a %U:%G %n' /var/lib/drosync /var/lib/drosync/config /etc/drosync
|
||||
```
|
||||
|
||||
## 11. 停止和卸载
|
||||
|
||||
停止自动同步:
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now drosync.service
|
||||
```
|
||||
|
||||
卸载 DEB:
|
||||
|
||||
```bash
|
||||
sudo apt-get remove drosync
|
||||
```
|
||||
|
||||
卸载不会自动删除 `/var/lib/drosync` 和 `/etc/drosync`,避免误删状态和凭据。确认不再需要后再由管理员单独清理。
|
||||
|
||||
## 12. 验收清单
|
||||
|
||||
- `drosync --json version` 显示 `linux-x86_64`;
|
||||
- `systemctl is-active drosync.service` 返回 `active`;
|
||||
- `drosync doctor` 通过 SQLite、IPC、凭据、控制面和 Scope 检查;
|
||||
- `plan` 的对象路径全部位于授权 prefix;
|
||||
- 上传后比对本地 SHA-256、远端 size/ETag/version;
|
||||
- 中断上传后可续传;
|
||||
- 并发覆盖测试生成冲突对象而不是覆盖原对象;
|
||||
- 审计中可追溯设备、session、item、操作和目标 key。
|
||||
17
docs/poc-baseline.md
Normal file
17
docs/poc-baseline.md
Normal file
@ -0,0 +1,17 @@
|
||||
# POC baseline
|
||||
|
||||
The POC branch is based on `d947a20b` (2026-08-19), immediately before the Fanyu delivery merge. It keeps the original dark interface, role navigation, projects, workflows, operators, storage and drosync pages.
|
||||
|
||||
The original drosync API/coordinator and access-key implementation are already in that baseline. The branch additionally preserves the drosync-only ARM64 packaging and signed-update changes from `f93654df`, plus these five changes from the former POC branch:
|
||||
|
||||
- `4e5166fe`: bucket-based mappings.
|
||||
- `eaec4c2c`: Linux package hardening.
|
||||
- `e19995ce`: headless systemd services.
|
||||
- `f7ca2755`: Chinese user manual.
|
||||
- `921fc372`: automatic scheduler reliability.
|
||||
|
||||
The resulting `tools/drosync`, `docs/drosync`, and `internal/api/object_sync.go` match the former `921fc372` POC versions, including executable script permissions. Fanyu business pages, global theme changes, role migrations and delivery bundles are excluded.
|
||||
|
||||
Deploy this branch to namespace `droboticflow`, public entry `http://120.48.164.19`, using the POC kubeconfig and the `dataflow-poc` image repository. Verify actual Pod image digests and API/Worker build metadata; changing the local branch does not change a running deployment.
|
||||
|
||||
Before this rollback, the branch was backed up as `backup/poc-before-fanyu-rollback-20260905`. Existing projects, users, runs and object-sync records are retained; reverting application code does not authorize deleting business data.
|
||||
@ -0,0 +1,64 @@
|
||||
# 新组织目录隔离部署验收(2026-10-10)
|
||||
|
||||
## 结果
|
||||
|
||||
已部署到 POC(命名空间 `droboticflow`,入口 `http://120.48.164.19`)。
|
||||
UBT 和 POC 主验收组织仍使用 `testxyz` 整桶连接。后续新建组织按组织标识
|
||||
自动创建 `testxyz/<slug>/`,自动配置独立的受限 IAM 用户、AK/SK 和产物存储。
|
||||
现有组织不自动加入新规则;组织改名不改变其已分配目录。
|
||||
|
||||
创建接口立即返回 202 / pending,后台自动配置和重试,管理页面自动刷新状态。
|
||||
存储就绪后才允许添加业务账号。本次真实验收:创建请求约 0.2 秒返回,约
|
||||
100 秒后后台自动转为 ready,未调用手动重试接口。密钥传播等待期间不回退
|
||||
到共享凭证或整桶授权。
|
||||
|
||||
## 已部署版本
|
||||
|
||||
维护发布分支 `poc-org-storage`,代码提交 `07d8566b`,构建版本 `1.0.19`。
|
||||
该分支基于已部署 POC API 镜像标签证实的 `109290ee`,保留线上数据契约。
|
||||
主分支 `poc` 已同步功能,提交 `3fb11571`。未执行全库 schema 切换。
|
||||
|
||||
| 服务 | 就绪副本 | 实际镜像 |
|
||||
|---|---|---|
|
||||
| droboticflow-api | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-api@sha256:faa2fe5335298a3206321397d538774b9e100f5b1c00fe3f6a52c1720c2a08f2` |
|
||||
| droboticflow-worker | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/droboticflow-worker@sha256:c556567d98627bab0f734a5fc0b8cfff406c92c6b726cf23aabf7e0c8040d695` |
|
||||
| drosync-coordinator | 2/2 | `ccr-29eug8s3-pub.cnc.bj.baidubce.com/dataflow-poc/drosync-coordinator@sha256:155b46cd5ee7c82e12fa67efc8f8203d67035c68ff3461e000f7d5450c8fec2e` |
|
||||
|
||||
API 和 Worker 的公开 health 接口均返回 `buildCommit: 07d8566b`、`status: ok`。
|
||||
Coordinator 的 `/health` 和 `/ready` 分别返回健康、ready=true,enabled=true。
|
||||
清除了部署中覆盖镜像版本信息的旧环境变量,运行时版本与镜像摘要已核对。
|
||||
|
||||
## 实测证据
|
||||
|
||||
| 检查 | 结果 |
|
||||
|---|---|
|
||||
| 新组织创建与后台 IAM 配置 | 202 后自动转 ready,账号创建返回 201 |
|
||||
| 新组织默认对象列表 | 定位到自己的顶层目录 |
|
||||
| 平台访问其他目录、读取/写入其他目录 | HTTP 403 |
|
||||
| 组织管理员扩大 rootPrefix 或创建替代的无目录限制连接 | HTTP 403 |
|
||||
| 授权目录内上传、读取、覆盖、删除 | 通过 |
|
||||
| 专用凭证直接访问 BOS:自己目录列举 | HTTP 200 |
|
||||
| 专用凭证直接访问 BOS:桶根、其他目录列举、其他目录读取/写入 | HTTP 403 |
|
||||
| 专用凭证直接访问 BOS:目录内上传、读取、覆盖、删除 | 通过 |
|
||||
| 现有 UBT 登录及整桶根目录列举 | 通过 |
|
||||
| 现有 POC 主验收组织登录及整桶根目录列举 | 通过 |
|
||||
| 现有 UBT / POC 存储连接 metadata 对比 | 未改变 |
|
||||
| 部署前记录的 13 个未结束同步 Session | 全部保留,状态未改变 |
|
||||
| 本地后端回归、路径/符号链接/挂载隔离、异步创建、并发状态保留 | 通过 |
|
||||
| 前端类型检查、255 项维护分支前端测试、构建 | 通过 |
|
||||
| 部署模板测试 | 9 项通过 |
|
||||
| 本地管理页面创建组织交互 | 自动显示 testxyz/company-a/ 和存储已就绪 |
|
||||
|
||||
后台为新组织配置专属目录的默认产物存储。其工作流容器仅挂载本次运行目录
|
||||
和只读的算子代码目录,拒绝共享 PVC/注册表凭据挂载;不能通过符号链接把其他
|
||||
运行目录的文件作为输出发布。旧组织保留原有行为。
|
||||
|
||||
## 清理与交付
|
||||
|
||||
两次真实 API 验收创建的临时组织、业务账号和存储连接已删除(回读均为 0)。
|
||||
其 IAM 密钥先由组织删除流程停用,再清理测试身份和策略;独立 SDK 探针资源
|
||||
也已清理。仅移除了本次测试的空目录标记和测试对象,目录保留登记仍用于防止
|
||||
名字复用造成数据继承。旁路 Deployment 已删除。
|
||||
|
||||
本地交互预览:`http://127.0.0.1:18090/`(独立测试数据,不是线上账号)。
|
||||
实现契约见 [新组织存储配置](new-organization-storage.md)。
|
||||
60
docs/security/new-organization-storage.md
Normal file
60
docs/security/new-organization-storage.md
Normal file
@ -0,0 +1,60 @@
|
||||
# New-organization storage provisioning
|
||||
|
||||
## Contract
|
||||
|
||||
Existing organizations, including UBT and POC Real Acceptance Organization, keep
|
||||
their current `testxyz` whole-bucket connections. Only organizations created after
|
||||
this deployment are enrolled. Their normalized organization slug becomes their
|
||||
immutable top-level root: e.g. `company-a` maps to `testxyz/company-a/`. Renaming
|
||||
an organization does not change its root.
|
||||
|
||||
The POC preset sets `ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID` to its existing
|
||||
POC connection. That credential remains server-side and is used for IAM management.
|
||||
New organizations receive a separate IAM user and AK/SK with only LIST/READ/WRITE
|
||||
on their root, plus READ of bucket metadata. No bucket-wide LIST or FULL_CONTROL
|
||||
is granted. Policy syntax follows the official [BOS IAM reference](https://intl.cloud.baidu.com/zh/doc/BOS/s/5k1rlj1lk-intl).
|
||||
|
||||
Creation reserves the directory permanently, rejects existing data, provisions
|
||||
cloud identity, saves the one-time secret, and verifies directory listing succeeds
|
||||
while root listing returns AccessDenied. A conditional empty directory marker
|
||||
materializes the folder. Artifacts default to that same root. Organization creation returns 202 with a pending state. A background reconciler
|
||||
retries pending provisioning every 15 seconds, and the console polls its status.
|
||||
Pending provisioning cannot add business users; the console also exposes a retry button. The
|
||||
lease prevents concurrent API replicas from issuing competing credentials.
|
||||
IAM propagation can take time; a failed verification retains the scoped connection
|
||||
for retry, never substitutes a shared key. An interrupted key issue is recovered
|
||||
only for an owned identity with no persisted tenant connection. Organization
|
||||
removal disables its IAM keys and retains directory reservations and BOS objects.
|
||||
|
||||
Scope is enforced for object operations and signed URLs, scans, sync staging,
|
||||
publication, cleanup and artifacts. New scoped-organization jobs only mount their
|
||||
run workspace and read-only operator catalog code. Shared PVC mounts and registry
|
||||
credential mounts are rejected. They cannot opt into raw shared filesystem scans.
|
||||
Files and symlinks outside the run directory cannot be externalized as artifacts.
|
||||
Old organizations keep the existing mount and storage behavior.
|
||||
|
||||
## Maintenance release
|
||||
|
||||
The live API image label identifies POC commit `109290ee`. The current `poc` head
|
||||
requires a separate global schema cutover which would interrupt existing sync
|
||||
sessions. This targeted maintenance release branches from the verified live POC
|
||||
ancestor as `poc-org-storage`. It carries the storage feature without the unrelated
|
||||
schema retirement. It never merges or deploys Fanyu/dev code. Build images only
|
||||
under `dataflow-poc`; client packages are not installed by this rollout.
|
||||
|
||||
Release build version is `1.0.19`. Earlier build candidates were refined for inline
|
||||
artifacts and automatic recovery from real IAM key-propagation delays.
|
||||
The main `poc` branch should receive the additive provisioning changes as well, so
|
||||
a future full cutover keeps the same new-organization policy.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- Preserve old UBT/POC connection metadata and whole-bucket listing.
|
||||
- Create a disposable new organization through the real super-admin API.
|
||||
- Verify its storage root, unique IAM key, default artifact policy and directory.
|
||||
- Login as its admin; default listing must start at its own root.
|
||||
- Verify inside upload/read/overwrite/delete; deny root/sibling lists and sibling
|
||||
reads/writes through both platform and direct BOS SDK calls.
|
||||
- Deny connection scope/credential widening and alternative unscoped connections.
|
||||
- Retain active POC sessions, and verify API/Worker/Coordinator rollouts and images.
|
||||
- Disable the disposable organization's keys and clean only its acceptance objects.
|
||||
@ -707,10 +707,34 @@ func (server *server) handleAdminOrganizationsCreate(writer http.ResponseWriter,
|
||||
renderValidationError(writer, validationIssue{Field: "status", Message: "status must be active or disabled"})
|
||||
return
|
||||
}
|
||||
if server.organizationStorageSourceConnectionID != "" {
|
||||
source, err := server.store.GetStorageConnection(request.Context(), server.organizationStorageSourceConnectionID)
|
||||
if err != nil || source == nil || source.Bucket == "" || source.RootPrefix != "" {
|
||||
renderJSON(writer, 503, map[string]string{"error": "default organization storage is unavailable"})
|
||||
return
|
||||
}
|
||||
item.StorageRootPrefix = slug + "/"
|
||||
item.StorageBucket = source.Bucket
|
||||
item.StorageConnectionID = "storage-" + item.ID
|
||||
item.StorageState = "pending"
|
||||
}
|
||||
if err := server.store.UpsertOrganization(request.Context(), item); err != nil {
|
||||
renderOrganizationMutationError(writer, err)
|
||||
return
|
||||
}
|
||||
if item.StorageRootPrefix != "" && server.organizationStorageAsync {
|
||||
go func(org domain.OrganizationRecord) {
|
||||
_ = server.provisionOrganizationStorage(context.Background(), &org)
|
||||
}(item)
|
||||
renderJSON(writer, http.StatusAccepted, item)
|
||||
return
|
||||
}
|
||||
if item.StorageRootPrefix != "" {
|
||||
if err := server.provisionOrganizationStorage(request.Context(), &item); err != nil {
|
||||
renderJSON(writer, http.StatusServiceUnavailable, map[string]any{"error": err.Error(), "organization": item})
|
||||
return
|
||||
}
|
||||
}
|
||||
renderJSON(writer, http.StatusCreated, item)
|
||||
}
|
||||
|
||||
@ -762,6 +786,39 @@ func (server *server) handleAdminOrganizationsUpdate(writer http.ResponseWriter,
|
||||
}
|
||||
|
||||
func (server *server) handleAdminOrganizationsDelete(writer http.ResponseWriter, request *http.Request) {
|
||||
org, lookupErr := server.store.GetOrganization(request.Context(), chi.URLParam(request, "id"))
|
||||
if lookupErr != nil {
|
||||
renderError(writer, 500, lookupErr)
|
||||
return
|
||||
}
|
||||
if org != nil && org.StorageRootPrefix != "" {
|
||||
owner := domain.NewID("storage-delete")
|
||||
acquired, err := server.store.AcquireOrganizationStorageLease(request.Context(), org.ID, owner, time.Now().Add(3*time.Minute))
|
||||
if err != nil || !acquired {
|
||||
renderJSON(writer, 409, map[string]string{"error": "organization storage provisioning is in progress; retry deletion later"})
|
||||
return
|
||||
}
|
||||
defer func() {
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(request.Context()), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = server.store.ReleaseOrganizationStorageLease(ctx, org.ID, owner)
|
||||
}()
|
||||
source, err := server.store.GetStorageConnection(request.Context(), server.organizationStorageSourceConnectionID)
|
||||
if err != nil || source == nil {
|
||||
renderJSON(writer, 503, map[string]string{"error": "cannot revoke organization storage"})
|
||||
return
|
||||
}
|
||||
org.Status = domain.OrganizationStatusDisabled
|
||||
if err := server.store.UpsertOrganization(request.Context(), *org); err != nil {
|
||||
renderError(writer, 500, err)
|
||||
return
|
||||
}
|
||||
if err := organizationStorageRevoke(request.Context(), *source, *org); err != nil {
|
||||
renderError(writer, 503, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
deleted, err := server.store.DeleteOrganizationCascade(request.Context(), chi.URLParam(request, "id"))
|
||||
if err != nil {
|
||||
renderError(writer, http.StatusBadRequest, err)
|
||||
@ -801,6 +858,10 @@ func (server *server) handleAdminOrganizationUsersCreate(writer http.ResponseWri
|
||||
renderJSON(writer, http.StatusNotFound, map[string]string{"error": "Organization not found"})
|
||||
return
|
||||
}
|
||||
if organization.StorageRootPrefix != "" && organization.StorageState != "ready" {
|
||||
renderJSON(writer, http.StatusConflict, map[string]string{"error": "organization storage provisioning must finish before adding users"})
|
||||
return
|
||||
}
|
||||
var payload adminUserMutationRequest
|
||||
if !decodeJSON(writer, request, &payload) {
|
||||
return
|
||||
|
||||
@ -299,6 +299,11 @@ func (server *server) resolveScanConfig(ctx context.Context, scanConfig *domain.
|
||||
if scanConfig == nil {
|
||||
return nil, nil
|
||||
}
|
||||
if principal, ok := ctx.Value(authContextKey{}).(*authenticatedPrincipal); ok && principal != nil && principal.Organization.StorageRootPrefix != "" {
|
||||
if scanConfig.ConnectionID != principal.Organization.StorageConnectionID || chooseDataSource(scanConfig.DataSource) != "BOS" {
|
||||
return nil, domain.ErrStorageScope
|
||||
}
|
||||
}
|
||||
copy := *scanConfig
|
||||
connection, err := server.getStorageConnection(ctx, copy.ConnectionID)
|
||||
if err != nil {
|
||||
@ -310,6 +315,17 @@ func (server *server) resolveScanConfig(ctx context.Context, scanConfig *domain.
|
||||
}
|
||||
return ©, nil
|
||||
}
|
||||
if err := connection.CheckStorageSource(copy.Bucket, copy.Prefix, copy.SourcePath); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
copy.StorageRootPrefix = connection.RootPrefix
|
||||
if connection.RootPrefix != "" && copy.Bucket == "" {
|
||||
copy.Bucket = connection.Bucket
|
||||
}
|
||||
if connection.RootPrefix != "" {
|
||||
copy.AccessKeyID = connection.AccessKeyID
|
||||
copy.SecretAccessKey = connection.SecretAccessKey
|
||||
}
|
||||
copy.Endpoint = server.storageEndpointForProvider(connection.Provider, connection.Endpoint)
|
||||
if strings.TrimSpace(copy.AccessKeyID) == "" {
|
||||
copy.AccessKeyID = connection.AccessKeyID
|
||||
@ -2132,6 +2148,10 @@ func buildSourceCollectionSignedPreviewURL(scanConfig *domain.ScanConfig, source
|
||||
if !ok {
|
||||
return "", fmt.Errorf("invalid BOS source path: %s", sourcePath)
|
||||
}
|
||||
scope := &domain.StorageConnectionRecord{Bucket: scanConfig.Bucket, RootPrefix: scanConfig.StorageRootPrefix}
|
||||
if err := scope.CheckStoragePath(bucket, object, false); err != nil {
|
||||
return "", err
|
||||
}
|
||||
accessKeyID := strings.TrimSpace(scanConfig.AccessKeyID)
|
||||
secretAccessKey := strings.TrimSpace(scanConfig.SecretAccessKey)
|
||||
if accessKeyID == "" || secretAccessKey == "" {
|
||||
|
||||
@ -27,6 +27,7 @@ import (
|
||||
type objectStorageBrowser interface {
|
||||
ListBuckets(context.Context) ([]domain.ObjectStorageBucketSummary, error)
|
||||
ListObjects(context.Context, string, string) ([]domain.ObjectStorageObjectSummary, error)
|
||||
ListObjectsRecursive(context.Context, string, string) ([]domain.ObjectStorageObjectSummary, error)
|
||||
CreateBucket(context.Context, string) (domain.ObjectStorageBucketSummary, error)
|
||||
DeleteBucket(context.Context, string) error
|
||||
PutObject(context.Context, string, string, []byte, string) (domain.ObjectStorageObjectSummary, error)
|
||||
@ -203,6 +204,10 @@ func objectStorageSummaryFromMeta(bucket string, key string, meta *bosapi.GetObj
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ListBuckets(ctx context.Context) ([]domain.ObjectStorageBucketSummary, error) {
|
||||
if browser.connection != nil && browser.connection.RootPrefix != "" {
|
||||
return []domain.ObjectStorageBucketSummary{{Name: browser.connection.Bucket}}, nil
|
||||
}
|
||||
|
||||
result, err := browser.client.ListBucketsWithContext(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@ -219,6 +224,11 @@ func (browser *bosObjectStorageBrowser) ListBuckets(ctx context.Context) ([]doma
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ListObjects(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
|
||||
prefix = browser.connection.StorageBrowsePrefix(prefix)
|
||||
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
items := make([]domain.ObjectStorageObjectSummary, 0)
|
||||
seen := make(map[string]struct{})
|
||||
marker := ""
|
||||
@ -243,6 +253,48 @@ func (browser *bosObjectStorageBrowser) ListObjects(ctx context.Context, bucket
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ListObjectsRecursive(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
|
||||
prefix = browser.connection.StorageBrowsePrefix(prefix)
|
||||
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
items := make([]domain.ObjectStorageObjectSummary, 0)
|
||||
seen := make(map[string]struct{})
|
||||
marker := ""
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
|
||||
for {
|
||||
result, err := browser.client.ListObjectsWithContext(ctx, normalizedBucket, &bosapi.ListObjectsArgs{
|
||||
Prefix: storageObjectRequestPrefix(prefix),
|
||||
MaxKeys: 1000,
|
||||
Marker: strings.TrimSpace(marker),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if result == nil {
|
||||
break
|
||||
}
|
||||
for _, item := range result.Contents {
|
||||
keyValue := strings.Trim(strings.TrimSpace(item.Key), "/")
|
||||
if keyValue == "" || keyValue == normalizedPrefix {
|
||||
continue
|
||||
}
|
||||
if _, exists := seen[keyValue]; exists {
|
||||
continue
|
||||
}
|
||||
seen[keyValue] = struct{}{}
|
||||
items = append(items, objectStorageSummaryFromListObject(normalizedBucket, item))
|
||||
}
|
||||
if !result.IsTruncated || strings.TrimSpace(result.NextMarker) == "" || strings.TrimSpace(result.NextMarker) == marker {
|
||||
break
|
||||
}
|
||||
marker = strings.TrimSpace(result.NextMarker)
|
||||
}
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ListObjectsPage(
|
||||
ctx context.Context,
|
||||
bucket string,
|
||||
@ -250,6 +302,11 @@ func (browser *bosObjectStorageBrowser) ListObjectsPage(
|
||||
marker string,
|
||||
limit int,
|
||||
) (objectStoragePage, error) {
|
||||
prefix = browser.connection.StorageBrowsePrefix(prefix)
|
||||
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return objectStoragePage{}, err
|
||||
}
|
||||
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
|
||||
if limit <= 0 {
|
||||
@ -304,6 +361,10 @@ func (browser *bosObjectStorageBrowser) ListObjectsPage(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) CreateBucket(_ context.Context, bucket string) (domain.ObjectStorageBucketSummary, error) {
|
||||
if browser.connection != nil && browser.connection.RootPrefix != "" {
|
||||
return domain.ObjectStorageBucketSummary{}, domain.ErrStorageScope
|
||||
}
|
||||
|
||||
if _, err := browser.client.PutBucket(strings.TrimSpace(bucket)); err != nil {
|
||||
return domain.ObjectStorageBucketSummary{}, err
|
||||
}
|
||||
@ -311,6 +372,10 @@ func (browser *bosObjectStorageBrowser) CreateBucket(_ context.Context, bucket s
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) DeleteBucket(_ context.Context, bucket string) error {
|
||||
if browser.connection != nil && browser.connection.RootPrefix != "" {
|
||||
return domain.ErrStorageScope
|
||||
}
|
||||
|
||||
return browser.client.DeleteBucket(strings.TrimSpace(bucket))
|
||||
}
|
||||
|
||||
@ -439,6 +504,10 @@ func writeObjectStorageRangeHeaders(writer http.ResponseWriter, summary domain.O
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) PutObject(ctx context.Context, bucket string, key string, payload []byte, contentType string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
args := &bosapi.PutObjectArgs{ContentType: detectObjectStorageContentType(objectKey, payload, contentType)}
|
||||
if _, err := browser.client.PutObjectFromBytesWithContext(ctx, strings.TrimSpace(bucket), objectKey, payload, args); err != nil {
|
||||
@ -457,6 +526,10 @@ func (browser *bosObjectStorageBrowser) PutObjectConditional(
|
||||
ifNoneMatch string,
|
||||
userMeta map[string]string,
|
||||
) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
args := &bosapi.PutObjectArgs{
|
||||
ContentType: detectObjectStorageContentType(objectKey, payload, contentType),
|
||||
@ -471,6 +544,10 @@ func (browser *bosObjectStorageBrowser) PutObjectConditional(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) PutObjectStream(ctx context.Context, bucket string, key string, reader io.Reader, contentType string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
args := &bosapi.PutObjectArgs{ContentType: detectObjectStorageContentType(objectKey, nil, contentType)}
|
||||
if _, err := browser.client.PutObjectFromStreamWithContext(ctx, strings.TrimSpace(bucket), objectKey, reader, args); err != nil {
|
||||
@ -502,6 +579,10 @@ func objectStorageUploadHeaders(contentType string) map[string]string {
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) PresignPutObject(_ context.Context, bucket string, key string, contentType string, expiresInSeconds int) (domain.ObjectStoragePresignedUploadResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStoragePresignedUploadResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
|
||||
headers := objectStorageUploadHeaders(contentType)
|
||||
@ -526,6 +607,10 @@ func (browser *bosObjectStorageBrowser) PresignPutObjectBound(
|
||||
metadata map[string]string,
|
||||
expiresInSeconds int,
|
||||
) (domain.ObjectStoragePresignedUploadResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStoragePresignedUploadResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
|
||||
headers := objectStorageUploadHeaders(contentType)
|
||||
@ -546,6 +631,10 @@ func (browser *bosObjectStorageBrowser) PresignPutObjectBound(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) InitiateMultipartUpload(_ context.Context, bucket string, key string, contentType string, expiresInSeconds int) (domain.ObjectStorageMultipartUploadCreateResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageMultipartUploadCreateResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
result, err := browser.client.InitiateMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(contentType), nil)
|
||||
if err != nil {
|
||||
@ -568,6 +657,10 @@ func (browser *bosObjectStorageBrowser) InitiateMultipartUploadBound(
|
||||
metadata map[string]string,
|
||||
expiresInSeconds int,
|
||||
) (domain.ObjectStorageMultipartUploadCreateResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageMultipartUploadCreateResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
result, err := browser.client.InitiateMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(contentType), nil)
|
||||
if err != nil {
|
||||
@ -583,6 +676,10 @@ func (browser *bosObjectStorageBrowser) InitiateMultipartUploadBound(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) PresignMultipartUploadPart(_ context.Context, bucket string, key string, uploadID string, partNumber int, expiresInSeconds int) (domain.ObjectStorageMultipartUploadPartURLResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageMultipartUploadPartURLResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
|
||||
params := map[string]string{
|
||||
@ -612,6 +709,10 @@ func (browser *bosObjectStorageBrowser) PresignMultipartUploadPartBound(
|
||||
sizeBytes int64,
|
||||
expiresInSeconds int,
|
||||
) (domain.ObjectStorageMultipartUploadPartURLResult, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageMultipartUploadPartURLResult{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
expiresInSeconds = normalizeObjectStoragePresignExpires(expiresInSeconds)
|
||||
params := map[string]string{
|
||||
@ -636,6 +737,10 @@ func (browser *bosObjectStorageBrowser) PresignMultipartUploadPartBound(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) CompleteMultipartUpload(ctx context.Context, bucket string, key string, uploadID string, parts []domain.ObjectStorageMultipartUploadPart) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
return browser.CompleteMultipartUploadBound(ctx, bucket, key, uploadID, parts, nil)
|
||||
}
|
||||
|
||||
@ -647,6 +752,10 @@ func (browser *bosObjectStorageBrowser) CompleteMultipartUploadBound(
|
||||
parts []domain.ObjectStorageMultipartUploadPart,
|
||||
metadata map[string]string,
|
||||
) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
uploadParts := make([]bosapi.UploadInfoType, 0, len(parts))
|
||||
for _, part := range parts {
|
||||
@ -659,6 +768,10 @@ func (browser *bosObjectStorageBrowser) CompleteMultipartUploadBound(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) AbortMultipartUpload(_ context.Context, bucket string, key string, uploadID string) error {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
objectKey := strings.Trim(strings.TrimSpace(key), "/")
|
||||
return browser.client.AbortMultipartUpload(strings.TrimSpace(bucket), objectKey, strings.TrimSpace(uploadID))
|
||||
}
|
||||
@ -691,6 +804,13 @@ func (browser *bosObjectStorageBrowser) PublishMultipartCopyConditional(
|
||||
ifNoneMatch string,
|
||||
userMeta map[string]string,
|
||||
) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, sourceKey, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
if err := browser.connection.CheckStoragePath(bucket, targetKey, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
sourceKey = exactObjectStorageKey(sourceKey)
|
||||
targetKey = exactObjectStorageKey(targetKey)
|
||||
@ -769,6 +889,10 @@ func (browser *bosObjectStorageBrowser) PublishMultipartCopyConditional(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) GetBucketVersioning(_ context.Context, bucket string) (string, error) {
|
||||
if browser.connection != nil && browser.connection.RootPrefix != "" && bucket != browser.connection.Bucket {
|
||||
return "", domain.ErrStorageScope
|
||||
}
|
||||
|
||||
result, err := browser.client.GetBucketVersioning(strings.TrimSpace(bucket))
|
||||
if err != nil {
|
||||
return "", err
|
||||
@ -780,6 +904,13 @@ func (browser *bosObjectStorageBrowser) GetBucketVersioning(_ context.Context, b
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) RenameObject(ctx context.Context, bucket string, key string, newKey string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
if err := browser.connection.CheckStoragePath(bucket, newKey, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := exactObjectStorageKey(key)
|
||||
destinationKey := exactObjectStorageKey(newKey)
|
||||
if objectKey == "" || destinationKey == "" {
|
||||
@ -798,6 +929,13 @@ func (browser *bosObjectStorageBrowser) RenameObject(ctx context.Context, bucket
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) MovePrefix(ctx context.Context, bucket string, prefix string, newPrefix string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
if err := browser.connection.CheckStoragePath(bucket, newPrefix, true); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
sourcePrefix := strings.Trim(strings.TrimSpace(prefix), "/")
|
||||
destinationPrefix := strings.Trim(strings.TrimSpace(newPrefix), "/")
|
||||
@ -900,10 +1038,18 @@ func objectStorageMoveDestination(sourceKey string, destinationPrefix string) st
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) DeleteObject(_ context.Context, bucket string, key string) error {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return browser.client.DeleteObject(strings.TrimSpace(bucket), exactObjectStorageKey(key))
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) DeletePrefix(ctx context.Context, bucket string, prefix string) error {
|
||||
if err := browser.connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
normalizedPrefix := strings.Trim(strings.TrimSpace(prefix), "/")
|
||||
if normalizedBucket == "" || normalizedPrefix == "" {
|
||||
@ -947,6 +1093,10 @@ func (browser *bosObjectStorageBrowser) DeletePrefix(ctx context.Context, bucket
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) GetObjectSummary(ctx context.Context, bucket string, key string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
objectKey := exactObjectStorageKey(key)
|
||||
meta, err := browser.client.GetObjectMetaWithContext(ctx, strings.TrimSpace(bucket), objectKey)
|
||||
if err != nil {
|
||||
@ -956,6 +1106,10 @@ func (browser *bosObjectStorageBrowser) GetObjectSummary(ctx context.Context, bu
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) getObjectSummaryAfterWrite(ctx context.Context, bucket string, key string) (domain.ObjectStorageObjectSummary, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return domain.ObjectStorageObjectSummary{}, err
|
||||
}
|
||||
|
||||
return retryObjectStorageSummary(ctx, objectStorageWriteMetadataRetryAttempts, objectStorageWriteMetadataRetryDelay, func() (domain.ObjectStorageObjectSummary, error) {
|
||||
return browser.GetObjectSummary(ctx, bucket, key)
|
||||
})
|
||||
@ -989,6 +1143,10 @@ func retryObjectStorageSummary(
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ReadObject(ctx context.Context, bucket string, key string) ([]byte, string, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
objectKey := exactObjectStorageKey(key)
|
||||
result, err := browser.client.GetObjectWithContext(ctx, strings.TrimSpace(bucket), objectKey, nil)
|
||||
if err != nil {
|
||||
@ -1003,6 +1161,10 @@ func (browser *bosObjectStorageBrowser) ReadObject(ctx context.Context, bucket s
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ReadObjectRange(ctx context.Context, bucket string, key string, start int64, end int64) (io.ReadCloser, string, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
objectKey := exactObjectStorageKey(key)
|
||||
result, err := browser.client.GetObjectWithContext(ctx, strings.TrimSpace(bucket), objectKey, nil, start, end)
|
||||
if err != nil {
|
||||
@ -1012,6 +1174,10 @@ func (browser *bosObjectStorageBrowser) ReadObjectRange(ctx context.Context, buc
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) ReadObjectPreview(ctx context.Context, bucket string, key string, maxBytes int64) ([]byte, string, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
if maxBytes <= 0 {
|
||||
return browser.ReadObject(ctx, bucket, key)
|
||||
}
|
||||
@ -1036,6 +1202,10 @@ func (browser *bosObjectStorageBrowser) ReadObjectPreview(ctx context.Context, b
|
||||
}
|
||||
|
||||
func (browser *bosObjectStorageBrowser) BuildPreviewURL(_ context.Context, bucket string, key string) (string, error) {
|
||||
if err := browser.connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
previewURL := strings.TrimSpace(browser.client.BasicGeneratePresignedUrl(strings.TrimSpace(bucket), strings.Trim(strings.TrimSpace(key), "/"), 3600))
|
||||
if previewURL == "" {
|
||||
return "", fmt.Errorf("failed to generate BOS preview url")
|
||||
@ -1232,7 +1402,7 @@ func (server *server) handleObjectStorageObjects(writer http.ResponseWriter, req
|
||||
renderValidationError(writer, validationIssue{Field: "bucket", Message: err.Error()})
|
||||
return
|
||||
}
|
||||
prefix := strings.Trim(strings.TrimSpace(query.Get("prefix")), "/")
|
||||
prefix := connection.StorageBrowsePrefix(strings.Trim(strings.TrimSpace(query.Get("prefix")), "/"))
|
||||
marker := strings.TrimSpace(query.Get("marker"))
|
||||
paginationRequested := query.Has("limit") || query.Has("marker")
|
||||
limit := 100
|
||||
@ -2055,3 +2225,93 @@ func (server *server) handleObjectStorageContent(writer http.ResponseWriter, req
|
||||
}
|
||||
writeSourceContent(writer, request, path.Base(key), contentType, payload)
|
||||
}
|
||||
|
||||
type objectStorageDirectoryManifestItem struct {
|
||||
Key string `json:"key"`
|
||||
RelativePath string `json:"relativePath"`
|
||||
SizeBytes int64 `json:"sizeBytes"`
|
||||
DownloadURL string `json:"downloadUrl"`
|
||||
}
|
||||
|
||||
type objectStorageDirectoryManifestResponse struct {
|
||||
Bucket string `json:"bucket"`
|
||||
BasePrefix string `json:"basePrefix"`
|
||||
TotalFiles int `json:"totalFiles"`
|
||||
TotalBytes int64 `json:"totalBytes"`
|
||||
Files []objectStorageDirectoryManifestItem `json:"files"`
|
||||
}
|
||||
|
||||
func (server *server) handleObjectStorageDirectoryManifest(writer http.ResponseWriter, request *http.Request) {
|
||||
connectionID := strings.TrimSpace(request.URL.Query().Get("connectionId"))
|
||||
browser, connection, cleanup, err := server.resolveObjectStorageBrowser(request.Context(), connectionID)
|
||||
if err != nil {
|
||||
renderError(writer, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
if cleanup != nil {
|
||||
defer cleanup()
|
||||
}
|
||||
if browser == nil || connection == nil {
|
||||
renderValidationError(writer, validationIssue{Field: "connectionId", Message: "connectionId is required"})
|
||||
return
|
||||
}
|
||||
bucket, err := resolveObjectStorageBucket(connection, request.URL.Query().Get("bucket"))
|
||||
if err != nil {
|
||||
renderValidationError(writer, validationIssue{Field: "bucket", Message: err.Error()})
|
||||
return
|
||||
}
|
||||
prefix := strings.Trim(strings.TrimSpace(request.URL.Query().Get("prefix")), "/")
|
||||
if bucket == "" {
|
||||
renderValidationError(writer, validationIssue{Field: "bucket", Message: "bucket is required"})
|
||||
return
|
||||
}
|
||||
|
||||
objects, err := browser.ListObjectsRecursive(request.Context(), bucket, prefix)
|
||||
if err != nil {
|
||||
renderObjectStorageWriteError(writer, err)
|
||||
return
|
||||
}
|
||||
|
||||
items := make([]objectStorageDirectoryManifestItem, 0, len(objects))
|
||||
var totalBytes int64
|
||||
basePrefixWithSlash := prefix
|
||||
if basePrefixWithSlash != "" && !strings.HasSuffix(basePrefixWithSlash, "/") {
|
||||
basePrefixWithSlash += "/"
|
||||
}
|
||||
|
||||
for _, obj := range objects {
|
||||
if obj.Kind == "directory" {
|
||||
continue
|
||||
}
|
||||
relPath := obj.Key
|
||||
if basePrefixWithSlash != "" && strings.HasPrefix(obj.Key, basePrefixWithSlash) {
|
||||
relPath = strings.TrimPrefix(obj.Key, basePrefixWithSlash)
|
||||
} else if prefix != "" && strings.HasPrefix(obj.Key, prefix) {
|
||||
relPath = strings.TrimPrefix(obj.Key, prefix)
|
||||
relPath = strings.TrimPrefix(relPath, "/")
|
||||
}
|
||||
if relPath == "" {
|
||||
relPath = path.Base(obj.Key)
|
||||
}
|
||||
downloadURL, urlErr := browser.BuildPreviewURL(request.Context(), bucket, obj.Key)
|
||||
if urlErr != nil {
|
||||
renderError(writer, http.StatusInternalServerError, fmt.Errorf("failed to presign %s: %w", obj.Key, urlErr))
|
||||
return
|
||||
}
|
||||
items = append(items, objectStorageDirectoryManifestItem{
|
||||
Key: obj.Key,
|
||||
RelativePath: relPath,
|
||||
SizeBytes: obj.SizeBytes,
|
||||
DownloadURL: downloadURL,
|
||||
})
|
||||
totalBytes += obj.SizeBytes
|
||||
}
|
||||
|
||||
renderJSON(writer, http.StatusOK, objectStorageDirectoryManifestResponse{
|
||||
Bucket: bucket,
|
||||
BasePrefix: prefix,
|
||||
TotalFiles: len(items),
|
||||
TotalBytes: totalBytes,
|
||||
Files: items,
|
||||
})
|
||||
}
|
||||
|
||||
@ -44,7 +44,10 @@ const (
|
||||
objectSyncReleasePreview = "preview"
|
||||
)
|
||||
|
||||
var objectSyncClientPlatforms = []string{"macos-arm64", "windows-x86_64", "linux-x86_64"}
|
||||
// Keep this list in lockstep with drosync_platform::release_platform_key_for.
|
||||
// S100 devices register as linux/arm64, so their preview artifact must be
|
||||
// required and exposed alongside the existing desktop targets.
|
||||
var objectSyncClientPlatforms = []string{"macos-arm64", "windows-x86_64", "linux-x86_64", "linux-arm64"}
|
||||
|
||||
func normalizedOrganizationID(value string) string {
|
||||
if trimmed := strings.TrimSpace(value); trimmed != "" {
|
||||
@ -527,6 +530,9 @@ func (server *server) objectSyncBrowser(ctx context.Context, space *domain.Objec
|
||||
if connection == nil || normalizedOrganizationID(connection.OrganizationID) != normalizedOrganizationID(space.OrganizationID) {
|
||||
return nil, nil, fmt.Errorf("storage connection not found in the sync space organization")
|
||||
}
|
||||
if err := connection.CheckStoragePath(space.Bucket, space.RootPrefix, true); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
browser, closeBrowser, err := storageConnectionBrowserResolver(ctx, server.storageConnectionWithResolvedEndpoint(connection))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
@ -643,6 +649,10 @@ func (server *server) handleObjectSyncSpaceCreate(writer http.ResponseWriter, re
|
||||
renderObjectSyncError(writer, request, http.StatusBadRequest, "invalid_storage_connection", "The storage connection is not available in this organization", false, nil)
|
||||
return
|
||||
}
|
||||
if err := connection.CheckStoragePath(input.Bucket, input.RootPrefix, true); err != nil {
|
||||
renderError(writer, http.StatusForbidden, err)
|
||||
return
|
||||
}
|
||||
policy := domain.DefaultObjectSyncSpacePolicy()
|
||||
if input.Policy != nil {
|
||||
policy = *input.Policy
|
||||
@ -1314,10 +1324,7 @@ func (server *server) handleObjectSyncSessionAdminAbort(writer http.ResponseWrit
|
||||
renderObjectSyncError(writer, request, http.StatusNotFound, "session_not_found", "Object sync session not found", false, nil)
|
||||
return
|
||||
}
|
||||
item.Status = domain.ObjectSyncSessionAborted
|
||||
item.UpdatedAt = nowISO()
|
||||
item.CompletedAt = item.UpdatedAt
|
||||
if err := target.UpsertObjectSyncSession(request.Context(), *item); err != nil {
|
||||
if err := abortObjectSyncSession(request.Context(), target, item); err != nil {
|
||||
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
|
||||
return
|
||||
}
|
||||
@ -2353,9 +2360,56 @@ func (server *server) externalObjectSyncItem(writer http.ResponseWriter, request
|
||||
renderObjectSyncError(writer, request, http.StatusNotFound, "item_not_found", "Object sync item not found for this device credential", false, nil)
|
||||
return nil, nil, principal, false
|
||||
}
|
||||
if session.Status == domain.ObjectSyncSessionAborted || session.Status == domain.ObjectSyncSessionExpired || session.Status == domain.ObjectSyncSessionFailed {
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "session_closed", "Session is no longer active", false, nil)
|
||||
return nil, nil, principal, false
|
||||
}
|
||||
return item, session, principal, true
|
||||
}
|
||||
|
||||
// Persist the terminal session before releasing leases so new client operations
|
||||
// cannot extend an aborted session. Retrying abort also retries lease cleanup.
|
||||
func abortObjectSyncSession(ctx context.Context, target store.ObjectSyncStore, session *domain.ObjectSyncSessionRecord) error {
|
||||
if session.Status == domain.ObjectSyncSessionCompleted || session.Status == domain.ObjectSyncSessionPartial || session.Status == domain.ObjectSyncSessionConflicted {
|
||||
return nil
|
||||
}
|
||||
session.Status = domain.ObjectSyncSessionAborted
|
||||
session.UpdatedAt = nowISO()
|
||||
session.CompletedAt = session.UpdatedAt
|
||||
if err := target.UpsertObjectSyncSession(ctx, *session); err != nil {
|
||||
return err
|
||||
}
|
||||
cursor := ""
|
||||
for {
|
||||
page, err := target.ListObjectSyncItems(ctx, domain.ObjectSyncListQuery{SessionID: session.ID, Cursor: cursor, Limit: 1000})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
updates := make([]domain.ObjectSyncItemRecord, 0, len(page.Items))
|
||||
for _, item := range page.Items {
|
||||
if item.LeaseID != "" {
|
||||
if err := target.ReleaseObjectSyncLease(ctx, item.LeaseID, session.ID, item.ID, item.FencingToken); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
switch item.Status {
|
||||
case domain.ObjectSyncItemCommitted, domain.ObjectSyncItemKeptBoth, domain.ObjectSyncItemUnchanged, domain.ObjectSyncItemSkipped, domain.ObjectSyncItemDenied, domain.ObjectSyncItemFailed, domain.ObjectSyncItemAborted, domain.ObjectSyncItemExpired:
|
||||
continue
|
||||
}
|
||||
item.Status = domain.ObjectSyncItemAborted
|
||||
item.UpdatedAt = nowISO()
|
||||
updates = append(updates, item)
|
||||
}
|
||||
if err := target.UpsertObjectSyncItems(ctx, updates); err != nil {
|
||||
return err
|
||||
}
|
||||
if page.NextCursor == "" {
|
||||
return nil
|
||||
}
|
||||
cursor = page.NextCursor
|
||||
}
|
||||
}
|
||||
|
||||
func objectSyncLeaseResourceHash(space domain.ObjectSyncSpaceRecord, key string) string {
|
||||
sum := sha256.Sum256([]byte(space.ConnectionID + "\x00" + space.Bucket + "\x00" + key))
|
||||
return hex.EncodeToString(sum[:])
|
||||
@ -2383,6 +2437,15 @@ func (server *server) handleExternalObjectSyncItemReserve(writer http.ResponseWr
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "space_not_found", "Session sync space is unavailable", false, nil)
|
||||
return
|
||||
}
|
||||
connection, err := server.store.GetStorageConnection(request.Context(), space.ConnectionID)
|
||||
if err != nil || connection == nil {
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "storage_unavailable", "Storage connection is unavailable", false, nil)
|
||||
return
|
||||
}
|
||||
if err := connection.CheckStoragePath(space.Bucket, item.TargetKey, false); err != nil {
|
||||
renderError(writer, http.StatusForbidden, err)
|
||||
return
|
||||
}
|
||||
leaseTemplate := domain.ObjectSyncLeaseRecord{
|
||||
ID: "osl_" + objectSyncLeaseResourceHash(*space, item.TargetKey),
|
||||
OrganizationID: session.OrganizationID,
|
||||
@ -2412,7 +2475,7 @@ func (server *server) handleExternalObjectSyncItemReserve(writer http.ResponseWr
|
||||
}
|
||||
item.LeaseID = lease.ID
|
||||
item.FencingToken = lease.FencingToken
|
||||
item.StagingKey = path.Join(".drosync-internal/staging", session.OrganizationID, session.ID, item.ID)
|
||||
item.StagingKey = connection.InternalStorageKey(path.Join(".drosync-internal/staging", session.OrganizationID, session.ID, item.ID))
|
||||
item.Status = domain.ObjectSyncItemReserved
|
||||
item.UpdatedAt = nowISO()
|
||||
session.Status = domain.ObjectSyncSessionRunning
|
||||
@ -2768,6 +2831,12 @@ func (server *server) handleExternalObjectSyncHeartbeat(writer http.ResponseWrit
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
switch item.Status {
|
||||
case domain.ObjectSyncItemReserved, domain.ObjectSyncItemUploading, domain.ObjectSyncItemUploaded, domain.ObjectSyncItemCommitting:
|
||||
default:
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "item_not_active", "Only active leased items may send heartbeats", false, nil)
|
||||
return
|
||||
}
|
||||
if item.LeaseID == "" {
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "lease_missing", "Item has no active lease", false, nil)
|
||||
return
|
||||
@ -2881,35 +2950,17 @@ func (server *server) handleExternalObjectSyncSessionFinish(writer http.Response
|
||||
}
|
||||
action := chi.URLParam(request, "action")
|
||||
if action == "abort" {
|
||||
session.Status = domain.ObjectSyncSessionAborted
|
||||
cursor := ""
|
||||
for {
|
||||
page, err := target.ListObjectSyncItems(request.Context(), domain.ObjectSyncListQuery{SessionID: session.ID, Cursor: cursor, Limit: 1000})
|
||||
if err != nil {
|
||||
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
|
||||
return
|
||||
}
|
||||
updates := make([]domain.ObjectSyncItemRecord, 0, len(page.Items))
|
||||
for _, item := range page.Items {
|
||||
switch item.Status {
|
||||
case domain.ObjectSyncItemCommitted, domain.ObjectSyncItemKeptBoth, domain.ObjectSyncItemUnchanged, domain.ObjectSyncItemSkipped, domain.ObjectSyncItemDenied, domain.ObjectSyncItemFailed, domain.ObjectSyncItemAborted, domain.ObjectSyncItemExpired:
|
||||
continue
|
||||
default:
|
||||
item.Status = domain.ObjectSyncItemAborted
|
||||
item.UpdatedAt = nowISO()
|
||||
updates = append(updates, item)
|
||||
}
|
||||
}
|
||||
if err := target.UpsertObjectSyncItems(request.Context(), updates); err != nil {
|
||||
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
|
||||
return
|
||||
}
|
||||
if page.NextCursor == "" {
|
||||
break
|
||||
}
|
||||
cursor = page.NextCursor
|
||||
if err := abortObjectSyncSession(request.Context(), target, session); err != nil {
|
||||
renderObjectSyncError(writer, request, http.StatusInternalServerError, "session_abort_failed", err.Error(), true, nil)
|
||||
return
|
||||
}
|
||||
renderJSON(writer, http.StatusOK, session)
|
||||
return
|
||||
} else if action == "complete" {
|
||||
if session.Status == domain.ObjectSyncSessionAborted || session.Status == domain.ObjectSyncSessionExpired || session.Status == domain.ObjectSyncSessionFailed {
|
||||
renderObjectSyncError(writer, request, http.StatusConflict, "session_closed", "Session is no longer active", false, nil)
|
||||
return
|
||||
}
|
||||
counts := domain.ObjectSyncSessionCounts{Manifest: session.Counts.Manifest}
|
||||
cursor := ""
|
||||
for {
|
||||
|
||||
99
internal/api/object_sync_abort_test.go
Normal file
99
internal/api/object_sync_abort_test.go
Normal file
@ -0,0 +1,99 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestObjectSyncAbortReleasesLeaseAndRejectsClosedOperations(t *testing.T) {
|
||||
for _, admin := range []bool{false, true} {
|
||||
name := "device"
|
||||
if admin {
|
||||
name = "admin"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
handler, data := objectSyncTestHandler(t, true)
|
||||
target, _ := store.AsObjectSyncStore(data)
|
||||
ctx := t.Context()
|
||||
org := domain.DefaultOrganizationID
|
||||
token := "abort-test-token"
|
||||
space := domain.ObjectSyncSpaceRecord{ID: "abort-space", OrganizationID: org, ConnectionID: "bos-connection", Bucket: "test", Enabled: true, Mode: "create_only", Policy: domain.DefaultObjectSyncSpacePolicy(), Capability: domain.ObjectSyncCapabilityRecord{State: domain.ObjectSyncCapabilityCreateOnly, ConditionalCreate: true}}
|
||||
cred := domain.ObjectSyncCredentialRecord{ID: "abort-credential", OrganizationID: org, DeviceID: "abort-device", Enabled: true, TokenHash: sessionTokenHash(token), Scopes: []domain.ObjectSyncCredentialScope{{SyncSpaceID: space.ID, AllowedRelativePrefixes: []string{""}, Actions: []string{"plan", "create"}}}}
|
||||
must := func(err error) {
|
||||
t.Helper()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
must(target.UpsertObjectSyncSpace(ctx, space))
|
||||
must(target.UpsertObjectSyncCredential(ctx, cred))
|
||||
must(target.UpsertObjectSyncDevice(ctx, domain.ObjectSyncDeviceRecord{ID: cred.DeviceID, OrganizationID: org, Status: "active"}))
|
||||
session := domain.ObjectSyncSessionRecord{ID: "abort-session", OrganizationID: org, SyncSpaceID: space.ID, CredentialID: cred.ID, DeviceID: cred.DeviceID, Status: domain.ObjectSyncSessionRunning}
|
||||
must(target.UpsertObjectSyncSession(ctx, session))
|
||||
lease, ok, err := target.AcquireObjectSyncLease(ctx, domain.ObjectSyncLeaseRecord{ID: "abort-lease", OrganizationID: org, ResourceHash: "abort-resource", OwnerSessionID: session.ID, OwnerItemID: "abort-item"}, time.Minute)
|
||||
must(err)
|
||||
if !ok {
|
||||
t.Fatal("lease not acquired")
|
||||
}
|
||||
item := domain.ObjectSyncItemRecord{ID: "abort-item", OrganizationID: org, SessionID: session.ID, Status: domain.ObjectSyncItemUploading, LeaseID: lease.ID, FencingToken: lease.FencingToken}
|
||||
committed := domain.ObjectSyncItemRecord{ID: "committed-item", OrganizationID: org, SessionID: session.ID, Status: domain.ObjectSyncItemCommitted}
|
||||
must(target.UpsertObjectSyncItems(ctx, []domain.ObjectSyncItemRecord{item, committed}))
|
||||
request := func(path, key string, management bool) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest(http.MethodPost, path, bytes.NewBufferString("{}"))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
r.Header.Set("Idempotency-Key", key)
|
||||
if !management {
|
||||
r.Header.Set(objectSyncCredentialHeader, token)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
abortPath := "/api/external/object-sync/sessions/" + session.ID + "/abort"
|
||||
if admin {
|
||||
abortPath = "/api/object-sync/sessions/" + session.ID + "/admin-abort"
|
||||
}
|
||||
w := request(abortPath, "abort-first", admin)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("abort %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
got, _ := target.GetObjectSyncItem(ctx, item.ID)
|
||||
if got.Status != domain.ObjectSyncItemAborted {
|
||||
t.Fatalf("item status %s", got.Status)
|
||||
}
|
||||
saved, _ := target.GetObjectSyncItem(ctx, committed.ID)
|
||||
if saved.Status != domain.ObjectSyncItemCommitted {
|
||||
t.Fatal("committed item changed")
|
||||
}
|
||||
next, acquired, err := target.AcquireObjectSyncLease(ctx, domain.ObjectSyncLeaseRecord{ID: lease.ID, OrganizationID: org, ResourceHash: lease.ResourceHash, OwnerSessionID: "next-session", OwnerItemID: "next-item"}, time.Minute)
|
||||
must(err)
|
||||
if !acquired || next.FencingToken <= lease.FencingToken {
|
||||
t.Fatal("lease not released / fencing not increased")
|
||||
}
|
||||
for _, action := range []string{"heartbeat", "reserve", "commit"} {
|
||||
w = request("/api/external/object-sync/items/"+item.ID+"/"+action, "closed-"+action, false)
|
||||
if w.Code != 409 {
|
||||
t.Fatalf("%s status %d %s", action, w.Code, w.Body.String())
|
||||
}
|
||||
}
|
||||
w = request(abortPath, "abort-repeat", admin)
|
||||
if w.Code != 200 {
|
||||
t.Fatalf("repeat abort %d", w.Code)
|
||||
}
|
||||
renewed, err := target.RenewObjectSyncLease(ctx, next.ID, "next-session", "next-item", next.FencingToken, time.Minute)
|
||||
must(err)
|
||||
if !renewed {
|
||||
t.Fatal("repeat abort released new owner's lease")
|
||||
}
|
||||
w = request("/api/external/object-sync/sessions/"+session.ID+"/complete", "closed-complete", false)
|
||||
if w.Code != 409 {
|
||||
t.Fatalf("closed complete %d", w.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
357
internal/api/organization_storage.go
Normal file
357
internal/api/organization_storage.go
Normal file
@ -0,0 +1,357 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"github.com/baidubce/bce-sdk-go/auth"
|
||||
"github.com/baidubce/bce-sdk-go/bce"
|
||||
"github.com/baidubce/bce-sdk-go/services/bos"
|
||||
bosapi "github.com/baidubce/bce-sdk-go/services/bos/api"
|
||||
"github.com/baidubce/bce-sdk-go/services/iam"
|
||||
iamapi "github.com/baidubce/bce-sdk-go/services/iam/api"
|
||||
"github.com/go-chi/chi/v5"
|
||||
)
|
||||
|
||||
type organizationStorageTransport struct{ ctx context.Context }
|
||||
|
||||
func (t organizationStorageTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
return http.DefaultTransport.RoundTrip(r.WithContext(t.ctx))
|
||||
}
|
||||
|
||||
func organizationStoragePolicy(bucket, root string) string {
|
||||
raw, _ := json.Marshal(map[string]any{"accessControlList": []any{
|
||||
map[string]any{"service": "bce:bos", "region": "*", "effect": "Allow", "permission": []string{"LIST", "READ", "WRITE"}, "resource": []string{bucket + "/" + root, bucket + "/" + root + "*"}},
|
||||
map[string]any{"service": "bce:bos", "region": "*", "effect": "Allow", "permission": []string{"READ"}, "resource": []string{bucket}},
|
||||
}})
|
||||
return string(raw)
|
||||
}
|
||||
func cloudStatus(err error, code int) bool {
|
||||
var e *bce.BceServiceError
|
||||
return errors.As(err, &e) && e.StatusCode == code
|
||||
}
|
||||
func samePolicy(a, b string) bool {
|
||||
var x, y any
|
||||
if json.Unmarshal([]byte(a), &x) != nil || json.Unmarshal([]byte(b), &y) != nil {
|
||||
return false
|
||||
}
|
||||
if m, ok := x.(map[string]any); ok {
|
||||
delete(m, "id")
|
||||
}
|
||||
if m, ok := y.(map[string]any); ok {
|
||||
delete(m, "id")
|
||||
}
|
||||
l, _ := json.Marshal(x)
|
||||
r, _ := json.Marshal(y)
|
||||
return string(l) == string(r)
|
||||
}
|
||||
|
||||
// Replaced only in tests. No master credential is copied to a tenant connection.
|
||||
var organizationStorageProvision = provisionBOSOrganizationStorage
|
||||
var organizationStorageVerify = verifyOrganizationBOSScope
|
||||
var organizationDirectoryEmpty = bosOrganizationDirectoryEmpty
|
||||
|
||||
func bosOrganizationDirectoryEmpty(ctx context.Context, c domain.StorageConnectionRecord, root string) (bool, error) {
|
||||
client, err := bos.NewClient(c.AccessKeyID, c.SecretAccessKey, c.Endpoint)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
client.Config.Retry = bce.NewNoRetryPolicy()
|
||||
page, err := client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{Prefix: root, MaxKeys: 1})
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return len(page.Contents) == 0 && len(page.CommonPrefixes) == 0, nil
|
||||
}
|
||||
|
||||
func provisionBOSOrganizationStorage(ctx context.Context, source domain.StorageConnectionRecord, org domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
|
||||
out := domain.StorageConnectionRecord{ID: org.StorageConnectionID, OrganizationID: org.ID, Name: "BOS Workspace", Provider: domain.StorageProviderBOS, Endpoint: source.Endpoint, Bucket: org.StorageBucket, RootPrefix: org.StorageRootPrefix, CreatedAt: nowISO(), UpdatedAt: nowISO()}
|
||||
root, err := domain.NormalizeStorageRoot(org.StorageRootPrefix)
|
||||
if err != nil || root == "" || source.Bucket != org.StorageBucket || source.RootPrefix != "" {
|
||||
return out, fmt.Errorf("invalid organization storage scope")
|
||||
}
|
||||
client, err := organizationIAMClient(ctx, source)
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
name := "poc-org-" + strings.TrimPrefix(org.ID, "org-")
|
||||
description := "Droboticflow organization " + org.ID
|
||||
user, err := client.GetUser(name)
|
||||
if cloudStatus(err, 404) {
|
||||
_, err = client.CreateUser(&iamapi.CreateUserArgs{Name: name, Description: description})
|
||||
} else if err == nil && user.Description != description {
|
||||
return out, fmt.Errorf("IAM identity ownership mismatch")
|
||||
}
|
||||
if err != nil {
|
||||
return out, iamOperationError("user provisioning", err)
|
||||
}
|
||||
policyName := strings.ReplaceAll(name, "-", "_")
|
||||
policy := organizationStoragePolicy(org.StorageBucket, root)
|
||||
existing, err := client.GetPolicy(policyName, "")
|
||||
if cloudStatus(err, 404) {
|
||||
_, err = client.CreatePolicy(&iamapi.CreatePolicyArgs{Name: policyName, Description: description, Document: policy})
|
||||
} else if err == nil && !samePolicy(existing.Document, policy) {
|
||||
return out, fmt.Errorf("IAM policy differs from the organization directory")
|
||||
}
|
||||
if err != nil {
|
||||
return out, iamOperationError("policy provisioning", err)
|
||||
}
|
||||
if err = client.AttachPolicyToUser(&iamapi.AttachPolicyToUserArgs{UserName: name, PolicyName: policyName, PolicyType: ""}); err != nil {
|
||||
return out, iamOperationError("policy attachment", err)
|
||||
}
|
||||
policies, err := client.ListUserAttachedPolicies(name)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("cannot verify IAM policies")
|
||||
}
|
||||
for _, p := range policies.Policies {
|
||||
if p.Name != policyName {
|
||||
return out, fmt.Errorf("IAM user has unexpected policies")
|
||||
}
|
||||
}
|
||||
groups, err := client.ListGroupsForUser(name)
|
||||
if err != nil || len(groups.Groups) != 0 {
|
||||
return out, fmt.Errorf("IAM user group isolation could not be verified")
|
||||
}
|
||||
keys, err := client.ListAccessKey(name)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("IAM key lookup failed")
|
||||
}
|
||||
for _, key := range keys.AccessKeys {
|
||||
if key.Enabled {
|
||||
if key.Secret == "" {
|
||||
// No persisted tenant connection exists when this provisioner runs.
|
||||
// Recover an interrupted key-creation response by revoking only
|
||||
// this owned, unissued key, then creating its replacement.
|
||||
if err := client.DeleteAccessKey(name, key.Id); err != nil {
|
||||
return out, fmt.Errorf("cannot recover unissued IAM key")
|
||||
}
|
||||
continue
|
||||
}
|
||||
out.AccessKeyID = key.Id
|
||||
out.SecretAccessKey = key.Secret
|
||||
break
|
||||
}
|
||||
}
|
||||
if out.AccessKeyID == "" {
|
||||
key, e := client.CreateAccessKey(name)
|
||||
if e != nil {
|
||||
return out, iamOperationError("key creation", e)
|
||||
}
|
||||
out.AccessKeyID = key.Id
|
||||
out.SecretAccessKey = key.Secret
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func verifyOrganizationBOSScope(ctx context.Context, c domain.StorageConnectionRecord) error {
|
||||
client, err := bos.NewClient(c.AccessKeyID, c.SecretAccessKey, c.Endpoint)
|
||||
if err != nil {
|
||||
return fmt.Errorf("BOS client initialization failed")
|
||||
}
|
||||
client.Config.Retry = bce.NewNoRetryPolicy()
|
||||
// IAM propagation is checked before enabling a tenant. Retry is bounded by the
|
||||
// caller's deadline; failures retain a pending organization for explicit retry.
|
||||
for attempt := 0; attempt < 45; attempt++ {
|
||||
_, err = client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{Prefix: c.RootPrefix, MaxKeys: 1})
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return iamOperationError("BOS authorized directory", err)
|
||||
}
|
||||
if _, err = client.ListObjectsWithContext(ctx, c.Bucket, &bosapi.ListObjectsArgs{MaxKeys: 1}); !bosScopeDenied(err) {
|
||||
return fmt.Errorf("BOS root listing was not denied")
|
||||
}
|
||||
_, err = client.PutObjectFromStringWithContext(ctx, c.Bucket, c.RootPrefix, "", &bosapi.PutObjectArgs{ContentType: "application/x-directory", IfNoneMatch: "*"})
|
||||
if err != nil && !cloudStatus(err, 412) {
|
||||
return fmt.Errorf("cannot create organization directory")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func bosScopeDenied(err error) bool {
|
||||
var e *bce.BceServiceError
|
||||
return errors.As(err, &e) && e.StatusCode == 403 && e.Code == "AccessDenied"
|
||||
}
|
||||
|
||||
func (s *server) provisionOrganizationStorage(ctx context.Context, org *domain.OrganizationRecord) error {
|
||||
if org.StorageRootPrefix == "" {
|
||||
return fmt.Errorf("existing organizations are not enrolled automatically")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
defer cancel()
|
||||
owner := domain.NewID("storage-claim")
|
||||
ok, err := s.store.AcquireOrganizationStorageLease(ctx, org.ID, owner, time.Now().Add(3*time.Minute))
|
||||
if err != nil || !ok {
|
||||
return fmt.Errorf("organization storage provisioning is already in progress")
|
||||
}
|
||||
defer func() {
|
||||
cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = s.store.ReleaseOrganizationStorageLease(cleanup, org.ID, owner)
|
||||
}()
|
||||
current, err := s.store.GetOrganization(ctx, org.ID)
|
||||
if err != nil || current == nil {
|
||||
return fmt.Errorf("organization is unavailable")
|
||||
}
|
||||
*org = *current
|
||||
if org.StorageState == "ready" {
|
||||
return nil
|
||||
}
|
||||
source, err := s.store.GetStorageConnection(ctx, s.organizationStorageSourceConnectionID)
|
||||
if err != nil || source == nil || source.Bucket != org.StorageBucket || source.RootPrefix != "" {
|
||||
return fmt.Errorf("organization storage source is unavailable")
|
||||
}
|
||||
if err = s.store.ReserveOrganizationStorageRoot(ctx, org.StorageBucket, org.StorageRootPrefix, org.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
existing, err := s.store.GetStorageConnection(ctx, org.StorageConnectionID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if existing == nil {
|
||||
// A claimed name must never inherit pre-existing data from the shared bucket.
|
||||
empty, e := organizationDirectoryEmpty(ctx, *s.storageConnectionWithResolvedEndpoint(source), org.StorageRootPrefix)
|
||||
if e != nil {
|
||||
return fmt.Errorf("cannot check organization directory")
|
||||
}
|
||||
if !empty {
|
||||
return fmt.Errorf("directory already contains data; choose another organization slug")
|
||||
}
|
||||
connection, e := organizationStorageProvision(ctx, *s.storageConnectionWithResolvedEndpoint(source), *org)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if e = s.store.UpsertStorageConnection(ctx, connection); e != nil {
|
||||
return fmt.Errorf("cannot persist organization storage; retry provisioning")
|
||||
}
|
||||
} else if existing.RootPrefix != org.StorageRootPrefix || existing.Bucket != org.StorageBucket || existing.OrganizationID != org.ID {
|
||||
return fmt.Errorf("organization connection scope mismatch")
|
||||
}
|
||||
saved, err := s.store.GetStorageConnection(ctx, org.StorageConnectionID)
|
||||
if err != nil || saved == nil {
|
||||
return fmt.Errorf("organization connection is unavailable")
|
||||
}
|
||||
if err := organizationStorageVerify(ctx, *saved); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.store.UpsertArtifactStorageSettings(ctx, org.ID, domain.ArtifactStorageSettings{ConnectionID: org.StorageConnectionID, Bucket: org.StorageBucket, PrefixTemplate: org.StorageRootPrefix + "artifacts/{projectId}/{runId}/{nodeId}"}); err != nil {
|
||||
return fmt.Errorf("cannot configure organization artifact storage")
|
||||
}
|
||||
if err := s.store.MarkOrganizationStorageReady(ctx, org.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
updated, err := s.store.GetOrganization(ctx, org.ID)
|
||||
if err != nil || updated == nil {
|
||||
return fmt.Errorf("organization is unavailable")
|
||||
}
|
||||
*org = *updated
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *server) handleAdminOrganizationStorageProvision(w http.ResponseWriter, r *http.Request) {
|
||||
org, err := s.store.GetOrganization(r.Context(), chi.URLParam(r, "id"))
|
||||
if err != nil || org == nil {
|
||||
renderJSON(w, 404, map[string]string{"error": "organization not found"})
|
||||
return
|
||||
}
|
||||
if err = s.provisionOrganizationStorage(r.Context(), org); err != nil {
|
||||
renderJSON(w, 503, map[string]any{"error": err.Error(), "organization": org})
|
||||
return
|
||||
}
|
||||
renderJSON(w, 200, org)
|
||||
}
|
||||
|
||||
func iamOperationError(stage string, err error) error {
|
||||
var e *bce.BceServiceError
|
||||
if errors.As(err, &e) {
|
||||
return fmt.Errorf("IAM %s failed (HTTP %d, %s)", stage, e.StatusCode, e.Code)
|
||||
}
|
||||
return fmt.Errorf("IAM %s failed", stage)
|
||||
}
|
||||
|
||||
func organizationIAMClient(ctx context.Context, source domain.StorageConnectionRecord) (*iam.Client, error) {
|
||||
client, err := iam.NewClientWithEndpoint(source.AccessKeyID, source.SecretAccessKey, "https://iam.bj.baidubce.com")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("IAM client initialization failed")
|
||||
}
|
||||
client.Config.Retry = bce.NewNoRetryPolicy()
|
||||
timeout := 15 * time.Second
|
||||
client.Config.HTTPClient = &http.Client{Transport: organizationStorageTransport{ctx}, Timeout: timeout}
|
||||
client.Config.HTTPClientTimeout = &timeout
|
||||
client.BceClient, err = bce.NewBceClientWithExclusiveHTTPClient(client.Config, &auth.BceV1Signer{})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("IAM client initialization failed")
|
||||
}
|
||||
return client, nil
|
||||
}
|
||||
|
||||
var organizationStorageRevoke = revokeBOSOrganizationStorage
|
||||
|
||||
func revokeBOSOrganizationStorage(ctx context.Context, source domain.StorageConnectionRecord, org domain.OrganizationRecord) error {
|
||||
client, err := organizationIAMClient(ctx, source)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
name := "poc-org-" + strings.TrimPrefix(org.ID, "org-")
|
||||
user, err := client.GetUser(name)
|
||||
if cloudStatus(err, 404) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return iamOperationError("identity lookup", err)
|
||||
}
|
||||
if user.Description != "Droboticflow organization "+org.ID {
|
||||
return fmt.Errorf("IAM identity ownership mismatch")
|
||||
}
|
||||
keys, err := client.ListAccessKey(name)
|
||||
if err != nil {
|
||||
return iamOperationError("key lookup", err)
|
||||
}
|
||||
for _, key := range keys.AccessKeys {
|
||||
if key.Enabled {
|
||||
if _, err := client.DisableAccessKey(name, key.Id); err != nil {
|
||||
return iamOperationError("key revocation", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Pending organizations are a durable queue. Leases serialize replicas and
|
||||
// the per-organization deadline bounds every IAM/BOS attempt.
|
||||
func (s *server) startOrganizationStorageReconciler() {
|
||||
go func() {
|
||||
ticker := time.NewTicker(15 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for range ticker.C {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
organizations, err := s.store.ListOrganizations(ctx)
|
||||
cancel()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
processed := 0
|
||||
for _, org := range organizations {
|
||||
if org.StorageRootPrefix != "" && org.StorageState == "pending" && org.Status == domain.OrganizationStatusActive {
|
||||
_ = s.provisionOrganizationStorage(context.Background(), &org)
|
||||
processed++
|
||||
if processed >= 20 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
180
internal/api/organization_storage_test.go
Normal file
180
internal/api/organization_storage_test.go
Normal file
@ -0,0 +1,180 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/config"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNewOrganizationsGetScopedStorageAndOldOnesStayWholeBucket(t *testing.T) {
|
||||
data := store.NewMemoryStore()
|
||||
source := domain.StorageConnectionRecord{ID: "source", OrganizationID: domain.DefaultOrganizationID, Name: "BOS", Provider: "BOS", Bucket: "testxyz", Endpoint: "https://bj.bcebos.com", AccessKeyID: "master-ak", SecretAccessKey: "master-sk"}
|
||||
if err := data.UpsertStorageConnection(t.Context(), source); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := domain.OrganizationRecord{ID: "old-ubt", Slug: "old-ubt", Name: "UBT", Status: "active"}
|
||||
data.UpsertOrganization(t.Context(), old)
|
||||
browser := newFakeObjectStorageBrowser()
|
||||
browser.objects = map[string]domain.ObjectStorageObjectSummary{}
|
||||
browser.content = map[string][]byte{}
|
||||
oldResolver, oldProvision := storageConnectionBrowserResolver, organizationStorageProvision
|
||||
oldVerify := organizationStorageVerify
|
||||
oldEmpty := organizationDirectoryEmpty
|
||||
organizationDirectoryEmpty = func(context.Context, domain.StorageConnectionRecord, string) (bool, error) { return true, nil }
|
||||
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
|
||||
t.Cleanup(func() {
|
||||
organizationDirectoryEmpty = oldEmpty
|
||||
organizationStorageVerify = oldVerify
|
||||
storageConnectionBrowserResolver = oldResolver
|
||||
organizationStorageProvision = oldProvision
|
||||
})
|
||||
storageConnectionBrowserResolver = func(context.Context, *domain.StorageConnectionRecord) (objectStorageBrowser, func(), error) {
|
||||
return browser, func() {}, nil
|
||||
}
|
||||
calls := 0
|
||||
fail := false
|
||||
organizationStorageProvision = func(_ context.Context, s domain.StorageConnectionRecord, o domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
|
||||
calls++
|
||||
if fail {
|
||||
return domain.StorageConnectionRecord{}, errors.New("IAM unavailable")
|
||||
}
|
||||
return domain.StorageConnectionRecord{ID: o.StorageConnectionID, OrganizationID: o.ID, Name: "BOS Workspace", Provider: "BOS", Bucket: o.StorageBucket, RootPrefix: o.StorageRootPrefix, AccessKeyID: "tenant-ak", SecretAccessKey: "tenant-sk"}, nil
|
||||
}
|
||||
h := NewHandlerWithConfig(data, nil, config.AppConfig{RequireSessionAuth: true, OrganizationStorageSourceConnectionID: "source"})
|
||||
root := bootstrapSuperAdminSession(t, h)
|
||||
created := adminCreateOrganization(t, h, root, "Company A", "company-a")
|
||||
if created.StorageRootPrefix != "company-a/" || created.StorageState != "ready" {
|
||||
t.Fatalf("organization %+v", created)
|
||||
}
|
||||
c, _ := data.GetStorageConnection(t.Context(), created.StorageConnectionID)
|
||||
if c == nil || c.RootPrefix != "company-a/" || c.AccessKeyID == source.AccessKeyID {
|
||||
t.Fatal("tenant inherited master credentials")
|
||||
}
|
||||
unchanged, _ := data.GetOrganization(t.Context(), old.ID)
|
||||
if unchanged.StorageRootPrefix != "" {
|
||||
t.Fatal("old organization changed")
|
||||
}
|
||||
savedSource, _ := data.GetStorageConnection(t.Context(), "source")
|
||||
if savedSource.RootPrefix != "" {
|
||||
t.Fatal("source was restricted")
|
||||
}
|
||||
r := serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+created.ID+"/storage/provision", []byte(`{}`), root)
|
||||
if r.Code != 200 || calls != 1 {
|
||||
t.Fatalf("idempotency: %d calls=%d", r.Code, calls)
|
||||
}
|
||||
// Organization admins cannot add a new unscoped connection.
|
||||
u := adminCreateUser(t, h, root, created.ID, "Admin", "scoped@example.com", "admin", "fixture-password")
|
||||
cookie := loginSession(t, h, u.Email, "fixture-password")
|
||||
r = serveJSONWithCookie(h, "POST", "/api/storage/connections", []byte(`{"name":"escape","provider":"BOS","bucket":"testxyz","endpoint":"https://bj.bcebos.com","accessKeyId":"ak","secretAccessKey":"sk"}`), cookie)
|
||||
if r.Code != 403 {
|
||||
t.Fatalf("unscoped create = %d", r.Code)
|
||||
}
|
||||
fail = true
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Pending","slug":"pending-org"}`), root)
|
||||
if r.Code != 503 {
|
||||
t.Fatal(r.Code)
|
||||
}
|
||||
var failure struct {
|
||||
Organization domain.OrganizationRecord `json:"organization"`
|
||||
}
|
||||
json.Unmarshal(r.Body.Bytes(), &failure)
|
||||
pending := failure.Organization
|
||||
if pending.StorageState != "pending" {
|
||||
t.Fatal("failed open")
|
||||
}
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+pending.ID+"/users", []byte(`{"name":"No","email":"no@example.com","role":"admin","password":"password"}`), root)
|
||||
if r.Code != http.StatusConflict {
|
||||
t.Fatalf("pending user create %d", r.Code)
|
||||
}
|
||||
fail = false
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+pending.ID+"/storage/provision", []byte(`{}`), root)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("retry: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
// A one-time secret is persisted before IAM propagation verification, so
|
||||
// retries do not mint or discard another tenant key.
|
||||
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return errors.New("IAM propagating") }
|
||||
beforeCalls := calls
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Propagation","slug":"propagation-test"}`), root)
|
||||
if r.Code != 503 {
|
||||
t.Fatal(r.Code)
|
||||
}
|
||||
json.Unmarshal(r.Body.Bytes(), &failure)
|
||||
propagating := failure.Organization
|
||||
stored, _ := data.GetStorageConnection(t.Context(), propagating.StorageConnectionID)
|
||||
if stored == nil || stored.SecretAccessKey != "tenant-sk" {
|
||||
t.Fatal("one-time secret was lost")
|
||||
}
|
||||
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations/"+propagating.ID+"/storage/provision", []byte(`{}`), root)
|
||||
if r.Code != 200 || calls != beforeCalls+1 {
|
||||
t.Fatalf("propagation retry: %d, calls=%d", r.Code, calls)
|
||||
}
|
||||
// A reservation survives org deletion; its old directory cannot be reassigned.
|
||||
if _, err := data.DeleteOrganizationCascade(t.Context(), created.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r = serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Replacement","slug":"company-a"}`), root)
|
||||
if r.Code != 503 {
|
||||
t.Fatalf("directory reassigned: %d", r.Code)
|
||||
}
|
||||
}
|
||||
func TestOrganizationStoragePolicyScope(t *testing.T) {
|
||||
var p struct {
|
||||
ACL []struct {
|
||||
Permission []string `json:"permission"`
|
||||
Resource []string `json:"resource"`
|
||||
} `json:"accessControlList"`
|
||||
}
|
||||
json.Unmarshal([]byte(organizationStoragePolicy("testxyz", "company-a/")), &p)
|
||||
if len(p.ACL) != 2 || len(p.ACL[0].Resource) != 2 || p.ACL[0].Resource[1] != "testxyz/company-a/*" || p.ACL[1].Permission[0] != "READ" {
|
||||
t.Fatalf("bad policy %s", organizationStoragePolicy("testxyz", "company-a/"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAsyncOrganizationCreationReturnsPendingThenBecomesReady(t *testing.T) {
|
||||
data := store.NewMemoryStore()
|
||||
data.UpsertStorageConnection(t.Context(), domain.StorageConnectionRecord{ID: "source", Bucket: "testxyz", Endpoint: "https://bj.bcebos.com", Provider: "BOS"})
|
||||
oldP, oldV, oldE := organizationStorageProvision, organizationStorageVerify, organizationDirectoryEmpty
|
||||
defer func() {
|
||||
organizationStorageProvision = oldP
|
||||
organizationStorageVerify = oldV
|
||||
organizationDirectoryEmpty = oldE
|
||||
}()
|
||||
proceed := make(chan struct{})
|
||||
organizationDirectoryEmpty = func(context.Context, domain.StorageConnectionRecord, string) (bool, error) { return true, nil }
|
||||
organizationStorageVerify = func(context.Context, domain.StorageConnectionRecord) error { return nil }
|
||||
organizationStorageProvision = func(_ context.Context, _ domain.StorageConnectionRecord, o domain.OrganizationRecord) (domain.StorageConnectionRecord, error) {
|
||||
<-proceed
|
||||
return domain.StorageConnectionRecord{ID: o.StorageConnectionID, OrganizationID: o.ID, Bucket: o.StorageBucket, RootPrefix: o.StorageRootPrefix, Provider: "BOS", AccessKeyID: "scoped", SecretAccessKey: "secret"}, nil
|
||||
}
|
||||
h := NewHandlerWithConfig(data, nil, config.AppConfig{RequireSessionAuth: true, OrganizationStorageSourceConnectionID: "source", OrganizationStorageReconcileEnabled: true})
|
||||
root := bootstrapSuperAdminSession(t, h)
|
||||
r := serveJSONWithCookie(h, "POST", "/api/admin/organizations", []byte(`{"name":"Async","slug":"async-scope"}`), root)
|
||||
if r.Code != 202 {
|
||||
close(proceed)
|
||||
t.Fatal(r.Code)
|
||||
}
|
||||
var org domain.OrganizationRecord
|
||||
json.Unmarshal(r.Body.Bytes(), &org)
|
||||
if org.StorageState != "pending" {
|
||||
close(proceed)
|
||||
t.Fatal("expected pending response")
|
||||
}
|
||||
close(proceed)
|
||||
deadline := time.Now().Add(time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
current, _ := data.GetOrganization(t.Context(), org.ID)
|
||||
if current.StorageState == "ready" {
|
||||
return
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
t.Fatal("background provisioning did not finish")
|
||||
}
|
||||
@ -46,6 +46,8 @@ type validationIssue struct {
|
||||
}
|
||||
|
||||
type server struct {
|
||||
organizationStorageAsync bool
|
||||
organizationStorageSourceConnectionID string
|
||||
store store.Store
|
||||
ai ai.Service
|
||||
operatorRunnerImage string
|
||||
@ -232,6 +234,9 @@ func listStorageBucketsFromConnection(ctx context.Context, connection *domain.St
|
||||
if connection == nil {
|
||||
return []string{}, nil
|
||||
}
|
||||
if connection.RootPrefix != "" {
|
||||
return []string{connection.Bucket}, nil
|
||||
}
|
||||
endpoint := storageEndpointForProvider(connection.Provider, connection.Endpoint)
|
||||
if normalizeStorageProvider(connection.Provider) == domain.StorageProviderBOS && endpoint == "" {
|
||||
return nil, fmt.Errorf("BOS endpoint is required for storage connection %s", strings.TrimSpace(connection.Name))
|
||||
@ -291,6 +296,10 @@ func listStorageObjectsFromConnection(ctx context.Context, connection *domain.St
|
||||
if connection == nil {
|
||||
return []string{}, nil
|
||||
}
|
||||
prefix = connection.StorageBrowsePrefix(prefix)
|
||||
if err := connection.CheckStoragePath(bucket, prefix, true); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
normalizedBucket := strings.TrimSpace(bucket)
|
||||
if normalizedBucket == "" {
|
||||
return []string{}, nil
|
||||
@ -1251,6 +1260,8 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
|
||||
operatorRunnerImage: appConfig.OperatorRunnerImage,
|
||||
workerAPIBaseURL: appConfig.WorkerAPIBaseURL,
|
||||
globalBOSEndpoint: appConfig.GlobalBOSEndpoint,
|
||||
organizationStorageSourceConnectionID: appConfig.OrganizationStorageSourceConnectionID,
|
||||
organizationStorageAsync: appConfig.OrganizationStorageReconcileEnabled,
|
||||
apiToken: appConfig.APIToken,
|
||||
requireSessionAuth: appConfig.RequireSessionAuth,
|
||||
sessionCookieSecure: appConfig.SessionCookieSecure,
|
||||
@ -1271,6 +1282,9 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
|
||||
buildTime: appConfig.BuildTime,
|
||||
image: appConfig.Image,
|
||||
}
|
||||
if instance.organizationStorageAsync {
|
||||
instance.startOrganizationStorageReconciler()
|
||||
}
|
||||
router := chi.NewRouter()
|
||||
router.Use(middleware.Compress(5))
|
||||
router.Use(instance.apiTokenAuth)
|
||||
@ -1289,6 +1303,7 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
|
||||
router.Use(instance.requireSuperAdmin)
|
||||
router.Get("/organizations", instance.handleAdminOrganizationsList)
|
||||
router.Post("/organizations", instance.handleAdminOrganizationsCreate)
|
||||
router.Post("/organizations/{id}/storage/provision", instance.handleAdminOrganizationStorageProvision)
|
||||
router.Put("/organizations/{id}", instance.handleAdminOrganizationsUpdate)
|
||||
router.Delete("/organizations/{id}", instance.handleAdminOrganizationsDelete)
|
||||
router.Get("/organizations/{id}/users", instance.handleAdminOrganizationUsersList)
|
||||
@ -1510,6 +1525,7 @@ func NewHandlerWithConfig(dataStore store.Store, aiService ai.Service, appConfig
|
||||
router.Get("/api/object-storage/objects", instance.handleObjectStorageObjects)
|
||||
router.Get("/api/object-storage/content", instance.handleObjectStorageContent)
|
||||
router.Head("/api/object-storage/content", instance.handleObjectStorageContent)
|
||||
router.Get("/api/object-storage/directory/manifest", instance.handleObjectStorageDirectoryManifest)
|
||||
router.Post("/api/object-storage/buckets", instance.handleObjectStorageBucketCreate)
|
||||
router.Delete("/api/object-storage/buckets/{name}", instance.handleObjectStorageBucketDelete)
|
||||
router.Post("/api/object-storage/objects", instance.handleObjectStorageObjectCreate)
|
||||
@ -3594,9 +3610,13 @@ func (server *server) validateScanConnectionReference(ctx context.Context, scanC
|
||||
if scanConfig == nil || strings.TrimSpace(scanConfig.ConnectionID) == "" {
|
||||
return nil
|
||||
}
|
||||
if _, err := server.getStorageConnection(ctx, scanConfig.ConnectionID); err != nil {
|
||||
connection, err := server.getStorageConnection(ctx, scanConfig.ConnectionID)
|
||||
if err != nil || connection == nil {
|
||||
return []validationIssue{{Field: "scanConfig.connectionId", Message: "connectionId must reference an existing storage connection"}}
|
||||
}
|
||||
if err := connection.CheckStorageSource(scanConfig.Bucket, scanConfig.Prefix, scanConfig.SourcePath); err != nil {
|
||||
return []validationIssue{{Field: "scanConfig.prefix", Message: err.Error()}}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -7421,6 +7441,7 @@ func (server *server) handleUserStoragePathFavoritesUpdate(writer http.ResponseW
|
||||
}
|
||||
|
||||
type storageConnectionPatch struct {
|
||||
RootPrefix *string `json:"rootPrefix"`
|
||||
Name *string `json:"name"`
|
||||
Description *string `json:"description"`
|
||||
Provider *domain.StorageProvider `json:"provider"`
|
||||
@ -7539,10 +7560,28 @@ func (server *server) handleStorageConnectionsList(writer http.ResponseWriter, r
|
||||
}
|
||||
|
||||
func (server *server) handleStorageConnectionsCreate(writer http.ResponseWriter, request *http.Request) {
|
||||
principal, authenticated := currentPrincipal(request)
|
||||
if authenticated && !principal.User.IsSuperAdmin && principal.Organization.StorageRootPrefix != "" {
|
||||
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "organization storage is managed by the platform"})
|
||||
return
|
||||
}
|
||||
var payload domain.CreateStorageConnectionInput
|
||||
if !decodeJSON(writer, request, &payload) {
|
||||
return
|
||||
}
|
||||
if payload.RootPrefix != "" {
|
||||
principal, ok := currentPrincipal(request)
|
||||
if !ok || !principal.User.IsSuperAdmin {
|
||||
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "only super admin can configure storage root"})
|
||||
return
|
||||
}
|
||||
root, err := domain.NormalizeStorageRoot(payload.RootPrefix)
|
||||
if err != nil || root == "" || strings.TrimSpace(payload.Bucket) == "" {
|
||||
renderValidationError(writer, validationIssue{Field: "rootPrefix", Message: "valid root prefix and bucket are required"})
|
||||
return
|
||||
}
|
||||
payload.RootPrefix = root
|
||||
}
|
||||
payload.Name = strings.TrimSpace(payload.Name)
|
||||
payload.Description = strings.TrimSpace(payload.Description)
|
||||
payload.Provider = normalizeStorageProvider(payload.Provider)
|
||||
@ -7613,6 +7652,20 @@ func (server *server) handleStorageConnectionsUpdate(writer http.ResponseWriter,
|
||||
if !decodeJSON(writer, request, &payload) {
|
||||
return
|
||||
}
|
||||
principal, isAuthenticated := currentPrincipal(request)
|
||||
superAdmin := isAuthenticated && principal.User.IsSuperAdmin
|
||||
if !superAdmin && (payload.RootPrefix != nil || (item.RootPrefix != "" && (payload.Bucket != nil || payload.Endpoint != nil || payload.Provider != nil || (payload.AccessKeyID != nil && strings.TrimSpace(*payload.AccessKeyID) != "") || (payload.SecretAccessKey != nil && strings.TrimSpace(*payload.SecretAccessKey) != "")))) {
|
||||
renderJSON(writer, http.StatusForbidden, map[string]string{"error": "only super admin can change a scoped storage connection"})
|
||||
return
|
||||
}
|
||||
if payload.RootPrefix != nil {
|
||||
root, err := domain.NormalizeStorageRoot(*payload.RootPrefix)
|
||||
if err != nil {
|
||||
renderValidationError(writer, validationIssue{Field: "rootPrefix", Message: err.Error()})
|
||||
return
|
||||
}
|
||||
item.RootPrefix = root
|
||||
}
|
||||
if payload.Name != nil {
|
||||
item.Name = strings.TrimSpace(*payload.Name)
|
||||
}
|
||||
@ -7651,6 +7704,10 @@ func (server *server) handleStorageConnectionsUpdate(writer http.ResponseWriter,
|
||||
renderValidationError(writer, *issue)
|
||||
return
|
||||
}
|
||||
if item.RootPrefix != "" && item.Bucket == "" {
|
||||
renderValidationError(writer, validationIssue{Field: "bucket", Message: "scoped connections require a bucket"})
|
||||
return
|
||||
}
|
||||
item.Endpoint = server.storageEndpointForProvider(item.Provider, item.Endpoint)
|
||||
item.UpdatedAt = nowISO()
|
||||
if err := server.store.UpsertStorageConnection(request.Context(), *item); err != nil {
|
||||
@ -8518,6 +8575,9 @@ func renderValidationError(writer http.ResponseWriter, issues ...validationIssue
|
||||
}
|
||||
|
||||
func renderError(writer http.ResponseWriter, status int, err error) {
|
||||
if errors.Is(err, domain.ErrStorageScope) {
|
||||
status = http.StatusForbidden
|
||||
}
|
||||
renderJSON(writer, status, map[string]string{"error": err.Error()})
|
||||
}
|
||||
|
||||
|
||||
@ -9295,6 +9295,10 @@ func (browser *fakeObjectStorageBrowser) ListObjects(_ context.Context, bucket s
|
||||
return items, nil
|
||||
}
|
||||
|
||||
func (browser *fakeObjectStorageBrowser) ListObjectsRecursive(ctx context.Context, bucket string, prefix string) ([]domain.ObjectStorageObjectSummary, error) {
|
||||
return browser.ListObjects(ctx, bucket, prefix)
|
||||
}
|
||||
|
||||
func (browser *fakeObjectStorageBrowser) CreateBucket(_ context.Context, bucket string) (domain.ObjectStorageBucketSummary, error) {
|
||||
browser.buckets[bucket] = struct{}{}
|
||||
return domain.ObjectStorageBucketSummary{Name: bucket}, nil
|
||||
@ -9612,6 +9616,25 @@ func TestObjectStorageBrowserSupportsConnectionCRUDAndPreview(t *testing.T) {
|
||||
t.Fatalf("unexpected content type: %s", got)
|
||||
}
|
||||
|
||||
manifestRecorder := httptest.NewRecorder()
|
||||
handler.ServeHTTP(
|
||||
manifestRecorder,
|
||||
httptest.NewRequest(http.MethodGet, "/api/object-storage/directory/manifest?connectionId="+connection.ID+"&prefix=algo", nil),
|
||||
)
|
||||
if manifestRecorder.Code != http.StatusOK {
|
||||
t.Fatalf("directory manifest status: %d body=%s", manifestRecorder.Code, manifestRecorder.Body.String())
|
||||
}
|
||||
var manifestResponse objectStorageDirectoryManifestResponse
|
||||
if err := json.NewDecoder(manifestRecorder.Body).Decode(&manifestResponse); err != nil {
|
||||
t.Fatalf("decode manifest response: %v", err)
|
||||
}
|
||||
if manifestResponse.TotalFiles == 0 || len(manifestResponse.Files) == 0 {
|
||||
t.Fatalf("expected manifest files, got %d", manifestResponse.TotalFiles)
|
||||
}
|
||||
if manifestResponse.Files[0].DownloadURL == "" {
|
||||
t.Fatalf("expected download url in manifest item: %+v", manifestResponse.Files[0])
|
||||
}
|
||||
|
||||
var multipartBody bytes.Buffer
|
||||
multipartWriter := multipart.NewWriter(&multipartBody)
|
||||
fileWriter, err := multipartWriter.CreateFormFile("file", "hello.txt")
|
||||
|
||||
110
internal/api/storage_scope_test.go
Normal file
110
internal/api/storage_scope_test.go
Normal file
@ -0,0 +1,110 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"errors"
|
||||
"github.com/baidubce/bce-sdk-go/services/bos"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestScopedBOSRejectsBeforeNetworkOrSigning(t *testing.T) {
|
||||
// A nil client makes any missed guard fail immediately; no remote service or
|
||||
// credential policy can accidentally hide a missing platform check.
|
||||
b := &bosObjectStorageBrowser{connection: &domain.StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}}
|
||||
ctx := context.Background()
|
||||
checks := map[string]func() error{
|
||||
"list": func() error { _, e := b.ListObjects(ctx, "testxyz", "other"); return e },
|
||||
"recursive": func() error { _, e := b.ListObjectsRecursive(ctx, "testxyz", "other"); return e },
|
||||
"page": func() error { _, e := b.ListObjectsPage(ctx, "testxyz", "other", "", 10); return e },
|
||||
"get": func() error { _, _, e := b.ReadObject(ctx, "testxyz", "other/x"); return e },
|
||||
"range": func() error { _, _, e := b.ReadObjectRange(ctx, "testxyz", "other/x", 0, 1); return e },
|
||||
"preview": func() error { _, _, e := b.ReadObjectPreview(ctx, "testxyz", "other/x", 10); return e },
|
||||
"head": func() error { _, e := b.GetObjectSummary(ctx, "testxyz", "other/x"); return e },
|
||||
"put": func() error { _, e := b.PutObject(ctx, "testxyz", "other/x", nil, ""); return e },
|
||||
"presign put": func() error { _, e := b.PresignPutObject(ctx, "testxyz", "other/x", "", 60); return e },
|
||||
"presign get": func() error { _, e := b.BuildPreviewURL(ctx, "testxyz", "other/x"); return e },
|
||||
"initiate": func() error { _, e := b.InitiateMultipartUpload(ctx, "testxyz", "other/x", "", 60); return e },
|
||||
"part": func() error { _, e := b.PresignMultipartUploadPart(ctx, "testxyz", "other/x", "id", 1, 60); return e },
|
||||
"complete": func() error { _, e := b.CompleteMultipartUpload(ctx, "testxyz", "other/x", "id", nil); return e },
|
||||
"abort": func() error { return b.AbortMultipartUpload(ctx, "testxyz", "other/x", "id") },
|
||||
"delete": func() error { return b.DeleteObject(ctx, "testxyz", "other/x") },
|
||||
"delete prefix": func() error { return b.DeletePrefix(ctx, "testxyz", "other") },
|
||||
"rename source": func() error { _, e := b.RenameObject(ctx, "testxyz", "other/x", "ubt/x"); return e },
|
||||
"rename target": func() error { _, e := b.RenameObject(ctx, "testxyz", "ubt/x", "other/x"); return e },
|
||||
"move": func() error { _, e := b.MovePrefix(ctx, "testxyz", "ubt/x", "other"); return e },
|
||||
}
|
||||
for name, fn := range checks {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
if e := fn(); !errors.Is(e, domain.ErrStorageScope) {
|
||||
t.Fatalf("expected scope denial, got %v", e)
|
||||
}
|
||||
})
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
renderError(w, 500, domain.ErrStorageScope)
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Fatal(w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopedConnectionCannotBeWidenedByOrganizationAdmin(t *testing.T) {
|
||||
t.Setenv("DROBOTICFLOW_REQUIRE_SESSION_AUTH", "true")
|
||||
data := store.NewMemoryStore()
|
||||
h := NewHandler(data, nil)
|
||||
root := bootstrapSuperAdminSession(t, h)
|
||||
org := adminCreateOrganization(t, h, root, "UBT", "ubt-scope")
|
||||
user := adminCreateUser(t, h, root, org.ID, "UBT Admin", "ubt-scope@example.com", "admin", "fixture-password")
|
||||
cookie := loginSession(t, h, user.Email, "fixture-password")
|
||||
c := domain.StorageConnectionRecord{ID: "scoped", OrganizationID: org.ID, Name: "UBT", Provider: domain.StorageProviderBOS, Bucket: "testxyz", RootPrefix: "ubt/", Endpoint: "https://bj.bcebos.com", AccessKeyID: "fixture", SecretAccessKey: "fixture"}
|
||||
if err := data.UpsertStorageConnection(t.Context(), c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, body := range []string{`{"rootPrefix":""}`, `{"rootPrefix":"other/"}`, `{"bucket":"other"}`, `{"endpoint":"https://example.com"}`, `{"accessKeyId":"broad"}`, `{"secretAccessKey":"broad"}`} {
|
||||
r := serveJSONWithCookie(h, http.MethodPut, "/api/storage/connections/scoped", []byte(body), cookie)
|
||||
if r.Code != 403 {
|
||||
t.Fatalf("mutation %s: %d %s", body, r.Code, r.Body.String())
|
||||
}
|
||||
}
|
||||
for _, target := range []string{
|
||||
"/api/object-storage/objects?connectionId=scoped&bucket=testxyz&prefix=other",
|
||||
"/api/object-storage/content?connectionId=scoped&bucket=testxyz&key=other/x",
|
||||
} {
|
||||
r := serveJSONWithCookie(h, http.MethodGet, target, nil, cookie)
|
||||
if r.Code != 403 {
|
||||
t.Fatalf("scope escape: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
}
|
||||
r := serveJSONWithCookie(h, http.MethodPut, "/api/storage/connections/scoped", []byte(`{"name":"UBT Storage"}`), cookie)
|
||||
if r.Code != 200 {
|
||||
t.Fatalf("rename: %d %s", r.Code, r.Body.String())
|
||||
}
|
||||
got, _ := data.GetStorageConnection(t.Context(), "scoped")
|
||||
if got.RootPrefix != "ubt/" {
|
||||
t.Fatal("scope lost")
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopedListingUsesAuthorizedRoot(t *testing.T) {
|
||||
endpoint := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Query().Get("prefix") != "ubt/" {
|
||||
t.Errorf("unexpected prefix: %s", r.URL.String())
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"name":"testxyz","prefix":"ubt/","contents":[{"key":"ubt/file.txt","size":3}],"isTruncated":false}`))
|
||||
}))
|
||||
defer endpoint.Close()
|
||||
client, err := bos.NewClient("fixture-key", "fixture-secret", endpoint.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b := &bosObjectStorageBrowser{client: client}
|
||||
b.connection = &domain.StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}
|
||||
page, err := b.ListObjectsPage(t.Context(), "testxyz", "", "", 10)
|
||||
if err != nil || len(page.Items) != 1 || page.Items[0].Key != "ubt/file.txt" {
|
||||
t.Fatalf("page=%+v err=%v", page, err)
|
||||
}
|
||||
}
|
||||
@ -10,6 +10,8 @@ import (
|
||||
)
|
||||
|
||||
type AppConfig struct {
|
||||
OrganizationStorageReconcileEnabled bool
|
||||
OrganizationStorageSourceConnectionID string
|
||||
Port int
|
||||
WorkerPort int
|
||||
CoordinatorPort int
|
||||
@ -129,7 +131,7 @@ func LoadValidated() (AppConfig, error) {
|
||||
return AppConfig{}, err
|
||||
}
|
||||
return AppConfig{
|
||||
Port: port,
|
||||
OrganizationStorageReconcileEnabled: strings.TrimSpace(os.Getenv("ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID")) != "", OrganizationStorageSourceConnectionID: strings.TrimSpace(os.Getenv("ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID")), Port: port,
|
||||
WorkerPort: workerPort,
|
||||
CoordinatorPort: coordinatorPort,
|
||||
MongoURI: mongoURI,
|
||||
|
||||
@ -8,6 +8,7 @@ type ScanNotificationSettings struct {
|
||||
}
|
||||
|
||||
type ScanConfig struct {
|
||||
StorageRootPrefix string `json:"-" bson:"-"`
|
||||
DataSource string `json:"dataSource,omitempty" bson:"dataSource,omitempty"`
|
||||
SourcePath string `json:"sourcePath,omitempty" bson:"sourcePath,omitempty"`
|
||||
ConnectionID string `json:"connectionId,omitempty" bson:"connectionId,omitempty"`
|
||||
|
||||
93
internal/domain/storage_scope.go
Normal file
93
internal/domain/storage_scope.go
Normal file
@ -0,0 +1,93 @@
|
||||
package domain
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var ErrStorageScope = errors.New("storage path is outside the authorized directory")
|
||||
|
||||
// NormalizeStorageRoot accepts a literal directory, never a wildcard or a path
|
||||
// that another consumer could interpret differently. Empty means legacy scope.
|
||||
func NormalizeStorageRoot(root string) (string, error) {
|
||||
root = strings.Trim(root, "/")
|
||||
if root == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !safeStorageKey(root) {
|
||||
return "", fmt.Errorf("invalid storage root prefix")
|
||||
}
|
||||
return root + "/", nil
|
||||
}
|
||||
|
||||
func safeStorageKey(key string) bool {
|
||||
if strings.ContainsAny(key, "\\%*?\x00\r\n") || strings.HasPrefix(key, "/") {
|
||||
return false
|
||||
}
|
||||
for _, part := range strings.Split(strings.TrimSuffix(key, "/"), "/") {
|
||||
if part == "" || part == "." || part == ".." {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return strings.TrimSpace(key) == key
|
||||
}
|
||||
|
||||
// CheckStoragePath is shared by the API, coordinator and worker. Directory
|
||||
// checks allow the root itself; object checks cannot address its sibling object.
|
||||
func (c *StorageConnectionRecord) CheckStoragePath(bucket, key string, directory bool) error {
|
||||
if c == nil || c.RootPrefix == "" {
|
||||
return nil
|
||||
}
|
||||
root, err := NormalizeStorageRoot(c.RootPrefix)
|
||||
if err != nil || root == "" || c.Bucket == "" || bucket != c.Bucket || !safeStorageKey(key) {
|
||||
return ErrStorageScope
|
||||
}
|
||||
if directory && strings.TrimSuffix(key, "/") == strings.TrimSuffix(root, "/") {
|
||||
return nil
|
||||
}
|
||||
if !strings.HasPrefix(key, root) {
|
||||
return ErrStorageScope
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *StorageConnectionRecord) StorageBrowsePrefix(prefix string) string {
|
||||
if c != nil && c.RootPrefix != "" && prefix == "" {
|
||||
return strings.TrimSuffix(c.RootPrefix, "/")
|
||||
}
|
||||
return prefix
|
||||
}
|
||||
|
||||
func (c *StorageConnectionRecord) InternalStorageKey(key string) string {
|
||||
if c == nil || c.RootPrefix == "" {
|
||||
return key
|
||||
}
|
||||
return strings.TrimSuffix(c.RootPrefix, "/") + "/" + key
|
||||
}
|
||||
|
||||
// CheckStorageSource validates the effective source URI as well as form fields.
|
||||
func (c *StorageConnectionRecord) CheckStorageSource(bucket, prefix, sourcePath string) error {
|
||||
if c == nil || c.RootPrefix == "" {
|
||||
return nil
|
||||
}
|
||||
if sourcePath != "" {
|
||||
if !strings.HasPrefix(sourcePath, "bos://") {
|
||||
return ErrStorageScope
|
||||
}
|
||||
parts := strings.SplitN(strings.TrimPrefix(sourcePath, "bos://"), "/", 2)
|
||||
if len(parts) != 2 {
|
||||
return ErrStorageScope
|
||||
}
|
||||
if err := c.CheckStoragePath(parts[0], parts[1], true); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if bucket != "" || prefix != "" {
|
||||
return c.CheckStoragePath(bucket, prefix, true)
|
||||
}
|
||||
if sourcePath == "" {
|
||||
return ErrStorageScope
|
||||
}
|
||||
return nil
|
||||
}
|
||||
33
internal/domain/storage_scope_test.go
Normal file
33
internal/domain/storage_scope_test.go
Normal file
@ -0,0 +1,33 @@
|
||||
package domain
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestStorageScopeBoundaries(t *testing.T) {
|
||||
c := &StorageConnectionRecord{Bucket: "testxyz", RootPrefix: "ubt/"}
|
||||
for _, key := range []string{"other/x", "ubt2/x", "ubt", "", "ubt/../other", "ubt/%2e%2e/other", "ubt//x", "/ubt/x", "ubt/./x", "ubt/a\\b"} {
|
||||
if c.CheckStoragePath("testxyz", key, false) == nil {
|
||||
t.Errorf("allowed %q", key)
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"ubt/x", "ubt/dataset/capture.mcap", "ubt/.drosync-internal/staging/x"} {
|
||||
if err := c.CheckStoragePath("testxyz", key, false); err != nil {
|
||||
t.Errorf("denied %q: %v", key, err)
|
||||
}
|
||||
}
|
||||
if c.CheckStoragePath("other", "ubt/x", false) == nil {
|
||||
t.Fatal("cross bucket allowed")
|
||||
}
|
||||
if c.CheckStoragePath("testxyz", "ubt", true) != nil {
|
||||
t.Fatal("root directory denied")
|
||||
}
|
||||
if c.CheckStorageSource("testxyz", "ubt", "bos://testxyz/other") == nil {
|
||||
t.Fatal("source URI bypass")
|
||||
}
|
||||
if c.InternalStorageKey(".drosync-internal/audit/x") != "ubt/.drosync-internal/audit/x" {
|
||||
t.Fatal("internal path escaped")
|
||||
}
|
||||
legacy := &StorageConnectionRecord{}
|
||||
if legacy.CheckStoragePath("testxyz", "anything", false) != nil {
|
||||
t.Fatal("legacy changed")
|
||||
}
|
||||
}
|
||||
@ -863,6 +863,11 @@ type UserRecord struct {
|
||||
}
|
||||
|
||||
type OrganizationRecord struct {
|
||||
StorageRootPrefix string `json:"storageRootPrefix,omitempty" bson:"storageRootPrefix,omitempty"`
|
||||
StorageBucket string `json:"storageBucket,omitempty" bson:"storageBucket,omitempty"`
|
||||
StorageState string `json:"storageState,omitempty" bson:"storageState,omitempty"`
|
||||
StorageConnectionID string `json:"storageConnectionId,omitempty" bson:"storageConnectionId,omitempty"`
|
||||
|
||||
ID string `json:"id" bson:"id"`
|
||||
Name string `json:"name" bson:"name"`
|
||||
Slug string `json:"slug" bson:"slug"`
|
||||
@ -1086,6 +1091,7 @@ type DashboardPOCReadiness struct {
|
||||
}
|
||||
|
||||
type StorageConnectionRecord struct {
|
||||
RootPrefix string `json:"rootPrefix,omitempty" bson:"rootPrefix,omitempty"`
|
||||
ID string `json:"id" bson:"id"`
|
||||
OrganizationID string `json:"organizationId,omitempty" bson:"organizationId,omitempty"`
|
||||
Name string `json:"name" bson:"name"`
|
||||
@ -1123,6 +1129,7 @@ type CreateProjectInput struct {
|
||||
}
|
||||
|
||||
type CreateStorageConnectionInput struct {
|
||||
RootPrefix string `json:"rootPrefix,omitempty" bson:"rootPrefix,omitempty"`
|
||||
OrganizationID string `json:"organizationId,omitempty"`
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description,omitempty"`
|
||||
|
||||
@ -38,6 +38,7 @@ const (
|
||||
)
|
||||
|
||||
type LaunchInput struct {
|
||||
ScopedWorkspace bool
|
||||
Image string
|
||||
Command []string
|
||||
Args []string
|
||||
@ -581,11 +582,17 @@ func CreateJobManifest(namespace string, imagePullSecretName string, workspacePV
|
||||
}
|
||||
sharedMemorySize = quantity.String()
|
||||
}
|
||||
if input.ScopedWorkspace && (len(input.PVCMounts) > 0 || input.RegistryAuth) {
|
||||
return nil, fmt.Errorf("scoped organizations cannot mount shared PVCs or registry credentials")
|
||||
}
|
||||
codeSubPath, err := cleanVolumeSubPath(input.CodeSubPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
codeMountPath := ""
|
||||
if input.ScopedWorkspace && codeSubPath != "" && !strings.HasPrefix(codeSubPath, "lightweight-operators/") && !strings.HasPrefix(codeSubPath, "operator-repos/") {
|
||||
return nil, fmt.Errorf("scoped organization code mount must use the operator catalog")
|
||||
}
|
||||
if codeSubPath != "" {
|
||||
codeMountPath, err = cleanMountPath(input.CodeMountPath)
|
||||
if err != nil {
|
||||
@ -725,26 +732,38 @@ func CreateJobManifest(namespace string, imagePullSecretName string, workspacePV
|
||||
},
|
||||
},
|
||||
})
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
|
||||
Name: "workspace",
|
||||
MountPath: workspaceMountPath,
|
||||
ReadOnly: input.CodeReadOnly,
|
||||
})
|
||||
if codeSubPath != "" {
|
||||
runSubPath, err := cleanVolumeSubPath(pathClean("system/runs/" + input.RunID))
|
||||
if !input.ScopedWorkspace {
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
|
||||
Name: "workspace",
|
||||
MountPath: workspaceMountPath,
|
||||
ReadOnly: input.CodeReadOnly,
|
||||
})
|
||||
} else {
|
||||
automount := false
|
||||
job.Spec.Template.Spec.AutomountServiceAccountToken = &automount
|
||||
runSubPath, err := cleanVolumeSubPath("system/runs/" + input.RunID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
|
||||
Name: "workspace",
|
||||
MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID,
|
||||
SubPath: runSubPath,
|
||||
})
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{Name: "workspace", MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID, SubPath: runSubPath})
|
||||
}
|
||||
if codeSubPath != "" {
|
||||
if !input.ScopedWorkspace {
|
||||
runSubPath, err := cleanVolumeSubPath(pathClean("system/runs/" + input.RunID))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
|
||||
Name: "workspace",
|
||||
MountPath: strings.TrimRight(workspaceMountPath, "/") + "/system/runs/" + input.RunID,
|
||||
SubPath: runSubPath,
|
||||
})
|
||||
}
|
||||
job.Spec.Template.Spec.Containers[0].VolumeMounts = append(job.Spec.Template.Spec.Containers[0].VolumeMounts, corev1.VolumeMount{
|
||||
Name: "workspace",
|
||||
MountPath: codeMountPath,
|
||||
SubPath: codeSubPath,
|
||||
ReadOnly: input.CodeReadOnly,
|
||||
ReadOnly: input.CodeReadOnly || input.ScopedWorkspace,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@ -930,3 +930,30 @@ func TestReadJobDiagnosticsIncludesPodStatusAndEvents(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopedOrganizationJobsCannotMountSharedWorkspace(t *testing.T) {
|
||||
input := LaunchInput{Image: "example/operator:1", RunID: "run-scoped", NodeID: "node", WorkflowID: "workflow", ScopedWorkspace: true}
|
||||
job, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mounts := job.Spec.Template.Spec.Containers[0].VolumeMounts
|
||||
for _, m := range mounts {
|
||||
if m.Name == "workspace" && (m.MountPath == "/workspace" || m.SubPath != "system/runs/run-scoped") {
|
||||
t.Fatalf("shared workspace exposed: %+v", m)
|
||||
}
|
||||
}
|
||||
if job.Spec.Template.Spec.AutomountServiceAccountToken == nil || *job.Spec.Template.Spec.AutomountServiceAccountToken {
|
||||
t.Fatal("service account exposed")
|
||||
}
|
||||
input.PVCMounts = []PVCMount{{Name: "escape", PVCName: "workspace", MountPath: "/other"}}
|
||||
if _, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input); err == nil {
|
||||
t.Fatal("shared PVC accepted")
|
||||
}
|
||||
input.PVCMounts = nil
|
||||
input.CodeSubPath = "private/other-org"
|
||||
input.CodeMountPath = "/app/code"
|
||||
if _, err := CreateJobManifest("poc", "registry", "workspace", "/workspace", input); err == nil {
|
||||
t.Fatal("private code path accepted")
|
||||
}
|
||||
}
|
||||
|
||||
@ -59,6 +59,10 @@ func isNotFound(err error) bool {
|
||||
}
|
||||
|
||||
func (publisher BOSPublisher) Head(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string) (ObjectMetadata, error) {
|
||||
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
}
|
||||
|
||||
client, err := publisher.client(connection)
|
||||
if err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
@ -108,6 +112,13 @@ func (publisher BOSPublisher) PublishConditional(
|
||||
ifNoneMatch string,
|
||||
userMeta map[string]string,
|
||||
) (ObjectMetadata, error) {
|
||||
if err := connection.CheckStoragePath(bucket, sourceKey, false); err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
}
|
||||
if err := connection.CheckStoragePath(bucket, targetKey, false); err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
}
|
||||
|
||||
client, err := publisher.client(connection)
|
||||
if err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
@ -200,6 +211,10 @@ func (publisher BOSPublisher) PutInternal(
|
||||
contentType string,
|
||||
userMeta map[string]string,
|
||||
) (ObjectMetadata, error) {
|
||||
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
}
|
||||
|
||||
client, err := publisher.client(connection)
|
||||
if err != nil {
|
||||
return ObjectMetadata{}, err
|
||||
@ -220,6 +235,10 @@ func (publisher BOSPublisher) PutInternal(
|
||||
}
|
||||
|
||||
func (publisher BOSPublisher) Delete(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key string) error {
|
||||
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
client, err := publisher.client(connection)
|
||||
if err != nil {
|
||||
return err
|
||||
@ -233,6 +252,10 @@ func (publisher BOSPublisher) Delete(ctx context.Context, connection domain.Stor
|
||||
}
|
||||
|
||||
func (publisher BOSPublisher) AbortMultipart(ctx context.Context, connection domain.StorageConnectionRecord, bucket, key, uploadID string) error {
|
||||
if err := connection.CheckStoragePath(bucket, key, false); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
client, err := publisher.client(connection)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@ -431,7 +431,7 @@ func (coordinator *Coordinator) sealAuditDay(ctx context.Context, organizationID
|
||||
if err != nil || connection == nil {
|
||||
return false, fmt.Errorf("audit seal storage connection is unavailable")
|
||||
}
|
||||
objectKey := path.Join(".drosync-internal/audit", organizationID, dayUTC+".json")
|
||||
objectKey := connection.InternalStorageKey(path.Join(".drosync-internal/audit", organizationID, dayUTC+".json"))
|
||||
object, err := coordinator.publisher.PutInternal(
|
||||
ctx,
|
||||
*connection,
|
||||
@ -1033,7 +1033,7 @@ func (coordinator *Coordinator) publishRelease(ctx context.Context, release *dom
|
||||
return err
|
||||
}
|
||||
defer reader.Close()
|
||||
manifestKey := path.Join(".drosync-internal/releases", release.OrganizationID, release.ID+".ndjson")
|
||||
manifestKey := connection.InternalStorageKey(path.Join(".drosync-internal/releases", release.OrganizationID, release.ID+".ndjson"))
|
||||
digest := hex.EncodeToString(hasher.Sum(nil))
|
||||
if _, err := coordinator.publisher.PutInternal(ctx, *connection, space.Bucket, manifestKey, reader, info.Size(), "application/x-ndjson", map[string]string{
|
||||
"drosync-release": release.ID,
|
||||
|
||||
@ -12,6 +12,9 @@ import (
|
||||
)
|
||||
|
||||
type MemoryStore struct {
|
||||
organizationStorageRoots map[string]string
|
||||
organizationStorageLeases map[string]organizationStorageLease
|
||||
|
||||
mu sync.Mutex
|
||||
organizations map[string]domain.OrganizationRecord
|
||||
projects map[string]domain.ProjectRecord
|
||||
@ -1824,6 +1827,7 @@ func (store *MemoryStore) CreateStorageConnection(_ context.Context, input domai
|
||||
defer store.mu.Unlock()
|
||||
now := nowISO()
|
||||
item := domain.StorageConnectionRecord{
|
||||
RootPrefix: input.RootPrefix,
|
||||
ID: domain.NewID("storage"),
|
||||
OrganizationID: input.OrganizationID,
|
||||
Name: input.Name,
|
||||
|
||||
@ -1798,6 +1798,7 @@ func (store *MongoStore) GetStorageConnection(ctx context.Context, id string) (*
|
||||
func (store *MongoStore) CreateStorageConnection(ctx context.Context, input domain.CreateStorageConnectionInput) (*domain.StorageConnectionRecord, error) {
|
||||
now := nowISO()
|
||||
item := domain.StorageConnectionRecord{
|
||||
RootPrefix: input.RootPrefix,
|
||||
ID: domain.NewID("storage"),
|
||||
OrganizationID: input.OrganizationID,
|
||||
Name: input.Name,
|
||||
|
||||
90
internal/store/organization_storage.go
Normal file
90
internal/store/organization_storage.go
Normal file
@ -0,0 +1,90 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
"time"
|
||||
)
|
||||
|
||||
type organizationStorageLease struct {
|
||||
owner string
|
||||
until time.Time
|
||||
}
|
||||
|
||||
func (s *MemoryStore) ReserveOrganizationStorageRoot(_ context.Context, bucket, root, org string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.organizationStorageRoots == nil {
|
||||
s.organizationStorageRoots = map[string]string{}
|
||||
}
|
||||
key := bucket + "/" + root
|
||||
if owner, ok := s.organizationStorageRoots[key]; ok && owner != org {
|
||||
return fmt.Errorf("directory is reserved by another organization")
|
||||
}
|
||||
s.organizationStorageRoots[key] = org
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) ReserveOrganizationStorageRoot(ctx context.Context, bucket, root, org string) error {
|
||||
_, err := s.db.Collection("organizationStorageRoots").UpdateOne(ctx, bson.M{"_id": bucket + "/" + root, "organizationId": org}, bson.M{"$setOnInsert": bson.M{"organizationId": org, "createdAt": time.Now().UTC()}}, options.Update().SetUpsert(true))
|
||||
if mongo.IsDuplicateKeyError(err) {
|
||||
return fmt.Errorf("directory is reserved by another organization")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (s *MemoryStore) AcquireOrganizationStorageLease(_ context.Context, id, owner string, until time.Time) (bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.organizationStorageLeases == nil {
|
||||
s.organizationStorageLeases = map[string]organizationStorageLease{}
|
||||
}
|
||||
if l, ok := s.organizationStorageLeases[id]; ok && l.until.After(time.Now()) {
|
||||
return false, nil
|
||||
}
|
||||
s.organizationStorageLeases[id] = organizationStorageLease{owner, until}
|
||||
return true, nil
|
||||
}
|
||||
func (s *MemoryStore) ReleaseOrganizationStorageLease(_ context.Context, id, owner string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.organizationStorageLeases[id].owner == owner {
|
||||
delete(s.organizationStorageLeases, id)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) AcquireOrganizationStorageLease(ctx context.Context, id, owner string, until time.Time) (bool, error) {
|
||||
_, err := s.db.Collection("organizationStorageLeases").UpdateOne(ctx, bson.M{"_id": id, "until": bson.M{"$lte": time.Now()}}, bson.M{"$set": bson.M{"owner": owner, "until": until}}, options.Update().SetUpsert(true))
|
||||
if mongo.IsDuplicateKeyError(err) {
|
||||
return false, nil
|
||||
}
|
||||
return err == nil, err
|
||||
}
|
||||
func (s *MongoStore) ReleaseOrganizationStorageLease(ctx context.Context, id, owner string) error {
|
||||
_, err := s.db.Collection("organizationStorageLeases").DeleteOne(ctx, bson.M{"_id": id, "owner": owner})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *MemoryStore) MarkOrganizationStorageReady(_ context.Context, id string) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
org, ok := s.organizations[id]
|
||||
if !ok {
|
||||
return fmt.Errorf("organization is unavailable")
|
||||
}
|
||||
org.StorageState = "ready"
|
||||
org.UpdatedAt = nowISO()
|
||||
s.organizations[id] = org
|
||||
return nil
|
||||
}
|
||||
func (s *MongoStore) MarkOrganizationStorageReady(ctx context.Context, id string) error {
|
||||
result, err := s.organizations.UpdateOne(ctx, bson.M{"id": id}, bson.M{"$set": bson.M{"storageState": "ready", "updatedAt": nowISO()}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if result.MatchedCount == 0 {
|
||||
return fmt.Errorf("organization is unavailable")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
23
internal/store/organization_storage_test.go
Normal file
23
internal/store/organization_storage_test.go
Normal file
@ -0,0 +1,23 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestReadyTransitionPreservesConcurrentOrganizationChanges(t *testing.T) {
|
||||
s := NewMemoryStore()
|
||||
org := domain.OrganizationRecord{ID: "scoped-org", Name: "Renamed", Status: domain.OrganizationStatusDisabled, StorageState: "pending"}
|
||||
s.UpsertOrganization(t.Context(), org)
|
||||
if err := s.MarkOrganizationStorageReady(t.Context(), org.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _ := s.GetOrganization(t.Context(), org.ID)
|
||||
if got.Status != domain.OrganizationStatusDisabled || got.Name != "Renamed" || got.StorageState != "ready" {
|
||||
t.Fatalf("lost concurrent changes: %+v", got)
|
||||
}
|
||||
s.DeleteOrganizationCascade(t.Context(), org.ID)
|
||||
if s.MarkOrganizationStorageReady(t.Context(), org.ID) == nil {
|
||||
t.Fatal("deleted organization revived")
|
||||
}
|
||||
}
|
||||
@ -21,6 +21,11 @@ type WorkerRunCounts struct {
|
||||
}
|
||||
|
||||
type Store interface {
|
||||
MarkOrganizationStorageReady(context.Context, string) error
|
||||
ReserveOrganizationStorageRoot(context.Context, string, string, string) error
|
||||
AcquireOrganizationStorageLease(context.Context, string, string, time.Time) (bool, error)
|
||||
ReleaseOrganizationStorageLease(context.Context, string, string) error
|
||||
|
||||
Ready(context.Context) error
|
||||
EnsureDefaultOrganization(context.Context) (*domain.OrganizationRecord, error)
|
||||
ListOrganizations(context.Context) ([]domain.OrganizationRecord, error)
|
||||
|
||||
@ -105,6 +105,16 @@ func artifactPrefix(policy domain.ArtifactStoragePolicy, run *domain.RunRecord,
|
||||
}
|
||||
|
||||
func (runtime *Runtime) artifactStoragePolicy(ctx context.Context, workflow *domain.WorkflowRecord) (*domain.ArtifactStoragePolicy, error) {
|
||||
if workflow != nil {
|
||||
org, err := runtime.store.GetOrganization(ctx, workflow.OrganizationID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if org != nil && org.StorageRootPrefix != "" {
|
||||
return &domain.ArtifactStoragePolicy{ConnectionID: org.StorageConnectionID, Bucket: org.StorageBucket, PrefixTemplate: org.StorageRootPrefix + "artifacts/{projectId}/{runId}/{nodeId}"}, nil
|
||||
}
|
||||
}
|
||||
|
||||
if workflow != nil && workflow.ArtifactStorage != nil && strings.TrimSpace(workflow.ArtifactStorage.ConnectionID) != "" {
|
||||
copy := *workflow.ArtifactStorage
|
||||
return ©, nil
|
||||
@ -195,6 +205,11 @@ func (runtime *Runtime) externalizeDirectory(ctx context.Context, publisher obje
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if connection.RootPrefix != "" {
|
||||
if err := validateScopedArtifactPath(filepath.Join(runtime.workspaceRoot, "system", "runs", run.ID), path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
key := basePrefix + "/files/" + normalizeArtifactKey(filepath.ToSlash(relative))
|
||||
meta, sha, err := putArtifactFile(ctx, publisher, connection, policy.Bucket, key, path)
|
||||
if err != nil {
|
||||
@ -243,6 +258,20 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
|
||||
result := make([]domain.ExecutionArtifact, len(artifacts))
|
||||
for index, artifact := range artifacts {
|
||||
result[index] = artifact
|
||||
if artifact.Payload != nil && artifact.Payload.Mode == domain.ArtifactTransferModeInline {
|
||||
if err := domain.ValidateArtifactPayload(*artifact.Payload, artifact.Type); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if artifact.Type == domain.PortTypeJSON && artifact.Path == "" && artifact.Data != nil {
|
||||
inline := domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: artifact.Data}
|
||||
if err := domain.ValidateArtifactPayload(inline, artifact.Type); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[index].Payload = &inline
|
||||
continue
|
||||
}
|
||||
if artifact.Payload != nil && artifact.Payload.Mode == domain.ArtifactTransferModeRef {
|
||||
if err := domain.ValidateArtifactPayload(*artifact.Payload, artifact.Type); err != nil {
|
||||
return nil, fmt.Errorf("artifact_ref_invalid: %w", err)
|
||||
@ -268,6 +297,11 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
|
||||
}
|
||||
continue
|
||||
}
|
||||
if connection != nil && connection.RootPrefix != "" && !strings.HasPrefix(artifact.Path, "bos://") {
|
||||
if err := validateScopedArtifactPath(filepath.Join(runtime.workspaceRoot, "system", "runs", run.ID), artifact.Path); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
info, statErr := os.Stat(artifact.Path)
|
||||
if statErr == nil && info.Mode().IsRegular() && artifact.Type == domain.PortTypeJSON && info.Size() <= domain.MaxInlinePayloadBytes {
|
||||
payloadBytes, readErr := os.ReadFile(artifact.Path)
|
||||
@ -294,6 +328,11 @@ func (runtime *Runtime) normalizeOutputArtifacts(ctx context.Context, workflow *
|
||||
connectionID = policy.ConnectionID
|
||||
}
|
||||
if connectionID != "" {
|
||||
if connection != nil {
|
||||
if err := connection.CheckStoragePath(parts[0], parts[1], false); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
ref := &domain.ArtifactReference{Provider: domain.StorageProviderBOS, ConnectionID: connectionID, Bucket: parts[0], Key: parts[1], SizeBytes: artifact.SizeBytes, Kind: domain.ArtifactRefKindObject}
|
||||
result[index].Payload = &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Type: artifact.Type, Ref: ref}
|
||||
result[index].Data = compactArtifactData(artifact.Data)
|
||||
@ -371,6 +410,9 @@ func (runtime *Runtime) resolveDirectoryRunInputs(ctx context.Context, run *doma
|
||||
resolvedByInputID[binding.WorkflowInputID] = &refCopy
|
||||
continue
|
||||
}
|
||||
if err := connection.CheckStoragePath(ref.Bucket, ref.Prefix, true); err != nil {
|
||||
return fmt.Errorf("input_resolution_failed: %w", err)
|
||||
}
|
||||
client, err := bos.NewClient(connection.AccessKeyID, connection.SecretAccessKey, connection.Endpoint)
|
||||
if err != nil {
|
||||
return fmt.Errorf("input_resolution_failed: %w", err)
|
||||
@ -429,3 +471,19 @@ func (runtime *Runtime) resolveDirectoryRunInputs(ctx context.Context, run *doma
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateScopedArtifactPath(root, candidate string) error {
|
||||
base, err := filepath.EvalSymlinks(root)
|
||||
if err != nil {
|
||||
return domain.ErrStorageScope
|
||||
}
|
||||
target, err := filepath.EvalSymlinks(candidate)
|
||||
if err != nil {
|
||||
return domain.ErrStorageScope
|
||||
}
|
||||
rel, err := filepath.Rel(base, target)
|
||||
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
|
||||
return domain.ErrStorageScope
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -841,8 +841,14 @@ func (runtime *Runtime) executeNode(ctx context.Context, run *domain.RunRecord,
|
||||
if workflow != nil {
|
||||
nodeTimeoutSeconds = workflow.NodeTimeoutSeconds
|
||||
}
|
||||
organization, orgErr := runtime.store.GetOrganization(ctx, run.OrganizationID)
|
||||
if orgErr != nil {
|
||||
return executionResult{}, nil, orgErr
|
||||
}
|
||||
scopedWorkspace := organization != nil && organization.StorageRootPrefix != ""
|
||||
launched, err := runtime.launcher.Launch(ctx, k8s.LaunchInput{
|
||||
Image: chooseImage(node, runtime.operatorRunnerImage),
|
||||
ScopedWorkspace: scopedWorkspace,
|
||||
Command: cloneStringSlice(node.Data.ContainerCommand),
|
||||
Args: cloneStringSlice(node.Data.ContainerArgs),
|
||||
RunID: run.ID,
|
||||
|
||||
@ -1739,7 +1739,40 @@ func stringList(value any) []string {
|
||||
return result
|
||||
}
|
||||
|
||||
func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *domain.WorkflowRecord, node domain.WorkflowNode) (domain.WorkflowNode, error) {
|
||||
func inputStorageConnectionID(bindings []domain.NodeInputBinding) (string, error) {
|
||||
connectionID := ""
|
||||
consider := func(payload *domain.ArtifactPayload) error {
|
||||
if payload == nil || payload.Mode != domain.ArtifactTransferModeRef || payload.Ref == nil {
|
||||
return nil
|
||||
}
|
||||
candidate := strings.TrimSpace(payload.Ref.ConnectionID)
|
||||
if candidate == "" {
|
||||
return nil
|
||||
}
|
||||
if connectionID != "" && connectionID != candidate {
|
||||
return fmt.Errorf("input_storage_ambiguous: node inputs reference multiple storage connections")
|
||||
}
|
||||
connectionID = candidate
|
||||
return nil
|
||||
}
|
||||
for index := range bindings {
|
||||
if err := consider(bindings[index].Payload); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for artifactIndex := range bindings[index].Artifacts {
|
||||
if err := consider(bindings[index].Artifacts[artifactIndex].Payload); err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
}
|
||||
return connectionID, nil
|
||||
}
|
||||
|
||||
func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *domain.WorkflowRecord, node domain.WorkflowNode, bindingSets ...[]domain.NodeInputBinding) (domain.WorkflowNode, error) {
|
||||
var inputBindings []domain.NodeInputBinding
|
||||
if len(bindingSets) > 0 {
|
||||
inputBindings = bindingSets[0]
|
||||
}
|
||||
copy, err := runtime.resolveBuiltinNodeRuntime(ctx, node)
|
||||
if err != nil {
|
||||
return node, err
|
||||
@ -1752,6 +1785,13 @@ func (runtime *Runtime) prepareExecutionNode(ctx context.Context, workflow *doma
|
||||
if err != nil {
|
||||
return node, err
|
||||
}
|
||||
if strings.TrimSpace(copy.Data.ConnectionID) == "" {
|
||||
connectionID, inputErr := inputStorageConnectionID(inputBindings)
|
||||
if inputErr != nil {
|
||||
return node, inputErr
|
||||
}
|
||||
copy.Data.ConnectionID = connectionID
|
||||
}
|
||||
copy, err = runtime.resolveNodeStorageConnection(ctx, copy)
|
||||
if err != nil {
|
||||
return node, err
|
||||
@ -1856,6 +1896,16 @@ func (runtime *Runtime) resolveBuiltinNodeRuntime(ctx context.Context, node doma
|
||||
}
|
||||
|
||||
func (runtime *Runtime) resolveScanWorkflow(ctx context.Context, workflow *domain.WorkflowRecord) (*domain.WorkflowRecord, error) {
|
||||
if workflow != nil && workflow.ScanConfig != nil {
|
||||
organization, err := runtime.store.GetOrganization(ctx, workflow.OrganizationID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if organization != nil && organization.StorageRootPrefix != "" && (workflow.ScanConfig.ConnectionID != organization.StorageConnectionID || (strings.TrimSpace(workflow.ScanConfig.DataSource) != "" && !strings.EqualFold(workflow.ScanConfig.DataSource, "BOS"))) {
|
||||
return nil, domain.ErrStorageScope
|
||||
}
|
||||
}
|
||||
|
||||
if workflow == nil || workflow.ScanConfig == nil || strings.TrimSpace(workflow.ScanConfig.ConnectionID) == "" {
|
||||
return workflow, nil
|
||||
}
|
||||
@ -1868,6 +1918,13 @@ func (runtime *Runtime) resolveScanWorkflow(ctx context.Context, workflow *domai
|
||||
}
|
||||
copy := *workflow
|
||||
scanConfig := *workflow.ScanConfig
|
||||
if err := connection.CheckStorageSource(scanConfig.Bucket, scanConfig.Prefix, scanConfig.SourcePath); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if connection.RootPrefix != "" {
|
||||
scanConfig.AccessKeyID = connection.AccessKeyID
|
||||
scanConfig.SecretAccessKey = connection.SecretAccessKey
|
||||
}
|
||||
scanConfig.Endpoint = runtime.storageConnectionEndpoint(connection)
|
||||
if strings.TrimSpace(scanConfig.AccessKeyID) == "" {
|
||||
scanConfig.AccessKeyID = connection.AccessKeyID
|
||||
@ -1906,6 +1963,9 @@ func (runtime *Runtime) resolveNodeStorageConnection(ctx context.Context, node d
|
||||
if sourcePath == "" {
|
||||
sourcePath = bosSourcePath(copy.Data.Bucket, copy.Data.Prefix)
|
||||
}
|
||||
if err := connection.CheckStorageSource(copy.Data.Bucket, copy.Data.Prefix, sourcePath); err != nil {
|
||||
return node, err
|
||||
}
|
||||
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "DATA_SOURCE", "BOS")
|
||||
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "SOURCE_PATH", sourcePath)
|
||||
copy.Data.Hyperparams = upsertNodeHyperparam(copy.Data.Hyperparams, "BOS_ENDPOINT", runtime.storageConnectionEndpoint(connection))
|
||||
@ -2763,7 +2823,7 @@ func (runtime *Runtime) processRun(ctx context.Context, claimed *domain.RunRecor
|
||||
// Use the node attempt for the Kubernetes Job suffix so a recovered
|
||||
// execution cannot collide with the vanished attempt name.
|
||||
runSnapshot.Attempt = executionAttempt
|
||||
nodeSnapshot, err := runtime.prepareExecutionNode(ctx, workflow, *node)
|
||||
nodeSnapshot, err := runtime.prepareExecutionNode(ctx, workflow, *node, inputBindings)
|
||||
if err != nil {
|
||||
results <- nodeExecutionOutcome{
|
||||
nodeID: node.ID,
|
||||
|
||||
@ -2834,6 +2834,75 @@ func TestRuntimeCommitsIncrementalSourceStateOnlyOnSuccessfulRuns(t *testing.T)
|
||||
})
|
||||
}
|
||||
|
||||
func TestInputStorageConnectionIDUsesDirectoryReferenceAndRejectsAmbiguity(t *testing.T) {
|
||||
payload := domain.ArtifactPayload{Mode: domain.ArtifactTransferModeRef, Type: domain.PortTypeFile, Ref: &domain.ArtifactReference{ConnectionID: "storage-source", Bucket: "bucket", Prefix: "capture", Kind: domain.ArtifactRefKindDirectory}}
|
||||
connectionID, err := inputStorageConnectionID([]domain.NodeInputBinding{{TargetPortID: "in-1", Payload: &payload}})
|
||||
if err != nil || connectionID != "storage-source" {
|
||||
t.Fatalf("connectionID=%q err=%v", connectionID, err)
|
||||
}
|
||||
other := payload
|
||||
refCopy := *payload.Ref
|
||||
refCopy.ConnectionID = "storage-other"
|
||||
other.Ref = &refCopy
|
||||
if _, err := inputStorageConnectionID([]domain.NodeInputBinding{{Payload: &payload}, {Payload: &other}}); err == nil {
|
||||
t.Fatal("multiple input storage connections must fail closed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareExecutionNodeResolvesStorageConnectionFromInputBindings(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dataStore := store.NewMemoryStore()
|
||||
connection, err := dataStore.CreateStorageConnection(ctx, domain.CreateStorageConnectionInput{
|
||||
Name: "BOS 动态绑定连接",
|
||||
Provider: domain.StorageProviderBOS,
|
||||
Endpoint: "https://bj.bcebos.com",
|
||||
AccessKeyID: "ak-input-binding-1234",
|
||||
SecretAccessKey: "sk-input-binding-5678",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("create storage connection: %v", err)
|
||||
}
|
||||
runtime, err := newTestRuntime(t, dataStore)
|
||||
if err != nil {
|
||||
t.Fatalf("create runtime: %v", err)
|
||||
}
|
||||
|
||||
node := domain.WorkflowNode{
|
||||
ID: "fanyu-v56-directory",
|
||||
Type: "operator",
|
||||
Data: domain.WorkflowNodeData{
|
||||
Label: "目录读取(多端口)",
|
||||
},
|
||||
}
|
||||
bindings := []domain.NodeInputBinding{
|
||||
{
|
||||
TargetPortID: "sourceDirectory",
|
||||
Payload: &domain.ArtifactPayload{
|
||||
Mode: domain.ArtifactTransferModeRef,
|
||||
Type: domain.PortTypeFile,
|
||||
Ref: &domain.ArtifactReference{
|
||||
ConnectionID: connection.ID,
|
||||
Bucket: "testxyz",
|
||||
Prefix: "datasets/capture1",
|
||||
Kind: domain.ArtifactRefKindDirectory,
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
prepared, err := runtime.prepareExecutionNode(ctx, nil, node, bindings)
|
||||
if err != nil {
|
||||
t.Fatalf("prepare execution node: %v", err)
|
||||
}
|
||||
params := nodeHyperparamsMap(prepared)
|
||||
if params["BOS_ENDPOINT"] != "https://bj.bcebos.com" {
|
||||
t.Fatalf("expected resolved endpoint, got %q", params["BOS_ENDPOINT"])
|
||||
}
|
||||
if params["BOS_ACCESS_KEY_ID"] != "ak-input-binding-1234" || params["BOS_SECRET_ACCESS_KEY"] != "sk-input-binding-5678" {
|
||||
t.Fatalf("expected resolved bos secrets, got %#v", params)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareExecutionNodeResolvesStorageConnectionSecrets(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
dataStore := store.NewMemoryStore()
|
||||
|
||||
50
internal/worker/storage_scope_test.go
Normal file
50
internal/worker/storage_scope_test.go
Normal file
@ -0,0 +1,50 @@
|
||||
package worker
|
||||
|
||||
import (
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestScopedArtifactPathsRejectSymlinkEscape(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
run := filepath.Join(root, "run")
|
||||
os.Mkdir(run, 0700)
|
||||
outside := filepath.Join(root, "secret")
|
||||
os.WriteFile(outside, []byte("fixture"), 0600)
|
||||
inside := filepath.Join(run, "result")
|
||||
os.WriteFile(inside, []byte("data"), 0600)
|
||||
if e := validateScopedArtifactPath(run, inside); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if e := validateScopedArtifactPath(run, outside); e == nil {
|
||||
t.Fatal("outside file accepted")
|
||||
}
|
||||
link := filepath.Join(run, "link")
|
||||
os.Symlink(outside, link)
|
||||
if e := validateScopedArtifactPath(run, link); e == nil {
|
||||
t.Fatal("symlink escape accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestScopedOrganizationKeepsInlineArtifacts(t *testing.T) {
|
||||
data := store.NewMemoryStore()
|
||||
org := domain.OrganizationRecord{ID: "scoped-org", StorageRootPrefix: "company-a/", StorageBucket: "testxyz", StorageConnectionID: "scoped-connection", StorageState: "ready"}
|
||||
data.UpsertOrganization(t.Context(), org)
|
||||
data.UpsertStorageConnection(t.Context(), domain.StorageConnectionRecord{ID: org.StorageConnectionID, OrganizationID: org.ID, RootPrefix: org.StorageRootPrefix, Bucket: org.StorageBucket, Provider: "BOS", Endpoint: "https://bj.bcebos.com"})
|
||||
rt := &Runtime{store: data}
|
||||
wf := &domain.WorkflowRecord{OrganizationID: org.ID}
|
||||
run := &domain.RunRecord{ID: "run-inline", OrganizationID: org.ID}
|
||||
artifacts := []domain.ExecutionArtifact{{ID: "inline", Type: domain.PortTypeJSON, Payload: &domain.ArtifactPayload{Mode: domain.ArtifactTransferModeInline, Type: domain.PortTypeJSON, Inline: map[string]any{"ok": true}}}, {ID: "json-data", Type: domain.PortTypeJSON, Data: map[string]any{"value": 1}}}
|
||||
out, err := rt.normalizeOutputArtifacts(t.Context(), wf, run, "node", artifacts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, a := range out {
|
||||
if a.Payload == nil || a.Payload.Mode != domain.ArtifactTransferModeInline {
|
||||
t.Fatalf("inline artifact lost: %+v", a)
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -140,6 +140,7 @@ data:
|
||||
WORKER_MISSING_JOB_MAX_RECOVERIES: "__WORKER_MISSING_JOB_MAX_RECOVERIES__"
|
||||
WORKER_API_BASE_URL: "__WORKER_API_BASE_URL__"
|
||||
BOS_ENDPOINT: "__BOS_ENDPOINT__"
|
||||
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID: "__ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID__"
|
||||
K8S_NAMESPACE: "__NAMESPACE__"
|
||||
K8S_IMAGE_PULL_SECRET_NAME: "__K8S_IMAGE_PULL_SECRET_NAME__"
|
||||
WORKSPACE_PVC_NAME: "__WORKSPACE_PVC_NAME__"
|
||||
@ -324,6 +325,11 @@ spec:
|
||||
configMapKeyRef:
|
||||
name: droboticflow-config
|
||||
key: WORKER_API_BASE_URL
|
||||
- name: ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: droboticflow-config
|
||||
key: ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID
|
||||
- name: BOS_ENDPOINT
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
|
||||
@ -49,3 +49,6 @@ export CUSTOM_OPERATOR_IMAGE_ARCHIVE_MAX_MIB="${CUSTOM_OPERATOR_IMAGE_ARCHIVE_MA
|
||||
|
||||
export AI_API_BASE_URL="${AI_API_BASE_URL:-}"
|
||||
export AI_MODEL="${AI_MODEL:-}"
|
||||
|
||||
# New organizations inherit a scoped child IAM identity, never this source key.
|
||||
export ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID="${ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID:-storage-485ec27477c334ada4372c3fca91154a}"
|
||||
|
||||
42
scripts/drosync-poc-abort-regression.py
Normal file
42
scripts/drosync-poc-abort-regression.py
Normal file
@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real POC abort/lease regression; creates only isolated sessions and no objects."""
|
||||
import argparse,json,pathlib,time,uuid,urllib.request,urllib.error
|
||||
p=argparse.ArgumentParser();p.add_argument('--server',required=True);p.add_argument('--token-file',required=True);p.add_argument('--space',required=True);p.add_argument('--prefix',required=True);p.add_argument('--output',required=True);a=p.parse_args()
|
||||
token=pathlib.Path(a.token_file).read_text().strip();results=[];sessions=[]
|
||||
def api(method,path,body=None):
|
||||
req=urllib.request.Request(a.server.rstrip('/')+path,data=json.dumps(body).encode() if body is not None else None,headers={'Authorization':'Bearer '+token,'Content-Type':'application/json','Idempotency-Key':str(uuid.uuid4())},method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req,timeout=30)as r:return r.status,json.load(r)
|
||||
except urllib.error.HTTPError as e:return e.code,json.load(e)
|
||||
def check(name,ok,detail):
|
||||
results.append({'name':name,'status':'PASS' if ok else 'FAIL','detail':detail});print(json.dumps(results[-1],ensure_ascii=False),flush=True)
|
||||
if not ok:raise AssertionError(name)
|
||||
def create(path):
|
||||
code,s=api('POST','/api/external/object-sync/sessions',{'syncSpaceId':a.space,'mappingId':str(uuid.uuid4()),'clientVersion':'1.0.0','protocolVersion':'v1'});assert code==201,s
|
||||
sid=s['id'];sessions.append(sid)
|
||||
code,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':True,'items':[{'clientItemId':str(uuid.uuid4()),'relativePath':path,'kind':'file','sizeBytes':0,'sha256':'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855','contentType':'application/octet-stream'}]});assert code==202,r
|
||||
code,r=api('POST',f'/api/external/object-sync/sessions/{sid}/plan',{});assert code==202,r
|
||||
for _ in range(120):
|
||||
code,r=api('GET',f'/api/external/object-sync/sessions/{sid}/items');items=r.get('items',[])
|
||||
if items and items[0]['status']=='planned':return sid,items[0]
|
||||
time.sleep(.5)
|
||||
raise RuntimeError('plan did not finish')
|
||||
try:
|
||||
path=a.prefix.rstrip('/')+'/abort-regression-'+uuid.uuid4().hex+'/empty.bin'
|
||||
sa,ia=create(path);sb,ib=create(path)
|
||||
c,r=api('POST',f"/api/external/object-sync/items/{ia['id']}/reserve",{});assert c==200,r;old=r.get('item',r)['fencingToken']
|
||||
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/reserve",{});check('competing reserve denied',c==409 and r.get('code')=='lease_held',{'http':c})
|
||||
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/abort',{});check('abort acknowledged',c==200,{'sessionId':sa,'http':c})
|
||||
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/reserve",{});new=r.get('item',r).get('fencingToken');check('immediate takeover with increased fencing',c==200 and new is not None and new>old,{'http':c,'oldFencing':old,'newFencing':new,'sessionId':sb})
|
||||
for action in ['heartbeat','reserve','commit']:
|
||||
c,r=api('POST',f"/api/external/object-sync/items/{ia['id']}/{action}",{});check('aborted '+action+' denied',c==409,{'http':c,'code':r.get('code')})
|
||||
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/abort',{});assert c==200,r
|
||||
c,r=api('POST',f"/api/external/object-sync/items/{ib['id']}/heartbeat",{});check('repeat abort preserves new owner',c==200 and r.get('renewed') is True,{'http':c})
|
||||
c,r=api('POST',f'/api/external/object-sync/sessions/{sa}/complete',{});check('aborted session cannot complete',c==409,{'http':c})
|
||||
except Exception as e:
|
||||
if not results or results[-1]['status']!='FAIL':results.append({'name':'execution','status':'FAIL','detail':str(e)})
|
||||
finally:
|
||||
for sid in sessions:api('POST',f'/api/external/object-sync/sessions/{sid}/abort',{})
|
||||
summary={'timestamp':time.time(),'scope':'two sessions, same existing device; no object bytes uploaded','results':results,'passed':sum(r['status']=='PASS' for r in results),'failed':sum(r['status']=='FAIL' for r in results)}
|
||||
pathlib.Path(a.output).write_text(json.dumps(summary,ensure_ascii=False,indent=2))
|
||||
raise SystemExit(1 if summary['failed'] else 0)
|
||||
30
scripts/drosync-poc-manifest-regression.py
Normal file
30
scripts/drosync-poc-manifest-regression.py
Normal file
@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Manifest validation tests: metadata only, no file upload or capacity claim."""
|
||||
import argparse,json,pathlib,time,uuid,urllib.request,urllib.error
|
||||
p=argparse.ArgumentParser();p.add_argument('--server',required=True);p.add_argument('--token-file',required=True);p.add_argument('--space',required=True);p.add_argument('--prefix',required=True);p.add_argument('--output',required=True);a=p.parse_args()
|
||||
token=pathlib.Path(a.token_file).read_text().strip();results=[];sid=None
|
||||
def api(method,path,body=None):
|
||||
req=urllib.request.Request(a.server.rstrip('/')+path,data=json.dumps(body).encode() if body is not None else None,headers={'Authorization':'Bearer '+token,'Content-Type':'application/json','Idempotency-Key':str(uuid.uuid4())},method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req,timeout=30)as r:return r.status,json.load(r)
|
||||
except urllib.error.HTTPError as e:return e.code,json.load(e)
|
||||
def check(name,expected,actual,detail=None):
|
||||
row={'name':name,'expected':expected,'actual':actual,'status':'PASS' if expected==actual else 'FAIL','detail':detail};results.append(row);print(json.dumps(row,ensure_ascii=False),flush=True)
|
||||
def item(**changes):
|
||||
v={'clientItemId':str(uuid.uuid4()),'relativePath':a.prefix.rstrip('/')+'/'+uuid.uuid4().hex+'.txt','sizeBytes':0,'kind':'file','sha256':'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'};v.update(changes);return v
|
||||
try:
|
||||
c,s=api('POST','/api/external/object-sync/sessions',{'syncSpaceId':a.space,'mappingId':str(uuid.uuid4()),'clientVersion':'1.0.0','protocolVersion':'v1'});assert c==201,s;sid=s['id'];limit=s['policySnapshot']['maxObjectBytes'];quota=s['policySnapshot']['maxSessionBytes']
|
||||
d=item()
|
||||
cases=[('zero page number',{'pageNumber':0,'items':[item()]},400),('empty page',{'items':[]},400),('1001 entries in one page',{'items':[item() for _ in range(1001)]},400),('duplicate client ID',{'items':[d,d]},400),('missing client ID',{'items':[item(clientItemId='')]},400),('nonhex SHA',{'items':[item(sha256='g'*64)]},400),('short SHA',{'items':[item(sha256='a'*63)]},400),('negative bytes',{'items':[item(sizeBytes=-1)]},413),('above object policy limit',{'items':[item(sizeBytes=limit+1)]},413),('non-file kind',{'items':[item(kind='directory')]},400),('parent traversal',{'items':[item(relativePath=a.prefix+'/../escape')]},400),('absolute path',{'items':[item(relativePath='/absolute')]},400),('empty path segment',{'items':[item(relativePath=a.prefix+'//file')]},400),('control character',{'items':[item(relativePath=a.prefix+'/line\nfeed')]},400),('above session byte quota',{'items':[item(sizeBytes=limit) for _ in range(quota//limit+1)]},413)]
|
||||
for name,body,expected in cases:
|
||||
c,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':False,**body});check(name,expected,c,{'code':r.get('code')})
|
||||
c,r=api('GET',f'/api/external/object-sync/sessions/{sid}/status');check('rejected pages preserve manifest count',0,r['counts'].get('manifest',0));check('rejected pages preserve total bytes',0,r.get('totalBytes',0))
|
||||
valid=[item() for _ in range(999)]+[item(relativePath=a.prefix+'/中文 空格/特殊#%.txt')]
|
||||
c,r=api('POST',f'/api/external/object-sync/sessions/{sid}/manifest-pages',{'pageNumber':1,'final':True,'items':valid});check('exactly 1000 entries with Unicode accepted',202,c)
|
||||
c,r=api('GET',f'/api/external/object-sync/sessions/{sid}/status');check('exact accepted count',1000,r['counts'].get('manifest'))
|
||||
except Exception as e:results.append({'name':'execution','status':'FAIL','detail':str(e)})
|
||||
finally:
|
||||
if sid:api('POST',f'/api/external/object-sync/sessions/{sid}/abort',{})
|
||||
summary={'scope':'metadata validation only; no 1TiB or 20TiB actual upload','sessionId':sid,'timestamp':time.time(),'results':results,'passed':sum(x['status']=='PASS'for x in results),'failed':sum(x['status']=='FAIL'for x in results)}
|
||||
pathlib.Path(a.output).write_text(json.dumps(summary,ensure_ascii=False,indent=2))
|
||||
raise SystemExit(1 if summary['failed'] else 0)
|
||||
@ -25,6 +25,7 @@ WORKER_POLL_INTERVAL_MS="${WORKER_POLL_INTERVAL_MS:-2000}"
|
||||
DROBOTICFLOW_SHARED_MEMORY_SIZE="${DROBOTICFLOW_SHARED_MEMORY_SIZE:-8Gi}"
|
||||
WORKER_API_BASE_URL="${WORKER_API_BASE_URL:-http://droboticflow-worker:3002}"
|
||||
BOS_ENDPOINT="${BOS_ENDPOINT:-}"
|
||||
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID="${ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID:-}"
|
||||
K8S_IMAGE_PULL_SECRET_NAME="${K8S_IMAGE_PULL_SECRET_NAME:-droboticflow-registry}"
|
||||
WORKSPACE_PVC_NAME="${WORKSPACE_PVC_NAME:-droboticflow-workspace}"
|
||||
WORKSPACE_MOUNT_PATH="${WORKSPACE_MOUNT_PATH:-/workspace}"
|
||||
@ -113,6 +114,7 @@ export \
|
||||
DROBOTICFLOW_SHARED_MEMORY_SIZE \
|
||||
WORKER_API_BASE_URL \
|
||||
BOS_ENDPOINT \
|
||||
ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID \
|
||||
K8S_IMAGE_PULL_SECRET_NAME \
|
||||
WORKSPACE_PVC_NAME \
|
||||
WORKSPACE_MOUNT_PATH \
|
||||
@ -168,6 +170,7 @@ replacements = {
|
||||
"__DROBOTICFLOW_SHARED_MEMORY_SIZE__": os.environ["DROBOTICFLOW_SHARED_MEMORY_SIZE"],
|
||||
"__WORKER_API_BASE_URL__": os.environ["WORKER_API_BASE_URL"],
|
||||
"__BOS_ENDPOINT__": os.environ["BOS_ENDPOINT"],
|
||||
"__ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID__": os.environ["ORGANIZATION_STORAGE_SOURCE_CONNECTION_ID"],
|
||||
"__K8S_IMAGE_PULL_SECRET_NAME__": os.environ["K8S_IMAGE_PULL_SECRET_NAME"],
|
||||
"__WORKSPACE_PVC_NAME__": os.environ["WORKSPACE_PVC_NAME"],
|
||||
"__WORKSPACE_MOUNT_PATH__": os.environ["WORKSPACE_MOUNT_PATH"],
|
||||
|
||||
@ -97,6 +97,18 @@ export function SuperAdminConsole() {
|
||||
});
|
||||
}, [loadUsers, selectedOrganizationId]);
|
||||
|
||||
const hasPendingStorage = organizations.some((item) => item.storageState === "pending");
|
||||
useEffect(() => {
|
||||
if (!hasPendingStorage) return;
|
||||
const timer = window.setInterval(() => {
|
||||
void adminApi
|
||||
.listOrganizations()
|
||||
.then((result) => setOrganizations(result.items))
|
||||
.catch(() => {});
|
||||
}, 5000);
|
||||
return () => window.clearInterval(timer);
|
||||
}, [hasPendingStorage]);
|
||||
|
||||
const createOrganization = async (event: React.FormEvent) => {
|
||||
event.preventDefault();
|
||||
const name = organizationDraft.name.trim();
|
||||
@ -112,7 +124,26 @@ export function SuperAdminConsole() {
|
||||
setSelectedOrganizationId(created.id);
|
||||
setOrganizationDraft(emptyOrganizationDraft);
|
||||
} catch (err) {
|
||||
setError(getActionErrorMessage(err, "创建组织失败"));
|
||||
const message = getActionErrorMessage(err, "创建组织失败");
|
||||
try {
|
||||
await loadOrganizations();
|
||||
} catch {
|
||||
/* Preserve the original provisioning failure. */
|
||||
}
|
||||
setError(message);
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
const retryStorageProvisioning = async (organization: Organization) => {
|
||||
setSaving(true);
|
||||
setError("");
|
||||
try {
|
||||
const updated = await adminApi.provisionOrganizationStorage(organization.id);
|
||||
setOrganizations((items) => items.map((item) => (item.id === updated.id ? updated : item)));
|
||||
} catch (err) {
|
||||
setError(getActionErrorMessage(err, "配置组织存储失败"));
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
@ -270,9 +301,12 @@ export function SuperAdminConsole() {
|
||||
<input
|
||||
value={organizationDraft.slug}
|
||||
onChange={(event) => setOrganizationDraft((item) => ({ ...item, slug: event.target.value }))}
|
||||
placeholder="slug"
|
||||
placeholder="组织标识,例如 company-a"
|
||||
className="w-full rounded-lg border border-[#2a2b2f] bg-[#0a0a0a] px-3 py-2 text-sm text-white outline-none focus:border-blue-500"
|
||||
/>
|
||||
<p className="text-xs text-gray-400">
|
||||
新组织将自动配置独立的顶层存储目录。组织标识确定后,目录不会随组织改名而改变。
|
||||
</p>
|
||||
<button
|
||||
type="submit"
|
||||
disabled={saving || !organizationDraft.name.trim() || !organizationDraft.slug.trim()}
|
||||
@ -303,6 +337,12 @@ export function SuperAdminConsole() {
|
||||
<span>
|
||||
<span className="block text-sm font-semibold text-white">{organization.name}</span>
|
||||
<span className="block text-xs text-gray-500">{organization.slug}</span>
|
||||
{organization.storageRootPrefix && (
|
||||
<span className="block text-xs text-blue-300">
|
||||
{organization.storageBucket}/{organization.storageRootPrefix} ·{" "}
|
||||
{organization.storageState === "ready" ? "存储已就绪" : "存储配置中,自动重试"}
|
||||
</span>
|
||||
)}
|
||||
</span>
|
||||
<span className="text-xs text-gray-400">{statusLabel(organization.status)}</span>
|
||||
</button>
|
||||
@ -317,6 +357,21 @@ export function SuperAdminConsole() {
|
||||
<div>
|
||||
<h2 className="text-xl font-bold text-white">{selectedOrganization.name}</h2>
|
||||
<div className="mt-1 text-xs text-gray-500">{selectedOrganization.id}</div>
|
||||
{selectedOrganization.storageRootPrefix && (
|
||||
<div className="mt-2 text-sm text-blue-300">
|
||||
授权目录:{selectedOrganization.storageBucket}/{selectedOrganization.storageRootPrefix}
|
||||
</div>
|
||||
)}
|
||||
{selectedOrganization.storageRootPrefix && selectedOrganization.storageState !== "ready" && (
|
||||
<button
|
||||
type="button"
|
||||
disabled={saving}
|
||||
onClick={() => void retryStorageProvisioning(selectedOrganization)}
|
||||
className="mt-2 rounded-lg bg-blue-600 px-3 py-2 text-sm text-white disabled:opacity-50"
|
||||
>
|
||||
重试配置存储
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex items-center gap-2">
|
||||
<button
|
||||
@ -380,6 +435,7 @@ export function SuperAdminConsole() {
|
||||
disabled={
|
||||
saving ||
|
||||
!selectedOrganizationId ||
|
||||
(Boolean(selectedOrganization?.storageRootPrefix) && selectedOrganization?.storageState !== "ready") ||
|
||||
!userDraft.name.trim() ||
|
||||
!userDraft.email.trim() ||
|
||||
!userDraft.password.trim()
|
||||
|
||||
@ -210,6 +210,7 @@ export default function ObjectStorageBrowser() {
|
||||
const initialSelectionAppliedRef = useRef(false);
|
||||
|
||||
const selectedConnection = connections.find((item) => item.id === selectedConnectionId) || null;
|
||||
const connectionRoot = normalizePrefix(selectedConnection?.rootPrefix);
|
||||
const previewURL = selectedItem
|
||||
? objectStorageApi.buildContentURL({
|
||||
connectionId: selectedConnectionId,
|
||||
@ -334,7 +335,7 @@ export default function ObjectStorageBrowser() {
|
||||
Boolean(initialBucket || initialPrefix || initialConnectionId) &&
|
||||
(!initialConnectionId || selectedConnectionId === initialConnectionId);
|
||||
initialSelectionAppliedRef.current = true;
|
||||
setCurrentPrefix(shouldApplyInitialSelection ? initialPrefix : "");
|
||||
setCurrentPrefix(shouldApplyInitialSelection && initialPrefix ? initialPrefix : connectionRoot);
|
||||
setItems([]);
|
||||
setObjectMarker("");
|
||||
setNextObjectMarker("");
|
||||
@ -349,7 +350,7 @@ export default function ObjectStorageBrowser() {
|
||||
return;
|
||||
}
|
||||
void reloadOverview(shouldApplyInitialSelection ? initialBucket : "");
|
||||
}, [selectedConnectionId]);
|
||||
}, [selectedConnectionId, connectionRoot]);
|
||||
|
||||
const reloadObjects = async (preferredKey = "", prefixOverride?: string, markerOverride = objectMarker) => {
|
||||
const requestId = objectsRequestIdRef.current + 1;
|
||||
@ -372,6 +373,7 @@ export default function ObjectStorageBrowser() {
|
||||
if (objectsRequestIdRef.current !== requestId) {
|
||||
return result;
|
||||
}
|
||||
setCurrentPrefix(result.prefix || connectionRoot);
|
||||
setItems(result.items || []);
|
||||
setNextObjectMarker(result.nextMarker || "");
|
||||
setHasMoreObjects(Boolean(result.hasMore));
|
||||
@ -586,15 +588,20 @@ export default function ObjectStorageBrowser() {
|
||||
});
|
||||
await loadConnections(created.id);
|
||||
} else if (selectedConnectionId) {
|
||||
await storageConnectionsApi.update(selectedConnectionId, {
|
||||
name,
|
||||
description: connectionDraft.description.trim(),
|
||||
provider: "BOS",
|
||||
endpoint,
|
||||
bucket,
|
||||
accessKeyId: connectionDraft.accessKeyId.trim(),
|
||||
secretAccessKey: connectionDraft.secretAccessKey.trim(),
|
||||
});
|
||||
await storageConnectionsApi.update(
|
||||
selectedConnectionId,
|
||||
selectedConnection?.rootPrefix
|
||||
? { name, description: connectionDraft.description.trim() }
|
||||
: {
|
||||
name,
|
||||
description: connectionDraft.description.trim(),
|
||||
provider: "BOS",
|
||||
endpoint,
|
||||
bucket,
|
||||
accessKeyId: connectionDraft.accessKeyId.trim(),
|
||||
secretAccessKey: connectionDraft.secretAccessKey.trim(),
|
||||
},
|
||||
);
|
||||
await loadConnections(selectedConnectionId);
|
||||
}
|
||||
setIsConnectionFormOpen(false);
|
||||
@ -791,7 +798,7 @@ export default function ObjectStorageBrowser() {
|
||||
setObjectActionError("请输入目标目录前缀。");
|
||||
return;
|
||||
}
|
||||
const destinationPrefix = moveDestinationInput === "/" ? "" : normalizePrefix(moveDestinationInput);
|
||||
const destinationPrefix = moveDestinationInput === "/" ? connectionRoot : normalizePrefix(moveDestinationInput);
|
||||
if (moveDestinationInput !== "/" && !destinationPrefix) {
|
||||
setObjectActionError("请输入目标目录前缀。");
|
||||
return;
|
||||
@ -1122,6 +1129,7 @@ export default function ObjectStorageBrowser() {
|
||||
<div className="space-y-2">
|
||||
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Endpoint</label>
|
||||
<input
|
||||
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
|
||||
value={connectionDraft.endpoint}
|
||||
onChange={(event) => setConnectionDraft((previous) => ({ ...previous, endpoint: event.target.value }))}
|
||||
className="w-full rounded-xl border border-[#2a2b2f] bg-[#0b0c10] px-4 py-3 text-sm text-white focus:border-blue-500 focus:outline-none"
|
||||
@ -1132,6 +1140,7 @@ export default function ObjectStorageBrowser() {
|
||||
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">绑定 Bucket</label>
|
||||
<input
|
||||
data-testid="connection-bucket-input"
|
||||
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
|
||||
value={connectionDraft.bucket}
|
||||
onChange={(event) => setConnectionDraft((previous) => ({ ...previous, bucket: event.target.value }))}
|
||||
className="w-full rounded-xl border border-[#2a2b2f] bg-[#0b0c10] px-4 py-3 text-sm text-white focus:border-blue-500 focus:outline-none"
|
||||
@ -1155,6 +1164,7 @@ export default function ObjectStorageBrowser() {
|
||||
<div className="space-y-2">
|
||||
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Access Key ID</label>
|
||||
<input
|
||||
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
|
||||
value={connectionDraft.accessKeyId}
|
||||
onChange={(event) =>
|
||||
setConnectionDraft((previous) => ({
|
||||
@ -1174,6 +1184,7 @@ export default function ObjectStorageBrowser() {
|
||||
<label className="text-xs uppercase tracking-[0.2em] text-gray-500">Secret Access Key</label>
|
||||
<input
|
||||
type="password"
|
||||
disabled={connectionFormMode === "edit" && Boolean(selectedConnection?.rootPrefix)}
|
||||
value={connectionDraft.secretAccessKey}
|
||||
onChange={(event) =>
|
||||
setConnectionDraft((previous) => ({
|
||||
@ -1250,6 +1261,11 @@ export default function ObjectStorageBrowser() {
|
||||
>
|
||||
{selectedConnection ? (
|
||||
<div className="space-y-4">
|
||||
{connectionRoot && (
|
||||
<div className="text-sm text-blue-200">
|
||||
授权目录:{selectedBucket}/{connectionRoot}/(由平台管理员配置)
|
||||
</div>
|
||||
)}
|
||||
<div>
|
||||
<div className="text-xs uppercase tracking-[0.2em] text-gray-500">连接名称</div>
|
||||
<div className="mt-2 text-sm font-medium text-white">{selectedConnection.name}</div>
|
||||
@ -1272,7 +1288,7 @@ export default function ObjectStorageBrowser() {
|
||||
</div>
|
||||
{selectedBucket ? (
|
||||
<div className="rounded-xl border border-emerald-500/20 bg-emerald-500/10 px-3 py-3 text-sm text-emerald-100">
|
||||
当前对象浏览和操作都限定在 bucket `{selectedBucket}` 下。
|
||||
当前对象浏览和操作都限定在 `{selectedBucket}/{connectionRoot ? `${connectionRoot}/` : ""}` 下。
|
||||
</div>
|
||||
) : (
|
||||
<div className="rounded-xl border border-amber-500/20 bg-amber-500/10 px-3 py-3 text-sm text-amber-100">
|
||||
@ -1318,32 +1334,34 @@ export default function ObjectStorageBrowser() {
|
||||
<div className="mt-4 flex flex-wrap items-center gap-2 text-xs text-gray-500">
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setCurrentPrefix("")}
|
||||
onClick={() => setCurrentPrefix(connectionRoot)}
|
||||
className={cn(
|
||||
"rounded-full border px-3 py-1.5",
|
||||
currentPrefix === ""
|
||||
currentPrefix === connectionRoot
|
||||
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
|
||||
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
|
||||
)}
|
||||
>
|
||||
根目录
|
||||
{connectionRoot ? "授权根目录" : "根目录"}
|
||||
</button>
|
||||
{prefixSegments.map((segment) => (
|
||||
<button
|
||||
key={segment.value}
|
||||
type="button"
|
||||
onClick={() => setCurrentPrefix(segment.value)}
|
||||
className={cn(
|
||||
"rounded-full border px-3 py-1.5",
|
||||
currentPrefix === segment.value
|
||||
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
|
||||
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
|
||||
)}
|
||||
>
|
||||
{segment.name}
|
||||
</button>
|
||||
))}
|
||||
{currentPrefix && (
|
||||
{prefixSegments
|
||||
.filter((segment) => !connectionRoot || segment.value.startsWith(connectionRoot + "/"))
|
||||
.map((segment) => (
|
||||
<button
|
||||
key={segment.value}
|
||||
type="button"
|
||||
onClick={() => setCurrentPrefix(segment.value)}
|
||||
className={cn(
|
||||
"rounded-full border px-3 py-1.5",
|
||||
currentPrefix === segment.value
|
||||
? "border-blue-500/40 bg-blue-500/10 text-blue-200"
|
||||
: "border-[#2a2b2f] bg-[#101114] hover:text-white",
|
||||
)}
|
||||
>
|
||||
{segment.name}
|
||||
</button>
|
||||
))}
|
||||
{currentPrefix && currentPrefix !== connectionRoot && (
|
||||
<button
|
||||
type="button"
|
||||
onClick={() => setCurrentPrefix(parentPrefix(currentPrefix))}
|
||||
@ -1634,7 +1652,8 @@ export default function ObjectStorageBrowser() {
|
||||
{selectedObjectKeys.length > 0
|
||||
? `将移动已勾选的 ${selectedObjectKeys.length} 项。`
|
||||
: "未勾选时移动当前选中的 1 项。"}
|
||||
输入 / 会移动到 bucket 根目录。目录会作为目录本身移动到目标目录下,并递归保留内部所有内容。
|
||||
输入 / 会移动到{connectionRoot ? "授权根目录" : "bucket 根目录"}
|
||||
。目录会作为目录本身移动到目标目录下,并递归保留内部所有内容。
|
||||
</div>
|
||||
</div>
|
||||
<button
|
||||
|
||||
@ -2,6 +2,8 @@ import { apiRequest } from "./client";
|
||||
import type { Organization, User, UserRole } from "../../types";
|
||||
|
||||
export const adminApi = {
|
||||
provisionOrganizationStorage: (id: string) =>
|
||||
apiRequest<Organization>(`/api/admin/organizations/${id}/storage/provision`, { method: "POST" }, {}),
|
||||
listOrganizations: () => apiRequest<{ items: Organization[] }>("/api/admin/organizations"),
|
||||
createOrganization: (payload: { name: string; slug: string; status?: Organization["status"] }) =>
|
||||
apiRequest<Organization>("/api/admin/organizations", { method: "POST" }, payload),
|
||||
|
||||
@ -2,6 +2,7 @@ import type { StorageConnection } from "../../types";
|
||||
import { apiRequest } from "./client";
|
||||
|
||||
type StorageConnectionPayload = {
|
||||
rootPrefix?: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
provider: "BOS";
|
||||
|
||||
@ -22,6 +22,10 @@ export interface User {
|
||||
}
|
||||
|
||||
export interface Organization {
|
||||
storageRootPrefix?: string;
|
||||
storageBucket?: string;
|
||||
storageState?: "pending" | "ready";
|
||||
storageConnectionId?: string;
|
||||
id: string;
|
||||
name: string;
|
||||
slug: string;
|
||||
@ -743,6 +747,7 @@ export interface ScanConfig {
|
||||
}
|
||||
|
||||
export interface StorageConnection {
|
||||
rootPrefix?: string;
|
||||
id: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
|
||||
16
tools/drobotic-pull/Cargo.toml
Normal file
16
tools/drobotic-pull/Cargo.toml
Normal file
@ -0,0 +1,16 @@
|
||||
[package]
|
||||
name = "drobotic-pull"
|
||||
version = "1.0.0"
|
||||
edition = "2021"
|
||||
description = "Fast, secure direct-from-BOS directory and file downloader"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1.0"
|
||||
clap = { version = "4.5", features = ["derive", "env"] }
|
||||
futures-util = "0.3"
|
||||
indicatif = { version = "0.17", features = ["tokio"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls", "stream", "cookies"] }
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
serde_json = "1.0"
|
||||
tokio = { version = "1.40", features = ["fs", "io-util", "macros", "rt-multi-thread", "sync", "time"] }
|
||||
urlencoding = "2.1"
|
||||
443
tools/drobotic-pull/src/main.rs
Normal file
443
tools/drobotic-pull/src/main.rs
Normal file
@ -0,0 +1,443 @@
|
||||
use anyhow::{anyhow, Context, Result};
|
||||
use clap::Parser;
|
||||
use futures_util::StreamExt;
|
||||
use indicatif::{MultiProgress, ProgressBar, ProgressStyle};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use tokio::sync::Semaphore;
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(
|
||||
name = "drobotic-pull",
|
||||
about = "High-speed, direct-from-BOS directory and file downloader with zero AK/SK exposure",
|
||||
version = "1.0.0"
|
||||
)]
|
||||
struct Cli {
|
||||
/// Droboticflow API base URL (e.g., http://120.48.164.19)
|
||||
#[arg(long, env = "DROBOTICFLOW_API_URL")]
|
||||
api: Option<String>,
|
||||
|
||||
/// Login email for platform authentication
|
||||
#[arg(long, env = "DROBOTICFLOW_EMAIL")]
|
||||
email: Option<String>,
|
||||
|
||||
/// Login password for platform authentication
|
||||
#[arg(long, env = "DROBOTICFLOW_PASSWORD")]
|
||||
password: Option<String>,
|
||||
|
||||
/// Storage connection ID (optional if only 1 connection exists)
|
||||
#[arg(long, env = "DROBOTICFLOW_CONNECTION_ID")]
|
||||
connection: Option<String>,
|
||||
|
||||
/// Storage bucket name (defaults to connection default bucket)
|
||||
#[arg(long)]
|
||||
bucket: Option<String>,
|
||||
|
||||
/// BOS remote directory prefix to download (e.g., ubbdata/2026_09_09-09_21_24)
|
||||
#[arg(long, short = 'p')]
|
||||
prefix: Option<String>,
|
||||
|
||||
/// Direct Manifest API URL (alternative to email/password)
|
||||
#[arg(long)]
|
||||
manifest_url: Option<String>,
|
||||
|
||||
/// Direct local manifest JSON file path
|
||||
#[arg(long)]
|
||||
manifest_file: Option<PathBuf>,
|
||||
|
||||
/// Session cookie string (e.g. droboticflow_session=...)
|
||||
#[arg(long, env = "DROBOTICFLOW_COOKIE")]
|
||||
cookie: Option<String>,
|
||||
|
||||
/// Local destination directory
|
||||
#[arg(long, short = 'o', default_value = "./downloads")]
|
||||
output: PathBuf,
|
||||
|
||||
/// Number of concurrent file downloads
|
||||
#[arg(long, short = 'j', default_value_t = 8)]
|
||||
threads: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct ManifestResponse {
|
||||
bucket: String,
|
||||
base_prefix: String,
|
||||
total_files: usize,
|
||||
total_bytes: u64,
|
||||
files: Vec<ManifestFileItem>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
struct ManifestFileItem {
|
||||
key: String,
|
||||
relative_path: String,
|
||||
size_bytes: u64,
|
||||
download_url: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct StorageConnectionSummary {
|
||||
id: String,
|
||||
name: String,
|
||||
#[serde(default)]
|
||||
bucket: String,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let args = Cli::parse();
|
||||
|
||||
println!("============================================================");
|
||||
println!(" drobotic-pull: Direct BOS Directory Downloader v1.0.0");
|
||||
println!("============================================================");
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(300))
|
||||
.cookie_store(true)
|
||||
.build()
|
||||
.context("create HTTP client")?;
|
||||
|
||||
let manifest = resolve_manifest(&client, &args).await?;
|
||||
|
||||
if manifest.files.is_empty() {
|
||||
println!("ℹ No files found under the specified prefix.");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
println!("📦 Remote Bucket : {}", manifest.bucket);
|
||||
println!("📁 Remote Prefix : {}", manifest.base_prefix);
|
||||
println!(
|
||||
"📄 Total Files : {} (Total Size: {})",
|
||||
manifest.total_files,
|
||||
format_bytes(manifest.total_bytes)
|
||||
);
|
||||
println!("💾 Local Output : {}", args.output.display());
|
||||
println!("⚡ Concurrency : {} workers", args.threads);
|
||||
println!("------------------------------------------------------------");
|
||||
|
||||
tokio::fs::create_dir_all(&args.output)
|
||||
.await
|
||||
.context("create local destination output directory")?;
|
||||
|
||||
let multi_progress = MultiProgress::new();
|
||||
|
||||
let total_style = ProgressStyle::default_bar()
|
||||
.template("{spinner:.green} [{elapsed_precise}] [{bar:40.cyan/blue}] {bytes}/{total_bytes} ({percent}%) | {msg} | {eta_precise} ETA")?
|
||||
.progress_chars("█▓▒░ ");
|
||||
|
||||
let total_progress = multi_progress.add(ProgressBar::new(manifest.total_bytes));
|
||||
total_progress.set_style(total_style);
|
||||
|
||||
let completed_files = Arc::new(AtomicU64::new(0));
|
||||
let skipped_files = Arc::new(AtomicU64::new(0));
|
||||
let failed_files = Arc::new(AtomicU64::new(0));
|
||||
let total_file_count = manifest.files.len() as u64;
|
||||
|
||||
let semaphore = Arc::new(Semaphore::new(args.threads));
|
||||
let mut tasks = Vec::with_capacity(manifest.files.len());
|
||||
|
||||
for item in manifest.files {
|
||||
let permit = semaphore.clone().acquire_owned().await.unwrap();
|
||||
let client_clone = client.clone();
|
||||
let output_root = args.output.clone();
|
||||
let total_bar = total_progress.clone();
|
||||
let completed_counter = completed_files.clone();
|
||||
let skipped_counter = skipped_files.clone();
|
||||
let failed_counter = failed_files.clone();
|
||||
|
||||
let task = tokio::spawn(async move {
|
||||
let _permit = permit;
|
||||
let result = download_file(&client_clone, &output_root, &item, &total_bar).await;
|
||||
|
||||
match result {
|
||||
Ok(DownloadOutcome::Downloaded) => {
|
||||
let completed = completed_counter.fetch_add(1, Ordering::Relaxed) + 1;
|
||||
let skipped = skipped_counter.load(Ordering::Relaxed);
|
||||
total_bar.set_message(format!("{}/{} files", completed + skipped, total_file_count));
|
||||
}
|
||||
Ok(DownloadOutcome::Skipped) => {
|
||||
let skipped = skipped_counter.fetch_add(1, Ordering::Relaxed) + 1;
|
||||
let completed = completed_counter.load(Ordering::Relaxed);
|
||||
total_bar.set_message(format!("{}/{} files", completed + skipped, total_file_count));
|
||||
}
|
||||
Err(err) => {
|
||||
failed_counter.fetch_add(1, Ordering::Relaxed);
|
||||
eprintln!("\n❌ Failed to download {}: {:?}", item.relative_path, err);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
tasks.push(task);
|
||||
}
|
||||
|
||||
for task in tasks {
|
||||
let _ = task.await;
|
||||
}
|
||||
|
||||
total_progress.finish_with_message("Download Complete");
|
||||
|
||||
let final_completed = completed_files.load(Ordering::Relaxed);
|
||||
let final_skipped = skipped_files.load(Ordering::Relaxed);
|
||||
let final_failed = failed_files.load(Ordering::Relaxed);
|
||||
|
||||
println!("------------------------------------------------------------");
|
||||
println!("✅ All operations finished!");
|
||||
println!(" - Downloaded : {} files", final_completed);
|
||||
println!(" - Skipped : {} files (already present & size matched)", final_skipped);
|
||||
if final_failed > 0 {
|
||||
println!(" - Failed : {} files (please retry)", final_failed);
|
||||
}
|
||||
println!("📂 Files saved to: {}", args.output.canonicalize().unwrap_or(args.output).display());
|
||||
println!("============================================================");
|
||||
|
||||
if final_failed > 0 {
|
||||
Err(anyhow!("{} files failed to download", final_failed))
|
||||
} else {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
enum DownloadOutcome {
|
||||
Downloaded,
|
||||
Skipped,
|
||||
}
|
||||
|
||||
async fn download_file(
|
||||
client: &reqwest::Client,
|
||||
output_root: &Path,
|
||||
item: &ManifestFileItem,
|
||||
total_bar: &ProgressBar,
|
||||
) -> Result<DownloadOutcome> {
|
||||
let clean_relative = item
|
||||
.relative_path
|
||||
.trim_start_matches('/')
|
||||
.replace('\\', "/");
|
||||
let target_path = output_root.join(clean_relative);
|
||||
|
||||
if let Some(parent) = target_path.parent() {
|
||||
tokio::fs::create_dir_all(parent)
|
||||
.await
|
||||
.context("create parent directory")?;
|
||||
}
|
||||
|
||||
// Check if file already exists with identical size
|
||||
if let Ok(metadata) = tokio::fs::metadata(&target_path).await {
|
||||
if metadata.len() == item.size_bytes {
|
||||
total_bar.inc(item.size_bytes);
|
||||
return Ok(DownloadOutcome::Skipped);
|
||||
}
|
||||
}
|
||||
|
||||
let temp_path = target_path.with_extension("drobotic-tmp");
|
||||
|
||||
// Retry up to 3 times on network failure
|
||||
let mut attempts = 0;
|
||||
let max_attempts = 3;
|
||||
|
||||
loop {
|
||||
attempts += 1;
|
||||
match stream_download(client, &item.download_url, &temp_path, item.size_bytes, total_bar).await {
|
||||
Ok(_) => {
|
||||
tokio::fs::rename(&temp_path, &target_path)
|
||||
.await
|
||||
.context("rename temp file to target")?;
|
||||
return Ok(DownloadOutcome::Downloaded);
|
||||
}
|
||||
Err(err) if attempts < max_attempts => {
|
||||
eprintln!("\n⚠️ Attempt {}/{} failed for {}: {}. Retrying...", attempts, max_attempts, item.relative_path, err);
|
||||
tokio::time::sleep(Duration::from_millis(500 * attempts as u64)).await;
|
||||
}
|
||||
Err(err) => {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn stream_download(
|
||||
client: &reqwest::Client,
|
||||
download_url: &str,
|
||||
temp_path: &Path,
|
||||
expected_size: u64,
|
||||
total_bar: &ProgressBar,
|
||||
) -> Result<()> {
|
||||
let response = client
|
||||
.get(download_url)
|
||||
.send()
|
||||
.await
|
||||
.context("send GET request to BOS")?;
|
||||
|
||||
if !response.status().is_success() {
|
||||
return Err(anyhow!("BOS returned HTTP error status: {}", response.status()));
|
||||
}
|
||||
|
||||
let mut file = tokio::fs::File::create(temp_path)
|
||||
.await
|
||||
.context("create temp file")?;
|
||||
|
||||
let mut stream = response.bytes_stream();
|
||||
let mut downloaded_bytes = 0u64;
|
||||
|
||||
while let Some(chunk_result) = stream.next().await {
|
||||
let chunk = chunk_result.context("read chunk from BOS stream")?;
|
||||
file.write_all(&chunk).await.context("write chunk to disk")?;
|
||||
let len = chunk.len() as u64;
|
||||
downloaded_bytes += len;
|
||||
total_bar.inc(len);
|
||||
}
|
||||
|
||||
file.flush().await.context("flush temp file")?;
|
||||
|
||||
if expected_size > 0 && downloaded_bytes != expected_size {
|
||||
return Err(anyhow!(
|
||||
"Incomplete download: expected {} bytes but got {}",
|
||||
expected_size,
|
||||
downloaded_bytes
|
||||
));
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn resolve_manifest(client: &reqwest::Client, args: &Cli) -> Result<ManifestResponse> {
|
||||
if let Some(file_path) = &args.manifest_file {
|
||||
println!("📖 Reading manifest from local file: {}", file_path.display());
|
||||
let content = tokio::fs::read_to_string(file_path)
|
||||
.await
|
||||
.context("read manifest file")?;
|
||||
let manifest: ManifestResponse = serde_json::from_str(&content)
|
||||
.context("parse manifest JSON")?;
|
||||
return Ok(manifest);
|
||||
}
|
||||
|
||||
if let Some(manifest_url) = &args.manifest_url {
|
||||
println!("🌐 Fetching manifest from URL: {}", manifest_url);
|
||||
let mut request = client.get(manifest_url);
|
||||
if let Some(cookie) = &args.cookie {
|
||||
request = request.header("Cookie", cookie);
|
||||
}
|
||||
let response = request.send().await.context("fetch manifest URL")?;
|
||||
if !response.status().is_success() {
|
||||
let status = response.status();
|
||||
let text = response.text().await.unwrap_or_default();
|
||||
return Err(anyhow!("Manifest request failed with status {}: {}", status, text));
|
||||
}
|
||||
let manifest: ManifestResponse = response.json().await.context("decode manifest JSON")?;
|
||||
return Ok(manifest);
|
||||
}
|
||||
|
||||
let api_base = args
|
||||
.api
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow!("Please specify --api or --manifest-url or --manifest-file"))?;
|
||||
let api_base = api_base.trim_end_matches('/');
|
||||
|
||||
let prefix = args
|
||||
.prefix
|
||||
.as_deref()
|
||||
.ok_or_else(|| anyhow!("Please specify --prefix (e.g. ubbdata/2026_09_09-09_21_24)"))?;
|
||||
|
||||
let mut session_cookie = args.cookie.clone();
|
||||
|
||||
// If email and password provided, perform login
|
||||
if let (Some(email), Some(password)) = (&args.email, &args.password) {
|
||||
println!("🔑 Authenticating as {} on {}...", email, api_base);
|
||||
let login_payload = serde_json::json!({
|
||||
"email": email,
|
||||
"password": password,
|
||||
});
|
||||
|
||||
let login_resp = client
|
||||
.post(format!("{}/api/auth/login", api_base))
|
||||
.json(&login_payload)
|
||||
.send()
|
||||
.await
|
||||
.context("send login request")?;
|
||||
|
||||
if !login_resp.status().is_success() {
|
||||
let text = login_resp.text().await.unwrap_or_default();
|
||||
return Err(anyhow!("Login failed: {}", text));
|
||||
}
|
||||
|
||||
// Extract cookie if returned
|
||||
if let Some(cookie_header) = login_resp.headers().get("set-cookie") {
|
||||
if let Ok(cookie_str) = cookie_header.to_str() {
|
||||
let first_cookie = cookie_str.split(';').next().unwrap_or(cookie_str);
|
||||
session_cookie = Some(first_cookie.to_string());
|
||||
}
|
||||
}
|
||||
println!("🔓 Authentication successful!");
|
||||
}
|
||||
|
||||
// Determine connection ID
|
||||
let connection_id = match &args.connection {
|
||||
Some(id) => id.clone(),
|
||||
None => {
|
||||
println!("🔍 Querying available storage connections...");
|
||||
let mut req = client.get(format!("{}/api/object-storage", api_base));
|
||||
if let Some(cookie) = &session_cookie {
|
||||
req = req.header("Cookie", cookie);
|
||||
}
|
||||
let resp = req.send().await.context("query storage connections")?;
|
||||
if !resp.status().is_success() {
|
||||
return Err(anyhow!("Failed to query storage connections: HTTP {}", resp.status()));
|
||||
}
|
||||
let connections: Vec<StorageConnectionSummary> = resp.json().await.context("parse connections")?;
|
||||
if connections.is_empty() {
|
||||
return Err(anyhow!("No storage connections found in this organization"));
|
||||
}
|
||||
let chosen = &connections[0];
|
||||
println!("💡 Using default storage connection: {} ({})", chosen.name, chosen.id);
|
||||
chosen.id.clone()
|
||||
}
|
||||
};
|
||||
|
||||
let mut manifest_url = format!(
|
||||
"{}/api/object-storage/directory/manifest?connectionId={}&prefix={}",
|
||||
api_base,
|
||||
urlencoding::encode(&connection_id),
|
||||
urlencoding::encode(prefix)
|
||||
);
|
||||
|
||||
if let Some(bucket) = &args.bucket {
|
||||
manifest_url.push_str(&format!("&bucket={}", urlencoding::encode(bucket)));
|
||||
}
|
||||
|
||||
println!("📋 Requesting directory manifest for '{}'...", prefix);
|
||||
let mut req = client.get(&manifest_url);
|
||||
if let Some(cookie) = &session_cookie {
|
||||
req = req.header("Cookie", cookie);
|
||||
}
|
||||
let resp = req.send().await.context("request directory manifest")?;
|
||||
if !resp.status().is_success() {
|
||||
let status = resp.status();
|
||||
let text = resp.text().await.unwrap_or_default();
|
||||
return Err(anyhow!("Directory manifest request failed ({}): {}", status, text));
|
||||
}
|
||||
|
||||
let manifest: ManifestResponse = resp.json().await.context("parse manifest response")?;
|
||||
Ok(manifest)
|
||||
}
|
||||
|
||||
fn format_bytes(bytes: u64) -> String {
|
||||
const KB: u64 = 1024;
|
||||
const MB: u64 = KB * 1024;
|
||||
const GB: u64 = MB * 1024;
|
||||
|
||||
if bytes >= GB {
|
||||
format!("{:.2} GB", bytes as f64 / GB as f64)
|
||||
} else if bytes >= MB {
|
||||
format!("{:.2} MB", bytes as f64 / MB as f64)
|
||||
} else if bytes >= KB {
|
||||
format!("{:.2} KB", bytes as f64 / KB as f64)
|
||||
} else {
|
||||
format!("{} B", bytes)
|
||||
}
|
||||
}
|
||||
18
tools/drosync/Cargo.lock
generated
18
tools/drosync/Cargo.lock
generated
@ -1194,7 +1194,7 @@ checksum = "d8b14ccef22fc6f5a8f4d7d768562a182c04ce9a3b3157b91390b52ddfdf1a76"
|
||||
|
||||
[[package]]
|
||||
name = "drosync-cli"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"clap",
|
||||
@ -1210,7 +1210,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-core"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"drosync-protocol",
|
||||
@ -1226,7 +1226,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-daemon"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"drosync-core",
|
||||
@ -1247,7 +1247,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-gui"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"drosync-daemon",
|
||||
"drosync-platform",
|
||||
@ -1259,7 +1259,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-platform"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"directories",
|
||||
"keyring",
|
||||
@ -1271,7 +1271,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-protocol"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"serde",
|
||||
@ -1280,7 +1280,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-state"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"drosync-core",
|
||||
"drosync-protocol",
|
||||
@ -1293,7 +1293,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-transfer"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"drosync-protocol",
|
||||
@ -1309,7 +1309,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "drosync-updater"
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"drosync-protocol",
|
||||
|
||||
@ -13,7 +13,7 @@ members = [
|
||||
]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.0.0"
|
||||
version = "1.0.19"
|
||||
edition = "2024"
|
||||
license = "Apache-2.0"
|
||||
rust-version = "1.89"
|
||||
|
||||
@ -79,8 +79,10 @@ enum MappingSubcommand {
|
||||
List,
|
||||
Add {
|
||||
local_root: PathBuf,
|
||||
#[arg(long)]
|
||||
space: String,
|
||||
#[arg(long, required_unless_present = "bucket", conflicts_with = "bucket")]
|
||||
space: Option<String>,
|
||||
#[arg(long, required_unless_present = "space", conflicts_with = "space")]
|
||||
bucket: Option<String>,
|
||||
#[arg(long)]
|
||||
remote_prefix: Option<String>,
|
||||
#[arg(long)]
|
||||
@ -291,6 +293,7 @@ async fn execute(app: &App) -> Result<Value> {
|
||||
MappingSubcommand::Add {
|
||||
local_root,
|
||||
space,
|
||||
bucket,
|
||||
remote_prefix,
|
||||
include_hidden,
|
||||
require_done,
|
||||
@ -298,7 +301,13 @@ async fn execute(app: &App) -> Result<Value> {
|
||||
let canonical = local_root
|
||||
.canonicalize()
|
||||
.with_context(|| format!("invalid mapping root {}", local_root.display()))?;
|
||||
let mut mapping = Mapping::new(canonical, space.clone());
|
||||
let mut mapping = match (space.as_ref(), bucket.as_ref()) {
|
||||
(Some(space), None) => Mapping::new(canonical, space.trim().to_owned()),
|
||||
(None, Some(bucket)) => {
|
||||
Mapping::for_bucket(canonical, bucket.trim().to_owned())
|
||||
}
|
||||
_ => return Err(anyhow!("exactly one of --space or --bucket is required")),
|
||||
};
|
||||
mapping.remote_prefix = remote_prefix.clone().unwrap_or_default();
|
||||
mapping.include_hidden = *include_hidden;
|
||||
mapping.require_done_marker = *require_done;
|
||||
@ -354,6 +363,7 @@ async fn execute(app: &App) -> Result<Value> {
|
||||
"protocolVersion": PROTOCOL_VERSION,
|
||||
"platform": drosync_platform::platform_name(),
|
||||
"architecture": drosync_platform::architecture(),
|
||||
"releasePlatform": drosync_platform::release_platform_key(),
|
||||
})),
|
||||
}
|
||||
}
|
||||
|
||||
@ -40,6 +40,8 @@ pub struct Mapping {
|
||||
pub local_root: PathBuf,
|
||||
pub sync_space_id: String,
|
||||
#[serde(default)]
|
||||
pub bucket: String,
|
||||
#[serde(default)]
|
||||
pub remote_prefix: String,
|
||||
#[serde(default)]
|
||||
pub include_hidden: bool,
|
||||
@ -54,11 +56,19 @@ impl Mapping {
|
||||
id: Uuid::new_v4().to_string(),
|
||||
local_root,
|
||||
sync_space_id,
|
||||
bucket: String::new(),
|
||||
remote_prefix: String::new(),
|
||||
include_hidden: false,
|
||||
require_done_marker: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn for_bucket(local_root: PathBuf, bucket: String) -> Self {
|
||||
let mut mapping = Self::new(local_root, String::new());
|
||||
mapping.bucket = bucket;
|
||||
mapping
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@ -369,4 +379,12 @@ mod tests {
|
||||
"nested/good.bin"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_mapping_defers_space_resolution_to_the_authenticated_daemon() {
|
||||
let mapping =
|
||||
Mapping::for_bucket(Path::new("/mapping").to_path_buf(), "bucket-a".to_owned());
|
||||
assert_eq!(mapping.bucket, "bucket-a");
|
||||
assert!(mapping.sync_space_id.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
@ -126,6 +126,43 @@ fn apply_remote_prefix(mapping: &Mapping, manifest: &mut ManifestItem) {
|
||||
}
|
||||
}
|
||||
|
||||
fn resolve_bucket_space(capabilities: &Value, bucket: &str) -> Result<(String, String)> {
|
||||
let requested = bucket.trim();
|
||||
if requested.is_empty() {
|
||||
return Err(anyhow!("bucket must not be empty"));
|
||||
}
|
||||
let spaces = capabilities
|
||||
.get("spaces")
|
||||
.and_then(Value::as_array)
|
||||
.ok_or_else(|| anyhow!("control plane did not return scoped sync spaces"))?;
|
||||
let matches = spaces
|
||||
.iter()
|
||||
.filter(|space| space.get("bucket").and_then(Value::as_str) == Some(requested))
|
||||
.filter_map(|space| {
|
||||
let id = space.get("id")?.as_str()?.trim();
|
||||
(!id.is_empty()).then(|| {
|
||||
(
|
||||
id.to_owned(),
|
||||
space
|
||||
.get("rootPrefix")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or_default()
|
||||
.to_owned(),
|
||||
)
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
match matches.as_slice() {
|
||||
[item] => Ok(item.clone()),
|
||||
[] => Err(anyhow!(
|
||||
"bucket {requested} is not available to this credential; ask an administrator to create or grant a sync space"
|
||||
)),
|
||||
_ => Err(anyhow!(
|
||||
"bucket {requested} maps to multiple sync spaces; use --space with the intended space id"
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn normalize_journal(mut journal: SyncJournal) -> SyncJournal {
|
||||
if journal.file_count == 0 && !journal.files.is_empty() {
|
||||
journal.file_count = u64::try_from(journal.files.len()).unwrap_or(u64::MAX);
|
||||
@ -813,12 +850,31 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
|
||||
.mappings()
|
||||
.map(|items| serde_json::to_value(items).expect("serializable mappings"))
|
||||
.map_err(Into::into),
|
||||
"mapping.add" => serde_json::from_value::<Mapping>(request.arguments)
|
||||
.map_err(Into::into)
|
||||
.and_then(|mapping| {
|
||||
state.put_mapping(&mapping)?;
|
||||
Ok(serde_json::to_value(mapping).expect("serializable mapping"))
|
||||
}),
|
||||
"mapping.add" => match serde_json::from_value::<Mapping>(request.arguments) {
|
||||
Ok(mut mapping) => {
|
||||
let resolved = if mapping.sync_space_id.trim().is_empty() {
|
||||
match load_transfer_client(state) {
|
||||
Ok(client) => match client.capabilities().await {
|
||||
Ok(capabilities) => {
|
||||
resolve_bucket_space(&capabilities, &mapping.bucket)
|
||||
}
|
||||
Err(error) => Err(error.into()),
|
||||
},
|
||||
Err(error) => Err(error),
|
||||
}
|
||||
} else {
|
||||
Ok((mapping.sync_space_id.clone(), String::new()))
|
||||
};
|
||||
resolved.and_then(|(space_id, space_root_prefix)| {
|
||||
mapping.sync_space_id = space_id;
|
||||
state.put_mapping(&mapping)?;
|
||||
let mut value = serde_json::to_value(mapping).expect("serializable mapping");
|
||||
value["spaceRootPrefix"] = Value::String(space_root_prefix);
|
||||
Ok(value)
|
||||
})
|
||||
}
|
||||
Err(error) => Err(error.into()),
|
||||
},
|
||||
"mapping.remove" => {
|
||||
let id = request
|
||||
.arguments
|
||||
@ -826,8 +882,17 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| anyhow!("id is required"));
|
||||
id.and_then(|value| {
|
||||
// Mark any non-terminal sessions for this mapping as aborted
|
||||
// before removing the mapping itself. This prevents the
|
||||
// scheduler from picking up stale "active" sessions that would
|
||||
// otherwise keep spawning new sync attempts.
|
||||
let aborted_count = state.mark_sessions_aborted_for_mapping(value)?;
|
||||
let removed = state.remove_mapping(value)?;
|
||||
Ok(serde_json::json!({"removed": removed}))
|
||||
// Clear pending scan journal so a restart won't resume work
|
||||
// for a mapping that no longer exists.
|
||||
let pending_key = format!("pending_journal:{value}");
|
||||
let _ = state.set_meta(&pending_key, "");
|
||||
Ok(serde_json::json!({"removed": removed, "abortedSessions": aborted_count}))
|
||||
})
|
||||
}
|
||||
"plan" | "sync" => {
|
||||
@ -948,6 +1013,9 @@ async fn dispatch(state: &mut StateStore, request: IpcRequest) -> IpcResponse {
|
||||
"name": "drosync-daemon",
|
||||
"version": env!("CARGO_PKG_VERSION"),
|
||||
"protocolVersion": drosync_protocol_version(),
|
||||
"platform": drosync_platform::platform_name(),
|
||||
"architecture": drosync_platform::architecture(),
|
||||
"releasePlatform": drosync_platform::release_platform_key(),
|
||||
})),
|
||||
_ => Err(anyhow!("unknown daemon command {}", request.command)),
|
||||
};
|
||||
@ -1006,14 +1074,25 @@ async fn handle_update(state: &mut StateStore, arguments: &Value) -> Result<Valu
|
||||
"offeredVersion": release.version,
|
||||
}));
|
||||
}
|
||||
let encoded_key = option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64")
|
||||
.ok_or_else(|| anyhow!("client was built without DROSYNC_UPDATE_PUBLIC_KEY_BASE64"))?;
|
||||
let expected_platform_key = drosync_platform::release_platform_key()
|
||||
.ok_or_else(|| anyhow!("this client target has no supported signed-update platform key"))?;
|
||||
let key_environment = drosync_platform::update_public_key_environment(expected_platform_key)
|
||||
.unwrap_or("DROSYNC_UPDATE_PUBLIC_KEY_BASE64");
|
||||
let encoded_key = drosync_platform::embedded_update_public_key().ok_or_else(|| {
|
||||
anyhow!("client was built without {key_environment} or DROSYNC_UPDATE_PUBLIC_KEY_BASE64")
|
||||
})?;
|
||||
let public_key = drosync_updater::decode_public_key(encoded_key)?;
|
||||
drosync_updater::verify_release(&release, &public_key)?;
|
||||
let platform_key = response
|
||||
.get("platformKey")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| anyhow!("update response omitted platformKey"))?;
|
||||
if platform_key != expected_platform_key {
|
||||
return Err(anyhow!(
|
||||
"update response platformKey {platform_key} does not match this client target {expected_platform_key}"
|
||||
));
|
||||
}
|
||||
drosync_updater::validate_release_platform(&release, platform_key)?;
|
||||
let url = release
|
||||
.artifacts
|
||||
.get(platform_key)
|
||||
@ -1073,7 +1152,9 @@ async fn handle_update(state: &mut StateStore, arguments: &Value) -> Result<Valu
|
||||
async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphore>) -> Result<()> {
|
||||
let (event_sender, mut event_receiver) = tokio::sync::mpsc::unbounded_channel();
|
||||
let mut watcher = notify::recommended_watcher(move |event: notify::Result<notify::Event>| {
|
||||
if let Ok(event) = event {
|
||||
if let Ok(event) = event
|
||||
&& event_triggers_sync(event.kind)
|
||||
{
|
||||
let _ = event_sender.send(event.paths);
|
||||
}
|
||||
})?;
|
||||
@ -1093,8 +1174,19 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
|
||||
let current = state.mappings()?.into_iter().map(|mapping| (mapping.id.clone(), mapping)).collect::<BTreeMap<_, _>>();
|
||||
for (id, mapping) in ¤t {
|
||||
if !mappings.contains_key(id) {
|
||||
watcher.watch(&mapping.local_root, RecursiveMode::Recursive)?;
|
||||
dirty.insert(id.clone(), std::time::Instant::now());
|
||||
match watcher.watch(&mapping.local_root, RecursiveMode::Recursive) {
|
||||
Ok(()) => {
|
||||
dirty.insert(id.clone(), std::time::Instant::now());
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::error!(
|
||||
mapping_id = %id,
|
||||
local_root = %mapping.local_root.display(),
|
||||
error = %error,
|
||||
"failed to watch mapping; scheduler will retry on refresh"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for (id, mapping) in &mappings {
|
||||
@ -1120,12 +1212,28 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
|
||||
.map(|(id, _)| id.clone())
|
||||
.collect::<Vec<_>>();
|
||||
for id in ready {
|
||||
let Some(mapping) = mappings.get(&id).cloned() else {
|
||||
if !mappings.contains_key(&id) {
|
||||
dirty.remove(&id);
|
||||
continue;
|
||||
}
|
||||
let _permit = transfer_gate.acquire().await?;
|
||||
let mut state = StateStore::open(&state_db)?;
|
||||
// Re-check the mapping against the database before syncing.
|
||||
// The in-memory cache is only refreshed every 60 s, so a
|
||||
// `mapping.remove` that arrived in the meantime would leave
|
||||
// a stale entry that keeps spawning sessions.
|
||||
let Some(mapping) = state.mapping(&id)? else {
|
||||
// Mapping was removed — tear down the watcher and
|
||||
// purge it from the in-memory caches immediately
|
||||
// instead of waiting for the next refresh tick.
|
||||
if let Some(old) = mappings.remove(&id) {
|
||||
let _ = watcher.unwatch(&old.local_root);
|
||||
}
|
||||
dirty.remove(&id);
|
||||
continue;
|
||||
};
|
||||
let _permit = transfer_gate.acquire().await?;
|
||||
let mut state = StateStore::open(&state_db)?;
|
||||
// Keep in-memory cache in sync with the database.
|
||||
mappings.insert(id.clone(), mapping.clone());
|
||||
match sync_mapping(&mut state, &mapping, false).await {
|
||||
Ok(value) if value.get("paused").and_then(Value::as_bool) == Some(true) => {
|
||||
dirty.insert(id, std::time::Instant::now());
|
||||
@ -1151,6 +1259,24 @@ async fn run_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphor
|
||||
}
|
||||
}
|
||||
|
||||
fn event_triggers_sync(kind: notify::EventKind) -> bool {
|
||||
matches!(
|
||||
kind,
|
||||
notify::EventKind::Create(_) | notify::EventKind::Modify(_) | notify::EventKind::Remove(_)
|
||||
)
|
||||
}
|
||||
|
||||
fn spawn_scheduler(state_db: PathBuf, transfer_gate: std::sync::Arc<Semaphore>) {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
if let Err(error) = run_scheduler(state_db.clone(), transfer_gate.clone()).await {
|
||||
tracing::error!(error = %error, "drosync scheduler stopped; restarting");
|
||||
tokio::time::sleep(Duration::from_secs(5)).await;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
const fn drosync_protocol_version() -> &'static str {
|
||||
"v1"
|
||||
}
|
||||
@ -1172,7 +1298,7 @@ pub async fn run(paths: &Paths) -> Result<()> {
|
||||
let state = StateStore::open(&paths.state_db)?;
|
||||
state.set_meta("data_dir", &paths.data_dir.to_string_lossy())?;
|
||||
let transfer_gate = std::sync::Arc::new(Semaphore::new(1));
|
||||
tokio::spawn(run_scheduler(paths.state_db.clone(), transfer_gate.clone()));
|
||||
spawn_scheduler(paths.state_db.clone(), transfer_gate.clone());
|
||||
loop {
|
||||
let (stream, _) = listener.accept().await?;
|
||||
let state_db = paths.state_db.clone();
|
||||
@ -1230,7 +1356,7 @@ async fn run_windows_pipe(paths: &Paths, pipe_name: &str) -> Result<()> {
|
||||
let state = StateStore::open(&paths.state_db)?;
|
||||
state.set_meta("data_dir", &paths.data_dir.to_string_lossy())?;
|
||||
let transfer_gate = std::sync::Arc::new(Semaphore::new(1));
|
||||
tokio::spawn(run_scheduler(paths.state_db.clone(), transfer_gate.clone()));
|
||||
spawn_scheduler(paths.state_db.clone(), transfer_gate.clone());
|
||||
let mut first_instance = true;
|
||||
loop {
|
||||
let mut options = ServerOptions::new();
|
||||
@ -1341,7 +1467,12 @@ pub fn default_log_path(data_dir: &Path) -> PathBuf {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{MULTIPART_MAX_PART, multipart_part_size};
|
||||
use super::{
|
||||
MULTIPART_MAX_PART, event_triggers_sync, multipart_part_size, resolve_bucket_space,
|
||||
};
|
||||
use notify::EventKind;
|
||||
use notify::event::{AccessKind, CreateKind, ModifyKind, RemoveKind};
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn one_tib_object_fits_bos_multipart_limits() {
|
||||
@ -1363,4 +1494,36 @@ mod tests {
|
||||
assert!(multipart_part_size(size) <= MULTIPART_MAX_PART);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_resolves_only_when_scoped_space_is_unique() {
|
||||
let capabilities = json!({"spaces": [
|
||||
{"id": "space-a", "bucket": "bucket-a", "rootPrefix": "tenant/ubt"},
|
||||
{"id": "space-b", "bucket": "bucket-b", "rootPrefix": ""}
|
||||
]});
|
||||
assert_eq!(
|
||||
resolve_bucket_space(&capabilities, " bucket-a ").unwrap(),
|
||||
("space-a".to_owned(), "tenant/ubt".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_resolution_fails_closed_for_missing_or_ambiguous_space() {
|
||||
let capabilities = json!({"spaces": [
|
||||
{"id": "space-a", "bucket": "bucket-a"},
|
||||
{"id": "space-b", "bucket": "bucket-a"}
|
||||
]});
|
||||
assert!(resolve_bucket_space(&capabilities, "bucket-a").is_err());
|
||||
assert!(resolve_bucket_space(&capabilities, "bucket-missing").is_err());
|
||||
assert!(resolve_bucket_space(&capabilities, " ").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn watcher_ignores_reads_but_tracks_content_changes() {
|
||||
assert!(!event_triggers_sync(EventKind::Access(AccessKind::Any)));
|
||||
assert!(!event_triggers_sync(EventKind::Other));
|
||||
assert!(event_triggers_sync(EventKind::Create(CreateKind::Any)));
|
||||
assert!(event_triggers_sync(EventKind::Modify(ModifyKind::Any)));
|
||||
assert!(event_triggers_sync(EventKind::Remove(RemoveKind::Any)));
|
||||
}
|
||||
}
|
||||
|
||||
@ -7,11 +7,16 @@ rust-version.workspace = true
|
||||
|
||||
[dependencies]
|
||||
directories.workspace = true
|
||||
keyring.workspace = true
|
||||
serde.workspace = true
|
||||
thiserror.workspace = true
|
||||
uuid.workspace = true
|
||||
|
||||
# RDK S100 is a headless Linux ARM64 target. Its systemd service receives a
|
||||
# token through a restricted environment file or `DROSYNC_SECRET_FILE`, rather
|
||||
# than pulling in a desktop D-Bus keyring implementation.
|
||||
[target.'cfg(not(all(target_os = "linux", target_arch = "aarch64")))'.dependencies]
|
||||
keyring.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile.workspace = true
|
||||
|
||||
|
||||
@ -3,7 +3,7 @@
|
||||
use directories::ProjectDirs;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::fs;
|
||||
use std::io;
|
||||
use std::io::{self, Write};
|
||||
use std::path::{Path, PathBuf};
|
||||
use thiserror::Error;
|
||||
use uuid::Uuid;
|
||||
@ -12,12 +12,19 @@ use uuid::Uuid;
|
||||
pub enum PlatformError {
|
||||
#[error("operating-system project directories are unavailable")]
|
||||
DirectoriesUnavailable,
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
#[error("credential storage failed: {0}")]
|
||||
Credential(#[from] keyring::Error),
|
||||
#[error("DROSYNC_SECRET_FILE is required for headless Linux ARM64 credentials")]
|
||||
HeadlessSecretFileRequired,
|
||||
#[error("I/O error: {0}")]
|
||||
Io(#[from] io::Error),
|
||||
#[error("secret file permissions must be 0600")]
|
||||
InsecureSecretPermissions,
|
||||
#[error("secret file must be a regular file, not a symlink or special file")]
|
||||
InvalidSecretFile,
|
||||
#[error("secret-file parent directory must not be group- or world-writable")]
|
||||
InsecureSecretDirectoryPermissions,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@ -42,6 +49,97 @@ pub fn architecture() -> &'static str {
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the release-manifest key for an operating-system/architecture pair.
|
||||
///
|
||||
/// The control plane uses this value as the key in a signed client release
|
||||
/// manifest. Keep it deliberately narrower than `std::env::consts`: an
|
||||
/// unsupported target must not accidentally select a package built for a
|
||||
/// different ABI.
|
||||
#[must_use]
|
||||
pub fn release_platform_key_for(
|
||||
operating_system: &str,
|
||||
target_architecture: &str,
|
||||
) -> Option<&'static str> {
|
||||
match (operating_system, target_architecture) {
|
||||
("linux", "x86_64") => Some("linux-x86_64"),
|
||||
("linux", "aarch64" | "arm64") => Some("linux-arm64"),
|
||||
("macos", "aarch64" | "arm64") => Some("macos-arm64"),
|
||||
("windows", "x86_64") => Some("windows-x86_64"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Return the release-manifest key compiled into this client.
|
||||
///
|
||||
/// This is separate from the two device-registration fields (`platform` and
|
||||
/// `architecture`), which remain wire-compatible with existing control-plane
|
||||
/// versions and are combined there into the same key.
|
||||
#[must_use]
|
||||
pub fn release_platform_key() -> Option<&'static str> {
|
||||
release_platform_key_for(std::env::consts::OS, std::env::consts::ARCH)
|
||||
}
|
||||
|
||||
/// Return the environment variable that may pin a public update key for a
|
||||
/// release platform. A generic `DROSYNC_UPDATE_PUBLIC_KEY_BASE64` remains a
|
||||
/// backwards-compatible fallback.
|
||||
#[must_use]
|
||||
pub fn update_public_key_environment(platform_key: &str) -> Option<&'static str> {
|
||||
match platform_key {
|
||||
"linux-arm64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64"),
|
||||
"linux-x86_64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64"),
|
||||
"macos-arm64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64"),
|
||||
"windows-x86_64" => Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Choose the target-specific update key when one was embedded, otherwise use
|
||||
/// the legacy generic key. This pure helper keeps the trust-key mapping
|
||||
/// testable without exposing compile-time configuration as a runtime setting.
|
||||
#[must_use]
|
||||
pub fn select_update_public_key(
|
||||
target_specific: Option<&'static str>,
|
||||
legacy: Option<&'static str>,
|
||||
) -> Option<&'static str> {
|
||||
match target_specific {
|
||||
Some(value) if !value.is_empty() => Some(value),
|
||||
_ => legacy,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
|
||||
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64");
|
||||
#[cfg(all(target_os = "linux", target_arch = "x86_64"))]
|
||||
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
|
||||
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64");
|
||||
#[cfg(all(target_os = "macos", target_arch = "aarch64"))]
|
||||
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
|
||||
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64");
|
||||
#[cfg(all(target_os = "windows", target_arch = "x86_64"))]
|
||||
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> =
|
||||
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64");
|
||||
#[cfg(not(any(
|
||||
all(target_os = "linux", target_arch = "aarch64"),
|
||||
all(target_os = "linux", target_arch = "x86_64"),
|
||||
all(target_os = "macos", target_arch = "aarch64"),
|
||||
all(target_os = "windows", target_arch = "x86_64"),
|
||||
)))]
|
||||
const TARGET_UPDATE_PUBLIC_KEY: Option<&str> = None;
|
||||
|
||||
/// Return the Ed25519 public key embedded for this exact release target.
|
||||
///
|
||||
/// Build scripts populate the target-specific environment variable and also
|
||||
/// preserve the legacy generic variable so previously configured CI continues
|
||||
/// to produce compatible clients.
|
||||
#[must_use]
|
||||
pub fn embedded_update_public_key() -> Option<&'static str> {
|
||||
select_update_public_key(
|
||||
TARGET_UPDATE_PUBLIC_KEY,
|
||||
option_env!("DROSYNC_UPDATE_PUBLIC_KEY_BASE64"),
|
||||
)
|
||||
}
|
||||
|
||||
pub fn paths() -> Result<Paths, PlatformError> {
|
||||
let dirs = ProjectDirs::from("com", "drobotics", "drosync")
|
||||
.ok_or(PlatformError::DirectoriesUnavailable)?;
|
||||
@ -57,15 +155,58 @@ pub fn paths() -> Result<Paths, PlatformError> {
|
||||
})
|
||||
}
|
||||
|
||||
/// Parse an explicitly configured secret-file path without reading process
|
||||
/// state. Keeping this small helper pure lets all target test suites exercise
|
||||
/// the headless credential contract.
|
||||
pub fn headless_secret_file_path(value: Option<&str>) -> Result<PathBuf, PlatformError> {
|
||||
value
|
||||
.map(str::trim)
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(PathBuf::from)
|
||||
.ok_or(PlatformError::HeadlessSecretFileRequired)
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
fn configured_headless_secret_file() -> Result<PathBuf, PlatformError> {
|
||||
let value = std::env::var("DROSYNC_SECRET_FILE").ok();
|
||||
headless_secret_file_path(value.as_deref())
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
fn configured_runtime_secret_file() -> Option<PathBuf> {
|
||||
std::env::var("DROSYNC_SECRET_FILE").ok().and_then(|value| {
|
||||
let trimmed = value.trim();
|
||||
(!trimmed.is_empty()).then(|| PathBuf::from(trimmed))
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
pub fn save_credential(profile: &str, token: &str) -> Result<(), PlatformError> {
|
||||
if let Some(path) = configured_runtime_secret_file() {
|
||||
return write_headless_secret(&path, token);
|
||||
}
|
||||
keyring::Entry::new("drosync", profile)?.set_password(token)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
pub fn save_credential(_profile: &str, token: &str) -> Result<(), PlatformError> {
|
||||
write_headless_secret(&configured_headless_secret_file()?, token)
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
pub fn load_credential(profile: &str) -> Result<String, PlatformError> {
|
||||
if let Some(path) = configured_runtime_secret_file() {
|
||||
return read_headless_secret(&path);
|
||||
}
|
||||
Ok(keyring::Entry::new("drosync", profile)?.get_password()?)
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
pub fn load_credential(_profile: &str) -> Result<String, PlatformError> {
|
||||
read_headless_secret(&configured_headless_secret_file()?)
|
||||
}
|
||||
|
||||
pub fn load_runtime_credential(profile: &str) -> Result<String, PlatformError> {
|
||||
if let Ok(value) = std::env::var("DROSYNC_TOKEN")
|
||||
&& !value.trim().is_empty()
|
||||
@ -80,11 +221,20 @@ pub fn load_runtime_credential(profile: &str) -> Result<String, PlatformError> {
|
||||
load_credential(profile)
|
||||
}
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
pub fn delete_credential(profile: &str) -> Result<(), PlatformError> {
|
||||
if let Some(path) = configured_runtime_secret_file() {
|
||||
return delete_headless_secret(&path);
|
||||
}
|
||||
keyring::Entry::new("drosync", profile)?.delete_credential()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
pub fn delete_credential(_profile: &str) -> Result<(), PlatformError> {
|
||||
delete_headless_secret(&configured_headless_secret_file()?)
|
||||
}
|
||||
|
||||
pub fn load_or_create_device_id(profile: &str) -> Result<String, PlatformError> {
|
||||
let paths = paths()?;
|
||||
let safe_profile = profile
|
||||
@ -115,14 +265,98 @@ pub fn load_or_create_device_id(profile: &str) -> Result<String, PlatformError>
|
||||
pub fn read_headless_secret(path: &Path) -> Result<String, PlatformError> {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
if fs::metadata(path)?.permissions().mode() & 0o077 != 0 {
|
||||
return Err(PlatformError::InsecureSecretPermissions);
|
||||
}
|
||||
ensure_secure_secret_file(path)?;
|
||||
}
|
||||
Ok(fs::read_to_string(path)?.trim().to_owned())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn ensure_secure_secret_file(path: &Path) -> Result<(), PlatformError> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let metadata = fs::symlink_metadata(path)?;
|
||||
if !metadata.file_type().is_file() {
|
||||
return Err(PlatformError::InvalidSecretFile);
|
||||
}
|
||||
if metadata.permissions().mode() & 0o777 != 0o600 {
|
||||
return Err(PlatformError::InsecureSecretPermissions);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn ensure_secure_secret_parent(path: &Path) -> Result<&Path, PlatformError> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let parent = path.parent().ok_or_else(|| {
|
||||
io::Error::new(
|
||||
io::ErrorKind::InvalidInput,
|
||||
"secret file must have a parent directory",
|
||||
)
|
||||
})?;
|
||||
let metadata = fs::metadata(parent)?;
|
||||
if !metadata.is_dir() {
|
||||
return Err(PlatformError::Io(io::Error::new(
|
||||
io::ErrorKind::NotADirectory,
|
||||
"secret-file parent is not a directory",
|
||||
)));
|
||||
}
|
||||
if metadata.permissions().mode() & 0o022 != 0 {
|
||||
return Err(PlatformError::InsecureSecretDirectoryPermissions);
|
||||
}
|
||||
Ok(parent)
|
||||
}
|
||||
|
||||
/// Write a headless credential by atomically replacing a 0600 file in an
|
||||
/// already-secure directory. The temporary file lives beside the destination,
|
||||
/// so the final rename is atomic on the same filesystem.
|
||||
#[cfg(unix)]
|
||||
pub fn write_headless_secret(path: &Path, token: &str) -> Result<(), PlatformError> {
|
||||
use std::fs::OpenOptions;
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
|
||||
let parent = ensure_secure_secret_parent(path)?;
|
||||
match fs::symlink_metadata(path) {
|
||||
Ok(metadata) => {
|
||||
if !metadata.file_type().is_file() {
|
||||
return Err(PlatformError::InvalidSecretFile);
|
||||
}
|
||||
}
|
||||
Err(error) if error.kind() == io::ErrorKind::NotFound => {}
|
||||
Err(error) => return Err(error.into()),
|
||||
}
|
||||
let temporary = parent.join(format!(".drosync-credential-{}.tmp", Uuid::new_v4()));
|
||||
let write_result = (|| -> Result<(), PlatformError> {
|
||||
let mut file = OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.mode(0o600)
|
||||
.open(&temporary)?;
|
||||
file.write_all(token.trim().as_bytes())?;
|
||||
file.write_all(b"\n")?;
|
||||
file.sync_all()?;
|
||||
drop(file);
|
||||
fs::rename(&temporary, path)?;
|
||||
secure_runtime_file(path)?;
|
||||
fs::File::open(parent)?.sync_all()?;
|
||||
Ok(())
|
||||
})();
|
||||
if write_result.is_err() {
|
||||
let _ = fs::remove_file(&temporary);
|
||||
}
|
||||
write_result
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
pub fn delete_headless_secret(path: &Path) -> Result<(), PlatformError> {
|
||||
ensure_secure_secret_file(path)?;
|
||||
fs::remove_file(path)?;
|
||||
if let Some(parent) = path.parent() {
|
||||
fs::File::open(parent)?.sync_all()?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
pub fn secure_runtime_file(path: &Path) -> Result<(), PlatformError> {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
@ -139,12 +373,18 @@ pub fn secure_runtime_file(_path: &Path) -> Result<(), PlatformError> {
|
||||
|
||||
#[cfg(all(test, unix))]
|
||||
mod tests {
|
||||
use super::{PlatformError, read_headless_secret, secure_runtime_file};
|
||||
use super::{
|
||||
PlatformError, delete_headless_secret, headless_secret_file_path, read_headless_secret,
|
||||
release_platform_key_for, secure_runtime_file, select_update_public_key,
|
||||
update_public_key_environment, write_headless_secret,
|
||||
};
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
use keyring::credential::CredentialPersistence;
|
||||
use std::fs;
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
use std::os::unix::net::UnixListener;
|
||||
|
||||
#[cfg(not(all(target_os = "linux", target_arch = "aarch64")))]
|
||||
#[test]
|
||||
fn operating_system_credential_store_is_persistent() {
|
||||
assert!(matches!(
|
||||
@ -163,9 +403,91 @@ mod tests {
|
||||
read_headless_secret(&secret),
|
||||
Err(PlatformError::InsecureSecretPermissions)
|
||||
));
|
||||
assert!(matches!(
|
||||
delete_headless_secret(&secret),
|
||||
Err(PlatformError::InsecureSecretPermissions)
|
||||
));
|
||||
|
||||
fs::set_permissions(&secret, fs::Permissions::from_mode(0o600)).unwrap();
|
||||
assert_eq!(read_headless_secret(&secret).unwrap(), "device-token");
|
||||
|
||||
fs::set_permissions(&secret, fs::Permissions::from_mode(0o400)).unwrap();
|
||||
assert!(matches!(
|
||||
read_headless_secret(&secret),
|
||||
Err(PlatformError::InsecureSecretPermissions)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn headless_secret_path_requires_an_explicit_nonempty_value() {
|
||||
assert!(matches!(
|
||||
headless_secret_file_path(None),
|
||||
Err(PlatformError::HeadlessSecretFileRequired)
|
||||
));
|
||||
assert!(matches!(
|
||||
headless_secret_file_path(Some(" ")),
|
||||
Err(PlatformError::HeadlessSecretFileRequired)
|
||||
));
|
||||
assert_eq!(
|
||||
headless_secret_file_path(Some(" /var/lib/drosync/token ")).unwrap(),
|
||||
std::path::PathBuf::from("/var/lib/drosync/token")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn headless_secret_atomic_write_read_and_delete_round_trip() {
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
fs::set_permissions(directory.path(), fs::Permissions::from_mode(0o700)).unwrap();
|
||||
let secret = directory.path().join("token");
|
||||
|
||||
write_headless_secret(&secret, "first-token\n").unwrap();
|
||||
assert_eq!(read_headless_secret(&secret).unwrap(), "first-token");
|
||||
assert_eq!(
|
||||
fs::metadata(&secret).unwrap().permissions().mode() & 0o777,
|
||||
0o600
|
||||
);
|
||||
|
||||
write_headless_secret(&secret, "second-token").unwrap();
|
||||
assert_eq!(read_headless_secret(&secret).unwrap(), "second-token");
|
||||
delete_headless_secret(&secret).unwrap();
|
||||
assert!(!secret.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn headless_secret_rejects_an_unsafe_parent_directory() {
|
||||
let directory = tempfile::tempdir().unwrap();
|
||||
fs::set_permissions(directory.path(), fs::Permissions::from_mode(0o777)).unwrap();
|
||||
let secret = directory.path().join("token");
|
||||
assert!(matches!(
|
||||
write_headless_secret(&secret, "device-token"),
|
||||
Err(PlatformError::InsecureSecretDirectoryPermissions)
|
||||
));
|
||||
}
|
||||
|
||||
#[cfg(all(target_os = "linux", target_arch = "aarch64"))]
|
||||
#[test]
|
||||
fn arm64_credential_store_round_trips_through_configured_secret_file() {
|
||||
use super::{delete_credential, load_credential, save_credential};
|
||||
|
||||
let configured = std::env::var("DROSYNC_SECRET_FILE")
|
||||
.expect("target test runner must provide DROSYNC_SECRET_FILE");
|
||||
let secret = std::path::PathBuf::from(configured);
|
||||
assert!(
|
||||
!secret.exists(),
|
||||
"target test secret file must start absent"
|
||||
);
|
||||
|
||||
save_credential("default", "arm64-device-token").expect("atomically persist token");
|
||||
assert_eq!(
|
||||
load_credential("default").expect("load persisted token"),
|
||||
"arm64-device-token"
|
||||
);
|
||||
assert_eq!(
|
||||
fs::metadata(&secret).unwrap().permissions().mode() & 0o777,
|
||||
0o600
|
||||
);
|
||||
delete_credential("default").expect("securely remove token");
|
||||
assert!(!secret.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@ -179,4 +501,60 @@ mod tests {
|
||||
0o600
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn release_platform_keys_cover_supported_client_targets() {
|
||||
assert_eq!(
|
||||
release_platform_key_for("linux", "x86_64"),
|
||||
Some("linux-x86_64")
|
||||
);
|
||||
assert_eq!(
|
||||
release_platform_key_for("linux", "aarch64"),
|
||||
Some("linux-arm64")
|
||||
);
|
||||
assert_eq!(
|
||||
release_platform_key_for("linux", "arm64"),
|
||||
Some("linux-arm64")
|
||||
);
|
||||
assert_eq!(
|
||||
release_platform_key_for("macos", "aarch64"),
|
||||
Some("macos-arm64")
|
||||
);
|
||||
assert_eq!(
|
||||
release_platform_key_for("windows", "x86_64"),
|
||||
Some("windows-x86_64")
|
||||
);
|
||||
assert_eq!(release_platform_key_for("linux", "i686"), None);
|
||||
assert_eq!(release_platform_key_for("windows", "aarch64"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn update_key_environment_is_platform_specific_with_legacy_fallback() {
|
||||
assert_eq!(
|
||||
update_public_key_environment("linux-arm64"),
|
||||
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64")
|
||||
);
|
||||
assert_eq!(
|
||||
update_public_key_environment("linux-x86_64"),
|
||||
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64")
|
||||
);
|
||||
assert_eq!(
|
||||
update_public_key_environment("macos-arm64"),
|
||||
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64")
|
||||
);
|
||||
assert_eq!(
|
||||
update_public_key_environment("windows-x86_64"),
|
||||
Some("DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64")
|
||||
);
|
||||
assert_eq!(update_public_key_environment("linux-riscv64"), None);
|
||||
assert_eq!(
|
||||
select_update_public_key(Some("target-key"), Some("legacy-key")),
|
||||
Some("target-key")
|
||||
);
|
||||
assert_eq!(
|
||||
select_update_public_key(None, Some("legacy-key")),
|
||||
Some("legacy-key")
|
||||
);
|
||||
assert_eq!(select_update_public_key(Some(""), None), None);
|
||||
}
|
||||
}
|
||||
|
||||
@ -340,6 +340,20 @@ impl StateStore {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Marks all non-terminal sessions for the given mapping as `"aborted"`.
|
||||
/// Returns the number of sessions updated.
|
||||
pub fn mark_sessions_aborted_for_mapping(
|
||||
&self,
|
||||
mapping_id: &str,
|
||||
) -> Result<usize, StateError> {
|
||||
Ok(self.connection.execute(
|
||||
"UPDATE sessions SET status = 'aborted', updated_at = CURRENT_TIMESTAMP
|
||||
WHERE mapping_id = ?1
|
||||
AND status NOT IN ('completed', 'partial', 'conflicted', 'failed', 'aborted', 'expired')",
|
||||
[mapping_id],
|
||||
)?)
|
||||
}
|
||||
|
||||
pub fn put_part(&self, part: &StoredPart) -> Result<(), StateError> {
|
||||
self.connection.execute(
|
||||
"INSERT INTO transfer_parts(item_id, part_number, etag, size_bytes)
|
||||
|
||||
@ -2,6 +2,7 @@
|
||||
|
||||
use base64::Engine;
|
||||
use drosync_protocol::ClientRelease;
|
||||
use drosync_updater::validate_signed_release_matrix;
|
||||
use ed25519_dalek::{Signer, SigningKey};
|
||||
use std::error::Error;
|
||||
use std::fs;
|
||||
@ -23,38 +24,13 @@ fn signing_key() -> Result<SigningKey, Box<dyn Error>> {
|
||||
}
|
||||
}
|
||||
|
||||
fn validate(release: &ClientRelease) -> Result<(), Box<dyn Error>> {
|
||||
if release.kind != "signed" {
|
||||
return Err("automatic update manifests must use kind=signed".into());
|
||||
}
|
||||
let expected_percent = match release.channel.as_str() {
|
||||
"canary" => 5,
|
||||
"pilot" => 20,
|
||||
"stable" => 100,
|
||||
_ => return Err("channel must be canary, pilot or stable".into()),
|
||||
};
|
||||
if !release.paused && release.rollout_percent != expected_percent {
|
||||
return Err("active rollout percentage does not match its channel".into());
|
||||
}
|
||||
for platform in ["windows-x86_64", "linux-x86_64", "macos-arm64"] {
|
||||
if !release.artifacts.contains_key(platform) || !release.sha256.contains_key(platform) {
|
||||
return Err(format!("manifest is missing the {platform} artifact or digest").into());
|
||||
}
|
||||
let digest = &release.sha256[platform];
|
||||
if digest.len() != 64 || !digest.bytes().all(|value| value.is_ascii_hexdigit()) {
|
||||
return Err(format!("invalid SHA-256 for {platform}").into());
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn main() -> Result<(), Box<dyn Error>> {
|
||||
let manifest_path = std::env::args_os()
|
||||
.nth(1)
|
||||
.map(PathBuf::from)
|
||||
.ok_or("usage: drosync-sign-manifest MANIFEST.json")?;
|
||||
let mut release: ClientRelease = serde_json::from_slice(&fs::read(&manifest_path)?)?;
|
||||
validate(&release)?;
|
||||
validate_signed_release_matrix(&release)?;
|
||||
release.signature.clear();
|
||||
let signature = signing_key()?.sign(&serde_json::to_vec(&release)?);
|
||||
release.signature = base64::engine::general_purpose::STANDARD.encode(signature.to_bytes());
|
||||
|
||||
@ -7,6 +7,15 @@ use sha2::{Digest, Sha256};
|
||||
use std::path::Path;
|
||||
use thiserror::Error;
|
||||
|
||||
/// Every newly signed manifest carries this complete supported-client matrix so
|
||||
/// its one Ed25519 signature cannot be replayed as a partial platform release.
|
||||
pub const REQUIRED_RELEASE_PLATFORMS: [&str; 4] = [
|
||||
"windows-x86_64",
|
||||
"linux-x86_64",
|
||||
"linux-arm64",
|
||||
"macos-arm64",
|
||||
];
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum UpdateError {
|
||||
#[error("invalid Ed25519 public key")]
|
||||
@ -27,6 +36,8 @@ pub enum UpdateError {
|
||||
Install,
|
||||
#[error("HTTPS is required for update downloads except on loopback")]
|
||||
InsecureURL,
|
||||
#[error("invalid signed update manifest: {0}")]
|
||||
InvalidManifest(String),
|
||||
}
|
||||
|
||||
pub fn decode_public_key(value: &str) -> Result<[u8; 32], UpdateError> {
|
||||
@ -36,6 +47,124 @@ pub fn decode_public_key(value: &str) -> Result<[u8; 32], UpdateError> {
|
||||
decoded.try_into().map_err(|_| UpdateError::InvalidKey)
|
||||
}
|
||||
|
||||
fn valid_sha256(value: &str) -> bool {
|
||||
value.len() == 64 && value.bytes().all(|character| character.is_ascii_hexdigit())
|
||||
}
|
||||
|
||||
/// Validate the signed-update fields that every client must understand.
|
||||
///
|
||||
/// The updater accepts already-signed historical manifests that predate
|
||||
/// `linux-arm64`, provided they contain a valid artifact for the running
|
||||
/// target. New release publication uses [`validate_signed_release_matrix`] to
|
||||
/// require the complete matrix.
|
||||
pub fn validate_signed_release(release: &ClientRelease) -> Result<(), UpdateError> {
|
||||
if release.kind != "signed" {
|
||||
return Err(UpdateError::InvalidManifest(
|
||||
"automatic updates require kind=signed".to_owned(),
|
||||
));
|
||||
}
|
||||
let expected_percent = match release.channel.as_str() {
|
||||
"canary" => 5,
|
||||
"pilot" => 20,
|
||||
"stable" => 100,
|
||||
_ => {
|
||||
return Err(UpdateError::InvalidManifest(
|
||||
"channel must be canary, pilot or stable".to_owned(),
|
||||
));
|
||||
}
|
||||
};
|
||||
if !release.paused && release.rollout_percent != expected_percent {
|
||||
return Err(UpdateError::InvalidManifest(
|
||||
"active rollout percentage does not match its channel".to_owned(),
|
||||
));
|
||||
}
|
||||
if release.artifacts.is_empty() {
|
||||
return Err(UpdateError::InvalidManifest(
|
||||
"manifest has no release artifacts".to_owned(),
|
||||
));
|
||||
}
|
||||
for (platform, artifact) in &release.artifacts {
|
||||
if artifact.is_empty() {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest is missing the {platform} artifact"
|
||||
)));
|
||||
}
|
||||
let digest = release
|
||||
.sha256
|
||||
.get(platform)
|
||||
.map(String::as_str)
|
||||
.unwrap_or_default();
|
||||
if !valid_sha256(digest) {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest has an invalid {platform} SHA-256"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate the complete release matrix required for new signed releases.
|
||||
pub fn validate_signed_release_matrix(release: &ClientRelease) -> Result<(), UpdateError> {
|
||||
validate_signed_release(release)?;
|
||||
for platform in REQUIRED_RELEASE_PLATFORMS {
|
||||
let artifact = release
|
||||
.artifacts
|
||||
.get(platform)
|
||||
.map(String::as_str)
|
||||
.unwrap_or_default();
|
||||
if artifact.is_empty() {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest is missing the {platform} artifact"
|
||||
)));
|
||||
}
|
||||
let digest = release
|
||||
.sha256
|
||||
.get(platform)
|
||||
.map(String::as_str)
|
||||
.unwrap_or_default();
|
||||
if !valid_sha256(digest) {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest has an invalid {platform} SHA-256"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Validate that a signed manifest contains the artifact for this exact ABI.
|
||||
pub fn validate_release_platform(
|
||||
release: &ClientRelease,
|
||||
platform_key: &str,
|
||||
) -> Result<(), UpdateError> {
|
||||
validate_signed_release(release)?;
|
||||
if !REQUIRED_RELEASE_PLATFORMS.contains(&platform_key) {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"unsupported release platform {platform_key}"
|
||||
)));
|
||||
}
|
||||
let artifact = release
|
||||
.artifacts
|
||||
.get(platform_key)
|
||||
.map(String::as_str)
|
||||
.unwrap_or_default();
|
||||
if artifact.is_empty() {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest is missing the {platform_key} artifact"
|
||||
)));
|
||||
}
|
||||
let digest = release
|
||||
.sha256
|
||||
.get(platform_key)
|
||||
.map(String::as_str)
|
||||
.unwrap_or_default();
|
||||
if !valid_sha256(digest) {
|
||||
return Err(UpdateError::InvalidManifest(format!(
|
||||
"manifest has an invalid {platform_key} SHA-256"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn download_verified(
|
||||
url: &str,
|
||||
destination: &Path,
|
||||
@ -145,6 +274,7 @@ pub async fn install_native_package(package: &Path) -> Result<(), UpdateError> {
|
||||
}
|
||||
|
||||
pub fn verify_release(release: &ClientRelease, public_key: &[u8; 32]) -> Result<(), UpdateError> {
|
||||
validate_signed_release(release)?;
|
||||
let mut unsigned = release.clone();
|
||||
unsigned.signature.clear();
|
||||
let payload = serde_json::to_vec(&unsigned)?;
|
||||
@ -169,7 +299,10 @@ pub fn verify_artifact(bytes: &[u8], expected_sha256: &str) -> Result<(), Update
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{verify_artifact, verify_release};
|
||||
use super::{
|
||||
REQUIRED_RELEASE_PLATFORMS, validate_release_platform, validate_signed_release,
|
||||
validate_signed_release_matrix, verify_artifact, verify_release,
|
||||
};
|
||||
use base64::Engine;
|
||||
use drosync_protocol::ClientRelease;
|
||||
use ed25519_dalek::{Signer, SigningKey};
|
||||
@ -185,14 +318,24 @@ mod tests {
|
||||
channel: "canary".to_owned(),
|
||||
protocol_version: "v1".to_owned(),
|
||||
minimum_server_version: "1.0.0".to_owned(),
|
||||
artifacts: BTreeMap::from([(
|
||||
"linux-x86_64".to_owned(),
|
||||
"https://downloads.example/drosync.deb".to_owned(),
|
||||
)]),
|
||||
sha256: BTreeMap::from([(
|
||||
"linux-x86_64".to_owned(),
|
||||
hex::encode(sha2::Sha256::digest(b"package")),
|
||||
)]),
|
||||
artifacts: REQUIRED_RELEASE_PLATFORMS
|
||||
.into_iter()
|
||||
.map(|platform| {
|
||||
(
|
||||
platform.to_owned(),
|
||||
format!("https://downloads.example/{platform}.package"),
|
||||
)
|
||||
})
|
||||
.collect::<BTreeMap<_, _>>(),
|
||||
sha256: REQUIRED_RELEASE_PLATFORMS
|
||||
.into_iter()
|
||||
.map(|platform| {
|
||||
(
|
||||
platform.to_owned(),
|
||||
hex::encode(sha2::Sha256::digest(b"package")),
|
||||
)
|
||||
})
|
||||
.collect::<BTreeMap<_, _>>(),
|
||||
signature: String::new(),
|
||||
rollout_percent: 5,
|
||||
paused: false,
|
||||
@ -221,4 +364,17 @@ mod tests {
|
||||
assert!(verify_release(&release, &public_key).is_err());
|
||||
assert!(verify_artifact(b"tampered", &release.sha256["linux-x86_64"]).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn signed_manifest_requires_linux_arm64_and_exact_platform_key() {
|
||||
let (mut release, _) = signed_release();
|
||||
release.artifacts.remove("linux-arm64");
|
||||
assert!(validate_signed_release(&release).is_ok());
|
||||
assert!(validate_signed_release_matrix(&release).is_err());
|
||||
|
||||
let (release, _) = signed_release();
|
||||
validate_release_platform(&release, "linux-arm64")
|
||||
.expect("ARM64 package belongs to the signed release matrix");
|
||||
assert!(validate_release_platform(&release, "linux-riscv64").is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@ -19,11 +19,26 @@ build_workspace() {
|
||||
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
|
||||
}
|
||||
|
||||
prepare_update_key() {
|
||||
local platform_key="$1"
|
||||
local specific_variable
|
||||
case "$platform_key" in
|
||||
linux-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64" ;;
|
||||
linux-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64" ;;
|
||||
macos-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64" ;;
|
||||
*) echo "unsupported preview update-key platform: $platform_key" >&2; exit 2 ;;
|
||||
esac
|
||||
local selected="${!specific_variable:-$DROSYNC_UPDATE_PUBLIC_KEY_BASE64}"
|
||||
export DROSYNC_UPDATE_PUBLIC_KEY_BASE64="$selected"
|
||||
export "$specific_variable=$selected"
|
||||
}
|
||||
|
||||
build_macos_preview() {
|
||||
[[ "$(uname -s)-$(uname -m)" == "Darwin-arm64" ]] || {
|
||||
echo "macos-arm64 preview must be built on an Apple Silicon runner" >&2
|
||||
exit 1
|
||||
}
|
||||
prepare_update_key macos-arm64
|
||||
build_workspace
|
||||
local app
|
||||
local package_version
|
||||
@ -58,9 +73,44 @@ build_linux_preview() {
|
||||
echo "linux-x86_64 preview must be built on a Linux x86_64 runner" >&2
|
||||
exit 1
|
||||
}
|
||||
build_workspace
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" "$ROOT_DIR/packaging/linux/build-deb.sh"
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" "$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
prepare_update_key linux-x86_64
|
||||
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release \
|
||||
-p drosync-cli -p drosync-daemon
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-deb.sh"
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
}
|
||||
|
||||
build_linux_arm64_preview() {
|
||||
[[ "$(uname -s)-$(uname -m)" == "Linux-x86_64" ]] || {
|
||||
echo "linux-arm64 preview must be cross-built on a Linux x86_64 runner" >&2
|
||||
exit 1
|
||||
}
|
||||
prepare_update_key linux-arm64
|
||||
local target="aarch64-unknown-linux-gnu"
|
||||
local linker="${DROSYNC_AARCH64_LINKER:-aarch64-linux-gnu-gcc}"
|
||||
local archiver="${DROSYNC_AARCH64_AR:-aarch64-linux-gnu-ar}"
|
||||
command -v "$linker" >/dev/null || {
|
||||
echo "missing ARM64 cross linker $linker (install gcc-aarch64-linux-gnu)" >&2
|
||||
exit 1
|
||||
}
|
||||
command -v "$archiver" >/dev/null || {
|
||||
echo "missing ARM64 archiver $archiver (install binutils-aarch64-linux-gnu)" >&2
|
||||
exit 1
|
||||
}
|
||||
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="$linker"
|
||||
export CC_aarch64_unknown_linux_gnu="$linker"
|
||||
export AR_aarch64_unknown_linux_gnu="$archiver"
|
||||
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release --target "$target" \
|
||||
-p drosync-cli -p drosync-daemon
|
||||
local binary_dir="$ROOT_DIR/target/$target/release"
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_PLATFORM_KEY=linux-arm64 \
|
||||
DROSYNC_DEB_ARCH=arm64 DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-deb.sh"
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" DROSYNC_VERSION="$VERSION" DROSYNC_PLATFORM_KEY=linux-arm64 \
|
||||
DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
}
|
||||
|
||||
case "$TARGET" in
|
||||
@ -70,8 +120,11 @@ case "$TARGET" in
|
||||
linux-x86_64)
|
||||
build_linux_preview
|
||||
;;
|
||||
linux-arm64)
|
||||
build_linux_arm64_preview
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 macos-arm64|linux-x86_64" >&2
|
||||
echo "usage: $0 macos-arm64|linux-x86_64|linux-arm64" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
@ -2,16 +2,75 @@
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
: "${DROSYNC_UPDATE_PUBLIC_KEY_BASE64:?set DROSYNC_UPDATE_PUBLIC_KEY_BASE64 from CI secrets}"
|
||||
TARGET="${1:-}"
|
||||
|
||||
prepare_update_key() {
|
||||
local platform_key="$1"
|
||||
local specific_variable
|
||||
case "$platform_key" in
|
||||
linux-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_ARM64" ;;
|
||||
linux-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_LINUX_X86_64" ;;
|
||||
macos-arm64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_MACOS_ARM64" ;;
|
||||
windows-x86_64) specific_variable="DROSYNC_UPDATE_PUBLIC_KEY_BASE64_WINDOWS_X86_64" ;;
|
||||
*) echo "unsupported update-key platform: $platform_key" >&2; exit 2 ;;
|
||||
esac
|
||||
local selected="${!specific_variable:-${DROSYNC_UPDATE_PUBLIC_KEY_BASE64:-}}"
|
||||
if [[ -z "$selected" ]]; then
|
||||
echo "set $specific_variable or DROSYNC_UPDATE_PUBLIC_KEY_BASE64 from CI secrets" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Keep the generic name populated for older targets while the platform crate
|
||||
# embeds the more specific variable for newly built clients.
|
||||
export DROSYNC_UPDATE_PUBLIC_KEY_BASE64="$selected"
|
||||
export "$specific_variable=$selected"
|
||||
}
|
||||
|
||||
require_linux_x86_64_builder() {
|
||||
[[ "$(uname -s)-$(uname -m)" == "Linux-x86_64" ]] || {
|
||||
echo "linux-arm64 cross builds require a Linux x86_64 builder" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
build_linux_arm64_headless() {
|
||||
require_linux_x86_64_builder
|
||||
prepare_update_key linux-arm64
|
||||
local target="aarch64-unknown-linux-gnu"
|
||||
local linker="${DROSYNC_AARCH64_LINKER:-aarch64-linux-gnu-gcc}"
|
||||
local archiver="${DROSYNC_AARCH64_AR:-aarch64-linux-gnu-ar}"
|
||||
command -v "$linker" >/dev/null || {
|
||||
echo "missing ARM64 cross linker $linker (install gcc-aarch64-linux-gnu)" >&2
|
||||
exit 1
|
||||
}
|
||||
command -v "$archiver" >/dev/null || {
|
||||
echo "missing ARM64 archiver $archiver (install binutils-aarch64-linux-gnu)" >&2
|
||||
exit 1
|
||||
}
|
||||
export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="$linker"
|
||||
export CC_aarch64_unknown_linux_gnu="$linker"
|
||||
export AR_aarch64_unknown_linux_gnu="$archiver"
|
||||
cargo build --locked --manifest-path "$ROOT_DIR/Cargo.toml" --release --target "$target" \
|
||||
-p drosync-cli -p drosync-daemon
|
||||
local binary_dir="$ROOT_DIR/target/$target/release"
|
||||
DROSYNC_PLATFORM_KEY=linux-arm64 DROSYNC_DEB_ARCH=arm64 DROSYNC_BINARY_DIR="$binary_dir" \
|
||||
DROSYNC_INCLUDE_GUI=0 "$ROOT_DIR/packaging/linux/build-deb.sh"
|
||||
DROSYNC_PLATFORM_KEY=linux-arm64 DROSYNC_BINARY_DIR="$binary_dir" DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
}
|
||||
|
||||
case "$TARGET" in
|
||||
linux-x86_64)
|
||||
prepare_update_key linux-x86_64
|
||||
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
|
||||
"$ROOT_DIR/packaging/linux/build-deb.sh"
|
||||
"$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
;;
|
||||
linux-arm64)
|
||||
build_linux_arm64_headless
|
||||
;;
|
||||
macos-arm64)
|
||||
[[ "$(uname -s)-$(uname -m)" == "Darwin-arm64" ]] || { echo "macos-arm64 must be built on an Apple Silicon runner" >&2; exit 1; }
|
||||
prepare_update_key macos-arm64
|
||||
cargo build --manifest-path "$ROOT_DIR/Cargo.toml" --release --workspace
|
||||
"$ROOT_DIR/packaging/macos/build-pkg.sh"
|
||||
;;
|
||||
@ -20,7 +79,7 @@ case "$TARGET" in
|
||||
exit 2
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 linux-x86_64|macos-arm64|windows-x86_64" >&2
|
||||
echo "usage: $0 linux-x86_64|linux-arm64|macos-arm64|windows-x86_64" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
|
||||
29
tools/drosync/packaging/linux/PORTABLE-SYSTEMD-README.md
Normal file
29
tools/drosync/packaging/linux/PORTABLE-SYSTEMD-README.md
Normal file
@ -0,0 +1,29 @@
|
||||
# drosync portable systemd bundle
|
||||
|
||||
This archive is an ABI-specific, no-container package for Linux systems with
|
||||
systemd. It contains the CLI and daemon; `linux-arm64` builds deliberately omit
|
||||
the desktop tray GUI so they can run on a headless S100.
|
||||
|
||||
Verify the adjacent SHA-256 file before extracting the archive. Then install it
|
||||
as root:
|
||||
|
||||
```bash
|
||||
sudo ./install-systemd-portable.sh --enable
|
||||
```
|
||||
|
||||
The portable binaries require Ubuntu's `libdbus-1-3` runtime package for the
|
||||
OS credential-store integration. Install it before running the installer:
|
||||
|
||||
```bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ca-certificates libdbus-1-3
|
||||
```
|
||||
|
||||
The installer uses `/opt/drosync` by default, creates the restricted `drosync`
|
||||
system account and `/var/lib/drosync`, writes `/etc/systemd/system/drosync.service`,
|
||||
and does not store access credentials. Configure credentials through the CLI or
|
||||
the existing `/etc/drosync/environment` / `DROSYNC_SECRET_FILE` mechanism.
|
||||
|
||||
Use `sudo ./uninstall-systemd-portable.sh` to stop and remove the service. The
|
||||
uninstaller leaves `/var/lib/drosync` and `/etc/drosync` intact so sync state and
|
||||
credentials are not deleted implicitly.
|
||||
@ -3,15 +3,56 @@ set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
VERSION="${DROSYNC_VERSION:-$(sed -n 's/^version = "\(.*\)"/\1/p' "$ROOT_DIR/Cargo.toml" | head -1)}"
|
||||
ARCH="${DROSYNC_DEB_ARCH:-amd64}"
|
||||
PLATFORM_KEY="${DROSYNC_PLATFORM_KEY:-}"
|
||||
BINARY_DIR="${DROSYNC_BINARY_DIR:-$ROOT_DIR/target/release}"
|
||||
if [[ -z "$PLATFORM_KEY" ]]; then
|
||||
case "$(uname -m)" in
|
||||
x86_64) PLATFORM_KEY="linux-x86_64" ;;
|
||||
aarch64|arm64) PLATFORM_KEY="linux-arm64" ;;
|
||||
*) echo "set DROSYNC_PLATFORM_KEY for this Linux architecture" >&2; exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
case "$PLATFORM_KEY" in
|
||||
linux-x86_64)
|
||||
DEFAULT_ARCH="amd64"
|
||||
DEFAULT_INCLUDE_GUI=1
|
||||
;;
|
||||
linux-arm64)
|
||||
DEFAULT_ARCH="arm64"
|
||||
DEFAULT_INCLUDE_GUI=0
|
||||
;;
|
||||
*)
|
||||
echo "unsupported Linux release platform: $PLATFORM_KEY" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
ARCH="${DROSYNC_DEB_ARCH:-$DEFAULT_ARCH}"
|
||||
INCLUDE_GUI="${DROSYNC_INCLUDE_GUI:-$DEFAULT_INCLUDE_GUI}"
|
||||
OUTPUT_DIR="${DROSYNC_OUTPUT_DIR:-$ROOT_DIR/dist}"
|
||||
PACKAGE_ROOT="$(mktemp -d)"
|
||||
trap 'rm -rf "$PACKAGE_ROOT"' EXIT
|
||||
|
||||
for binary in drosync drosync-daemon; do
|
||||
if [[ ! -x "$BINARY_DIR/$binary" ]]; then
|
||||
echo "missing executable $BINARY_DIR/$binary; build CLI and daemon for $PLATFORM_KEY first" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
case "$INCLUDE_GUI" in
|
||||
0|1) ;;
|
||||
*) echo "DROSYNC_INCLUDE_GUI must be 0 or 1" >&2; exit 2 ;;
|
||||
esac
|
||||
if [[ "$INCLUDE_GUI" == 1 && ! -x "$BINARY_DIR/drosync-gui" ]]; then
|
||||
echo "missing executable $BINARY_DIR/drosync-gui while DROSYNC_INCLUDE_GUI=1" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$PACKAGE_ROOT/DEBIAN" "$PACKAGE_ROOT/usr/bin" "$PACKAGE_ROOT/lib/systemd/system" "$PACKAGE_ROOT/usr/lib/systemd/user"
|
||||
cp "$ROOT_DIR/target/release/drosync" "$PACKAGE_ROOT/usr/bin/drosync"
|
||||
cp "$ROOT_DIR/target/release/drosync-daemon" "$PACKAGE_ROOT/usr/bin/drosync-daemon"
|
||||
cp "$ROOT_DIR/target/release/drosync-gui" "$PACKAGE_ROOT/usr/bin/drosync-gui"
|
||||
cp "$BINARY_DIR/drosync" "$PACKAGE_ROOT/usr/bin/drosync"
|
||||
cp "$BINARY_DIR/drosync-daemon" "$PACKAGE_ROOT/usr/bin/drosync-daemon"
|
||||
if [[ "$INCLUDE_GUI" == 1 ]]; then
|
||||
cp "$BINARY_DIR/drosync-gui" "$PACKAGE_ROOT/usr/bin/drosync-gui"
|
||||
fi
|
||||
sed -e "s/^Version:.*/Version: $VERSION/" -e "s/^Architecture:.*/Architecture: $ARCH/" \
|
||||
"$ROOT_DIR/packaging/linux/debian/control" >"$PACKAGE_ROOT/DEBIAN/control"
|
||||
cp "$ROOT_DIR/packaging/linux/debian/postinst" "$PACKAGE_ROOT/DEBIAN/postinst"
|
||||
@ -21,6 +62,10 @@ cp "$ROOT_DIR/packaging/linux/drosync.service" "$PACKAGE_ROOT/lib/systemd/system
|
||||
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$PACKAGE_ROOT/usr/lib/systemd/user/drosync.service"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
dpkg-deb --root-owner-group --build "$PACKAGE_ROOT" "$OUTPUT_DIR/drosync_${VERSION}_${ARCH}.deb"
|
||||
(
|
||||
cd "$OUTPUT_DIR"
|
||||
sha256sum "drosync_${VERSION}_${ARCH}.deb" >"drosync_${VERSION}_${ARCH}.deb.sha256"
|
||||
)
|
||||
if [[ -n "${DROSYNC_DEB_SIGN_KEY:-}" ]]; then
|
||||
dpkg-sig --sign builder -k "$DROSYNC_DEB_SIGN_KEY" "$OUTPUT_DIR/drosync_${VERSION}_${ARCH}.deb"
|
||||
fi
|
||||
|
||||
@ -4,14 +4,62 @@ set -euo pipefail
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
VERSION="${DROSYNC_VERSION:-1.0.0}"
|
||||
OUTPUT_DIR="${DROSYNC_OUTPUT_DIR:-$ROOT_DIR/dist}"
|
||||
PLATFORM_KEY="${DROSYNC_PLATFORM_KEY:-}"
|
||||
BINARY_DIR="${DROSYNC_BINARY_DIR:-$ROOT_DIR/target/release}"
|
||||
if [[ -z "$PLATFORM_KEY" ]]; then
|
||||
case "$(uname -m)" in
|
||||
x86_64) PLATFORM_KEY="linux-x86_64" ;;
|
||||
aarch64|arm64) PLATFORM_KEY="linux-arm64" ;;
|
||||
*) echo "set DROSYNC_PLATFORM_KEY for this Linux architecture" >&2; exit 2 ;;
|
||||
esac
|
||||
fi
|
||||
case "$PLATFORM_KEY" in
|
||||
linux-x86_64) DEFAULT_INCLUDE_GUI=1 ;;
|
||||
linux-arm64) DEFAULT_INCLUDE_GUI=0 ;;
|
||||
*) echo "unsupported Linux release platform: $PLATFORM_KEY" >&2; exit 2 ;;
|
||||
esac
|
||||
INCLUDE_GUI="${DROSYNC_INCLUDE_GUI:-$DEFAULT_INCLUDE_GUI}"
|
||||
STAGE="$(mktemp -d)"
|
||||
trap 'rm -rf "$STAGE"' EXIT
|
||||
|
||||
mkdir -p "$STAGE/drosync-$VERSION" "$OUTPUT_DIR"
|
||||
cp "$ROOT_DIR/target/release/drosync" "$ROOT_DIR/target/release/drosync-daemon" "$ROOT_DIR/target/release/drosync-gui" "$STAGE/drosync-$VERSION/"
|
||||
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$STAGE/drosync-$VERSION/"
|
||||
tar -C "$STAGE" -czf "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz" "drosync-$VERSION"
|
||||
sha256sum "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz" >"$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz.sha256"
|
||||
if [[ -n "${DROSYNC_GPG_SIGN_KEY:-}" ]]; then
|
||||
gpg --batch --yes --local-user "$DROSYNC_GPG_SIGN_KEY" --detach-sign --armor "$OUTPUT_DIR/drosync-${VERSION}-linux-x86_64.tar.gz"
|
||||
for binary in drosync drosync-daemon; do
|
||||
if [[ ! -x "$BINARY_DIR/$binary" ]]; then
|
||||
echo "missing executable $BINARY_DIR/$binary; build CLI and daemon for $PLATFORM_KEY first" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
case "$INCLUDE_GUI" in
|
||||
0|1) ;;
|
||||
*) echo "DROSYNC_INCLUDE_GUI must be 0 or 1" >&2; exit 2 ;;
|
||||
esac
|
||||
if [[ "$INCLUDE_GUI" == 1 && ! -x "$BINARY_DIR/drosync-gui" ]]; then
|
||||
echo "missing executable $BINARY_DIR/drosync-gui while DROSYNC_INCLUDE_GUI=1" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PACKAGE_NAME="drosync-$VERSION"
|
||||
PACKAGE_ROOT="$STAGE/$PACKAGE_NAME"
|
||||
mkdir -p "$PACKAGE_ROOT" "$OUTPUT_DIR"
|
||||
cp "$BINARY_DIR/drosync" "$PACKAGE_ROOT/drosync"
|
||||
cp "$BINARY_DIR/drosync-daemon" "$PACKAGE_ROOT/drosync-daemon"
|
||||
if [[ "$INCLUDE_GUI" == 1 ]]; then
|
||||
cp "$BINARY_DIR/drosync-gui" "$PACKAGE_ROOT/drosync-gui"
|
||||
fi
|
||||
cp "$ROOT_DIR/packaging/linux/drosync.service" "$PACKAGE_ROOT/drosync.service"
|
||||
cp "$ROOT_DIR/packaging/linux/drosync-user.service" "$PACKAGE_ROOT/drosync-user.service"
|
||||
cp "$ROOT_DIR/packaging/linux/drosync-portable.service" "$PACKAGE_ROOT/drosync-portable.service"
|
||||
cp "$ROOT_DIR/packaging/linux/install-systemd-portable.sh" "$PACKAGE_ROOT/install-systemd-portable.sh"
|
||||
cp "$ROOT_DIR/packaging/linux/uninstall-systemd-portable.sh" "$PACKAGE_ROOT/uninstall-systemd-portable.sh"
|
||||
cp "$ROOT_DIR/packaging/linux/PORTABLE-SYSTEMD-README.md" "$PACKAGE_ROOT/README-systemd-portable.md"
|
||||
chmod 0755 "$PACKAGE_ROOT/install-systemd-portable.sh" "$PACKAGE_ROOT/uninstall-systemd-portable.sh"
|
||||
|
||||
ARCHIVE="$OUTPUT_DIR/drosync-${VERSION}-${PLATFORM_KEY}.tar.gz"
|
||||
tar --sort=name --mtime='@0' --owner=0 --group=0 --numeric-owner \
|
||||
-C "$STAGE" -czf "$ARCHIVE" "$PACKAGE_NAME"
|
||||
(
|
||||
cd "$OUTPUT_DIR"
|
||||
sha256sum "$(basename "$ARCHIVE")" >"$(basename "$ARCHIVE").sha256"
|
||||
)
|
||||
if [[ -n "${DROSYNC_GPG_SIGN_KEY:-}" ]]; then
|
||||
gpg --batch --yes --local-user "$DROSYNC_GPG_SIGN_KEY" --detach-sign --armor "$ARCHIVE"
|
||||
fi
|
||||
|
||||
@ -4,6 +4,7 @@ Section: utils
|
||||
Priority: optional
|
||||
Architecture: amd64
|
||||
Maintainer: D-Robotics <support@example.invalid>
|
||||
Depends: ca-certificates, debsig-verify, libsecret-1-0
|
||||
Depends: ca-certificates, debsig-verify, libdbus-1-3, libsecret-1-0
|
||||
Description: Secure one-way local-to-BOS synchronization client
|
||||
drosync provides a CLI, daemon, tray GUI, crash recovery, and signed updates.
|
||||
drosync provides a CLI, daemon, crash recovery, and signed updates. Some
|
||||
packages additionally include an optional tray GUI.
|
||||
|
||||
@ -8,4 +8,10 @@ if ! getent passwd drosync >/dev/null 2>&1; then
|
||||
adduser --system --ingroup drosync --home /var/lib/drosync --no-create-home drosync
|
||||
fi
|
||||
install -d -o drosync -g drosync -m 0700 /var/lib/drosync
|
||||
install -d -o drosync -g drosync -m 0700 /var/lib/drosync/config
|
||||
install -d -o root -g root -m 0755 /etc/drosync
|
||||
if [ ! -e /etc/drosync/environment ]; then
|
||||
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' >/etc/drosync/environment
|
||||
chmod 0644 /etc/drosync/environment
|
||||
fi
|
||||
systemctl daemon-reload
|
||||
|
||||
24
tools/drosync/packaging/linux/drosync-portable.service
Normal file
24
tools/drosync/packaging/linux/drosync-portable.service
Normal file
@ -0,0 +1,24 @@
|
||||
[Unit]
|
||||
Description=drosync portable system synchronization daemon
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=@DROSYNC_INSTALL_ROOT@/drosync-daemon
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
User=drosync
|
||||
Group=drosync
|
||||
Environment=HOME=/var/lib/drosync
|
||||
Environment=XDG_DATA_HOME=/var/lib/drosync
|
||||
Environment=XDG_CONFIG_HOME=/var/lib/drosync/config
|
||||
EnvironmentFile=-/etc/drosync/environment
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=read-only
|
||||
ReadWritePaths=/var/lib/drosync
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@ -12,7 +12,7 @@ User=drosync
|
||||
Group=drosync
|
||||
Environment=HOME=/var/lib/drosync
|
||||
Environment=XDG_DATA_HOME=/var/lib/drosync
|
||||
Environment=XDG_CONFIG_HOME=/etc/drosync
|
||||
Environment=XDG_CONFIG_HOME=/var/lib/drosync/config
|
||||
EnvironmentFile=-/etc/drosync/environment
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
|
||||
81
tools/drosync/packaging/linux/install-systemd-portable.sh
Executable file
81
tools/drosync/packaging/linux/install-systemd-portable.sh
Executable file
@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
|
||||
echo "run this installer as root (for example: sudo ./install-systemd-portable.sh)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
INSTALL_ROOT="/opt/drosync"
|
||||
ENABLE=0
|
||||
|
||||
usage() {
|
||||
echo "usage: $0 [--prefix /opt/drosync] [--enable]" >&2
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--prefix)
|
||||
[[ $# -ge 2 ]] || { usage; exit 2; }
|
||||
INSTALL_ROOT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--enable)
|
||||
ENABLE=1
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$INSTALL_ROOT" in
|
||||
/*) ;;
|
||||
*) echo "--prefix must be an absolute path" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
for binary in drosync drosync-daemon; do
|
||||
[[ -x "$SOURCE_DIR/$binary" ]] || {
|
||||
echo "portable bundle is missing executable $binary" >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
if ! ldconfig -p 2>/dev/null | grep -Fq 'libdbus-1.so.3'; then
|
||||
echo "missing runtime dependency libdbus-1.so.3; install it with: apt-get install libdbus-1-3" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! getent group drosync >/dev/null 2>&1; then
|
||||
groupadd --system drosync
|
||||
fi
|
||||
if ! getent passwd drosync >/dev/null 2>&1; then
|
||||
useradd --system --gid drosync --home-dir /var/lib/drosync --no-create-home --shell /usr/sbin/nologin drosync
|
||||
fi
|
||||
install -d -o drosync -g drosync -m 0700 /var/lib/drosync
|
||||
install -d -o drosync -g drosync -m 0700 /var/lib/drosync/config
|
||||
install -d -m 0755 /etc/drosync "$INSTALL_ROOT"
|
||||
if [[ ! -e /etc/drosync/environment ]]; then
|
||||
printf '%s\n' 'DROSYNC_SECRET_FILE=/var/lib/drosync/token' >/etc/drosync/environment
|
||||
chmod 0644 /etc/drosync/environment
|
||||
fi
|
||||
install -m 0755 "$SOURCE_DIR/drosync" "$INSTALL_ROOT/drosync"
|
||||
install -m 0755 "$SOURCE_DIR/drosync-daemon" "$INSTALL_ROOT/drosync-daemon"
|
||||
if [[ -f "$SOURCE_DIR/drosync-gui" ]]; then
|
||||
install -m 0755 "$SOURCE_DIR/drosync-gui" "$INSTALL_ROOT/drosync-gui"
|
||||
fi
|
||||
sed "s|@DROSYNC_INSTALL_ROOT@|$INSTALL_ROOT|g" "$SOURCE_DIR/drosync-portable.service" \
|
||||
>/etc/systemd/system/drosync.service
|
||||
chmod 0644 /etc/systemd/system/drosync.service
|
||||
systemctl daemon-reload
|
||||
if [[ "$ENABLE" == 1 ]]; then
|
||||
systemctl enable --now drosync.service
|
||||
fi
|
||||
echo "installed drosync portable systemd bundle to $INSTALL_ROOT"
|
||||
27
tools/drosync/packaging/linux/uninstall-systemd-portable.sh
Executable file
27
tools/drosync/packaging/linux/uninstall-systemd-portable.sh
Executable file
@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
|
||||
echo "run this uninstaller as root" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
INSTALL_ROOT="/opt/drosync"
|
||||
if [[ "${1:-}" == "--prefix" ]]; then
|
||||
[[ $# -eq 2 ]] || { echo "usage: $0 [--prefix /opt/drosync]" >&2; exit 2; }
|
||||
INSTALL_ROOT="$2"
|
||||
elif [[ $# -ne 0 ]]; then
|
||||
echo "usage: $0 [--prefix /opt/drosync]" >&2
|
||||
exit 2
|
||||
fi
|
||||
case "$INSTALL_ROOT" in
|
||||
/*) ;;
|
||||
*) echo "--prefix must be an absolute path" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
systemctl disable --now drosync.service >/dev/null 2>&1 || true
|
||||
rm -f /etc/systemd/system/drosync.service
|
||||
systemctl daemon-reload
|
||||
rm -f "$INSTALL_ROOT/drosync" "$INSTALL_ROOT/drosync-daemon" "$INSTALL_ROOT/drosync-gui"
|
||||
rmdir "$INSTALL_ROOT" 2>/dev/null || true
|
||||
echo "uninstalled drosync portable systemd bundle from $INSTALL_ROOT"
|
||||
53
tools/drosync/packaging/linux/verify-arm64-portable.sh
Executable file
53
tools/drosync/packaging/linux/verify-arm64-portable.sh
Executable file
@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
WORK_DIR="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK_DIR"' EXIT
|
||||
|
||||
BIN_DIR="$WORK_DIR/bin"
|
||||
OUTPUT_DIR="$WORK_DIR/dist"
|
||||
VERSION="0.0.0-arm64-package-check"
|
||||
mkdir -p "$BIN_DIR"
|
||||
for binary in drosync drosync-daemon; do
|
||||
printf '#!/bin/sh\nexit 0\n' >"$BIN_DIR/$binary"
|
||||
chmod 0755 "$BIN_DIR/$binary"
|
||||
done
|
||||
|
||||
DROSYNC_VERSION="$VERSION" \
|
||||
DROSYNC_OUTPUT_DIR="$OUTPUT_DIR" \
|
||||
DROSYNC_PLATFORM_KEY=linux-arm64 \
|
||||
DROSYNC_BINARY_DIR="$BIN_DIR" \
|
||||
DROSYNC_INCLUDE_GUI=0 \
|
||||
"$ROOT_DIR/packaging/linux/build-portable.sh"
|
||||
|
||||
ARCHIVE="$OUTPUT_DIR/drosync-${VERSION}-linux-arm64.tar.gz"
|
||||
CHECKSUM="$ARCHIVE.sha256"
|
||||
[[ -f "$ARCHIVE" && -f "$CHECKSUM" ]]
|
||||
sha256sum --check "$CHECKSUM"
|
||||
|
||||
CONTENTS="$WORK_DIR/contents.txt"
|
||||
tar -tzf "$ARCHIVE" >"$CONTENTS"
|
||||
PACKAGE_ROOT="drosync-$VERSION"
|
||||
for path in \
|
||||
"$PACKAGE_ROOT/drosync" \
|
||||
"$PACKAGE_ROOT/drosync-daemon" \
|
||||
"$PACKAGE_ROOT/drosync-portable.service" \
|
||||
"$PACKAGE_ROOT/install-systemd-portable.sh" \
|
||||
"$PACKAGE_ROOT/uninstall-systemd-portable.sh" \
|
||||
"$PACKAGE_ROOT/README-systemd-portable.md"; do
|
||||
grep -Fqx "$path" "$CONTENTS"
|
||||
done
|
||||
if grep -Fq "drosync-gui" "$CONTENTS"; then
|
||||
echo "ARM64 portable package must not require or include drosync-gui" >&2
|
||||
exit 1
|
||||
fi
|
||||
tar -xOf "$ARCHIVE" "$PACKAGE_ROOT/drosync-portable.service" \
|
||||
| grep -Fqx 'ExecStart=@DROSYNC_INSTALL_ROOT@/drosync-daemon'
|
||||
tar -xOf "$ARCHIVE" "$PACKAGE_ROOT/install-systemd-portable.sh" \
|
||||
| grep -Fqx ' systemctl enable --now drosync.service'
|
||||
bash -n "$ROOT_DIR/packaging/linux/build-portable.sh" \
|
||||
"$ROOT_DIR/packaging/linux/install-systemd-portable.sh" \
|
||||
"$ROOT_DIR/packaging/linux/uninstall-systemd-portable.sh"
|
||||
|
||||
echo "validated ARM64 no-GUI systemd portable package: $ARCHIVE"
|
||||
@ -26,11 +26,12 @@
|
||||
"paused": { "type": "boolean" },
|
||||
"artifacts": {
|
||||
"type": "object",
|
||||
"required": ["windows-x86_64", "linux-x86_64", "macos-arm64"],
|
||||
"required": ["windows-x86_64", "linux-x86_64", "linux-arm64", "macos-arm64"],
|
||||
"additionalProperties": { "type": "string", "format": "uri" }
|
||||
},
|
||||
"sha256": {
|
||||
"type": "object",
|
||||
"required": ["windows-x86_64", "linux-x86_64", "linux-arm64", "macos-arm64"],
|
||||
"additionalProperties": { "type": "string", "pattern": "^[a-f0-9]{64}$" }
|
||||
},
|
||||
"signature": { "type": "string", "contentEncoding": "base64" }
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user