diff --git a/docs/cluster-poc-deployment.md b/docs/cluster-poc-deployment.md index 0adc1e1..2651fb8 100644 --- a/docs/cluster-poc-deployment.md +++ b/docs/cluster-poc-deployment.md @@ -1,8 +1,8 @@ # WishForge 部署在 POC 集群 -把原本跑在 Mac 上的 WishForge(数据库、编排、对象存储、控制面、网页、执行器、发布 Runner)整体迁到 DroboticFlow 所在的百度云 POC 集群(`https://120.48.181.105:6443`)。执行器仍然使用 Claude CLI,模型改走公司地瓜网关(Anthropic 协议,`https://ai-api.d-robotics.cc`,客户端自动拼接 `/v1/messages`),默认 `deepseek-pro@latest`,小模型槽位 `deepseek-flash@latest`。 +WishForge(数据库、编排、对象存储、控制面、网页、执行器、发布 Runner、备份)只运行在 DroboticFlow 所在的百度云 POC 集群(`https://120.48.181.105:6443`)。它在 2026-11-01 从 Mac 迁入,Mac 上的部署和数据已于 2026-11-02 删除;操作者的电脑只保存集群设置的输入(见「数据与密钥」),不运行任何服务。执行器仍然使用 Claude CLI,模型改走公司地瓜网关(Anthropic 协议,`https://ai-api.d-robotics.cc`,客户端自动拼接 `/v1/messages`),默认 `deepseek-pro@latest`,小模型槽位 `deepseek-flash@latest`。 -## 当前进度(2026-11-01) +## 当前进度(2026-11-02) | 步骤 | 状态 | |---|---| @@ -10,14 +10,13 @@ | 命名空间、节点数据目录、本地卷、访问规则、准入策略 | 完成,权限已逐项核对 | | Gitea 部署密钥 | 已添加到 `longtao.wu/droboticflow`(写权限),集群内拉取已验证 | | Secret 与 ConfigMap | 已创建,网关 Key 已写入 | -| 演练迁移(`migrate --rehearsal`) | 完成:逐表行数一致;Runner 保持空闲,Mac 照常服务 | | 演练验收 | 隔离标签与日志脱敏修复已部署并在新运行中验证;发布后重跑愿望优化、幂等提交、实现、暂停/继续、独立检查、审核门禁和取消,通过并完成产物独立验收,见[部署后验收](cluster-poc-postdeploy-2026-11-01.md)。该轮暴露的取消期间检查 Pod 回收竞态已修复并复现验证通过,见[检查 Pod 清理验收](cluster-poc-checkpod-cleanup-2026-11-01.md) | | 公网入口(独立负载均衡 + 公网 IP + 域名) | **`https://wishforge.d-robotics.cc`** 已启用正式证书,校验通过;负载均衡 `180.76.227.59`,HTTP 跳转 HTTPS。本机解析正常,其他内网设备需确认公司 DNS 的记录,见下文 | -| 正式切换(`migrate --final`) | 2026-11-01 完成:Mac 服务已停,数据逐表一致导入,集群 Runner 启动;之后 Mac 只作回退 | +| 从 Mac 迁入 | 2026-11-01 演练后正式切换,数据逐表一致导入;2026-11-02 删除 Mac 部署、数据和备份副本,迁移命令也已移除 | | 项目 | 2026-11-01 旧项目全部删除,只保留两个:「DroboticFlow POC」(`poc` 分支 → POC 集群)和「DroboticFlow Fanyu · dev」(`dev` 分支 → Fanyu 集群),见下文「两个项目」 | -| 每日备份 | 2026-11-01 已部署:节点上每日快照、Mac 每日 10:17 复制(launchd 已安装并试跑成功),在演练副本上完成一次恢复演练,见下文「备份与恢复」 | +| 每日备份 | 节点上每日快照,恢复在集群内完成,见下文「备份与恢复」 | -演练验收: +迁入前后的验收: | 项目 | 结果 | |---|---| @@ -34,8 +33,6 @@ | 合入、构建、发布 | 正式切换后由第一个真实愿望验收(演练副本不向 `poc` 分支推送、不发布) | | Pod 重启后恢复:运行停在代码审核时删除 Pod,约 25 秒恢复就绪;数据行数、运行状态、待审核记录和 Temporal 工作流不变,之后取消正常完成 | 通过 | -Mac 上的服务仍在运行,切换前的临时公网链接由 Cloudflare 隧道提供(地址以维护者给出的为准),切换完成后停用。 - 公网入口是 WishForge 专属负载均衡的公网 IP `180.76.227.59`,以及公司自己的域名 `wishforge.d-robotics.cc`(公网 DNS 在阿里云,A 记录指向这个 IP;不使用 sslip.io、Cloudflare 等第三方域名)。HTTPS 时域名和 IP 各有一张 Let's Encrypt 证书,由 Caddy 自动续签,HTTP 的 GET/HEAD 跳转到 HTTPS,其他方法拒绝;控制台以域名为主地址。也可以用 `public --http` 改成纯 HTTP,这时密码和会话明文传输。 2026-10-30 查询时,公司内网 DNS(`10.9.0.10`、`10.48.6.10`,hobot.cc 的 AD 域控)自己托管的 `d-robotics.cc` 区域没有这条记录,返回 NXDOMAIN;仅在公网 DNS 加记录不一定能让内网用户访问。2026-11-01 本机已能正常解析并通过域名访问。其他内网设备若仍解析失败,需要 IT 在内网区域补同样的 A 记录 `wishforge` → `180.76.227.59`;在那之前可用 IP 访问。`public --domain` 只要求公网解析正确,本机 DNS 解析不到时会给出提醒。 @@ -49,7 +46,7 @@ Mac 上的服务仍在运行,切换前的临时公网链接由 Cloudflare 隧 - 两个项目的环境都不预授权:**每次发布都要有人批准**。愿望一律用 Claude CLI 执行,从目标分支开始;受保护检查都在 POC 集群的检查 Pod 里跑。 - 两个项目沿用旧 POC 项目的「后端代码」划分(除 `src/`、`tests/frontend/`、`**/*.md` 外都算后端),前端账号只能改前端。POC 项目的成员沿用旧项目(管理员和 `qianduan`);Fanyu 项目先只有管理员,要给其他账号,在账号管理页面勾选项目。 -- 每个环境有固定的 profile:POC 的在 Mac 的 `~/.config/wishforge/poc-profile.json`,Fanyu 的在 `.wishforge/cluster-poc/fanyu-profile.json`(由 `fanyu-access` 生成)。profile 钉死分支、集群地址、命名空间及其 UID、各 Deployment 的 UID、镜像仓库和访问地址;任何一项对不上,构建和发布都会拒绝。 +- 每个环境有固定的 profile:POC 的在 `.wishforge/cluster-poc/poc-profile.json`,Fanyu 的在 `.wishforge/cluster-poc/fanyu-profile.json`(由 `fanyu-access` 生成)。profile 钉死分支、集群地址、命名空间及其 UID、各 Deployment 的 UID、镜像仓库和访问地址;任何一项对不上,构建和发布都会拒绝。 - Fanyu 的 api 镜像同时跑在 api、annotation-processor、annotation-validator 三个 Deployment 上:一次发布把三者都换成新 api 镜像(顺序 coordinator → worker → api → processor → validator),逐个核对 rollout 和 Pod 摘要;回滚时每个 Deployment 恢复成它自己原来的镜像。headpose、wilor 不在发布范围内。 - Fanyu 集群里 WishForge 的身份([fanyu-access.yaml](../infra/k8s/cluster-poc/fanyu-access.yaml)):Service Account `droboticflow-fanyu/wishforge`,只能读取并更新那 5 个 Deployment 的镜像、查看 rollout、在 `droboticflow-fanyu` 里创建构建 Pod,读不到 Secret 和 ConfigMap;准入策略只允许它创建固定 Kaniko 镜像、只挂临时目录和镜像仓库凭据、不带令牌的构建 Pod。它的长期令牌只在 WishForge 集群的 Secret `wishforge-fanyu-kube` 里,只挂进 executor(构建)和 runner-fanyu(发布)。删除 Fanyu 集群里的 Secret `wishforge-token` 即可吊销。 @@ -65,11 +62,11 @@ node scripts/cluster-poc.mjs fanyu-access --kubeconfig ~/workSpace/droboticflow/ node scripts/cluster-poc.mjs projects --replace ``` -它删除每个项目及其愿望、运行、发布、环境、Runner、证据和审核记录,以及运行事件、对象存储里的产物和托管的 Git 仓库;保留账号、成员、用量账本和审计日志。两个 Runner 的注册信息保存在 Mac 的 `.wishforge/cluster-poc/projects.json`(0600),之后的 `secrets`/`apply` 从这里生成两个 Runner 的设置。 +它删除每个项目及其愿望、运行、发布、环境、Runner、证据和审核记录,以及运行事件、对象存储里的产物和托管的 Git 仓库;保留账号、成员、用量账本和审计日志。两个 Runner 的注册信息保存在操作者电脑的 `.wishforge/cluster-poc/projects.json`(0600),之后的 `secrets`/`apply` 从这里生成两个 Runner 的设置。 ## 架构 -命名空间 `wishforge` 中一个单副本 StatefulSet([platform.yaml](../infra/k8s/cluster-poc/platform.yaml)),固定在 CPU 节点 192.168.80.249(由本地卷的节点亲和性决定)。Pod 里的容器共享回环网络,结构与 Mac 相同: +命名空间 `wishforge` 中一个单副本 StatefulSet([platform.yaml](../infra/k8s/cluster-poc/platform.yaml)),固定在 CPU 节点 192.168.80.249(由本地卷的节点亲和性决定)。Pod 里的容器共享回环网络: | 容器 | 内容 | 监听 | |---|---|---| @@ -92,11 +89,11 @@ node scripts/cluster-poc.mjs projects --replace - 验证器每次重建工作区目录后会换一个新的检查 Pod;命令超时或被取消时连同 Pod 一起删除;Pod 自身最长存活 2 小时; - 起不来或 exec 连不上时,运行停在「基础设施」等待,不会当作检查失败。 -与 Mac 的差异:Mac 上检查在 Seatbelt 中断网运行;集群里检查 Pod 可以访问外网和集群内网(集群不执行网络策略)。这是选择「不放宽集群安全」换来的代价,证据中记录为 `checkSandbox: pod`。 +注意:检查 Pod 可以访问外网和集群内网(集群不执行网络策略)。这是选择「不放宽集群安全」换来的代价,证据中记录为 `checkSandbox: pod`。 ### 身份与权限([rbac.yaml](../infra/k8s/cluster-poc/rbac.yaml)、[admission.yaml](../infra/k8s/cluster-poc/admission.yaml)) -WishForge 在集群里只有一个身份 `wishforge/wishforge`,它的短期令牌只挂进 executor 和 runner 两个容器,取代 Mac 上的集群管理员 kubeconfig: +WishForge 在集群里只有一个身份 `wishforge/wishforge`,它的短期令牌只挂进 executor 和 runner 两个容器: - `wishforge-checks`:创建、删除检查 Pod 并 exec; - `droboticflow`:只能读取和更新 `droboticflow-api`、`droboticflow-worker`、`drosync-coordinator` 三个 Deployment,创建 Kaniko 构建 Pod;读不到任何 Secret 或 ConfigMap; @@ -108,10 +105,9 @@ WishForge 在集群里只有一个身份 `wishforge/wishforge`,它的短期令 - 节点 192.168.80.249 的本地盘 `/home/cce/wishforge`([storage.yaml](../infra/k8s/cluster-poc/storage.yaml)),回收策略 Retain,各容器只挂自己的子目录。检查 Pod 另有一个只指向其中 `exec/` 目录的本地卷。 - 每个容器只读自己的设置文件(`wishforge-env` 中的 `platform.env`、`executor.env`、`runner.env`),executor 拿不到数据库和对象存储的密码。 -- 从 Mac 原样带过去的只有持久数据依赖的密钥和令牌(`CARRIED`,见 [cluster-poc.mjs](../scripts/cluster-poc.mjs));数据库和对象存储使用新的随机密码,只保存在 Mac 私有的 `.wishforge/cluster-poc/generated.json`(0600)。 -- 网关 Key 只写进集群 Secret `wishforge-claude`,不在 Mac 上另存。 -- 集群版生成新的本机管理员密码;个人账号随数据库迁移,沿用原密码。会话密钥更新,所有人需要重新登录一次。 -- 项目设置里的受保护安装、检查命令是 `poc-project-check.mjs` 的绝对路径。从 Mac 迁来的项目写的是 Mac 上的 Node 和仓库路径,platform 容器每次启动、控制面开始服务之前把它们改成镜像里的 `/usr/local/bin/node /opt/wishforge/scripts/poc-project-check.mjs install|verify`([cluster-poc-project.ts](../scripts/cluster-poc-project.ts));别的命令和已经指向镜像的命令不动。检查命令变化会让旧的检查证据不能再用于发布,迁移前已要求没有待发布的版本,所以没有影响。 +- 集群设置的输入只在操作者电脑的 `.wishforge/cluster-poc/`(0700,文件 0600):持久数据依赖的密钥和令牌 `carried.env`(`CARRIED`,见 [cluster-poc.mjs](../scripts/cluster-poc.mjs),其中 `WF_VAULT_KEY` 解密数据库里的凭据)、Runner 签名密钥 `runner-ed25519.pem`、套餐 `plans.json`、POC 与 Fanyu 的 profile、Gitea 主机密钥 `known_hosts`、数据库与对象存储密码 `generated.json`、两个 Runner 的注册 `projects.json`。`secrets`/`apply` 只读这些文件和 POC 集群的 kubeconfig。**这个目录要另外妥善备份**:快照里没有它,丢了就无法解密快照里的凭据,也无法重建 Pod 设置。 +- 网关 Key 只写进集群 Secret `wishforge-claude`,不在本地另存。 +- 项目的受保护安装、检查命令是镜像里的 `/usr/local/bin/node /opt/wishforge/scripts/poc-project-check.mjs install|verify`;platform 容器每次启动时把仍指向别处的这类命令改回镜像里的路径([cluster-poc-project.ts](../scripts/cluster-poc-project.ts))。 ### 实现要点 @@ -124,7 +120,7 @@ WishForge 在集群里只有一个身份 `wishforge/wishforge`,它的短期令 - SWC 的原生模块缓存拒绝父目录对其他用户可写的位置,platform 容器的 `HOME` 放在持久卷里的私有目录。 - 网页启动器只透传 `WF_LOCAL_API_TOKEN`,不透传 `WF_DEV_AUTH_TOKEN`。集群设置必须把前者设成控制面所接受的后者;否则管理员密码校验和会话都成功,项目接口仍返回 401。凭据只在 platform 容器内使用,不下发到 executor 或 runner。 - OpenSSH 按用户数据库里的家目录找 `~/.ssh`,不看 `$HOME`,所以 Git 的 SSH 目录挂在镜像用户的家目录 `/home/wishforge`。 -- 受保护检查的工具链除了 Node、npm、Go、Git,还有 python3(带 NumPy、OpenCV、PyYAML)和 ffmpeg:DroboticFlow 的 Go 测试(`internal/worker`)用它们运行算子 runner `backend/operator-runner/main.py` 的视频和数据集算子。Mac 上检查用的是 `/opt/homebrew` 里的 python3 和 ffmpeg;镜像里装 Debian 的 `python3`、`ffmpeg`,Python 包按 [check-python-requirements.txt](../infra/images/check-python-requirements.txt) 固定版本和哈希安装。缺了它们,这些测试会失败或等到超时。 +- 受保护检查的工具链除了 Node、npm、Go、Git,还有 python3(带 NumPy、OpenCV、PyYAML)和 ffmpeg:DroboticFlow 的 Go 测试(`internal/worker`)用它们运行算子 runner `backend/operator-runner/main.py` 的视频和数据集算子。镜像里装 Debian 的 `python3`、`ffmpeg`,Python 包按 [check-python-requirements.txt](../infra/images/check-python-requirements.txt) 固定版本和哈希安装。缺了它们,这些测试会失败或等到超时。 ## 需要操作者完成 @@ -146,7 +142,7 @@ WishForge 在集群里只有一个身份 `wishforge/wishforge`,它的短期令 kubectl --kubeconfig ~/.config/droboticflow/kubeconfigs/poc.config -n wishforge delete pod wishforge-0 ``` -3. **授权启动与迁移**:Claude Code 的自动权限检查可能拦截 `apply` 和 `migrate`,这时在交互式 `claude` 终端中为这些命令加权限规则,或者自己在终端里执行下面的命令。 +3. **授权启动**:Claude Code 的自动权限检查可能拦截 `apply`,这时在交互式 `claude` 终端中为这些命令加权限规则,或者自己在终端里执行下面的命令。 ## 命令顺序 @@ -162,12 +158,6 @@ node scripts/cluster-poc.mjs secrets 容器只在启动时读取设置文件(其中也写着检查 Pod 使用的镜像)。`apply` 每次都会先刷新这些 Secret 和 ConfigMap,内容有变化时 Pod 模板上的摘要随之改变,Pod 自动按新设置重建;重新构建镜像后运行 `apply` 即可,平台和检查 Pod 会用同一个镜像。 -演练迁移:复制数据库快照、Git、POC 构建记录和对象存储,Temporal 为空,Runner 保持空闲,Mac 照常服务: - -```bash -node scripts/cluster-poc.mjs migrate --rehearsal -``` - ```bash node scripts/cluster-poc.mjs status ``` @@ -178,7 +168,7 @@ node scripts/cluster-poc.mjs status kubectl --kubeconfig ~/.config/droboticflow/kubeconfigs/poc.config -n wishforge port-forward pod/wishforge-0 13000:3000 ``` -保持它运行,在浏览器打开 `http://127.0.0.1:13000`。网页只接受登记过的来源地址,本机端口必须是 13000(Mac 上自己的控制台占着 3000)。演练副本只用来验收登录和浏览:不要在上面提愿望(有直接合入权限的账号会推送到 `poc`,而 Mac 也在服务),正式迁移时它的数据会被 Mac 的数据覆盖。 +保持它运行,在浏览器打开 `http://127.0.0.1:13000`。网页只接受登记过的来源地址,本机端口必须是 13000。 集群版管理员密码在 Pod 的私有文件里,复制到剪贴板(不打印): @@ -210,10 +200,7 @@ node scripts/cluster-poc.mjs public --http ## 备份与恢复 -集群里没有可跨节点挂载的存储(没有 StorageClass),备份分两层: - -- **节点上**:Pod 里的 `backup` 容器每天 03:20(北京时间)取一份快照,写到数据盘的 `backups//`,保留最近 7 份。Pod 启动时若最新快照超过一天,会在约一分钟后补一份;失败一小时后重试。 -- **Mac 上**:`backup` 把最新快照复制到 `.wishforge/cluster-poc/backups//`(0700,保留 14 份),逐个文件核对大小和 SHA-256。`backup --schedule` 安装 launchd 任务,每天 10:17 自动执行(Mac 睡眠时在唤醒后补跑),日志在 `.wishforge/cluster-poc/backup.log`;Pod 最近一次快照失败或最新快照超过 36 小时,命令在复制后报错退出;由 launchd 运行时,任何失败都会弹出 macOS 通知「WishForge 集群备份失败」。 +Pod 里的 `backup` 容器每天 03:20(北京时间)取一份快照,写到数据盘的 `backups//`,保留最近 7 份。Pod 启动时若最新快照超过一天,会在约一分钟后补一份;失败一小时后重试。`status` 显示节点上的快照、最新时间和最近一次结果。 一份快照包括: @@ -221,77 +208,38 @@ node scripts/cluster-poc.mjs public --http |---|---| | `postgres.dump` | `pg_dump -Fc`。与逐表行数取自同一个数据库快照(`pg_export_snapshot`),行数记在 `manifest.json`,恢复后据此核对 | | `temporal.db` | Temporal 的 SQLite,经 SQLite 在线备份复制并通过 `integrity_check` | -| `git.tgz`、`minio.tgz`、`poc-state.tgz`、`runner-state.tgz`、`platform-state.tgz` | 对应数据目录;`platform-state` 不含可重建的 `home` 缓存,`minio` 不含 MinIO 启动时会清理的 `.minio.sys/tmp` | +| `git.tgz`、`minio.tgz`、`poc-state.tgz`、`runner-state.tgz`、`runner-state-fanyu.tgz`、`platform-state.tgz` | 对应数据目录;`platform-state` 不含可重建的 `home` 缓存,`minio` 不含 MinIO 启动时会清理的 `.minio.sys/tmp` | | `manifest.json` | 时间、镜像、行数,以及每个文件的大小和 SHA-256 | -不包括运行工作区和检查缓存(`exec/`)、Claude 的家目录、Caddy 的证书(会自动重新签发)。快照里有加密保存的凭据和管理员密码文件;解密这些凭据要用 Mac 上 `.env.local` 里的 `WF_VAULT_KEY` 等密钥,恢复也依赖 `.wishforge/cluster-poc/generated.json`,这些不在快照里,要和快照一起保管好。 +不包括运行工作区和检查缓存(`exec/`)、Claude 的家目录、Caddy 的证书(会自动重新签发)。解密快照里的凭据需要 `carried.env` 里的 `WF_VAULT_KEY`,见「数据与密钥」。 `backup` 容器只读挂载它要保存的目录,只写 `backups/`;通过回环连接数据库(需要数据库密码),没有集群令牌、设置文件或密钥。镜像带有与数据库同版本的 `pg_dump`、`pg_restore`(从固定的 postgres 镜像复制)。 -立即取一份快照并复制到 Mac: +恢复:用节点上的一份快照替换集群里的全部数据(数据库、Temporal、Git、对象存储,以及平台、POC 和两个 Runner 的记录),全程在集群内完成: + +1. 停掉 WishForge Pod,起一个只挂数据卷的恢复 Pod; +2. 先按 `manifest.json` 逐个核对快照文件的大小和 SHA-256,核对不过就什么都不动、把 Pod 启动回来; +3. 清空并解包各数据目录,放回 Temporal 的数据库; +4. 维护模式启动 Pod(只有数据库、Temporal、对象存储在跑),由 `backup` 容器经回环把 `postgres.dump` 导入全新的 PostgreSQL; +5. 逐表行数与快照记录一致才正常启动。节点上的快照保留不动。 ```bash -node scripts/cluster-poc.mjs backup --now +node scripts/cluster-poc.mjs restore --runner ``` -每天自动复制(取消用 `--unschedule`): - -```bash -node scripts/cluster-poc.mjs backup --schedule -``` - -`status` 会显示节点上的快照数、最新时间和最近一次结果。 - -恢复:用 Mac 上的一份快照替换集群里的全部数据(数据库、Temporal、Git、对象存储,以及平台、POC 和 Runner 的记录),流程与 `migrate` 相同——停 Pod、清空并解包目录、维护模式下导入数据库、逐表行数与快照记录一致才启动。节点上已有的快照保留。正式切换后要让 Runner 工作,加 `--runner`: - -```bash -node scripts/cluster-poc.mjs restore .wishforge/cluster-poc/backups/ -``` - -2026-11-01 在演练副本上的验收(集群空闲、快照后无新写入): - -| 项目 | 结果 | -|---|---| -| 部署后 `backup` 容器约一分钟内自动取第一份快照(29.2 MB);`backup --now` 再取一份,均复制到 Mac 并逐文件核对通过 | 通过 | -| launchd 任务手动触发一次,退出码 0 | 通过 | -| `restore` 最新快照:约 3 分钟,逐表行数一致后启动;Git、POC 记录、Runner 记录、管理员凭据文件逐字节一致,MinIO 对象一致;登录页 200,未登录 API 401 | 通过 | -| 首次 `restore` 在上传时遇到集群连接超时 | 当时 Pod 停在 0 副本,已手动恢复,数据未动;随后改为每次上传核对 SHA-256 并重试、上传失败时自动重启 Pod | - -这台 Mac 到集群的连接会丢掉 `kubectl exec` 输出流的末尾,下载速度约 150 kB/s。所以 `backup` 逐个文件下载:容器端发完文件后等 Mac 关闭输入才退出,Mac 收齐全部字节后才关闭。一份快照大约要 4 分钟。 - -节点故障时,在新节点上执行 `bootstrap`(先把 [storage.yaml](../infra/k8s/cluster-poc/storage.yaml) 里的节点改成新节点)、`apply`,再 `restore`。 - -## 正式切换 - -切换窗口约 30 分钟,期间停止写入: - -1. Mac 上确认没有排队、执行中的运行,没有就绪或部署中的发布(`migrate` 会检查,不满足时拒绝执行)。 -2. 停止 Mac 服务,确认 `stopCompleted=true`: - - ```bash - node scripts/local-poc-service.mjs stop - ``` - -3. 正式迁移:会短暂停止 Mac 上 WishForge 的 MinIO 和 Temporal 容器以取得一致副本(之后重新启动作为回退);导入后逐表比对行数,一致才启动;Runner 开始工作: - - ```bash - node scripts/cluster-poc.mjs migrate --final - ``` - -4. 用个人账号登录新地址验收,然后停用 Cloudflare 临时链接。Mac 上的数据保留不动。 +不加 `--runner` 时两个 Runner 保持空闲,不会部署任何东西。 ## 回退 -- 切换前任何一步失败:Mac 照常服务,集群里的副本可以随时用 `migrate` 重新覆盖。 -- 切换后、集群尚无新写入:`kubectl -n wishforge scale statefulset wishforge --replicas=0`,再在 Mac 上 `node scripts/local-poc-service.mjs start`。 -- 集群已有新写入:先缩容冻结集群,把数据反向导回 Mac 后再启动 Mac 服务。导回的项目检查命令是镜像里的路径,启动前在 Mac 上运行 `node --import tsx scripts/setup-local-poc.ts`,把它们恢复成 Mac 的路径。 +- 数据出错:用出错前的快照 `restore`。 +- 新镜像有问题:把 `.wishforge/cluster-poc/image.json` 换回上一个镜像的记录后 `apply`。 - 删除 `wishforge-public` 会回收公网 IP,删除后在百度云控制台核对 BLB 和 EIP 已经释放。 ## 已知限制 - IDE(Theia、OCT)没有迁移,集群版网页上的 IDE 入口不可用。 - 检查 Pod 有网络(见上文)。 -- 数据在单节点本地盘上。节点上的快照和数据在同一块盘上,节点损坏时只能用 Mac 上的副本恢复;Mac 每天复制一次,所以最多丢失约一天的数据,且 Mac 长期关机期间没有异地副本。 +- **数据和快照都在同一个节点的同一块本地盘上**。快照能防误删、误改和软件故障,但节点或磁盘损坏时数据和快照会一起丢失;集群里没有可跨节点挂载的存储(没有 StorageClass),目前没有异地副本。 - Temporal 使用开发服务,Web UI 已关闭。 - 模型名取自网关的 `/v1/models`(2026-10-30):`deepseek-pro@latest`、`deepseek-flash@latest` 是网关的别名,都已用 Anthropic 协议实测可用;`[1m]` 后缀让 Claude CLI 按 1M 上下文处理。网关文档里推荐的 `deepseek-v4-flash` 不在列表中。 diff --git a/docs/local-browser-access.md b/docs/local-browser-access.md index cbf0621..602f0e1 100644 --- a/docs/local-browser-access.md +++ b/docs/local-browser-access.md @@ -1,5 +1,7 @@ # 本机工作空间的协作者访问 +> **已停用(2026-11-02)**:WishForge 只运行在 POC 集群,Mac 本机部署及其数据已删除。现行部署、备份与恢复见 [cluster-poc-deployment.md](cluster-poc-deployment.md);本文仅作历史记录。 + 本机模式面向获准的可信操作者。网页可监听局域网地址;控制 API、数据库、Temporal、Provisioner 等服务仍监听本机回环地址。平台在服务端验证本机管理员或个人账号登录后,以对应身份调用控制 API;个人账号还携带账号 ID 和密码版本,由控制面按当前权限授权。浏览器收不到控制 API 凭据。 ## 配置与启动 @@ -37,7 +39,7 @@ npm run local:network -- --address 192.168.1.20 --apply ## 登录与退出 -协作者通过局域网或经批准的 HTTPS 入口访问,使用管理员在「成员与权限」页创建的个人账号登录,并只授予所需项目与操作权限。不要共享本机管理员账号。登录会话为加密、HttpOnly、SameSite=Strict 的 cookie,1 小时到期;HTTPS 入口额外设置 Secure,退出会清除会话 cookie。公网入口的配置、限制与验收见 [local-public-entry.md](local-public-entry.md)。 +协作者通过局域网或经批准的 HTTPS 入口访问,使用管理员在「成员与权限」页创建的个人账号登录,并只授予所需项目与操作权限。不要共享本机管理员账号。登录会话为加密、HttpOnly、SameSite=Strict 的 cookie,1 小时到期;HTTPS 入口额外设置 Secure,退出会清除会话 cookie。 本机管理员密码是 256 位随机值;个人账号使用各自的随机初始密码。登录限流用于限制 scrypt 校验开销。15 分钟内错误超过 8 次后,密码校验在全局范围内每 2 秒最多一次,而不是锁定 15 分钟,局域网中其他设备输错密码不会把管理员锁在外面。限流不信任客户端提供的转发 IP,状态存在当前 Web 进程内,进程重启会清空。更换凭据文件(尤其 passwordHash 或 sessionSecret)会使旧会话失效。所有写入请求必须携带与访问入口完全一致的 Origin;未在白名单的 Host 或跨站请求被拒绝。 diff --git a/docs/local-poc-operations.md b/docs/local-poc-operations.md index 2e64c2d..8743f58 100644 --- a/docs/local-poc-operations.md +++ b/docs/local-poc-operations.md @@ -1,8 +1,8 @@ # 本机许愿服务运行说明 -本机运行 WishForge 网页、控制面、Claude CLI 和发布 Runner。用户通过网页登录后提交愿望;项目固定接入 Gitea 的 `longtao.wu/droboticflow` 仓库 `poc` 分支,目标环境为既有 POC。Claude 修改独立工作区,受保护检查通过后由交付流程处理 Git 提交和发布。 +> **已停用(2026-11-02)**:WishForge 只运行在 POC 集群,Mac 本机部署及其数据已删除。现行部署、备份与恢复见 [cluster-poc-deployment.md](cluster-poc-deployment.md);本文仅作历史记录。 -迁到 POC 集群运行的部署、迁移与回退见 [cluster-poc-deployment.md](cluster-poc-deployment.md);切换完成前,本文描述的 Mac 服务仍是正式入口,也是迁移的数据来源。 +本机运行 WishForge 网页、控制面、Claude CLI 和发布 Runner。用户通过网页登录后提交愿望;项目固定接入 Gitea 的 `longtao.wu/droboticflow` 仓库 `poc` 分支,目标环境为既有 POC。Claude 修改独立工作区,受保护检查通过后由交付流程处理 Git 提交和发布。 ## 后台运行 @@ -26,7 +26,7 @@ node scripts/local-poc-service.mjs start ## 入口与登录 -本机入口为 `http://localhost:3000`;同一可达局域网的其他设备使用 `http://本机局域网IP:3000`。是否跨网可达由现有路由和防火墙决定。当前部署另有经批准的 Cloudflare 临时 HTTPS 隧道,地址以维护者提供的链接和私有入口白名单为准;它只代理 Web,不公开内部 API 或 IDE。百度云独立公网 IP 入口的实现与上线闸门见 [local-public-entry.md](local-public-entry.md),不能把清单已生成当成公网入口已开通。 +本机入口为 `http://localhost:3000`;同一可达局域网的其他设备使用 `http://本机局域网IP:3000`。是否跨网可达由现有路由和防火墙决定。当前部署另有经批准的 Cloudflare 临时 HTTPS 隧道,地址以维护者提供的链接和私有入口白名单为准;它只代理 Web,不公开内部 API 或 IDE。 本机管理员的登录凭据保存在 `.wishforge/local-access.json`,只允许本机当前用户读取(0600)。文件含用户名、密码和会话签名材料,不应上传到 Gitea、放入网页静态目录或复制到日志。文件不存在时可运行 `node scripts/setup-local-access.mjs` 创建;重复运行保留已有凭据。`.env.local` 的 `WF_LOCAL_ACCESS_FILE` 指向该文件的绝对路径。 diff --git a/docs/local-public-entry.md b/docs/local-public-entry.md deleted file mode 100644 index bbf36da..0000000 --- a/docs/local-public-entry.md +++ /dev/null @@ -1,151 +0,0 @@ -# 本机 WishForge 的公网入口 - -## 当前状态(2026-10-29) - -本机已有 Cloudflare 临时 HTTPS 隧道,最终地址以维护者提供的链接及私有 `.env.local` 中的 `WF_LOCAL_ALLOWED_ORIGINS` 为准。它只转发 Web,不直接公开 API、数据库或 IDE。隧道停止、Mac 休眠或断网会中断访问。 - -**百度云独立公网 IP 入口尚未开通。** 本次完成了脚本、配置、镜像及真实二进制校验,但在目标集群的 NetworkPolicy 验收处停止: - -- 指定的测试端口可以正常访问,正向控制通过。 -- 另一个本应被默认拒绝的入站端口仍可访问;等待策略传播并重复检查 8 次仍为 HTTP 200。 -- 删除临时出站放行策略后,出站连接仍可访问;重复检查 8 次仍为 HTTP 200。 -- 因此不能把“已创建 NetworkPolicy 对象”当作网络隔离已生效。 - -没有创建新的 BLB/EIP,没有改动原 `droboticflow-api-public`,没有追加新 origin,没有安装或启动本机 Caddy/frpc LaunchAgent。网关 Deployment、Pod、ConfigMap、专用隧道 Secret 和网络策略已撤回。`wishforge-access` 命名空间保留:CCE 自动同步了镜像拉取凭据,清理程序没有擅自删除这些自动管理对象;没有运行中的工作负载或公网 Service。 - -后续需要先由运维确认并启用目标 CCE 网络模式支持的 NetworkPolicy 执行能力,然后重新验收;或者选择具备独立网络边界的中转环境。**不能为完成公网访问而跳过此闸门,或擅自修改整个 POC 的网络组件。** - -## 设计边界 - -平台和数据继续留在 Mac。集群只运行单副本 FRP 服务端和回环代理校验进程;浏览器 HTTPS 经 TCP 隧道直达 Mac 的 Caddy,再反代 `127.0.0.1:3000`。 - -| 公网端口 | 网关端口 | Mac 回环端口 | 用途 | -|---|---|---|---| -| 80/TCP | 18080 | 18080 | ACME HTTP-01;普通 GET/HEAD 跳转到标准 HTTPS 端口,其他方法拒绝 | -| 443/TCP | 18443 | 18443 | 用户 HTTPS | -| 7000/TCP | 7000 | frpc 主动连接 | 双向 TLS 加随机令牌的隧道控制及工作连接 | - -不使用 sslip.io,不新增云主机、磁盘或数据库,不把 4000、4300 等内部端口加入隧道。只有在下述验收通过后,才可以申请独立的 BLB/EIP。 - -### 隧道与权限 - -- FRP 固定 v0.71.1,容器镜像引用固定摘要。 -- 使用专用私有 CA 双向 TLS,客户端明确校验服务器名称 `wishforge-entry.internal`;这个名称仅是私有传输证书的身份,不需要公网 DNS。 -- 服务端只允许 18080/18443。回环 `NewProxy` 钩子进一步限定两个固定名称的 TCP 代理,拒绝 STCP、UDP、其他端口及分组代理;钩子不可用时注册失败。 -- 一个副本,Deployment 使用 `Recreate`,避免两个 FRP 实例分担同一个客户端的控制连接与用户连接。 -- 就绪检查针对 7000 和策略进程,不依赖客户端注册后才出现的代理端口,避免启动死锁。 -- 非 root、禁止提权、只读根文件系统、无 ServiceAccount 令牌、无宿主机挂载、无 Docker socket。 -- 命名空间默认拒绝出入网络;服务端不需要主动访问业务 Pod、数据库或 Kubernetes API。 - -## 文件与本机状态 - -- `infra/k8s/local-entry/kustomization.yaml`:非公网网关资源;**不包含**收费的公网 Service。 -- `infra/k8s/local-entry/public-service.yaml`:单独的新 LoadBalancer 清单,不能指向旧业务的 LB ID 或 selector。 -- `scripts/configure-local-entry.mjs`:精确 origin 的预览、追加、按回执撤销,以及固定 Caddy/frpc 配置生成。 -- `scripts/local-entry-service.mjs`:仅管理自己的 Caddy/frpc 进程,不改变原 platform/runner 服务组。 - -私有状态位于已被 Git 忽略的 `.wishforge/local-entry/`:目录 0700,秘密文件 0600。 - -- `bin/`:核对官方摘要后的 Caddy、frpc。 -- `mtls/`:专用 CA、传输证书及令牌。CA 私钥和客户端私钥不上传集群;只把服务端需要的材料放到专用 Secret。 -- `caddy-staging/`、`caddy-production/`:独立的 ACME 账户与证书状态,不得混用。 -- `receipts/`:环境文件私有备份及本次追加 origin 的回执。 -- `logs/`:独立日志,按文件轮转;不启用 HTTP 请求访问日志或调试级认证日志。 -- `network-verification.json`:本次集群网络隔离实测结果。`false` 不能当作可发布状态。 - -所有启动脚本在 import 时都不会启动服务;ConfigMap 挂载脚本需按真实路径识别入口,不能用未解析的 `argv` 路径直接比较。 - -## 重新实施前必须满足 - -1. 集群网络隔离重新实测通过:有正向对照,禁止的入站端口被拦截,撤销允许规则后出站也被拦截。仅检查 API 对象或网络插件名称不够。 -2. 入口对象名和命名空间 UID 经检查仍属于本任务;恢复时不能覆盖他人后来创建的对象。 -3. 新资源仅引用该入口镜像、Secret 与 selector;不修改 DroboticFlow 或其他业务资源。 -4. 用户已经确认新增 LB/EIP 费用,且未改变申请范围。 - -之后才能单独创建公网 Service。其配置显式选择 `small1`、标准 BGP、按流量计费及 5 Mbps 初始峰值上限,避免默认的 100 Mbps。**5 Mbps 不是账单封顶;Mac 离线时 LB/EIP 仍可能计费。** - -## 新 IP 分配后的配置流程 - -以下命令只在前置条件通过、维护者取得本次专属新 IP 后执行;占位符不是实际可用地址。 - -先生成 staging 配置,不修改 Web origin: - -```bash -node scripts/configure-local-entry.mjs --config <新公网IPv4> --staging -``` - -验证所需私有证书文件及 Caddy/frpc 配置: - -```bash -node scripts/local-entry-service.mjs validate -``` - -安装独立用户服务(需本机登录的 macOS 用户): - -```bash -node scripts/local-entry-service.mjs install -``` - -staging 仅用于确认 ACME 验证路径,不能作为浏览器信任验收。验证通过后,停止入口服务、确认 `stopCompleted=true`,重新生成不带 `--staging` 的配置,再启动并检查生产证书: - -```bash -node scripts/local-entry-service.mjs stop -``` - -```bash -node scripts/configure-local-entry.mjs --config <新公网IPv4> -``` - -```bash -node scripts/local-entry-service.mjs start -``` - -Caddy 显式使用 Let's Encrypt `shortlived` profile,不使用内部 CA。公网 IP 证书需要自动续签,状态目录必须保留。`default_sni` 使没有 SNI 的 IP 访问也能选中证书;不得用跳过 TLS 校验的请求代替验收。首次签发成功不等于已经观察到自然续签成功。 - -只在可信 HTTPS 已通过验证后,预览并添加该 origin: - -```bash -node scripts/configure-local-entry.mjs --address <新公网IPv4> -``` - -```bash -node scripts/configure-local-entry.mjs --address <新公网IPv4> --apply -``` - -工具只追加 `https://<新IP>`,保留 localhost、LAN、Cloudflare 及其他原有入口;不打印环境文件内容、不改凭据。随后在所有愿望、构建与发布空闲且 Runner 为新鲜 idle 时,按原运维流程重启本机服务,使新 origin 生效。`doctor.ready` 或 `/usage.activeRuns=0` 单独不足以证明安全空闲。 - -## 验收与运维 - -```bash -node scripts/local-entry-service.mjs status -``` - -检查可信 IP 证书、证书到期时间、HTTPS `/login`、未登录 API 拒绝、错误 Host/Origin 拒绝、HTTP 重定向不泄露本机端口,以及原 POC `/api/health` 和备用入口。 - -登录后的 Cookie、写入、SSE、WebSocket 检查必须使用已获授权的真实会话;不得读取管理员密码、伪造会话或擅自创建测试账号。没有验证的项目须明确标注,不得把匿名登录页可访问描述成完整业务验收。 - -服务由独立 LaunchAgent `cc.d-robotics.wishforge.local-entry` 管理,只随当前用户登录运行。不改变系统休眠设置;合盖、休眠和网络中断仍会影响访问和续签。 - -停止入口不会停止 WishForge: - -```bash -node scripts/local-entry-service.mjs stop -``` - -按回执撤销本次新增的 origin,不整份覆盖旧环境文件: - -```bash -node scripts/configure-local-entry.mjs --revert <回执文件> --apply -``` - -删除本次公网 Service 前必须确认它的 UID 和关联的 LB/EIP 属于本任务。删除会使地址失效,且地址可能无法恢复;需核对云端实际资源回收情况,不能仅凭 Kubernetes 对象消失就认定停止计费。命名空间中出现其他人或系统管理的额外对象时,不直接整体删除。 - -## 离线检查 - -```bash -node --test scripts/configure-local-entry.test.mjs scripts/local-entry-service.test.mjs -``` - -```bash -kubectl kustomize infra/k8s/local-entry -``` diff --git a/infra/k8s/cluster-poc/platform.yaml b/infra/k8s/cluster-poc/platform.yaml index d9ee958..ac0e546 100644 --- a/infra/k8s/cluster-poc/platform.yaml +++ b/infra/k8s/cluster-poc/platform.yaml @@ -226,9 +226,9 @@ spec: - { name: data, mountPath: /var/lib/wishforge/runner-state, subPath: runner-state-fanyu } - { name: data, mountPath: /var/lib/wishforge/poc-state, subPath: poc-state } - { name: runner-fanyu-tmp, mountPath: /tmp } - # Daily snapshots of the data onto this node's disk (scripts/cluster-poc-backup.mjs); the - # operator Mac copies them off the node (cluster-poc.mjs backup). It reads the data it saves - # read-only and the database through loopback; it has no token, settings or keys. + # Daily snapshots of the data onto this node's disk (scripts/cluster-poc-backup.mjs), which + # cluster-poc.mjs restore puts back. It reads the data it saves read-only and the database + # through loopback; it has no token, settings or keys. - name: backup image: ${WISHFORGE_IMAGE} args: [backup] @@ -247,6 +247,7 @@ spec: - { name: data, mountPath: /mnt/data/platform-state, subPath: platform-state, readOnly: true } - { name: data, mountPath: /mnt/data/poc-state, subPath: poc-state, readOnly: true } - { name: data, mountPath: /mnt/data/runner-state, subPath: runner-state, readOnly: true } + - { name: data, mountPath: /mnt/data/runner-state-fanyu, subPath: runner-state-fanyu, readOnly: true } - { name: data, mountPath: /mnt/data/temporal, subPath: temporal, readOnly: true } - { name: backup-tmp, mountPath: /tmp } - name: caddy diff --git a/infra/k8s/local-entry/frps.toml b/infra/k8s/local-entry/frps.toml deleted file mode 100644 index 2d06326..0000000 --- a/infra/k8s/local-entry/frps.toml +++ /dev/null @@ -1,21 +0,0 @@ -bindAddr = "0.0.0.0" -bindPort = 7000 -proxyBindAddr = "0.0.0.0" -allowPorts = [{ single = 18080 }, { single = 18443 }] -maxPortsPerClient = 2 -transport.tls.force = true -transport.tls.certFile = "/run/entry-secret/server.crt" -transport.tls.keyFile = "/run/entry-secret/server.key" -transport.tls.trustedCaFile = "/run/entry-secret/ca.crt" -auth.method = "token" -auth.additionalScopes = ["HeartBeats", "NewWorkConns"] -auth.tokenSource.type = "file" -auth.tokenSource.file.path = "/run/entry-secret/token" -log.to = "console" -log.level = "info" -log.disablePrintColor = true -[[httpPlugins]] -name = "wishforge-web-only" -addr = "127.0.0.1:19000" -path = "/validate" -ops = ["NewProxy"] diff --git a/infra/k8s/local-entry/gateway.yaml b/infra/k8s/local-entry/gateway.yaml deleted file mode 100644 index 9b94816..0000000 --- a/infra/k8s/local-entry/gateway.yaml +++ /dev/null @@ -1,123 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: wishforge-access - labels: - app.kubernetes.io/part-of: wishforge-local-entry - pod-security.kubernetes.io/enforce: restricted - pod-security.kubernetes.io/enforce-version: v1.34 ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: local-entry - namespace: wishforge-access - labels: - app.kubernetes.io/part-of: wishforge-local-entry -spec: - replicas: 1 - strategy: - type: Recreate - selector: - matchLabels: - app.kubernetes.io/name: wishforge-local-entry - template: - metadata: - labels: - app.kubernetes.io/name: wishforge-local-entry - app.kubernetes.io/part-of: wishforge-local-entry - spec: - automountServiceAccountToken: false - terminationGracePeriodSeconds: 20 - nodeSelector: - kubernetes.io/os: linux - kubernetes.io/arch: amd64 - securityContext: - runAsNonRoot: true - runAsUser: 10001 - runAsGroup: 10001 - fsGroup: 10001 - seccompProfile: - type: RuntimeDefault - containers: - - name: frps - image: docker.io/fatedier/frps:v0.71.1@sha256:48720c81f048e842f7f0b8053753c1a8514053d59d2dee8a953f67995de73a18 - args: ['-c', '/etc/entry/frps.toml'] - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: [ALL] - resources: - requests: { cpu: 100m, memory: 64Mi } - limits: { cpu: 500m, memory: 256Mi } - ports: - - { name: tunnel, containerPort: 7000 } - - { name: web-http, containerPort: 18080 } - - { name: web-https, containerPort: 18443 } - readinessProbe: - tcpSocket: { port: tunnel } - periodSeconds: 5 - livenessProbe: - tcpSocket: { port: tunnel } - initialDelaySeconds: 15 - periodSeconds: 15 - volumeMounts: - - { name: config, mountPath: /etc/entry, readOnly: true } - - { name: credentials, mountPath: /run/entry-secret, readOnly: true } - - name: proxy-policy - image: docker.io/library/node:24-alpine@sha256:8a32c8cf649a49915a66e81e7a609223c708d0adabf288ee3f4077d765542f85 - command: ['node', '/etc/entry/proxy-policy.mjs'] - securityContext: - allowPrivilegeEscalation: false - readOnlyRootFilesystem: true - capabilities: - drop: [ALL] - resources: - requests: { cpu: 25m, memory: 32Mi } - limits: { cpu: 100m, memory: 128Mi } - readinessProbe: - exec: - command: ['node', '/etc/entry/proxy-policy.mjs', '--check'] - timeoutSeconds: 3 - periodSeconds: 10 - livenessProbe: - exec: - command: ['node', '/etc/entry/proxy-policy.mjs', '--check'] - timeoutSeconds: 3 - initialDelaySeconds: 15 - periodSeconds: 20 - volumeMounts: - - { name: config, mountPath: /etc/entry, readOnly: true } - volumes: - - name: config - configMap: { name: entry-config } - - name: credentials - secret: - secretName: entry-mtls - defaultMode: 0440 ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: default-deny - namespace: wishforge-access -spec: - podSelector: {} - policyTypes: [Ingress, Egress] ---- -apiVersion: networking.k8s.io/v1 -kind: NetworkPolicy -metadata: - name: entry-ports - namespace: wishforge-access -spec: - podSelector: - matchLabels: - app.kubernetes.io/name: wishforge-local-entry - policyTypes: [Ingress] - ingress: - - ports: - - { protocol: TCP, port: 7000 } - - { protocol: TCP, port: 18080 } - - { protocol: TCP, port: 18443 } diff --git a/infra/k8s/local-entry/kustomization.yaml b/infra/k8s/local-entry/kustomization.yaml deleted file mode 100644 index 9ac06c0..0000000 --- a/infra/k8s/local-entry/kustomization.yaml +++ /dev/null @@ -1,10 +0,0 @@ -apiVersion: kustomize.config.k8s.io/v1beta1 -kind: Kustomization -namespace: wishforge-access -resources: - - gateway.yaml -configMapGenerator: - - name: entry-config - files: - - frps.toml - - proxy-policy.mjs diff --git a/infra/k8s/local-entry/proxy-policy.mjs b/infra/k8s/local-entry/proxy-policy.mjs deleted file mode 100644 index ff86da6..0000000 --- a/infra/k8s/local-entry/proxy-policy.mjs +++ /dev/null @@ -1,76 +0,0 @@ -import http from 'node:http'; -import { realpathSync } from 'node:fs'; -import { pathToFileURL } from 'node:url'; - -export const POLICY_PORT = 19000; -const ports = new Map([ - ['wishforge-http', 18080], - ['wishforge-https', 18443], -]); - -/** FRP's NewProxy hook: only the two named web transports, never another proxy type. */ -export function proxyDecision(request) { - const proxy = request?.content; - const allowed = - request?.op === 'NewProxy' && - proxy?.proxy_type === 'tcp' && - ports.has(proxy.proxy_name) && - ports.get(proxy.proxy_name) === proxy.remote_port && - !proxy.group && - !proxy.group_key && - !proxy.custom_domains?.length && - !proxy.subdomain && - !proxy.multiplexer; - return allowed - ? { reject: false, unchange: true } - : { reject: true, reject_reason: 'Only the configured WishForge HTTP and HTTPS TCP proxies are allowed' }; -} - -/** Loopback only; no request bodies or authentication material enter logs. */ -export function policyServer() { - return http.createServer({ requestTimeout: 5000, headersTimeout: 5000, maxHeaderSize: 8192 }, async (req, res) => { - const reply = (status, body) => { - res.writeHead(status, { 'Content-Type': 'application/json', 'Cache-Control': 'no-store' }); - res.end(JSON.stringify(body)); - }; - if (req.method === 'GET' && req.url === '/healthz') return reply(200, { ready: true }); - if (req.method !== 'POST' || req.url?.split('?')[0] !== '/validate') - return reply(403, { reject: true, reject_reason: 'Unsupported operation' }); - try { - const chunks = []; - let size = 0; - for await (const chunk of req) { - size += chunk.length; - if (size > 16384) { - reply(413, { reject: true, reject_reason: 'Request too large' }); - req.destroy(); - return; - } - chunks.push(chunk); - } - reply(200, proxyDecision(JSON.parse(Buffer.concat(chunks).toString('utf8')))); - } catch { - if (!res.headersSent) reply(400, { reject: true, reject_reason: 'Invalid request' }); - } - }); -} - -/** ConfigMap mounts use symlinks; Node resolves the main module but keeps the original argv path. */ -export function isPolicyMain(moduleUrl, script = process.argv[1]) { - try { - return Boolean(script && moduleUrl === pathToFileURL(realpathSync(script)).href); - } catch { - return false; - } -} - -if (isPolicyMain(import.meta.url)) { - if (process.argv[2] === '--check') { - const response = await fetch(`http://127.0.0.1:${POLICY_PORT}/healthz`, { signal: AbortSignal.timeout(1500) }); - if (!response.ok || !(await response.json()).ready) process.exitCode = 1; - } else { - const server = policyServer(); - server.listen(POLICY_PORT, '127.0.0.1'); - for (const signal of ['SIGTERM', 'SIGINT']) process.once(signal, () => server.close()); - } -} diff --git a/infra/k8s/local-entry/public-service.yaml b/infra/k8s/local-entry/public-service.yaml deleted file mode 100644 index 21564db..0000000 --- a/infra/k8s/local-entry/public-service.yaml +++ /dev/null @@ -1,25 +0,0 @@ -# Apply separately, only after the isolated gateway and NetworkPolicy checks pass. -# This creates a NEW billable BLB/EIP. It never references an existing business load balancer. -apiVersion: v1 -kind: Service -metadata: - name: local-entry-public - namespace: wishforge-access - labels: - app.kubernetes.io/part-of: wishforge-local-entry - annotations: - service.beta.kubernetes.io/cce-load-balancer-lb-name: wishforge-access-local-entry - service.beta.kubernetes.io/cce-load-balancer-internal-vpc: 'false' - service.beta.kubernetes.io/cce-elastic-ip-route-type: BGP - service.beta.kubernetes.io/cce-elastic-ip-billing-method: ByTraffic - service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps: '5' - service.beta.kubernetes.io/blb-performance-level: small1 -spec: - type: LoadBalancer - externalTrafficPolicy: Cluster - selector: - app.kubernetes.io/name: wishforge-local-entry - ports: - - { name: http, protocol: TCP, port: 80, targetPort: web-http } - - { name: https, protocol: TCP, port: 443, targetPort: web-https } - - { name: tunnel, protocol: TCP, port: 7000, targetPort: tunnel } diff --git a/scripts/cluster-poc-backup.mjs b/scripts/cluster-poc-backup.mjs index c1386ce..8f090a5 100644 --- a/scripts/cluster-poc-backup.mjs +++ b/scripts/cluster-poc-backup.mjs @@ -1,11 +1,12 @@ #!/usr/bin/env node // Daily snapshots of WishForge's data in the POC cluster (docs/cluster-poc-deployment.md), taken by -// the WishForge Pod's backup container onto the node's disk; `cluster-poc.mjs backup` copies the -// newest one off the node to the operator Mac. Importing it starts nothing. +// the WishForge Pod's backup container onto the node's disk; `cluster-poc.mjs restore` puts one +// back. Importing it starts nothing. // // node scripts/cluster-poc-backup.mjs take one snapshot a day, until Kubernetes stops the container // node scripts/cluster-poc-backup.mjs once take one snapshot now // node scripts/cluster-poc-backup.mjs latest print the newest complete snapshot's manifest and the last result +// node scripts/cluster-poc-backup.mjs verify DIR check a snapshot's files against its manifest import { spawn } from 'node:child_process'; import { createHash } from 'node:crypto'; import { createReadStream } from 'node:fs'; @@ -20,10 +21,10 @@ export const DATA_ROOT = '/mnt/data'; /** The database beside it on the Pod's loopback; the password comes from PGPASSWORD. */ const DATABASE = { host: '127.0.0.1', port: 5432, user: 'wishforge', database: 'wishforge' }; /** The directories saved as archives, beside the database dump and Temporal's database. */ -export const ARCHIVES = ['git', 'minio', 'platform-state', 'poc-state', 'runner-state']; +export const ARCHIVES = ['git', 'minio', 'platform-state', 'poc-state', 'runner-state', 'runner-state-fanyu']; /** Not saved: the platform container's caches, rebuilt when it starts, and MinIO's scratch space and trash. */ const ARCHIVE_EXCLUDES = { 'platform-state': ['./home'], minio: ['./.minio.sys/tmp'] }; -/** Snapshots kept on the node; the operator Mac keeps its own copies longer. */ +/** Snapshots kept on the node. */ export const KEEP = 7; /** 03:20 in Beijing, when nobody is expected to be working. */ export const SCHEDULE = { hourUtc: 19, minute: 20 }; @@ -287,11 +288,17 @@ export async function main(argv = process.argv.slice(2)) { await once(BACKUP_ROOT); return 0; } + if (argv[0] === 'verify' && argv.length === 2) { + const manifest = JSON.parse(await readFile(path.join(argv[1], 'manifest.json'), 'utf8')); + await verifyFiles(argv[1], manifest.files); + console.log(`[backup] snapshot ${manifest.name}: every file matches its manifest`); + return 0; + } if (argv[0] === 'latest' && argv.length === 1) { process.stdout.write(`${JSON.stringify(await latest())}\n`); return 0; } - throw new Error('Usage: cluster-poc-backup.mjs [once|latest]'); + throw new Error('Usage: cluster-poc-backup.mjs [once|latest|verify DIR]'); } if (process.argv[1] && pathToFileURL(path.resolve(process.argv[1])).href === import.meta.url) diff --git a/scripts/cluster-poc-backup.test.mjs b/scripts/cluster-poc-backup.test.mjs index 54ca146..9bd2132 100644 --- a/scripts/cluster-poc-backup.test.mjs +++ b/scripts/cluster-poc-backup.test.mjs @@ -1,11 +1,10 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; -import { execFileSync } from 'node:child_process'; import { mkdir, mkdtemp, readdir, rm, stat, writeFile } from 'node:fs/promises'; import os from 'node:os'; import path from 'node:path'; import { nextRun, retention, snapshotName, takeSnapshot, verifyFiles, sha256File } from './cluster-poc-backup.mjs'; -import { failureNotification, launchAgentPlist } from './cluster-poc.mjs'; +import { restoreScript } from './cluster-poc.mjs'; test('snapshots are taken daily at 03:20 Beijing time and named by their UTC time', () => { assert.equal(nextRun(new Date('2026-11-01T10:00:00Z')).toISOString(), '2026-11-01T19:20:00.000Z'); @@ -22,7 +21,7 @@ test('the newest complete snapshots are kept; older ones and leftover partial on assert.deepEqual(retention(names.slice(0, 1), 7).remove, []); }); -test('a copied snapshot is accepted only when every file has its recorded size and SHA-256', async t => { +test('a snapshot is restored only when every file has its recorded size and SHA-256', async t => { const directory = await mkdtemp(path.join(os.tmpdir(), 'wf-backup-verify-')); t.after(() => rm(directory, { recursive: true, force: true })); await writeFile(path.join(directory, 'git.tgz'), 'archive'); @@ -47,36 +46,16 @@ test('a snapshot that fails leaves no partial snapshot or lock behind, and a hel assert.ok(await stat(path.join(root, '.lock'))); }); -test('the Mac copies the newest snapshot every day through a launchd job it can read back', async t => { - const plist = launchAgentPlist({ - node: '/opt/homebrew/bin/node', - script: '/repo/scripts/cluster-poc.mjs', - root: '/repo & co', - pathEnv: '/opt/homebrew/bin:/usr/bin', - home: '/Users/someone', - log: '/repo/.wishforge/cluster-poc/backup.log', - }); - assert.match(plist, /Hour<\/key>10<\/integer>/); - assert.match(plist, /backup<\/string>/); - assert.match(plist, /\/repo & co/); - if (process.platform !== 'darwin') return; - const directory = await mkdtemp(path.join(os.tmpdir(), 'wf-backup-plist-')); - t.after(() => rm(directory, { recursive: true, force: true })); - const file = path.join(directory, 'job.plist'); - await writeFile(file, plist); - const parsed = JSON.parse(execFileSync('plutil', ['-convert', 'json', '-o', '-', file], { encoding: 'utf8' })); - assert.deepEqual(parsed.ProgramArguments, ['/opt/homebrew/bin/node', '/repo/scripts/cluster-poc.mjs', 'backup']); - assert.equal(parsed.WorkingDirectory, '/repo & co'); - assert.deepEqual(parsed.StartCalendarInterval, { Hour: 10, Minute: 17 }); - assert.equal(parsed.EnvironmentVariables.WF_BACKUP_NOTIFY, '1'); -}); - -test('a failing daily copy is announced in a notification whose text cannot break out of its quotes', () => { - const script = failureNotification('Backup needs attention: "x" \\ end\nsecond line'); - assert.equal( - script, - 'display notification "Backup needs attention: \\"x\\" \\\\ end second line" with title "WishForge 集群备份失败" sound name "Basso"', - ); - if (process.platform === 'darwin') - execFileSync('osacompile', ['-o', path.join(os.tmpdir(), `wf-notify-${process.pid}.scpt`), '-e', script]); +test('a restore checks the snapshot before it empties anything, and unpacks only what the snapshot holds', () => { + const script = restoreScript('20261102T192000Z', ['git', 'minio']).split('\n'); + const verify = script.findIndex(line => line.includes('cluster-poc-backup.mjs verify')); + const firstReset = script.findIndex(line => line.includes('-mindepth 1 -delete')); + assert.ok(verify >= 0 && verify < firstReset); + assert.ok(script.indexOf('touch /tmp/verified') < firstReset); + assert.ok(script.some(line => line.startsWith("mkdir -p 'runner-state-fanyu'"))); + assert.ok(!script.some(line => line.includes("-C 'runner-state-fanyu'"))); + assert.ok(!script.some(line => /backups'? &&/.test(line) && line.includes('-delete')), 'snapshots are never emptied'); + assert.equal(script.at(-2), 'touch /tmp/restored'); + assert.throws(() => restoreScript('../x'), /Not a snapshot/); + assert.throws(() => restoreScript('20261102T192000Z', ['postgres']), /Unknown archive/); }); diff --git a/scripts/cluster-poc.mjs b/scripts/cluster-poc.mjs index b899ca1..38975cd 100644 --- a/scripts/cluster-poc.mjs +++ b/scripts/cluster-poc.mjs @@ -1,6 +1,7 @@ #!/usr/bin/env node // Operator CLI of WishForge's deployment in the POC cluster (docs/cluster-poc-deployment.md). // Every command names its cluster explicitly and prints no secret. Importing it starts nothing. +// WishForge runs in the cluster only; this machine keeps the operator's inputs (STATE) and nothing else. // // node scripts/cluster-poc.mjs image build and push the image in the cluster (Kaniko) // node scripts/cluster-poc.mjs bootstrap namespaces, storage, access rules @@ -8,30 +9,38 @@ // node scripts/cluster-poc.mjs gateway-key [--stdin] the model gateway key, from the clipboard or stdin // node scripts/cluster-poc.mjs secrets Secrets and ConfigMaps from this Mac's settings // node scripts/cluster-poc.mjs apply [--maintenance] [--runner] the WishForge Pod -// node scripts/cluster-poc.mjs migrate --rehearsal|--final copy this Mac's data into it // node scripts/cluster-poc.mjs public [--production] the public HTTPS entry (billed load balancer) // node scripts/cluster-poc.mjs fanyu-access --kubeconfig WishForge's identity in Fanyu's cluster // node scripts/cluster-poc.mjs projects --replace delete every project; create the POC and Fanyu projects -// node scripts/cluster-poc.mjs backup [--now] copy the newest daily snapshot to this Mac -// node scripts/cluster-poc.mjs backup --schedule do that every day (launchd) -// node scripts/cluster-poc.mjs restore [--runner] replace the cluster's data with a copied snapshot +// node scripts/cluster-poc.mjs restore [--runner] replace the cluster's data with a node snapshot // node scripts/cluster-poc.mjs status show the deployment's state import { spawn } from 'node:child_process'; import { createHash, randomBytes } from 'node:crypto'; import { createReadStream, createWriteStream } from 'node:fs'; -import { chmod, mkdir, mkdtemp, readFile, readdir, rename, rm, stat, writeFile } from 'node:fs/promises'; +import { chmod, mkdir, mkdtemp, readFile, rename, rm, stat, writeFile } from 'node:fs/promises'; import os from 'node:os'; import path from 'node:path'; import { fileURLToPath, pathToFileURL } from 'node:url'; import { Resolver, getServers, resolve4 } from 'node:dns/promises'; +import { isIP } from 'node:net'; import { setTimeout as delay } from 'node:timers/promises'; -import { publicIPv4 } from './configure-local-entry.mjs'; -import { ARCHIVES, COUNTS_SQL, retention, verifyFiles } from './cluster-poc-backup.mjs'; +import { ARCHIVES, COUNTS_SQL } from './cluster-poc-backup.mjs'; export const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); export const STATE = path.join(ROOT, '.wishforge/cluster-poc'); export const KUBECONFIG = path.join(os.homedir(), '.config/droboticflow/kubeconfigs/poc.config'); -export const POC_PROFILE = path.join(os.homedir(), '.config/wishforge/poc-profile.json'); +/** + * The operator's inputs to the cluster's settings, kept private here: the keys and tokens the data + * depends on (carried.env, see CARRIED), the Runners' signing key, the plan catalog, the POC's + * profile and Gitea's pinned host keys. + */ +export const INPUTS = { + carried: path.join(STATE, 'carried.env'), + signingKey: path.join(STATE, 'runner-ed25519.pem'), + plans: path.join(STATE, 'plans.json'), + knownHosts: path.join(STATE, 'known_hosts'), +}; +export const POC_PROFILE = path.join(STATE, 'poc-profile.json'); export const CLUSTER = 'https://120.48.181.105:6443'; export const NAMESPACE = 'wishforge'; export const CHECK_NAMESPACE = 'wishforge-checks'; @@ -82,31 +91,6 @@ function capture(executable, args, options = {}) { child.stdin.end(options.input); }); } -/** A command whose stdin comes from `from` or whose stdout goes to `to`: binary-safe, never in memory. */ -function pipeFile(executable, args, { from, to, env }) { - return new Promise((resolve, reject) => { - const child = spawn(executable, args, { - cwd: ROOT, - env, - stdio: [from ? 'pipe' : 'ignore', to ? 'pipe' : 'ignore', 'pipe'], - }); - const err = []; - child.stderr.on('data', chunk => err.push(chunk)); - child.once('error', reject); - if (from) createReadStream(from).on('error', reject).pipe(child.stdin); - const written = to - ? new Promise((done, fail) => - child.stdout - .pipe(createWriteStream(to, { mode: 0o600 })) - .on('finish', done) - .on('error', fail), - ) - : Promise.resolve(); - child.once('close', code => - written.then(() => resolve({ code, stderr: Buffer.concat(err).toString('utf8') }), reject), - ); - }); -} export async function kubectl(args, { input, timeoutSeconds = 60, allowFailure = false } = {}) { const result = await capture( 'kubectl', @@ -384,6 +368,25 @@ const DOMAIN = /^(?=.{1,253}$)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2 const PUBLIC_ORIGIN = new RegExp(`^https?://(?:\\d{1,3}(?:\\.\\d{1,3}){3}|${DOMAIN.source.slice(1, -1)})$`); /** Public resolvers that show what the internet, Let's Encrypt included, sees of a domain. */ const PUBLIC_RESOLVERS = ['223.5.5.5', '119.29.29.29']; +/** A publicly routable IPv4 address: never a URL, private, shared or documentation address. */ +export function publicIPv4(value) { + if (isIP(value) !== 4) throw new Error('A public IPv4 address is required'); + const [a, b, c] = value.split('.').map(Number); + if ( + a === 0 || + a === 10 || + a === 127 || + a >= 224 || + (a === 100 && b >= 64 && b <= 127) || + (a === 169 && b === 254) || + (a === 172 && b >= 16 && b <= 31) || + (a === 192 && (b === 168 || b === 0 || (b === 88 && c === 99))) || + (a === 198 && (b === 18 || b === 19 || (b === 51 && c === 100))) || + (a === 203 && b === 0 && c === 113) + ) + throw new Error('The entry must use a publicly routable IPv4 address'); + return value; +} /** A domain name for the public entry, as the operator's DNS points it at the load balancer. */ export function publicDomain(name) { const domain = String(name ?? '') @@ -397,7 +400,7 @@ export const PORT_FORWARD_ORIGIN = 'http://127.0.0.1:13000'; /** The settings files of the platform, executor and Runner containers; see {@link CARRIED}. */ export function clusterEnvironments({ local, runner, runnerFanyu, image, generated, origins = [] }) { const missing = [ - ...CARRIED.filter(key => !local[key]).map(key => `.env.local ${key}`), + ...CARRIED.filter(key => !local[key]).map(key => `carried.env ${key}`), ...RUNNER_CARRIED.filter(key => !runner[key]).map(key => `runner.env ${key}`), ...(runnerFanyu ? RUNNER_CARRIED.filter(key => !runnerFanyu[key]).map(key => `Fanyu's Runner ${key}`) : []), ]; @@ -500,19 +503,6 @@ export function envFile(values) { return `${lines.join('\n')}\n`; } -/** The POC files as the cluster's containers use them: the same bindings, at their own paths. */ -export function clusterPocFiles({ profile, binding }) { - return { - profile: { ...profile, kubeconfig: `${PATHS.config}/kubeconfig` }, - binding: { - ...binding, - profileFile: `${PATHS.pocState}/poc-profile.json`, - stateDirectory: `${PATHS.pocState}/poc`, - gitHome: PATHS.gitHome, - }, - }; -} - /** WishForge's identity at the POC cluster's address, by its short-lived projected token. */ export function clusterKubeconfig(certificateAuthorityData) { if (!/^[A-Za-z0-9+/=]+$/.test(certificateAuthorityData)) throw new Error('Invalid cluster CA data'); @@ -806,28 +796,15 @@ async function gatewayKey(flag) { console.log(`The gateway key is in the cluster (Secret wishforge-claude)${flag ? '.' : '; clear your clipboard.'}`); } -/** - * The projects the cluster delivers and their Runners: once `projects` replaced them, the - * cluster's own (STATE/projects.json); before, the Mac's POC project, migrated with its data. - */ -async function clusterProjects(local) { +/** The projects the cluster delivers and their Runners (projects --replace). */ +async function clusterProjects() { const pocProfile = { ...JSON.parse(await readFile(POC_PROFILE, 'utf8')), kubeconfig: `${PATHS.config}/kubeconfig` }; const fanyuProfile = await readFile(FANYU_PROFILE, 'utf8').then(JSON.parse, () => undefined); const profiles = { 'poc-profile.json': pocProfile, ...(fanyuProfile ? { 'fanyu-profile.json': fanyuProfile } : {}), }; - const replaced = await readFile(PROJECTS_FILE, 'utf8').then(JSON.parse, () => undefined); - if (!replaced) { - const runner = await readEnv(path.join(ROOT, '.wishforge/local-poc/runner.env')); - const binding = JSON.parse(await readFile(local.WF_LOCAL_POC_BINDING_FILE, 'utf8')); - return { - profiles, - bindings: clusterPocFiles({ profile: pocProfile, binding }).binding, - runner, - keys: { 'runner-public.json': await readFile(runner.WF_RUNNER_PUBLIC_KEYS, 'utf8') }, - }; - } + const replaced = JSON.parse(await readFile(PROJECTS_FILE, 'utf8')); for (const binding of replaced.bindings) if (!profiles[path.basename(binding.profileFile)]) throw new Error(`No ${path.basename(binding.profileFile)} for project ${binding.projectId}; run fanyu-access`); @@ -853,8 +830,8 @@ async function clusterProjects(local) { async function secrets() { await assertCluster(); - const local = await readEnv(path.join(ROOT, '.env.local')); - const projects = await clusterProjects(local); + const local = await readEnv(INPUTS.carried); + const projects = await clusterProjects(); const { runner, runnerFanyu } = projects; const image = (await imageRecord()).reference; const generated = await generatedSecrets(); @@ -874,10 +851,8 @@ async function secrets() { 'jsonpath={.clusters[0].cluster.certificate-authority-data}', ]) ).stdout.trim(); - const hosts = (await capture('ssh-keygen', ['-F', 'gitea.d-robotics.cc'])).stdout - .split('\n') - .filter(line => line && !line.startsWith('#')); - if (!hosts.length) throw new Error('No pinned host key for gitea.d-robotics.cc in ~/.ssh/known_hosts'); + const hosts = (await readFile(INPUTS.knownHosts, 'utf8')).split('\n').filter(line => line && !line.startsWith('#')); + if (!hosts.length) throw new Error(`No pinned host key for gitea.d-robotics.cc in ${INPUTS.knownHosts}`); // The Pod mounts the Claude settings; until gateway-key stores the key, they name the gateway only. if (!(await exists('secret', 'wishforge-claude', NAMESPACE))) { await applyObjects([ @@ -894,9 +869,9 @@ async function secrets() { 'runner.env': envFile(environments.runner), ...(environments.runnerFanyu ? { 'runner-fanyu.env': envFile(environments.runnerFanyu) } : {}), }), - secret('wishforge-keys', { 'runner-ed25519.pem': await readFile(local.WF_RUNNER_SIGNING_KEY_FILE, 'utf8') }), + secret('wishforge-keys', { 'runner-ed25519.pem': await readFile(INPUTS.signingKey, 'utf8') }), configMap('wishforge-config', { - 'plans.json': await readFile(local.WF_PLAN_CATALOG_FILE, 'utf8'), + 'plans.json': await readFile(INPUTS.plans, 'utf8'), ...projects.keys, ...Object.fromEntries( Object.entries(projects.profiles).map(([name, profile]) => [name, `${JSON.stringify(profile, null, 2)}\n`]), @@ -914,8 +889,9 @@ async function secrets() { }), ]; await applyObjects(objects); - console.log('Secrets and ConfigMaps are in place: three settings files, keys, POC files and the Caddyfile.'); - return createHash('sha256').update(JSON.stringify(objects)).digest('hex').slice(0, 32); + const digest = createHash('sha256').update(JSON.stringify(objects)).digest('hex').slice(0, 32); + console.log(`Secrets and ConfigMaps are in place: settings, keys, POC files and the Caddyfile (settings ${digest}).`); + return digest; } async function apply(...flags) { @@ -978,33 +954,13 @@ async function status() { } } -// ------------------------------------------------------------------------------ migration +// ------------------------------------------------------------------------------- restore -const MAC = { - postgres: 'wishforge-dev-postgres-1', - minio: 'wishforge-dev-minio-1', - temporal: 'wishforge-dev-temporal-1', -}; -const docker = (args, options = {}) => - capture('docker', ['--context', 'colima', ...args], { - env: { PATH: process.env.PATH, HOME: os.homedir() }, - ...options, - }); -const macSql = async sql => { - const result = await docker(['exec', MAC.postgres, 'psql', '-U', 'wishforge', '-d', 'wishforge', '-At', '-c', sql]); - if (result.code !== 0) throw new Error(`Reading the Mac database failed: ${result.stderr.trim()}`); - return result.stdout.trim(); -}; -/** What the Mac is still doing: a migration starts only when nothing runs, deploys or is dispatched. */ +/** What the cluster is still doing: projects are replaced and data restored only when nothing runs. */ export const UNFINISHED_SQL = `select kind || ' ' || id || ' ' || coalesce(data->>'status', '') from entities where (kind = 'run' and data->>'status' in ('queued', 'running')) or (kind = 'runner_task' and data->>'status' in ('pending', 'dispatching', 'unknown')) or (kind = 'release' and data->>'status' in ('ready', 'deploying', 'unknown'))`; - -async function macService() { - const result = await capture(process.execPath, [path.join(ROOT, 'scripts/local-poc-service.mjs'), 'status']); - return JSON.parse(result.stdout); -} async function waitFor(description, check, minutes = 10) { for (const until = Date.now() + minutes * 60_000; Date.now() < until; await delay(5000)) if (await check()) return; throw new Error(`Timed out waiting for ${description}`); @@ -1014,7 +970,7 @@ const podGone = async name => await kubectl(['-n', NAMESPACE, 'get', 'pod', name, '--ignore-not-found', '-o', 'name'], { allowFailure: true }) ).stdout.trim() === ''; -/** A restricted Pod with the data volume, to put files in place while the WishForge Pod is stopped. */ +/** A restricted Pod with the data volume, to put a snapshot's files in place while the WishForge Pod is stopped. */ const restorePod = image => ({ apiVersion: 'v1', kind: 'Pod', @@ -1057,77 +1013,70 @@ const restorePod = image => ({ }, }); +const SNAPSHOT = /^\d{8}T\d{6}Z$/; /** - * Replaces the cluster's data, with the WishForge Pod stopped: each tarball's directory, the - * PostgreSQL and Temporal directories and the run workspaces are emptied first. The database is - * then restored from `dump` into a fresh PostgreSQL started with nothing else running, and the Pod - * starts only if every table's row count is `counts`. + * Puts snapshot `name`'s files in place, in the restore Pod: each file is checked against the + * manifest first, and only then are the directories emptied and unpacked. `archives` are the + * directories the snapshot holds; the others are emptied. /tmp/verified and /tmp/restored mark how + * far it got, since kubectl can lose an exec's status. */ -async function replaceData({ image, tarballs, dump, counts, runner }) { - // ---- put the files in place while the WishForge Pod is stopped +export function restoreScript(name, archives = ARCHIVES) { + if (!SNAPSHOT.test(name)) throw new Error(`Not a snapshot name: ${name}`); + if (archives.some(archive => !ARCHIVES.includes(archive))) throw new Error('Unknown archive'); + const snapshot = `/mnt/data/backups/${name}`; + const reset = ['postgres', 'temporal', 'exec/runs', ...ARCHIVES]; + return [ + 'set -e', + `node /opt/wishforge/scripts/cluster-poc-backup.mjs verify '${snapshot}'`, + 'touch /tmp/verified', + 'cd /mnt/data', + ...reset.map(dir => `mkdir -p '${dir}' && find '${dir}' -mindepth 1 -delete && chmod 0700 '${dir}'`), + ...archives.map(dir => `tar -xzf '${snapshot}/${dir}.tgz' -C '${dir}' --no-same-owner --no-same-permissions`), + `cp '${snapshot}/temporal.db' temporal/temporal.db && chmod 0600 temporal/temporal.db`, + 'touch /tmp/restored', + 'echo restored: $(find minio -type f | wc -l) object storage files, $(find git -type f | wc -l) Git files', + ].join('\n'); +} + +/** + * Replaces the cluster's data with a snapshot on the node, with the WishForge Pod stopped. The + * database is then restored into a fresh PostgreSQL started with nothing else running, by the + * backup container, which reads the snapshot and reaches the database on loopback; the Pod starts + * only if every table's row count is `counts`. + */ +async function replaceData({ image, snapshot, archives, counts, runner }) { + const script = restoreScript(snapshot, archives); if (await exists('statefulset', 'wishforge', NAMESPACE)) { await kubectl(['-n', NAMESPACE, 'scale', 'statefulset', 'wishforge', '--replicas=0']); await waitFor('the WishForge Pod to stop', () => podGone('wishforge-0')); } await kubectl(['-n', NAMESPACE, 'delete', 'pod', 'wishforge-restore', '--ignore-not-found', '--wait=true']); await kubectl(['create', '-f', '-'], { input: JSON.stringify(restorePod(image)) }); - const inRestorePod = (script, options) => - kubectl(['-n', NAMESPACE, 'exec', 'wishforge-restore', '-c', 'restore', '--', '/bin/sh', '-c', script], options); - // This Mac's link to the cluster drops connections: each upload is checked, and tried again. - const upload = async file => { - const target = `/tmp/${path.basename(file)}`; - const sum = await sha256File(file); - for (let attempt = 1; ; attempt++) { - await kubectl(['-n', NAMESPACE, 'cp', file, `wishforge-restore:${target}`, '-c', 'restore'], { - timeoutSeconds: 600, - allowFailure: true, - }); - const check = await inRestorePod(`sha256sum '${target}' 2>/dev/null`, { allowFailure: true }); - if (check.stdout.startsWith(sum)) return; - if (attempt === 5) throw new Error(`Uploading ${path.basename(file)} failed ${attempt} times`); - await delay(3000 * attempt); - } - }; - let changed = false; + const inRestorePod = (command, options) => + kubectl(['-n', NAMESPACE, 'exec', 'wishforge-restore', '-c', 'restore', '--', '/bin/sh', '-c', command], options); try { await kubectl(['-n', NAMESPACE, 'wait', '--for=condition=Ready', 'pod/wishforge-restore', '--timeout=600s'], { timeoutSeconds: 620, }); - const reset = [...new Set(['postgres', 'temporal', 'exec/runs', ...Object.keys(tarballs)])]; - const script = [ - 'set -e', - 'cd /mnt/data', - ...reset.map(dir => `mkdir -p '${dir}' && find '${dir}' -mindepth 1 -delete && chmod 0700 '${dir}'`), - ...Object.keys(tarballs).map( - dir => `tar -xzf '/tmp/${dir.replaceAll('/', '-')}.tgz' -C '${dir}' --no-same-owner --no-same-permissions`, - ), - 'touch /tmp/restored', - 'echo restored: $(find minio -type f | wc -l) object storage files, $(find git -type f | wc -l) Git files', - ].join('\n'); - for (const file of Object.values(tarballs)) await upload(file); - changed = true; - // kubectl can lose an exec's status: the script's own mark says whether it finished. - const restored = await inRestorePod(script, { timeoutSeconds: 600, allowFailure: true }); - const finished = await inRestorePod('test -f /tmp/restored && echo finished', { allowFailure: true }); - if (finished.stdout.trim() !== 'finished') + const restored = await inRestorePod(script, { timeoutSeconds: 1200, allowFailure: true }); + const marks = (await inRestorePod('ls /tmp', { allowFailure: true })).stdout; + if (!/\brestored\b/.test(marks)) { + if (/\bverified\b/.test(marks)) + throw new Error( + `Unpacking snapshot ${snapshot} did not finish (${restored.stderr.trim().slice(0, 400)}); the Pod stays stopped, run this again`, + ); + await kubectl(['-n', NAMESPACE, 'scale', 'statefulset', 'wishforge', '--replicas=1'], { allowFailure: true }); throw new Error( - `Unpacking the data did not finish (${restored.stderr.trim().slice(0, 400)}); the Pod stays stopped, run this again`, + `Snapshot ${snapshot} did not pass its check (${restored.stderr.trim().slice(0, 400)}); nothing was changed and the WishForge Pod is starting again`, ); - console.log(restored.stdout.trim() || 'Data unpacked.'); - } catch (error) { - if (changed) throw error; - await kubectl(['-n', NAMESPACE, 'delete', 'pod', 'wishforge-restore', '--ignore-not-found', '--wait=true'], { - allowFailure: true, - }); - await kubectl(['-n', NAMESPACE, 'scale', 'statefulset', 'wishforge', '--replicas=1'], { allowFailure: true }); - throw new Error(`${error.message}\nNothing was changed; the WishForge Pod is starting again.`); + } + console.log(restored.stdout.trim() || 'Files restored.'); } finally { await kubectl(['-n', NAMESPACE, 'delete', 'pod', 'wishforge-restore', '--ignore-not-found', '--wait=true'], { allowFailure: true, }); } - // ---- the database, into a fresh PostgreSQL started with nothing else running await apply('--maintenance'); await kubectl(['-n', NAMESPACE, 'scale', 'statefulset', 'wishforge', '--replicas=1']); await waitFor('PostgreSQL in the cluster', async () => { @@ -1148,26 +1097,16 @@ async function replaceData({ image, tarballs, dump, counts, runner }) { '-d', 'wishforge', ], - { - allowFailure: true, - }, + { allowFailure: true }, ); return ready.code === 0; }); - const restored = await pipeFile( - 'kubectl', - [ - '--kubeconfig', - KUBECONFIG, - '-n', - NAMESPACE, - 'exec', - '-i', - 'wishforge-0', - '-c', - 'postgres', - '--', + // kubectl can lose an exec's status: the row counts say whether the restore took. + const restored = await kubectl( + backupExec([ 'pg_restore', + '-h', + '127.0.0.1', '-U', 'wishforge', '-d', @@ -1175,10 +1114,10 @@ async function replaceData({ image, tarballs, dump, counts, runner }) { '--no-owner', '--no-privileges', '--exit-on-error', - ], - { from: dump, env: { PATH: process.env.PATH, HOME: os.homedir() } }, + `/var/lib/wishforge/backups/${snapshot}/postgres.dump`, + ]), + { timeoutSeconds: 1800, allowFailure: true }, ); - if (restored.code !== 0) throw new Error(`pg_restore failed: ${restored.stderr.slice(0, 2000)}`); const clusterCounts = ( await kubectl([ '-n', @@ -1199,104 +1138,13 @@ async function replaceData({ image, tarballs, dump, counts, runner }) { ]) ).stdout.trim(); if (clusterCounts !== counts) - throw new Error(`Row counts differ after the restore:\nSource: ${counts}\nCluster: ${clusterCounts}`); + throw new Error( + `Row counts differ after the restore (${restored.stderr.trim().slice(0, 800)}):\nSnapshot: ${counts}\nCluster: ${clusterCounts}`, + ); console.log(`Database restored; row counts match: ${counts}`); await apply(...(runner ? ['--runner'] : [])); } -/** - * Copies the Mac's WishForge into the cluster. --rehearsal copies a snapshot while the Mac keeps - * serving: the database, Git, POC records and object storage, but an empty Temporal and a Runner - * that stays idle, so the copy can act on nothing. --final copies everything after the Mac's - * service is stopped, including Temporal, and starts the Runner. Either replaces what the cluster - * held; Claude homes and the cluster's administrator credentials are kept. - */ -async function migrate(mode) { - if (!['--rehearsal', '--final'].includes(mode)) throw new Error('Usage: cluster-poc.mjs migrate --rehearsal|--final'); - const final = mode === '--final'; - await assertCluster(); - const image = (await imageRecord()).reference; - const unfinished = await macSql(UNFINISHED_SQL); - if (unfinished) throw new Error(`The Mac is still working; wait until these finish:\n${unfinished}`); - if (final && (await macService()).state !== 'stopped') - throw new Error('Stop the Mac service first (node scripts/local-poc-service.mjs stop)'); - const directory = await mkdtemp(path.join(os.tmpdir(), 'wishforge-migrate-')); - await chmod(directory, 0o700); - try { - // ---- export from the Mac - // The Mac may still write during a rehearsal: a dump counts only when no row changed around it. - const dump = path.join(directory, 'wishforge.dump'); - let macCounts; - for (let attempt = 1; !macCounts; attempt++) { - const before = await macSql(COUNTS_SQL); - const dumped = await pipeFile( - 'docker', - [ - '--context', - 'colima', - 'exec', - MAC.postgres, - 'pg_dump', - '-U', - 'wishforge', - '-d', - 'wishforge', - '-Fc', - '--no-owner', - '--no-privileges', - ], - { to: dump, env: { PATH: process.env.PATH, HOME: os.homedir() } }, - ); - if (dumped.code !== 0) throw new Error(`pg_dump failed: ${dumped.stderr}`); - if ((await macSql(COUNTS_SQL)) === before) macCounts = before; - else if (attempt === 5) - throw new Error('The Mac database kept changing during the dump; try again when it is quiet'); - } - const stopped = []; - try { - for (const service of final ? ['minio', 'temporal'] : []) { - const result = await docker(['stop', MAC[service]]); - if (result.code !== 0) throw new Error(`Stopping ${MAC[service]} failed: ${result.stderr}`); - stopped.push(service); - } - for (const [service, source] of [['minio', '/data'], ...(final ? [['temporal', '/var/lib/temporal']] : [])]) { - const copy = await docker(['cp', `${MAC[service]}:${source}/.`, path.join(directory, service)]); - if (copy.code !== 0) throw new Error(`Copying ${service} data failed: ${copy.stderr}`); - } - } finally { - for (const service of stopped) await docker(['start', MAC[service]]); - } - const archives = { - git: path.join(ROOT, '.wishforge/git'), - 'runner-state': path.join(ROOT, '.wishforge/local-poc/runner-state'), - 'poc-state/poc': path.join(os.homedir(), '.local/share/wishforge/poc'), - minio: path.join(directory, 'minio'), - ...(final ? { temporal: path.join(directory, 'temporal') } : {}), - }; - const tarballs = {}; - for (const [target, source] of Object.entries(archives)) { - const file = path.join(directory, `${target.replaceAll('/', '-')}.tgz`); - const made = await capture('tar', ['--no-mac-metadata', '-czf', file, '-C', source, '.'], { - env: { PATH: process.env.PATH, COPYFILE_DISABLE: '1' }, - }); - if (made.code !== 0) throw new Error(`Packing ${target} failed: ${made.stderr}`); - tarballs[target] = file; - } - console.log( - `Exported from the Mac: database (${((await stat(dump)).size / 1e6).toFixed(1)} MB), ${Object.keys(tarballs).join(', ')}.`, - ); - - await replaceData({ image, tarballs, dump, counts: macCounts, runner: final }); - console.log( - final - ? 'Migration complete; the cluster now serves WishForge.' - : 'Rehearsal copy is running; its Runner stays idle.', - ); - } finally { - await rm(directory, { recursive: true, force: true }); - } -} - // ------------------------------------------------------------------------------ public entry /** The load balancer's public IPv4, once CCE has assigned it; private addresses it also lists are skipped. */ @@ -1373,18 +1221,11 @@ async function publicEntry(...args) { ); } -// ---------------------------------------------------------------------------------- backups +// ------------------------------------------------------------------------------- snapshots -const BACKUPS = path.join(STATE, 'backups'); -/** Snapshots this Mac keeps; the node keeps a week (cluster-poc-backup.mjs). */ -export const LOCAL_KEEP = 14; -/** A snapshot this much older than the schedule says the Pod's backup is not working. */ -const STALE_HOURS = 36; -export const LAUNCH_AGENT = 'cc.d-robotics.wishforge.cluster-backup'; -const LAUNCH_AGENT_FILE = path.join(os.homedir(), 'Library/LaunchAgents', `${LAUNCH_AGENT}.plist`); const backupExec = args => ['-n', NAMESPACE, 'exec', 'wishforge-0', '-c', 'backup', '--', ...args]; /** - * Copies one file out of the backup container. kubectl drops whatever of an exec's output is + * Copies one file out of one of the WishForge Pod's containers. kubectl drops whatever of an exec's output is * still in flight when the remote command exits, so the remote side sends the file and then waits * for this end to close its input, which happens only once every byte has arrived. */ @@ -1440,168 +1281,41 @@ async function download(remote, local, bytes, container = 'backup') { } } -const xml = value => String(value).replace(/[<>&'"]/g, character => `&#${character.charCodeAt(0)};`); /** - * The launchd job that runs `backup` every day at 10:17, after the Pod's 03:20 snapshot; a Mac - * asleep then runs it when it wakes. + * Replaces everything the cluster holds with one of the node's daily snapshots (`status` lists + * them): the database, Temporal, Git, object storage and the platform's, POC's and Runners' + * records. Starts the Runners only with --runner. The node keeps its snapshots. */ -export function launchAgentPlist({ node, script, root, pathEnv, home, log, hour = 10, minute = 17 }) { - const string = value => `${xml(value)}`; - return ` - - - - Label${string(LAUNCH_AGENT)} - ProgramArguments - ${string(node)}${string(script)}${string('backup')} - WorkingDirectory${string(root)} - EnvironmentVariables - PATH${string(pathEnv)}HOME${string(home)}WF_BACKUP_NOTIFY${string('1')} - StartCalendarInterval - Hour${hour}Minute${minute} - StandardOutPath${string(log)} - StandardErrorPath${string(log)} - - -`; -} - -async function scheduleBackup(enable) { - const domain = `gui/${process.getuid()}`; - await capture('launchctl', ['bootout', `${domain}/${LAUNCH_AGENT}`]); - if (!enable) { - await rm(LAUNCH_AGENT_FILE, { force: true }); - console.log('The daily copy of the cluster backup is no longer scheduled.'); - return; - } - await privateState(); - const log = path.join(STATE, 'backup.log'); - await mkdir(path.dirname(LAUNCH_AGENT_FILE), { recursive: true }); - await writeFile( - LAUNCH_AGENT_FILE, - launchAgentPlist({ - node: process.execPath, - script: path.join(ROOT, 'scripts/cluster-poc.mjs'), - root: ROOT, - pathEnv: process.env.PATH, - home: os.homedir(), - log, - }), - { mode: 0o644 }, - ); - const loaded = await capture('launchctl', ['bootstrap', domain, LAUNCH_AGENT_FILE]); - if (loaded.code !== 0) throw new Error(`launchctl bootstrap failed: ${loaded.stderr.trim()}`); - console.log(`Every day at 10:17 this Mac copies the newest cluster snapshot to ${BACKUPS}; log: ${log}`); -} - -/** - * Copies the newest snapshot the Pod's backup container took (with --now, one it takes first) to - * this Mac, checks every file against its manifest, and keeps {@link LOCAL_KEEP}. Fails, after - * copying, when the Pod's last attempt failed or its newest snapshot is stale. - */ -/** A macOS notification, so a failing daily copy run by launchd does not go unnoticed in its log. */ -export function failureNotification(message) { - const quote = value => `"${String(value).replace(/\s+/g, ' ').slice(0, 200).replace(/["\\]/g, '\\$&')}"`; - return `display notification ${quote(message)} with title ${quote('WishForge 集群备份失败')} sound name "Basso"`; -} - -async function backup(...flags) { - try { - await copyBackup(...flags); - } catch (error) { - if (process.env.WF_BACKUP_NOTIFY === '1') await capture('osascript', ['-e', failureNotification(error.message)]); - throw error; - } -} - -async function copyBackup(...flags) { - if (flags.length === 1 && ['--schedule', '--unschedule'].includes(flags[0])) - return scheduleBackup(flags[0] === '--schedule'); - if (flags.some(flag => flag !== '--now')) - throw new Error('Usage: cluster-poc.mjs backup [--now|--schedule|--unschedule]'); - await assertCluster(); - const startedAt = Date.now(); - // The snapshot's own record says how it went: kubectl can lose an exec's last output and status. - if (flags.includes('--now')) - await kubectl(backupExec(['node', 'scripts/cluster-poc-backup.mjs', 'once']), { - timeoutSeconds: 3600, - allowFailure: true, - }); - const { manifest, last } = JSON.parse( - (await kubectl(backupExec(['node', 'scripts/cluster-poc-backup.mjs', 'latest']))).stdout, - ); - if (flags.includes('--now') && !(last?.ok && Date.parse(last.at) >= startedAt)) - throw new Error(`The snapshot failed${last && !last.ok ? `: ${last.error}` : ''}`); - if (!manifest) - throw new Error(`The cluster has no snapshot yet${last?.error ? `; the last attempt failed: ${last.error}` : ''}`); - await privateState(); - await mkdir(BACKUPS, { recursive: true, mode: 0o700 }); - const target = path.join(BACKUPS, manifest.name); - if (await stat(path.join(target, 'manifest.json')).catch(() => undefined)) - console.log(`Snapshot ${manifest.name} is already on this Mac.`); - else { - const partial = path.join(BACKUPS, `.partial-${manifest.name}`); - await rm(partial, { recursive: true, force: true }); - await mkdir(partial, { mode: 0o700 }); - try { - for (const [file, { bytes }] of Object.entries(manifest.files)) - await download(`/var/lib/wishforge/backups/${manifest.name}/${file}`, path.join(partial, file), bytes); - await writeFile(path.join(partial, 'manifest.json'), `${JSON.stringify(manifest, null, 2)}\n`, { mode: 0o600 }); - await verifyFiles(partial, manifest.files); - await rename(partial, target); - } catch (error) { - await rm(partial, { recursive: true, force: true }); - throw error; - } - const bytes = Object.values(manifest.files).reduce((sum, file) => sum + file.bytes, 0); - console.log(`Copied snapshot ${manifest.name} (${(bytes / 1e6).toFixed(1)} MB, verified) to ${target}`); - } - for (const entry of retention(await readdir(BACKUPS), LOCAL_KEEP).remove) - await rm(path.join(BACKUPS, entry), { recursive: true, force: true }); - const problems = []; - if (last && !last.ok) problems.push(`the Pod's last snapshot attempt failed at ${last.at}: ${last.error}`); - const hours = (Date.now() - Date.parse(manifest.takenAt)) / 3600_000; - if (hours > STALE_HOURS) problems.push(`the newest snapshot is ${Math.floor(hours)} hours old`); - if (problems.length) throw new Error(`Backup needs attention: ${problems.join('; ')}`); -} - -/** - * Replaces everything the cluster holds with a snapshot copied to this Mac (see `backup`): the - * database, Temporal, Git, object storage and the platform's, POC's and Runner's records. Starts - * the Runner only with --runner. The node's own snapshots are kept. - */ -async function restore(source, ...flags) { - if (!source || source.startsWith('--') || flags.some(flag => flag !== '--runner')) - throw new Error('Usage: cluster-poc.mjs restore [--runner]'); - const directory = path.resolve(source); - const manifest = JSON.parse(await readFile(path.join(directory, 'manifest.json'), 'utf8')); - if (!manifest.database?.counts) throw new Error('The snapshot records no row counts to check the restore against'); - for (const file of ['postgres.dump', 'temporal.db', ...ARCHIVES.map(name => `${name}.tgz`)]) - if (!manifest.files[file]) throw new Error(`The snapshot has no ${file}`); - await verifyFiles(directory, manifest.files); +async function restore(name, ...flags) { + if (!name || name.startsWith('--') || flags.some(flag => flag !== '--runner')) + throw new Error('Usage: cluster-poc.mjs restore [--runner]'); + restoreScript(name); await assertCluster(); const image = (await imageRecord()).reference; + const { snapshots } = JSON.parse( + (await kubectl(backupExec(['node', 'scripts/cluster-poc-backup.mjs', 'latest']))).stdout, + ); + if (!snapshots.includes(name)) + throw new Error(`The node has no snapshot ${name}; it has ${snapshots.join(', ') || 'none'}`); + const remote = `/var/lib/wishforge/backups/${name}/manifest.json`; const work = await mkdtemp(path.join(os.tmpdir(), 'wishforge-restore-')); - await chmod(work, 0o700); try { - const temporal = path.join(work, 'temporal.tgz'); - const packed = await capture('tar', ['--no-mac-metadata', '-czf', temporal, '-C', directory, 'temporal.db'], { - env: { PATH: process.env.PATH, COPYFILE_DISABLE: '1' }, - }); - if (packed.code !== 0) throw new Error(`Packing Temporal's database failed: ${packed.stderr}`); - const tarballs = { - ...Object.fromEntries(ARCHIVES.map(name => [name, path.join(directory, `${name}.tgz`)])), - temporal, - }; - console.log(`Restoring snapshot ${manifest.name} (taken ${manifest.takenAt}) …`); + const size = (await kubectl(backupExec(['/bin/sh', '-c', `wc -c < ${remote}`]))).stdout.trim(); + await download(remote, path.join(work, 'manifest.json'), Number(size)); + const manifest = JSON.parse(await readFile(path.join(work, 'manifest.json'), 'utf8')); + if (!manifest.database?.counts) throw new Error('The snapshot records no row counts to check the restore against'); + for (const file of ['postgres.dump', 'temporal.db']) + if (!manifest.files[file]) throw new Error(`The snapshot has no ${file}`); + const archives = ARCHIVES.filter(archive => manifest.files[`${archive}.tgz`]); + console.log(`Restoring snapshot ${name} (taken ${manifest.takenAt}) …`); await replaceData({ image, - tarballs, - dump: path.join(directory, 'postgres.dump'), + snapshot: name, + archives, counts: manifest.database.counts, runner: flags.includes('--runner'), }); - console.log(`Restored snapshot ${manifest.name}${flags.includes('--runner') ? '' : '; the Runner stays idle'}.`); + console.log(`Restored snapshot ${name}${flags.includes('--runner') ? '' : '; the Runners stay idle'}.`); } finally { await rm(work, { recursive: true, force: true }); } @@ -1831,8 +1545,6 @@ const COMMANDS = { 'gateway-key': gatewayKey, secrets, apply, - migrate, - backup, restore, 'fanyu-access': fanyuAccess, projects: projectsCommand, diff --git a/scripts/cluster-poc.test.mjs b/scripts/cluster-poc.test.mjs index 333b3d9..adaa3c0 100644 --- a/scripts/cluster-poc.test.mjs +++ b/scripts/cluster-poc.test.mjs @@ -12,7 +12,7 @@ import { publicService, clusterEnvironments, clusterKubeconfig, - clusterPocFiles, + publicIPv4, envFile, fanyuKubeconfig, fanyuProfile, @@ -107,22 +107,23 @@ test('settings files hold one value per line and read back the same', () => { assert.throws(() => envFile({ A: "it's" }), /single-line/); }); -test('POC files keep their bindings and only move to the containers’ paths', () => { - const files = clusterPocFiles({ - profile: { id: 'droboticflow-poc', kubeconfig: '/Users/x/poc.config', namespaceUid: 'u' }, - binding: { projectId: 'p', profileFile: '/Users/x/p.json', stateDirectory: '/Users/x/s', gitHome: '/Users/x' }, - }); - assert.deepEqual(files.profile, { - id: 'droboticflow-poc', - kubeconfig: '/etc/wishforge/config/kubeconfig', - namespaceUid: 'u', - }); - assert.equal(files.binding.projectId, 'p'); - assert.equal(files.binding.gitHome, '/home/wishforge'); - const config = JSON.parse(clusterKubeconfig('Q0E=')); - assert.equal(config.clusters[0].cluster.server, 'https://120.48.181.105:6443'); - assert.deepEqual(config.users[0].user, { tokenFile: '/var/run/wishforge/kube/token' }); - assert.throws(() => clusterKubeconfig('not base64!'), /CA/); +test('the public entry takes a public IPv4 only, never a URL, private or documentation address', () => { + assert.equal(publicIPv4('120.48.164.20'), '120.48.164.20'); + for (const value of [ + 'https://8.8.8.8', + 'localhost', + '127.0.0.1', + '10.0.0.1', + '192.168.1.2', + '172.16.2.1', + '100.64.1.2', + '169.254.169.254', + '203.0.113.1', + '198.51.100.1', + '224.0.0.1', + '8.8.8.8\n', + ]) + assert.throws(() => publicIPv4(value), value); }); test('the public entry serves a Let’s Encrypt IP certificate and redirects to the standard HTTPS port', () => { @@ -210,7 +211,7 @@ test('the WishForge Pod is restricted everywhere and only Caddy listens beyond l ); assert.deepEqual( backup.volumeMounts.filter(mount => mount.readOnly).map(mount => mount.subPath), - ['git', 'minio', 'platform-state', 'poc-state', 'runner-state', 'temporal'], + ['git', 'minio', 'platform-state', 'poc-state', 'runner-state', 'runner-state-fanyu', 'temporal'], ); assert.deepEqual([...new Set(mounts('backup'))], ['data', 'backup-tmp']); assert.equal(backup.env.find(item => item.name === 'WF_IMAGE').value, IMAGE); diff --git a/scripts/configure-local-entry.mjs b/scripts/configure-local-entry.mjs deleted file mode 100644 index d4e61e5..0000000 --- a/scripts/configure-local-entry.mjs +++ /dev/null @@ -1,248 +0,0 @@ -#!/usr/bin/env node -import { constants } from 'node:fs'; -import { lstat, mkdir, open, readFile, writeFile } from 'node:fs/promises'; -import { randomUUID } from 'node:crypto'; -import { isIP } from 'node:net'; -import path from 'node:path'; -import { fileURLToPath, pathToFileURL } from 'node:url'; -import { parseEnv } from 'node:util'; -import { replaceEnvFile, setEnvValues } from './env-file.mjs'; - -export const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); -const KEY = 'WF_LOCAL_ALLOWED_ORIGINS'; -export const entryRoot = (root = ROOT) => path.join(root, '.wishforge/local-entry'); - -export function publicIPv4(value) { - if (isIP(value) !== 4) throw new Error('A public IPv4 address is required'); - const [a, b, c] = value.split('.').map(Number); - if ( - a === 0 || - a === 10 || - a === 127 || - a >= 224 || - (a === 100 && b >= 64 && b <= 127) || - (a === 169 && b === 254) || - (a === 172 && b >= 16 && b <= 31) || - (a === 192 && (b === 168 || b === 0 || (b === 88 && c === 99))) || - (a === 198 && (b === 18 || b === 19 || (b === 51 && c === 100))) || - (a === 203 && b === 0 && c === 113) - ) - throw new Error('The entry must use a publicly routable IPv4 address'); - return value; -} - -export async function privateDirectory(directory) { - await mkdir(directory, { recursive: true, mode: 0o700 }); - const stat = await lstat(directory); - if (!stat.isDirectory() || stat.isSymbolicLink() || stat.uid !== process.getuid() || stat.mode & 0o077) - throw new Error('The entry state directory must be operator-owned and private'); -} -export async function readPrivate(file) { - const handle = await open(file, constants.O_RDONLY | constants.O_NOFOLLOW); - try { - const stat = await handle.stat(); - if (!stat.isFile() || stat.uid !== process.getuid() || stat.mode & 0o077) - throw new Error('Configuration must be an operator-owned private file'); - return await handle.readFile('utf8'); - } finally { - await handle.close(); - } -} -function originsIn(source) { - const env = parseEnv(source); - if ( - env.WF_LOCAL_ACCESS !== 'true' || - env.WF_WEB_SERVE_MODE !== 'production' || - env.WF_DEPLOYMENT_MODE !== 'development' - ) - throw new Error('An authenticated production Web build in local development mode is required'); - const origins = (env[KEY] ?? '') - .split(',') - .map(value => value.trim()) - .filter(Boolean); - for (const origin of origins) { - const url = new URL(origin); - if (!['http:', 'https:'].includes(url.protocol) || url.origin !== origin || url.username || url.password) - throw new Error('Existing allowed origins must be exact HTTP(S) origins'); - } - if (!origins.length) throw new Error('Keep the existing local entry origins configured'); - return origins; -} -export function entryPlan(source, address) { - const origin = `https://${publicIPv4(address)}`; - const previous = originsIn(source); - const origins = [...new Set([...previous, origin])]; - return { - origin, - previous, - origins, - added: !previous.includes(origin), - source: setEnvValues(source, { [KEY]: origins.join(',') }), - }; -} - -/** Preview is read-only. Applying only adds this origin; its receipt can remove only that addition. */ -export async function configureEntry({ root = ROOT, address, apply = false, revert }) { - const file = path.join(root, '.env.local'); - const before = await readPrivate(file); - if (revert) { - const receipts = path.join(entryRoot(root), 'receipts'); - const receiptFile = path.resolve(revert); - if (path.dirname(receiptFile) !== receipts) throw new Error('Use a receipt from this entry state directory'); - const receipt = JSON.parse(await readPrivate(receiptFile)); - if ( - receipt.version !== 1 || - receipt.added !== true || - receipt.origin !== `https://${publicIPv4(new URL(receipt.origin).hostname)}` - ) - throw new Error('The receipt does not describe an origin this tool added'); - const previous = originsIn(before); - const origins = previous.filter(value => value !== receipt.origin); - const changed = origins.length !== previous.length; - if (apply && changed) await replaceEnvFile(file, before, setEnvValues(before, { [KEY]: origins.join(',') })); - return { - origin: receipt.origin, - removed: changed, - applied: apply && changed, - origins, - restartRequired: apply && changed, - }; - } - const { source: next, ...summary } = entryPlan(before, address); - if (!apply || !summary.added) return { ...summary, applied: false }; - const state = entryRoot(root); - await privateDirectory(state); - const receipts = path.join(state, 'receipts'); - await privateDirectory(receipts); - const id = randomUUID(); - await writeFile(path.join(receipts, `${id}.env`), before, { flag: 'wx', mode: 0o600 }); - const receipt = path.join(receipts, `${id}.json`); - await writeFile(receipt, JSON.stringify({ version: 1, origin: summary.origin, added: true }, null, 2), { - flag: 'wx', - mode: 0o600, - }); - await replaceEnvFile(file, before, next); - return { ...summary, applied: true, receipt, restartRequired: true }; -} - -/** All user HTTPS terminates on this Mac; the cluster transports opaque TCP. */ -export function entryConfigs(address, state, staging = false) { - const ip = publicIPv4(address); - const ca = `https://acme${staging ? '-staging' : ''}-v02.api.letsencrypt.org/directory`; - const caddy = `{ - admin off - skip_install_trust - default_bind 127.0.0.1 - http_port 18080 - https_port 18443 - default_sni ${ip} - auto_https disable_redirects - servers { - protocols h1 h2 - } -} -http://${ip}:18080 { - bind 127.0.0.1 - @safe method GET HEAD - redir @safe https://${ip}{uri} 308 - respond 405 -} -https://${ip}:18443 { - bind 127.0.0.1 - tls { - issuer acme { - dir ${ca} - test_dir ${ca} - profile shortlived - disable_tlsalpn_challenge - alt_http_port 18080 - } - } - reverse_proxy 127.0.0.1:3000 { - header_up X-Forwarded-Proto https - header_up -Forwarded - } -} -`; - const file = name => JSON.stringify(path.join(state, 'mtls', name)); - const frpc = `serverAddr = "${ip}" -serverPort = 7000 -loginFailExit = false -transport.protocol = "tcp" -transport.tls.enable = true -transport.tls.serverName = "wishforge-entry.internal" -transport.tls.certFile = ${file('client.crt')} -transport.tls.keyFile = ${file('client.key')} -transport.tls.trustedCaFile = ${file('ca.crt')} -auth.method = "token" -auth.additionalScopes = ["HeartBeats", "NewWorkConns"] -auth.tokenSource.type = "file" -auth.tokenSource.file.path = ${file('token')} -log.to = "console" -log.level = "info" -log.disablePrintColor = true -[[proxies]] -name = "wishforge-http" -type = "tcp" -localIP = "127.0.0.1" -localPort = 18080 -remotePort = 18080 -[[proxies]] -name = "wishforge-https" -type = "tcp" -localIP = "127.0.0.1" -localPort = 18443 -remotePort = 18443 -`; - return { caddy, frpc }; -} - -export async function writeEntryConfigs({ root = ROOT, address, staging = false }) { - const state = entryRoot(root); - await privateDirectory(state); - const configs = entryConfigs(address, state, staging); - for (const [name, content] of [ - ['Caddyfile', configs.caddy], - ['frpc.toml', configs.frpc], - ]) { - const file = path.join(state, name); - try { - const before = await readPrivate(file); - await replaceEnvFile(file, before, content); - } catch (error) { - if (error.code !== 'ENOENT') throw error; - await writeFile(file, content, { flag: 'wx', mode: 0o600 }); - } - } - const settings = path.join(state, 'entry.json'); - const content = JSON.stringify({ version: 1, address: publicIPv4(address), staging }, null, 2); - try { - const before = await readPrivate(settings); - await replaceEnvFile(settings, before, content); - } catch (error) { - if (error.code !== 'ENOENT') throw error; - await writeFile(settings, content, { flag: 'wx', mode: 0o600 }); - } - return { address, staging, state }; -} - -if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { - const args = process.argv.slice(2); - try { - let result; - if (args[0] === '--address' && args[1] && args.length <= 3 && (!args[2] || args[2] === '--apply')) - result = await configureEntry({ address: args[1], apply: args[2] === '--apply' }); - else if (args[0] === '--revert' && args[1] && args.length <= 3 && (!args[2] || args[2] === '--apply')) - result = await configureEntry({ revert: args[1], apply: args[2] === '--apply' }); - else if (args[0] === '--config' && args[1] && args.length <= 3 && (!args[2] || args[2] === '--staging')) - result = await writeEntryConfigs({ address: args[1], staging: args[2] === '--staging' }); - else - throw new Error( - 'Usage: configure-local-entry.mjs --address [--apply] | --revert [--apply] | --config [--staging]', - ); - console.log(JSON.stringify(result, null, 2)); - } catch (error) { - console.error(error.message); - process.exitCode = 1; - } -} diff --git a/scripts/configure-local-entry.test.mjs b/scripts/configure-local-entry.test.mjs deleted file mode 100644 index 0dd2fe1..0000000 --- a/scripts/configure-local-entry.test.mjs +++ /dev/null @@ -1,115 +0,0 @@ -import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { chmod, lstat, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises'; -import os from 'node:os'; -import path from 'node:path'; -import { - configureEntry, - entryConfigs, - entryPlan, - entryRoot, - publicIPv4, - writeEntryConfigs, -} from './configure-local-entry.mjs'; - -const IP = '120.48.164.20'; -const SOURCE = [ - 'WF_DEPLOYMENT_MODE=development', - 'WF_LOCAL_ACCESS=true', - 'WF_WEB_SERVE_MODE=production', - 'WF_LOCAL_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000,https://previous.trycloudflare.com', - 'WF_DEV_AUTH_TOKEN=fixture-private-value', - '', -].join('\n'); -async function fixture(t) { - const root = await mkdtemp(path.join(os.tmpdir(), 'wf-entry-config-test-')); - t.after(() => rm(root, { recursive: true, force: true })); - await writeFile(path.join(root, '.env.local'), SOURCE, { mode: 0o600 }); - return root; -} -test('only accepts public IPv4, never a URL, private or documentation address', () => { - assert.equal(publicIPv4(IP), IP); - for (const value of [ - 'https://8.8.8.8', - 'localhost', - '127.0.0.1', - '10.0.0.1', - '192.168.1.2', - '172.16.2.1', - '100.64.1.2', - '169.254.169.254', - '203.0.113.1', - '198.51.100.1', - '224.0.0.1', - '::1', - '8.8.8.8\n', - ]) - assert.throws(() => publicIPv4(value)); -}); -test('previews without writing, retains old origins and never returns private environment text', async t => { - const root = await fixture(t); - const result = await configureEntry({ root, address: IP }); - assert.equal(result.applied, false); - assert.equal(await readFile(path.join(root, '.env.local'), 'utf8'), SOURCE); - assert.ok(result.origins.includes('https://previous.trycloudflare.com')); - assert.ok(!JSON.stringify(result).includes('fixture-private-value')); - await assert.rejects(lstat(entryRoot(root)), { code: 'ENOENT' }); - assert.throws(() => entryPlan(SOURCE.replace('WF_LOCAL_ACCESS=true', 'WF_LOCAL_ACCESS=false'), IP)); - assert.throws(() => entryPlan(SOURCE.replace('WF_WEB_SERVE_MODE=production', 'WF_WEB_SERVE_MODE=development'), IP)); -}); -test('applies once and reverts only its addition, preserving later changes', async t => { - const root = await fixture(t); - const result = await configureEntry({ root, address: IP, apply: true }); - assert.equal(result.applied, true); - assert.equal((await lstat(result.receipt)).mode & 0o777, 0o600); - const file = path.join(root, '.env.local'); - const applied = await readFile(file, 'utf8'); - assert.ok(applied.includes('fixture-private-value')); - assert.ok(applied.includes(`https://${IP}`)); - assert.equal((await configureEntry({ root, address: IP, apply: true })).applied, false); - await writeFile(file, `${applied.replace(`https://${IP}`, `https://${IP},https://later.example`)}OTHER=new\n`); - const preview = await configureEntry({ root, revert: result.receipt }); - assert.equal(preview.applied, false); - const reverted = await configureEntry({ root, revert: result.receipt, apply: true }); - assert.equal(reverted.applied, true); - const after = await readFile(file, 'utf8'); - assert.ok(!after.includes(IP)); - assert.ok(after.includes('https://later.example')); - assert.ok(after.includes('https://previous.trycloudflare.com')); - assert.ok(after.includes('OTHER=new')); - assert.equal((await lstat(file)).mode & 0o777, 0o600); - await assert.rejects(configureEntry({ root, revert: path.join(root, 'elsewhere.json'), apply: true })); -}); -test('refuses shared or symbolic-link environment files', async t => { - const root = await fixture(t); - const file = path.join(root, '.env.local'); - await chmod(file, 0o644); - await assert.rejects(configureEntry({ root, address: IP, apply: true })); - await rm(file); - await writeFile(path.join(root, 'elsewhere'), SOURCE, { mode: 0o600 }); - await symlink(path.join(root, 'elsewhere'), file); - await assert.rejects(configureEntry({ root, address: IP, apply: true })); -}); -test('generates fixed loopback targets, explicit IP ACME and distinct staging storage settings', async t => { - const root = await fixture(t); - const result = await writeEntryConfigs({ root, address: IP, staging: true }); - const { caddy, frpc } = entryConfigs(IP, result.state, true); - assert.match(caddy, /default_bind 127\.0\.0\.1/); - assert.match(caddy, new RegExp(`default_sni ${IP.replaceAll('.', '\\.')}`)); - assert.match(caddy, /acme-staging-v02/); - assert.match(caddy, /profile shortlived/); - assert.match(caddy, /disable_tlsalpn_challenge/); - assert.match(caddy, /auto_https disable_redirects/); - assert.match(caddy, /reverse_proxy 127\.0\.0\.1:3000/); - assert.match(caddy, /header_up X-Forwarded-Proto https/); - assert.ok(!caddy.includes('tls internal')); - assert.ok(!caddy.includes('strict_sni_host')); - assert.match(frpc, /transport\.tls\.trustedCaFile/); - assert.match(frpc, /transport\.tls\.certFile/); - assert.match(frpc, /auth\.tokenSource\.file\.path/); - assert.equal((frpc.match(/\[\[proxies\]\]/g) ?? []).length, 2); - for (const unsafe of ['4000', '4300', 'visitors', 'plugins', 'webServer']) assert.ok(!frpc.includes(unsafe)); - await writeEntryConfigs({ root, address: IP, staging: false }); - assert.ok(!(await readFile(path.join(result.state, 'Caddyfile'), 'utf8')).includes('acme-staging')); - assert.equal((await lstat(path.join(result.state, 'entry.json'))).mode & 0o777, 0o600); -}); diff --git a/scripts/local-entry-service.mjs b/scripts/local-entry-service.mjs deleted file mode 100644 index af2e7a5..0000000 --- a/scripts/local-entry-service.mjs +++ /dev/null @@ -1,305 +0,0 @@ -#!/usr/bin/env node -// A separate LaunchAgent for the public entry. Importing this module starts nothing. -import { spawn, spawnSync } from 'node:child_process'; -import { existsSync, lstatSync, readFileSync, renameSync, writeFileSync } from 'node:fs'; -import { readdir, readFile } from 'node:fs/promises'; -import { homedir } from 'node:os'; -import { X509Certificate } from 'node:crypto'; -import path from 'node:path'; -import { pathToFileURL } from 'node:url'; -import { setTimeout as delay } from 'node:timers/promises'; -import { ROOT, entryRoot, entryConfigs, privateDirectory, readPrivate, publicIPv4 } from './configure-local-entry.mjs'; -import { minimalEnvironment, rotateAppend } from './local-poc-service.mjs'; - -export const LABEL = 'cc.d-robotics.wishforge.local-entry'; -const STATE = entryRoot(); -const NODE = '/opt/homebrew/bin/node'; -const DOMAIN = `gui/${process.getuid?.()}`; -const SERVICE = `${DOMAIN}/${LABEL}`; -const PLIST = path.join(homedir(), 'Library/LaunchAgents', `${LABEL}.plist`); -const STATE_FILE = path.join(STATE, 'service-state.json'); -const log = (name, message) => rotateAppend(path.join(STATE, 'logs', `${name}.log`), message); - -export function entryEnvironment(state, staging, network = {}) { - const proxies = Object.fromEntries( - ['HTTP_PROXY', 'HTTPS_PROXY', 'NO_PROXY'].filter(key => network[key]).map(key => [key, network[key]]), - ); - return { - ...minimalEnvironment(state), - TMPDIR: path.join(state, 'tmp'), - XDG_DATA_HOME: path.join(state, staging ? 'caddy-staging' : 'caddy-production'), - XDG_CONFIG_HOME: path.join(state, 'caddy-config'), - ...proxies, - }; -} -export function entryCommands(state) { - return [ - { - name: 'caddy', - binary: path.join(state, 'bin/caddy'), - args: ['run', '--config', path.join(state, 'Caddyfile'), '--adapter', 'caddyfile'], - }, - { name: 'frpc', binary: path.join(state, 'bin/frpc'), args: ['-c', path.join(state, 'frpc.toml')] }, - ]; -} -const xml = value => - String(value).replace( - /[<>&"']/g, - char => ({ '<': '<', '>': '>', '&': '&', '"': '"', "'": ''' })[char], - ); -export function entryPlist(root = ROOT, network = {}) { - const env = entryEnvironment(entryRoot(root), false, network); - return ` - - -Label${LABEL} -ProgramArguments${[NODE, path.join(root, 'scripts/local-entry-service.mjs'), 'run'].map(value => `${xml(value)}`).join('')} -WorkingDirectory${xml(root)} -EnvironmentVariables${Object.entries(env) - .map(([key, value]) => `${key}${xml(value)}`) - .join('')} -RunAtLoadKeepAlive -ThrottleInterval30ExitTimeOut25 -Umask63ProcessTypeBackground -StandardOutPath/dev/nullStandardErrorPath/dev/null -\n`; -} -function launchctl(args, checked = true) { - const result = spawnSync('/bin/launchctl', args, { encoding: 'utf8', timeout: 35000, env: minimalEnvironment() }); - if (checked && result.status !== 0) throw new Error(`launchctl ${args[0]} failed (exit ${result.status})`); - return result; -} -const registered = () => launchctl(['print', SERVICE], false).status === 0; -async function settings() { - const data = JSON.parse(await readPrivate(path.join(STATE, 'entry.json'))); - if (data.version !== 1 || typeof data.staging !== 'boolean') throw new Error('Invalid entry configuration'); - publicIPv4(data.address); - return data; -} -async function preflight() { - if (process.platform !== 'darwin') throw new Error('The entry service uses a macOS user LaunchAgent'); - await privateDirectory(STATE); - const data = await settings(); - const expected = entryConfigs(data.address, STATE, data.staging); - for (const [name, content] of [ - ['Caddyfile', expected.caddy], - ['frpc.toml', expected.frpc], - ]) - if ((await readPrivate(path.join(STATE, name))) !== content) - throw new Error(`Regenerate the managed ${name} before starting`); - for (const name of ['ca.crt', 'client.crt', 'client.key', 'token']) await readPrivate(path.join(STATE, 'mtls', name)); - for (const { binary } of entryCommands(STATE)) { - const stat = lstatSync(binary); - if (!stat.isFile() || stat.isSymbolicLink() || stat.uid !== process.getuid() || stat.mode & 0o022) - throw new Error('Entry binaries must be operator-owned regular files, not writable by others'); - } - const env = entryEnvironment(STATE, data.staging, process.env); - for (const directory of [env.TMPDIR, env.XDG_DATA_HOME, env.XDG_CONFIG_HOME, path.join(STATE, 'logs')]) - await privateDirectory(directory); - return { data, env }; -} -async function validate() { - const { env } = await preflight(); - for (const [binary, args] of [ - [path.join(STATE, 'bin/caddy'), ['validate', '--config', path.join(STATE, 'Caddyfile'), '--adapter', 'caddyfile']], - [path.join(STATE, 'bin/frpc'), ['verify', '-c', path.join(STATE, 'frpc.toml')]], - ]) { - const result = spawnSync(binary, args, { env, cwd: ROOT, timeout: 30000, encoding: 'utf8' }); - if (result.status !== 0) - throw new Error(`${path.basename(binary)} configuration validation failed (exit ${result.status})`); - } - return { valid: true }; -} - -/** Signals only children this supervisor spawned, never PIDs loaded from a stale state file. */ -export async function stopChildren(children, kill = process.kill, wait = delay) { - const active = child => child.pid && child.exitCode === null && child.signalCode === null; - const signal = name => { - for (const child of children.filter(active)) { - try { - kill(-child.pid, name); - } catch (error) { - if (error.code !== 'ESRCH') throw error; - } - } - }; - signal('SIGTERM'); - for (let attempt = 0; attempt < 30 && children.some(active); attempt++) await wait(200); - signal('SIGKILL'); -} -async function supervise() { - const { data, env } = await preflight(); - const children = []; - let stopping; - const stop = () => (stopping ??= stopChildren(children)); - for (const signal of ['SIGTERM', 'SIGINT']) process.once(signal, () => void stop()); - const ended = []; - for (const command of entryCommands(STATE)) { - const child = spawn(command.binary, command.args, { - cwd: ROOT, - env, - shell: false, - detached: true, - stdio: ['ignore', 'pipe', 'pipe'], - }); - children.push(child); - child.stdout.on('data', bytes => log(command.name, bytes)); - child.stderr.on('data', bytes => log(command.name, bytes)); - ended.push( - new Promise(resolve => { - child.once('error', () => { - log('supervisor', `${command.name} failed to start\n`); - resolve(); - }); - child.once('close', code => { - log('supervisor', `${command.name} exited ${code}\n`); - resolve(); - }); - }), - ); - } - writeFileSync( - STATE_FILE, - JSON.stringify({ - supervisorPid: process.pid, - children: children.map((child, index) => ({ name: entryCommands(STATE)[index].name, pid: child.pid })), - address: data.address, - staging: data.staging, - startedAt: new Date().toISOString(), - }), - { mode: 0o600 }, - ); - await Promise.race(ended); - await stop(); - await Promise.all(ended); -} - -async function certificates(directory, address) { - const found = []; - async function visit(dir, depth) { - if (depth > 8) return; - for (const entry of await readdir(dir, { withFileTypes: true }).catch(() => [])) { - const file = path.join(dir, entry.name); - if (entry.isDirectory()) await visit(file, depth + 1); - else if (entry.isFile() && entry.name.endsWith('.crt')) { - try { - const cert = new X509Certificate(await readFile(file)); - if (cert.checkIP(address)) - found.push({ - subjectAltName: cert.subjectAltName, - issuer: cert.issuer, - validTo: cert.validTo, - expiresInHours: Math.floor((Date.parse(cert.validTo) - Date.now()) / 3600000), - }); - } catch { - /* An unrelated certificate is not an entry certificate. */ - } - } - } - } - await visit(directory, 0); - return found; -} -async function status() { - const printed = launchctl(['print', SERVICE], false); - const pid = printed.status === 0 ? Number(printed.stdout.match(/^\s*pid = (\d+)$/m)?.[1]) || null : null; - let data; - try { - data = await settings(); - } catch { - /* Not yet configured. */ - } - let saved; - try { - saved = JSON.parse(readFileSync(STATE_FILE, 'utf8')); - } catch { - /* No matching supervisor state. */ - } - let httpsStatus = null; - if (pid && data && !data.staging) { - try { - httpsStatus = ( - await fetch(`https://${data.address}/login`, { redirect: 'manual', signal: AbortSignal.timeout(7000) }) - ).status; - } catch { - /* Unavailable or an untrusted/expired certificate. */ - } - } - return { - label: LABEL, - installed: existsSync(PLIST), - registered: printed.status === 0, - pid, - children: pid && saved?.supervisorPid === pid ? saved.children : [], - address: data?.address, - staging: data?.staging, - httpsStatus, - ready: Boolean(pid && httpsStatus === 200), - certificates: data ? await certificates(entryEnvironment(STATE, data.staging).XDG_DATA_HOME, data.address) : [], - }; -} -export function entryAction(args) { - if (!args.length) return 'status'; - if (args.length !== 1 || !['install', 'start', 'stop', 'status', 'validate', 'run'].includes(args[0])) - throw new Error('Usage: local-entry-service.mjs install|start|stop|status|validate'); - return args[0]; -} -export async function main(args = process.argv.slice(2)) { - const action = entryAction(args); - if (action === 'run') return supervise(); - if (action === 'status') return status(); - if (action === 'validate') return validate(); - if (action === 'stop') { - const printed = launchctl(['print', SERVICE], false); - const pid = printed.status === 0 ? Number(printed.stdout.match(/^\s*pid = (\d+)$/m)?.[1]) || null : null; - const pids = pid ? [pid] : []; - try { - const saved = JSON.parse(await readPrivate(STATE_FILE)); - if (pid && saved.supervisorPid === pid) - pids.push( - ...saved.children - .filter(child => ['caddy', 'frpc'].includes(child.name) && Number.isSafeInteger(child.pid) && child.pid > 1) - .map(child => child.pid), - ); - } catch { - /* No current supervisor state to wait for. */ - } - const alive = value => { - try { - process.kill(value, 0); - return true; - } catch (error) { - return error.code !== 'ESRCH'; - } - }; - launchctl(['disable', SERVICE]); - if (registered()) launchctl(['bootout', DOMAIN, PLIST]); - for (let attempt = 0; attempt < 60 && (registered() || pids.some(alive)); attempt++) await delay(250); - return { label: LABEL, stopCompleted: !registered() && !pids.some(alive) }; - } - await validate(); - if (action === 'install') { - const content = entryPlist(ROOT, process.env); - if (registered()) throw new Error('Stop this entry service before installing its LaunchAgent'); - const temporary = `${PLIST}.tmp`; - writeFileSync(temporary, content, { mode: 0o600 }); - const result = spawnSync('/usr/bin/plutil', ['-lint', temporary], { encoding: 'utf8', timeout: 10000 }); - if (result.status !== 0) throw new Error('Invalid entry LaunchAgent plist'); - renameSync(temporary, PLIST); - } - if (!existsSync(PLIST)) throw new Error('Install the entry service first'); - launchctl(['enable', SERVICE]); - if (!registered()) launchctl(['bootstrap', DOMAIN, PLIST]); - launchctl(['kickstart', SERVICE]); - return status(); -} -if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) - main() - .then(result => { - if (result) console.log(JSON.stringify(result, null, 2)); - if (result?.stopCompleted === false) process.exitCode = 1; - }) - .catch(error => { - console.error(error.message); - process.exitCode = 1; - }); diff --git a/scripts/local-entry-service.test.mjs b/scripts/local-entry-service.test.mjs deleted file mode 100644 index f2518be..0000000 --- a/scripts/local-entry-service.test.mjs +++ /dev/null @@ -1,148 +0,0 @@ -import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { mkdtemp, readFile, rm, symlink } from 'node:fs/promises'; -import os from 'node:os'; -import path from 'node:path'; -import { fileURLToPath } from 'node:url'; -import { once } from 'node:events'; -import { parseAllDocuments } from 'yaml'; -import { - entryAction, - entryCommands, - entryEnvironment, - entryPlist, - LABEL, - stopChildren, -} from './local-entry-service.mjs'; -import { isPolicyMain, policyServer, proxyDecision } from '../infra/k8s/local-entry/proxy-policy.mjs'; - -test('recognizes the policy entrypoint through a ConfigMap-style symlink without starting on import', async t => { - const directory = await mkdtemp(path.join(os.tmpdir(), 'wf-policy-link-')); - t.after(() => rm(directory, { recursive: true, force: true })); - const moduleUrl = new URL('../infra/k8s/local-entry/proxy-policy.mjs', import.meta.url); - const link = path.join(directory, 'proxy-policy.mjs'); - await symlink(fileURLToPath(moduleUrl), link); - assert.equal(isPolicyMain(moduleUrl.href, link), true); - assert.equal(isPolicyMain(import.meta.url, link), false); - assert.equal(isPolicyMain(moduleUrl.href, path.join(directory, 'missing')), false); -}); - -test('entry commands and environment do not inherit platform credentials or control its services', () => { - const env = entryEnvironment('/private/state', true, { - WF_DEV_AUTH_TOKEN: 'never-copy', - HOME: '/Users/operator', - HTTPS_PROXY: 'http://127.0.0.1:8089', - }); - assert.ok(!('WF_DEV_AUTH_TOKEN' in env)); - assert.equal(env.HOME, '/private/state'); - assert.equal(env.HTTPS_PROXY, 'http://127.0.0.1:8089'); - assert.ok(env.XDG_DATA_HOME.endsWith('caddy-staging')); - assert.ok(entryEnvironment('/private/state', false).XDG_DATA_HOME.endsWith('caddy-production')); - const commands = entryCommands('/private/state'); - assert.deepEqual( - commands.map(command => command.name), - ['caddy', 'frpc'], - ); - assert.ok(commands.every(command => command.binary.startsWith('/private/state/bin/'))); - const plist = entryPlist('/work/repo&space', { WF_DEV_AUTH_TOKEN: 'never-copy' }); - assert.match(plist, /repo&space/); - assert.ok(plist.includes(LABEL)); - assert.ok(!plist.includes('never-copy')); - assert.ok(!plist.includes('cc.d-robotics.wishforge.local-poc')); - assert.equal(entryAction([]), 'status'); - assert.throws(() => entryAction(['start', '--anything'])); -}); -test('stops only live children it owns, never already finished processes', async () => { - const children = [ - { pid: 10001, exitCode: null, signalCode: null }, - { pid: 10002, exitCode: 0, signalCode: null }, - ]; - const sent = []; - await stopChildren( - children, - (pid, signal) => { - sent.push([pid, signal]); - children[0].signalCode = signal; - }, - async () => {}, - ); - assert.deepEqual(sent, [[-10001, 'SIGTERM']]); -}); -test('FRP registration only accepts the two named TCP ports', () => { - const base = { op: 'NewProxy', content: { proxy_name: 'wishforge-http', proxy_type: 'tcp', remote_port: 18080 } }; - assert.deepEqual(proxyDecision(base), { reject: false, unchange: true }); - assert.equal( - proxyDecision({ ...base, content: { proxy_name: 'wishforge-https', proxy_type: 'tcp', remote_port: 18443 } }) - .reject, - false, - ); - for (const change of [ - { proxy_type: 'stcp' }, - { proxy_type: 'udp' }, - { remote_port: 4000 }, - { proxy_name: 'other' }, - { group: 'shared' }, - { custom_domains: ['anything.example'] }, - ]) - assert.equal(proxyDecision({ ...base, content: { ...base.content, ...change } }).reject, true); - for (const request of [null, {}, { ...base, op: 'Login' }]) assert.equal(proxyDecision(request).reject, true); -}); -test('policy endpoint rejects malformed input and reports health without logging content', async t => { - const server = policyServer(); - server.listen(0, '127.0.0.1'); - await once(server, 'listening'); - t.after(() => { - server.closeAllConnections(); - server.close(); - }); - const origin = `http://127.0.0.1:${server.address().port}`; - assert.equal((await fetch(`${origin}/healthz`)).status, 200); - assert.equal((await fetch(`${origin}/other`)).status, 403); - const invalid = await fetch(`${origin}/validate`, { method: 'POST', body: '{' }); - assert.equal(invalid.status, 400); - assert.equal((await invalid.json()).reject, true); - const allowed = await fetch(`${origin}/validate?op=NewProxy`, { - method: 'POST', - body: JSON.stringify({ - op: 'NewProxy', - content: { proxy_name: 'wishforge-https', proxy_type: 'tcp', remote_port: 18443 }, - }), - }); - assert.equal((await allowed.json()).reject, false); -}); -test('cluster manifests are isolated and do not implicitly create the billable Service', async () => { - const directory = new URL('../infra/k8s/local-entry/', import.meta.url); - const raw = await readFile(new URL('gateway.yaml', directory), 'utf8'); - const resources = parseAllDocuments(raw).map(document => document.toJS()); - const deploy = resources.find(resource => resource.kind === 'Deployment'); - assert.equal(deploy.metadata.namespace, 'wishforge-access'); - assert.equal(deploy.spec.replicas, 1); - assert.equal(deploy.spec.strategy.type, 'Recreate'); - const pod = deploy.spec.template.spec; - assert.equal(pod.automountServiceAccountToken, false); - assert.equal(pod.securityContext.runAsNonRoot, true); - assert.ok( - pod.containers.every( - container => - container.image.includes('@sha256:') && - container.securityContext.readOnlyRootFilesystem && - !container.securityContext.allowPrivilegeEscalation, - ), - ); - assert.equal(pod.containers[0].readinessProbe.tcpSocket.port, 'tunnel'); - assert.ok(!pod.volumes.some(volume => volume.hostPath || volume.persistentVolumeClaim)); - const kustomization = await readFile(new URL('kustomization.yaml', directory), 'utf8'); - assert.ok(!kustomization.includes('public-service.yaml')); - const service = parseAllDocuments(await readFile(new URL('public-service.yaml', directory), 'utf8'))[0].toJS(); - assert.deepEqual( - service.spec.ports.map(port => port.port), - [80, 443, 7000], - ); - assert.equal(service.metadata.annotations['service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps'], '5'); - assert.equal(service.metadata.annotations['service.beta.kubernetes.io/cce-elastic-ip-billing-method'], 'ByTraffic'); - assert.equal(service.metadata.annotations['service.beta.kubernetes.io/cce-elastic-ip-route-type'], 'BGP'); - assert.ok(!('service.beta.kubernetes.io/cce-load-balancer-id' in service.metadata.annotations)); - assert.ok(!service.spec.loadBalancerIP); - assert.ok(!raw.includes('droboticflow-api')); - assert.ok(!raw.includes('wishforge.io/tenant')); -});