forked from longtao.wu/droboticflow
Restore local dev toolkit (recovered after working copy loss)
- scripts/local/env-local.sh + run-operator.sh (verified working again) - local-samples for json_to_csv + video_slice - docs/local-dev-guide.zh-CN.md + docs/release-checklist.zh-CN.md
This commit is contained in:
parent
2a19c943ef
commit
8369b7e951
4
.gitignore
vendored
4
.gitignore
vendored
@ -14,3 +14,7 @@ docs.zip
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.operator-source-repos/
|
||||
|
||||
# 本地单算子调试产物
|
||||
local-runs/
|
||||
.venv/
|
||||
|
||||
149
docs/local-dev-guide.zh-CN.md
Normal file
149
docs/local-dev-guide.zh-CN.md
Normal file
@ -0,0 +1,149 @@
|
||||
# 本地开发调试指南(macOS / 隔离环境)
|
||||
|
||||
本指南描述如何在不接触公司集群的前提下,在本机完成单功能开发与调试。环境搭建背景见仓库根目录 `README.md`,业务概念见 `docs/platform-usage-manual.zh-CN.md`。
|
||||
|
||||
## 0. 隔离边界(先读这个)
|
||||
|
||||
| 事项 | 规则 |
|
||||
| --- | --- |
|
||||
| MongoDB | 只用本机容器 `fanyu-mongo`(`127.0.0.1:27017`),绝不指向公司实例 |
|
||||
| Kubernetes | 不导出 `KUBECONFIG`、不使用 `fanyu.conf`;发布走 `docs/release-checklist.zh-CN.md` 且须同事确认 |
|
||||
| BOS / AI | 本地全部留空禁用;数据源一律用本地路径(`local-samples/`) |
|
||||
| 镜像仓库 | 本地构建演练不登录、不推送;推送发布见发布清单 |
|
||||
|
||||
## 1. 环境准备(已完成则跳过)
|
||||
|
||||
```bash
|
||||
# 工具链(一次性)
|
||||
brew install go colima docker docker-buildx ffmpeg mongosh
|
||||
colima start --cpu 4 --memory 6 --vm-type vz --vz-rosetta
|
||||
|
||||
# 本地 MongoDB(只绑 127.0.0.1;--restart 保证 colima/VM 重启后自动跟随启动)
|
||||
docker run -d --name fanyu-mongo --restart unless-stopped -p '127.0.0.1:27017:27017' -v fanyu-mongo-data:/data/db mongo:7
|
||||
|
||||
# Python 算子依赖(公司网络下 pip 默认源不可用,用阿里镜像)
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com -r requirements-dev.txt
|
||||
|
||||
# Go 模块代理(公司网络下 proxy.golang.org 不可用)
|
||||
go env -w GOPROXY=https://goproxy.cn,direct
|
||||
|
||||
# 前端依赖
|
||||
npm install
|
||||
```
|
||||
|
||||
注意事项(踩过的坑):
|
||||
|
||||
- **Go 进程不读取 `.env.local`**:仓库没有 godotenv,`MONGO_URI` 等变量必须 shell 导出。
|
||||
- **colima VM 内 DNS 可能失效**(`systemd-resolved` 未安装导致 resolv.conf 指向不存在的 stub)。症状:`docker pull` 报 `lookup ... on [::1]:53: connection refused`。修复:`colima ssh -- sudo sh -c 'printf "nameserver 202.4.130.100\nnameserver 202.4.130.101\n" > /etc/resolv.conf'`。
|
||||
- **Docker Hub 直连超时**:改走国内镜像 `docker pull docker.m.daocloud.io/library/mongo:7` 再 `docker tag` 回 `mongo:7`。
|
||||
- **colima 每次重建 VM 后要重配两样**:VM 内 DNS(见上文修复命令)和 docker daemon 的 registry mirror(`colima ssh -- sudo sh -c 'printf ... > /etc/docker/daemon.json && systemctl restart docker'`)。建议保持 `--vm-type vz --vz-rosetta` 参数启动。
|
||||
- 日常启停:`colima start` / `colima stop`。Mongo 容器已设 `--restart unless-stopped`,会随 docker daemon 自动跟随启动;只有 colima VM 被**删除重建**(`colima delete`)后容器才会丢,需要按上面命令重建。
|
||||
- **API 报 `server selection timeout ... connection refused`** = 27017 没人监听:`docker start fanyu-mongo`(先确认 `colima status` 是 running)。API 进程本身不会死,Mongo 恢复后立即自愈,无需重启 `npm run dev`。
|
||||
|
||||
## 2. 启动本地平台(API + 前端)
|
||||
|
||||
```bash
|
||||
source scripts/local/env-local.sh # 每个新 shell 都要先 source
|
||||
npm run dev # API :3001 + Vite :5173
|
||||
```
|
||||
|
||||
- 浏览器打开 `http://127.0.0.1:5173`,首次用「创建超管账号」入口建一个本地管理员(如 `admin@local.dev`)。
|
||||
- 已验证可用:登录/会话、CSRF 双提交写保护、算子目录(111 个)、存储连接、项目、工作流(含节点)全套 CRUD。
|
||||
- **预期边界**:不启动 worker(`npm run dev:full` 本地起不来——worker 启动必须有 kubeconfig)。页面上点"运行"后 Run 会停在 `queued`,这是轻量层的正常行为。如需本地端到端,需另行搭建本地迷你 K8s(kind/colima-k8s),属于后续升级项。
|
||||
|
||||
写接口的 curl 调试需带 CSRF(与线上一致):
|
||||
|
||||
```bash
|
||||
curl -c /tmp/c.txt -X POST http://127.0.0.1:3001/api/auth/login \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"email":"admin@local.dev","password":"..."}'
|
||||
csrf=$(grep droboticflow_csrf /tmp/c.txt | awk '{print $NF}')
|
||||
curl -b /tmp/c.txt -X POST http://127.0.0.1:3001/api/projects \
|
||||
-H 'Content-Type: application/json' -H "X-Droboticflow-CSRF-Token: $csrf" \
|
||||
-d '{"name":"...","storageConnectionId":"<id>","storageBucket":"..."}'
|
||||
```
|
||||
|
||||
## 3. 单算子一键调试(核心工作流)
|
||||
|
||||
```bash
|
||||
scripts/local/run-operator.sh <slug> [输入artifact列表JSON文件]
|
||||
```
|
||||
|
||||
- `<slug>` 是 `operator-repos/builtin/` 下的目录名;脚本自动读取 `operator.json` / `runtime_contract.json` 的 handler,用与 worker 在 K8S Job 里**完全相同的方式**调用 `backend/operator-runner/main.py`。
|
||||
- 产物固定写入 `local-runs/<slug>/<时间戳>/`(已 .gitignore):`outputs/`、`result.json`、`stage-manifest.json`。
|
||||
- 默认输入:`local-samples/<slug>/input.json`(存在时)。
|
||||
- 附加超参:`OPERATOR_EXTRA_HYPERPARAMS='[{"key":"SEGMENT_DURATION_SECONDS","value":"4"}]'`。
|
||||
- 已内置样例:`json_to_csv`(3 行 JSON→CSV)与 `video_slice`(本地 10 秒测试视频→3 段虚拟切片)。
|
||||
|
||||
**改算子源码的调试循环**(不需要任何镜像构建):
|
||||
|
||||
```bash
|
||||
# 1. 直接改 operator-repos/builtin/<slug>/src/operator.py
|
||||
# 2. run-operator.sh 默认通过 DROBOTICFLOW_OPERATOR_SOURCE 直指这份源码,立即验证
|
||||
scripts/local/run-operator.sh <slug>
|
||||
# 3. 验证通过后,同步镜像副本(否则 operators:manifests:check 会挂):
|
||||
npm run operators:repos
|
||||
```
|
||||
|
||||
## 4. 分层测试命令
|
||||
|
||||
| 层 | 命令 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 单算子 | `scripts/local/run-operator.sh <slug>` | 真实执行,秒级 |
|
||||
| 单算子单测 | `.venv/bin/python -m pytest backend/operator-runner -k <关键词>` | 或 `tests/backend/<某算子>_test.py` |
|
||||
| Go 全量 | `npm test`(= `go test ./...`) | 不需要 Mongo/集群 |
|
||||
| Python 全量 | `npm run test:python` | 用系统 python3;建议先 `source .venv/bin/activate` |
|
||||
| 前端 | `npm run test:frontend` | node --test,无需浏览器 |
|
||||
| 总闸门 | `npm run verify:local` | lint + 全部测试 + 构建 + k8s 清单 dry-run |
|
||||
|
||||
## 5. 算子发布链本地演练(无 docker)
|
||||
|
||||
内置算子的正式发布流程是"改码 → 生成仓库清单 → 构建镜像 → digest 钉版 → 目录同步 → reconcile"。本地可以演练全部**不碰 docker** 的环节:
|
||||
|
||||
```bash
|
||||
npm run operators:repos # 同步 operator-repos 与 backend 镜像副本
|
||||
python3 scripts/update_operator_release_manifests.py --slug <slug> \
|
||||
--image <registry/ref:tag> --image-id <registry/ref@sha256:digest> # 钉版(digest 来自实际构建)
|
||||
npm run operators:images # 重新生成算子目录/镜像契约
|
||||
npm run operators:images:check # 一致性闸门,必须通过
|
||||
```
|
||||
|
||||
## 6. 镜像构建演练(本地、不推送)
|
||||
|
||||
公司网络下四个拦路虎及解法:
|
||||
|
||||
1. **基础镜像拉不下来**(Docker Hub 直连超时):先 `docker pull docker.m.daocloud.io/library/<镜像>` 再 `docker tag` 回标准名(golang:1.24-bookworm、node:22-alpine、debian:bookworm-slim、python:3.11-slim 都需要)。
|
||||
2. **构建时 `go mod download` 超时**:Dockerfile 里默认 `GOPROXY=proxy.golang.org` 被墙,构建时必须传 `--build-arg GOPROXY=https://goproxy.cn,direct`。
|
||||
3. **apt/pip 下载超时**(operator-runner 镜像重灾区):用本机代理 `127.0.0.1:7890`。colima 容器里访问宿主机代理要用 VM 网关地址(`colima ssh -- ip route | grep default`,当前为 `192.168.5.2`),以 `--build-arg http_proxy=http://192.168.5.2:7890` 注入(docker 的 http_proxy/https_proxy build-arg 无需 Dockerfile 声明即自动生效)。
|
||||
4. **QEMU 交叉编译 Go 会段错误**:colima 需用 `--vm-type vz --vz-rosetta` 启动(Rosetta 模拟 amd64 接近原生速度)。
|
||||
|
||||
因此公司网络内的本地构建演练用 docker 原生命令(`scripts/build-and-push-images.sh` 未透传 GOPROXY/代理,直接跑会卡在 go mod download 或 pip):
|
||||
|
||||
```bash
|
||||
PROXY=http://192.168.5.2:7890 # colima VM 看到的宿主机代理地址
|
||||
|
||||
docker build -f Dockerfile.worker --platform linux/amd64 \
|
||||
--build-arg GOPROXY=https://goproxy.cn,direct -t droboticflow-worker:localtest .
|
||||
docker build -f Dockerfile.drosync-coordinator --platform linux/amd64 \
|
||||
--build-arg GOPROXY=https://goproxy.cn,direct -t drosync-coordinator:localtest .
|
||||
docker build -f Dockerfile.api --platform linux/amd64 \
|
||||
--build-arg GOPROXY=https://goproxy.cn,direct \
|
||||
--build-arg http_proxy=$PROXY --build-arg https_proxy=$PROXY \
|
||||
-t droboticflow-api:localtest .
|
||||
docker build -f Dockerfile.operator-runner --platform linux/amd64 \
|
||||
--build-arg http_proxy=$PROXY --build-arg https_proxy=$PROXY \
|
||||
-t droboticflow-operator-runner:localtest .
|
||||
|
||||
# 验证产物架构
|
||||
docker image inspect droboticflow-operator-runner:localtest --format '{{.Os}}/{{.Architecture}}'
|
||||
```
|
||||
|
||||
> 注:真正推送发布时走 `scripts/build-and-push-images.sh`(见发布清单)——如果届时也卡在 go mod download / pip,可先与同事确认是否把代理 build-arg 透传进脚本,或临时在本机代理开着的状态下用上面的原生命令构建后手动 `docker push`。
|
||||
|
||||
## 7. 常见问题
|
||||
|
||||
- **`go run` 卡在下载依赖**:确认 `go env GOPROXY` 是 `https://goproxy.cn,direct`。
|
||||
- **pytest 报 numpy/cv2 缺失**:用 `.venv/bin/python -m pytest` 而不是裸 `pytest`。
|
||||
- **算子报"无法获取真实输入文件"**:检查 input.json 里 `items[].path`(文件级 path 是算子找源的入口,不是 artifact 顶层 path)。
|
||||
- **API 起不来报 Mongo 连接失败**:`docker start fanyu-mongo`,确认 `mongosh --host 127.0.0.1` 能 ping。
|
||||
109
docs/release-checklist.zh-CN.md
Normal file
109
docs/release-checklist.zh-CN.md
Normal file
@ -0,0 +1,109 @@
|
||||
# 发布清单与回滚预案(公司集群 droboticflow-fanyu)
|
||||
|
||||
> 本清单在**本地全绿 + 同事确认之后**才执行。发布顺序不可颠倒;每一步失败就停下排查,不带病前进。
|
||||
|
||||
## 0. 前置闸门(发布日前逐项打勾)
|
||||
|
||||
- [ ] `npm run verify:local` 全绿(lint + Go + Python + 前端 + 构建 + 清单 dry-run)
|
||||
- [ ] 涉及的算子用 `scripts/local/run-operator.sh <slug>` 实测通过
|
||||
- [ ] 改过 `operator-repos/**` 的,`npm run operators:images:check` 通过
|
||||
- [ ] 代码已 push 到 Gitea(无账号时:`git format-patch origin/dev -o /tmp/` 交同事代推/评审)
|
||||
- [ ] **同事已在 Gitea/补丁上评审确认**(必要条件,不满足则不发)
|
||||
- [ ] 已确认目标 namespace:`droboticflow-fanyu`;已和同事约定发布时间窗口(避开采集高峰)
|
||||
- [ ] 集群当前无进行中的重要 Run(平台监控页确认)
|
||||
|
||||
## 1. 记录回滚锚点(发布前必做)
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=/Users/d-robotics/Desktop/fanyu/fanyu.conf # 仅发布 shell 会话内
|
||||
kubectl -n droboticflow-fanyu get deploy \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .spec.template.spec.containers[*]}{" image: "}{.image}{"\n"}{end}{end}' \
|
||||
| tee /tmp/rollback-images-$(date +%Y%m%d-%H%M).txt
|
||||
```
|
||||
|
||||
> 注意:api/worker 的镜像记录在 Deployment env(`DROBOTICFLOW_IMAGE`、`OPERATOR_RUNNER_IMAGE`)里且带 digest;回滚清单要同时抓 env 里的这几项。把这份文件连同本次发布说明一起存档。
|
||||
|
||||
## 2. 构建并推送镜像(公网端点)
|
||||
|
||||
```bash
|
||||
docker login ccr-29eug8s3-pub.cnc.bj.baidubce.com
|
||||
# 凭据:用户 ccr$fanyu(见本地 source.md;只存在本机 docker 凭据库,绝不进代码)
|
||||
REGISTRY_HOST=ccr-29eug8s3-pub.cnc.bj.baidubce.com \
|
||||
REGISTRY_NAMESPACE=fanyu \
|
||||
IMAGE_TAG=$(git rev-parse --short HEAD) \
|
||||
./scripts/build-and-push-images.sh
|
||||
```
|
||||
|
||||
- 产出 4 个镜像:`droboticflow-api` / `droboticflow-worker` / `drosync-coordinator` / `droboticflow-operator-runner`(自动交叉构建 linux/amd64)。
|
||||
- 取 digest 备用:
|
||||
|
||||
```bash
|
||||
for img in droboticflow-api droboticflow-worker drosync-coordinator droboticflow-operator-runner; do
|
||||
docker buildx imagetools inspect \
|
||||
ccr-29eug8s3-pub.cnc.bj.baidubce.com/fanyu/$img:$(git rev-parse --short HEAD) \
|
||||
| grep Digest
|
||||
done
|
||||
```
|
||||
|
||||
- 只改了算子(没改平台代码)时可以只发算子镜像,其余保持旧 digest 不动——**最小变更原则**。
|
||||
- 公司网络下若卡在 go mod download / pip:以 `--build-arg http_proxy=http://<colima网关>:7890` 注入本机代理构建(见 local-dev-guide 第 6 节),构建完手动 `docker push`。
|
||||
|
||||
## 3. 部署到公司集群(VPC 端点 + digest 钉版)
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=/Users/d-robotics/Desktop/fanyu/fanyu.conf
|
||||
VPC=ccr-29eug8s3-vpc.cnc.bj.baidubce.com
|
||||
TAG=$(git rev-parse --short HEAD)
|
||||
|
||||
K8S_NAMESPACE=droboticflow-fanyu \
|
||||
API_IMAGE=$VPC/fanyu/droboticflow-api@sha256:<api-digest> \
|
||||
WORKER_IMAGE=$VPC/fanyu/droboticflow-worker@sha256:<worker-digest> \
|
||||
DROSYNC_COORDINATOR_IMAGE=$VPC/fanyu/drosync-coordinator@sha256:<drosync-digest> \
|
||||
OPERATOR_RUNNER_IMAGE=$VPC/fanyu/droboticflow-operator-runner@sha256:<runner-digest> \
|
||||
CREATE_REGISTRY_SECRET=false \
|
||||
./scripts/deploy-k8s.sh
|
||||
```
|
||||
|
||||
要点:
|
||||
|
||||
- `MONGO_URI` 不传:脚本自动从集群现有 `droboticflow-mongo-secret` 回读。
|
||||
- `CREATE_REGISTRY_SECRET=false`:复用集群现有 `droboticflow-registry` 拉取密钥,命令行不出现任何明文凭据。
|
||||
- 脚本内部是 `kubectl apply` + `rollout status` 等待,可安全重跑。
|
||||
- GPU/Qwen 相关变量(`OPERATOR_GPU_NODE_SELECTOR` 等)当前由集群 ConfigMap 提供,不在本次发布中改动。
|
||||
|
||||
## 4. 发布后验证
|
||||
|
||||
```bash
|
||||
# 1) rollout 就绪 + 健康
|
||||
kubectl -n droboticflow-fanyu get pods
|
||||
curl -s http://120.48.180.80/api/health && curl -s http://120.48.180.80/api/ready
|
||||
|
||||
# 2) 构建 commit 与预期一致
|
||||
curl -s http://120.48.180.80/api/health | python3 -m json.tool | grep -i commit
|
||||
|
||||
# 3) 平台冒烟:登录、跑一条最小工作流(或按变更范围跑对应验收)
|
||||
```
|
||||
|
||||
## 5. 回滚预案
|
||||
|
||||
症状(任一):健康检查失败、run 大面积失败、镜像拉取失败。
|
||||
|
||||
```bash
|
||||
# 用第 1 步存档的旧 digest 重跑同一脚本即回滚(rolling update 回旧镜像)
|
||||
export KUBECONFIG=/Users/d-robotics/Desktop/fanyu/fanyu.conf
|
||||
K8S_NAMESPACE=droboticflow-fanyu \
|
||||
API_IMAGE=<旧api镜像@digest> WORKER_IMAGE=<旧worker镜像@digest> \
|
||||
DROSYNC_COORDINATOR_IMAGE=<旧drosync镜像@digest> \
|
||||
OPERATOR_RUNNER_IMAGE=<旧runner镜像@digest> \
|
||||
CREATE_REGISTRY_SECRET=false ./scripts/deploy-k8s.sh
|
||||
```
|
||||
|
||||
- 算子级回滚:`git revert` 对应 `operator-repos/builtin/<slug>/release.json` 的提交,重跑 `npm run operators:images` + reconcile,把线上算子目录拨回旧 digest。
|
||||
- 数据安全:deploy 只滚动更新 Deployment,不动 Mongo 数据与 BOS 对象;回滚后 Run 历史仍完整可查。
|
||||
- 回滚后立即通知同事并在发布记录里写明原因。
|
||||
|
||||
## 6. 代码评审通道(Gitea 权限待定期间)
|
||||
|
||||
- 有推送权限:`git push origin feature/<名字>` → Gitea 上发起合并到 `dev` → 同事评审。
|
||||
- 暂无权限:`git format-patch origin/dev -o /tmp/patches/`,把补丁目录交同事 `git am` 代推;或直接把分支推到个人 fork。
|
||||
- 线上 `dev` 分支与集群部署版本可能存在少量提交差,发布前与同事对齐目标基线,避免把未验证提交带上线。
|
||||
14
local-samples/json_to_csv/input.json
Normal file
14
local-samples/json_to_csv/input.json
Normal file
@ -0,0 +1,14 @@
|
||||
[
|
||||
{
|
||||
"name": "sample-data.json",
|
||||
"type": "json",
|
||||
"path": "local-samples/json_to_csv/sample-data.json",
|
||||
"data": {
|
||||
"items": [
|
||||
{"filename": "a.mp4", "duration": 12.5, "label": "walk"},
|
||||
{"filename": "b.mp4", "duration": 30.0, "label": "run"},
|
||||
{"filename": "c.mp4", "duration": 7.25, "label": "jump"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
7
local-samples/json_to_csv/sample-data.json
Normal file
7
local-samples/json_to_csv/sample-data.json
Normal file
@ -0,0 +1,7 @@
|
||||
{
|
||||
"items": [
|
||||
{"filename": "a.mp4", "duration": 12.5, "label": "walk"},
|
||||
{"filename": "b.mp4", "duration": 30.0, "label": "run"},
|
||||
{"filename": "c.mp4", "duration": 7.25, "label": "jump"}
|
||||
]
|
||||
}
|
||||
16
local-samples/video_slice/input.json
Normal file
16
local-samples/video_slice/input.json
Normal file
@ -0,0 +1,16 @@
|
||||
[
|
||||
{
|
||||
"name": "sample-video.mp4",
|
||||
"type": "json",
|
||||
"data": {
|
||||
"items": [
|
||||
{
|
||||
"filename": "sample-video.mp4",
|
||||
"path": "/Users/d-robotics/Desktop/fanyu/droboticflow/local-samples/video_slice/sample-video.mp4",
|
||||
"duration": 10.0,
|
||||
"label": "test"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
BIN
local-samples/video_slice/sample-video.mp4
Normal file
BIN
local-samples/video_slice/sample-video.mp4
Normal file
Binary file not shown.
28
scripts/local/env-local.sh
Executable file
28
scripts/local/env-local.sh
Executable file
@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
# 本地开发环境变量(source 方式使用:source scripts/local/env-local.sh)
|
||||
#
|
||||
# 隔离边界(务必遵守):
|
||||
# - MONGO_URI 只指向本机 127.0.0.1 的本地 MongoDB 容器,绝不指向公司实例
|
||||
# - 不导出 KUBECONFIG,不使用公司集群凭据
|
||||
# - BOS/AI 变量全部留空:本地不访问百度 BOS,AI 生成功能禁用
|
||||
# - 数据源一律使用本地路径(local-samples/ 或自建目录)
|
||||
|
||||
export MONGO_URI="mongodb://127.0.0.1:27017/droboticflow-local"
|
||||
export MONGO_DB="droboticflow-local"
|
||||
export MONGO_CONNECT_RETRIES=5
|
||||
export MONGO_CONNECT_RETRY_DELAY_MS=1000
|
||||
|
||||
# API / 前端
|
||||
export PORT=3001
|
||||
export VITE_DEV_API_BASE_URL="http://127.0.0.1:3001"
|
||||
export VITE_DEV_WORKER_BASE_URL="http://127.0.0.1:3002"
|
||||
|
||||
# 可选外部依赖:本地全部禁用
|
||||
unset BOS_ENDPOINT || true
|
||||
unset AI_API_KEY || true
|
||||
unset AI_API_BASE_URL || true
|
||||
unset AI_MODEL || true
|
||||
unset KUBECONFIG || true
|
||||
|
||||
echo "[env-local] 本地环境已加载:Mongo=127.0.0.1:27017/droboticflow-local, API=:3001"
|
||||
echo "[env-local] 提醒:Go 进程不读取 .env.local,环境变量必须像这样 source 导出"
|
||||
131
scripts/local/run-operator.sh
Executable file
131
scripts/local/run-operator.sh
Executable file
@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env bash
|
||||
# 单算子本地一键调试:模拟 worker 在 Kubernetes Job 里调用 operator-runner 的同一套方式。
|
||||
#
|
||||
# 用法:
|
||||
# scripts/local/run-operator.sh <slug> [输入文件或目录]
|
||||
#
|
||||
# 示例:
|
||||
# scripts/local/run-operator.sh json_to_csv local-samples/json_to_csv/input.json
|
||||
# scripts/local/run-operator.sh video_slice local-samples/video_slice/input.json
|
||||
#
|
||||
# 说明:
|
||||
# - <slug> 是 operator-repos/builtin/ 下的目录名(如 video_slice)
|
||||
# - 输入文件是 JSON Artifact 列表(形如 [{"name":..., "path":..., "type":...}, ...]);
|
||||
# 若省略则使用 local-samples/<slug>/input.json(存在时),否则用空输入
|
||||
# - 产物固定写到 ./local-runs/<slug>/<时间戳>/,包含 outputs/、result.json、stage-manifest.json
|
||||
# - 调试 operator-repos 源码副本改动时无需任何额外参数:脚本默认用
|
||||
# DROBOTICFLOW_OPERATOR_SOURCE 直接指向 operator-repos/builtin/<slug>/src/operator.py
|
||||
# - 附加超参:OPERATOR_EXTRA_HYPERPARAMS='[{"key":"K","value":"V"}]' 变量传入
|
||||
# - 使用仓库 .venv 里的 Python(含 numpy/opencv 等算子依赖);.venv 不存在时回退 python3
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
slug="${1:-}"
|
||||
if [[ -z "$slug" ]]; then
|
||||
echo "用法: scripts/local/run-operator.sh <slug> [输入artifact列表JSON文件]" >&2
|
||||
echo "示例: scripts/local/run-operator.sh json_to_csv local-samples/json_to_csv/input.json" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
op_dir="operator-repos/builtin/$slug"
|
||||
if [[ ! -f "$op_dir/operator.json" ]]; then
|
||||
echo "错误: 找不到算子 $slug($op_dir/operator.json 不存在)" >&2
|
||||
echo "可用算子: ls operator-repos/builtin/" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 从 operator.json / runtime_contract.json 读取算子名和 handler
|
||||
op_name="$(python3 -c "import json; print(json.load(open('$op_dir/operator.json'))['name'])")"
|
||||
handler="$(python3 -c "import json; print(json.load(open('$op_dir/runtime_contract.json'))['handler'])")"
|
||||
|
||||
# 输入 artifact 列表:命令行 > local-samples 默认 > 空列表
|
||||
input_file="${2:-}"
|
||||
if [[ -z "$input_file" && -f "local-samples/$slug/input.json" ]]; then
|
||||
input_file="local-samples/$slug/input.json"
|
||||
fi
|
||||
if [[ -n "$input_file" && ! -f "$input_file" ]]; then
|
||||
echo "错误: 输入文件不存在: $input_file" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run_dir="$REPO_ROOT/local-runs/$slug/$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$run_dir/outputs"
|
||||
|
||||
# 组装 NODE_DATA_JSON:label 用于日志显示,outputs 占位一个端口
|
||||
extra_hyperparams="${OPERATOR_EXTRA_HYPERPARAMS:-[]}"
|
||||
node_data_json="$(slug_label="$op_name" extra_hyperparams="$extra_hyperparams" python3 - <<'PY'
|
||||
import json, os
|
||||
print(json.dumps({
|
||||
"label": os.environ["slug_label"],
|
||||
"hyperparams": json.loads(os.environ["extra_hyperparams"]),
|
||||
"outputs": [{"id": "out-1", "name": "output", "type": "json"}],
|
||||
}, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
|
||||
# 输入 artifact:文件形式给 INPUT_ARTIFACTS_PATH(>64KiB 或含路径引用时更贴近真实)
|
||||
if [[ -n "$input_file" ]]; then
|
||||
cp "$input_file" "$run_dir/input-artifacts.json"
|
||||
input_env=("INPUT_ARTIFACTS_PATH=$run_dir/input-artifacts.json")
|
||||
else
|
||||
input_env=("INPUT_ARTIFACTS_JSON=[]")
|
||||
fi
|
||||
|
||||
# Python 解释器:优先仓库 .venv(有 numpy/opencv/h5py 等依赖)
|
||||
if [[ -x ".venv/bin/python" ]]; then
|
||||
PY=".venv/bin/python"
|
||||
else
|
||||
PY="python3"
|
||||
echo "[提示] .venv 不存在,使用系统 python3;视频/科学计算类算子可能缺依赖" >&2
|
||||
echo "[提示] 建议执行: python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt" >&2
|
||||
fi
|
||||
|
||||
echo "====== 单算子本地调试 ======"
|
||||
echo "算子: $slug / $op_name"
|
||||
echo "handler: $handler"
|
||||
echo "输入: ${input_file:-(空输入)}"
|
||||
echo "产物目录: $run_dir"
|
||||
echo "============================"
|
||||
|
||||
env \
|
||||
RUN_ID="run-local-$slug" \
|
||||
WORKFLOW_ID="wf-local" \
|
||||
NODE_ID="node-local-$slug" \
|
||||
NODE_LABEL="$op_name" \
|
||||
NODE_DATA_JSON="$node_data_json" \
|
||||
"${input_env[@]}" \
|
||||
OUTPUT_DIR="$run_dir/outputs" \
|
||||
RESULT_OUTPUT_PATH="$run_dir/result.json" \
|
||||
STAGE_MANIFEST_PATH="$run_dir/stage-manifest.json" \
|
||||
DROBOTICFLOW_OPERATOR_HANDLER="$handler" \
|
||||
DROBOTICFLOW_OPERATOR_SOURCE="$op_dir/src/operator.py" \
|
||||
"$PY" backend/operator-runner/main.py
|
||||
|
||||
echo
|
||||
echo "====== 执行完成 ======"
|
||||
echo "result.json: $run_dir/result.json"
|
||||
echo "stage-manifest: $run_dir/stage-manifest.json"
|
||||
echo "outputs/: $run_dir/outputs/"
|
||||
if [[ -f "$run_dir/result.json" ]]; then
|
||||
python3 - "$run_dir/result.json" <<'PY'
|
||||
import json, sys
|
||||
try:
|
||||
result = json.load(open(sys.argv[1]))
|
||||
except Exception as exc:
|
||||
print(f"(result.json 解析失败: {exc})")
|
||||
raise SystemExit(0)
|
||||
artifacts = result.get("artifacts", [])
|
||||
print(f"产物 artifact 数: {len(artifacts)}")
|
||||
for artifact in artifacts:
|
||||
print(f" - {artifact.get('name')} (type={artifact.get('type')}, path={artifact.get('path')})")
|
||||
logs = result.get("logs", [])
|
||||
errors = [entry for entry in logs if entry.get("level", "").lower() in ("error", "warn")]
|
||||
if errors:
|
||||
print(f"日志中的告警/错误 {len(errors)} 条(前 3 条):")
|
||||
for entry in errors[:3]:
|
||||
print(f" [{entry.get('level')}] {entry.get('message', '')[:160]}")
|
||||
PY
|
||||
fi
|
||||
Loading…
x
Reference in New Issue
Block a user