75 Commits

Author SHA1 Message Date
Codex
f4e147e355 🐛 Enforce V4 delivery for legacy release runs 2026-11-08 10:00:31 -08:00
Codex
7b7da08a44 ⚡ Decouple Capture status reconciliation from source discovery 2026-11-02 10:09:18 -08:00
d-robotics
23087c63b8 🐛 Advance materialization cleanup fairly across queues and pages 2026-10-15 13:52:00 +08:00
d-robotics
05f2dc1499 🚦 Prioritize bounded recovery runs 2026-10-01 03:08:27 -07:00
d-robotics
4a364b1352 🐛 Fix review binding races and unblock worker dispatch
(cherry picked from commit ea9dd2fc79cac1a3c2c02fe34ad9491119a4a848)
2026-09-29 08:54:35 -07:00
eust-w
21277a1887 ✨ Archive old terminal runs to bound org-wide scan cost
An organization's run history grows unbounded and nothing ever shrinks
it — completed/failed/cancelled runs stay in the runs collection
forever, only removed if their whole project is later deleted. That
made every org-wide scan (most notably validateDeletionReferences'
deletion-safety check, and any other query through
ListRunArtifactsByOrganization) get proportionally slower over time as
an org's run count kept climbing, independent of the size of whatever
was actually being operated on. Live on Fanyu this showed up as project
deletion getting slower every day, driven entirely by an unrelated
capture-processing backlog continuously adding new runs to the same
organization.

Adds RunRecord.ArchivedAt and a periodic worker sweep
(archiveTerminalRuns, mirroring the existing orphan-runtime
reconciliation's throttle pattern) that marks completed/failed/cancelled
runs whose EndedAt is older than WORKER_RUN_ARCHIVE_MIN_AGE_SECONDS
(default 30 days) as archived, checking at most every
WORKER_RUN_ARCHIVE_CHECK_INTERVAL_SECONDS (default 1 hour). Archiving
never deletes anything — ListRunArtifactsByOrganization now excludes
archived runs so the hot-path org-wide scans stay bounded, but
ListRunArtifactsForProject (used when a project is actually being
deleted) still includes them, since a project's own deletion must still
clean up every run it ever had regardless of archive state. A run's
data is only actually removed when its project is deleted, unchanged
from today.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 14:02:34 +08:00
eust-w
e48c4ccab3 🐛 Stabilize worker readiness and verify multi-supplier delivery 2026-09-21 12:36:08 +08:00
eust-w
29ed9d2475 🗂️ Add reviewable slice discard policy and filtered delivery 2026-09-16 20:44:46 +08:00
eust-w
640b703c91 🛠️ Add resumable project cascade deletion with scoped cleanup and audit 2026-09-08 16:46:51 +08:00
eust-w
7000f68af5 ♻️ Retire single-run execution in favor of continuous Capture production 2026-09-08 13:17:03 +08:00
eust-w
4e761d6369 ✨ Report active Capture subdirectory in workflow run history 2026-09-07 03:53:45 +08:00
eust-w
f8239c12a6 🐛 Preserve actual Job image digest when recovering executions 2026-09-06 13:34:58 +08:00
eust-w
1eb1eb754a ✨ Upgrade Fanyu Capture production flow 2026-08-31 21:25:47 +08:00
eust-w
c26aa00ab5 🐛 Terminalize orphaned active runs 2026-08-28 08:39:42 +08:00
eust-w
5389843f1e ✨ Add per-capture automatic workflow execution 2026-08-28 00:31:04 +08:00
eust-w
f786bdc3b1 📌 Preserve environment-specific operator image pins 2026-08-27 13:02:42 +08:00
eust-w
cd8204d8e7 ✨ Materialize synchronized slices after annotation 2026-08-27 03:51:29 +08:00
eust-w
109e0b31c4 🔗 Normalize artifact value and reference presentation 2026-08-22 17:26:47 +08:00
eust-w
b0734fec38 ⚡ Add opt-in Qwen node-local cache 2026-08-22 06:02:27 +08:00
eust-w
13bd8fc384 🐛 Refresh Fanyu template and runtime evidence 2026-08-21 13:12:44 +08:00
eust-w
6aa4058e36 🚀 Build PTS-grounded Fanyu human workflow 2026-08-21 12:29:07 +08:00
eust-w
7cf37084e3 🐛 Respect active run leases during reconciliation 2026-08-21 06:05:39 +08:00
eust-w
17adaaa8ae ✨ feat: implement dual-track workflow execution isolation (debug vs production) 2026-08-20 09:00:11 +08:00
eust-w
33b52dcfd0 🚀 Implement dual-track store and workflow coverage updates 2026-08-20 01:24:16 +08:00
eust-w
f31fd7ae55 🔀 Merge Fanyu full delivery baseline 2026-08-19 16:22:21 +08:00
eust-w
7e663344f2 ✨ Add versioned workflow input contracts 2026-08-19 12:57:20 +08:00
eust-w
0324b189e6 ✨ Add pluginized human-loop annotation workflows 2026-08-18 21:52:20 +08:00
eust-w
8aaa50fe3f ✨ Connect native annotation workflow gate 2026-08-18 12:22:11 +08:00
eust-w
93662f828e 🐛 Finalize reviewed artifact delivery 2026-08-18 11:35:44 +08:00
eust-w
48a5a0a899 🐛 Preserve artifacts across human review gates 2026-08-18 11:07:09 +08:00
eust-w
4afdc1c869 🛠️ Refresh builtin runtime images safely 2026-08-17 16:04:44 +08:00
eust-w
9051f412c9 📊 Add end-to-end data operations lifecycle 2026-07-23 21:11:15 +08:00
eust-w
090b034247 🛟 preserve jobs across worker interruptions 2026-07-13 20:39:16 +08:00
eust-w
1678e33d8f 🔄 sync progress before stale job reconciliation 2026-07-13 18:27:57 +08:00
eust-w
4fd8796b9d 📈 persist progress during active run reconciliation 2026-07-13 18:19:48 +08:00
eust-w
bf498237d8 🧭 distinguish cancelled workflow status 2026-07-13 18:10:00 +08:00
eust-w
b59a692be8 🧹 clean runtime after terminal failures 2026-07-13 12:50:21 +08:00
eust-w
6fe111021b 🚦 converge worker runs and harden POC acceptance 2026-07-13 12:31:36 +08:00
eust-w
f0959b4e41 ✨ lock active workflows and report runner builds 2026-07-11 01:11:48 +08:00
eust-w
66a15d4cf4 ⚡ tighten worker scheduling and runtime visibility 2026-07-11 00:10:34 +08:00
eust-w
bf36bf6086 ✨ optimize POC workflow performance and reliability 2026-07-10 17:16:38 +08:00
wangbin
db6f97b25e 流式上报运行中节点的 GPU+日志,修 RUNNING 态资源监控 GPU=0 与 sam 节点日志不可见
根因:launcher 的 GPU 采样 goroutine 把样本缓存在闭包内、pod 日志一次性 Stream 全量,
两者都只在节点 completion(Launch 返回)时落进 LaunchResult;worker 阻塞在 launcher.Launch
上,运行中无任何样本/日志持久化。builtin 节点靠 operator-runner 的 ResourceSampler
自报告,但 custom-container 节点(如 SAM3)绕过 main.py,运行中 resourceSeries/logs 全空。

分治修法(各用其 RBAC,核心执行循环与 completion 路径零改动,稳定不冒险):
- GPU(worker,worker SA 有 pods/exec):节点 dispatch goroutine 内为 custom-container
  节点起 runNodeProgressReporter,每 15s 调 launcher.SamplePodGPUForRun exec nvidia-smi,
  经 store.AppendNodeExecutionResourceSample(mongo $push positional 到
  nodeExecutions[nodeId].resourceSeries)就地追加,不重写整条 run、不碰 currentRun。
  completion 时 2146 用 final samples 自然覆盖(终态正确)。goroutine 在 executeNode 返回后
  progressCancel+Wait 再送 outcome,确保无 live 样本落在 final 之后。
- Logs(API,api SA 有 pods/log 但无 pods/exec):handleRunsLogs 对 in-flight run 找 running
  节点,launcher.TailNodePodLogs(GetLogs TailLines+Container=operator-runner)live-tail
  pod stdout,按行解析成 LogEntry 合并进响应,运行中即可见 sam 执行日志。
- handleRunsResources 读时调 rebuildRunAggregates 折叠 nodeExecutions(含 worker $push 的
  live 样本)进 top-level ResourceSeries——idempotent 于 worker 自身折叠,仅运行中增值。
- 前端 Monitoring logs modal:对 in-flight run 5s 轮询 /api/runs/:id/logs,配合 API live-tail
  实时刷新;run 终态或 modal 关闭即停。

新增:k8s.SamplePodGPUForRun / TailNodePodLogs / findRunningNodePod(复用 samplePodGPU);
store.AppendNodeExecutionResourceSample(interface+mongo $push+memory);worker
shouldReportNodeProgress/mapGPUSample/runNodeProgressReporter + jobLauncher 接口方法。
回归:store/worker/api/k8s 全测试通过(新增 3 个针对性测试)+ 192 前端测试 + tsc + vet。

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-10 12:08:38 +08:00
wangbin
b7c798c342 🐛 修复 ProcessOnce claim 与 activeRuns 竞态: 释放误占 lease, 不再回退 Queued
排查 "执行期间 status=queued" 现象, 结论: 非 active bug, 但修复了唯一能产生该现象的
潜在缺陷.

根因复测: completed run 6a4f6c3d (=run-b03e3098) 的 nodeExecutions 显示 sam3 为
attempt=1 单一干净派发, 全程持久化, 无重复派发 → 证实 "sam3 不在 nodeExecutions
+ status=queued" 是字段大小写误读 (currentNodeID vs currentNodeId), 平台健康.
ProcessAvailable 流程在 claim 时原子置 Running, 派发后 if-launched 块持久化各节点,
正常流程无 "执行期间 queued" 窗口.

唯一能产生 "Queued + currentNodeId 保留" 态的代码点是 ProcessOnce:2205
(tryMarkRunActive 失败分支): 回退 status=Queued 却不清 lease (defer 在 return 后才
注册, 不执行) → 留下 "Queued + currentNodeId + lease 有效" 不一致态 (正匹配误读现象),
且会卡 ~15min (lease 被占, ProcessAvailable 的 claim filter 过不去). ProcessOnce
当前为死代码 (无内部调用方/前端不调/API proxy 日志 3h 零次), 非 active 触发.

修复 (tryMark 失败分支): 改调 ReleaseRunLease 释放误占 lease + 保留 status=Running
(tryMark 失败 = 本 worker 另一 goroutine 正处理它, Running 才正确), 让 reconcile /
活跃处理方推进 run. 用现有 store 方法, 零新接口面.

回归测试 TestProcessOnceSkippedLeaseActiveReleasesLeaseAndKeepsRunning:
旧代码 FAIL ("expected run to stay Running ... got queued"), 新代码 PASS, 全套 worker
测试无回归.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-09 21:09:55 +08:00
eust-w
5e4185ff8b 🐛 Fix workflow lost-update + reconcile race during long GPU runs
Two platform bugs surfaced by the SAM3 video-operator run:

P6 — workflow PUT reverted mid-run (lost update). The worker held a workflow
snapshot for the whole run duration and bumped run state via UpsertWorkflow
(full-document $set), clobbering concurrent PUT edits to Nodes — e.g. a node's
containerCommand reverted to its pre-run value while a long GPU run was in
flight. Add store.UpdateWorkflowRunState, a field-level $set of only
status/lastRunId/updatedAt, and switch the 5 worker run-state writes to it.
PUT edits to Nodes/Edges now survive a long-running workflow.

P7 — run.status "queued" while the pod was already running (reconcile race).
RunStatusRunning + CurrentNodeID are persisted BEFORE the k8s job is created
(async dispatch goroutine). reconcileActiveRuns had a freshness guard only on
the Queued branch; a freshly-dispatched Running run was Inspect→JobStateMissing
→ re-queued + its nodeExecution dropped, while the pod kept running (orphan,
GPU still 44-91%). Add the inverse freshness guard for the Running branch
(skip Inspect only while the active nodeExecution is fresh) and clean up
orphan pods via TerminateRun before re-queueing in the Missing branch.

- store.go/mongo.go/memory.go: UpdateWorkflowRunState field-level update
- runtime.go: Running-branch freshness guard; TerminateRun on JobStateMissing;
  5 run-state writes use UpdateWorkflowRunState; jobLauncher + testLauncher
  gain TerminateRun

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-09 16:28:37 +08:00
eust-w
220581dd79 ✨ Require explicit lightweight operator runtime 2026-07-08 18:42:31 +08:00
eust-w
d99ebf1045 ✨ Enforce operator release manifests and AI lifecycle 2026-07-02 21:19:40 +08:00
eust-w
4cdddaf512 🧹 Slim worker process-once responses 2026-07-02 20:15:27 +08:00
eust-w
8f09af0b1e ✨ Harden production workflow execution 2026-07-02 12:54:48 +08:00
eust-w
6b0dc63dcf 🐛 Fix POC runtime digest and strict readiness gates 2026-07-01 05:47:56 +08:00
eust-w
f1ed2d262f 🐛 Fix tenant-scoped audit approvals 2026-06-30 21:36:23 +08:00