📝 Record the 2026-10-20 internal test lane revision and tickets 06–15
Some checks failed
web-ci / checks (push) Has been cancelled

Recovered from uncommitted work in the dev-mainline worktree (written
2026-10-19/20, never committed). Revises ADR 0004 and the spec/map: the
lane verifies annotation → QC → final acceptance only, uses node-local
storage with no BOS credentials, and promotes the same image to
production. Adds the two domain terms and tickets 06–15. Docs only; no
code, configuration or cluster changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
d-robotics 2026-10-27 00:09:40 +08:00
parent b542d71119
commit de96cfc2be
16 changed files with 586 additions and 8 deletions

View File

@ -1,6 +1,6 @@
# 独立内部测试环境落地计划
Status: proposed
Status: superseded — 2026-10-20 ADR-0004 修订替代了专用节点、新云盘、合成素材先行、Ingress 与 5 账号/6 样本等前提;见 [spec.md](spec.md) 当前修订
Date: 2026-09-27
Scope: 本轮仅制定计划,不部署、不申请付费资源、不复制客户样本、不提交 PR。

View File

@ -25,3 +25,7 @@ Blocked by: 无(触发=用户启动执行模式/部署后线上验证)
## 验证命令
- 复现栈/测试通道浏览器手工证据记录到本文件 `## Comments`
## Comments
- 2026-10-20:内部测试环境修订后,第 1 项(无 movi 服务时的浏览器播放)由 [12](12-cluster-http-verifier.md) 的浏览器播放场景覆盖,结果作为 `video-playback` 测试环境部分的证据;本票其余内容不变。

View File

@ -0,0 +1,49 @@
# Render the environment on a pinned node with local storage and a private entry
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Environment. Code: `scripts/internal_test_lane/environment.py`, `tests/internal_test_lane/test_environment.py`.
Today the renderer creates an RWX workspace PVC and an Ingress, forces the Secure cookie and does
not pin nodes. The cluster has only `csi-bos` (no-provisioner) and no IngressClass.
- Replace the RWX PVC and Ingress with static local PVs for Mongo and workspace: `Retain`, RWO,
node affinity on `kubernetes.io/hostname`, explicit claim binding, per-batch names as today.
- Config gains `node` and `hostRoot`. Refuse an empty node, a `hostRoot` outside a dedicated
internal-test prefix, `droboticflow-fanyu`, and any production PV/PVC name.
- Pin API, processor, Mongo and initializer to the node.
- Render a one-time directory-preparation pod (pattern: `k8s/qwen-node-cache.prepare-dir.template.yaml`)
that creates only this batch's directories under `hostRoot`, owned for the Mongo user.
- Entry mode `private-forward`: ClusterIP only, no Ingress or TLS secret, and
`DROBOTICFLOW_SESSION_COOKIE_SECURE=false` only in this mode; other modes keep `true`.
- Processor runs `ANNOTATION_PROCESSOR_ROLE=all`, concurrency 1.
- Refuse any `BOS_*`, `ANNOTATION_WORKSPACE_*` or csi-bos reference.
- `internal-test-quota` keeps its stable name and fits API, processor, Mongo, initializer and the
preparation pod with finite limits.
- Keep default-deny; allow only app pods → Mongo. Whether port-forward needs an extra rule under
Cilium stays an explicit unverified item; never open namespace-external ingress to get it working.
Capacity (PV size and ResourceQuota do not limit what is actually written to a local disk, and
node `.4`'s disk is shared with kubelet and images; filling it evicts pods on that node, which can
include production operator Jobs):
- Config gains a capacity budget: bundle source bytes, an output factor for slice media and results
(default 2.0 × source until a real batch measures it), Mongo, dump, and retention of 3 archived
batches plus the current one. The renderer prints the total.
- The directory-preparation pod reports `df` for the filesystem holding `hostRoot`. The operator
imports that report; the tool refuses to prepare when free space minus the budget would leave less
than 25% of the filesystem free (kubelet hard eviction defaults to 10%). The threshold can be
raised, not set below 15%.
- The same preflight is required before seed, baseline start, upgrade and archive. A crossed
threshold means stop new work (operator scales API and processor to 0) and record it; nothing is
deleted automatically.
- Every pod gets ephemeral-storage requests and limits, so log and container-layer growth evicts the
test pod rather than its neighbours.
- Use a filesystem project quota on `hostRoot` when the node filesystem supports it. Otherwise the
batch record states "disk usage not enforced by the filesystem; bounded by preflight only".
Done when: public-boundary tests cover the rendered shape, the budget arithmetic, the threshold
refusal and each other refusal, existing tests are
updated, and the deployment section of `docs/internal-test-lane.md` matches. No cluster access.

View File

@ -0,0 +1,24 @@
# Derive test runtime config from a production snapshot
Type: task
Status: ready-for-agent
Blocked by: 06
Spec: Environment (runtime configuration).
- Input: a read-only snapshot of the production `droboticflow-api` and
`droboticflow-annotation-processor` Deployments plus the ConfigMaps they reference. Secrets are
never read; only references appear.
- Copy non-secret env from the snapshot, then apply an explicit override allowlist:
- isolation: namespace / `K8S_NAMESPACE`, Mongo URI and database (test Secret),
`WORKSPACE_MOUNT_PATH`, removal of `BOS_ENDPOINT` and every `ANNOTATION_WORKSPACE_*` key,
the private-forward cookie setting;
- capacity: replicas, `ANNOTATION_PROCESSOR_ROLE`, concurrency keys, resources.
- `DROBOTICFLOW_DATABASE_BOOTSTRAP` follows production (currently `verify-existing`).
- A reference to a production Secret or ConfigMap that the test namespace lacks is refused, not dropped.
- Emit `config-diff.json` per batch (key, class, before/after; secret refs shown as references);
`batch.json` binds its SHA256.
- Refuse snapshots from another namespace or with a missing Deployment.
Done when: tests cover the allowlist, refusal of unlisted differences and stable diff output, and
the guide explains the diff and why other differences disqualify a batch as evidence.

View File

@ -0,0 +1,28 @@
# Bundle and seed v2: production projects, two suppliers, five business roles
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Data and accounts.
Today `scripts/internal_test_lane/lifecycle.py` (around lines 155, 201, 476) requires exactly six
samples, three each for BLF-data and SL-data, and `internal/testlane/plan.go` (around lines
133–302) hardcodes six samples, two dictionaries, those two projects, one supplier and the roles
admin / supplier_manager / annotator / quality_reviewer / acceptance_reviewer.
- New schema `droboticflow.internal-test-seed/v2`: projects come from the bundle (name,
workflow-definition snapshot, three dictionaries each); 1–12 samples, at most 3 per project;
kinds `scratch`, `snapshot` at `annotation`, `snapshot` at `quality_review`.
- Suppliers `test-a` and `test-b`, both authorized on every test project; tasks start unassigned
in the shared project pool.
- Accounts: annotator and quality reviewer per supplier, plus a `test-a` acceptance reviewer with
supplier final-acceptance authorization. No platform admin, no supplier manager.
`credentials.py` emits exactly these five.
- The seeded database must start under the production bootstrap mode (`verify-existing`).
- v1 bundles exist only as local synthetic fixtures: refuse v1 with a clear message. Keep every
existing refusal (non-empty target, non-test URI, plan/manifest hash mismatch).
- `tests/internal_test_lane/make_synthetic_bundle.py` produces v2.
Done when: Go and Python tests are updated, a fresh local Mongo seeds from a synthetic v2 bundle,
and the guide documents v2.

View File

@ -0,0 +1,42 @@
# Export samples from a restored backup and download their media
Type: task
Status: ready-for-agent
Blocked by: 08
Spec: Data and accounts, run-sequence steps 1–2.
Restoring the backup (today's `scripts/local/fetch-mongo-snapshot.sh` execs into a production API
pod, stages the archive in chunks under that pod's `/tmp`, then runs `docker rm -f
fanyu-mongo-restore`, recreates 27018 and restores with `--drop`; its default backup directory is
the fixed `fanyu-v58-ca15d60d`):
- Leave that script unchanged for its existing 27018 use; add a separate restore path for the lane.
- List backup directories under `/workspace/system/backups/` and show the chosen one's name, date
and archive size. If the newest backup is older than the selection window, stop: taking a new
production backup is a separate approved operation.
- Read the archive by byte ranges straight from the mounted file, with no staging in the production
pod. If staging cannot be avoided, check the archive size against the pod's free ephemeral space
first and refuse above a threshold. Verify the full SHA256 locally.
- Restore into a new container with an explicit unique name and an explicit unused loopback port.
Refuse when the name exists or the port is in use, and refuse 27017–27020 (existing local
stacks). Never `docker rm -f` an existing container and never `--drop` into one.
Exporting:
- Connect only to an explicit loopback Mongo URI; refuse non-loopback hosts and the existing
local containers on 27017/27018.
- `candidates`: per project active in the backup's recent window (default 14 days, flag), list
tasks by stage with Capture name, part count, required media keys and sizes. No account names
or phone numbers.
- `export`: from chosen task IDs (at most 3 per project, 12 total) write a v2 bundle skeleton:
project config, workflow-definition snapshot, dictionaries, slice/action content for snapshot
kinds, and required media (both cameras, both PTS, IMU) as BOS keys.
- Download only those keys, read-only, with credentials from the untracked local file (same
approach as `scripts/local/download-capture.sh`); record size and SHA256 into bundle assets;
resume safely; never write to BOS.
- Print the total size before downloading; refuse above a cap (default 20 GiB).
Done when: tests use a synthetic Mongo fixture, a mocked BOS client and a mocked `kubectl`/`docker`
boundary that prove the refusals (existing container, busy port, staging over threshold). This ticket does not touch
a real backup or BOS.

View File

@ -0,0 +1,31 @@
# release.py: selectable policy, validator coverage, worker guard
Type: task
Status: ready-for-agent
Blocked by: none
Spec: Promotion. Code: `scripts/release.py` (policy hardcoded at line 19), `docs/release/`.
- `--policy <path>`; without it, behaviour stays on `docs/release/fanyu-policy.json`, so existing
records keep working. Historical records with another policy hash follow the existing read-only rules.
- The Fanyu policy adds `droboticflow-annotation-validator`: same image patch, its own role env
(`validation`) and resources preserved; the cluster check verifies each Deployment's role.
- New `docs/release/internal-test-policy.json`: namespace `droboticflow-internal-test`, local-PV
workspace, no bucket/BOS checks, API + processor (role `all`). Each policy refuses the other
namespace by name and UID.
- Split `annotation-delivery` in the Fanyu policy's `requiredEvidence` into `annotation-results`
and `delivery-packaging`. `delivery-packaging` is satisfied by either:
- a packaging test record from an environment that runs the worker and package operators (the
test lane cannot produce it), or
- an unchanged-delivery-inputs check that `release.py` computes itself: the diff between the
production API build commit and the candidate over a policy-declared path list is empty. The path
list (result-JSON writers, release partition, part-index delivery, package operators) is derived
from the code and reviewed with the policy. A non-empty diff requires the packaging test record.
Records under the old policy hash stay historical; an in-progress record must be prepared again.
- Worker guard: read the production worker's build commit from the read-only snapshot; refuse
the plan if `internal/worker` or `cmd/worker` differ between it and the candidate, pointing to
separate verification.
Done when: `tests/release` (simulated Kubernetes) covers policy selection, three-Deployment patch
and partial failure, the worker guard, cross-policy refusal, and both `delivery-packaging` paths
(empty diff accepted, non-empty diff without a packaging record refused), and `docs/release/README.md` is updated.

View File

@ -0,0 +1,36 @@
# Bind test-lane batch reports to release evidence
Type: task
Status: ready-for-agent
Blocked by: 10, 13
Spec: Promotion.
- `release.py evidence <id> --check <c> --test-lane-batch <batch-id>` reads the local batch
record and requires: state `business-verified`, `candidateImage` equal to the release image
digest, a config diff limited to allowed classes, and unchanged report hashes. It stores the
batch ID, digests and report hash; anything else is refused. Batches in `failed` or
`outcome-unknown` are always refused.
- Start point: for the upgrade-rehearsal part of `change-safety`, the batch's `baselineImage` must
equal the `before` image the release plan recorded for each Deployment it covers. Production may run
different starting digests (today the validator runs `7699d455…` while api and processor run
`1a1ead7a…`); a Deployment whose `before` image differs from `baselineImage` is listed as not
covered by the rehearsal.
- Each check needs named scenarios in the report, all `passed`. A scenario that is missing, failed or
unverified refuses the check.
| Release check | Required scenarios |
| --- | --- |
| `role-login` | five business-role logins; role-boundary refusals (admin logins come from production smoke) |
| `supplier-permissions` | cross-supplier claim refused; other supplier's annotation invisible |
| `review-acceptance` | QC pass; QC return → rework resubmit; final acceptance |
| `slice-recovery` | processor restart with work resumed; in-flight tasks resumed after upgrade |
| `video-playback` | browser playback for both cameras: first frame, playback advancing (test-lane part) |
| `annotation-results` | accepted results contain normal, discard, invalid and blank cases as entered |
| `change-safety` (rehearsal part) | start-point match; in-flight state created on baseline; resumed on candidate |
`delivery-packaging` cannot be satisfied by a batch (see ticket 10).
- Production-side smoke evidence (ticket 14) is still required by `verify`.
Done when: tests cover digest mismatch, start-point mismatch, a partially covered Deployment, a
missing or failed scenario, an unverified or failed batch, a tampered report and a disallowed check.

View File

@ -0,0 +1,29 @@
# HTTP business verifier for cluster batches
Type: task
Status: ready-for-agent
Blocked by: 08
Spec: Batch flow steps 2 and 4. Today `tests/internal_test_lane/verify_local.py` reads fixed
local paths and assumes one supplier.
- Inputs: loopback base URL (port-forward), the private accounts file and the executed plan.
Refuse any non-loopback host.
- Scenarios: five logins and role-boundary refusals; `test-a` claims and `test-b` can neither
claim nor see that annotation; slicing → normal / discard / invalid / blank → submit → QC return
→ rework resubmit → QC pass → final acceptance; snapshot samples; frame range.
- Browser playback with Playwright (already a dependency): for both cameras in normal and focus
layout, the first frame renders (video ready and a non-blank sampled frame) and playback time
advances within ±20% of wall-clock time over 5 s. Record what happens without a movi service; this
covers item 1 of ticket 05 for the test lane. HTTP media responses alone never pass this scenario.
- `--phase in-flight` stops after leaving claimed, slicing-pending and submission-pending tasks;
`--phase after-upgrade` resumes exactly those tasks.
- Processor-restart recovery: the tool prints the restart step for the operator, then waits and
checks the work resumes; it never restarts pods itself.
- Every scenario has a stable ID and a `passed` / `failed` / `unverified` status, matching the table
in ticket 11.
- One final JSON report: batch ID, observed build commit/digest (cross-checked with `batch.json`),
sample keys, task and item IDs, per-check status. No credentials; never overwrites.
Done when: it runs against a fresh local stack with two suppliers, and tests cover report shape
and refusals.

View File

@ -0,0 +1,26 @@
# Batch flow: baseline-to-candidate upgrade rehearsal
Type: task
Status: ready-for-agent
Blocked by: 06, 07, 10, 12
Spec: Batch flow.
- Batch config gains pinned `baselineImage` and `candidateImage`; `batch.json` binds both.
`baselineImage` comes from a read-only snapshot of the production Deployments taken at prepare
time, and the snapshot's per-Deployment images are recorded for the start-point check in ticket 11.
- States: prepared → seeded → baseline-running → in-flight-created → upgraded →
business-verified → archived. Each transition imports operator-provided evidence as today;
the tool never probes the cluster.
- The upgrade runs `release.py` with the internal-test policy, the same code path as production.
- If the candidate needs a migration that production would run as a separate reviewed step, the
rehearsal runs that same procedure in the test namespace; otherwise its change-safety part
cannot be recorded.
- `status` shows which digest ran in each phase.
- Failure path: any phase may move to `failed` or `outcome-unknown`. Either one means stop new work
(operator scales API and processor to 0), archive with that state kept (ticket 15), and never
promote the batch to `business-verified`. The next attempt is a new batch with new identities; a
contaminated batch is not resumed.
Done when: lifecycle tests cover transitions, refusals and both failure states, a local end-to-end run uses two locally
built binaries (production commit → dev-mainline HEAD), and the guide is updated.

View File

@ -0,0 +1,31 @@
# Post-promotion production smoke, no customer-data writes by default
Type: task
Status: ready-for-agent
Blocked by: 12
Spec: Promotion (production smoke).
- Uses the untracked production verification accounts file. Before any write mode, confirm those
accounts belong to the frozen internal organization.
- An HTTP method does not decide whether a call writes. Task detail GET runs interrupted-slice
recovery and can finalize a pending submission (`internal/api/annotation_operations.go`, around
line 999); frame-range GET persists a timeline index on a cache miss
(`internal/api/annotation_timeline_index.go`, around line 102). Keep an endpoint list in the tool,
each entry classified by server-side effect: none / possible write / forbidden.
- Default mode promises no customer-data writes, not zero writes: logins (admin roles when accounts
are supplied), project and task lists, opening an existing internal-organization task, browser
playback (reusing ticket 12's check), frame range. Endpoints marked "possible write" are called
only on internal-organization objects, and the report lists every such call and the object it
touched. Calls outside the list, and any call on a non-internal organization, are refused.
- BOS path: the smoke asserts that frame range loads from the BOS source of an internal-organization
task (this exercises the BOS read and HEAD). Cache invalidation after a source ETag change would
need a production object to change, so it stays unverified.
- Write mode needs `--write --confirm <token>` and explicit internal-organization task IDs:
claim → slicing → submit → QC → acceptance. Refuse customer organization IDs, report how many
internal tasks remain, never create tasks.
- Report JSON bound to the release ID and the observed build commit.
Done when: tests against a mocked HTTP server cover unlisted-endpoint refusal, organization
refusal, the side-effect report (including cold frame-range cache and an interrupted task) and the
confirm token. This ticket does not access production.

View File

@ -0,0 +1,17 @@
# Archive cluster batches to local storage
Type: task
Status: ready-for-agent
Blocked by: 06
Spec: Batch flow step 5.
- Render an archive pod in the test namespace that runs `mongodump` of the batch database into
the batch directory on the node; render the pull commands into
`local-data/internal-test-lane/batches/<batch-id>/` with a SHA256 manifest.
- Dumps count toward the capacity budget in ticket 06, and archiving requires a passing preflight.
- `archive` evidence additionally requires the pulled dump, reports and config diff with hashes.
- Retention: latest 3 archived batches plus the current one. `cleanup-plan` also lists the
node-side directories for a separately approved cluster cleanup; the tool never deletes remotely.
Done when: tests cover manifest verification and retention selection.

View File

@ -128,3 +128,82 @@ Q1–Q3 均已采用推荐项。
- [01 environment](issues/01-environment.md), [02 seed](issues/02-seed.md), [03 lifecycle](issues/03-lifecycle.md) implemented and locally verified.
- [04 integration](issues/04-integration.md): core five-role scratch/snapshot workflows verified through final acceptance; standard delivery-package worker/graph remains uncovered. See [verification and two-axis review](verification.md).
- No customer assets or production runtime changes. Company deployment, real sample copy and production publication remain separately authorized operations.
## 2026-10-19 修订讨论
背景:客户已投产,用户不能再用客户账号测试。希望基于 `codex/dev-mainline-20261019` 在线上单开测试环境,用少量真实客户样本模拟全流程;通过后用脚本把同一版本发布到生产。本节不构成部署、复制或账号创建授权。
### 2026-10-19 Round 1(Q1–Q10 全部采用推荐项)
- Q1:修订 ADR-0004 而非另起方案。三处修订:首次部署即用真实客户样本(不再先走合成素材阶段);全流程覆盖标准交付打包;测试结果作为生产发布验收证据。其余既定决策不变。
- Q2:“推送到生产”定义为**版本晋升**:测试环境运行 dev-mainline 已提交 commit 构建的不可变镜像 digest,通过后经 `scripts/release.py` 将同一 digest 发布到生产;测试数据、账号、配置、存储卷不进生产;测试批次验收报告挂入 release 记录作为写入类证据;每次生产部署仍需当次授权。
- Q3:测试环境运行配置从生产只读导出非机密项,仅覆盖隔离字段(namespace、数据库、存储、入口)与容量字段(并发 1、资源上限);每批记录与生产的配置差异,差异外不一致的项不能作为晋升证据。
- Q4:纳入标准交付打包:测试 namespace 内独立工作流 worker(仅限测试 namespace 创建 Job)、算子镜像、生产打包工作流快照;不启用 `fanyu-high-capacity` 档,打包并发 1,产物只写测试存储、不进客户交付出口。
- Q5:新建测试专用 BOS bucket,用生产只读凭证复制选中 Capture;测试环境的 csi-bos 存储卷与 AK/SK 只对测试 bucket 有权限。不只读挂载生产 bucket,不改用 local PV。
- Q6:测试 namespace 内独立 Mongo 实例(不在生产 Mongo 上另建库);从生产只读导出选中项目配置(项目、工作流定义快照、三类字典)及样本,以新身份生成测试任务副本;不复制客户账号、会话、领取、在途处理和正式审查记录。
- Q7:每个在产项目 3 条(待切片、保留切片/动作快照、质检阶段快照),总数 ≤ 12;精确清单与体积只读盘点后由用户确认。
- Q8:两个测试供应商 `test-a`、`test-b`,各含供应商管理员/标注员/质检员,另加平台管理员和最终验收员,约 8 个账号(替代原 5 账号)。
- Q9:首版仅本机 127.0.0.1 端口转发,供用户本人使用;不建公网入口、不改生产 Ingress/DNS。
- Q10:“多个 pod”指一套测试环境内多个组件,一次一个候选版本;不并行多套环境。
### 集群只读盘点(2026-10-19)
- 三个 Ready 节点:`10.189.16.33`(56 CPU / ~220 GiB / 2 GPU,requests CPU 15%、limits 37%);`10.189.16.4`、`10.189.16.5`(各 8 CPU / ~31 GiB,后者 CPU limits 已超售至 185%)。均无污点。采样时实际 CPU 使用接近 0。
- 存储类仅 `csi-bos`(no-provisioner,静态 PV);无 IngressClass。生产工作区 `droboticflow-workspace-pv` 200Gi 为 csi-bosplugin。
- `droboticflow-internal-test` namespace 于 2026-09-27 已建,仅有 `internal-test-quota`(pods 3、requests 1 CPU/2Gi、limits 3 CPU/10.5Gi、存储 20Gi、PVC 8)、`internal-test-default-deny` 网络策略及 CCE 自动镜像拉取 Secret;无工作负载。配额不足以容纳 worker 与算子 Job。
- 生产 `droboticflow-fanyu` 应用组件:api ×4 与 annotation-processor ×2(同镜像,build commit 630a95c8 = 当前 dev-mainline HEAD,processor role=revision);annotation-validator ×2(另一镜像,963b530f,role=validation,2026-10-17 建);worker ×2(worker 镜像,963b530f);drosync-coordinator ×2;GPU 服务 headpose/wilor 无就绪副本。`release.py` 策略只覆盖 api 与 processor。
- **已有生产内测试组织**:所有生产应用组件的 `ANNOTATION_WORKSPACE_MAPPINGS` 含 `org-internal-0bd5aa2e…`,读取生产 bucket `internal-test/blf-20260927-…/assets`,写 `system/runs/run-internal-*`,交付至 `internal-test/…/deliveries/TEST-internal-BLF__…`。来源为 2026-09-27 release 记录 `test-readonly-fix-20260927`(API-only 配置变更,7 个测试账号覆盖 5 角色)。这是生产库内隔离组织方案,与 ADR-0004 的独立环境方案并存,待决定其去留。
### 2026-10-19 Round 2(Q11–Q15 采用推荐项;Q16 被用户否决)
- Q11:保留既有生产内测试组织 `org-internal-…`,冻结并收窄为**晋升后生产冒烟**:每次晋升后用 1 条样本验证生产自身配置、凭证与存储;不再新增样本或项目。
- Q12:候选版本为一个 dev-mainline commit 构建的镜像;测试环境各组件 1 副本;drosync-coordinator 不在测试环境和候选范围内。(组件范围已被 2026-10-20 范围收窄修订,见下文。)
- Q13:每批先以生产当前 digest 启动并导入样本、制造在途状态,再用**同一晋升脚本**指向测试环境升级到候选版本,演练迁移与在途恢复,产出 change-safety 证据。修订 ADR-0004“一批一版本”为“一批 = 生产版本 → 候选版本”。
- Q14:测试环境固定到 `10.189.16.4`(无生产应用 Pod),不给生产节点加污点、不改生产调度;按需上调测试配额;观察到争用再申请专用节点。
- Q15:测试 Mongo 为测试 namespace 内 Pod,主版本与生产一致(待只读核实),数据放 `.4` 本地盘专用目录的静态卷。
- Q16(否决):用户无法也不愿创建 BOS 子用户或测试 bucket。
### 2026-10-20 用户范围收窄
- 本轮只测试**标注 → 质检 → 最终验收**;Capture 接入及算子处理不在本问题范围。
- 不需要平台管理员的全局管理功能。
- 用户希望直接从 BOS 读取相应文件。代码核实:该链路并非只读——标注处理器在领取后自行生成切片媒体,送检与终审发布结果文件,均写入 `WORKSPACE_MOUNT_PATH` 下;算子 Job 与 worker 确实不参与。依据 [annotation-workspace-mappings.md](../../docs/annotation-workspace-mappings.md):“只读委托仅解决素材与帧范围加载,不支持生成切片产物”。
- 由此 Round 1 Q4(纳入标准交付打包)失效;Q5、Q8、Q12、Q15 中依赖测试 bucket、worker 或平台管理员的部分待 Round 3 重新确认。
- 事实:本地 Capture 样例(单 part、480p)约 235 MB;12 条源素材约数 GB。生产 worker(963b530f)与 api(630a95c8)之间 5 个提交未改 worker 代码。
### 2026-10-20 Round 3(Q16′–Q21 全部采用推荐项)
- Q16′(替代 Q16、Q5):以本机已有 BOS 凭证、按 `scripts/local/download-capture.sh` 方式一次性只读下载选中 Capture,校验 SHA256 后上传到 `.4` 本地盘测试工作区。测试环境为纯本地工作区模式:不挂载 BOS、不持有任何 BOS 凭证,切片与结果写本地盘。bosfs 读取与 BOS ETag 版本检查不在测试环境覆盖范围,由 Q11 晋升后生产冒烟覆盖。不采用只读挂载生产 bucket,不写入生产 bucket 的 `internal-test/` 前缀。
- Q17(撤回 Round 1 Q4):不含 worker、算子 Job 与标准交付打包。测试环境组件为 API ×1、标注处理器 ×1(role `all`,并发 1)、Mongo ×1。终审通过的任务停在待交付。
- Q18(修订 Q12):晋升只更新 API 镜像的三个组件 api、annotation-processor、annotation-validator;worker 保持现版本。候选改动 `internal/worker`、`cmd/worker` 时晋升脚本拒绝并要求另行验证。需扩展 `release.py` 策略纳入 validator。
- Q19(修订 Q8):5 个业务账号——test-a 标注员与质检员、test-b 标注员与质检员、1 名归属 test-a 且开通供应商最终验收授权的最终验收员;无平台管理员与供应商管理员。项目、供应商、成员与任务由 seed 命令建立。管理类登录由生产冒烟只读检查。
- Q20:样本选择、项目配置与字典导出、Mongo 版本确认均基于 `scripts/local/fetch-mongo-snapshot.sh` 拉取的最新备份,在本地独立容器完成,不查询线上库。选样清单交用户确认。
- Q21:测试批次报告记录候选镜像 digest 与配置差异摘要;`release.py evidence` 接收测试报告时校验 digest 一致且批次为 business-verified,否则拒绝。测试环境无法产出的证据(含交付打包)保持 unverified。
### 2026-10-20 Round 4(Q22–Q25 全部采用推荐项)
- Q22:逐步单独授权,前一步通过才询问下一步:① 拉取最新 Mongo 备份到新的本地容器(不动 27018)并选样;② 确认清单后从 BOS 只读下载样本;③ 从 dev-mainline 干净 worktree 构建候选 API 镜像并推送到公司镜像仓库;④ 测试 namespace 写入(配额、`.4` 专用目录、静态卷、Mongo、上传素材、seed、删除初始化 Pod),写前复核集群与 namespace UID;⑤ 以生产当前 digest 启动、制造在途状态、用晋升脚本升级到候选并跑全流程;⑥ 按 `release.py` 流程晋升生产并做生产冒烟。
- Q23:晋升后生产冒烟默认只读(全角色登录、列表、内部组织任务、视频播放与帧范围);仅当候选改动领取、切片、送检或审查写路径时,经当次批准在内部组织剩余任务上做写入冒烟;任务耗尽时报告,不自动补充,不使用客户任务。
- Q24:归档时 `mongodump`、结果报告与配置差异摘要写 `.4` 本地盘并拉回 `local-data/internal-test-lane/batches/<batch-id>/`;保留最近 3 批加当前批;旧批次本地产物预览后确认删除;集群侧清理另列清单与授权。
- Q25:复现特定客户问题时可追加该 Capture 为样本,按 ①→②→④ 流程逐次批准,活动总数 ≤ 12;新身份,不回写来源任务。
- 前沿已清空,待用户确认达成共同理解后修订 ADR-0004 与 spec 并拆分实施票。
### 2026-10-20 共同理解确认
- 用户确认四轮决定达成共同理解。已记录:[ADR-0004 2026-10-20 修订](../../docs/adr/0004-isolated-internal-test-lane.md)、[spec.md 当前修订](spec.md)、实施票 [06](issues/06-renderer-local-node.md)–[15](issues/15-batch-archive.md)(05 为主线既有 movi 冒烟票);[部署计划](deployment-plan.md) 标记为 superseded;术语“内部测试环境”“版本晋升”写入 [CONTEXT.md](../../CONTEXT.md)。
- 依赖:06、08、10、14 无前置;07←06;09←08;12←08;15←06;13←06/07/10/12;11←10/13。既有 05(movi 直连播放冒烟,暂缓)可在本环境部署后执行。
- 未开始实施,未提交;集群、BOS、备份拉取与生产晋升均未执行,按 spec 的 Run sequence 逐步授权。
### 2026-10-20 外部审阅采纳
用户采纳外部审阅的 7 点及补充项,第 6 点选方案 A。均已代码核实并写入票据与 spec:
- 备份恢复:现有 `fetch-mongo-snapshot.sh` 会在生产 API Pod 的 `/tmp` 分块暂存归档,并删除重建 27018 容器、`--drop` 恢复;默认备份目录固定为 `fanyu-v58-ca15d60d`。改为独立容器名与端口、拒绝覆盖、按字节区间直接读取不在生产 Pod 暂存、展示备份日期,过旧则停止([09](issues/09-backup-sample-export.md))。
- 生产冒烟:任务详情 GET 会恢复中断切片并可能完成待定送检,帧范围 GET 缓存缺失时写入索引;“只允许 GET”不等于只读。改为按服务端副作用分类端点,默认承诺不写客户数据,内部组织上的副作用写入列入报告([14](issues/14-production-smoke.md),新增依赖 12)。
- 本地盘容量:预算 + `df` 预检 + 剩余 25% 停止阈值 + 临时存储上限;文件系统配额视节点支持,否则记录为仅预检约束([06](issues/06-renderer-local-node.md)、[15](issues/15-batch-archive.md))。
- 升级起点:`baselineImage` 须等于发布计划记录的各 Deployment 旧镜像;当前 validator(`7699d455…`)与 api/processor(`1a1ead7a…`)起点不同,不一致者列为演练未覆盖([11](issues/11-release-evidence-binding.md)、[13](issues/13-upgrade-rehearsal.md))。
- 检查项→必需场景映射,缺项、失败、未验证一律拒绝([11](issues/11-release-evidence-binding.md)、[12](issues/12-cluster-http-verifier.md))。
- 发布证据:`annotation-delivery` 拆为 `annotation-results` 与 `delivery-packaging`;后者由打包测试记录,或 `release.py` 自行核对策略声明的交付输入路径未改动满足([10](issues/10-release-policy-scope.md))。ADR-0004 修订已补记。
- 播放与 ETag:浏览器(Playwright)首帧与播放推进;ETag 变化后的缓存失效保持未验证;05 号票第 1 项由 12 覆盖。
- 失败路径:`failed` / `outcome-unknown` 停机、保留状态归档、重开新批次([13](issues/13-upgrade-rehearsal.md))。
- 依赖更新:14←12。

View File

@ -1,4 +1,166 @@
# Internal test lane — implementation baseline
# Internal test lane
Status: ready-for-agent
Current revision: 2026-10-20
Code baseline: `codex/dev-mainline-20261019` @ `630a95c8`
Decisions: [map.md](map.md) (2026-10-19/20, Rounds 1–4) and the 2026-10-20 revision of
[ADR-0004](../../docs/adr/0004-isolated-internal-test-lane.md). This revision authorizes local
implementation and local verification of tickets 06–15 only. Pulling a backup, downloading
from BOS, pushing images, writing to the cluster, creating accounts there and promoting to
production are separately approved steps (see Run sequence).
## User-facing outcome
Before promoting a dev-mainline candidate, the developer runs one batch in the isolated test
environment. Up to 12 real customer samples, chosen from the newest production backup and
downloaded once read-only from BOS, are claimed, sliced, annotated, submitted, reviewed and
accepted by five test business accounts in two test suppliers. The batch first runs the
production image, then upgrades to the candidate with the same promotion script used for
production. A business-verified report bound to the candidate digest becomes release evidence;
that digest is then promoted to production's API-image components and checked by a production
smoke that is read-only by default.
## Scope
In: annotation → quality review → final acceptance, including slice generation, source-index
validation, save, submission, rework return, processor restart recovery, supplier claim
exclusivity and invisibility, browser video playback (first frame, playback advancing) and frame
range from the local workspace.
Out, and recorded as unverified wherever release evidence asks for it: workflow worker,
operator Jobs, standard delivery packaging, Capture ingestion, cache invalidation after a BOS
source ETag change, load and capacity, platform-admin and supplier-admin
management.
## Environment
- Namespace `droboticflow-internal-test` (exists; only `internal-test-quota` and
`internal-test-default-deny` today). Every pod is pinned to node `10.189.16.4`; production
nodes get no taints and production scheduling is unchanged.
- Components: API ×1; annotation processor ×1 with role `all`, concurrency 1; Mongo ×1 with
the production major version; an initializer pod during import only; a one-time
directory-preparation pod.
- Storage: static local PVs on a dedicated directory of node `.4` for Mongo and workspace,
`Retain`, node affinity, RWO. No csi-bos volume, BOS endpoint, storage connection,
workspace mapping or BOS credential exists in the namespace.
- Entry: ClusterIP only, reached through `kubectl port-forward` bound to `127.0.0.1`. The
session cookie may drop `Secure` only in this explicit private-forward mode.
- Runtime configuration is derived from a read-only snapshot of the production API and
processor Deployments. Only isolation fields (namespace, database, workspace/storage,
entry/cookie) and capacity fields (replicas, processor role, concurrency, resources) are
overridden; the database bootstrap mode follows production. Each batch records the diff;
any other difference disqualifies the batch as promotion evidence.
- Quota fits API, processor, Mongo and initializer with finite CPU, memory and storage limits.
- Disk: PV size and quota do not bound real writes to the node's local disk, which kubelet and
images share. A budget (samples, slice output, Mongo, dumps, 3 retained batches plus the current
one) is checked against a `df` preflight before seed, start, upgrade and archive; below 25% free
after the budget, work stops. Pods carry ephemeral-storage limits; a filesystem quota is used where
the node supports it, otherwise the batch record says the limit is preflight-only.
## Data and accounts
- Selection and export read a backup restored into a new local container with its own name and
loopback port; the restore refuses an existing container name, a busy port and 27017–27020, and
never drops into an existing database. The archive is read by byte ranges from the backup mount,
without staging inside the production pod. The chosen backup's name and date are shown; if the
newest one is older than the selection window, work stops, because a new production backup is a
separately approved operation. The live database is never queried.
- Per in-production project, up to 3 samples: scratch (pending annotation), annotation
snapshot (existing slices and actions), quality-review snapshot. At most 12 in total. The
user confirms the final list (Capture, project, template kind, size) before download.
- Media is downloaded read-only from production BOS on the developer machine, SHA256-verified
and bundled. Project config, the workflow-definition snapshot and the three dictionaries come
from the backup. Customer accounts, sessions, leases, in-flight work, formal review records
and notification targets are not copied.
- Suppliers `test-a` and `test-b`. Accounts: annotator and quality reviewer for each supplier,
plus one final acceptance reviewer in `test-a` with supplier final-acceptance authorization.
All work identities are new; source IDs are kept only for traceability. The seeded database
must start under the production bootstrap mode.
- Reproduction samples may be added later through the same export → download → import path,
approved each time, keeping the active total at 12 or fewer.
## Batch flow
1. Prepare a batch with `baselineImage` (production's current API digest, from a read-only
Deployment snapshot that also records each Deployment's image) and `candidateImage`.
2. Seed, start on `baselineImage`, and create in-flight state: claimed tasks, slicing pending,
submission pending.
3. Upgrade to `candidateImage` with `release.py` and the internal-test policy.
4. Run the HTTP verifier and the browser playback check on the candidate: five roles, supplier
isolation, normal and rework paths, processor restart recovery. Record `business-verified` with the candidate digest and
the config diff.
5. Archive: `mongodump`, reports and config diff pulled to
`local-data/internal-test-lane/batches/<batch-id>/`. Keep the latest 3 batches plus the
current one; older local outputs are removed only after preview and confirmation. Cluster-side
cleanup is a separate approved list.
Any phase can end in `failed` or `outcome-unknown`: stop new work, archive with that state kept,
never mark the batch `business-verified`, and start over with a new batch.
## Promotion
- `release.py` takes an explicit policy; without one it keeps today's Fanyu behaviour.
- The Fanyu policy covers `droboticflow-api`, `droboticflow-annotation-processor` and
`droboticflow-annotation-validator`. The worker is not touched; promotion is refused when the
candidate differs from the production worker's build commit under `internal/worker` or
`cmd/worker`. Shared-package compatibility with the old worker stays a change-safety review item.
- The Fanyu policy's `annotation-delivery` becomes `annotation-results` (satisfiable by a batch) and
`delivery-packaging` (a packaging test record, or `release.py`'s own check that the policy-declared
delivery-input paths are unchanged between the production API commit and the candidate).
- A test-lane report counts as evidence only if its candidate digest equals the release image
digest, the batch is `business-verified`, and every scenario the requested check needs is present
and `passed` (mapping in ticket 11). For the rehearsal part of `change-safety`, `baselineImage`
must also equal each covered Deployment's `before` image in the release plan; Deployments with a
different starting digest are listed as not covered. Evidence the batch cannot produce stays
unverified.
- After promotion, the production smoke runs in the frozen internal organization and by default
writes no customer data. Some GETs write (task detail recovery, frame-range index on a cache miss),
so endpoints are classified by server-side effect, possible writes are confined to
internal-organization objects and listed in the report. A write smoke runs only when the candidate changes claim, slicing, submission or
review write paths, approved each time, on remaining internal tasks only. Customer tasks are
never used and internal tasks are never replenished automatically.
## Run sequence (each step approved separately)
1. Pull the newest backup (name and date shown) into a new local container, select samples, user
confirms the list.
2. Download the confirmed samples from BOS, read-only.
3. Build the candidate API image from a clean dev-mainline worktree and push it to the company
registry.
4. Test-namespace writes: quota, node directory, PVs/PVCs, Mongo, upload, seed, delete the
initializer. Re-verify the cluster and namespace UID first.
5. Start on the production digest, create in-flight state, upgrade, verify, record evidence.
6. Promote with `release.py` prepare → check → plan → approval → deploy → verify, then run the
production smoke.
## Verification seams
- Public-boundary tests for the renderer, lifecycle CLI, seed planner and `release.py`
(simulated Kubernetes, as today).
- Local end-to-end on a fresh Mongo port and API port (not 27017/27018/3001): a v2 bundle from
synthetic media, two suppliers, five roles, baseline → candidate upgrade, and a verifier report
bound to the digest.
- Cluster behaviour (local PV binding on `.4`, port-forward under Cilium default-deny, quota)
stays unverified until run-sequence step 4 is approved.
## Tickets
[06](issues/06-renderer-local-node.md) ·
[07](issues/07-runtime-config-parity.md) ·
[08](issues/08-bundle-seed-v2.md) ·
[09](issues/09-backup-sample-export.md) ·
[10](issues/10-release-policy-scope.md) ·
[11](issues/11-release-evidence-binding.md) ·
[12](issues/12-cluster-http-verifier.md) ·
[13](issues/13-upgrade-rehearsal.md) ·
[14](issues/14-production-smoke.md) ·
[15](issues/15-batch-archive.md)
Existing [05](issues/05-movi-smoke-and-sl3-evidence.md) (movi direct-playback smoke, deferred by the user)
fits this environment once it is deployed: it has no movi service.
## 2026-09-27 baseline (implemented; superseded where it conflicts)
Status: implemented 2026-09-27 (local synthetic verification); 2026-10-12 drift catch-up re-verified (suites + five-role HTTP flow; movi browser smoke pending), see verification.md new section and final section
Baseline: 17e7de7e (implementation); drift-reviewed 2026-10-12 at 5a956624
@ -9,7 +171,7 @@ The current turn authorizes local implementation and synthetic validation only:
no customer database, BOS assets, production deployment or live credentials are accessed.
Actual cluster provisioning and the six real sample transfers remain separately approved operations.
## User-facing outcome
### User-facing outcome
An internal developer can prepare a small, independently runnable environment
using the same application/business UI, five isolated role accounts and a fixed
@ -17,7 +179,7 @@ six-source sample pack (three BLF-data, three SL-data). They can reproduce a
known state or start from slicing, archive results and explicitly prepare the
next batch/version without putting customer work into a test queue.
## Delivery components
### Delivery components
1. Offline environment renderer and configuration validation. Fixed test namespace,
distinct host, authenticated dedicated Mongo, per-batch database and PVCs,
@ -35,7 +197,7 @@ next batch/version without putting customer work into a test queue.
prepared, seeded and business-verified states; no claim of deployed/accepted
solely because manifests were rendered or records inserted.
## Explicit behavior
### Explicit behavior
- Test supplier `test`; platform admin plus supplier manager, annotator, quality
reviewer and acceptance reviewer. Customer identities are not cloned.
@ -66,7 +228,7 @@ next batch/version without putting customer work into a test queue.
- Same immutable app image is used for eventual promotion, with production runtime
settings retained. Candidate success does not authorize production release.
## Confirmed verification seams
### Confirmed verification seams
The user explicitly accepted these on 2026-09-27:
@ -79,7 +241,7 @@ typecheck/build, then full relevant suites at the end. Review against baseline
17e7de7e along standards and specification axes. Existing full-suite failures
must be reported separately; do not claim a wholly green repository.
## Current-turn acceptance
### Current-turn acceptance
Local commands, manifests, initialization, lifecycle and synthetic workflow evidence
are runnable and documented. No company access is required to demonstrate refusal
@ -87,7 +249,7 @@ checks and business behavior. Real sample inventory, exact company allocation,
external ingress/storage prerequisites, cluster deployment and capacity validation
remain explicitly unverified until separately authorized; do not fabricate them.
## 2026-10-12 漂移复核与追进度工单(用户批准落档)
### 2026-10-12 漂移复核与追进度工单(用户批准落档)
9-27 实现完整存在且未被后续演进破坏:Go 包测试 2026-10-12 重跑全绿(`go test ./internal/testlane/...`),种子命令 `cmd/internal-test-seed` 与 `scripts/internal_test_lane.py` 冒烟正常,五角色账号模型与当前 `internal/api/auth.go` 服务端校验完全一致;实现后零维护提交、零破坏。但距验证基线 `17e7de7e` 约 174 个提交已进入主线,差距如下:

View File

@ -177,6 +177,12 @@ _Avoid_:客户公共池(内部测试不参与客户任务的竞争领取)
**测试任务副本**:以经批准的客户样本为来源、拥有独立身份和处理历史的内部测试任务;副本的修改不回写来源任务。
_Avoid_:供应商任务副本(不代表给每家生产供应商复制客户任务)
**内部测试环境**:与客户生产隔离、一次运行一个候选版本的完整平台,用经批准的少量客户样本模拟客户生产处理流程;其中的账号、任务、审查和交付不计入客户生产、统计或交付出口。
_Avoid_:测试 pod(一套环境包含多个组件)、生产内测试项目(与客户共用业务数据,不构成环境隔离)
**版本晋升**:把已在内部测试环境验收的同一候选版本发布到客户生产;只带走版本本身,不带走测试数据、账号或配置,每次发布仍需单独授权。测试通过不等于获准发布。
_Avoid_:推送 pod、迁移测试环境
**撤销供应商项目授权**:停止供应商领取该项目的新任务,但保留其既有任务的标注、质检与返工处理资格,直至存量完成;不等于停用账号或释放已领取任务。
**生产统计异常**:仅指当前正式处理中的执行故障,包括运行失败、源采集处理失败、切片或送检处理作业失败、交付打包失败。供应商归属不同、部分未领取、归属待核对、审核退回、业务升级处理及正常等待不单独计为异常;这些状态保留在明细,不增加红色异常数量。

View File

@ -15,3 +15,17 @@ status: accepted
Q7–Q13 随后明确:必要素材复制到测试存储;5 个独立测试角色账号;固定 6 份样本模板,最多 6 条活动测试任务,手动重建;切片并发 1,资源不足暂停;受控命令管理,按批次切换版本,保留最近 3 批结果并在确认后清理更旧测试产物。
本决策不授权创建账号、复制样本、部署或修改客户数据。具体来源清单、资源额度、账号使用者和执行审批见[需求讨论](../../.scratch/internal-test-lane/map.md),完整方案确认后再推进实施准备。
## 2026-10-20 修订
客户已投产,内部不能再借客户账号验证;候选版本须先在测试环境走完人工标注链路,再把同一镜像晋升到生产。以下决定替代上文及 2026-09-27 部署计划中与之冲突的部分,逐条依据见需求讨论中 2026-10-19/20 四轮记录。
- **范围**:只验证标注 → 质检 → 最终验收。工作流 worker、算子 Job、标准交付打包和 Capture 接入不在测试环境中运行;对应发布证据保持未验证。
- **存储**:不创建 BOS 子用户或测试 bucket。选中样本从生产 BOS 一次性只读下载,导入测试节点本地盘;测试环境是纯本地工作区,不挂载 BOS、不持有任何 BOS 凭证,切片与结果只写本地盘。未采用只读挂载生产 bucket:测试命名空间须持有全桶读写凭证,隔离只剩只读挂载和应用前缀检查。未采用写入生产 bucket 的 `internal-test/` 前缀:测试产物会进入客户生产存储。代价是 BOS 读取与 ETag 版本检查只能在晋升后的生产冒烟中验证。
- **资源**:固定在当前没有生产应用 Pod 的节点 `10.189.16.4`,不新增节点、不改生产调度;Mongo 与工作区用该节点本地盘上的静态卷。生产算子 Job 仍可能调度到该节点,观察到争用再申请专用节点。
- **样本与账号**:从最新生产备份选样,不查询线上库;每个在产项目 3 条、总数不超过 12。两个测试供应商、5 个业务角色账号,不设平台管理员和供应商管理员,用以覆盖多供应商领取与不可见。
- **版本**:每批先以生产当前镜像启动并制造在途状态,再用与生产相同的晋升脚本升级到候选版本。一批对应“生产版本 → 候选版本”,替代上文“按批次一次运行一个版本”。
- **晋升**:只晋升到使用同一镜像的 api、annotation-processor、annotation-validator;worker 不随之更新,候选改动 worker 代码时拒绝晋升。测试报告只有与本次发布镜像 digest 一致时才作为发布证据。原 `annotation-delivery` 证据拆为标注结果与交付打包两项:前者可由测试批次满足,后者需要打包测试,或经核对候选版本未改动交付输入。
- **生产内测试组织**:2026-09-27 起生产库中的内部测试组织冻结,只用于晋升后的生产冒烟,默认只读。
每次执行仍逐步单独授权:拉取备份、下载样本、构建推送镜像、写入测试命名空间、启动与升级演练、晋升生产。