diff --git a/.scratch/verification-flow/work.md b/.scratch/verification-flow/work.md new file mode 100644 index 00000000..a56eaa28 --- /dev/null +++ b/.scratch/verification-flow/work.md @@ -0,0 +1,8 @@ +# Targeted development verification + +Goal: implement the approved fast development / task / integration verification workflow. +Agent: Codex, current unified-worktree conversation +Base: codex/review-focus-layout-stability @ c8aa1ae7; continue the user's unified worktree without moving other work. +Scope: AGENTS.md, verification guidance, npm aliases, local selector/runner and its tests. +Out of scope: changing release policy, bypassing approvals, production actions or capacity acceptance. +Risk: false-positive verification; test unmapped changes, skips, empty tests, timeouts and affected-scope expansion. diff --git a/AGENTS.md b/AGENTS.md index 985ca8bb..b6370370 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -80,11 +80,17 @@ These principles guide implementation choices. Specific business contracts and t - Deploying to any company cluster requires the user's explicit approval immediately before the deployment; verification work must stop when that approval is absent. - Cluster verification is read-only against customer data: use isolated synthetic fixtures or existing records without mutation, and never delete, overwrite, or add customer data. If a test requires a write or destructive operation, stop and obtain explicit approval for that exact operation. +## Development verification + +- Before choosing checks for a change, read `docs/agents/verification.md`. Start with `npm run verify:plan -- --base `; use quick checks while editing and affected task checks once before delivery. Expand only for changed dependencies, failures or uncovered risk. +- Reuse unchanged evidence within a task using the recorded source, dependency, configuration and environment identities. Report skipped, timed-out and unmapped coverage as pending; quick checks never replace release evidence. +- Keep local implementation and verification moving within the authorized scope. The existing push, PR, production-data and deployment approval rules still apply. + ## Local Environment - Local replica stack (Vite 5173, Go API 3001, snapshot Mongo 27018) is the default verification surface before any PR. - Verification commands and known pitfalls (upstream test debt, session auth flag, workspace mount) are documented in `docs/HANDOFF.md`. -- Comparing test failures against the known-failure baseline ALWAYS goes through `npm run test:frontend:gate` / `npm run test:go:gate` (they diff against `tests/known-failures.json`; only NEW failures fail). Never hand-diff failure lists with stash/scratch files. Formatting gate: `npm run format:check`. +- Comparing test failures against the known-failure baseline ALWAYS goes through `npm run test:frontend:gate` / `npm run test:go:gate` (they diff against `tests/known-failures.json`; only NEW failures fail). Never hand-diff failure lists with stash/scratch files. Quick edits check changed-file formatting; the full formatting gate remains `npm run format:check` at integration/release scope. - The company cluster and BOS bucket are read-only by default; any write requires explicit user approval. ## Agent skills diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 41ebd522..be445728 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -19,6 +19,10 @@ - 质检员 `fanyu-full-qc@droboticflow.local`、验收员 `fanyu-full-acceptance@droboticflow.local`、管理员 `fanyu-admin@droboticflow.local` - **红线**:27018/27017 都是本地容器,随便写;公司集群(kubeconfig `local-data/fanyu.conf`)和 BOS 桶默认只读,任何写操作需用户明确确认。 +## 日常验证入口 + +开发时先按[分层验证流程](agents/verification.md)选择检查。`verify:plan` 查看范围,`verify:quick` 用于编辑反馈,`verify:task` 用于本批业务流程验收;完整集成与发布检查独立执行。不要每次修改都重复本手册的所有验证命令。 + ## 1. 前端(5173)怎么改、怎么验证 **改动即生效**:Vite dev server 直连 `src/` 源码,保存文件浏览器自动热更(HMR)。没有"重新构建前端"这一步。 diff --git a/docs/agents/verification.md b/docs/agents/verification.md new file mode 100644 index 00000000..a41c0533 --- /dev/null +++ b/docs/agents/verification.md @@ -0,0 +1,45 @@ +# 分层验证 + +适用于日常功能开发、修复及本批改动交付。发布仍使用 [release.py 流程](../release/README.md)。 + +## 工作步骤 + +1. 开工时用几句话确定业务结果、兼容范围、任务基线和风险。普通小改动不另写长计划;在已授权范围内连续完成实现、验证、修正。 +2. `npm run verify:plan -- --base <任务起点提交>` 查看范围。默认 `HEAD` 只包含未提交改动;交付已提交任务时必须指定任务起点,不能用干净树的空计划证明已提交代码通过。 +3. 编辑过程中运行 `npm run verify:quick -- --base <任务起点提交>`。相关单元测试被合并、去重执行;不默认启动浏览器、Mongo 或长时压测。 +4. 本批完成后运行 `npm run verify:task -- --base <任务起点提交>`,集中验证受影响的真实业务流程。失败只修复并重跑有关部分;新改动影响其他范围时扩大验证。 +5. 跨模块集成或公共依赖变化时运行 `verify:integration`。最终报告列出通过、失败、待验证和日志路径;部署单独执行发布门禁。 + +命令支持 `--plan`(不执行)及 `--budget-seconds N`。默认执行预算:quick 180 秒、task 600 秒、integration 3600 秒;预算耗尽会停止进程组并报告待验证,不自动重跑或改成通过。冷编译、安装依赖和机器负载会影响耗时,预算不是延迟承诺。 + +## 风险与完成标准 + +| 变更 | 开发检查 | 本批完成还需确认 | +| --- | --- | --- | +| 文档、纯样式 | 差异/格式检查 | 界面样式实际查看正常与错误状态;文案遵循现有最简规则 | +| 交互、状态和页面切换 | 类型、有关单元回归 | 一条覆盖进入、操作、退出的真实浏览器流程 | +| 领取、权限、保存、校验、返工 | 相关接口和规则回归 | 隔离存储、冲突、失败恢复及旧数据兼容 | +| 查询、轮询、缓存 | 结果等价回归 | 相同代表性规模的查询量、扫描量、耗时、取消及并发对照 | +| 工作流、迁移、资源或容量 | 专项验证 | 在途任务、回退和恢复;生产写入按具体授权执行 | + +风险取决于行为和影响面,不按行数判断。共享工作台页面可能同时影响多个流程,脚本会扩大映射;确认只是文案/样式时可记录实际差异与人工检查,执行有关检查并说明未运行项,不把整批列为全通过。 + +120 人完整作业、8 小时运行、媒体容量和节点故障属于专项验收:容量承诺、相关架构或资源变更时执行,不绑定每次普通功能提交。发布基础门禁未因本规则减少。 + +## 选择与兜底 + +- 映射的唯一配置为 `scripts/verification-scopes.json`;当前覆盖审查切换、源校验/个人池、搜索分页、标注编辑、审核决定和统计面板。 +- Git 检测包含起点以来提交、暂存、未暂存、未跟踪文件,重命名检查新旧路径。改动的前端测试和 Go 同名测试自动加入。CSS 不因单纯样式修改启动全套类型/业务测试。 +- 未映射的前端代码扩大到前端基线门禁,未映射 Go 实现扩大到所属包;浏览器流程仍需人工核对。其他未映射行为报告待验证,补充映射或专项证据后才算交付完成。 +- 退出码 0 仅表示所选检查通过;1 表示检查失败;2 表示超时、跳过、未映射、执行中源码变化或入口异常。报告中的 deferred/notCovered 始终保留,不把快速通过解释成完整通过。 +- 遇到已有测试债务,使用既有 `test:frontend:gate` / `test:go:gate` 比较具体用例。不因失败而扩大 known-failures 基线,不手工忽略断言。 + +## 减少重复与环境准备 + +- 同一批同一输入只做一次成功验证;优先复用已有 npm 测试组、Go 编译/测试缓存和本地隔离 Mongo 服务,不为每次小改动重装依赖、重建镜像或重启整套副本。 +- 浏览器/数据库测试串行运行,继续使用各自独立合成数据和清理机制。共享服务不等于共享业务状态;公司集群和 BOS 不用于这些写入测试。 +- 日志及 JSON 报告写入忽略目录 `local-data/test-results/verify-*`,包含命令、基线、源码摘要、环境摘要、工具版本、安装锁文件摘要、耗时和未覆盖项。原始环境值不写入报告。 +- 同任务内复用记录前,核对源码/测试/夹具、依赖安装、配置、工具版本、测试环境均未变化。源码摘要和环境摘要不是依赖安装或外部数据状态的证明。此入口不做跨运行的自动“免测缓存”;Go 自有缓存正常使用。浏览器、数据库和生产证据按各自环境重新核验。 +- 更换依赖、修复失败、修改夹具、出现新问题或范围扩大时,重跑相关检查。测试报错、跳过和未启动永远不是可复用的成功证据。 + +此流程不增加日常审批。普通本地修改、测试及本地提交持续推进;推送、PR、公司部署、生产数据和资源修改仍遵循 `AGENTS.md` 的授权边界。 diff --git a/docs/release/README.md b/docs/release/README.md index 0f087434..c3972ae9 100644 --- a/docs/release/README.md +++ b/docs/release/README.md @@ -1,5 +1,8 @@ # 泛宇发布试行流程 +日常开发按[分层验证流程](../agents/verification.md)执行,不在每次编辑后重复本发布流程。 +本页发布门禁及 `fanyu-policy.json` 保持独立;快速检查报告不能作为缺失发布证据的替代。 + ## 开始或恢复工作 发布、部署故障、回退、上下文恢复时先读本页。状态的权威来源是 diff --git a/package.json b/package.json index e2d54bc5..593e3a80 100644 --- a/package.json +++ b/package.json @@ -93,7 +93,12 @@ "test:review:decisions": "node --test tests/frontend/review-decision-controller.test.mjs tests/frontend/review-decision-completion.test.mjs tests/frontend/rework-resubmission.test.mjs", "test:browser:review-decisions": "REVIEW_DECISION_BROWSER_TEST=1 go test ./internal/api -run '^TestReviewDecisionControllerBrowserReplica$' -count=1 -v && node tests/browser/review-decision-failure.mjs && node tests/browser/review-decision-start-race.mjs", "test:review:drafts": "node --test tests/frontend/review-draft-persistence.test.mjs tests/frontend/rework-verdict-keys.test.mjs tests/frontend/review-draft-autosave.test.mjs tests/frontend/acceptance-legacy-compat.test.mjs", - "test:browser:review-drafts": "REVIEW_DRAFT_BROWSER_TEST=1 go test ./internal/api -run '^TestReviewDraftPersistenceBrowserReplica$' -count=1 -v" + "test:browser:review-drafts": "REVIEW_DRAFT_BROWSER_TEST=1 go test ./internal/api -run '^TestReviewDraftPersistenceBrowserReplica$' -count=1 -v", + "verify:plan": "python3 scripts/verify-change.py --plan", + "verify:quick": "python3 scripts/verify-change.py --mode quick", + "verify:task": "python3 scripts/verify-change.py --mode task", + "verify:integration": "python3 scripts/verify-change.py --mode integration", + "test:verification": "python3 -m unittest discover -s tests/tooling -p test_verification.py" }, "dependencies": { "@tailwindcss/vite": "^4.1.14", diff --git a/scripts/verification-scopes.json b/scripts/verification-scopes.json new file mode 100644 index 00000000..bc761e9c --- /dev/null +++ b/scripts/verification-scopes.json @@ -0,0 +1,249 @@ +{ + "version": 1, + "scopes": [ + { + "name": "review-navigation", + "paths": [ + "^src/(pages/AnnotationOperations\\.tsx|hooks/useReviewContinuation\\.ts|services/api/review-continuation\\.ts)$", + "^internal/api/review_(continuation|trace)", + "^tests/browser/review-trace\\.mjs$" + ], + "frontend": [ + "tests/frontend/annotation-project-loading.test.mjs" + ], + "go": { + "./internal/api": [ + "^TestReviewContinuation" + ] + }, + "task": [ + { + "argv": [ + "go", + "test", + "./internal/api", + "-run", + "^TestReviewTraceBrowserReplica$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "REVIEW_TRACE_BROWSER_TEST": "1" + }, + "tests": true + } + ] + }, + { + "name": "source-and-buffer", + "paths": [ + "^src/(pages/AnnotationOperations\\.tsx|services/api/annotation-operations\\.ts)$", + "^internal/(api|store)/annotation_(source|work_buffer|claim_read|claim_visit|random_claim)", + "^internal/store/human_claim", + "^tests/browser/annotation-source-validation-replica\\.mjs$" + ], + "frontend": [ + "tests/frontend/annotation-buffer-*.test.mjs", + "tests/frontend/annotation-source-validation-ui.test.mjs" + ], + "go": { + "./internal/api": [ + "^TestSourceValidation", + "^TestPendingSourceValidation", + "^TestWorkBufferRetains", + "^TestRandomClaim" + ], + "./internal/store": [ + "^TestSourceAdmissionMemoryClaimFence$" + ] + }, + "task": [ + { + "argv": [ + "go", + "test", + "./internal/store", + "-run", + "^TestSourceAdmissionMongo(ClaimFence|BatchMatchesLegacyPipeline)$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "SOURCE_ADMISSION_LOCAL_MONGO": "1" + }, + "tests": true + }, + { + "argv": [ + "go", + "test", + "./internal/api", + "-run", + "^TestSourceValidationBrowserReplica$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "SOURCE_VALIDATION_BROWSER_TEST": "1" + }, + "tests": true + } + ], + "warnings": [ + "涉及准入/轮询:核对真实规模下查询次数、扫描量、超时与并发;局部回归不等于容量通过。" + ] + }, + { + "name": "pool-search", + "paths": [ + "^src/pages/AnnotationOperations\\.tsx$", + "^src/.*(pool|Pool|human-loop)", + "^internal/(api|store)/(human_pool|admin_annotation_pool|human_loop)", + "^tests/browser/(review-pool-search|admin-annotation-pool)" + ], + "frontend": [ + "tests/frontend/annotation-pool-*.test.mjs", + "tests/frontend/review-pool-preview.test.mjs" + ], + "go": { + "./internal/api": [ + "^TestHumanPoolSearch", + "^TestHumanPoolQCProjectScope" + ] + }, + "task": [ + { + "argv": [ + "go", + "test", + "./internal/api", + "-run", + "^TestReviewPoolSearchBrowserReplica$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "REVIEW_POOL_BROWSER_TEST": "1" + }, + "tests": true + } + ] + }, + { + "name": "annotation-editing", + "paths": [ + "^src/(pages/AnnotationOperations\\.tsx|components/CaptureAnnotationWorkbench\\.tsx)$", + "^src/.*(slice-edit|SliceEdit|annotation-persistence)", + "^internal/api/(annotation_draft|annotation_slice|slice_recovery|slice_edit)", + "^tests/browser/(rework-boundary|slice-edit)" + ], + "frontend": [ + "tests/frontend/annotation-persistence.test.mjs", + "tests/frontend/slice-edit-session.test.mjs", + "tests/frontend/annotation-authoritative-mutation.test.mjs" + ], + "go": {}, + "task": [ + { + "argv": [ + "go", + "test", + "./internal/api", + "-run", + "^TestReworkBoundaryBrowserReplica$", + "-count=1", + "-timeout=3m", + "-v" + ], + "env": { + "REWORK_BOUNDARY_BROWSER_TEST": "1" + }, + "tests": true + } + ] + }, + { + "name": "review-decisions", + "paths": [ + "^src/(pages/AnnotationOperations\\.tsx|components/CaptureAnnotationWorkbench\\.tsx)$", + "^src/.*([Rr]eview[Dd]raft|review-draft|[Rr]eview[Dd]ecision|review-decision)", + "^internal/api/review_(draft|decision|confirmation)", + "^tests/browser/review-(draft|decision|authoring)" + ], + "frontend": [ + "tests/frontend/review-decision-controller.test.mjs", + "tests/frontend/review-decision-completion.test.mjs", + "tests/frontend/review-draft-persistence.test.mjs", + "tests/frontend/review-draft-autosave.test.mjs" + ], + "go": {}, + "task": [ + { + "argv": [ + "npm", + "run", + "test:browser:review-drafts" + ], + "tests": true + }, + { + "argv": [ + "npm", + "run", + "test:browser:review-decisions" + ], + "tests": true + }, + { + "argv": [ + "npm", + "run", + "test:browser:review-permissions" + ], + "tests": true + } + ] + }, + { + "name": "dashboards", + "paths": [ + "^src/.*([Dd]ashboard|WorkforceStatistics|annotation-output)", + "^internal/(api|store)/(production_(dashboard|stats)|annotation_output|workforce)", + "^internal/(outputsnapshot|workforcestats)/", + "^tests/browser/.*dashboard" + ], + "frontend": [ + "tests/frontend/dashboard-presentation.test.mjs", + "tests/frontend/production-quality-dates.test.mjs" + ], + "go": { + "./internal/api": [ + "^TestProductionDashboardScopeIsolationAndFullCorpus$" + ] + }, + "task": [ + { + "argv": [ + "npm", + "run", + "test:browser:dashboard" + ], + "tests": true + }, + { + "argv": [ + "python3", + "scripts/test_dashboard_stability.py" + ] + } + ], + "warnings": [ + "统计/查询改动须额外记录同数据规模的性能对照;完整 120 人容量另行验收。" + ] + } + ] +} diff --git a/scripts/verify-change.py b/scripts/verify-change.py new file mode 100644 index 00000000..7ff8e4e8 --- /dev/null +++ b/scripts/verify-change.py @@ -0,0 +1,290 @@ +#!/usr/bin/env python3 +"""Select local development checks; never substitutes for release.py evidence.""" +import argparse +import ast +import hashlib +import json +import os +from pathlib import Path +import re +import signal +import subprocess +import sys +import time + +ROOT = Path(__file__).resolve().parents[1] +POLICY = ROOT / 'scripts/verification-scopes.json' + + +def git(root, *args): + return subprocess.check_output(['git', *args], cwd=root, stderr=subprocess.PIPE).decode().strip('\n') + + +def changed_files(root, base): + revision = git(root, 'rev-parse', '--verify', base + '^{commit}') + changed = git(root, 'diff', '--name-only', '--no-renames', '-z', revision, '--').split('\0') + untracked = git(root, 'ls-files', '--others', '--exclude-standard', '-z').split('\0') + return revision, sorted(set(filter(None, changed + untracked))) + + +def command(argv, env=None, tests=False): + return {'argv': argv, 'env': env or {}, 'tests': tests} + + +def tooling_package_change(root, base): + try: + old = json.loads(git(root, 'show', base + ':package.json')) + new = json.loads((root/'package.json').read_text()) + except (OSError, ValueError, subprocess.CalledProcessError): + return False + old_scripts, new_scripts = old.pop('scripts', {}), new.pop('scripts', {}) + names = {name for name in set(old_scripts) | set(new_scripts) if old_scripts.get(name) != new_scripts.get(name)} + allowed = {'verify:plan', 'verify:quick', 'verify:task', 'verify:integration', 'test:verification'} + return old == new and bool(names) and names <= allowed + + +def plan_checks(root, files, policy, mode, base, tooling_package=False): + quick = [command(['git', 'diff', '--check', base, '--'])] + task = [] + warnings = [] + unresolved = [] + scopes = [] + frontend = set() + go_patterns = {} + matched = set() + for scope in policy['scopes']: + hits = [f for f in files if any(re.search(p, f) for p in scope['paths'])] + if not hits: + continue + scopes.append(scope['name']) + matched.update(hits) + for pattern in scope.get('frontend', []): + found = list(root.glob(pattern)) + if not found: + unresolved.append('测试映射不存在: ' + pattern) + frontend.update(str(p.relative_to(root)) for p in found) + for pkg, patterns in scope.get('go', {}).items(): + go_patterns.setdefault(pkg, set()).update(patterns) + task.extend(scope.get('task', [])) + warnings.extend(scope.get('warnings', [])) + web = any((f.startswith('src/') and not f.endswith('.css') or re.match(r'(package(-lock)?\.json|tsconfig.*|vite\.config.*)$', f)) and not (f == 'package.json' and tooling_package) for f in files) + if web: + quick.append(command(['npm', 'run', 'lint:web'])) + formatting = [f for f in files if (root/f).is_file() and + (f.startswith('src/') and re.search(r'\.(ts|tsx|css|js)$', f) or + f.startswith('tests/frontend/') and f.endswith('.mjs'))] + if formatting: + quick.append(command(['node', 'node_modules/prettier/bin/prettier.cjs', '--check', *formatting])) + go_files = [f for f in files if f.endswith('.go') and (root/f).is_file()] + if go_files: + quick.append(command([sys.executable, 'scripts/verify-change.py', '--gofmt', *go_files])) + for f in files: + path = root/f + if f.startswith('tests/frontend/') and f.endswith('.test.mjs') and path.exists(): + frontend.add(f) + continue + if f.endswith('.go'): + pkg = './' + str(Path(f).parent) + test_file = path if f.endswith('_test.go') else path.with_name(path.stem + '_test.go') + fast = [] + if test_file.exists(): + names = re.findall(r'^func (Test\w+)\(', test_file.read_text(), re.M) + fast = [name for name in names if not re.search(r'Browser|Mongo|Capacity|Soak|Load|Local|Benchmark', name)] + if fast: + go_patterns.setdefault(pkg, set()).update('^' + name + '$' for name in fast) + if len(fast) != len(names) and f not in matched: + unresolved.append(f + ': 外部环境测试需要显式任务级映射,不能将跳过当通过。') + if len(fast) != len(names): + warnings.append(f + ': 外部环境/容量测试不计入快速检查;核对任务级覆盖。') + if f not in matched and (not f.endswith('_test.go') or not fast): + # Package-wide fallback is explicit at task level, not every edit. + task.append(command(['go', 'test', pkg, '-timeout=5m', '-v'], tests=True)) + continue + if f in matched or f.endswith('.md') or f.startswith('docs/') and f.endswith(('.png', '.jpg', '.svg')): + continue + if f.startswith('src/') and f.endswith('.css'): + warnings.append(f + ': 样式改动需人工查看受影响页面及异常状态。') + continue + if f.startswith('src/'): + task.append(command(['npm', 'run', 'test:frontend:gate'])) + warnings.append(f + ': 未建立业务映射,任务级扩大到前端基线门禁;仍需人工确认浏览器流程。') + continue + if f.startswith('tests/browser/'): + unresolved.append(f + ': 为该浏览器夹具补充可运行的任务级映射。') + continue + if f == 'package.json' and tooling_package: + quick.append(command([sys.executable, '-m', 'unittest', 'discover', '-s', 'tests/tooling', '-p', 'test_verification.py'], tests=True)) + continue + if f in ('package.json', 'package-lock.json', 'go.mod', 'go.sum') or re.match(r'(tsconfig|vite\.config)', f): + task.extend([command(['npm', 'run', 'test:frontend:gate']), command(['npm', 'run', 'test:go:gate']), + command(['npm', 'run', 'build:web']), command(['npm', 'run', 'test:production:boot'])]) + warnings.append('依赖/公共配置变化:任务级扩大到集成检查,核对安装与锁文件。') + continue + if f in ('scripts/verify-change.py', 'scripts/verification-scopes.json') or f.startswith('tests/tooling/'): + quick.append(command([sys.executable, '-m', 'unittest', 'discover', '-s', 'tests/tooling', '-p', 'test_verification.py'], tests=True)) + continue + if path.exists() and f.endswith('.py'): + quick.append(command([sys.executable, 'scripts/verify-change.py', '--python-syntax', f])) + elif path.exists() and f.endswith('.mjs'): + quick.append(command(['node', '--check', f])) + elif path.exists() and f.endswith('.sh'): + quick.append(command(['bash', '-n', f])) + unresolved.append(f + ': 尚无行为验证映射;补充映射/专项证据,语法通过不代表功能通过。') + if frontend: + quick.append(command(['node', '--test', *sorted(frontend)], tests=True)) + for pkg, patterns in sorted(go_patterns.items()): + quick.append(command(['go', 'test', pkg, '-run', '|'.join(sorted(patterns)), '-timeout=2m', '-v'], tests=True)) + integration = [command(['npm', 'run', x]) for x in + ['lint:web', 'format:check', 'test:frontend:gate', 'test:go:gate', 'build:web', 'test:production:boot']] + def unique(commands): + out, seen = [], set() + for c in commands: + c = {**command(c['argv']), **c} + key = json.dumps(c, sort_keys=True) + if key not in seen: + out.append(c) + seen.add(key) + return out + quick, task = unique(quick), unique(task) + # Full suite gates subsume targeted frontend/Go tests, not tooling checks. + integration_extra = [c for c in quick if c['argv'][:2] not in (['node', '--test'], ['go', 'test']) + and c['argv'] != ['npm', 'run', 'lint:web'] + and 'node_modules/prettier/bin/prettier.cjs' not in c['argv']] + selected = quick if mode == 'quick' else unique(integration_extra + task + integration) if mode == 'integration' else unique(quick + task) + return {'mode': mode, 'files': files, 'scopes': scopes, 'commands': selected, + 'deferred': task if mode == 'quick' else [], 'unmapped': sorted(set(unresolved)), + 'notes': sorted(set(warnings)), 'notCovered': [ + '生产业务写验收、生产数据处置及正式发布门禁', '120 人完整作业、8 小时运行、节点故障及真实媒体容量', + '业务影响需人工核对;文件映射不能证明所有间接依赖均已覆盖']} + + +def fingerprint(root): + digest = hashlib.sha256() + paths = git(root, 'ls-files', '-z', '--cached', '--others', '--exclude-standard').split('\0') + for name in sorted(set(filter(None, paths))): + p = root/name + digest.update(name.encode()) + if p.is_symlink(): + digest.update(os.readlink(p).encode()) + elif p.is_file(): + digest.update(p.read_bytes()) + else: + digest.update(b'') + return digest.hexdigest() + + +def run_check(check, root, logfile, timeout): + start = time.monotonic() + env = dict(os.environ) + # Avoid accidentally enabling unrelated opt-in suites in an inherited shell. + for key in list(env): + if key.startswith(('TEST_', 'RUN_')) or key in ('MONGO_URI', 'DATABASE_URL') or re.search(r'(_BROWSER_TEST|_LOCAL_MONGO|_MONGO_HTTP|_MONGO_STORE|_LOAD_TEST|_CAPACITY_TEST)$', key): + env.pop(key) + env.update(check['env']) + status, code = 'failed', None + with logfile.open('w') as stream: + try: + child = subprocess.Popen(check['argv'], cwd=root, env=env, stdout=stream, + stderr=subprocess.STDOUT, start_new_session=True) + try: + code = child.wait(timeout=timeout) + status = 'passed' if code == 0 else 'failed' + except (subprocess.TimeoutExpired, KeyboardInterrupt): + status = 'incomplete' + try: + os.killpg(child.pid, signal.SIGTERM) + except ProcessLookupError: + pass + try: + child.wait(timeout=2) + except subprocess.TimeoutExpired: + pass + # Kill lingering grandchildren even if the parent exited on TERM. + try: + os.killpg(child.pid, signal.SIGKILL) + except ProcessLookupError: + pass + child.wait() + except OSError as error: + stream.write(str(error)) + output = logfile.read_text() + if status == 'passed' and check.get('tests'): + if re.search(r'--- SKIP:|\[no test files\]|no tests to run|# SKIP|# skipped [1-9]|ℹ skipped [1-9]|skipped=[1-9]', output): + status = 'incomplete' + if re.search(r'Ran 0 tests|# tests 0|ℹ tests 0', output): + status = 'incomplete' + return {**check, 'status': status, 'exitCode': code, 'seconds': round(time.monotonic()-start, 3), 'log': str(logfile)} + + +def runtime_identity(root): + versions = {'python': sys.version.split()[0]} + for name, argv in [('node', ['node', '--version']), ('go', ['go', 'version'])]: + try: + versions[name] = subprocess.check_output(argv, cwd=root, stderr=subprocess.PIPE, timeout=5).decode().strip() + except (OSError, subprocess.SubprocessError): + versions[name] = 'unavailable' + lock = root/'node_modules/.package-lock.json' + versions['installedNpmLockDigest'] = hashlib.sha256(lock.read_bytes()).hexdigest() if lock.is_file() else 'unavailable' + return versions + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--mode', choices=['quick', 'task', 'integration'], default='quick') + parser.add_argument('--base', default='HEAD', help='包含该提交以来的提交、暂存、未暂存及未跟踪改动;默认仅当前未提交改动') + parser.add_argument('--plan', action='store_true', help='只打印计划,不运行检查') + parser.add_argument('--budget-seconds', type=int, help='总时间预算,超时报告待验证') + parser.add_argument('--gofmt', nargs='+', help=argparse.SUPPRESS) + parser.add_argument('--python-syntax', help=argparse.SUPPRESS) + args = parser.parse_args(argv) + if args.gofmt: + result = subprocess.check_output(['gofmt', '-l', *args.gofmt], cwd=ROOT).decode() + print(result, end='') + return 1 if result.strip() else 0 + if args.python_syntax: + ast.parse((ROOT/args.python_syntax).read_text()) + return 0 + budget = args.budget_seconds if args.budget_seconds is not None else {'quick': 180, 'task': 600, 'integration': 3600}[args.mode] + if budget <= 0: + parser.error('--budget-seconds must be positive') + base, files = changed_files(ROOT, args.base) + plan = plan_checks(ROOT, files, json.loads(POLICY.read_text()), args.mode, base, tooling_package_change(ROOT, base)) + print(json.dumps(plan, ensure_ascii=False, indent=2), flush=True) + if args.plan: + return 0 + folder = ROOT/'local-data/test-results'/f'verify-{time.time_ns()}' + folder.mkdir(parents=True, mode=0o700) + before = fingerprint(ROOT) + report = {**plan, 'base': base, 'head': git(ROOT, 'rev-parse', 'HEAD'), 'inputDigest': before, 'runtime': runtime_identity(ROOT), + 'environmentDigest': hashlib.sha256(json.dumps(dict(os.environ), sort_keys=True).encode()).hexdigest(), + 'startedAt': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()), 'results': []} + start = time.monotonic() + for index, check in enumerate(plan['commands']): + remaining = budget - (time.monotonic()-start) + if remaining <= 0: + report['results'].append({**check, 'status': 'not-run', 'reason': 'budget exhausted'}) + continue + result = run_check(check, ROOT, folder/f'{index:02}.log', remaining) + report['results'].append(result) + print(f"{result['status']}: {' '.join(check['argv'])} ({result['seconds']}s)", flush=True) + if result['status'] != 'passed': + report['results'].extend({**c, 'status': 'not-run', 'reason': 'previous check did not pass'} for c in plan['commands'][index+1:]) + break + report['inputsChangedDuringRun'] = fingerprint(ROOT) != before + statuses = {r['status'] for r in report['results']} + code = 1 if 'failed' in statuses else 2 if (statuses - {'passed'} or plan['unmapped'] or report['inputsChangedDuringRun']) else 0 + report['status'] = 'failed' if code == 1 else 'incomplete' if code == 2 else 'selected-checks-passed' + report['seconds'] = round(time.monotonic()-start, 3) + (folder/'report.json').write_text(json.dumps(report, ensure_ascii=False, indent=2)+'\n') + print(f"{report['status']}; 报告: {folder/'report.json'}") + if plan['deferred']: + print('任务级检查尚未执行;本结果不代表发布/容量验收通过。') + return code + + +if __name__ == '__main__': + try: + sys.exit(main()) + except (OSError, ValueError, subprocess.CalledProcessError) as error: + print(f'检查入口失败: {type(error).__name__}', file=sys.stderr) + sys.exit(2) diff --git a/tests/tooling/test_verification.py b/tests/tooling/test_verification.py new file mode 100644 index 00000000..4e661219 --- /dev/null +++ b/tests/tooling/test_verification.py @@ -0,0 +1,147 @@ +import importlib.util +import json +import os +from pathlib import Path +import subprocess +import sys +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location('verification', ROOT/'scripts/verify-change.py') +v = importlib.util.module_from_spec(spec) +spec.loader.exec_module(v) +POLICY = json.loads((ROOT/'scripts/verification-scopes.json').read_text()) + + +class VerificationTest(unittest.TestCase): + def plan(self, files, mode='quick'): + return v.plan_checks(ROOT, files, POLICY, mode, 'HEAD') + + def test_documentation_is_small(self): + plan = self.plan(['docs/example.md', 'AGENTS.md']) + self.assertEqual(len(plan['commands']), 1) + self.assertEqual(plan['deferred'], []) + self.assertEqual(plan['unmapped'], []) + + def test_common_page_expands_and_deduplicates(self): + plan = self.plan(['src/pages/AnnotationOperations.tsx']) + self.assertGreaterEqual(len(plan['scopes']), 4) + self.assertTrue(plan['deferred']) + self.assertFalse(any('go' == c['argv'][0] and c['env'] for c in plan['commands'])) + node_tests = [c for c in plan['commands'] if c['argv'][:2] == ['node', '--test']] + self.assertEqual(len(node_tests), 1) + self.assertEqual(len(node_tests[0]['argv'][2:]), len(set(node_tests[0]['argv'][2:]))) + + def test_source_task_includes_real_mongo_and_browser(self): + plan = self.plan(['internal/store/annotation_source_admission.go'], 'task') + self.assertTrue(any(c['env'].get('SOURCE_ADMISSION_LOCAL_MONGO') for c in plan['commands'])) + self.assertTrue(any(c['env'].get('SOURCE_VALIDATION_BROWSER_TEST') for c in plan['commands'])) + self.assertIn('source-and-buffer', plan['scopes']) + self.assertEqual(plan['unmapped'], []) + + def test_script_alias_only_does_not_trigger_dependency_suite(self): + plan = v.plan_checks(ROOT, ['package.json'], POLICY, 'quick', 'HEAD', tooling_package=True) + self.assertEqual(plan['deferred'], []) + self.assertFalse(any('lint:web' in c['argv'] for c in plan['commands'])) + self.assertTrue(any('unittest' in c['argv'] for c in plan['commands'])) + + def test_css_does_not_run_business_suites(self): + plan = self.plan(['src/example.css']) + self.assertEqual(plan['deferred'], []) + self.assertFalse(any('lint:web' in c['argv'] for c in plan['commands'])) + + def test_integration_does_not_repeat_targeted_frontend_tests(self): + plan = self.plan(['src/pages/AnnotationOperations.tsx'], 'integration') + self.assertFalse(any(c['argv'][:2] == ['node', '--test'] for c in plan['commands'])) + self.assertEqual(sum('test:frontend:gate' in c['argv'] for c in plan['commands']), 1) + + def test_integration_keeps_tooling_regressions(self): + plan = self.plan(['scripts/verify-change.py'], 'integration') + self.assertTrue(any('unittest' in c['argv'] for c in plan['commands'])) + + def test_new_frontend_falls_back_at_task_level(self): + plan = self.plan(['src/components/FutureShared.tsx']) + self.assertTrue(any('test:frontend:gate' in c['argv'] for c in plan['deferred'])) + self.assertFalse(any('test:frontend:gate' in c['argv'] for c in plan['commands'])) + + def test_unknown_script_is_not_silently_covered(self): + self.assertTrue(self.plan(['scripts/new-data-migration.py'])['unmapped']) + self.assertTrue(self.plan(['tests/browser/new-flow.mjs'])['unmapped']) + + def test_changed_frontend_test_is_always_included(self): + f = 'tests/frontend/annotation-buffer-summary.test.mjs' + plan = self.plan([f]) + self.assertTrue(any(f in c['argv'] for c in plan['commands'])) + + def test_deleted_go_file_gets_package_fallback(self): + plan = self.plan(['internal/store/removed.go']) + self.assertTrue(any('./internal/store' in c['argv'] for c in plan['deferred'])) + + def test_scope_tests_and_browser_names_exist(self): + for scope in POLICY['scopes']: + for pattern in scope.get('frontend', []): + self.assertTrue(list(ROOT.glob(pattern)), pattern) + for c in scope.get('task', []): + if c['argv'][0] == 'go': + package = ROOT/c['argv'][2] + self.assertTrue(package.is_dir()) + if c['argv'][:2] == ['npm', 'run']: + self.assertIn(c['argv'][2], json.loads((ROOT/'package.json').read_text())['scripts']) + + def test_changes_include_committed_staged_unstaged_untracked_and_renamed(self): + with tempfile.TemporaryDirectory() as d: + root = Path(d) + def git(*args): + return subprocess.check_output(['git', *args], cwd=root, stderr=subprocess.DEVNULL).decode().strip() + git('init') + git('config', 'user.email', 'test@example.invalid') + git('config', 'user.name', 'Test') + (root/'a').write_text('initial') + git('add', 'a'); git('commit', '-m', 'initial') + base = git('rev-parse', 'HEAD') + git('mv', 'a', 'b'); git('commit', '-m', 'rename') + (root/'staged').write_text('staged'); git('add', 'staged') + (root/'b').write_text('modified') + (root/'new').write_text('untracked') + _, files = v.changed_files(root, base) + self.assertEqual(files, ['a', 'b', 'new', 'staged']) + before = v.fingerprint(root) + (root/'new').write_text('changed during check') + self.assertNotEqual(before, v.fingerprint(root)) + with self.assertRaises(subprocess.CalledProcessError): + v.changed_files(root, 'does-not-exist') + + def test_runner_failure_skip_empty_and_timeout_never_pass(self): + with tempfile.TemporaryDirectory() as d: + for code, expected in [ + ('print("ok")', 'passed'), + ('raise SystemExit(3)', 'failed'), + ('print("--- SKIP: TestOptIn")', 'incomplete'), + ('print("testing: warning: no tests to run")', 'incomplete'), + ('import time; time.sleep(3)', 'incomplete'), + ]: + result = v.run_check(v.command([sys.executable, '-c', code], tests=True), Path(d), Path(d)/'run.log', .1) + self.assertEqual(result['status'], expected, code) + + def test_runner_only_enables_declared_opt_ins(self): + from unittest.mock import patch + with tempfile.TemporaryDirectory() as d, patch.dict(os.environ, {'RUN_OUTPUT_SNAPSHOT_BENCHMARK': '1', 'TEST_WORKFORCE_MONGO_URI': 'remote', 'MONGO_URI': 'remote'}): + code = 'import os; assert "RUN_OUTPUT_SNAPSHOT_BENCHMARK" not in os.environ; assert "TEST_WORKFORCE_MONGO_URI" not in os.environ; assert "MONGO_URI" not in os.environ; assert os.environ["SOURCE_ADMISSION_LOCAL_MONGO"] == "1"' + c = v.command([sys.executable, '-c', code], env={'SOURCE_ADMISSION_LOCAL_MONGO': '1'}) + self.assertEqual(v.run_check(c, Path(d), Path(d)/'run.log', 2)['status'], 'passed') + + def test_timeout_terminates_descendants(self): + with tempfile.TemporaryDirectory() as d: + marker = Path(d)/'should-not-exist' + nested = f'import time; from pathlib import Path; time.sleep(1); Path({str(marker)!r}).touch()' + code = f'import subprocess,sys,time; subprocess.Popen([sys.executable,"-c",{nested!r}]); time.sleep(5)' + result = v.run_check(v.command([sys.executable, '-c', code]), Path(d), Path(d)/'run.log', .1) + self.assertEqual(result['status'], 'incomplete') + import time + time.sleep(1.1) + self.assertFalse(marker.exists()) + + +if __name__ == '__main__': + unittest.main()