feat(跨层 X.1+X.3+X.4): 反馈采集全链路 + 可观测面板 + 量化坐实 (M1-M5+X 全闭环)
Some checks are pending
CI / quality-gate (push) Waiting to run
CI / type-drift (push) Waiting to run

跨层共享项落地(设计文档 §19.4 X.1/X.3/X.4):

X.1 时序三档接线 + X.3 v_t 全链路贯通:
- shared/types.ts: TaskFeedback/ToolOutcome/EvolutionMetrics/MetricsSources
  + hitRate/avgInfoGain/avgSkillQuality/maxWorkflowRisk/evolutionBrickRate 纯函数
- core/feedback-collector.ts: FeedbackCollector(recordPrompt/recordBoardFingerprint/
  recordToolResult/finalize/lastFinalized/pending/clear) + derive* 纯变换
  (deriveTaskSignature/deriveDistillInput/deriveSkillExtractInput/deriveSkillQuery/
  derivePriorCorrection/deriveProposals + inferOutcome/inferActionClass/
  inferValidatedBy/isIrreversibleTool) + getSharedCollector/initSharedCollector 共享单例
- extensions/feedback-bank.ts: hooks tool_result+message_end+before_agent_start
  → 采集 → finalize → session_before_compact 喂 derive* + feedback_recall 工具
  (仿 orchestration-bank/skill-bank 范式)
- **4 bank 默认 derive* 读共享 collector lastFinalized 生产自驱**
  (experience/skill/orchestration/variant bank) — X.1+X.3 生产闭环

X.4 进化可观测面板:
- core/evolution-metrics.ts: aggregateMetrics/renderMetrics 纯函数
  (5 层注册表聚合 → EvolutionMetrics:规模/命中率/infoGain/risk/砖板率/forbidden 拒绝)
- bin/evolution-metrics.ts CLI(brickRate!=0 时 exit 1 fail-closed 自检)

X.1+X.3 全链路评估(tests/eval/feedback-chain-eval.test.ts, 合成 session 轨迹→
feedback-bank hooks→collector→derive*→真 5 层 manager→断 v_t 全链路 end-to-end):
  §4.6 分级正确率 8/8=100% (success+ground-truth/可逆/不可逆/partial/empty/mixed)
  同签名检索命中 100% + 注入块可见 100%
  5 链 end-to-end (skill/workflow/variant/tool-growth)
  forbidden §11.3/§4.8 拒绝 (命门 safety-gate/flash 工具)

X.4 评估(tests/eval/evolution-metrics-eval.test.ts, 合成 5 层注册表→精确指标):
  experienceHitRate=0.600/avgInfoGain=0.650/maxRisk=1 精确
  brickRate=0 EXACT 构造铁律 forbiddenRejected=7 精确

结论: X.1(4/5 时序钩子全接, stage-wise θ=L5.*) + X.3(v_t 全链路贯通: tool_result
采集→derive* 蒸馏→5 层入库→检索→注入→约束, §4.6 100% + 5 链 end-to-end + 4 bank
生产自驱) + X.4(可观测面板精确指标, brickRate=0 构造铁律) 全部完成。诚实边界
(M1-M5.* 跟进): live-LLM 读注入包真受益端到端门 + stage-wise 真 θ 更新 + 真 git
worktree 变体 + 真 CI §11.4 执行 = M1-M5.* 工程项(机制层全闭环+量化坐实)。

验证: tsc 0 / 全量 62 文件 3018 passed 5 skipped / sk- 明文 0 / 真 pi T01 boot
过(barrel 28 扩展含 L1-L5 + 跨层反馈采集)
多 Agent 并行: K(feedback-collector+derive*)+L(feedback-bank+chain eval)+M(evolution-metrics)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
d-robotics 2026-08-17 17:32:37 +08:00
parent 80e89c9841
commit 74f4cd428e
18 changed files with 3758 additions and 27 deletions

126
bin/evolution-metrics.ts Normal file
View File

@ -0,0 +1,126 @@
#!/usr/bin/env node
/**
* @file bin/evolution-metrics.ts
* @module bin/evolution-metrics
*
* X.4 进化可观测面板 CLI(设计文档 §13 证据 / §16 评估 / §19.4 X.4)。
*
* 用法:
* npx tsx bin/evolution-metrics.ts [metrics-sources.json]
*
* - 不带参数:用内置确定性合成 5 层注册表快照演示面板输出。
* - 带参数:读 JSON 文件(序列化 MetricsSources),聚合 + 渲染面板。
*
* 生产接线(X.* 跟进):真实 5 层管理器是 in-memory 单例,进程内聚合;
* 本 CLI 演示纯函数聚合 + 渲染。生产可由 session/registry 钩子落
* MetricsSources 快照到 JSON,再用本 CLI 离线查看;或直接调
* aggregateMetrics(renderMetrics) 渲染实时快照。不读真实单例。
*/
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import type {
ExperienceEntry,
GrownTool,
HardwareSkill,
MetricsSources,
Variant,
WorkflowGraph,
} from "../src/shared/types.js";
import {
aggregateMetrics,
renderMetrics,
} from "../src/core/evolution-metrics.js";
// ============================================================================
// 内置确定性合成 5 层注册表快照(演示用;固定数字,无 RNG)
// ============================================================================
const beta = { soc: "rdk-x5", kernel: "6.1.83", image: "v1", mac: "AA:01" };
const sig = { actionClass: "flash", target: "bootloader", constraints: ["aarch64"] };
/** 5 经验,hitCounts [3,0,2,1,0] → hitRate = 0.6 */
const DEMO_EXPERIENCES: ExperienceEntry[] = [
{ id: "e0", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "dd if=img of=/dev/mmcblk0", evidenceRef: "s1", contextTags: ["flash"], validity: "repeatable", usage: "actionable", createdAt: 0, hitCount: 3, validatedBy: "physical-ground-truth" },
{ id: "e1", boardFingerprint: beta, taskSignature: sig, outcome: "failure", actionable: "n/a", antiPattern: "don't skip verify", evidenceRef: "s2", contextTags: ["flash"], validity: "irreversible-fail", usage: "anti-pattern", createdAt: 1, hitCount: 0, validatedBy: "physical-ground-truth" },
{ id: "e2", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "i2cdetect -y 0", evidenceRef: "s3", contextTags: ["i2c"], validity: "repeatable", usage: "actionable", createdAt: 2, hitCount: 2, validatedBy: "executable-stdout" },
{ id: "e3", boardFingerprint: beta, taskSignature: sig, outcome: "partial", actionable: "probe pmic", evidenceRef: "s4", contextTags: ["i2c"], validity: "once", usage: "hint", createdAt: 3, hitCount: 1, validatedBy: "executable-stdout" },
{ id: "e4", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "read dmesg", evidenceRef: "s5", contextTags: ["dmesg"], validity: "inferred", usage: "hint", createdAt: 4, hitCount: 0, validatedBy: "model-claim" },
];
/** 4 技能:avgInfoGain=0.65, avgQualityScore=0.75, 3 active */
const DEMO_SKILLS: HardwareSkill[] = [
{ id: "sk0", name: "i2c-recover", trigger: { kind: "event", pattern: "i2c.*no such device", actionClass: "i2c-probe" }, procedure: [{ tool: "i2cdetect", argsTemplate: "0x3a" }], granularity: "event-driven", provenance: ["s1"], quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.8, createdAt: 0, hitCount: 4, active: true },
{ id: "sk1", name: "reg-read", trigger: { kind: "state", statePredicate: "reg locked", actionClass: "reg-read" }, procedure: [{ tool: "devmem", argsTemplate: "0x40020000" }], granularity: "event-driven", provenance: ["s2"], quality: { rubricScore: 0.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.6, createdAt: 1, hitCount: 2, active: true },
{ id: "sk2", name: "old-probe", trigger: { kind: "event", pattern: "timeout", actionClass: "probe" }, procedure: [{ tool: "scan", argsTemplate: "" }], granularity: "task-level", provenance: ["s3"], quality: { rubricScore: 1.0, execPassRate: 0.0, execSamples: 5 }, infoGain: 0.9, createdAt: 2, hitCount: 0, active: false, supersededBy: "sk0" },
{ id: "sk3", name: "dt-parse", trigger: { kind: "event", pattern: "dtb error", actionClass: "device-tree" }, procedure: [{ tool: "fdtdump", argsTemplate: "" }], granularity: "event-driven", provenance: ["s4"], quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.3, createdAt: 3, hitCount: 1, active: true },
];
/** 3 workflow,risks [1,0,1] → maxRisk=1, 2 active */
const DEMO_WORKFLOWS: WorkflowGraph[] = [
{ id: "w0", taskClass: "flash", nodes: [{ id: "n0", tool: "verify", argsTemplate: "", irreversibility: "irreversible" }], edges: [], risk: 1, provenance: ["s1"], createdAt: 0, hitCount: 3, active: true },
{ id: "w1", taskClass: "i2c", nodes: [{ id: "n1", tool: "i2cdetect", argsTemplate: "", irreversibility: "reversible" }], edges: [], risk: 0, provenance: ["s3"], createdAt: 1, hitCount: 0, active: false, supersededBy: "w2" },
{ id: "w2", taskClass: "i2c", nodes: [{ id: "n2", tool: "i2cget", argsTemplate: "", irreversibility: "irreversible" }], edges: [], risk: 1, provenance: ["s3"], createdAt: 2, hitCount: 2, active: true },
];
/** 5 grownTools,2 promoted */
const DEMO_GROWN_TOOLS: GrownTool[] = [
{ id: "g0", name: "dump-reg", toolClass: "read", definitionHash: "h0", stage: "promoted", usageCount: 10, passCount: 9, validity: "repeatable", createdAt: 0, lastUsed: 5 },
{ id: "g1", name: "scan-i2c", toolClass: "probe", definitionHash: "h1", stage: "promoted", usageCount: 12, passCount: 11, validity: "repeatable", createdAt: 1, lastUsed: 6 },
{ id: "g2", name: "parse-dtb", toolClass: "analyze", definitionHash: "h2", stage: "verified", usageCount: 4, passCount: 3, validity: "repeatable", createdAt: 2, lastUsed: 3 },
{ id: "g3", name: "probe-gpio", toolClass: "probe", definitionHash: "h3", stage: "draft", usageCount: 1, passCount: 1, validity: "once", createdAt: 3, lastUsed: 1 },
{ id: "g4", name: "old-tool", toolClass: "read", definitionHash: "h4", stage: "retired", usageCount: 20, passCount: 15, validity: "repeatable", createdAt: 4, lastUsed: 0, supersededBy: "g0" },
];
/** 4 variants,1 promoted */
const DEMO_VARIANTS: Variant[] = [
{ id: "v0", name: "faster-probe", targetExtension: "env-probe", scaffoldClass: "probe", baseRef: "HEAD", factoryOverride: "o0", stage: "promoted", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: true, allPassed: true }, createdAt: 0, shadowResults: { succVariant: 15, succBaseline: 10, N: 20 } },
{ id: "v1", name: "better-dt", targetExtension: "device-tree", scaffoldClass: "analyze", baseRef: "HEAD", factoryOverride: "o1", stage: "candidate", invariants: { tests: false, tsc: false, noSecrets: true, modelLaunch: false, sandboxGate: true, brickRateZero: false, riskNonIncrease: true, allPassed: false }, createdAt: 1 },
{ id: "v2", name: "risky-build", targetExtension: "build-tracker", scaffoldClass: "analyze", baseRef: "HEAD", factoryOverride: "o2", stage: "rolled-back", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: false, allPassed: false }, createdAt: 2 },
{ id: "v3", name: "old-variant", targetExtension: "pdf-reader", scaffoldClass: "read", baseRef: "HEAD", factoryOverride: "o3", stage: "retired", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: true, allPassed: true }, createdAt: 3, supersededBy: "v0" },
];
const DEMO_SOURCES: MetricsSources = {
experiences: DEMO_EXPERIENCES,
skills: DEMO_SKILLS,
workflows: DEMO_WORKFLOWS,
grownTools: DEMO_GROWN_TOOLS,
variants: DEMO_VARIANTS,
forbiddenRejected: 7,
};
// ============================================================================
// CLI 入口
// ============================================================================
function loadSourcesFromJson(path: string): MetricsSources {
const raw = readFileSync(resolve(path), "utf8");
const parsed = JSON.parse(raw) as MetricsSources;
// fail-closed:基本结构校验(5 层数组必须存在)
if (!parsed || !Array.isArray(parsed.experiences) || !Array.isArray(parsed.skills)
|| !Array.isArray(parsed.workflows) || !Array.isArray(parsed.grownTools)
|| !Array.isArray(parsed.variants)) {
throw new Error(`evolution-metrics: JSON 文件 ${path} 缺 5 层数组之一(MetricsSources 结构不完整)`);
}
return parsed;
}
function main(): void {
const arg = process.argv[2];
const sources = arg ? loadSourcesFromJson(arg) : DEMO_SOURCES;
const ts = Number(process.env.PBA_METRICS_TS ?? 0);
const m = aggregateMetrics(sources, ts);
const out = renderMetrics(m);
// stdout 仅渲染表,stderr 打元信息(便于管道)
const origin = arg ? `file:${arg}` : "demo-synthetic";
process.stderr.write(`[evolution-metrics] sources=${origin} ts=${ts}\n`);
process.stdout.write(out + "\n");
// 构造不变式自检(fail-closed):brickRate 必须为 0
if (m.brickRate !== 0) {
process.stderr.write(`[evolution-metrics] CRITICAL: brickRate=${m.brickRate} 非零(构造不变式破坏,§4.8/§11.4)\n`);
process.exit(1);
}
}
main();

View File

@ -42,7 +42,7 @@ export interface LaunchResult {
* const { session } = await new PhysicalByteAgent().launch();
*/
export class PhysicalByteAgent {
/** 已装配的域扩展工厂列表(27 个,按依赖顺序)。 */
/** 已装配的域扩展工厂列表(28 个,按依赖顺序)。 */
static readonly extensions = extensionFactories;
/**

View File

@ -0,0 +1,108 @@
/**
* @module core/evolution-metrics
*
* 跨层共享 X.4(设计文档 §13 证据 / §16 评估 / §19.4):进化可观测面板。
*
* 5 层注册表聚合 → EvolutionMetrics:规模/命中率/infoGain/risk/砖板率/forbidden 拒绝。
* 砖板率构造恒 0(L3+L5 forbidden 全拒+只长读/探+sandbox+§11.4 门);maxRisk 须 ≤ default(§4.8)。
*
* 纯逻辑,零 pi 运行时依赖。CLI/工具接线在 scripts/ 或 metrics-bank 扩展。
*
* ┌──────────────────────────────────────────────────────────────────┐
* │ 公共契约(interface EvolutionMetricsAggregator)签名稳定 —— │
* │ 扩展/评估 agent 据此编码,实现(EvolutionMetricsAggregatorImpl) │
* │ 由 X 填充真实 body,**不得改 interface 签名**。 │
* └──────────────────────────────────────────────────────────────────┘
*/
import type { EvolutionMetrics, MetricsSources } from "../shared/types.js";
import {
avgInfoGain,
avgSkillQuality,
evolutionBrickRate,
hitRate,
maxWorkflowRisk,
} from "../shared/types.js";
// ============================================================================
// EvolutionMetricsAggregator —— 公共契约(签名稳定)
// ============================================================================
/**
* X.4 进化可观测聚合器。公共契约(签名稳定):
* - aggregate(sources):5 层注册表快照 → EvolutionMetrics。
* - 砖板率恒 0(构造不变式);maxRisk = max(workflow risk)(§4.8)。
*/
export interface EvolutionMetricsAggregator {
aggregate(sources: MetricsSources, ts?: number): EvolutionMetrics;
}
// ============================================================================
// 纯函数(已实现,命名导出;CLI/工具/评估复用)
// ============================================================================
/**
* X.4 聚合(纯函数):5 层注册表快照 → EvolutionMetrics。
* 砖板率构造恒 0(§4.8/§11.4);maxRisk = max workflow risk(须 ≤ default)。
* @param sources 5 层注册表快照。
* @param ts 时间戳(默认 0,eval 用 task index)。
* @returns EvolutionMetrics。
*/
export function aggregateMetrics(sources: MetricsSources, ts: number = 0): EvolutionMetrics {
const { experiences, skills, workflows, grownTools, variants } = sources;
const forbiddenRejected = sources.forbiddenRejected ?? 0;
return {
experienceCount: experiences.length,
experienceHitRate: hitRate(experiences),
skillCount: skills.length,
activeSkillCount: skills.filter((s) => s.active).length,
workflowCount: workflows.length,
activeWorkflowCount: workflows.filter((w) => w.active).length,
grownToolCount: grownTools.length,
promotedToolCount: grownTools.filter((t) => t.stage === "promoted").length,
variantCount: variants.length,
promotedVariantCount: variants.filter((v) => v.stage === "promoted").length,
maxRisk: maxWorkflowRisk(workflows),
brickRate: evolutionBrickRate(),
forbiddenRejected,
avgInfoGain: avgInfoGain(skills),
avgQualityScore: avgSkillQuality(skills),
ts,
};
}
/**
* X.4 渲染指标表(纯函数):EvolutionMetrics → 可读多行串(CLI/日志用)。
* @param m 指标。
* @returns 多行串。
*/
export function renderMetrics(m: EvolutionMetrics): string {
const lines: string[] = [];
lines.push("=== Evolution Metrics ===");
lines.push(`experienceCount=${m.experienceCount} hitRate=${m.experienceHitRate.toFixed(3)}`);
lines.push(`skillCount=${m.skillCount} active=${m.activeSkillCount} avgInfoGain=${m.avgInfoGain.toFixed(3)} avgQuality=${m.avgQualityScore.toFixed(3)}`);
lines.push(`workflowCount=${m.workflowCount} active=${m.activeWorkflowCount} maxRisk=${m.maxRisk}`);
lines.push(`grownToolCount=${m.grownToolCount} promoted=${m.promotedToolCount}`);
lines.push(`variantCount=${m.variantCount} promoted=${m.promotedVariantCount}`);
lines.push(`brickRate=${m.brickRate} (构造恒 0) forbiddenRejected=${m.forbiddenRejected}`);
return lines.join("\n");
}
// ============================================================================
// EvolutionMetricsAggregator 实现
// ============================================================================
/** 默认聚合器实现(委托纯函数 aggregateMetrics)。 */
export class EvolutionMetricsAggregatorImpl implements EvolutionMetricsAggregator {
aggregate(sources: MetricsSources, ts: number = 0): EvolutionMetrics {
return aggregateMetrics(sources, ts);
}
}
/** 默认工厂:返回聚合器。 */
export function createEvolutionMetricsAggregator(): EvolutionMetricsAggregator {
return new EvolutionMetricsAggregatorImpl();
}
// 再导出纯函数(一个入口)。
export { avgInfoGain, avgSkillQuality, evolutionBrickRate, hitRate, maxWorkflowRisk };

View File

@ -0,0 +1,438 @@
/**
* @module core/feedback-collector
*
* 跨层共享 X.1+X.3(设计文档 §12 时序 / §4.6 可信度 / §13 证据 / §19.4):
* 反馈采集 + v_t 全链路贯通。
*
* 时序三档接线(X.1):
* - `tool_result` 钩子 → recordToolResult(每工具结果入账)。
* - `before_agent_start` → recordPrompt(捕获 prompt 派生 σ)+ recordBoardFingerprint(env-probe)。
* - `message_end` → finalize(结算一轮 turn → TaskFeedback)。
* - `session_before_compact` → 读 finalize 产物喂 derive*(蒸馏/先验校正/提案)。
*
* v_t 全链路(X.3):反馈采集 → 蒸馏(derive*)→ 入库(5 层 Registry)→ 检索 → 注入 → 约束。
*
* 纯逻辑,零 pi 运行时依赖。derive* 是纯变换(prompt/toolOutcomes → 各层入参),
* 可单测 + 不需 live LLM(live 受益 = M1-M5.\* 端到端门)。
*
* ┌──────────────────────────────────────────────────────────────────┐
* │ 公共契约(interface FeedbackCollector + derive* 签名)稳定 —— │
* │ 扩展/评估 agent 据此编码,实现(FeedbackCollectorImpl)由 X 填充 │
* │ 真实 body,**不得改 interface/签名**。 │
* └──────────────────────────────────────────────────────────────────┘
*/
import type {
BoardFingerprint,
DifficultyCorrection,
SkillExtractInput,
SkillTrigger,
TaskFeedback,
TaskSignature,
ToolOutcome,
} from "../shared/types.js";
import type { DistillInput } from "./experience-store.js";
import type { VariantProposal } from "./variant-manager.js";
// ============================================================================
// 命名常量 —— 消灭魔法串
// ============================================================================
/** §13 结果摘要截断长度(防大回包爆上下文)。 */
export const RESULT_SUMMARY_MAX = 200;
/** §4.5 默认 actionClass(推导不出时的兜底)。 */
export const DEFAULT_ACTION_CLASS = "unknown";
// ============================================================================
// FeedbackCollector —— 公共契约(签名稳定)
// ============================================================================
/**
* 反馈采集器(一轮 turn 累积 tool_result → 结算 TaskFeedback)。
*
* 公共契约(签名稳定):
* - recordPrompt(prompt):捕获用户消息(before_agent_start)。
* - recordBoardFingerprint(beta):捕获板指纹(env-probe)。
* - recordToolResult(outcome):累积工具结果(tool_result 钩子)。
* - finalize(actionClassHint?):结算一轮 → TaskFeedback(含 outcome/步数/失败推导);未收到 prompt→null。
* - pending()/clear()。
*/
export interface FeedbackCollector {
recordPrompt(prompt: string): void;
recordBoardFingerprint(beta: BoardFingerprint): void;
recordToolResult(outcome: ToolOutcome): void;
finalize(actionClassHint?: string): TaskFeedback | null;
/** 读取最近一次 finalize 的 TaskFeedback(不清空;供其他 bank 的 derive* 在 session_before_compact 读,避免二次 finalize 返 null)。 */
lastFinalized(): TaskFeedback | null;
pending(): readonly ToolOutcome[];
clear(): void;
}
// ============================================================================
// derive* —— 纯变换(公共契约,签名稳定;X 填充真实 body)
// ============================================================================
/**
* X.3 从 feedback 派生 taskSignature(纯函数)。
* actionClass 从 prompt+toolNames 启发;target 从 prompt 首个名词/地址;constraints 从板 SoC+arch。
* @param feedback 任务反馈。
* @returns TaskSignature。
*/
export function deriveTaskSignature(feedback: TaskFeedback): TaskSignature {
const actionClass = feedback.actionClassHint || inferActionClass(feedback.prompt, feedback.toolOutcomes.map((o) => o.toolName));
const target = inferTarget(feedback.prompt);
const constraints = [feedback.boardFingerprint.soc, feedback.boardFingerprint.kernel];
return { actionClass, target, constraints };
}
/**
* X.3 从 feedback 派生 DistillInput(纯函数,喂 ExperienceStore.append)。
* outcome/irreversible/validatedBy 从 toolOutcomes 推导(§4.6 分级在 distillEntry 内做)。
* @param feedback 任务反馈。
* @returns DistillInput。
*/
export function deriveDistillInput(feedback: TaskFeedback): DistillInput {
const irreversible = feedback.toolOutcomes.some((o) => isIrreversibleTool(o.toolName));
const validatedBy = inferValidatedBy(feedback.toolOutcomes);
return {
boardFingerprint: feedback.boardFingerprint,
taskSignature: deriveTaskSignature(feedback),
outcome: feedback.outcome,
actionable: feedback.outcome === "success" ? inferActionable(feedback) : "",
antiPattern: feedback.outcome !== "success" ? inferAntiPattern(feedback) : undefined,
evidenceRef: `turn@${feedback.turnTs}`,
contextTags: [feedback.actionClassHint, feedback.boardFingerprint.soc],
validatedBy,
irreversible,
createdAt: feedback.turnTs,
};
}
/**
* X.3 从 feedback 派生技能抽取入参(纯函数,喂 SkillManager.extract→register)。
* @param feedback 任务反馈。
* @returns SkillExtractInput(成功才抽技能;失败返 antiPattern 倾向的 trigger)。
*/
export function deriveSkillExtractInput(feedback: TaskFeedback): SkillExtractInput {
const actionClass = feedback.actionClassHint || inferActionClass(feedback.prompt, feedback.toolOutcomes.map((o) => o.toolName));
const trigger: SkillTrigger = {
kind: "event",
actionClass,
pattern: feedback.outcome === "success" ? undefined : `${actionClass}-failure`,
};
return {
trigger,
procedure: feedback.toolOutcomes.map((o) => ({ tool: o.toolName, argsTemplate: JSON.stringify(o.input).slice(0, 80) })),
granularity: "task-level",
provenance: [`turn@${feedback.turnTs}`],
rubricScore: feedback.outcome === "success" ? 0.8 : 0.3,
execResults: feedback.toolOutcomes.map((o) => ({ passed: o.passed })),
createdAt: feedback.turnTs,
};
}
/**
* X.3 从 feedback 派生技能检索查询(纯函数,喂 SkillManager.resolve)。
* @param feedback 任务反馈。
* @returns { trigger, sigma } 或 null(无 prompt→null)。
*/
export function deriveSkillQuery(feedback: TaskFeedback): { trigger: SkillTrigger; sigma: TaskSignature } | null {
if (!feedback.prompt) return null;
const sigma = deriveTaskSignature(feedback);
const trigger: SkillTrigger = { kind: "event", actionClass: sigma.actionClass };
return { trigger, sigma };
}
/**
* X.3 从 feedback 派生难度先验校正(纯函数,喂 WorkflowOptimizer.correctPrior)。
* @param feedback 任务反馈。
* @returns DifficultyCorrection。
*/
export function derivePriorCorrection(feedback: TaskFeedback): DifficultyCorrection {
return {
taskClass: feedback.actionClassHint || inferActionClass(feedback.prompt, feedback.toolOutcomes.map((o) => o.toolName)),
estimated: feedback.steps > 6 ? "high" : feedback.steps > 3 ? "mid" : "low",
actualSteps: feedback.steps,
failures: feedback.failures,
boardFingerprint: feedback.boardFingerprint,
};
}
/**
* X.3 从 feedback 派生变体提案(纯函数,喂 VariantManager.propose)。
* 失败 turn + 同类反复 → 提案改对应扩展(非命门)。
* @param feedback 任务反馈。
* @returns VariantProposal[](空数组若无有意义提案)。
*/
export function deriveProposals(feedback: TaskFeedback): VariantProposal[] {
if (feedback.outcome === "success") return [];
const target = inferTargetExtension(feedback.toolOutcomes.map((o) => o.toolName));
if (!target) return [];
return [
{
name: `fix-${target}-${feedback.turnTs}`,
targetExtension: target,
description: `improve ${target} handling for ${feedback.actionClassHint} failures`,
baseRef: `turn@${feedback.turnTs}`,
factoryOverride: `patch:${feedback.actionClassHint}`,
createdAt: feedback.turnTs,
},
];
}
// ============================================================================
// 纯函数(已实现,命名导出;扩展/评估复用)—— outcome/actionClass 推导
// ============================================================================
/**
* §4.5 从 toolOutcomes 推导任务整体 outcome(纯函数)。
* 全 passed=success;有 fail 但有 pass=partial;全 fail=failure。
* @param outcomes 工具结果集。
* @returns success/partial/failure。
*/
export function inferOutcome(outcomes: readonly ToolOutcome[]): "success" | "failure" | "partial" {
if (outcomes.length === 0) return "failure";
const passed = outcomes.filter((o) => o.passed).length;
if (passed === outcomes.length) return "success";
if (passed === 0) return "failure";
return "partial";
}
/**
* §4.5 从 prompt + toolNames 启发推导 actionClass(纯函数)。
* flash/reg/i2c/build/dt/gpio 等关键词命中;无→unknown。
* @param prompt 用户消息。
* @param toolNames 工具名集。
* @returns actionClass。
*/
export function inferActionClass(prompt: string, toolNames: readonly string[]): string {
const text = `${prompt} ${toolNames.join(" ")}`.toLowerCase();
if (/flash|burn|烧|镜像/.test(text)) return "flash";
if (/reg|寄存器|0x[0-9a-f]+/.test(text)) return "reg-read";
if (/i2c|i2cdetect|i2cget/.test(text)) return "i2c-probe";
if (/build|make|gcc|编译/.test(text)) return "build";
if (/dt|device.tree|设备树/.test(text)) return "dt-edit";
if (/gpio|gpioget|gpioset/.test(text)) return "gpio-config";
if (/jtag|gdb|debug/.test(text)) return "jtag-debug";
return DEFAULT_ACTION_CLASS;
}
/**
* 从 prompt 提取 target(首个名词/地址,简化)。
* @param prompt 用户消息。
* @returns target 串。
*/
export function inferTarget(prompt: string): string {
const m = prompt.match(/(0x[0-9a-fA-F]+|\b\w+\b)/);
return m?.[1] ?? "target";
}
/**
* 从 toolOutcomes 推导 validatedBy(§13 证据强度)。
* 有 hardware-verify/gpio/i2c 读回值 → physical-ground-truth;exit 0 + stdout → executable-stdout;否则 model-claim。
* @param outcomes 工具结果集。
* @returns ValidationKind。
*/
export function inferValidatedBy(outcomes: readonly ToolOutcome[]): "physical-ground-truth" | "executable-stdout" | "model-claim" | "none" {
const names = outcomes.map((o) => o.toolName.toLowerCase());
if (names.some((n) => /verify|gpio|i2c|reg/.test(n))) return "physical-ground-truth";
if (outcomes.some((o) => !o.isError && o.exitCode === 0)) return "executable-stdout";
if (outcomes.length > 0) return "model-claim";
return "none";
}
/** 是否含不可逆工具(flash/erase/dd 等)→ §4.6 irreversible-fail 路径。 */
export function isIrreversibleTool(toolName: string): boolean {
return /flash|erase|burn|dd if|mkfs|fuse/.test(toolName.toLowerCase());
}
/** 从成功 turn 推导 actionable 句(简化:用 toolNames 串)。 */
export function inferActionable(feedback: TaskFeedback): string {
return feedback.toolOutcomes.map((o) => o.toolName).join("→");
}
/** 从失败 turn 推导 antiPattern 句(简化)。 */
export function inferAntiPattern(feedback: TaskFeedback): string {
const failed = feedback.toolOutcomes.filter((o) => !o.passed).map((o) => o.toolName);
return `avoid ${failed.join("/") || "noop"} sequence for ${feedback.actionClassHint}`;
}
/** 从 toolNames 推导可改扩展(非命门;命门→null)。 */
export function inferTargetExtension(toolNames: readonly string[]): string | null {
const joined = toolNames.join(" ").toLowerCase();
if (/i2c|gpio|peripheral/.test(joined)) return "peripheral-probe";
if (/dt|device.tree/.test(joined)) return "device-tree";
if (/build|make/.test(joined)) return "build-tracker";
if (/pdf|datasheet/.test(joined)) return "pdf-reader";
return null;
}
// ============================================================================
// FeedbackCollector 实现 —— 【X 待填充】body 为桩,签名不得改
// ============================================================================
/**
* §13 兜底板指纹:未收到 recordBoardFingerprint 时 finalize 用此(全 "unknown")。
* 真实 β 来自 env-probe(feedback-bank 扩展在 before_agent_start 注入);此仅 fallback,
* 保证 finalize 永不因 beta===null 而抛(§4.1 β 是结构必填字段)。
*/
const DEFAULT_BETA: BoardFingerprint = Object.freeze({
soc: "unknown",
kernel: "unknown",
image: "unknown",
mac: "unknown",
});
/**
* 默认 FeedbackCollector 实现(in-memory 累积)。
*
* 状态:prompt(一轮 turn 的用户消息,latest wins;finalize 后清空)+ beta(板指纹,
* latest wins;**跨 turn 持久**,clear 才清)+ outcomes(tool_result 累积数组)。
*
* 不变式(§12 时序三档):
* - recordPrompt:before_agent_start 捕获 prompt(派生 σ 用);latest wins(一轮一 prompt)。
* - recordBoardFingerprint:env-probe 注入 β;latest wins;跨 turn 持久(板不变)。
* - recordToolResult:tool_result 钩子每工具结果入账(caller 负责截断 resultSummary)。
* - finalize:message_end 结算一轮 → TaskFeedback(outcome/步数/失败由 inferOutcome 推导);
* prompt===null → null(无任务可蒸馏);结算后清 prompt+outcomes,**保 beta**(新 turn 起始)。
* - pending:未结算 outcomes 的只读快照。
* - clear:全清(prompt+outcomes+beta,测试/重置)。
*
* turnTs 用 Date.now()(仅可观测,非 grading 负载字段;测试断言结构非精确 ts)。
* derive* 已实现(纯变换)。公共契约签名稳定。
*/
export class FeedbackCollectorImpl implements FeedbackCollector {
private prompt: string | null = null;
private beta: BoardFingerprint | null = null;
private readonly outcomes: ToolOutcome[] = [];
/** 最近一次 finalize 的 TaskFeedback(供其他 bank 的 derive* 在 session_before_compact 读,避免二次 finalize 返 null)。 */
private lastFb: TaskFeedback | null = null;
/**
* §12 before_agent_start:捕获用户消息(派生 σ 的 prompt)。latest wins——一轮
* turn 只记一条 prompt(后到的 recordPrompt 覆盖前值,未 finalize 前可修正)。
* @param prompt 用户消息。
*/
recordPrompt(prompt: string): void {
this.prompt = prompt;
}
/**
* §12 env-probe:注入板指纹 β。latest wins;**跨 turn 持久**(finalize 不清 beta,
* 板物理身份不随 turn 变)。真实 β 来自 env-probe 扩展,feedback-bank 接线注入。
* @param beta 板指纹。
*/
recordBoardFingerprint(beta: BoardFingerprint): void {
this.beta = beta;
}
/**
* §12 tool_result 钩子:累积单次工具结果。caller 负责 resultSummary 截断
* (RESULT_SUMMARY_MAX),此处只 push(不二次截断,保 caller 控制权)。
* @param outcome 工具结果。
*/
recordToolResult(outcome: ToolOutcome): void {
this.outcomes.push(outcome);
}
/**
* §12 message_end:结算一轮 turn → TaskFeedback(喂 derive* 蒸馏/先验/提案)。
*
* - prompt===null → null(无任务可蒸馏,未收到 recordPrompt)。
* - outcome = inferOutcome(outcomes)(全 passed=success/有 fail=partial/全 fail=failure/空=failure)。
* - steps = outcomes.length;failures = outcomes.filter(!passed).length。
* - actionClassHint = hint ?? inferActionClass(prompt, toolNames)(§4.5 启发)。
* - turnTs = Date.now()(可观测,非 grading 负载)。
* - boardFingerprint = beta ?? DEFAULT_BETA(未注入 β 时兜底,永不 null)。
* - 结算后清 prompt + outcomes(新 turn 起始);**保 beta**(板跨 turn 持久)。
*
* @param actionClassHint 可选动作类提示(覆盖 inferActionClass 推导)。
* @returns TaskFeedback 或 null(无 prompt)。
*/
finalize(actionClassHint?: string): TaskFeedback | null {
if (this.prompt === null) return null;
const prompt = this.prompt;
const outcomes = [...this.outcomes];
const outcome = inferOutcome(outcomes);
const steps = outcomes.length;
const failures = outcomes.filter((o) => !o.passed).length;
const hint = actionClassHint ?? inferActionClass(prompt, outcomes.map((o) => o.toolName));
const feedback: TaskFeedback = {
prompt,
boardFingerprint: this.beta ?? DEFAULT_BETA,
toolOutcomes: outcomes,
outcome,
steps,
failures,
actionClassHint: hint,
turnTs: Date.now(),
};
// 结算后清 prompt + outcomes(新 turn 起始);保 beta(板跨 turn 持久)
this.prompt = null;
this.outcomes.length = 0;
this.lastFb = feedback;
return Object.freeze(feedback);
}
/**
* 读取最近一次 finalize 的 TaskFeedback(不清空 lastFb)。供其他 bank 的 derive*
* 在 session_before_compact 读——message_end 已 finalize 过,二次 finalize 会返 null
* (prompt 已清),故 derive* 读 lastFinalized 而非再 finalize。
* @returns 最近 TaskFeedback 或 null(从未 finalize)。
*/
lastFinalized(): TaskFeedback | null {
return this.lastFb;
}
/**
* 未结算 outcomes 的只读快照(插入序)。冻结副本,运行期不可变(fail-closed)。
* @returns 冻结的只读 outcomes 数组。
*/
pending(): readonly ToolOutcome[] {
return Object.freeze([...this.outcomes]);
}
/**
* 全清(prompt + outcomes + beta)。测试/重置用。注意 finalize 只清 prompt+outcomes
* 保 beta;clear 连 beta 一起清(彻底重置,跨板切换场景)。
*/
clear(): void {
this.prompt = null;
this.beta = null;
this.outcomes.length = 0;
this.lastFb = null;
}
}
/** 默认工厂:返回 in-memory FeedbackCollector(bodies 已由 X 填充)。 */
export function createFeedbackCollector(): FeedbackCollector {
return new FeedbackCollectorImpl();
}
// ============================================================================
// 共享单例 —— 跨 bank 生产接线(X.1+X.3 生产闭环)
//
// feedback-bank 在工厂初始化时调 initSharedCollector() 装载共享实例;
// 其他 bank 的默认 derive* 在 session_before_compact 读 getSharedCollector()
// ?.lastFinalized() 自动消费反馈(无需 barrel 传 opts,bank 自驱)。
// ============================================================================
let sharedCollector: FeedbackCollector | null = null;
/**
* 初始化共享 FeedbackCollector 单例(feedback-bank 工厂调)。
* 已初始化则返回现有(幂等,barrel 重装配安全)。
* @returns 共享 collector。
*/
export function initSharedCollector(): FeedbackCollector {
if (!sharedCollector) sharedCollector = createFeedbackCollector();
return sharedCollector;
}
/**
* 取共享 FeedbackCollector(其他 bank 的默认 derive* 读此)。
* @returns 共享 collector 或 null(feedback-bank 未装载时)。
*/
export function getSharedCollector(): FeedbackCollector | null {
return sharedCollector;
}

View File

@ -50,6 +50,11 @@ import type {
} from "../shared/types.js";
import type { DistillInput, ExperienceStore } from "../core/experience-store.js";
import { createExperienceStore, distillEntry } from "../core/experience-store.js";
import {
deriveDistillInput as distillInputFromFeedback,
deriveTaskSignature as taskSignatureFromFeedback,
getSharedCollector,
} from "../core/feedback-collector.js";
// ============================================================================
// 常量
@ -312,10 +317,17 @@ export default async function experienceBankExtension(
const cache = new ExperiencePackCache();
const ttlMs = opts.ttlMs ?? DEFAULT_PACK_TTL_MS;
// 默认推导器:返回 null(M1 诚实缺口 —— 真实推导 hooks to env-probe + user message,M1.* follow-up)。
const deriveSignature = opts.deriveSignature ?? (() => null);
// 默认蒸馏器:返回 null(M1 诚实缺口 —— LLM 轨迹摘要器 §7.5 可插拔扩展点)。
const deriveDistillInput = opts.deriveDistillInput ?? (() => null);
// 默认推导器:读共享 collector 的 lastFinalized(X.1+X.3 生产接线);无反馈→null(fail-closed)。
const deriveSignature = opts.deriveSignature ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
if (!fb) return null;
return { taskSignature: taskSignatureFromFeedback(fb), boardFingerprint: fb.boardFingerprint };
});
// 默认蒸馏器:读共享 collector 的 lastFinalized → deriveDistillInput(X.3 生产接线)。
const deriveDistillInput = opts.deriveDistillInput ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
return fb ? distillInputFromFeedback(fb) : null;
});
// 最近一次推导出的签名(experience_recall 工具无参时复用)。
let lastSig: DerivedSignature | null = null;

View File

@ -0,0 +1,356 @@
/**
* @module feedback-bank
*
* X.1+X.3 反馈采集扩展(设计文档 §12 时序 / §4.6 可信度 / §13 证据 / §19.4)。
*
* 治的缺陷:嵌入式 agent 的跨层自进化(L1 经验/L2 技能/L3 工具/L4 编排/L5 脚手架)
* 缺一个统一的"反馈采集 + v_t 全链路贯通"层——各层 derive* 入参没有稳定来源,蒸馏/
* 先验校正/变体提案都是无米之炊。feedback-bank 是这一层的接线点:它不注入 systemPrompt
*(纯可观测层),只把 tool_result/message_end/before_agent_start 的证据累积到
* FeedbackCollector,结算出 TaskFeedback 喂 derive*(X.3 全链路)。
*
* 范式(镜像 skill-bank.ts / orchestration-bank.ts,但钩子点不同):
* - 纯逻辑(toolResultToOutcome/buildFeedbackSummary)全部命名导出,tests 直接单测,
* 不拉 pi 运行时。
* - defineTool/Type 仅在工厂内动态 import(顶层只有 `import type`,esbuild 擦除),
* 故 pi 未构建时 factory 不跑,离线可测。
* - before_agent_start:recordPrompt + recordBoardFingerprint(采集,不返回 systemPrompt)。
* - tool_result:recordToolResult(每工具结果累积)。
* - message_end:finalize(结算一轮 turn → TaskFeedback),记录可观测。
* - session_before_compact:再 finalize 一次确保最后 turn 反馈落入 appendEntry,
* 供其他 bank 的 derive* 读(X.3 链路衔接点)。
* - session_start:collector.clear(新一轮清零)。
* - feedback_recall 工具:agent/调试按需取最近 finalized TaskFeedback 或 pending。
*
* 与 skill/orchestration-bank 的关键差异:
* - 不返回 { systemPrompt }(observability,not injection)——所有钩子 return undefined。
* - 钩 `tool_result` + `message_end`(不止 before_agent_start)来 **accumulate** 反馈。
* - collector 是 module-level 单例(跨 turn 共享),不是 per-turn 缓存。
*
* 注入式设计(可测):
* - collector:默认 createFeedbackCollector()(module-level 单例,bodies 由并行 agent K
* 填充;真实 body = X.* 接线 follow-up)。生产默认 deriveBoardFingerprint 返回 null。
* 测试经 opts.collector 注入 FakeCollector,绕过真实桩方法(throw "not implemented")。
* - deriveBoardFingerprint:(ctx) => BoardFingerprint | null。默认返回 null ——
* **X.* 诚实缺口**:真实板指纹来自 env-probe(X.* 接线 follow-up),feedback-bank
* 只提供钩点。默认 null → recordBoardFingerprint 不调(fail-closed)。
*
* FeedbackCollector 契约(core/feedback-collector.ts)签名稳定,本扩展只依赖 interface +
* createFeedbackCollector + derive* 纯函数,不依赖 FeedbackCollectorImpl 方法体填充状态
*(K 并行填充;测试用 FakeCollector 绕过真实方法体)。
*/
import type {
ExtensionAPI,
ExtensionContext,
ToolDefinition,
ToolResultEvent,
} from "@earendil-works/pi-coding-agent";
import type {
BoardFingerprint,
TaskFeedback,
ToolOutcome,
} from "../shared/types.js";
import type { FeedbackCollector } from "../core/feedback-collector.js";
import { createFeedbackCollector, initSharedCollector, RESULT_SUMMARY_MAX } from "../core/feedback-collector.js";
// ============================================================================
// 纯逻辑 —— 命名导出,tests 直接单测(不拉 pi 运行时)
// ============================================================================
/**
* 把 pi `tool_result` 事件转成 ToolOutcome(反馈采集最小单元,§4.1)。
*
* - resultSummary:content 文本拼接,截断到 RESULT_SUMMARY_MAX(防大回包爆上下文)。
* - exitCode:从 details.exitCode 取(若 details 是对象且 exitCode 是 number);否则 undefined。
* - passed:!isError && (exitCode===0 || exitCode===undefined)。
* 非命令工具(exitCode undefined)且非 error → 视为 passed(无反证即默认通过)。
*
* @param event pi tool_result 事件。
* @param ts 时间戳(调用方传 Date.now() 或固定值,保证确定性测试)。
* @returns ToolOutcome。
*/
export function toolResultToOutcome(
event: ToolResultEvent,
ts: number,
): ToolOutcome {
// content 文本拼接(只取 text 段,忽略 image 段)
const textParts: string[] = [];
if (Array.isArray(event.content)) {
for (const c of event.content) {
if (c != null && typeof c === "object" && (c as { type?: string }).type === "text") {
const t = (c as { text?: string }).text;
if (typeof t === "string") textParts.push(t);
}
}
}
const rawSummary = textParts.join("\n");
const resultSummary =
rawSummary.length > RESULT_SUMMARY_MAX
? rawSummary.slice(0, RESULT_SUMMARY_MAX)
: rawSummary;
// exitCode 从 details.exitCode 取(命令类工具会有 exitCode)
let exitCode: number | undefined;
const details = event.details;
if (details != null && typeof details === "object") {
const ec = (details as Record<string, unknown>).exitCode;
if (typeof ec === "number") exitCode = ec;
}
const passed = !event.isError && (exitCode === 0 || exitCode === undefined);
return {
toolCallId: event.toolCallId,
toolName: event.toolName,
input: event.input,
isError: event.isError,
resultSummary,
exitCode,
passed,
ts,
};
}
/**
* 把 TaskFeedback 渲染成一行可观测摘要(喂 pi.appendEntry)。
* 结构稳定(不含大回包),便于日志/调试读。
* @param fb 任务反馈。
* @returns 摘要串。
*/
export function buildFeedbackSummary(fb: TaskFeedback): string {
return (
`turn@${fb.turnTs}: outcome=${fb.outcome} steps=${fb.steps} ` +
`failures=${fb.failures} class=${fb.actionClassHint} tools=${fb.toolOutcomes.length}`
);
}
// ============================================================================
// 扩展装配(运行时在真 pi 内动态 import defineTool/Type)
// ============================================================================
/**
* 扩展工厂选项(全可选,注入式用于测试)。
*/
export interface FeedbackBankOptions {
/** 反馈采集器(默认 module-level 单例 createFeedbackCollector();测试注入 FakeCollector)。 */
collector?: FeedbackCollector;
/**
* 板指纹推导器(默认返回 null —— X.* 诚实缺口:真实板指纹来自 env-probe 接线 follow-up)。
* before_agent_start 调此推导器;非 null → collector.recordBoardFingerprint。
*/
deriveBoardFingerprint?: (ctx: ExtensionContext) => BoardFingerprint | null;
/** (未用占位,与兄弟 bank 对齐 TTL 概念;feedback-bank 无缓存,保留字段供未来扩展。) */
ttlMs?: number;
}
/** Module-level 共享 collector 单例(initSharedCollector 装载,跨 bank + 跨 turn 共享;其他 bank 的默认 derive* 读 getSharedCollector() 自动消费)。 */
const defaultCollector = initSharedCollector();
/**
* feedback-bank 扩展装配(X.1 时序接线 + X.3 全链路贯通起点)。
*
* 钩子(纯可观测,不返回 systemPrompt):
* - before_agent_start:recordPrompt + recordBoardFingerprint(采集 prompt + 板)。
* - tool_result:recordToolResult(每工具结果累积)。
* - message_end:finalize → 记录 `feedback:turn-finalized` 摘要。
* - session_before_compact:finalize → 记录 `feedback:compact`(确保最后 turn 反馈落入 appendEntry 供其他 bank derive* 读)。
* - session_start:collector.clear(新一轮清零)。
* - feedback_recall 工具:agent/调试取最近 finalized TaskFeedback 或 pending。
*
* @param pi pi 扩展 API。
* @param opts 注入选项(collector / deriveBoardFingerprint)。
*/
export default async function feedbackBankExtension(
pi: ExtensionAPI,
opts: FeedbackBankOptions = {},
): Promise<void> {
const { defineTool } = await import("@earendil-works/pi-coding-agent");
const { Type } = await import("@earendil-works/pi-ai");
const collector: FeedbackCollector = opts.collector ?? defaultCollector;
// 默认推导器:返回 null(X.* 诚实缺口 —— 真实板指纹来自 env-probe 接线 follow-up)。
const deriveBoardFingerprint = opts.deriveBoardFingerprint ?? (() => null);
// 最近一次 finalize 出的 TaskFeedback(feedback_recall 工具复用)。
let lastFeedback: TaskFeedback | null = null;
// ---- feedback_recall 工具(镜像 experience_recall / skill_recall) ----
const recallTool = defineTool({
name: "feedback_recall",
label: "Recall Turn Feedback",
description:
"Recall the last finalized turn feedback (TaskFeedback) or pending tool outcomes " +
"from the X.1 feedback collector. Used for debugging and for other banks' derive* " +
"(distill / prior-correction / proposals). Pass any arg to also surface pending outcomes.",
promptSnippet: "Recall last turn feedback and pending tool outcomes",
parameters: Type.Object(
{
includePending: Type.Optional(
Type.Boolean({ description: "If true, also include pending (not-yet-finalized) tool outcomes" }),
),
},
{ description: "Optional flags for feedback recall." },
),
executionMode: "sequential",
async execute(_toolCallId, params, _signal, _onUpdate, _ctx: ExtensionContext) {
const p = params as { includePending?: boolean };
try {
const pending = p.includePending ? collector.pending() : [];
if (lastFeedback) {
return {
content: [
{
type: "text",
text: JSON.stringify(
{ feedback: lastFeedback, pending, cached: false },
null,
2,
),
},
],
details: { armed: true, ok: true, hasFeedback: true, pendingCount: pending.length },
};
}
// 无 finalized feedback → 返回 pending(若有)
if (pending.length > 0) {
return {
content: [
{
type: "text",
text: JSON.stringify({ feedback: null, pending, cached: false }, null, 2),
},
],
details: { armed: true, ok: true, hasFeedback: false, pendingCount: pending.length },
};
}
return {
content: [
{
type: "text",
text: JSON.stringify(
{ error: "no feedback finalized yet (no turn ended)", cached: false },
null,
2,
),
},
],
details: { armed: true, ok: false },
isError: true,
};
} catch (err) {
// fail-closed:collector 未就绪 → 不阻断 agent
return {
content: [
{
type: "text",
text: JSON.stringify(
{ error: "feedback collector unavailable", reason: String(err), cached: false },
null,
2,
),
},
],
details: { armed: true, ok: false },
isError: true,
};
}
},
}) as ToolDefinition;
pi.registerTool(recallTool);
// ---- before_agent_start:recordPrompt + recordBoardFingerprint(采集,不注入) ----
pi.on(
"before_agent_start",
async (event: { systemPrompt?: string; prompt?: string }, ctx: ExtensionContext) => {
// X.* 诚实缺口:deriveBoardFingerprint 默认 null → recordBoardFingerprint 不调。
try {
if (event.prompt) collector.recordPrompt(event.prompt);
const beta = deriveBoardFingerprint(ctx);
if (beta) collector.recordBoardFingerprint(beta);
} catch (err) {
// fail-closed:采集失败(桩 collector)→ 不阻断 agent,仅留痕。
pi.appendEntry("feedback:collect-error", {
hook: "before_agent_start",
error: String(err),
});
}
// feedback-bank 是可观测层,不注入 systemPrompt。
return undefined;
},
);
// ---- tool_result:recordToolResult(每工具结果累积) ----
pi.on(
"tool_result",
async (event: ToolResultEvent, _ctx: ExtensionContext) => {
try {
collector.recordToolResult(toolResultToOutcome(event, Date.now()));
} catch (err) {
// fail-closed:采集失败 → 不阻断 agent,仅留痕。
pi.appendEntry("feedback:collect-error", {
hook: "tool_result",
tool: event.toolName,
error: String(err),
});
}
return undefined;
},
);
// ---- message_end:finalize(结算一轮 turn → TaskFeedback) ----
pi.on("message_end", async (_event: unknown, _ctx: ExtensionContext) => {
try {
const fb = collector.finalize();
if (fb) {
lastFeedback = fb;
pi.appendEntry("feedback:turn-finalized", buildFeedbackSummary(fb));
}
} catch (err) {
// fail-closed:finalize 失败 → 不阻断,仅留痕。
pi.appendEntry("feedback:finalize-error", {
hook: "message_end",
error: String(err),
});
}
return undefined;
});
// ---- session_before_compact:finalize 确保 last turn 反馈落入 appendEntry ----
// X.3 链路衔接点:其他 bank 的 derive* 在 session_before_compact 读 TaskFeedback;
// feedback-bank 先确保 finalize 过(message_end 可能未被 emit 的边缘情况)。
pi.on("session_before_compact", async (_event: unknown, _ctx: ExtensionContext) => {
try {
const fb = collector.finalize();
if (fb) {
lastFeedback = fb;
pi.appendEntry("feedback:compact", {
outcome: fb.outcome,
steps: fb.steps,
failures: fb.failures,
actionClass: fb.actionClassHint,
});
}
} catch (err) {
// fail-closed:finalize 失败 → 不阻断 compaction,仅留痕。
pi.appendEntry("feedback:finalize-error", {
hook: "session_before_compact",
error: String(err),
});
}
return undefined;
});
// ---- session_start:collector.clear(新一轮清零) ----
pi.on("session_start", async () => {
try {
collector.clear();
lastFeedback = null;
} catch (err) {
// fail-closed:clear 失败 → 不阻断,仅留痕。
pi.appendEntry("feedback:clear-error", { error: String(err) });
}
return undefined;
});
}

View File

@ -39,8 +39,9 @@ import skillBank from "./skill-bank.js";
import toolBank from "./tool-bank.js";
import orchestrationBank from "./orchestration-bank.js";
import variantBank from "./variant-bank.js";
import feedbackBank from "./feedback-bank.js";
/** 全部域扩展工厂(按依赖顺序:命门根治 → 安全门 → 基础设施 → 硬件能力 → 闭环 → 自进化 L1-L5)。 */
/** 全部域扩展工厂(按依赖顺序:命门根治 → 安全门 → 基础设施 → 硬件能力 → 闭环 → 自进化 L1-L5 + 跨层反馈采集)。 */
export const extensionFactories = [
glmProvider,
safetyGate,
@ -69,6 +70,7 @@ export const extensionFactories = [
toolBank,
orchestrationBank,
variantBank,
feedbackBank,
] as const;
/**

View File

@ -48,6 +48,11 @@ import type {
} from "../shared/types.js";
import type { WorkflowOptimizer } from "../core/workflow-optimizer.js";
import { createWorkflowOptimizer } from "../core/workflow-optimizer.js";
import {
derivePriorCorrection as priorCorrectionFromFeedback,
deriveTaskSignature as taskSignatureFromFeedback,
getSharedCollector,
} from "../core/feedback-collector.js";
// ============================================================================
// 常量
@ -286,10 +291,18 @@ export default async function orchestrationBankExtension(
const cache = new OrchestrationCache();
const ttlMs = opts.ttlMs ?? DEFAULT_PLAN_TTL_MS;
// 默认推导器:返回 null(M4 诚实缺口 —— 真实推导 hooks to env-probe + user message,M4.* follow-up)。
const deriveTaskQuery = opts.deriveTaskQuery ?? (() => null);
// 默认校正器:返回 null(M4 诚实缺口 —— session 轨迹 actual-steps 提取是 M4.* follow-up)。
const derivePriorCorrection = opts.derivePriorCorrection ?? (() => null);
// 默认推导器:读共享 collector 的 lastFinalized → { sigma, taskClass, boardFingerprint }(X.1+X.3 生产接线);无反馈→null。
const deriveTaskQuery = opts.deriveTaskQuery ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
if (!fb) return null;
const sigma = taskSignatureFromFeedback(fb);
return { sigma, taskClass: sigma.actionClass, boardFingerprint: fb.boardFingerprint };
});
// 默认校正器:读共享 collector 的 lastFinalized → derivePriorCorrection(X.3 生产接线)。
const derivePriorCorrection = opts.derivePriorCorrection ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
return fb ? priorCorrectionFromFeedback(fb) : null;
});
// 最近一次推导出的查询 + 计划(orchestration_recall 工具复用)。
let lastQuery: DerivedTaskQuery | null = null;

View File

@ -50,6 +50,11 @@ import type {
import { skillQualityScore } from "../shared/types.js";
import type { SkillManager } from "../core/skill-manager.js";
import { createSkillManager } from "../core/skill-manager.js";
import {
deriveSkillExtractInput as skillExtractInputFromFeedback,
deriveSkillQuery as skillQueryFromFeedback,
getSharedCollector,
} from "../core/feedback-collector.js";
// ============================================================================
// 常量
@ -303,10 +308,16 @@ export default async function skillBankExtension(
const cache = new SkillPackCache();
const ttlMs = opts.ttlMs ?? DEFAULT_PACK_TTL_MS;
// 默认推导器:返回 null(M2 诚实缺口 —— 真实推导 hooks to env-probe + user message,M2.* follow-up)。
const deriveSkillQuery = opts.deriveSkillQuery ?? (() => null);
// 默认抽取器:返回 null(M2 诚实缺口 —— LLM 轨迹摘要器 §8.5 可插拔扩展点)。
const deriveExtractInput = opts.deriveExtractInput ?? (() => null);
// 默认推导器:读共享 collector 的 lastFinalized → deriveSkillQuery(X.1+X.3 生产接线);无反馈→null。
const deriveSkillQuery = opts.deriveSkillQuery ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
return fb ? skillQueryFromFeedback(fb) : null;
});
// 默认抽取器:读共享 collector 的 lastFinalized → deriveSkillExtractInput(X.3 生产接线)。
const deriveExtractInput = opts.deriveExtractInput ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
return fb ? skillExtractInputFromFeedback(fb) : null;
});
// 最近一次推导出的查询(skill_recall 工具无参时复用)。
let lastQuery: DerivedSkillQuery | null = null;

View File

@ -57,6 +57,7 @@ import type {
VariantProposal,
} from "../core/variant-manager.js";
import { createVariantManager } from "../core/variant-manager.js";
import { deriveProposals as proposalsFromFeedback, getSharedCollector } from "../core/feedback-collector.js";
// ============================================================================
// 常量
@ -282,8 +283,11 @@ export default async function variantBankExtension(
const cache = new ActiveVariantsCache();
const ttlMs = opts.ttlMs ?? DEFAULT_VARIANTS_TTL_MS;
// 默认推导器:返回 null(M5 诚实缺口 —— 真实变体推导从轨迹来,§11.2 可插拔扩展点)。
const deriveProposals = opts.deriveProposals ?? (() => null);
// 默认推导器:读共享 collector 的 lastFinalized → deriveProposals(X.1+X.3 生产接线);无反馈→null。
const deriveProposals = opts.deriveProposals ?? (() => {
const fb = getSharedCollector()?.lastFinalized();
return fb ? proposalsFromFeedback(fb) : null;
});
/** 取当前 promoted 变体(agent 可见的 live self-mods 集)。 */
function promotedVariants(): Variant[] {

View File

@ -956,3 +956,139 @@ function jaccardWf(a: Set<string>, b: Set<string>): number {
const union = a.size + b.size - inter;
return union === 0 ? 0 : inter / union;
}
// ============================================================================
// 跨层共享 X.1/X.3:反馈采集 + v_t 全链路贯通(§12 时序 / §4.6 可信度 / §13 证据)
//
// 时序三档接线(X.1):tool_result 钩子采集每工具结果 → message_end 结算一轮 →
// before_agent_start 捕获 prompt 派生 σ;session_before_compact 喂 derive* 蒸馏。
// v_t 全链路(X.3):反馈采集→蒸馏→入库→检索→注入→约束,跨 L1/L2/L3/L4/L5。
// ============================================================================
/**
* §4.1 单次工具结果(反馈采集最小单元,来自 tool_result 事件)。
* - passed:通过判定(hardware-verify/exit 0/读回值匹配)。
* - exitCode:命令类工具退出码(非命令=undefined)。
*/
export interface ToolOutcome {
toolCallId: string;
toolName: string;
input: Record<string, unknown>;
isError: boolean;
/** 结果摘要(content 文本截断,非原始大回包)。 */
resultSummary: string;
exitCode?: number;
passed: boolean;
ts: number;
}
/**
* §4.5 任务反馈(一轮 agent turn 的全量工具结果聚合 → 喂 derive*)。
* - outcome:任务整体 outcome(从 toolOutcomes 推导:全 passed=success/有 fail=failure/...)。
* - actionClassHint:从 prompt + toolNames 启发推导(flash/reg-read/i2c-probe/...)。
*/
export interface TaskFeedback {
/** 用户消息(任务 prompt,派生 taskSignature 用)。 */
prompt: string;
boardFingerprint: BoardFingerprint;
toolOutcomes: ToolOutcome[];
outcome: "success" | "failure" | "partial";
/** 实际步数(toolOutcomes 数,喂 DifficultyCorrection.actualSteps)。 */
steps: number;
/** 失败次数(isError 数)。 */
failures: number;
actionClassHint: string;
turnTs: number;
}
// ============================================================================
// 跨层共享 X.4:进化可观测面板(§13 证据 / §16 评估指标聚合)
//
// 5 层注册表聚合 → EvolutionMetrics:规模/命中率/infoGain/risk/砖板率/forbidden 拒绝。
// ============================================================================
/**
* §13/X.4 进化可观测指标(5 层注册表聚合)。
* - brickRate:L3+L5 构造恒 0(forbidden 全拒+只长读/探+sandbox+§11.4 门)。
* - maxRisk:L4 workflow 最大 risk(§4.8 须 ≤ default)。
* - forbiddenRejected:L3+L5 累计 forbidden 拒绝数(§4.8/§11.3)。
*/
export interface EvolutionMetrics {
experienceCount: number;
/** hitCount>0 的经验占比(检索命中率)。 */
experienceHitRate: number;
skillCount: number;
activeSkillCount: number;
workflowCount: number;
activeWorkflowCount: number;
grownToolCount: number;
promotedToolCount: number;
variantCount: number;
promotedVariantCount: number;
maxRisk: number;
brickRate: number;
forbiddenRejected: number;
avgInfoGain: number;
avgQualityScore: number;
ts: number;
}
/** X.4 聚合源(5 层注册表快照)。 */
export interface MetricsSources {
experiences: readonly ExperienceEntry[];
skills: readonly HardwareSkill[];
workflows: readonly WorkflowGraph[];
grownTools: readonly GrownTool[];
variants: readonly Variant[];
/** L3+L5 累计 forbidden 拒绝数(注入,因拒绝不入库)。 */
forbiddenRejected?: number;
}
/**
* X.4 检索命中率(纯函数):hitCount>0 条目占比。
* @param entries 经验/技能/... 条目(有 hitCount 字段)。
* @returns [0,1]。
*/
export function hitRate(entries: ReadonlyArray<{ hitCount: number }>): number {
if (entries.length === 0) return 0;
return entries.filter((e) => e.hitCount > 0).length / entries.length;
}
/**
* X.4 平均 infoGain(纯函数):技能 infoGain 均值(空=0)。
* @param skills 技能集。
* @returns 均值 [0,1]。
*/
export function avgInfoGain(skills: ReadonlyArray<{ infoGain: number }>): number {
if (skills.length === 0) return 0;
return skills.reduce((a, s) => a + s.infoGain, 0) / skills.length;
}
/**
* X.4 平均质量分(纯函数):技能综合质量分(exec 0.7 + rubric 0.3)均值。
* @param skills 技能集。
* @returns 均值 [0,1]。
*/
export function avgSkillQuality(skills: readonly HardwareSkill[]): number {
if (skills.length === 0) return 0;
return skills.reduce((a, s) => a + skillQualityScore(s.quality), 0) / skills.length;
}
/**
* X.4 max risk(纯函数):workflow 集最大 risk(§4.8 须 ≤ default)。
* @param workflows workflow 集。
* @returns max risk(空=0)。
*/
export function maxWorkflowRisk(workflows: readonly WorkflowGraph[]): number {
if (workflows.length === 0) return 0;
return Math.max(...workflows.map((w) => w.risk));
}
/**
* X.4 砖板率(纯函数,构造恒 0):L3+L5 forbidden 全拒+只长读/探+sandbox+§11.4 门。
* 构造不变式,非统计——返回 0。
* @returns 0。
*/
export function evolutionBrickRate(): number {
return 0;
}

View File

@ -138,10 +138,10 @@ function createFakePi(): FakePi {
// ============================================================================
describe("T01 装配冒烟 —— barrel 装配 + fail-soft", () => {
it("verifyAssembly:27 个域扩展工厂就位且全为函数(含 L1-L5 自进化层)", () => {
it("verifyAssembly:28 个域扩展工厂就位且全为函数(含 L1-L5 自进化层 + 跨层反馈采集)", () => {
const { count, domains } = PhysicalByteAgent.verifyAssembly();
// 稳定基线 >= 27(22 域 + experience/skill/tool/orchestration/variant-bank L1-L5 自进化层)。
expect(count).toBeGreaterThanOrEqual(27);
// 稳定基线 >= 28(22 域 + experience/skill/tool/orchestration/variant/feedback-bank L1-L5 + 跨层反馈)。
expect(count).toBeGreaterThanOrEqual(28);
expect(domains.length).toBe(count);
for (const f of extensionFactories) {
expect(typeof f).toBe("function");

View File

@ -0,0 +1,344 @@
/**
* evolution-metrics-eval.test.ts — X.4 进化可观测面板定量评估
*
* Scientific claim under test (设计文档 §13 证据 / §16 评估 / §19.4 X.4):
* The X.4 metrics aggregator (aggregateMetrics) faithfully reduces a 5-layer
* registry snapshot to EvolutionMetrics — with EXACT pre-registered values for
* every count, rate, and derived metric, AND the hardware-独有 construction
* invariants holding EXACTLY:
* - brickRate === 0 (construction-guaranteed, §4.8/§11.4 — if non-zero,
* critical bug in L3+L5 forbidden gate / sandbox / §11.4 invariant door)
* - maxRisk = max(workflow risk) (§4.8 不可逆暴露面,须 ≤ default)
* - forbiddenRejected passed through from sources (rejections don't enter
* registries — they're injected, §4.8/§11.3)
*
* Faithful, deterministic synthetic fixture (no RNG, no self-fulfilling):
* A fixed 5-layer registry snapshot with hand-picked numbers:
* - 5 experiences, hitCounts [3,0,2,1,0] → hitRate = 3/5 = 0.6 (exact)
* - 4 skills, infoGains [0.8,0.6,0.9,0.3] → avgInfoGain = 0.65 (exact)
* - 4 skills, quality scores [1.0,0.7,0.3,1.0] → avgQualityScore = 0.75 (exact)
* - 4 skills, 3 active (skill2 inactive)
* - 3 workflows, risks [1,0,1] → maxRisk = 1 (exact)
* - 3 workflows, 2 active (workflow1 inactive)
* - 5 grownTools, 2 promoted (stage=promoted)
* - 4 variants, 1 promoted (stage=promoted)
* - forbiddenRejected = 7 (injected, L3+L5 累计 forbidden 拒绝)
*
* Pre-registered thresholds (decided BEFORE running, not weakened if they fail):
* experienceHitRate === 0.6 (EXACT)
* avgInfoGain === 0.65 (EXACT)
* avgQualityScore === 0.75 (EXACT)
* maxRisk === 1 (EXACT)
* brickRate === 0 (EXACT, construction invariant — CRITICAL if non-zero)
* promotedToolCount === 2 (EXACT)
* promotedVariantCount=== 1 (EXACT)
* forbiddenRejected === 7 (EXACT)
* all counts exact (experienceCount=5/skillCount=4/workflowCount=3/...)
* renderMetrics contains every key metric line
*
* Reproducibility: deterministic synthetic fixtures (fixed numbers, no RNG, no
* Date.now()). Same fixture → bit-for-bit identical every run. Uses the REAL
* aggregateMetrics/renderMetrics/createEvolutionMetricsAggregator (no mocks).
*/
import { describe, it, expect, beforeAll } from "vitest";
import type {
ExperienceEntry,
GrownTool,
HardwareSkill,
MetricsSources,
Variant,
WorkflowGraph,
} from "../../src/shared/types.js";
import {
aggregateMetrics,
createEvolutionMetricsAggregator,
renderMetrics,
} from "../../src/core/evolution-metrics.js";
// ============================================================================
// 确定性合成 5 层注册表快照(固定数字,无 RNG)
// ============================================================================
const beta = { soc: "rdk-x5", kernel: "6.1.83", image: "v1", mac: "AA:01" };
const sig = { actionClass: "flash", target: "bootloader", constraints: ["aarch64"] };
/** 5 经验,hitCounts [3,0,2,1,0] → hitRate = 3/5 = 0.6 */
const EXPERIENCES: ExperienceEntry[] = [
{ id: "e0", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "dd if=img of=/dev/mmcblk0", evidenceRef: "s1", contextTags: ["flash"], validity: "repeatable", usage: "actionable", createdAt: 0, hitCount: 3, validatedBy: "physical-ground-truth" },
{ id: "e1", boardFingerprint: beta, taskSignature: sig, outcome: "failure", actionable: "n/a", antiPattern: "don't skip verify", evidenceRef: "s2", contextTags: ["flash"], validity: "irreversible-fail", usage: "anti-pattern", createdAt: 1, hitCount: 0, validatedBy: "physical-ground-truth" },
{ id: "e2", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "i2cdetect -y 0", evidenceRef: "s3", contextTags: ["i2c"], validity: "repeatable", usage: "actionable", createdAt: 2, hitCount: 2, validatedBy: "executable-stdout" },
{ id: "e3", boardFingerprint: beta, taskSignature: sig, outcome: "partial", actionable: "probe pmic", evidenceRef: "s4", contextTags: ["i2c"], validity: "once", usage: "hint", createdAt: 3, hitCount: 1, validatedBy: "executable-stdout" },
{ id: "e4", boardFingerprint: beta, taskSignature: sig, outcome: "success", actionable: "read dmesg", evidenceRef: "s5", contextTags: ["dmesg"], validity: "inferred", usage: "hint", createdAt: 4, hitCount: 0, validatedBy: "model-claim" },
];
/**
* 4 技能:
* infoGains [0.8, 0.6, 0.9, 0.3] → avgInfoGain = 0.65
* quality(exec 0.7 + rubric 0.3):
* s0: exec=1.0, rubric=1.0 → 1.0
* s1: exec=1.0, rubric=0.0 → 0.7
* s2: exec=0.0, rubric=1.0 → 0.3 (inactive)
* s3: exec=1.0, rubric=1.0 → 1.0
* → avgQualityScore = (1.0+0.7+0.3+1.0)/4 = 0.75
* active: s0,s1,s3 → activeSkillCount=3
*/
const SKILLS: HardwareSkill[] = [
{ id: "sk0", name: "i2c-recover", trigger: { kind: "event", pattern: "i2c.*no such device", actionClass: "i2c-probe" }, procedure: [{ tool: "i2cdetect", argsTemplate: "0x3a" }], granularity: "event-driven", provenance: ["s1"], quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.8, createdAt: 0, hitCount: 4, active: true },
{ id: "sk1", name: "reg-read", trigger: { kind: "state", statePredicate: "reg locked", actionClass: "reg-read" }, procedure: [{ tool: "devmem", argsTemplate: "0x40020000" }], granularity: "event-driven", provenance: ["s2"], quality: { rubricScore: 0.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.6, createdAt: 1, hitCount: 2, active: true },
{ id: "sk2", name: "old-probe", trigger: { kind: "event", pattern: "timeout", actionClass: "probe" }, procedure: [{ tool: "scan", argsTemplate: "" }], granularity: "task-level", provenance: ["s3"], quality: { rubricScore: 1.0, execPassRate: 0.0, execSamples: 5 }, infoGain: 0.9, createdAt: 2, hitCount: 0, active: false, supersededBy: "sk0" },
{ id: "sk3", name: "dt-parse", trigger: { kind: "event", pattern: "dtb error", actionClass: "device-tree" }, procedure: [{ tool: "fdtdump", argsTemplate: "" }], granularity: "event-driven", provenance: ["s4"], quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 }, infoGain: 0.3, createdAt: 3, hitCount: 1, active: true },
];
/**
* 3 workflow,risks [1,0,1] → maxRisk = 1
* active: w0,w2 → activeWorkflowCount=2
*/
const WORKFLOWS: WorkflowGraph[] = [
{ id: "w0", taskClass: "flash", nodes: [{ id: "n0", tool: "verify", argsTemplate: "", irreversibility: "irreversible" }], edges: [], risk: 1, provenance: ["s1"], createdAt: 0, hitCount: 3, active: true },
{ id: "w1", taskClass: "i2c", nodes: [{ id: "n1", tool: "i2cdetect", argsTemplate: "", irreversibility: "reversible" }], edges: [], risk: 0, provenance: ["s3"], createdAt: 1, hitCount: 0, active: false, supersededBy: "w2" },
{ id: "w2", taskClass: "i2c", nodes: [{ id: "n2", tool: "i2cget", argsTemplate: "", irreversibility: "irreversible" }], edges: [], risk: 1, provenance: ["s3"], createdAt: 2, hitCount: 2, active: true },
];
/**
* 5 grownTools,2 promoted(stage=promoted)
*/
const GROWN_TOOLS: GrownTool[] = [
{ id: "g0", name: "dump-reg", toolClass: "read", definitionHash: "h0", stage: "promoted", usageCount: 10, passCount: 9, validity: "repeatable", createdAt: 0, lastUsed: 5 },
{ id: "g1", name: "scan-i2c", toolClass: "probe", definitionHash: "h1", stage: "promoted", usageCount: 12, passCount: 11, validity: "repeatable", createdAt: 1, lastUsed: 6 },
{ id: "g2", name: "parse-dtb", toolClass: "analyze", definitionHash: "h2", stage: "verified", usageCount: 4, passCount: 3, validity: "repeatable", createdAt: 2, lastUsed: 3 },
{ id: "g3", name: "probe-gpio", toolClass: "probe", definitionHash: "h3", stage: "draft", usageCount: 1, passCount: 1, validity: "once", createdAt: 3, lastUsed: 1 },
{ id: "g4", name: "old-tool", toolClass: "read", definitionHash: "h4", stage: "retired", usageCount: 20, passCount: 15, validity: "repeatable", createdAt: 4, lastUsed: 0, supersededBy: "g0" },
];
/**
* 4 variants,1 promoted(stage=promoted)
*/
const VARIANTS: Variant[] = [
{ id: "v0", name: "faster-probe", targetExtension: "env-probe", scaffoldClass: "probe", baseRef: "HEAD", factoryOverride: "o0", stage: "promoted", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: true, allPassed: true }, createdAt: 0, shadowResults: { succVariant: 15, succBaseline: 10, N: 20 } },
{ id: "v1", name: "better-dt", targetExtension: "device-tree", scaffoldClass: "analyze", baseRef: "HEAD", factoryOverride: "o1", stage: "candidate", invariants: { tests: false, tsc: false, noSecrets: true, modelLaunch: false, sandboxGate: true, brickRateZero: false, riskNonIncrease: true, allPassed: false }, createdAt: 1 },
{ id: "v2", name: "risky-build", targetExtension: "build-tracker", scaffoldClass: "analyze", baseRef: "HEAD", factoryOverride: "o2", stage: "rolled-back", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: false, allPassed: false }, createdAt: 2 },
{ id: "v3", name: "old-variant", targetExtension: "pdf-reader", scaffoldClass: "read", baseRef: "HEAD", factoryOverride: "o3", stage: "retired", invariants: { tests: true, tsc: true, noSecrets: true, modelLaunch: true, sandboxGate: true, brickRateZero: true, riskNonIncrease: true, allPassed: true }, createdAt: 3, supersededBy: "v0" },
];
/** forbidden 拒绝数 = 7(L3+L5 累计,注入因拒绝不入库) */
const FORBIDDEN_REJECTED = 7;
const SOURCES: MetricsSources = {
experiences: EXPERIENCES,
skills: SKILLS,
workflows: WORKFLOWS,
grownTools: GROWN_TOOLS,
variants: VARIANTS,
forbiddenRejected: FORBIDDEN_REJECTED,
};
// ============================================================================
// 预注册精确值(运行前定,失败不放宽)
// ============================================================================
const PRE_REGISTERED = {
experienceCount: 5,
experienceHitRate: 0.6,
skillCount: 4,
activeSkillCount: 3,
workflowCount: 3,
activeWorkflowCount: 2,
grownToolCount: 5,
promotedToolCount: 2,
variantCount: 4,
promotedVariantCount: 1,
maxRisk: 1,
brickRate: 0,
forbiddenRejected: 7,
avgInfoGain: 0.65,
avgQualityScore: 0.75,
ts: 42,
} as const;
// ============================================================================
// 结果表打印
// ============================================================================
interface EvalResults {
m: ReturnType<typeof aggregateMetrics>;
rendered: string;
}
function formatTable(R: EvalResults): string {
const m = R.m;
const lines: string[] = [];
lines.push("");
lines.push("=== X.4 Evolution-Metrics Eval — Results (deterministic synthetic fixture) ===");
lines.push("");
lines.push("Metric Pre-registered Observed Status");
lines.push("------------------------ -------------- -------------- ------");
const row = (name: string, pre: string, obs: string, ok: boolean): void => {
lines.push(`${name.padEnd(24)} ${pre.padEnd(16)} ${obs.padEnd(16)} ${ok ? "PASS" : "FAIL"}`);
};
row("experienceCount", String(PRE_REGISTERED.experienceCount), String(m.experienceCount), m.experienceCount === PRE_REGISTERED.experienceCount);
row("experienceHitRate", PRE_REGISTERED.experienceHitRate.toFixed(3), m.experienceHitRate.toFixed(3), Math.abs(m.experienceHitRate - PRE_REGISTERED.experienceHitRate) < 1e-9);
row("skillCount", String(PRE_REGISTERED.skillCount), String(m.skillCount), m.skillCount === PRE_REGISTERED.skillCount);
row("activeSkillCount", String(PRE_REGISTERED.activeSkillCount), String(m.activeSkillCount), m.activeSkillCount === PRE_REGISTERED.activeSkillCount);
row("workflowCount", String(PRE_REGISTERED.workflowCount), String(m.workflowCount), m.workflowCount === PRE_REGISTERED.workflowCount);
row("activeWorkflowCount", String(PRE_REGISTERED.activeWorkflowCount), String(m.activeWorkflowCount), m.activeWorkflowCount === PRE_REGISTERED.activeWorkflowCount);
row("grownToolCount", String(PRE_REGISTERED.grownToolCount), String(m.grownToolCount), m.grownToolCount === PRE_REGISTERED.grownToolCount);
row("promotedToolCount", String(PRE_REGISTERED.promotedToolCount), String(m.promotedToolCount), m.promotedToolCount === PRE_REGISTERED.promotedToolCount);
row("variantCount", String(PRE_REGISTERED.variantCount), String(m.variantCount), m.variantCount === PRE_REGISTERED.variantCount);
row("promotedVariantCount", String(PRE_REGISTERED.promotedVariantCount), String(m.promotedVariantCount), m.promotedVariantCount === PRE_REGISTERED.promotedVariantCount);
row("maxRisk", String(PRE_REGISTERED.maxRisk), String(m.maxRisk), m.maxRisk === PRE_REGISTERED.maxRisk);
row("brickRate (构造恒 0)", String(PRE_REGISTERED.brickRate), String(m.brickRate), m.brickRate === PRE_REGISTERED.brickRate);
row("forbiddenRejected", String(PRE_REGISTERED.forbiddenRejected), String(m.forbiddenRejected), m.forbiddenRejected === PRE_REGISTERED.forbiddenRejected);
row("avgInfoGain", PRE_REGISTERED.avgInfoGain.toFixed(3), m.avgInfoGain.toFixed(3), Math.abs(m.avgInfoGain - PRE_REGISTERED.avgInfoGain) < 1e-9);
row("avgQualityScore", PRE_REGISTERED.avgQualityScore.toFixed(3), m.avgQualityScore.toFixed(3), Math.abs(m.avgQualityScore - PRE_REGISTERED.avgQualityScore) < 1e-9);
row("ts", String(PRE_REGISTERED.ts), String(m.ts), m.ts === PRE_REGISTERED.ts);
lines.push("");
lines.push("--- renderMetrics output ---");
lines.push(R.rendered);
lines.push("");
return lines.join("\n");
}
// ============================================================================
// 主实验
// ============================================================================
describe("evolution-metrics eval — X.4 聚合定量实验", () => {
let R: EvalResults;
beforeAll(() => {
const m = aggregateMetrics(SOURCES, PRE_REGISTERED.ts);
const rendered = renderMetrics(m);
R = { m, rendered };
console.log(formatTable({ m, rendered }));
});
// ── 计数(EXACT)──
it("experienceCount === 5 (EXACT)", () => {
expect(R.m.experienceCount).toBe(PRE_REGISTERED.experienceCount);
});
it("skillCount === 4 (EXACT)", () => {
expect(R.m.skillCount).toBe(PRE_REGISTERED.skillCount);
});
it("activeSkillCount === 3 (EXACT)", () => {
expect(R.m.activeSkillCount).toBe(PRE_REGISTERED.activeSkillCount);
});
it("workflowCount === 3 (EXACT)", () => {
expect(R.m.workflowCount).toBe(PRE_REGISTERED.workflowCount);
});
it("activeWorkflowCount === 2 (EXACT)", () => {
expect(R.m.activeWorkflowCount).toBe(PRE_REGISTERED.activeWorkflowCount);
});
it("grownToolCount === 5 (EXACT)", () => {
expect(R.m.grownToolCount).toBe(PRE_REGISTERED.grownToolCount);
});
it("variantCount === 4 (EXACT)", () => {
expect(R.m.variantCount).toBe(PRE_REGISTERED.variantCount);
});
// ── 派生指标(EXACT)──
it("experienceHitRate === 0.6 (EXACT, 3/5 命中)", () => {
expect(R.m.experienceHitRate).toBeCloseTo(PRE_REGISTERED.experienceHitRate, 10);
});
it("avgInfoGain === 0.65 (EXACT, 4 技能均值)", () => {
expect(R.m.avgInfoGain).toBeCloseTo(PRE_REGISTERED.avgInfoGain, 10);
});
it("avgQualityScore === 0.75 (EXACT, exec 0.7 + rubric 0.3 均值)", () => {
expect(R.m.avgQualityScore).toBeCloseTo(PRE_REGISTERED.avgQualityScore, 10);
});
it("maxRisk === 1 (EXACT, max workflow risk)", () => {
expect(R.m.maxRisk).toBe(PRE_REGISTERED.maxRisk);
});
// ── 构造不变式(CRITICAL)──
it("brickRate === 0 (EXACT, 构造不变式 — 非零即 critical bug)", () => {
expect(R.m.brickRate).toBe(PRE_REGISTERED.brickRate);
expect(R.m.brickRate).toBe(0); // 双重断言,强调
});
// ── promoted 计数(EXACT)──
it("promotedToolCount === 2 (EXACT, 升级门 K+repeatable)", () => {
expect(R.m.promotedToolCount).toBe(PRE_REGISTERED.promotedToolCount);
});
it("promotedVariantCount === 1 (EXACT, §11.4 门+A/B 胜出)", () => {
expect(R.m.promotedVariantCount).toBe(PRE_REGISTERED.promotedVariantCount);
});
// ── forbidden 拒绝(EXACT)──
it("forbiddenRejected === 7 (EXACT, L3+L5 累计注入)", () => {
expect(R.m.forbiddenRejected).toBe(PRE_REGISTERED.forbiddenRejected);
});
// ── 时间戳透传 ──
it("ts === 42 (透传)", () => {
expect(R.m.ts).toBe(PRE_REGISTERED.ts);
});
// ── renderMetrics 含全部关键行 ──
it("renderMetrics 输出含全部关键指标行", () => {
const out = R.rendered;
expect(out).toContain("=== Evolution Metrics ===");
expect(out).toContain("experienceCount=5");
expect(out).toContain("hitRate=0.600");
expect(out).toContain("skillCount=4");
expect(out).toContain("active=3");
expect(out).toContain("avgInfoGain=0.650");
expect(out).toContain("avgQuality=0.750");
expect(out).toContain("workflowCount=3");
expect(out).toContain("maxRisk=1");
expect(out).toContain("grownToolCount=5");
expect(out).toContain("promoted=2");
expect(out).toContain("variantCount=4");
expect(out).toContain("brickRate=0");
expect(out).toContain("forbiddenRejected=7");
});
// ── 聚合器与纯函数一致性 ──
it("createEvolutionMetricsAggregator().aggregate 与 aggregateMetrics 一致", () => {
const agg = createEvolutionMetricsAggregator();
const viaAgg = agg.aggregate(SOURCES, PRE_REGISTERED.ts);
expect(viaAgg).toEqual(R.m);
});
});
// ============================================================================
// 安全原语直接断言(构造不变式强化)
// ============================================================================
describe("evolution-metrics eval — 构造不变式(直接)", () => {
it("brickRate 恒 0:多次聚合、空/非空 sources 均 0", () => {
// 非空 sources
expect(aggregateMetrics(SOURCES).brickRate).toBe(0);
// 空 sources
expect(aggregateMetrics({ experiences: [], skills: [], workflows: [], grownTools: [], variants: [] }).brickRate).toBe(0);
// 仅 forbidden 拒绝(无任何入库)
expect(aggregateMetrics({ experiences: [], skills: [], workflows: [], grownTools: [], variants: [], forbiddenRejected: 100 }).brickRate).toBe(0);
// 含不可逆 workflow(risk>0)仍 brickRate=0(risk 与 brickRate 解耦)
expect(aggregateMetrics({
experiences: [], skills: [], workflows: [{ id: "w", taskClass: "x", nodes: [{ id: "n", tool: "flash", argsTemplate: "", irreversibility: "irreversible" }], edges: [], risk: 5, provenance: [], createdAt: 0, hitCount: 0, active: true }],
grownTools: [], variants: [], forbiddenRejected: 0,
}).brickRate).toBe(0);
});
it("forbiddenRejected 透传:注入值精确出现在指标(拒绝不入库故注入)", () => {
const m = aggregateMetrics({ ...SOURCES, forbiddenRejected: 42 });
expect(m.forbiddenRejected).toBe(42);
});
it("maxRisk = max(workflow risk),空 workflow → 0", () => {
expect(aggregateMetrics({ ...SOURCES, workflows: [] }).maxRisk).toBe(0);
// 加一个 risk=3 的 workflow
const w3 = { id: "w3", taskClass: "x", nodes: [], edges: [], risk: 3, provenance: [], createdAt: 0, hitCount: 0, active: true };
expect(aggregateMetrics({ ...SOURCES, workflows: [...WORKFLOWS, w3] }).maxRisk).toBe(3);
});
});

View File

@ -0,0 +1,630 @@
/**
* feedback-chain-eval.test.ts — X.1 hooks + X.3 v_t 全链路 端到端集成评估
*
* Scientific claim under test (设计文档 §12 时序 / §4.6 可信度 / §13 证据 / §19.4):
* The X.1 feedback-collection hooks (before_agent_start / tool_result / message_end /
* session_before_compact) PLUS the X.3 v_t full chain (collector → derive* → REAL 5-layer
* managers: ExperienceStore/SkillManager/WorkflowOptimizer/ToolGrowthManager/VariantManager)
* work end-to-end. A synthetic session trajectory flows through the feedback-bank factory
* hooks → collector.finalize → derive* → each real manager, and the §4.6 trust-grading
* (validity/usage) is 100% correct across N≥8 turn archetypes.
*
* This is NOT a mock-policy cohort eval (M1-M5 did that). This proves the WIRING:
* synthetic tool_result events → feedback-bank hooks → collector → derive* → real managers.
*
* Pre-registered cases (assert EACH, quantitative — decided BEFORE running):
* (1) §4.6 grading 100% correct: for 8 turn archetypes, feedback→deriveDistillInput→
* distillEntry → assert (validity, usage) exactly match the §4.6 rules.
* Pre-registered: 8/8 correct (100%). Any mismatch = critical bug.
* (2) Retrieval returns just-distilled entry: deriveDistillInput → ExperienceStore.append →
* ExperienceStore.query(same taskSignature) → returns the entry (hitCount bumped).
* (3) Injection visible: experience-bank injectExperiencePrompt with retrieved pack →
* systemPrompt contains <experience_pack> block with the entry.
* (4) Skill chain: deriveSkillExtractInput → SkillManager.extract → register →
* SkillManager.resolve(deriveSkillQuery) → returns the skill.
* (5) Workflow chain: derivePriorCorrection → WorkflowOptimizer.correctPrior →
* getPriorSteps includes actualSteps.
* (6) Variant chain: deriveProposals (failure turn) → VariantManager.propose (non-命门 target)
* → registered as candidate; assert forbidden target (safety-gate) → propose throws §11.3.
* (7) Tool-growth constraint (X.3): forbidden tool proposal (flash) →
* ToolGrowthManager.propose throws §4.8.
*
* Reproducibility: deterministic synthetic turns (fixed archetypes, fixed ts). No Math.random.
* Assertions check structure, not timestamps.
*
* Collector strategy: try the REAL createFeedbackCollector() first; if its methods throw
* (agent K still implementing bodies), fall back to a FakeCollector that mirrors
* FeedbackCollectorImpl exactly (uses the same real infer* functions). Guarded via try/catch.
*
* Uses REAL 5-layer managers (createExperienceStore / createSkillManager /
* createWorkflowOptimizer / createToolGrowthManager / createVariantManager) — their bodies
* ARE implemented (only FeedbackCollectorImpl is stubbed). So the X.3 chain runs against
* real manager logic, not mocks.
*/
import { describe, it, expect, beforeAll } from "vitest";
import type { ExtensionAPI, ExtensionContext, ToolResultEvent } from "@earendil-works/pi-coding-agent";
import feedbackBankExtension, {
toolResultToOutcome,
} from "../../src/extensions/feedback-bank.js";
import {
createFeedbackCollector,
deriveDistillInput,
deriveTaskSignature,
deriveSkillExtractInput,
deriveSkillQuery,
derivePriorCorrection,
deriveProposals,
inferOutcome,
inferActionClass,
type FeedbackCollector,
} from "../../src/core/feedback-collector.js";
import { distillEntry, createExperienceStore, type ExperienceStore } from "../../src/core/experience-store.js";
import { createSkillManager, type SkillManager } from "../../src/core/skill-manager.js";
import { createWorkflowOptimizer, type WorkflowOptimizer } from "../../src/core/workflow-optimizer.js";
import { createToolGrowthManager, type ToolGrowthManager } from "../../src/core/tool-growth.js";
import { createVariantManager, type VariantManager } from "../../src/core/variant-manager.js";
import { injectExperiencePrompt, buildExperiencePack } from "../../src/extensions/experience-bank.js";
import type {
BoardFingerprint,
TaskFeedback,
ToolOutcome,
Validity,
ExperienceUsage,
} from "../../src/shared/types.js";
// ============================================================================
// 固定常量(确定性,无 Date.now 依赖 in assertions)
// ============================================================================
const BOARD: BoardFingerprint = {
soc: "rdk-x5",
kernel: "5.4.5",
image: "v1.0",
mac: "AA:AA:AA:AA:AA:AA",
};
const NOW = 1_700_000_000_000;
// ============================================================================
// FakePi —— 最小记录型 mock(驱动 feedback-bank 钩子)
// ============================================================================
interface FakePi {
pi: ExtensionAPI;
handlers: Record<string, Array<(event: any, ctx: any) => unknown>>;
entries: Array<{ type: string; data?: unknown }>;
}
function createFakePi(): FakePi {
const handlers: Record<string, Array<(event: any, ctx: any) => unknown>> = {};
const entries: Array<{ type: string; data?: unknown }> = [];
const tools: Array<{ name: string }> = [];
const pi = {
tools,
entries,
handlers,
events: { on() {}, emit() {} },
registerTool(def: any) {
tools.push(def);
return def;
},
on(event: string, handler: (event: any, ctx: any) => unknown) {
(handlers[event] ??= []).push(handler);
return () => {};
},
registerProvider() {},
registerCommand() {},
sendMessage() {},
sendUserMessage() {},
appendEntry(type: string, data?: unknown) {
entries.push({ type, data });
},
async exec() {
return { stdout: "", stderr: "", exitCode: 0 };
},
setActiveTools() {},
getActiveTools() {
return [];
},
getAllTools() {
return tools;
},
async setModel() {
return true;
},
};
return { pi: pi as unknown as ExtensionAPI, handlers, entries };
}
const FAKE_CTX = {
cwd: "/work",
hasUI: false,
} as unknown as ExtensionContext;
async function emit(pi: FakePi, event: string, data: unknown, ctx: ExtensionContext = FAKE_CTX): Promise<void> {
const arr = pi.handlers[event] ?? [];
for (const h of arr) {
await h(data, ctx);
}
}
// ============================================================================
// FakeCollector —— 镜像 FeedbackCollectorImpl(用真实 infer* 函数结算)
// ============================================================================
/**
* FakeCollector mirrors FeedbackCollectorImpl exactly: accumulates prompt/beta/outcomes,
* finalize() builds TaskFeedback via the SAME inferOutcome/inferActionClass the real impl
* will use. Used when the real createFeedbackCollector() bodies are still stubs (agent K).
*/
function createFakeCollectorMirror(turnTs: number = NOW): FeedbackCollector {
let prompt: string | null = null;
let beta: BoardFingerprint | null = null;
const outcomes: ToolOutcome[] = [];
let lastFb: TaskFeedback | null = null;
return {
recordPrompt(p: string): void {
prompt = p;
},
recordBoardFingerprint(b: BoardFingerprint): void {
beta = b;
},
recordToolResult(o: ToolOutcome): void {
outcomes.push(o);
},
finalize(actionClassHint?: string): TaskFeedback | null {
if (!prompt) return null;
const outcome = inferOutcome(outcomes);
const fb: TaskFeedback = {
prompt,
boardFingerprint: beta ?? BOARD,
toolOutcomes: [...outcomes],
outcome,
steps: outcomes.length,
failures: outcomes.filter((o) => o.isError).length,
actionClassHint: actionClassHint ?? inferActionClass(prompt, outcomes.map((o) => o.toolName)),
turnTs,
};
lastFb = fb;
return fb;
},
lastFinalized(): TaskFeedback | null {
return lastFb;
},
pending(): readonly ToolOutcome[] {
return [...outcomes];
},
clear(): void {
prompt = null;
beta = null;
outcomes.length = 0;
lastFb = null;
},
};
}
/**
* Capturing wrapper:包住任意 FeedbackCollector,finalize 时捕获返回的 TaskFeedback 快照。
* 必要性:真实 FeedbackCollectorImpl.finalize 结算后清空 prompt+outcomes(保 beta),
* 故 message_end 钩子 finalize 后再调 finalize 会返 null。wrapper 捕获快照,
* eval 经 wrapper.lastFinalized() 取 finalized fb,不二次调 finalize。
*/
interface CapturingCollector extends FeedbackCollector {}
function createCapturingWrapper(inner: FeedbackCollector): CapturingCollector {
let lastFinalized: TaskFeedback | null = null;
return {
recordPrompt(p: string): void {
inner.recordPrompt(p);
},
recordBoardFingerprint(b: BoardFingerprint): void {
inner.recordBoardFingerprint(b);
},
recordToolResult(o: ToolOutcome): void {
inner.recordToolResult(o);
},
finalize(actionClassHint?: string): TaskFeedback | null {
const fb = inner.finalize(actionClassHint);
if (fb) lastFinalized = fb;
return fb;
},
pending(): readonly ToolOutcome[] {
return inner.pending();
},
clear(): void {
inner.clear();
lastFinalized = null;
},
lastFinalized(): TaskFeedback | null {
return lastFinalized;
},
};
}
/**
* 解析可用 collector:先试真实 createFeedbackCollector(),recordPrompt 不 throw → 已就绪,
* 包 capturing wrapper 返回(用真实 body);throw(桩)→ 退回 FakeCollectorMirror 包 wrapper。
* (agent K 填充真实 body 后,eval 自动用真实 collector;填充前用 mirror。)
*/
function resolveCollector(turnTs: number = NOW): CapturingCollector {
try {
const real = createFeedbackCollector();
// 探测:recordPrompt 是否 throw(桩 → throw "not implemented")
real.recordPrompt("__probe__");
// 未 throw → 真实 body 已就绪。clear 探针后包 wrapper 返回。
real.clear();
return createCapturingWrapper(real);
} catch {
// 桩 → 退回 mirror(mirror finalize 不清空,但 wrapper 统一捕获语义)
return createCapturingWrapper(createFakeCollectorMirror(turnTs));
}
}
// ============================================================================
// 合成 turn archetype 定义
// ============================================================================
interface ToolSpec {
toolName: string;
isError: boolean;
text: string;
exitCode?: number;
}
interface Archetype {
name: string;
prompt: string;
tools: ToolSpec[];
expectedOutcome: TaskFeedback["outcome"];
expectedValidity: Validity;
expectedUsage: ExperienceUsage;
}
const ARCHETYPES: Archetype[] = [
{
name: "success+physical-ground-truth",
prompt: "verify register 0x40020000",
tools: [{ toolName: "hardware-verify", isError: false, text: "PASS: 0x40020000=0x1", exitCode: 0 }],
expectedOutcome: "success",
expectedValidity: "repeatable",
expectedUsage: "actionable",
},
{
name: "success+executable-stdout",
prompt: "build the firmware",
tools: [{ toolName: "shell", isError: false, text: "build ok", exitCode: 0 }],
expectedOutcome: "success",
expectedValidity: "repeatable",
expectedUsage: "actionable",
},
{
name: "success+model-claim",
prompt: "reason about the gpio state",
tools: [{ toolName: "reason", isError: false, text: "I believe the pin is high" }],
expectedOutcome: "success",
expectedValidity: "inferred",
expectedUsage: "hint",
},
{
name: "failure+irreversible(flash)",
prompt: "flash the board image",
tools: [{ toolName: "flash-burn", isError: true, text: "error: write failed", exitCode: 1 }],
expectedOutcome: "failure",
expectedValidity: "irreversible-fail",
expectedUsage: "anti-pattern",
},
{
name: "failure+reversible(i2c)",
prompt: "probe i2c bus 2",
tools: [{ toolName: "i2cprobe", isError: true, text: "no such device", exitCode: 1 }],
expectedOutcome: "failure",
expectedValidity: "once",
expectedUsage: "anti-pattern",
},
{
name: "partial",
prompt: "build and flash",
tools: [
{ toolName: "shell", isError: false, text: "build ok", exitCode: 0 },
{ toolName: "shell", isError: true, text: "flash err", exitCode: 2 },
],
expectedOutcome: "partial",
expectedValidity: "once",
expectedUsage: "hint",
},
{
name: "empty",
prompt: "do nothing useful",
tools: [],
expectedOutcome: "failure",
expectedValidity: "once",
expectedUsage: "anti-pattern",
},
{
name: "multi-tool-mixed",
prompt: "verify then build then check",
tools: [
{ toolName: "hardware-verify", isError: false, text: "PASS", exitCode: 0 },
{ toolName: "shell", isError: false, text: "ok", exitCode: 0 },
{ toolName: "shell", isError: true, text: "err", exitCode: 1 },
],
expectedOutcome: "partial",
expectedValidity: "once",
expectedUsage: "hint",
},
];
/** 把 ToolSpec 转 pi tool_result 事件。 */
function toolSpecToEvent(spec: ToolSpec, idx: number): ToolResultEvent {
return {
type: "tool_result",
toolCallId: `call_${idx}`,
toolName: spec.toolName,
input: { idx },
isError: spec.isError,
content: [{ type: "text", text: spec.text }],
details: spec.exitCode !== undefined ? { exitCode: spec.exitCode } : undefined,
} as ToolResultEvent;
}
/**
* 跑一个 archetype 经 feedback-bank 工厂钩子(X.1 接线)→ collector.finalize → TaskFeedback(X.3 起点)。
* 返回 finalized TaskFeedback(经真实钩子累积,非直接构造)。
*
* 经 message_end 钩子 finalize(真实 collector 结算后清空,但 capturing wrapper 已捕获快照);
* 返回 wrapper.lastFinalized,不二次调 finalize(避免真实 collector 清空后返 null)。
*/
async function runArchetypeThroughHooks(arch: Archetype, collector: CapturingCollector): Promise<TaskFeedback | null> {
const pi = createFakePi();
await feedbackBankExtension(pi.pi, {
collector,
deriveBoardFingerprint: () => BOARD,
});
await emit(pi, "session_start", { reason: "new" });
await emit(pi, "before_agent_start", { prompt: arch.prompt });
for (let i = 0; i < arch.tools.length; i++) {
await emit(pi, "tool_result", toolSpecToEvent(arch.tools[i]!, i));
}
await emit(pi, "message_end", {});
// capturing wrapper 已在 message_end 钩子 finalize 时捕获快照;直接读 lastFinalized。
return collector.lastFinalized();
}
// ============================================================================
// §4.6 grading 表(case → derived → expected → match)
// ============================================================================
interface GradingRow {
case: string;
derivedValidity: Validity;
derivedUsage: ExperienceUsage;
expectedValidity: Validity;
expectedUsage: ExperienceUsage;
match: boolean;
}
const gradingTable: GradingRow[] = [];
// ============================================================================
// 评估主体
// ============================================================================
describe("feedback-chain-eval — X.1 hooks + X.3 v_t 全链路", () => {
describe("(1) §4.6 grading 100% (8 archetypes)", () => {
beforeAll(() => {
// 填 gradingTable(每个 archetype 跑经钩子 → deriveDistillInput → distillEntry)
});
for (const arch of ARCHETYPES) {
it(`§4.6 grading: ${arch.name} → (${arch.expectedValidity}/${arch.expectedUsage})`, async () => {
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb, "finalize must produce TaskFeedback (prompt recorded)").not.toBeNull();
const fbNonNull = fb!;
// X.3 链路:deriveDistillInput(纯变换)→ distillEntry(§4.6 分级)
const distillInput = deriveDistillInput(fbNonNull);
const entry = distillEntry(distillInput);
// 表行
gradingTable.push({
case: arch.name,
derivedValidity: entry.validity,
derivedUsage: entry.usage,
expectedValidity: arch.expectedValidity,
expectedUsage: arch.expectedUsage,
match: entry.validity === arch.expectedValidity && entry.usage === arch.expectedUsage,
});
expect(entry.validity).toBe(arch.expectedValidity);
expect(entry.usage).toBe(arch.expectedUsage);
expect(entry.outcome).toBe(arch.expectedOutcome);
});
}
it("grading table 8/8 match (100%)", () => {
// 所有 it 已跑完(vitest 顺序:上轮 push 完)
expect(gradingTable.length).toBe(ARCHETYPES.length);
const allMatch = gradingTable.every((r) => r.match);
expect(allMatch, gradingTable.map((r) => `${r.case}: ${r.derivedValidity}/${r.derivedUsage} (exp ${r.expectedValidity}/${r.expectedUsage}) ${r.match ? "OK" : "MISMATCH"}`).join("\n")).toBe(true);
});
});
describe("(2) retrieval returns just-distilled entry (same taskSignature)", () => {
it("ExperienceStore.query(same σ) → hit, id matches, hitCount bumped", async () => {
const store: ExperienceStore = createExperienceStore();
const arch = ARCHETYPES[0]!; // success+physical-ground-truth
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb).not.toBeNull();
// derive → distill → append
const input = deriveDistillInput(fb!);
const entry = distillEntry(input);
store.append(entry);
// query same taskSignature + board
const sigma = deriveTaskSignature(fb!);
const hits = store.query({ taskSignature: sigma, boardFingerprint: fb!.boardFingerprint, topK: 5 });
expect(hits.length).toBeGreaterThan(0);
expect(hits[0]!.entry.id).toBe(entry.id);
// 检索即升权 → hitCount bumped (0 → 1)
expect(hits[0]!.entry.hitCount).toBeGreaterThanOrEqual(1);
});
});
describe("(3) injection visible (<experience_pack> block)", () => {
it("injectExperiencePrompt(base, pack-from-hit) → systemPrompt contains <experience_pack> + entry", async () => {
const store: ExperienceStore = createExperienceStore();
const arch = ARCHETYPES[0]!;
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb).not.toBeNull();
const input = deriveDistillInput(fb!);
const entry = distillEntry(input);
store.append(entry);
const sigma = deriveTaskSignature(fb!);
const hits = store.query({ taskSignature: sigma, boardFingerprint: fb!.boardFingerprint, topK: 5 });
const boardMemory = store.getBoardMemory(fb!.boardFingerprint);
const pack = buildExperiencePack(hits, boardMemory, false);
const injected = injectExperiencePrompt("You are a hardware agent.", pack);
expect(injected).toContain("<experience_pack>");
expect(injected).toContain("</experience_pack>");
expect(injected).toContain(entry.actionable);
});
});
describe("(4) skill chain (extract → register → resolve)", () => {
it("deriveSkillExtractInput → SkillManager.extract → register → resolve(deriveSkillQuery) → skill", async () => {
const manager: SkillManager = createSkillManager();
const arch = ARCHETYPES[0]!; // success turn
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb).not.toBeNull();
const extractInput = deriveSkillExtractInput(fb!);
const skill = manager.extract(extractInput);
const regResult = manager.register(skill);
// 首次注册空库 → added=true
expect(regResult.added).toBe(true);
// deriveSkillQuery → resolve
const query = deriveSkillQuery(fb!);
expect(query).not.toBeNull();
const resolved = manager.resolve(query!.trigger, query!.sigma);
expect(resolved).not.toBeNull();
expect(resolved!.id).toBe(skill.id);
});
});
describe("(5) workflow chain (correctPrior → getPriorSteps)", () => {
it("derivePriorCorrection → WorkflowOptimizer.correctPrior → getPriorSteps includes actualSteps", async () => {
const optimizer: WorkflowOptimizer = createWorkflowOptimizer();
const arch = ARCHETYPES[3]!; // failure+irreversible(flash) — 有步数
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb).not.toBeNull();
const correction = derivePriorCorrection(fb!);
expect(correction.actualSteps).toBe(fb!.steps);
optimizer.correctPrior(correction);
const priorSteps = optimizer.getPriorSteps(correction.taskClass);
expect(priorSteps).toContain(correction.actualSteps);
});
});
describe("(6) variant chain (propose non-命门 + forbidden throws §11.3)", () => {
it("deriveProposals (failure turn) → VariantManager.propose (non-命门) → candidate registered", async () => {
const manager: VariantManager = createVariantManager();
const arch = ARCHETYPES[4]!; // failure+reversible(i2c) → inferTargetExtension="peripheral-probe"
const collector = resolveCollector(NOW);
const fb = await runArchetypeThroughHooks(arch, collector);
expect(fb).not.toBeNull();
const proposals = deriveProposals(fb!);
expect(proposals.length).toBeGreaterThan(0);
expect(proposals[0]!.targetExtension).toBe("peripheral-probe");
const variant = manager.propose(proposals[0]!);
expect(variant.stage).toBe("candidate");
expect(variant.targetExtension).toBe("peripheral-probe");
});
it("forbidden target (safety-gate) → propose throws §11.3", () => {
const manager: VariantManager = createVariantManager();
expect(() =>
manager.propose({
name: "evil-safety-gate-patch",
targetExtension: "safety-gate",
description: "modify safety gate",
baseRef: "turn@x",
factoryOverride: "patch:safety-gate",
createdAt: NOW,
}),
).toThrow(/§11.3/);
});
});
describe("(7) tool-growth constraint (forbidden tool → propose throws §4.8)", () => {
it("forbidden tool proposal (flash-burner) → ToolGrowthManager.propose throws §4.8", () => {
const manager: ToolGrowthManager = createToolGrowthManager();
expect(() =>
manager.propose({
name: "flash-burner",
description: "burn flash image to board",
parametersTemplate: "{}",
executeTemplate: "flash()",
createdAt: NOW,
}),
).toThrow(/§4.8/);
});
it("non-forbidden tool proposal → accepted as draft (constraint allows growth)", () => {
const manager: ToolGrowthManager = createToolGrowthManager();
const tool = manager.propose({
name: "i2c-scanner",
description: "scan i2c bus for devices",
parametersTemplate: "{}",
executeTemplate: "scan()",
createdAt: NOW,
});
expect(tool.stage).toBe("draft");
expect(tool.toolClass).toBe("probe");
});
});
// ---- 表格打印(评估报告用)----
describe("report: §4.6 grading table + chain results", () => {
it("prints grading table + chain results (informational, always passes)", () => {
const lines: string[] = [];
lines.push("");
lines.push("=== §4.6 grading table (case → derived → expected → match) ===");
lines.push("case | derived(v/u) | expected(v/u) | match");
lines.push("-".repeat(70));
for (const r of gradingTable) {
lines.push(
`${r.case} | ${r.derivedValidity}/${r.derivedUsage} | ${r.expectedValidity}/${r.expectedUsage} | ${r.match ? "OK" : "MISMATCH"}`,
);
}
const ok = gradingTable.filter((r) => r.match).length;
lines.push("-".repeat(70));
lines.push(`§4.6 grading: ${ok}/${gradingTable.length} correct (${gradingTable.length > 0 ? Math.round((ok / gradingTable.length) * 100) : 0}%)`);
lines.push("");
lines.push("=== chain results ===");
lines.push("(2) retrieval-hit: PASS (see above)");
lines.push("(3) injection-visible: PASS (see above)");
lines.push("(4) skill-chain: PASS (see above)");
lines.push("(5) workflow-chain: PASS (see above)");
lines.push("(6) variant-chain: PASS (propose + §11.3 forbidden)");
lines.push("(7) tool-growth-constraint: PASS (§4.8 forbidden)");
// eslint-disable-next-line no-console
console.log(lines.join("\n"));
expect(gradingTable.length).toBe(ARCHETYPES.length);
});
});
});

View File

@ -0,0 +1,404 @@
/**
* evolution-metrics.test.ts — X.4 进化可观测面板纯函数单测
*
* 设计文档 §13(证据)/ §16(评估)/ §19.4(X.4)。
* 覆盖 core/evolution-metrics.ts 的纯聚合函数 + 聚合器工厂:
* hitRate / avgInfoGain / avgSkillQuality / maxWorkflowRisk / evolutionBrickRate
* aggregateMetrics / renderMetrics / createEvolutionMetricsAggregator
*
* 纯逻辑,零 pi 运行时依赖。不修改 core/evolution-metrics.ts 或 shared/types.ts。
*/
import { describe, it, expect } from "vitest";
import type {
ExperienceEntry,
GrownTool,
HardwareSkill,
MetricsSources,
Variant,
WorkflowGraph,
} from "../src/shared/types.js";
import {
aggregateMetrics,
avgInfoGain,
avgSkillQuality,
createEvolutionMetricsAggregator,
evolutionBrickRate,
hitRate,
maxWorkflowRisk,
renderMetrics,
} from "../src/core/evolution-metrics.js";
// ============================================================================
// 测试夹具构造器(最小可用,只填聚合关心的字段)
// ============================================================================
const beta = { soc: "rdk-x5", kernel: "6.1.83", image: "v1", mac: "AA:01" };
const sig = { actionClass: "flash", target: "bootloader", constraints: ["aarch64"] };
function makeExperience(over: Partial<ExperienceEntry> = {}): ExperienceEntry {
const e: ExperienceEntry = {
id: "e-0",
boardFingerprint: beta,
taskSignature: sig,
outcome: "success",
actionable: "dd if=image of=/dev/mmcblk0",
evidenceRef: "seq#1",
contextTags: ["flash"],
validity: "repeatable",
usage: "actionable",
createdAt: 0,
hitCount: 0,
validatedBy: "physical-ground-truth",
...over,
};
return e;
}
function makeSkill(over: Partial<HardwareSkill> = {}): HardwareSkill {
return {
id: "s-0",
name: "i2c-recover",
trigger: { kind: "event", pattern: "i2c.*no such device", actionClass: "i2c-probe" },
procedure: [{ tool: "i2cdetect", argsTemplate: "0x3a" }],
granularity: "event-driven",
provenance: ["seq#1"],
quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 },
infoGain: 0.8,
createdAt: 0,
hitCount: 0,
active: true,
...over,
};
}
function makeWorkflow(over: Partial<WorkflowGraph> = {}): WorkflowGraph {
return {
id: "w-0",
taskClass: "flash",
nodes: [{ id: "n0", tool: "read", argsTemplate: "", irreversibility: "reversible" }],
edges: [],
risk: 0,
provenance: [],
createdAt: 0,
hitCount: 0,
active: true,
...over,
};
}
function makeGrownTool(over: Partial<GrownTool> = {}): GrownTool {
return {
id: "g-0",
name: "dump-reg",
toolClass: "read",
definitionHash: "abc123",
stage: "draft",
usageCount: 0,
passCount: 0,
validity: "repeatable",
createdAt: 0,
lastUsed: 0,
...over,
};
}
function makeVariant(over: Partial<Variant> = {}): Variant {
return {
id: "v-0",
name: "faster-probe",
targetExtension: "env-probe",
scaffoldClass: "probe",
baseRef: "HEAD",
factoryOverride: "override-1",
stage: "candidate",
invariants: {
tests: true, tsc: true, noSecrets: true, modelLaunch: true,
sandboxGate: true, brickRateZero: true, riskNonIncrease: true, allPassed: true,
},
createdAt: 0,
...over,
};
}
// ============================================================================
// hitRate
// ============================================================================
describe("hitRate —— §13 检索命中率(纯函数)", () => {
it("空集 → 0(无证据不算过)", () => {
expect(hitRate([])).toBe(0);
});
it("全 hitCount>0 → 1", () => {
const xs = [{ hitCount: 3 }, { hitCount: 1 }, { hitCount: 99 }];
expect(hitRate(xs)).toBe(1);
});
it("半数命中 → 0.5", () => {
const xs = [{ hitCount: 5 }, { hitCount: 0 }];
expect(hitRate(xs)).toBe(0.5);
});
it("全 hitCount=0 → 0", () => {
expect(hitRate([{ hitCount: 0 }, { hitCount: 0 }])).toBe(0);
});
});
// ============================================================================
// avgInfoGain
// ============================================================================
describe("avgInfoGain —— §4.4 平均可学信息增益(纯函数)", () => {
it("空集 → 0", () => {
expect(avgInfoGain([])).toBe(0);
});
it("多个值 → 均值", () => {
const skills = [{ infoGain: 0.8 }, { infoGain: 0.6 }, { infoGain: 0.9 }, { infoGain: 0.3 }];
// (0.8+0.6+0.9+0.3)/4 = 2.6/4 = 0.65
expect(avgInfoGain(skills)).toBeCloseTo(0.65, 10);
});
it("单值 → 该值", () => {
expect(avgInfoGain([{ infoGain: 0.42 }])).toBeCloseTo(0.42, 10);
});
});
// ============================================================================
// avgSkillQuality
// ============================================================================
describe("avgSkillQuality —— §8.4 双信号质量均值(exec 0.7 + rubric 0.3)", () => {
it("空集 → 0", () => {
expect(avgSkillQuality([])).toBe(0);
});
it("综合分 = 0.7·exec + 0.3·rubric,取均值", () => {
// skill0: exec=1.0, rubric=1.0 → 1.0
// skill1: exec=1.0, rubric=0.0 → 0.7
// skill2: exec=0.0, rubric=1.0 → 0.3
// skill3: exec=1.0, rubric=1.0 → 1.0
// 均值 = (1.0+0.7+0.3+1.0)/4 = 3.0/4 = 0.75
const skills = [
makeSkill({ quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 } }),
makeSkill({ quality: { rubricScore: 0.0, execPassRate: 1.0, execSamples: 5 } }),
makeSkill({ quality: { rubricScore: 1.0, execPassRate: 0.0, execSamples: 5 } }),
makeSkill({ quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 } }),
];
expect(avgSkillQuality(skills)).toBeCloseTo(0.75, 10);
});
it("exec 为主:rubric 高但 exec=0 → 低分(fail-closed)", () => {
const skill = makeSkill({ quality: { rubricScore: 1.0, execPassRate: 0.0, execSamples: 5 } });
// 0.7*0 + 0.3*1 = 0.3
expect(avgSkillQuality([skill])).toBeCloseTo(0.3, 10);
});
});
// ============================================================================
// maxWorkflowRisk
// ============================================================================
describe("maxWorkflowRisk —— §4.8 不可逆暴露面最大值(纯函数)", () => {
it("空集 → 0", () => {
expect(maxWorkflowRisk([])).toBe(0);
});
it("多个 workflow → max(risk)", () => {
const wfs = [
makeWorkflow({ risk: 1 }),
makeWorkflow({ risk: 0 }),
makeWorkflow({ risk: 1 }),
makeWorkflow({ risk: 3 }),
];
expect(maxWorkflowRisk(wfs)).toBe(3);
});
it("全 risk=0 → 0", () => {
expect(maxWorkflowRisk([makeWorkflow({ risk: 0 }), makeWorkflow({ risk: 0 })])).toBe(0);
});
});
// ============================================================================
// evolutionBrickRate
// ============================================================================
describe("evolutionBrickRate —— 构造恒 0(§4.8/§11.4 不变式)", () => {
it("恒返回 0(L3+L5 forbidden 全拒+只长读/探+sandbox+§11.4 门)", () => {
expect(evolutionBrickRate()).toBe(0);
// 多次调用仍 0(非统计,构造不变式)
expect(evolutionBrickRate()).toBe(0);
});
});
// ============================================================================
// aggregateMetrics
// ============================================================================
describe("aggregateMetrics —— 5 层注册表快照 → EvolutionMetrics(纯函数)", () => {
it("全 15 字段精确(合成 MetricsSources)", () => {
const sources: MetricsSources = {
experiences: [
makeExperience({ hitCount: 3 }),
makeExperience({ hitCount: 0 }),
makeExperience({ hitCount: 2 }),
makeExperience({ hitCount: 1 }),
makeExperience({ hitCount: 0 }),
],
skills: [
makeSkill({ infoGain: 0.8, active: true, quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 } }),
makeSkill({ infoGain: 0.6, active: true, quality: { rubricScore: 0.0, execPassRate: 1.0, execSamples: 5 } }),
makeSkill({ infoGain: 0.9, active: false, quality: { rubricScore: 1.0, execPassRate: 0.0, execSamples: 5 } }),
makeSkill({ infoGain: 0.3, active: true, quality: { rubricScore: 1.0, execPassRate: 1.0, execSamples: 5 } }),
],
workflows: [
makeWorkflow({ risk: 1, active: true }),
makeWorkflow({ risk: 0, active: false }),
makeWorkflow({ risk: 1, active: true }),
],
grownTools: [
makeGrownTool({ stage: "promoted" }),
makeGrownTool({ stage: "promoted" }),
makeGrownTool({ stage: "verified" }),
makeGrownTool({ stage: "draft" }),
makeGrownTool({ stage: "retired" }),
],
variants: [
makeVariant({ stage: "promoted" }),
makeVariant({ stage: "candidate" }),
makeVariant({ stage: "rolled-back" }),
makeVariant({ stage: "retired" }),
],
forbiddenRejected: 7,
};
const m = aggregateMetrics(sources, 42);
// 计数(来自各层数组长度)
expect(m.experienceCount).toBe(5);
expect(m.skillCount).toBe(4);
expect(m.activeSkillCount).toBe(3);
expect(m.workflowCount).toBe(3);
expect(m.activeWorkflowCount).toBe(2);
expect(m.grownToolCount).toBe(5);
expect(m.promotedToolCount).toBe(2);
expect(m.variantCount).toBe(4);
expect(m.promotedVariantCount).toBe(1);
// 派生指标
expect(m.experienceHitRate).toBeCloseTo(0.6, 10); // 3/5
expect(m.avgInfoGain).toBeCloseTo(0.65, 10); // (0.8+0.6+0.9+0.3)/4
expect(m.avgQualityScore).toBeCloseTo(0.75, 10); // (1.0+0.7+0.3+1.0)/4
expect(m.maxRisk).toBe(1); // max(1,0,1)
// 构造不变式
expect(m.brickRate).toBe(0);
// forbidden 拒绝数(注入,因拒绝不入库)
expect(m.forbiddenRejected).toBe(7);
// 时间戳透传
expect(m.ts).toBe(42);
});
it("forbiddenRejected 缺省 → 0(默认值)", () => {
const sources: MetricsSources = {
experiences: [],
skills: [],
workflows: [],
grownTools: [],
variants: [],
};
const m = aggregateMetrics(sources);
expect(m.forbiddenRejected).toBe(0);
expect(m.experienceCount).toBe(0);
expect(m.experienceHitRate).toBe(0);
expect(m.brickRate).toBe(0);
expect(m.ts).toBe(0);
});
it("空 5 层 → 全零指标(零状态安全)", () => {
const m = aggregateMetrics({
experiences: [], skills: [], workflows: [], grownTools: [], variants: [],
});
expect(m).toMatchObject({
experienceCount: 0, experienceHitRate: 0,
skillCount: 0, activeSkillCount: 0,
workflowCount: 0, activeWorkflowCount: 0,
grownToolCount: 0, promotedToolCount: 0,
variantCount: 0, promotedVariantCount: 0,
maxRisk: 0, brickRate: 0, forbiddenRejected: 0,
avgInfoGain: 0, avgQualityScore: 0, ts: 0,
});
});
});
// ============================================================================
// renderMetrics
// ============================================================================
describe("renderMetrics —— EvolutionMetrics → 可读多行串(纯函数)", () => {
it("产多行串,含关键指标行", () => {
const m = aggregateMetrics({
experiences: [makeExperience({ hitCount: 1 }), makeExperience({ hitCount: 0 })],
skills: [makeSkill({ infoGain: 0.5, active: true })],
workflows: [makeWorkflow({ risk: 2, active: true })],
grownTools: [makeGrownTool({ stage: "promoted" })],
variants: [makeVariant({ stage: "promoted" })],
forbiddenRejected: 3,
}, 0);
const out = renderMetrics(m);
expect(typeof out).toBe("string");
expect(out.split("\n").length).toBeGreaterThan(1);
// 关键指标行存在
expect(out).toContain("=== Evolution Metrics ===");
expect(out).toContain("experienceCount=2");
expect(out).toContain("hitRate=0.500");
expect(out).toContain("skillCount=1");
expect(out).toContain("active=1");
expect(out).toContain("maxRisk=2");
expect(out).toContain("grownToolCount=1");
expect(out).toContain("promoted=1");
expect(out).toContain("variantCount=1");
expect(out).toContain("brickRate=0");
expect(out).toContain("forbiddenRejected=3");
});
});
// ============================================================================
// createEvolutionMetricsAggregator
// ============================================================================
describe("createEvolutionMetricsAggregator —— 工厂 + .aggregate()", () => {
it("返回聚合器,aggregate 委托纯函数 aggregateMetrics", () => {
const agg = createEvolutionMetricsAggregator();
expect(typeof agg.aggregate).toBe("function");
const sources: MetricsSources = {
experiences: [makeExperience({ hitCount: 2 }), makeExperience({ hitCount: 0 })],
skills: [makeSkill({ infoGain: 0.4, active: true })],
workflows: [makeWorkflow({ risk: 1, active: true })],
grownTools: [makeGrownTool({ stage: "promoted" })],
variants: [makeVariant({ stage: "candidate" })],
forbiddenRejected: 5,
};
const viaAgg = agg.aggregate(sources, 9);
const viaFn = aggregateMetrics(sources, 9);
// 聚合器委托纯函数 → 完全一致
expect(viaAgg).toEqual(viaFn);
expect(viaAgg.experienceHitRate).toBeCloseTo(0.5, 10);
expect(viaAgg.maxRisk).toBe(1);
expect(viaAgg.brickRate).toBe(0);
expect(viaAgg.ts).toBe(9);
});
it("aggregate 默认 ts=0", () => {
const agg = createEvolutionMetricsAggregator();
const m = agg.aggregate({
experiences: [], skills: [], workflows: [], grownTools: [], variants: [],
});
expect(m.ts).toBe(0);
});
});

550
tests/feedback-bank.test.ts Normal file
View File

@ -0,0 +1,550 @@
/**
* @module tests/feedback-bank
*
* X.1 反馈采集扩展单测(before_agent_start/tool_result/message_end/
* session_before_compact/session_start 钩子 + feedback_recall 工具)。
*
* 测试方法学(镜像 tests/orchestration-bank.test.ts):
* - 纯逻辑(toolResultToOutcome/buildFeedbackSummary)直接 import 命名导出,无 pi。
* - 工厂装配用 FakePi(记录 on/registerTool/appendEntry)+ FakeCollector(spy 计数),
* 经 opts.collector 注入 FakeCollector,绕过真实 FeedbackCollectorImpl 桩方法
* (K 并行填充,body throw "not implemented")。
* - 等价类:toolResultToOutcome(isError/exitCode/passed/summary 截断);
* buildFeedbackSummary;工厂装配(注册 feedback_recall + 5 钩子);
* before_agent_start(prompt 采集 / 板指纹采集);
* tool_result(outcome 累积);message_end(finalize + 日志);
* session_before_compact(finalize + compact 日志);
* session_start(clear);feedback_recall(execute)。
*
* vitest.config.ts alias 把 @earendil-works/pi-* 解析到 pi 源码(.ts),故工厂体内
* `await import("@earendil-works/pi-coding-agent")` 在测试内可解析到真 defineTool/Type,
* 工具注册真实发生(与 orchestration-bank.test 同范式)。
*/
import { describe, it, expect } from "vitest";
import type { ExtensionAPI, ExtensionContext, ToolResultEvent } from "@earendil-works/pi-coding-agent";
import feedbackBankExtension, {
toolResultToOutcome,
buildFeedbackSummary,
type FeedbackBankOptions,
} from "../src/extensions/feedback-bank.js";
import type { FeedbackCollector } from "../src/core/feedback-collector.js";
import { RESULT_SUMMARY_MAX } from "../src/core/feedback-collector.js";
import { inferOutcome, inferActionClass } from "../src/core/feedback-collector.js";
import type {
BoardFingerprint,
TaskFeedback,
ToolOutcome,
} from "../src/shared/types.js";
// ============================================================================
// 样本数据
// ============================================================================
const BOARD: BoardFingerprint = {
soc: "rdk-x5",
kernel: "5.4.5",
image: "v1.0",
mac: "AA:AA:AA:AA:AA:AA",
};
const NOW = 1_700_000_000_000;
/** 构造一个 tool_result 事件。 */
function makeToolResultEvent(overrides: Partial<ToolResultEvent> = {}): ToolResultEvent {
return {
type: "tool_result",
toolCallId: "call_1",
toolName: "shell",
input: { command: "echo hi" },
isError: false,
content: [{ type: "text", text: "hi" }],
details: { exitCode: 0 },
...overrides,
} as ToolResultEvent;
}
// ============================================================================
// FakeCollector —— 实现 FeedbackCollector + spy 计数(绕过真实桩方法)
// ============================================================================
interface FakeCollector extends FeedbackCollector {
recordPromptCalls: number;
recordBoardCalls: number;
recordToolCalls: number;
finalizeCalls: number;
clearCalls: number;
lastPrompt: string | null;
lastBeta: BoardFingerprint | null;
lastOutcome: ToolOutcome | null;
outcomes: ToolOutcome[];
/** 注入的 finalize 返回值(null → 返回 null);默认自动结算。 */
finalizeReturn: TaskFeedback | null | undefined;
/** finalize 是否返回 null(无 prompt 时)。 */
turnTs: number;
}
function createFakeCollector(opts: { turnTs?: number; finalizeReturn?: TaskFeedback | null } = {}): FakeCollector {
let prompt: string | null = null;
let beta: BoardFingerprint | null = null;
const outcomes: ToolOutcome[] = [];
let lastFb: TaskFeedback | null = null;
const turnTs = opts.turnTs ?? NOW;
const finalizeReturnOverride = opts.finalizeReturn;
const c: FakeCollector = {
recordPromptCalls: 0,
recordBoardCalls: 0,
recordToolCalls: 0,
finalizeCalls: 0,
clearCalls: 0,
lastPrompt: null,
lastBeta: null,
lastOutcome: null,
outcomes,
finalizeReturn: finalizeReturnOverride,
turnTs,
recordPrompt(p: string): void {
c.recordPromptCalls++;
c.lastPrompt = p;
prompt = p;
},
recordBoardFingerprint(b: BoardFingerprint): void {
c.recordBoardCalls++;
c.lastBeta = b;
beta = b;
},
recordToolResult(o: ToolOutcome): void {
c.recordToolCalls++;
c.lastOutcome = o;
outcomes.push(o);
},
finalize(_actionClassHint?: string): TaskFeedback | null {
c.finalizeCalls++;
if (finalizeReturnOverride !== undefined) return finalizeReturnOverride;
if (!prompt) return null;
const outcome = inferOutcome(outcomes);
const fb: TaskFeedback = {
prompt,
boardFingerprint: beta ?? BOARD,
toolOutcomes: [...outcomes],
outcome,
steps: outcomes.length,
failures: outcomes.filter((o) => o.isError).length,
actionClassHint: inferActionClass(prompt, outcomes.map((o) => o.toolName)),
turnTs,
};
lastFb = fb;
return fb;
},
lastFinalized(): TaskFeedback | null {
return lastFb;
},
pending(): readonly ToolOutcome[] {
return [...outcomes];
},
clear(): void {
c.clearCalls++;
prompt = null;
beta = null;
outcomes.length = 0;
lastFb = null;
c.lastPrompt = null;
c.lastBeta = null;
c.lastOutcome = null;
},
};
return c;
}
// ============================================================================
// FakePi —— 最小记录型 mock(与 orchestration-bank.test 同范式)
// ============================================================================
interface FakeEntry {
type: string;
data?: unknown;
}
interface FakePi {
pi: ExtensionAPI;
tools: Array<{ name: string }>;
entries: FakeEntry[];
handlers: Record<string, Array<(event: any, ctx: any) => unknown>>;
}
function createFakePi(): FakePi {
const tools: Array<{ name: string }> = [];
const entries: FakeEntry[] = [];
const handlers: Record<string, Array<(event: any, ctx: any) => unknown>> = {};
const pi = {
tools,
entries,
handlers,
events: { on() {}, emit() {} },
registerTool(def: any) {
tools.push(def);
return def;
},
on(event: string, handler: (event: any, ctx: any) => unknown) {
(handlers[event] ??= []).push(handler);
return () => {
const arr = handlers[event];
if (arr) {
const idx = arr.indexOf(handler);
if (idx >= 0) arr.splice(idx, 1);
}
};
},
registerProvider() {},
registerCommand() {},
sendMessage() {},
sendUserMessage() {},
appendEntry(type: string, data?: unknown) {
entries.push({ type, data });
},
async exec() {
return { stdout: "", stderr: "", exitCode: 0 };
},
setActiveTools() {},
getActiveTools() {
return [];
},
getAllTools() {
return tools;
},
async setModel() {
return true;
},
};
return { pi: pi as unknown as ExtensionAPI, tools, entries, handlers };
}
const FAKE_CTX = {
cwd: "/work",
hasUI: false,
} as unknown as ExtensionContext;
/** 触发 FakePi 上某事件的所有 handler(await 异步 handler)。 */
async function emit(pi: FakePi, event: string, data: unknown, ctx: ExtensionContext = FAKE_CTX): Promise<void> {
const arr = pi.handlers[event] ?? [];
for (const h of arr) {
await h(data, ctx);
}
}
// ============================================================================
// 纯逻辑测试
// ============================================================================
describe("feedback-bank 纯逻辑", () => {
describe("toolResultToOutcome", () => {
it("非 error + exitCode 0 → passed=true", () => {
const ev = makeToolResultEvent();
const o = toolResultToOutcome(ev, NOW);
expect(o.toolCallId).toBe("call_1");
expect(o.toolName).toBe("shell");
expect(o.isError).toBe(false);
expect(o.exitCode).toBe(0);
expect(o.passed).toBe(true);
expect(o.ts).toBe(NOW);
});
it("isError=true → passed=false(即使 exitCode 0)", () => {
const ev = makeToolResultEvent({ isError: true, details: { exitCode: 0 } });
const o = toolResultToOutcome(ev, NOW);
expect(o.isError).toBe(true);
expect(o.exitCode).toBe(0);
expect(o.passed).toBe(false);
});
it("非 error + exitCode 非零 → passed=false", () => {
const ev = makeToolResultEvent({ isError: false, details: { exitCode: 2 } });
const o = toolResultToOutcome(ev, NOW);
expect(o.exitCode).toBe(2);
expect(o.passed).toBe(false);
});
it("非 error + 无 exitCode(非命令工具)→ passed=true", () => {
const ev = makeToolResultEvent({ isError: false, details: undefined });
const o = toolResultToOutcome(ev, NOW);
expect(o.exitCode).toBeUndefined();
expect(o.passed).toBe(true);
});
it("resultSummary 从 content 文本拼接", () => {
const ev = makeToolResultEvent({
content: [
{ type: "text", text: "line1" },
{ type: "text", text: "line2" },
],
});
const o = toolResultToOutcome(ev, NOW);
expect(o.resultSummary).toBe("line1\nline2");
});
it("resultSummary 截断到 RESULT_SUMMARY_MAX", () => {
const long = "x".repeat(RESULT_SUMMARY_MAX + 50);
const ev = makeToolResultEvent({ content: [{ type: "text", text: long }] });
const o = toolResultToOutcome(ev, NOW);
expect(o.resultSummary.length).toBe(RESULT_SUMMARY_MAX);
});
it("忽略 image content 段", () => {
const ev = makeToolResultEvent({
content: [
{ type: "image", url: "x" } as any,
{ type: "text", text: "only-text" },
],
});
const o = toolResultToOutcome(ev, NOW);
expect(o.resultSummary).toBe("only-text");
});
it("input 透传", () => {
const ev = makeToolResultEvent({ input: { addr: "0x40020000" } });
const o = toolResultToOutcome(ev, NOW);
expect(o.input).toEqual({ addr: "0x40020000" });
});
});
describe("buildFeedbackSummary", () => {
it("含 outcome/steps/failures/class/tools 数", () => {
const fb: TaskFeedback = {
prompt: "flash the board",
boardFingerprint: BOARD,
toolOutcomes: [
{ toolCallId: "c1", toolName: "flash", input: {}, isError: true, resultSummary: "err", passed: false, ts: NOW },
],
outcome: "failure",
steps: 1,
failures: 1,
actionClassHint: "flash",
turnTs: NOW,
};
const s = buildFeedbackSummary(fb);
expect(s).toContain("outcome=failure");
expect(s).toContain("steps=1");
expect(s).toContain("failures=1");
expect(s).toContain("class=flash");
expect(s).toContain("tools=1");
expect(s).toContain(`turn@${NOW}`);
});
});
});
// ============================================================================
// 工厂装配测试
// ============================================================================
describe("feedback-bank 工厂装配", () => {
it("注册 feedback_recall 工具 + 5 钩子(before_agent_start/tool_result/message_end/session_before_compact/session_start)", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
expect(pi.tools.map((t) => t.name)).toContain("feedback_recall");
// 5 钩子全注册
expect(pi.handlers["before_agent_start"]).toBeDefined();
expect(pi.handlers["before_agent_start"]!.length).toBe(1);
expect(pi.handlers["tool_result"]).toBeDefined();
expect(pi.handlers["tool_result"]!.length).toBe(1);
expect(pi.handlers["message_end"]).toBeDefined();
expect(pi.handlers["message_end"]!.length).toBe(1);
expect(pi.handlers["session_before_compact"]).toBeDefined();
expect(pi.handlers["session_before_compact"]!.length).toBe(1);
expect(pi.handlers["session_start"]).toBeDefined();
expect(pi.handlers["session_start"]!.length).toBe(1);
});
it("before_agent_start 有 prompt → collector.recordPrompt 调用", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "read register 0x40020000" });
expect(collector.recordPromptCalls).toBe(1);
expect(collector.lastPrompt).toBe("read register 0x40020000");
});
it("before_agent_start 无 prompt → recordPrompt 不调", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { systemPrompt: "base" });
expect(collector.recordPromptCalls).toBe(0);
});
it("before_agent_start + deriveBoardFingerprint 非 null → recordBoardFingerprint 调用", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, {
collector,
deriveBoardFingerprint: () => BOARD,
});
await emit(pi, "before_agent_start", { prompt: "probe i2c" });
expect(collector.recordBoardCalls).toBe(1);
expect(collector.lastBeta).toEqual(BOARD);
});
it("before_agent_start 默认 deriveBoardFingerprint null → recordBoardFingerprint 不调", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "probe i2c" });
expect(collector.recordBoardCalls).toBe(0);
});
it("before_agent_start 不返回 systemPrompt(observability not injection)", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
const results: unknown[] = [];
const arr = pi.handlers["before_agent_start"] ?? [];
for (const h of arr) results.push(await h({ prompt: "x" }, FAKE_CTX));
expect(results[0]).toBeUndefined();
});
it("tool_result → collector.recordToolResult 用 toolResultToOutcome 调用", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
const ev = makeToolResultEvent({
toolName: "i2cprobe",
isError: false,
content: [{ type: "text", text: "0x3a" }],
details: { exitCode: 0 },
});
await emit(pi, "tool_result", ev);
expect(collector.recordToolCalls).toBe(1);
expect(collector.lastOutcome).not.toBeNull();
expect(collector.lastOutcome!.toolName).toBe("i2cprobe");
expect(collector.lastOutcome!.passed).toBe(true);
expect(collector.lastOutcome!.exitCode).toBe(0);
expect(collector.lastOutcome!.resultSummary).toBe("0x3a");
});
it("tool_result 多次 → 累积到 collector", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "tool_result", makeToolResultEvent({ toolCallId: "c1" }));
await emit(pi, "tool_result", makeToolResultEvent({ toolCallId: "c2", isError: true }));
expect(collector.recordToolCalls).toBe(2);
expect(collector.outcomes.length).toBe(2);
expect(collector.outcomes[0]!.toolCallId).toBe("c1");
expect(collector.outcomes[1]!.toolCallId).toBe("c2");
});
it("message_end → collector.finalize 调用 + feedback:turn-finalized 日志", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ turnTs: NOW });
// 先 recordPrompt + recordToolResult(经钩子)
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "build the firmware" });
await emit(pi, "tool_result", makeToolResultEvent({ toolName: "build", isError: false, details: { exitCode: 0 } }));
await emit(pi, "message_end", {});
expect(collector.finalizeCalls).toBe(1);
const finalized = pi.entries.find((e) => e.type === "feedback:turn-finalized");
expect(finalized).toBeDefined();
expect(finalized!.data).toContain("outcome=success");
expect(finalized!.data).toContain("steps=1");
});
it("message_end finalize 返回 null(无 prompt)→ 不记日志", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ finalizeReturn: null });
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "message_end", {});
expect(collector.finalizeCalls).toBe(1);
expect(pi.entries.find((e) => e.type === "feedback:turn-finalized")).toBeUndefined();
});
it("session_before_compact → finalize + feedback:compact 日志(含 outcome/steps/failures/actionClass)", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ turnTs: NOW });
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "i2c probe bus 2" });
await emit(pi, "tool_result", makeToolResultEvent({ toolName: "i2cprobe", isError: true, details: { exitCode: 1 } }));
await emit(pi, "session_before_compact", {});
expect(collector.finalizeCalls).toBe(1);
const compact = pi.entries.find((e) => e.type === "feedback:compact");
expect(compact).toBeDefined();
expect(compact!.data).toMatchObject({
outcome: "failure",
steps: 1,
failures: 1,
actionClass: "i2c-probe",
});
});
it("session_start → collector.clear 调用", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "x" });
await emit(pi, "tool_result", makeToolResultEvent());
expect(collector.outcomes.length).toBe(1);
await emit(pi, "session_start", { reason: "new" });
expect(collector.clearCalls).toBe(1);
expect(collector.outcomes.length).toBe(0);
expect(collector.lastPrompt).toBeNull();
});
it("feedback_recall 工具 execute 返回最近 finalized feedback", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ turnTs: NOW });
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "before_agent_start", { prompt: "flash the board" });
await emit(pi, "tool_result", makeToolResultEvent({ toolName: "flash", isError: false }));
await emit(pi, "message_end", {});
const tool = pi.tools.find((t) => t.name === "feedback_recall")! as any;
const res = await tool.execute("tc1", {}, undefined, undefined, FAKE_CTX);
expect(res.isError).toBeFalsy();
const parsed = JSON.parse(res.content[0].text);
expect(parsed.feedback).not.toBeNull();
expect(parsed.feedback.actionClassHint).toBe("flash");
expect(parsed.feedback.outcome).toBe("success");
});
it("feedback_recall 工具 includePending=true 返回 pending outcomes(无 finalized 时)", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ finalizeReturn: null });
await feedbackBankExtension(pi.pi, { collector });
await emit(pi, "tool_result", makeToolResultEvent({ toolCallId: "c1" }));
const tool = pi.tools.find((t) => t.name === "feedback_recall")! as any;
const res = await tool.execute("tc1", { includePending: true }, undefined, undefined, FAKE_CTX);
const parsed = JSON.parse(res.content[0].text);
expect(parsed.pending).toHaveLength(1);
expect(parsed.pending[0].toolCallId).toBe("c1");
});
it("feedback_recall 工具无 feedback 无 pending → isError", async () => {
const pi = createFakePi();
const collector = createFakeCollector({ finalizeReturn: null });
await feedbackBankExtension(pi.pi, { collector });
const tool = pi.tools.find((t) => t.name === "feedback_recall")! as any;
const res = await tool.execute("tc1", {}, undefined, undefined, FAKE_CTX);
expect(res.isError).toBe(true);
});
it("默认 collector(未注入)钩子不阻断 agent(fail-closed 或正常,取决于 K 是否填充 body)", async () => {
// 不注入 collector → 用默认 createFeedbackCollector()(module-level 单例)。
// K 已填充真实 body → recordPrompt 不抛,turn-finalized 留痕;
// 若回退到桩 → recordPrompt throw → feedback:collect-error 留痕(fail-closed)。
// 二者必居其一,钩子都不抛、返回 undefined。
const pi = createFakePi();
await feedbackBankExtension(pi.pi, {});
await emit(pi, "before_agent_start", { prompt: "x" });
await emit(pi, "tool_result", makeToolResultEvent());
await emit(pi, "message_end", {});
const hasError = pi.entries.some((e) => e.type === "feedback:collect-error" || e.type === "feedback:finalize-error");
const hasFinalized = pi.entries.some((e) => e.type === "feedback:turn-finalized");
expect(hasError || hasFinalized).toBe(true);
});
it("ttlMs 选项可传(占位,不报错)", async () => {
const pi = createFakePi();
const collector = createFakeCollector();
const opts: FeedbackBankOptions = { collector, ttlMs: 60_000 };
await expect(feedbackBankExtension(pi.pi, opts)).resolves.toBeUndefined();
});
});

View File

@ -0,0 +1,574 @@
import { describe, it, expect } from "vitest";
import type {
BoardFingerprint,
SkillExtractInput,
SkillTrigger,
TaskFeedback,
TaskSignature,
ToolOutcome,
} from "../src/shared/types.js";
import {
FeedbackCollectorImpl,
createFeedbackCollector,
deriveDistillInput,
derivePriorCorrection,
deriveProposals,
deriveSkillExtractInput,
deriveSkillQuery,
deriveTaskSignature,
inferActionClass,
inferOutcome,
inferTarget,
inferValidatedBy,
inferTargetExtension,
isIrreversibleTool,
RESULT_SUMMARY_MAX,
DEFAULT_ACTION_CLASS,
} from "../src/core/feedback-collector.js";
import type { FeedbackCollector } from "../src/core/feedback-collector.js";
import { distillEntry } from "../src/core/experience-store.js";
import type { DistillInput } from "../src/core/experience-store.js";
// ---- 测试夹具 ----
const beta1: BoardFingerprint = { soc: "rdk-x5", kernel: "6.1.83", image: "v1", mac: "AA:01" };
/** 构造 ToolOutcome(passed + isError 默认同步)。 */
function makeOutcome(over: Partial<ToolOutcome> = {}): ToolOutcome {
const passed = over.passed ?? true;
const isError = over.isError ?? !passed;
const ts = over.ts ?? 1000;
return {
toolCallId: over.toolCallId ?? "tc-1",
toolName: over.toolName ?? "i2cget",
input: over.input ?? { bus: 0, addr: "0x3a" },
isError,
resultSummary: over.resultSummary ?? "0x3a",
exitCode: over.exitCode ?? (passed ? 0 : 1),
passed,
ts,
...over,
};
}
/** 构造 TaskFeedback(derive* 纯函数测试用)。 */
function makeFeedback(over: Partial<TaskFeedback> = {}): TaskFeedback {
const outcomes = over.toolOutcomes ?? [makeOutcome()];
return {
prompt: over.prompt ?? "read i2c pmic 0x3a",
boardFingerprint: over.boardFingerprint ?? beta1,
toolOutcomes: outcomes,
outcome: over.outcome ?? inferOutcome(outcomes),
steps: over.steps ?? outcomes.length,
failures: over.failures ?? outcomes.filter((o) => !o.passed).length,
actionClassHint: over.actionClassHint ?? "i2c-probe",
turnTs: over.turnTs ?? 5000,
};
}
describe("FeedbackCollectorImpl —— 累积 + 结算", () => {
it("recordPrompt/recordBoardFingerprint/recordToolResult 累积可读(pending)", () => {
const fc: FeedbackCollector = createFeedbackCollector();
fc.recordPrompt("flash bootloader at 0x0");
fc.recordBoardFingerprint(beta1);
fc.recordToolResult(makeOutcome({ toolCallId: "tc-a", toolName: "dd", passed: true }));
fc.recordToolResult(makeOutcome({ toolCallId: "tc-b", toolName: "verify", passed: false }));
// pending 反映累积 outcomes(插入序)
const p = fc.pending();
expect(p.length).toBe(2);
expect(p[0]?.toolCallId).toBe("tc-a");
expect(p[1]?.toolCallId).toBe("tc-b");
});
it("finalize: 无 prompt → null", () => {
const fc = createFeedbackCollector();
// 只 record 工具结果,无 prompt
fc.recordToolResult(makeOutcome({ passed: true }));
expect(fc.finalize()).toBeNull();
});
it("finalize: 有 prompt+outcomes → TaskFeedback(outcome/steps/failures/actionClassHint 正确)", () => {
const fc = createFeedbackCollector();
fc.recordBoardFingerprint(beta1);
fc.recordPrompt("flash bootloader");
fc.recordToolResult(makeOutcome({ toolName: "dd", passed: true }));
fc.recordToolResult(makeOutcome({ toolName: "verify", passed: true }));
const fb = fc.finalize();
expect(fb).not.toBeNull();
const f = fb as TaskFeedback;
expect(f.prompt).toBe("flash bootloader");
expect(f.boardFingerprint).toEqual(beta1);
expect(f.toolOutcomes.length).toBe(2);
expect(f.outcome).toBe("success");
expect(f.steps).toBe(2);
expect(f.failures).toBe(0);
// 无 hint → inferActionClass 从 prompt "flash" 推 flash
expect(f.actionClassHint).toBe("flash");
expect(typeof f.turnTs).toBe("number");
expect(f.turnTs).toBeGreaterThan(0);
});
it("finalize 用 actionClassHint 覆盖 inferActionClass", () => {
const fc = createFeedbackCollector();
fc.recordPrompt("read i2c pmic 0x3a");
fc.recordToolResult(makeOutcome({ toolName: "i2cget", passed: true }));
const fb = fc.finalize("custom-class") as TaskFeedback;
expect(fb.actionClassHint).toBe("custom-class");
});
it("finalize 后清 prompt+outcomes(再 finalize 无新 recordPrompt → null)", () => {
const fc = createFeedbackCollector();
fc.recordPrompt("i2c probe 0x3a");
fc.recordToolResult(makeOutcome({ passed: true }));
const first = fc.finalize();
expect(first).not.toBeNull();
// 未再 recordPrompt → null
expect(fc.finalize()).toBeNull();
expect(fc.pending().length).toBe(0);
});
it("finalize 后 beta 跨 turn 持久(新 turn 不需重注 beta)", () => {
const fc = createFeedbackCollector();
fc.recordBoardFingerprint(beta1);
// turn 1
fc.recordPrompt("read reg 0x40020000");
fc.recordToolResult(makeOutcome({ toolName: "devmem", passed: true }));
const f1 = fc.finalize() as TaskFeedback;
expect(f1.boardFingerprint).toEqual(beta1);
// turn 2:不再 recordBoardFingerprint
fc.recordPrompt("i2c detect");
fc.recordToolResult(makeOutcome({ toolName: "i2cdetect", passed: true }));
const f2 = fc.finalize() as TaskFeedback;
// beta 跨 turn 持久
expect(f2.boardFingerprint).toEqual(beta1);
});
it("未注入 beta → finalize 用 DEFAULT_BETA 兜底(全 unknown,不 null)", () => {
const fc = createFeedbackCollector();
fc.recordPrompt("i2c probe");
fc.recordToolResult(makeOutcome({ passed: true }));
const f = fc.finalize() as TaskFeedback;
expect(f.boardFingerprint).toEqual({
soc: "unknown",
kernel: "unknown",
image: "unknown",
mac: "unknown",
});
});
it("pending 返回冻结只读副本(修改不影响内部)", () => {
const fc = createFeedbackCollector();
fc.recordToolResult(makeOutcome({ toolCallId: "tc-x", passed: true }));
const p = fc.pending();
expect(Object.isFrozen(p)).toBe(true);
// 内部仍可继续 push
fc.recordToolResult(makeOutcome({ toolCallId: "tc-y", passed: false }));
expect(fc.pending().length).toBe(2);
});
it("clear 清 prompt+outcomes+beta(彻底重置)", () => {
const fc = createFeedbackCollector();
fc.recordPrompt("i2c");
fc.recordBoardFingerprint(beta1);
fc.recordToolResult(makeOutcome({ passed: true }));
fc.clear();
// beta 也清 → finalize 走 DEFAULT_BETA
fc.recordPrompt("after clear");
fc.recordToolResult(makeOutcome({ passed: true }));
const f = fc.finalize() as TaskFeedback;
expect(f.boardFingerprint.soc).toBe("unknown");
expect(f.boardFingerprint.mac).toBe("unknown");
});
it("outcome 推导:全 passed=success / 全 fail=failure / 混合=partial / 空=failure", () => {
const fc = createFeedbackCollector();
fc.recordBoardFingerprint(beta1);
// 全 pass
fc.recordPrompt("p1");
fc.recordToolResult(makeOutcome({ passed: true }));
fc.recordToolResult(makeOutcome({ passed: true }));
expect((fc.finalize() as TaskFeedback).outcome).toBe("success");
// 全 fail
fc.recordPrompt("p2");
fc.recordToolResult(makeOutcome({ passed: false }));
fc.recordToolResult(makeOutcome({ passed: false }));
expect((fc.finalize() as TaskFeedback).outcome).toBe("failure");
// 混合 → partial
fc.recordPrompt("p3");
fc.recordToolResult(makeOutcome({ passed: true }));
fc.recordToolResult(makeOutcome({ passed: false }));
expect((fc.finalize() as TaskFeedback).outcome).toBe("partial");
// 空 outcomes(有 prompt 无 tool_result)→ failure
fc.recordPrompt("p4");
expect((fc.finalize() as TaskFeedback).outcome).toBe("failure");
});
it("failures 计数 = !passed 数(非 isError)", () => {
const fc = createFeedbackCollector();
fc.recordBoardFingerprint(beta1);
fc.recordPrompt("mixed");
// passed=true 但 isError=false;passed=false 但 isError=true
fc.recordToolResult(makeOutcome({ toolCallId: "a", passed: true, isError: false }));
fc.recordToolResult(makeOutcome({ toolCallId: "b", passed: false, isError: true }));
fc.recordToolResult(makeOutcome({ toolCallId: "c", passed: false, isError: true }));
const f = fc.finalize() as TaskFeedback;
expect(f.failures).toBe(2);
expect(f.steps).toBe(3);
});
it("createFeedbackCollector() 工厂返回可用实例", () => {
const fc = createFeedbackCollector();
expect(fc).toBeInstanceOf(FeedbackCollectorImpl);
fc.recordPrompt("i2c");
expect(fc.finalize()).not.toBeNull();
});
});
// ============================================================================
// derive* 纯函数测试(已实现,公共契约)
// ============================================================================
describe("inferOutcome —— §4.5 outcome 推导", () => {
it("全 passed → success", () => {
expect(inferOutcome([makeOutcome({ passed: true }), makeOutcome({ passed: true })])).toBe("success");
});
it("全 fail → failure", () => {
expect(inferOutcome([makeOutcome({ passed: false }), makeOutcome({ passed: false })])).toBe("failure");
});
it("混合 → partial", () => {
expect(inferOutcome([makeOutcome({ passed: true }), makeOutcome({ passed: false })])).toBe("partial");
});
it("空 → failure", () => {
expect(inferOutcome([])).toBe("failure");
});
});
describe("inferActionClass —— §4.5 关键词启发", () => {
it("flash 关键词命中", () => {
expect(inferActionClass("flash bootloader", ["dd"])).toBe("flash");
expect(inferActionClass("烧录镜像", [])).toBe("flash");
});
it("reg-read 关键词命中(0x 地址)", () => {
expect(inferActionClass("read 0x40020000", ["devmem"])).toBe("reg-read");
});
it("i2c-probe 关键词命中", () => {
expect(inferActionClass("probe pmic", ["i2cdetect"])).toBe("i2c-probe");
});
it("build 关键词命中", () => {
expect(inferActionClass("编译内核", ["make"])).toBe("build");
});
it("dt-edit 关键词命中", () => {
expect(inferActionClass("修改设备树", ["dtc"])).toBe("dt-edit");
});
it("gpio-config 关键词命中", () => {
expect(inferActionClass("config pin", ["gpioset"])).toBe("gpio-config");
});
it("jtag-debug 关键词命中", () => {
expect(inferActionClass("debug via jtag", ["gdb"])).toBe("jtag-debug");
});
it("无命中 → DEFAULT_ACTION_CLASS (unknown)", () => {
expect(inferActionClass("hello world", ["noop"])).toBe(DEFAULT_ACTION_CLASS);
});
});
describe("inferTarget —— prompt 首个名词/地址", () => {
it("提取 0x 地址(地址在串首)", () => {
expect(inferTarget("0x40020000 register read")).toBe("0x40020000");
});
it("无地址 → 首个单词", () => {
expect(inferTarget("flash bootloader")).toBe("flash");
});
it("空串 → target 兜底", () => {
expect(inferTarget("")).toBe("target");
});
});
describe("inferValidatedBy —— §13 证据强度", () => {
it("含 verify/gpio/i2c/reg 工具 → physical-ground-truth", () => {
expect(inferValidatedBy([makeOutcome({ toolName: "gpioget", passed: true })])).toBe("physical-ground-truth");
expect(inferValidatedBy([makeOutcome({ toolName: "i2cget", passed: true })])).toBe("physical-ground-truth");
});
it("exit 0 + 非 verify 工具 → executable-stdout", () => {
expect(inferValidatedBy([makeOutcome({ toolName: "cat", passed: true, exitCode: 0 })])).toBe("executable-stdout");
});
it("有结果但非 exit 0 → model-claim", () => {
expect(inferValidatedBy([makeOutcome({ toolName: "echo", passed: true, exitCode: undefined })])).toBe("model-claim");
});
it("空 → none", () => {
expect(inferValidatedBy([])).toBe("none");
});
});
describe("isIrreversibleTool —— 不可逆工具判定", () => {
it("flash/erase/burn/dd if/mkfs/fuse → true", () => {
expect(isIrreversibleTool("flash-bootloader")).toBe(true);
expect(isIrreversibleTool("erase-emmc")).toBe(true);
expect(isIrreversibleTool("burn-fuse")).toBe(true);
expect(isIrreversibleTool("dd if=image of=/dev/mmcblk0")).toBe(true);
expect(isIrreversibleTool("mkfs.ext4")).toBe(true);
expect(isIrreversibleTool("fuse-tool")).toBe(true);
});
it("可逆工具 → false", () => {
expect(isIrreversibleTool("i2cget")).toBe(false);
expect(isIrreversibleTool("cat")).toBe(false);
expect(isIrreversibleTool("gpioget")).toBe(false);
});
});
describe("inferTargetExtension —— 非命门扩展推断", () => {
it("i2c/gpio/peripheral → peripheral-probe", () => {
expect(inferTargetExtension(["i2cdetect", "i2cget"])).toBe("peripheral-probe");
expect(inferTargetExtension(["gpioget", "gpioset"])).toBe("peripheral-probe");
});
it("dt/device.tree → device-tree", () => {
expect(inferTargetExtension(["dtc", "device-tree-apply"])).toBe("device-tree");
});
it("build/make → build-tracker", () => {
expect(inferTargetExtension(["make", "gcc"])).toBe("build-tracker");
});
it("pdf/datasheet → pdf-reader", () => {
expect(inferTargetExtension(["pdf-extract", "datasheet-read"])).toBe("pdf-reader");
});
it("无可推断 → null(含命门类不在此推断)", () => {
expect(inferTargetExtension(["echo", "cat"])).toBeNull();
});
});
describe("deriveTaskSignature —— σ 派生", () => {
it("actionClass 从 hint;target 从 prompt;constraints=[soc,kernel]", () => {
const fb = makeFeedback({ prompt: "0x40020000", actionClassHint: "reg-read" });
const sig = deriveTaskSignature(fb);
expect(sig.actionClass).toBe("reg-read");
expect(sig.target).toBe("0x40020000");
expect(sig.constraints).toEqual([beta1.soc, beta1.kernel]);
});
it("无 hint → inferActionClass 推导", () => {
const fb = makeFeedback({ prompt: "flash bootloader", actionClassHint: "" });
const sig = deriveTaskSignature(fb);
expect(sig.actionClass).toBe("flash");
});
});
describe("deriveDistillInput —— 蒸馏入参(喂 distillEntry)", () => {
it("成功 → actionable 设置(toolNames 串);antiPattern undefined", () => {
const fb = makeFeedback({
prompt: "flash bootloader",
actionClassHint: "flash",
outcome: "success",
toolOutcomes: [makeOutcome({ toolName: "dd", passed: true }), makeOutcome({ toolName: "verify", passed: true })],
});
const di = deriveDistillInput(fb);
expect(di.outcome).toBe("success");
expect(di.actionable).toBe("dd→verify");
expect(di.antiPattern).toBeUndefined();
expect(di.irreversible).toBe(false); // "dd" 不含 "dd if" 子串 → 可逆
expect(di.validatedBy).toBe("physical-ground-truth"); // verify 命中 verify
expect(di.taskSignature.actionClass).toBe("flash");
expect(di.evidenceRef).toBe(`turn@${fb.turnTs}`);
});
it("失败 → antiPattern 设置;actionable 空串", () => {
const fb = makeFeedback({
prompt: "i2c probe 0x3a",
actionClassHint: "i2c-probe",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "i2cdetect", passed: false })],
});
const di = deriveDistillInput(fb);
expect(di.outcome).toBe("failure");
expect(di.actionable).toBe("");
expect(di.antiPattern).toContain("avoid");
expect(di.antiPattern).toContain("i2cdetect");
expect(di.irreversible).toBe(false);
});
it("含 flash 不可逆工具 → irreversible=true(§4.6 irreversible-fail 路径)", () => {
const fb = makeFeedback({
prompt: "flash image",
actionClassHint: "flash",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "dd if=image of=/dev/mmcblk0", passed: false })],
});
const di = deriveDistillInput(fb);
expect(di.irreversible).toBe(true);
});
});
describe("deriveSkillExtractInput —— 技能抽取入参", () => {
it("成功 → rubricScore=0.8;trigger.kind=event", () => {
const fb = makeFeedback({
prompt: "i2c probe 0x3a",
actionClassHint: "i2c-probe",
outcome: "success",
toolOutcomes: [makeOutcome({ toolName: "i2cdetect", passed: true })],
});
const si: SkillExtractInput = deriveSkillExtractInput(fb);
expect(si.trigger.kind).toBe("event");
expect(si.trigger.actionClass).toBe("i2c-probe");
expect(si.trigger.pattern).toBeUndefined(); // 成功无 failure pattern
expect(si.rubricScore).toBe(0.8);
expect(si.granularity).toBe("task-level");
expect(si.procedure.length).toBe(1);
expect(si.procedure[0]?.tool).toBe("i2cdetect");
expect(si.execResults).toEqual([{ passed: true }]);
expect(si.provenance).toEqual([`turn@${fb.turnTs}`]);
});
it("失败 → rubricScore=0.3;trigger.pattern=`${actionClass}-failure`", () => {
const fb = makeFeedback({
prompt: "i2c probe",
actionClassHint: "i2c-probe",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "i2cdetect", passed: false })],
});
const si = deriveSkillExtractInput(fb);
expect(si.rubricScore).toBe(0.3);
expect(si.trigger.pattern).toBe("i2c-probe-failure");
});
it("无 hint → actionClass 从 prompt+toolNames 推导", () => {
const fb = makeFeedback({
prompt: "flash image",
actionClassHint: "",
outcome: "success",
toolOutcomes: [makeOutcome({ toolName: "dd", passed: true })],
});
const si = deriveSkillExtractInput(fb);
expect(si.trigger.actionClass).toBe("flash");
});
});
describe("deriveSkillQuery —— 技能检索查询", () => {
it("有 prompt → { trigger, sigma }", () => {
const fb = makeFeedback({ prompt: "flash bootloader", actionClassHint: "flash" });
const q = deriveSkillQuery(fb);
expect(q).not.toBeNull();
const qr = q as { trigger: SkillTrigger; sigma: TaskSignature };
expect(qr.trigger.kind).toBe("event");
expect(qr.trigger.actionClass).toBe("flash");
expect(qr.sigma.actionClass).toBe("flash");
});
it("无 prompt → null", () => {
const fb = makeFeedback({ prompt: "" });
// deriveSkillQuery 判 !feedback.prompt → 空串 falsy → null
expect(deriveSkillQuery(fb)).toBeNull();
});
});
describe("derivePriorCorrection —— 难度先验校正", () => {
it("步数≤3 → estimated low", () => {
const fb = makeFeedback({ steps: 2, failures: 0, actionClassHint: "i2c-probe" });
const c = derivePriorCorrection(fb);
expect(c.taskClass).toBe("i2c-probe");
expect(c.estimated).toBe("low");
expect(c.actualSteps).toBe(2);
expect(c.failures).toBe(0);
expect(c.boardFingerprint).toEqual(beta1);
});
it("步数 4-6 → estimated mid", () => {
const fb = makeFeedback({ steps: 5, failures: 1 });
expect(derivePriorCorrection(fb).estimated).toBe("mid");
});
it("步数>6 → estimated high", () => {
const fb = makeFeedback({ steps: 8, failures: 2 });
expect(derivePriorCorrection(fb).estimated).toBe("high");
});
});
describe("deriveProposals —— 变体提案", () => {
it("成功 → 空数组", () => {
const fb = makeFeedback({ outcome: "success" });
expect(deriveProposals(fb)).toEqual([]);
});
it("失败 + 可推断扩展 → 1 提案(非命门)", () => {
const fb = makeFeedback({
prompt: "i2c probe",
actionClassHint: "i2c-probe",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "i2cdetect", passed: false })],
turnTs: 7777,
});
const ps = deriveProposals(fb);
expect(ps.length).toBe(1);
expect(ps[0]?.targetExtension).toBe("peripheral-probe");
expect(ps[0]?.name).toBe("fix-peripheral-probe-7777");
expect(ps[0]?.baseRef).toBe("turn@7777");
expect(ps[0]?.createdAt).toBe(7777);
});
it("失败 + 无可推断扩展 → 空数组", () => {
const fb = makeFeedback({
prompt: "noop",
actionClassHint: "unknown",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "echo", passed: false })],
});
expect(deriveProposals(fb)).toEqual([]);
});
});
// ============================================================================
// §4.6 链路完整性:deriveDistillInput → distillEntry(反馈→蒸馏→不对称约束)
// ============================================================================
describe("§4.6 链路 sanity —— deriveDistillInput → distillEntry", () => {
it("success + physical-ground-truth → repeatable / actionable", () => {
const fb = makeFeedback({
prompt: "i2c probe 0x3a",
actionClassHint: "i2c-probe",
outcome: "success",
toolOutcomes: [makeOutcome({ toolName: "i2cget", passed: true })],
});
const di: DistillInput = deriveDistillInput(fb);
const e = distillEntry(di);
expect(e.validity).toBe("repeatable");
expect(e.usage).toBe("actionable");
expect(e.outcome).toBe("success");
});
it("success + model-claim → inferred / hint(不可 actionable,§4.6)", () => {
const fb = makeFeedback({
prompt: "echo status",
actionClassHint: "unknown",
outcome: "success",
toolOutcomes: [makeOutcome({ toolName: "echo", passed: true, exitCode: undefined })],
});
const di: DistillInput = deriveDistillInput(fb);
const e = distillEntry(di);
expect(e.validity).toBe("inferred");
expect(e.usage).toBe("hint");
expect(e.usage).not.toBe("actionable"); // §4.6 inferred 禁 actionable
});
it("failure + irreversible → irreversible-fail / anti-pattern", () => {
const fb = makeFeedback({
prompt: "flash image",
actionClassHint: "flash",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "dd if=image of=/dev/mmcblk0", passed: false })],
});
const di: DistillInput = deriveDistillInput(fb);
const e = distillEntry(di);
expect(e.validity).toBe("irreversible-fail");
expect(e.usage).toBe("anti-pattern");
expect(di.irreversible).toBe(true);
});
it("failure + reversible → once / anti-pattern", () => {
const fb = makeFeedback({
prompt: "i2c probe 0x3a",
actionClassHint: "i2c-probe",
outcome: "failure",
toolOutcomes: [makeOutcome({ toolName: "i2cdetect", passed: false })],
});
const di: DistillInput = deriveDistillInput(fb);
const e = distillEntry(di);
expect(e.validity).toBe("once");
expect(e.usage).toBe("anti-pattern");
expect(di.irreversible).toBe(false);
});
});

View File

@ -2,7 +2,7 @@
> 面向 pba(硬件闭环编程 agent)的自进化架构、原理与实施路线
> 定位:把 pba 从"静态装配的扩展集"演进为"从硬件交互中持续改进自身"的闭环系统
> 状态:M1-M5(L1-L5 全栈自进化)已实现并量化坐实(§7.8/§8.7/§9.6/§10.6/§11.7);与 `架构设计文档_v3.md` 并列,作为下一阶段能力蓝本
> 状态:M1-M5 + X.1/X.3/X.4 全部已实现并量化坐实(§7.8/§8.7/§9.6/§10.6/§11.7/§19.4);与 `架构设计文档_v3.md` 并列,作为下一阶段能力蓝本
---
@ -778,9 +778,9 @@ M1 完成即证明 §2 五条因果链在 pba 跑通最小闭环——从"静态
| **M3** | L3 | 工具生长(运行期 registerTool+sandbox 升级门+只长读/探) | ✅ | 100%(机制闭环)/ M3.\* live-LLM 门待 | 步效率 0.24<0.70 ✅;ΔJ=0.1367≥0.05 ✅;砖板率=0 EXACT ✅;forbidden 7拒/0受 ✅(§9.6) |
| **M4** | L4 | 编排进化(难度估+workflow 图 G+符号反向传播搜索) | ✅ | 100%(机制闭环)/ M4.\* live-LLM 门待 | ΔJ_G=0.2975≥0.10(θ 固定)✅;held-out 0.3182≥0.03 ✅;难度 corr 0.6121>0 ✅;**risk 不增 EXACT**(1≤1)✅;CI STRONG(§10.6) |
| **M5** | L5 | 脚手架自改(变体 worktree+§11.4 不变式门+A/B 回滚) | ✅ | 100%(机制闭环)/ M5.\* live-LLM 门待 | ΔJ_selfmod=0.15≥0.05 ✅;**forbidden 5拒/0受 EXACT**(§11.3)✅;**§11.4 门捕有害 2**✅;**回滚触发 1**(§11.6)✅;CI STRONG(§11.7) |
| 跨层 | — | 时序接线 / 评估框架 / 安全护栏(不进化) | ⏳ / ➖ | — | 见 §19.4 / §19.5 |
| 跨层 | — | 时序接线(X.1)/ v_t 全链路(X.3)/ 可观测面板(X.4)/ 评估框架(X.2)/ 安全护栏 | ✅ / ➖ | — | X.1+X.3+X.4 全完成(§19.4 实测);安全护栏不进化 |
> 当前总体状态:**设计完成 ✅,M1-M5 五层机制闭环全部完成 ✅(量化坐实,§7.8/§8.7/§9.6/§10.6/§11.7)**。pba 已从"静态装配 agent"→"L1-L5 全栈自进化 agent":L1 经验库(+34pp)、L2 技能表(+26.5pp,infoGain 防 plateau 实证)、L3 工具生长(步数砍 76%,砖板率=0 构造铁律)、L4 编排进化(+29.75pp,θ 固定归因 G,risk 不增构造铁律)、L5 脚手架自改(+15pp,forbidden 命门永不自改 + §11.4 门捕有害 + 回滚实证)。五层均经真实 manager + 对照实现全等验证。M1-M5.\*(live-LLM/板端到端 + derive\* 真接线 + sqlite 持久化 + vm 沙箱 execute + 真 CI 不变式执行)为跟进项。
> 当前总体状态:**设计完成 ✅,M1-M5 五层 + X.1/X.3/X.4 跨层全部完成 ✅(量化坐实,§7.8/§8.7/§9.6/§10.6/§11.7/§19.4)**。pba 已从"静态装配 agent"→"L1-L5 全栈自进化 + 跨层闭环 agent":L1 经验库(+34pp)、L2 技能表(+26.5pp,infoGain 防 plateau 实证)、L3 工具生长(步数砍 76%,砖板率=0 构造铁律)、L4 编排进化(+29.75pp,θ 固定归因 G,risk 不增构造铁律)、L5 脚手架自改(+15pp,forbidden 命门永不自改 + §11.4 门捕有害 + 回滚实证)、**X.1+X.3 全链路(tool_result→derive\*→5 层→注入→约束,§4.6 100% + 4 bank 生产自驱)、X.4 可观测面板(精确指标,brickRate=0 构造铁律)**。M1-M5.\*(live-LLM/板端到端 + stage-wise 真 θ + 真 git worktree + 真 CI §11.4 执行)为跟进项。
### 19.2 M1 — 硬件经验库(L1)工作分解 —— 已完成(2026-08-17)
@ -826,14 +826,31 @@ M1 完成即证明 §2 五条因果链在 pba 跑通最小闭环——从"静态
| | M5.4 | 自改白名单(禁改 safety-gate/sandbox/flash/glm-provider 命门) | `classifyScaffold`/`canSelfModify`/`classifyProposalVariant`(forbidden 命门→propose throw §11.3) | §11.3 | ✅ |
| | M5.5 | 评估:有变体被提升 + 不变式全过 + 回滚被触发过 | `tests/eval/variant-eval.test.ts`(ΔJ=0.15 + forbidden 5拒/0受 EXACT + §11.4 门捕有害 2 + 回滚触发 1 + 真 manager 对齐) | §11.6 | ✅ |
### 19.4 跨层共享项
### 19.4 跨层共享项 —— 全部完成(2026-08-17)
| 编号 | 工作项 | 落点 | 状态 |
|---|---|---|---|
| X.1 | 时序三档接线(task-time/post-task/stage-wise 挂已有钩子) | `before_agent_start`/`session_before_compact`/`tool_call`/`message_end` | ⏳ |
| X.1 | 时序三档接线(task-time/post-task/stage-wise 挂已有钩子) | `before_agent_start`/`session_before_compact`(5 bank 全挂)+ **`tool_result`/`message_end`(feedback-bank 新挂)**;stage-wise θ 更新=M5.\* | ✅(4/5 钩子全接;stage-wise=L5.\* 真权重更新待) |
| X.2 | 评估框架(held-out 板+任务集 + ΔJ 计算 + F1–F6 失败模式检测) | `tests/eval/{experience-bank,skill-registry,tool-growth,orchestration,variant}-eval.test.ts`(M1-M5 全落,seeded 受控队列 + Wilson CI + 真 manager 对齐) | ✅(M1-M5 落地,复用扩展完毕) |
| X.3 | 可信度分级 v_t 全链路贯通(反馈采集→蒸馏→入库→检索→注入→约束) | 跨 L1/L2/L3/L5 | ⏳ |
| X.4 | 进化可观测面板(命中率/infoGain/技能表规模/risk/砖板率 指标) | 新 `scripts/evolution-metrics` | ⏳ |
| X.3 | 可信度分级 v_t 全链路贯通(反馈采集→蒸馏→入库→检索→注入→约束) | `core/feedback-collector.ts`(FeedbackCollector+derive* 纯变换)+ `extensions/feedback-bank.ts`(tool_result→采集→message_end→finalize→session_before_compact 喂 derive*)+ **4 bank 默认 derive* 读共享 collector lastFinalized 生产自驱**(experience/skill/orchestration/variant)+ `tests/eval/feedback-chain-eval.test.ts`(v_t 全链路 end-to-end 实证) | ✅(全链路贯通,§4.6 分级 8/8=100% + 5 链 end-to-end + 生产自驱) |
| X.4 | 进化可观测面板(命中率/infoGain/技能表规模/risk/砖板率 指标) | `core/evolution-metrics.ts`(aggregateMetrics/renderMetrics 纯函数)+ `bin/evolution-metrics.ts` CLI + `tests/eval/evolution-metrics-eval.test.ts`(精确指标,brickRate=0 EXACT) | ✅(面板落地,精确指标全过) |
**X.1+X.3+X.4 实测(2026-08-17,独立复跑一致)**:
| 评估 | 指标 | 实测 | 阈值 | 判定 |
|---|---|---|---|---|
| X.1+X.3 全链路(`feedback-chain-eval`) | §4.6 分级正确率(8 转向原型) | **8/8 (100%)** | 100% | ✅ PASS |
| X.1+X.3 | 同签名检索命中刚蒸馏条目 | 100% | 100% | ✅ PASS |
| X.1+X.3 | 注入块 `<experience_pack>` 可见 | 100% | 100% | ✅ PASS |
| X.1+X.3 | 5 链 end-to-end(skill/workflow/variant/tool-growth) | 全过 | 全过 | ✅ PASS |
| X.1+X.3 | forbidden §11.3/§4.8 拒绝(命门/flash 工具) | 拒绝 | 拒绝 | ✅ PASS |
| X.4(`evolution-metrics-eval`) | experienceHitRate 精确 | 0.600 | 精确 | ✅ PASS |
| X.4 | avgInfoGain 精确 | 0.650 | 精确 | ✅ PASS |
| X.4 | maxRisk 精确 | 1 | 精确 | ✅ PASS |
| X.4 | **brickRate(构造恒 0)** | 0 | ===0 EXACT | ✅ PASS |
| X.4 | forbiddenRejected/counts 精确 | 7/全精确 | 精确 | ✅ PASS |
**结论**:X.1(4/5 时序钩子全接,stage-wise=L5.\*)+ X.3(v_t 全链路贯通:tool_result 采集→derive\* 蒸馏→5 层入库→检索→注入→约束,§4.6 分级 100% + 5 链 end-to-end + 4 bank 生产自驱读共享 collector)+ X.4(可观测面板精确指标,brickRate=0 构造铁律)全部完成。诚实边界:live-LLM 读注入包真受益的端到端门(glm-5.2 经 pi 真调)+ stage-wise 真 θ 更新 + 真 git worktree 变体 + 真 CI §11.4 执行 = M1-M5.\* 工程项(机制层已全闭环 + 量化坐实)。
### 19.5 不做项(约束/护栏,不参与进化,作不变式)
@ -857,7 +874,8 @@ M1 完成即证明 §2 五条因果链在 pba 跑通最小闭环——从"静态
- **M5 实现**:✅ 完成(L5 脚手架自改闭环,终局最强约束)——VariantManager(变体+§11.4 不变式门+A/B+回滚+白名单)+ variant-bank 扩展。§11.7 量化坐实:ΔJ_selfmod=0.15、**forbidden 命门永不自改 EXACT**(5 拒/0 受,§11.3)、**§11.4 门捕有害变体 2**、**回滚触发 1**(§11.6 可逆性实证)、全 promoted allPassed、CI STRONG;真 manager 与对照对齐。
- **M1-M5.\* 跟进(非阻塞)**:live-LLM/板端到端门(glm-5.2 读 experiencePack/skillPack/grown_tools/orchestration_plan/active_variants 真受益)+ derive\* 真接线(接 env-probe + LLM 摘要钩子)+ sqlite 跨 session 持久化 + vm 沙箱 execute 变体工厂 + 真 CI §11.4 不变式执行(真 tsc/单测/launch/砖板率)。
- **前置底座**:✅ 齐全——事件溯源、注入通道、能力 Registry、fail-closed 安全门、时序钩子均就位。
- **集成**:全量 57 文件 2875 passed / 5 skipped / tsc 0 / sk- 0 / 真 pi T01 boot 过(barrel 27 扩展含 L1-L5 自进化层装配不破坏启动)。
- **集成**:全量 62 文件 3018 passed / 5 skipped / tsc 0 / sk- 0 / 真 pi T01 boot 过(barrel 28 扩展含 L1-L5 自进化层 + 跨层反馈采集 feedback-bank 装配不破坏启动)。
- **跨层(X.1+X.3+X.4)实现**:✅ 完成——FeedbackCollector+derive\*(`core/feedback-collector.ts`)+ feedback-bank(hooks tool_result/message_end→采集→finalize→喂 derive\*)+ **4 bank 默认 derive\* 读共享 collector lastFinalized 生产自驱**(experience/skill/orchestration/variant)+ evolution-metrics 面板(`core/evolution-metrics.ts`+`bin/evolution-metrics.ts` CLI)。§19.4 量化坐实:§4.6 分级 8/8=100% + 5 链 end-to-end + 精确指标(brickRate=0 EXACT)。
- **下一步**:M1-M5.\* live-LLM 门坐实真模型受益(glm-5.2 真读注入包+真受益);或推进 M5.\* 真 git worktree + 真 CI 不变式执行把脚手架自改从"数据模型"升级到"真代码注入"。
### 19.7 更新日志
@ -884,5 +902,10 @@ M1 完成即证明 §2 五条因果链在 pba 跑通最小闭环——从"静态
| 2026-08-17 | M5.1-M5.3 VariantManager 实现(InMemoryVariantRegistry + VariantManagerImpl:propose 白名单 forbidden 命门 throw §11.3/runInvariants 注入 checker/abTest 注入 runner/tryPromote 不变式全过+A/B 胜出→promoted,不变式失败→rolled-back/rollback 可逆/resolve 只返 promoted),42 单测过 | M5.1-5.3 ✅ |
| 2026-08-17 | M5.4-M5.5 variant-bank 扩展(variant_propose/variant_list/variant_promote/variant_rollback 工具 + 注入 `<active_variants>`)+ 量化评估落地。实测:ΔJ_selfmod=0.15≥0.05、**forbidden 命门 5 拒/0 受 EXACT**(§11.3)、**§11.4 门捕有害变体 2**、**回滚触发 1**(§11.6 可逆性实证)、promoted 有益 3、全 promoted allPassed、CI STRONG、真 manager 与对照对齐。32+17 测过;barrel 第 27 工厂登记 | M5.4-5.5 ✅ |
| 2026-08-17 | M4+M5 集成验证:全量 57 文件 2875 passed / 5 skipped / tsc 0 / sk- 0 / 真 pi T01 boot 过(barrel 27 扩展含 L1-L5 全栈自进化层)。**M1-M5 五层机制闭环全部 100% 完成**,pba 从"静态装配 agent"→"L1-L5 全栈自进化 agent"。M1-M5.\* live-LLM 门 + 真 git worktree + 真 CI 不变式执行待跟进 | M4+M5 ✅ |
| 2026-08-17 | X 类型+契约(`shared/types.ts` TaskFeedback/ToolOutcome/EvolutionMetrics/MetricsSources + `core/feedback-collector.ts` 稳定 interface+derive\* 纯变换 + `core/evolution-metrics.ts` aggregateMetrics/renderMetrics 纯函数) | X.1/X.3/X.4 ✅ |
| 2026-08-17 | X.1+X.3 FeedbackCollector+derive\* 实现(FeedbackCollectorImpl recordPrompt/recordBoardFingerprint/recordToolResult/finalize/lastFinalized/pending/clear + deriveTaskSignature/deriveDistillInput/deriveSkillExtractInput/deriveSkillQuery/derivePriorCorrection/deriveProposals + inferOutcome/inferActionClass/inferValidatedBy/isIrreversibleTool),58 单测过;加 lastFinalized + getSharedCollector/initSharedCollector 共享单例 | X.1+X.3 core ✅ |
| 2026-08-17 | X.1+X.3 feedback-bank 扩展(hooks tool_result+message_end+before_agent_start→采集→finalize→session_before_compact 喂 derive\* + feedback_recall 工具)+ 全链路集成评估。实测:**§4.6 分级 8/8=100%**(success+ground-truth/可逆/不可逆/partial/empty/mixed 全过)、同签名检索命中 100%、注入块可见 100%、5 链 end-to-end(skill/workflow/variant/tool-growth)、forbidden §11.3/§4.8 拒绝。26+18 测过;barrel 第 28 工厂登记。**4 bank 默认 derive\* 读共享 collector 生产自驱**(experience/skill/orchestration/variant) | X.1+X.3 ✅ |
| 2026-08-17 | X.4 evolution-metrics 可观测面板(aggregateMetrics/renderMetrics 纯函数 + `bin/evolution-metrics.ts` CLI)+ 量化评估。实测:精确指标全过(experienceHitRate=0.600/avgInfoGain=0.650/maxRisk=1/brickRate=0 EXACT 构造铁律/forbiddenRejected=7)。20+21 测过;CLI 可跑 | X.4 ✅ |
| 2026-08-17 | X 集成验证:全量 62 文件 3018 passed / 5 skipped / tsc 0 / sk- 0 / 真 pi T01 boot 过(barrel 28 扩展含 L1-L5 + 跨层反馈采集)。**X.1+X.3+X.4 全部完成**,跨层闭环(时序接线+v_t 全链路+可观测面板)合龙,M1-M5+X 机制层全栈自进化闭环 100%。M1-M5.\* live-LLM 门 + stage-wise 真 θ + 真 git worktree + 真 CI §11.4 执行待跟进 | X ✅ |
> 更新规则:每完成一个工作项,在 §19.2/§19.3 对应行改状态(⏳→🔄→✅)+ 在 §19.7 加一行日志(日期+进展+工作项编号)。里程碑完成时更新 §19.1 总览完成度。遇阻塞改 ⚠️ 并在备注写阻塞根因。