* feat(eval): add state manager bridging ActionEngine for eval Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add shared types for whiteboard layout eval harness * feat(eval): add SSE chat client for whiteboard eval * feat(eval): add Playwright capture module for whiteboard screenshots * feat(eval): add VLM scorer for whiteboard layout evaluation * feat(eval): add report generator for whiteboard eval results * feat(eval): add 8 constructed scenarios for whiteboard layout eval * feat(eval): add minimal whiteboard render page for Playwright screenshots Creates app/eval/whiteboard/page.tsx — a headless client page that seeds the stageStore with a synthetic slide scene, exposes window.__setElements() for Playwright to inject PPTElement[], and renders them via ScreenElement inside a 1000×562.5px white canvas. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add main runner for whiteboard layout eval Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add eval:whiteboard script, install tsx, gitignore results * fix(eval): fix TS errors, lint, and prettier formatting * fix(eval): address code review — add cue_user/empty turn guards, validate VLM output, fix empty array crash * refactor(eval): replace synthetic scenarios with realistic ones - Replace 8 generic scenarios with 6 that match real usage patterns - Multi-agent discussion with short user replies (嗯, 明白了, 继续) - Include real slide scene data as initialStoreState - Generated agent configs with Chinese names and proper roles - Cover: physics, math, finance, primary school, economics, medical Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: extract shared agent loop from use-chat-sessions Extract the core agent loop logic into lib/chat/agent-loop.ts as a pure async function with callback injection. Both the frontend React hook and the eval harness now share the same loop — SSE parsing, exit conditions (END/cue_user/empty turns/max turns), and director state accumulation. The frontend wires StreamBuffer callbacks for UI pacing; the eval wires ActionEngine + message accumulation for headless execution. If loop logic changes in the shared module, both consumers automatically stay in sync. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(eval): use project LLM infrastructure, fix eval page and model config - Rewrite scorer to use resolveModel() + generateText() from AI SDK instead of raw fetch — supports all providers (OpenAI, Google, Anthropic) - Model config via env vars (EVAL_CHAT_MODEL, EVAL_SCORER_MODEL), matching the pattern from outline-language eval - Fix eval page: bootstrap store before SceneProvider mounts - Fix __dirname for tsx CJS mode - Remove --api-key/--scorer-model CLI args (use env vars instead) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): organize results by model/timestamp * fix: remove double turnCount increment in shared agent loop The extracted agent-loop.ts had both `turnCount = directorState?.turnCount ?? turnCount + 1` (line 190) and a redundant `turnCount++` (line 215), causing multi-agent scenarios to hit maxTurns at half the expected number of iterations. Also removes unused processSSEStream import from use-chat-sessions.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): organize screenshots by scenario subdirectory Results structure: results/<model>/<timestamp>/<scenario>/run0_turn1.png Report files stay at the timestamp level. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): revise scorer rubric and add rescore mode - Replace space_utilization with rendering_correctness and content_completeness - Rubric now evaluates from a teacher's perspective (empty space is normal) - readability emphasizes font size consistency - Add --rescore flag to re-score existing screenshots without re-running chat - Increase maxOutputTokens to 2000, add JSON parse error recovery - Score errors no longer abort the entire scenario Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): sharpen scorer rubric with teacher-perspective examples The rubric now catches specific classroom whiteboard failure modes: - overlap now explicitly penalizes writing over existing content when empty space is available (spatial planning failure) - rendering_correctness calls out diagram accuracy (e.g., parabola drawn as V-shape), raw subscripts (G_x), Chinese inside LaTeX math mode - content_completeness emphasizes canvas edge clipping and bare unlabeled diagrams - readability penalizes text styled as UI components (gray card backgrounds) - overall instructed to weight overlap and rendering_correctness more heavily - explicit note to ignore the "N" page UI element Also increases maxOutputTokens to 3000 since longer rubric produces longer justifications. Reporter now guards against null scores (scorer failures no longer crash report generation). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review findings Critical fixes: - use-chat-sessions: restore agent_end handling, currentMessageId fallback for text_delta/action events with missing messageId, and re-throw on SSE error events (previously silently pushed to buffer only). - eval runner: serialize ActionEngine executions via promise chain. void-fire-and-forget raced with ensureWhiteboardOpen's 2s delay and could insert elements out of order or before the first element was committed to the store. Important fixes: - CHAT_MODEL default: 'openai/gpt-4o-mini' -> 'openai:gpt-4o-mini' (parseModelString splits on ':', not '/'). - CheckpointResult.score is now VlmScore | null; removed the 'as unknown as' cast that hid the null contract from consumers. - Delete dead code: eval/whiteboard-layout/chat-client.ts and components/chat/process-sse-stream.ts (both unused after the shared agent loop refactor). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
222 lines
6.7 KiB
TypeScript
222 lines
6.7 KiB
TypeScript
/**
|
|
* Agent Loop — Shared core logic for the frontend-driven multi-agent loop.
|
|
*
|
|
* Extracted from use-chat-sessions.ts so both the frontend hook and the
|
|
* eval harness share the same loop logic. No React dependency — pure
|
|
* async function with callback injection for environment-specific behavior.
|
|
*
|
|
* The loop runs per-user-message: the director dispatches agents one at a
|
|
* time, each agent generates a response, and the loop continues until the
|
|
* director says END, cues the user, or maxTurns is reached.
|
|
*/
|
|
|
|
import type { StatelessEvent, DirectorState } from '@/lib/types/chat';
|
|
import { createLogger } from '@/lib/logger';
|
|
|
|
const log = createLogger('AgentLoop');
|
|
|
|
// ==================== Types ====================
|
|
|
|
/** Store state snapshot sent with each /api/chat request */
|
|
export interface AgentLoopStoreState {
|
|
stage: unknown;
|
|
scenes: unknown[];
|
|
currentSceneId: string | null;
|
|
mode: string;
|
|
whiteboardOpen: boolean;
|
|
}
|
|
|
|
/** Request template — fields that stay constant across loop iterations */
|
|
export interface AgentLoopRequest {
|
|
config: {
|
|
agentIds: string[];
|
|
sessionType?: string;
|
|
agentConfigs?: Record<string, unknown>[];
|
|
[key: string]: unknown;
|
|
};
|
|
userProfile?: { nickname?: string; bio?: string };
|
|
apiKey: string;
|
|
baseUrl?: string;
|
|
model?: string;
|
|
providerType?: string;
|
|
}
|
|
|
|
/** Per-iteration outcome extracted from the done event */
|
|
export interface AgentLoopIterationResult {
|
|
directorState?: DirectorState;
|
|
totalAgents: number;
|
|
agentHadContent: boolean;
|
|
cueUserReceived: boolean;
|
|
}
|
|
|
|
/** Callbacks injected by the caller (frontend or eval) */
|
|
export interface AgentLoopCallbacks {
|
|
/** Get fresh store state for each iteration (whiteboard may have changed) */
|
|
getStoreState: () => AgentLoopStoreState;
|
|
|
|
/** Get current messages for the request */
|
|
getMessages: () => unknown[];
|
|
|
|
/**
|
|
* Make the HTTP request to /api/chat.
|
|
* Returns a Response object (or equivalent with .body ReadableStream).
|
|
*/
|
|
fetchChat: (body: Record<string, unknown>, signal: AbortSignal) => Promise<Response>;
|
|
|
|
/**
|
|
* Process a single SSE event. Called for every event in the stream.
|
|
* The callback should handle action execution, text accumulation,
|
|
* message construction, and UI updates.
|
|
*/
|
|
onEvent: (event: StatelessEvent) => void;
|
|
|
|
/**
|
|
* Called after all SSE events for one iteration have been processed
|
|
* and the stream is closed.
|
|
*
|
|
* Must return the iteration result (extracted from the 'done' event).
|
|
* The frontend waits for buffer drain here before reading the result
|
|
* from loopDoneDataRef. The eval harness returns a result it
|
|
* accumulated during onEvent calls.
|
|
*/
|
|
onIterationEnd: () => Promise<AgentLoopIterationResult | null>;
|
|
}
|
|
|
|
/** Final outcome of the agent loop */
|
|
export interface AgentLoopOutcome {
|
|
/** Why the loop stopped */
|
|
reason: 'end' | 'cue_user' | 'max_turns' | 'aborted' | 'empty_turns' | 'no_done';
|
|
/** Accumulated director state */
|
|
directorState?: DirectorState;
|
|
/** Number of iterations completed */
|
|
turnCount: number;
|
|
}
|
|
|
|
// ==================== Core Loop ====================
|
|
|
|
/**
|
|
* Run the agent loop — shared between frontend and eval.
|
|
*
|
|
* Each iteration: refresh state → POST /api/chat → process SSE events
|
|
* → check exit conditions → repeat.
|
|
*/
|
|
export async function runAgentLoop(
|
|
request: AgentLoopRequest,
|
|
callbacks: AgentLoopCallbacks,
|
|
signal: AbortSignal,
|
|
maxTurns: number,
|
|
): Promise<AgentLoopOutcome> {
|
|
let directorState: DirectorState | undefined = undefined;
|
|
let turnCount = 0;
|
|
let consecutiveEmptyTurns = 0;
|
|
|
|
while (turnCount < maxTurns) {
|
|
if (signal.aborted) {
|
|
return { reason: 'aborted', directorState, turnCount };
|
|
}
|
|
|
|
// Refresh store state each iteration — agent actions may have changed
|
|
// whiteboard, scene, or mode between turns
|
|
const freshStoreState = callbacks.getStoreState();
|
|
const currentMessages = callbacks.getMessages();
|
|
|
|
// Build request body
|
|
const body: Record<string, unknown> = {
|
|
messages: currentMessages,
|
|
storeState: freshStoreState,
|
|
config: request.config,
|
|
directorState,
|
|
userProfile: request.userProfile,
|
|
apiKey: request.apiKey,
|
|
baseUrl: request.baseUrl,
|
|
model: request.model,
|
|
providerType: request.providerType,
|
|
};
|
|
|
|
// Fetch
|
|
const response = await callbacks.fetchChat(body, signal);
|
|
|
|
if (!response.ok) {
|
|
const errorText = await response.text();
|
|
throw new Error(`API error: ${response.status} - ${errorText}`);
|
|
}
|
|
|
|
// Parse SSE stream and process events
|
|
const reader = response.body?.getReader();
|
|
if (!reader) throw new Error('No response body');
|
|
|
|
const decoder = new TextDecoder();
|
|
let sseBuffer = '';
|
|
|
|
try {
|
|
while (true) {
|
|
const { done, value } = await reader.read();
|
|
if (done) break;
|
|
|
|
sseBuffer += decoder.decode(value, { stream: true });
|
|
const parts = sseBuffer.split('\n\n');
|
|
sseBuffer = parts.pop() || '';
|
|
|
|
for (const part of parts) {
|
|
const line = part.trim();
|
|
if (!line.startsWith('data: ')) continue;
|
|
|
|
try {
|
|
const event: StatelessEvent = JSON.parse(line.slice(6));
|
|
callbacks.onEvent(event);
|
|
} catch {
|
|
// Skip malformed events (heartbeats, etc.)
|
|
}
|
|
}
|
|
}
|
|
} finally {
|
|
reader.releaseLock();
|
|
}
|
|
|
|
if (signal.aborted) {
|
|
return { reason: 'aborted', directorState, turnCount };
|
|
}
|
|
|
|
// Post-iteration: wait for buffer drain (frontend) or collect results (eval)
|
|
const iterationResult = await callbacks.onIterationEnd();
|
|
|
|
// Check exit conditions
|
|
if (!iterationResult) {
|
|
return { reason: 'no_done', directorState, turnCount };
|
|
}
|
|
|
|
// Update accumulated director state
|
|
directorState = iterationResult.directorState;
|
|
turnCount = directorState?.turnCount ?? turnCount + 1;
|
|
|
|
// Director said USER — stop loop
|
|
if (iterationResult.cueUserReceived) {
|
|
return { reason: 'cue_user', directorState, turnCount };
|
|
}
|
|
|
|
// Director said END — no agent spoke
|
|
if (iterationResult.totalAgents === 0) {
|
|
return { reason: 'end', directorState, turnCount };
|
|
}
|
|
|
|
// Track consecutive empty responses
|
|
if (!iterationResult.agentHadContent) {
|
|
consecutiveEmptyTurns++;
|
|
if (consecutiveEmptyTurns >= 2) {
|
|
log.warn(
|
|
`[AgentLoop] ${consecutiveEmptyTurns} consecutive empty agent responses, stopping loop`,
|
|
);
|
|
return { reason: 'empty_turns', directorState, turnCount };
|
|
}
|
|
} else {
|
|
consecutiveEmptyTurns = 0;
|
|
}
|
|
}
|
|
|
|
// maxTurns reached
|
|
if (turnCount >= maxTurns) {
|
|
log.info(`[AgentLoop] Max turns (${maxTurns}) reached`);
|
|
}
|
|
return { reason: 'max_turns', directorState, turnCount };
|
|
}
|