openmaic-custom/lib/chat/agent-loop.ts
wyuc ea0e8126a5
feat: whiteboard layout quality eval harness (#425)
* feat(eval): add state manager bridging ActionEngine for eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add shared types for whiteboard layout eval harness

* feat(eval): add SSE chat client for whiteboard eval

* feat(eval): add Playwright capture module for whiteboard screenshots

* feat(eval): add VLM scorer for whiteboard layout evaluation

* feat(eval): add report generator for whiteboard eval results

* feat(eval): add 8 constructed scenarios for whiteboard layout eval

* feat(eval): add minimal whiteboard render page for Playwright screenshots

Creates app/eval/whiteboard/page.tsx — a headless client page that
seeds the stageStore with a synthetic slide scene, exposes
window.__setElements() for Playwright to inject PPTElement[], and
renders them via ScreenElement inside a 1000×562.5px white canvas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add main runner for whiteboard layout eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add eval:whiteboard script, install tsx, gitignore results

* fix(eval): fix TS errors, lint, and prettier formatting

* fix(eval): address code review — add cue_user/empty turn guards, validate VLM output, fix empty array crash

* refactor(eval): replace synthetic scenarios with realistic ones

- Replace 8 generic scenarios with 6 that match real usage patterns
- Multi-agent discussion with short user replies (嗯, 明白了, 继续)
- Include real slide scene data as initialStoreState
- Generated agent configs with Chinese names and proper roles
- Cover: physics, math, finance, primary school, economics, medical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract shared agent loop from use-chat-sessions

Extract the core agent loop logic into lib/chat/agent-loop.ts as a pure
async function with callback injection. Both the frontend React hook and
the eval harness now share the same loop — SSE parsing, exit conditions
(END/cue_user/empty turns/max turns), and director state accumulation.

The frontend wires StreamBuffer callbacks for UI pacing; the eval wires
ActionEngine + message accumulation for headless execution. If loop logic
changes in the shared module, both consumers automatically stay in sync.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(eval): use project LLM infrastructure, fix eval page and model config

- Rewrite scorer to use resolveModel() + generateText() from AI SDK
  instead of raw fetch — supports all providers (OpenAI, Google, Anthropic)
- Model config via env vars (EVAL_CHAT_MODEL, EVAL_SCORER_MODEL),
  matching the pattern from outline-language eval
- Fix eval page: bootstrap store before SceneProvider mounts
- Fix __dirname for tsx CJS mode
- Remove --api-key/--scorer-model CLI args (use env vars instead)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): organize results by model/timestamp

* fix: remove double turnCount increment in shared agent loop

The extracted agent-loop.ts had both `turnCount = directorState?.turnCount ?? turnCount + 1`
(line 190) and a redundant `turnCount++` (line 215), causing multi-agent scenarios to hit
maxTurns at half the expected number of iterations.

Also removes unused processSSEStream import from use-chat-sessions.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): organize screenshots by scenario subdirectory

Results structure: results/<model>/<timestamp>/<scenario>/run0_turn1.png
Report files stay at the timestamp level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): revise scorer rubric and add rescore mode

- Replace space_utilization with rendering_correctness and content_completeness
- Rubric now evaluates from a teacher's perspective (empty space is normal)
- readability emphasizes font size consistency
- Add --rescore flag to re-score existing screenshots without re-running chat
- Increase maxOutputTokens to 2000, add JSON parse error recovery
- Score errors no longer abort the entire scenario

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): sharpen scorer rubric with teacher-perspective examples

The rubric now catches specific classroom whiteboard failure modes:
- overlap now explicitly penalizes writing over existing content when empty space is available (spatial planning failure)
- rendering_correctness calls out diagram accuracy (e.g., parabola drawn as V-shape), raw subscripts (G_x), Chinese inside LaTeX math mode
- content_completeness emphasizes canvas edge clipping and bare unlabeled diagrams
- readability penalizes text styled as UI components (gray card backgrounds)
- overall instructed to weight overlap and rendering_correctness more heavily
- explicit note to ignore the "N" page UI element

Also increases maxOutputTokens to 3000 since longer rubric produces longer justifications.
Reporter now guards against null scores (scorer failures no longer crash report generation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address code review findings

Critical fixes:
- use-chat-sessions: restore agent_end handling, currentMessageId fallback
  for text_delta/action events with missing messageId, and re-throw on
  SSE error events (previously silently pushed to buffer only).
- eval runner: serialize ActionEngine executions via promise chain.
  void-fire-and-forget raced with ensureWhiteboardOpen's 2s delay and
  could insert elements out of order or before the first element was
  committed to the store.

Important fixes:
- CHAT_MODEL default: 'openai/gpt-4o-mini' -> 'openai:gpt-4o-mini'
  (parseModelString splits on ':', not '/').
- CheckpointResult.score is now VlmScore | null; removed the
  'as unknown as' cast that hid the null contract from consumers.
- Delete dead code: eval/whiteboard-layout/chat-client.ts and
  components/chat/process-sse-stream.ts (both unused after the
  shared agent loop refactor).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
2026-04-18 15:29:54 +08:00

222 lines
6.7 KiB
TypeScript

/**
* Agent Loop — Shared core logic for the frontend-driven multi-agent loop.
*
* Extracted from use-chat-sessions.ts so both the frontend hook and the
* eval harness share the same loop logic. No React dependency — pure
* async function with callback injection for environment-specific behavior.
*
* The loop runs per-user-message: the director dispatches agents one at a
* time, each agent generates a response, and the loop continues until the
* director says END, cues the user, or maxTurns is reached.
*/
import type { StatelessEvent, DirectorState } from '@/lib/types/chat';
import { createLogger } from '@/lib/logger';
const log = createLogger('AgentLoop');
// ==================== Types ====================
/** Store state snapshot sent with each /api/chat request */
export interface AgentLoopStoreState {
stage: unknown;
scenes: unknown[];
currentSceneId: string | null;
mode: string;
whiteboardOpen: boolean;
}
/** Request template — fields that stay constant across loop iterations */
export interface AgentLoopRequest {
config: {
agentIds: string[];
sessionType?: string;
agentConfigs?: Record<string, unknown>[];
[key: string]: unknown;
};
userProfile?: { nickname?: string; bio?: string };
apiKey: string;
baseUrl?: string;
model?: string;
providerType?: string;
}
/** Per-iteration outcome extracted from the done event */
export interface AgentLoopIterationResult {
directorState?: DirectorState;
totalAgents: number;
agentHadContent: boolean;
cueUserReceived: boolean;
}
/** Callbacks injected by the caller (frontend or eval) */
export interface AgentLoopCallbacks {
/** Get fresh store state for each iteration (whiteboard may have changed) */
getStoreState: () => AgentLoopStoreState;
/** Get current messages for the request */
getMessages: () => unknown[];
/**
* Make the HTTP request to /api/chat.
* Returns a Response object (or equivalent with .body ReadableStream).
*/
fetchChat: (body: Record<string, unknown>, signal: AbortSignal) => Promise<Response>;
/**
* Process a single SSE event. Called for every event in the stream.
* The callback should handle action execution, text accumulation,
* message construction, and UI updates.
*/
onEvent: (event: StatelessEvent) => void;
/**
* Called after all SSE events for one iteration have been processed
* and the stream is closed.
*
* Must return the iteration result (extracted from the 'done' event).
* The frontend waits for buffer drain here before reading the result
* from loopDoneDataRef. The eval harness returns a result it
* accumulated during onEvent calls.
*/
onIterationEnd: () => Promise<AgentLoopIterationResult | null>;
}
/** Final outcome of the agent loop */
export interface AgentLoopOutcome {
/** Why the loop stopped */
reason: 'end' | 'cue_user' | 'max_turns' | 'aborted' | 'empty_turns' | 'no_done';
/** Accumulated director state */
directorState?: DirectorState;
/** Number of iterations completed */
turnCount: number;
}
// ==================== Core Loop ====================
/**
* Run the agent loop — shared between frontend and eval.
*
* Each iteration: refresh state → POST /api/chat → process SSE events
* → check exit conditions → repeat.
*/
export async function runAgentLoop(
request: AgentLoopRequest,
callbacks: AgentLoopCallbacks,
signal: AbortSignal,
maxTurns: number,
): Promise<AgentLoopOutcome> {
let directorState: DirectorState | undefined = undefined;
let turnCount = 0;
let consecutiveEmptyTurns = 0;
while (turnCount < maxTurns) {
if (signal.aborted) {
return { reason: 'aborted', directorState, turnCount };
}
// Refresh store state each iteration — agent actions may have changed
// whiteboard, scene, or mode between turns
const freshStoreState = callbacks.getStoreState();
const currentMessages = callbacks.getMessages();
// Build request body
const body: Record<string, unknown> = {
messages: currentMessages,
storeState: freshStoreState,
config: request.config,
directorState,
userProfile: request.userProfile,
apiKey: request.apiKey,
baseUrl: request.baseUrl,
model: request.model,
providerType: request.providerType,
};
// Fetch
const response = await callbacks.fetchChat(body, signal);
if (!response.ok) {
const errorText = await response.text();
throw new Error(`API error: ${response.status} - ${errorText}`);
}
// Parse SSE stream and process events
const reader = response.body?.getReader();
if (!reader) throw new Error('No response body');
const decoder = new TextDecoder();
let sseBuffer = '';
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
sseBuffer += decoder.decode(value, { stream: true });
const parts = sseBuffer.split('\n\n');
sseBuffer = parts.pop() || '';
for (const part of parts) {
const line = part.trim();
if (!line.startsWith('data: ')) continue;
try {
const event: StatelessEvent = JSON.parse(line.slice(6));
callbacks.onEvent(event);
} catch {
// Skip malformed events (heartbeats, etc.)
}
}
}
} finally {
reader.releaseLock();
}
if (signal.aborted) {
return { reason: 'aborted', directorState, turnCount };
}
// Post-iteration: wait for buffer drain (frontend) or collect results (eval)
const iterationResult = await callbacks.onIterationEnd();
// Check exit conditions
if (!iterationResult) {
return { reason: 'no_done', directorState, turnCount };
}
// Update accumulated director state
directorState = iterationResult.directorState;
turnCount = directorState?.turnCount ?? turnCount + 1;
// Director said USER — stop loop
if (iterationResult.cueUserReceived) {
return { reason: 'cue_user', directorState, turnCount };
}
// Director said END — no agent spoke
if (iterationResult.totalAgents === 0) {
return { reason: 'end', directorState, turnCount };
}
// Track consecutive empty responses
if (!iterationResult.agentHadContent) {
consecutiveEmptyTurns++;
if (consecutiveEmptyTurns >= 2) {
log.warn(
`[AgentLoop] ${consecutiveEmptyTurns} consecutive empty agent responses, stopping loop`,
);
return { reason: 'empty_turns', directorState, turnCount };
}
} else {
consecutiveEmptyTurns = 0;
}
}
// maxTurns reached
if (turnCount >= maxTurns) {
log.info(`[AgentLoop] Max turns (${maxTurns}) reached`);
}
return { reason: 'max_turns', directorState, turnCount };
}