wyuc ea0e8126a5
feat: whiteboard layout quality eval harness (#425)
* feat(eval): add state manager bridging ActionEngine for eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add shared types for whiteboard layout eval harness

* feat(eval): add SSE chat client for whiteboard eval

* feat(eval): add Playwright capture module for whiteboard screenshots

* feat(eval): add VLM scorer for whiteboard layout evaluation

* feat(eval): add report generator for whiteboard eval results

* feat(eval): add 8 constructed scenarios for whiteboard layout eval

* feat(eval): add minimal whiteboard render page for Playwright screenshots

Creates app/eval/whiteboard/page.tsx — a headless client page that
seeds the stageStore with a synthetic slide scene, exposes
window.__setElements() for Playwright to inject PPTElement[], and
renders them via ScreenElement inside a 1000×562.5px white canvas.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add main runner for whiteboard layout eval

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): add eval:whiteboard script, install tsx, gitignore results

* fix(eval): fix TS errors, lint, and prettier formatting

* fix(eval): address code review — add cue_user/empty turn guards, validate VLM output, fix empty array crash

* refactor(eval): replace synthetic scenarios with realistic ones

- Replace 8 generic scenarios with 6 that match real usage patterns
- Multi-agent discussion with short user replies (嗯, 明白了, 继续)
- Include real slide scene data as initialStoreState
- Generated agent configs with Chinese names and proper roles
- Cover: physics, math, finance, primary school, economics, medical

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract shared agent loop from use-chat-sessions

Extract the core agent loop logic into lib/chat/agent-loop.ts as a pure
async function with callback injection. Both the frontend React hook and
the eval harness now share the same loop — SSE parsing, exit conditions
(END/cue_user/empty turns/max turns), and director state accumulation.

The frontend wires StreamBuffer callbacks for UI pacing; the eval wires
ActionEngine + message accumulation for headless execution. If loop logic
changes in the shared module, both consumers automatically stay in sync.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(eval): use project LLM infrastructure, fix eval page and model config

- Rewrite scorer to use resolveModel() + generateText() from AI SDK
  instead of raw fetch — supports all providers (OpenAI, Google, Anthropic)
- Model config via env vars (EVAL_CHAT_MODEL, EVAL_SCORER_MODEL),
  matching the pattern from outline-language eval
- Fix eval page: bootstrap store before SceneProvider mounts
- Fix __dirname for tsx CJS mode
- Remove --api-key/--scorer-model CLI args (use env vars instead)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): organize results by model/timestamp

* fix: remove double turnCount increment in shared agent loop

The extracted agent-loop.ts had both `turnCount = directorState?.turnCount ?? turnCount + 1`
(line 190) and a redundant `turnCount++` (line 215), causing multi-agent scenarios to hit
maxTurns at half the expected number of iterations.

Also removes unused processSSEStream import from use-chat-sessions.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): organize screenshots by scenario subdirectory

Results structure: results/<model>/<timestamp>/<scenario>/run0_turn1.png
Report files stay at the timestamp level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): revise scorer rubric and add rescore mode

- Replace space_utilization with rendering_correctness and content_completeness
- Rubric now evaluates from a teacher's perspective (empty space is normal)
- readability emphasizes font size consistency
- Add --rescore flag to re-score existing screenshots without re-running chat
- Increase maxOutputTokens to 2000, add JSON parse error recovery
- Score errors no longer abort the entire scenario

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(eval): sharpen scorer rubric with teacher-perspective examples

The rubric now catches specific classroom whiteboard failure modes:
- overlap now explicitly penalizes writing over existing content when empty space is available (spatial planning failure)
- rendering_correctness calls out diagram accuracy (e.g., parabola drawn as V-shape), raw subscripts (G_x), Chinese inside LaTeX math mode
- content_completeness emphasizes canvas edge clipping and bare unlabeled diagrams
- readability penalizes text styled as UI components (gray card backgrounds)
- overall instructed to weight overlap and rendering_correctness more heavily
- explicit note to ignore the "N" page UI element

Also increases maxOutputTokens to 3000 since longer rubric produces longer justifications.
Reporter now guards against null scores (scorer failures no longer crash report generation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address code review findings

Critical fixes:
- use-chat-sessions: restore agent_end handling, currentMessageId fallback
  for text_delta/action events with missing messageId, and re-throw on
  SSE error events (previously silently pushed to buffer only).
- eval runner: serialize ActionEngine executions via promise chain.
  void-fire-and-forget raced with ensureWhiteboardOpen's 2s delay and
  could insert elements out of order or before the first element was
  committed to the store.

Important fixes:
- CHAT_MODEL default: 'openai/gpt-4o-mini' -> 'openai:gpt-4o-mini'
  (parseModelString splits on ':', not '/').
- CheckpointResult.score is now VlmScore | null; removed the
  'as unknown as' cast that hid the null contract from consumers.
- Delete dead code: eval/whiteboard-layout/chat-client.ts and
  components/chat/process-sse-stream.ts (both unused after the
  shared agent loop refactor).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
2026-04-18 15:29:54 +08:00
..