* test(orchestration): add answer-content eval for agent first-reply answering
The existing director question-answering eval only checks that the director
ROUTES to the teacher; it never generates the dispatched agent's reply, so it
cannot catch the case where routing is correct but the agent's first sentence
drifts (greets / opens a lecture / pivots) and only a later turn answers.
This adds an answer-content eval that runs the real agent-generate inputs
(buildStructuredPrompt + convertMessagesToOpenAI), parses the structured output
with the app's runtime parser, and uses an LLM judge for two signals:
leads_with_answer (first sentence addresses the literal ask) and
answered_anywhere (addressed at all). Their gap quantifies "drift-then-answer".
A/B mirrors the answering-runner rule-13 strip: baseline removes the
"Answering the User's Question" section from agent-system, with_rule is as-shipped.
Scenarios are synthetic/anonymized (opening-lecture override, ignored
format/capability/navigation request, ignored correction, frustration re-ask,
adjacent pivot, vague-clarify) plus clean controls; no real user data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(orchestration): give answer-content scenarios live-shaped classroom context
The minimal store state (empty scenes, null stage, no profile) made the
assembled agent prompt far thinner than the live agent-generate path and
suppressed the slide-narration "lecture reflex" that drives opener drift.
Each scenario now carries a topical slide scene (title + body + latex elements
with ids), a stage (name/description/languageDirective), and a student profile,
so buildStructuredPrompt emits a "# Current State", "# Student Profile" and
"# Language" of the same shape and bulk as production. The runner reads this
context from the scenario instead of a hardcoded empty state.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agent): respond to the user's turn before continuing the lecture
Generalize the agent-system "Answering the User's Question" section into
"Responding to the User's Turn": the user's latest message always takes priority
over continuing the planned lecture. Beyond factual questions it now explicitly
covers the interaction requests agents were ignoring — navigation/pacing
("skip to next page"), format/language ("explain in Chinese"), capability
requests it cannot fulfill ("make a video" -> say so + offer an alternative),
corrections, frustration re-asks, and vague asks (ask one clarifying question).
Strengthen the answer-content eval that measures this:
- the navigation case now uses a 3-slide deck so "skip to next page" is well-posed;
runner supports multi-scene state and strips the renamed section for the A/B;
- judge is fairer to legitimate behavior: explaining primarily in the requested
language while keeping technical terms standard counts; a verbal transition to
the next slide counts as honoring navigation (no page-turn action exists).
Result (gemini-3-flash-preview, n=10): mean leads-with-answer rises from 42%
(section stripped) to 93% (as-shipped); all 12 scenarios pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agent): don't over-respond to bare acknowledgements; guard empty eval set
Code-review follow-ups on the "Responding to the User's Turn" change:
- Prompt: the absolute "every user message needs a direct response / treat
continuing the lecture as a failure" wording had no carve-out for bare
acknowledgements ("ok" / "嗯" / "good" / "thanks"), risking the agent halting
the lesson to manufacture a reply, and conflicted with agent-initiated
discussions where no user turn exists. Add a bare-acknowledgement carve-out,
allow a brief "好的"/"Sure" before the answer, and scope the override to actual
requests (and to "if there is a user turn to respond to").
- Eval: answer-content-runner exited 0 with a PASS verdict and NaN% aggregates
when the scenario set was empty (e.g. an EVAL_SCENARIO typo). Guard it: error
and exit 1 when no scenario matches.
Re-run unchanged otherwise (gemini-3-flash-preview, n=10): mean leads-with-answer
42% (stripped) -> 94% (as-shipped), 11/12 scenarios pass; the chem opener is
variance-prone around the 70% bar and not over-tuned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(eval): cr-loop cleanups on answer-content-runner
- ScenarioAgentSpec extends the shared ScenarioAgent (./types) instead of
re-declaring id/name/role/priority.
- Drop the dead expectedPreFix plumbing from ScenarioResult (it was copied in
but never read; it stays as an informational annotation in the scenarios).
- extractTexts: replace the hand-rolled JSON fallback with the app's
finalizeParser, which recovers plain-prose / unclosed-array output the same
way the runtime does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: tighten acknowledgement carve-out; don't surface action-JSON as text (cr round 2)
Round-2 review follow-ups.
Prompt: the bare-acknowledgement carve-out was too loose — it could swallow
ack+request hybrids ("嗯,那为什么X?") as acks (re-introducing drift), double-listed
"好的" as both an allowed lead-in and a no-response ack, and listed pacing words
("继续"/"go on") that collide with the navigation bullet. Restrict it to a
STANDALONE acknowledgement with nothing else attached; route an embedded
question/request to that request; route pacing words to the navigation bullet.
Eval: extractTexts gated finalizeParser recovery on !state.jsonStarted, so an
actions-only / unclosed array yields "no text" (the judge's intended empty case)
instead of finalizeParser's raw-buffer fallback surfacing action JSON as fake speech.
Re-run (gemini-3-flash-preview, n=5): no regression — 11/12 pass, meta-requests
100/100/80/100; the chem opener remains variance-prone around the 70% bar.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address review — honest slide-nav, strict judge booleans, errors fail (cr)
Review follow-ups from @cosarah:
- Navigation (P2): agents have no slide-navigation action, so "do it" + a verbal
transition let "skip to next page" pass the eval while the UI stayed on the
same slide. Reframe prompt, judge, and scenario so a correct reply ACKNOWLEDGES
the request and is HONEST that it cannot change the slide itself (continue
verbally / tell the user how to navigate); pretending it flipped the slide no
longer counts.
- Judge booleans (P2): answer-content-judge coerced fields with Boolean(...), so a
string "false" became true. Parse strictly (real bool or "true"/"false");
malformed output is flagged as an error.
- Error accounting (P3): aggregate excluded errored samples from the denominator,
letting a scenario pass on one good sample. Errors now count as failures
(denominator = all requested samples).
Re-run (gemini-3-flash-preview, n=10): navigation 0% -> 100% under the honest
framing; aggregate leads-with-answer ~93%, no regression. Opener scenarios
(math/chem) remain variance-prone near the 70% bar.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(eval): judge first-sentence lead; use canonical teacher actions (cr round 2)
Second review round from @cosarah:
- leads_with_answer was judged against texts[0] (the whole first JSON text
block), so "Today we'll discuss parabolas. The formula is x=-b/(2a)." counted
as leading even though the first sentence drifted. Now extract the first ~2
sentences as the "opening" and have the judge make the call: a brief
acknowledgement of the user before the answer counts as leading, but a greeting
/ self-intro / topic preamble before the answer does not. (Two sentences +
semantic judge handles "Sorry, let me clarify. m is inertial mass." vs
"Welcome! …" without a brittle keyword list.)
- The runner hardcoded TEACHER_ACTIONS, omitting canonical teacher actions
(play_video, wb_draw_chart). Use getActionsForRole('teacher') so the available-
actions section matches production.
Re-run (gemini-3-flash-preview agent, n=10): 11/12 pass, aggregate leads ~90%.
The chem opener now correctly fails (~10%): its "Welcome! Today we…" persona
greets first, so at sentence granularity it is genuine first-sentence drift
(answered-anywhere ~90%) — exactly what the stricter lead measurement should
catch.
Note: the judge model is deepseek-v4-pro in this run because the anthropic
gateway is currently returning Bad Request in our environment; the eval is
judge-model-agnostic via EVAL_JUDGE_MODEL.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(eval): judge the opening across all text blocks, not just texts[0] (cr round 3)
Review follow-up from @cosarah: leadFromTexts only looked inside the first
text block, so a valid multi-block reply like
[{"type":"text","content":"Sure."},{"type":"text","content":"The derivative is 2x."}]
was judged on "Sure." alone and failed leads_with_answer despite following the
allowed acknowledgement-then-answer pattern. Build the opening from the first ~2
speech sentences across ALL ordered text chunks.
Re-run (gemini-3-flash-preview, n=10, deepseek judge): 11/12 pass, aggregate
leads ~93%. The chem opener stays sub-threshold (~50%) — its "Welcome! Today
we…" persona greets before answering about half the time, a genuine
first-sentence drift.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>