* test(orchestration): add premature-END regression eval (#598 follow-up)
Adds eval/orchestration/ following the outline-language pattern: a runner
that A/B-tests the director against the same scenario with two prompt
variants — current main (post-#554) and a synthesised pre-#554 baseline
that strips the role-aware summary labels AND the new system.md rules
10/11/12 together.
Pass criterion is now framed as a regression guard rather than a fixture
discrimination test: every scenario's post-fix END rate must stay below
EVAL_END_THRESHOLD (default 20%). The pre-vs-post Δ is reported as
informational data, since #554's reviewer feedback was that earlier
fixtures didn't discriminate.
Empirical finding across 7 shipped model configs (gpt-4.1-mini,
gpt-4o-mini, gpt-5.4-nano, qwen-plus, qwen3.5-flash, deepseek-chat,
deepseek-v4-flash, gemini-2.5-flash, claude-haiku-4-5) with 5 scenarios
modelled on #511 (incl. the exact Tiananmen-3D objection trace, soft
pushback after a long resolved-looking discussion, topic pivot, brief
acknowledgement, and explicit teacher-signals-end-then-user-objects):
no scenario produced a non-zero END rate in either variant. Useful
data for the #554/#598 discussion — the prompt-layer rules don't
measurably change behavior on these shipped models with these prompts,
but the eval is now wired so future regressions show up.
* test(orchestration): add director question-answering eval (#598 follow-up)
Adds an A/B eval probing whether the director routes correctly when the
conversation contains an unanswered user question — both in the first-turn
drift case (agents started answering but drifted onto adjacent topics,
no frustration yet) and in the escalated frustration case (user has
already complained multiple times).
The prod symptom this targets ("答非所问"): agents reply on adjacent
topics rather than answering the literal question, and the director
keeps picking peer agents for "variety" instead of routing back to
the teacher to actually answer.
Scenarios: 5 synthesized cases across math, biology, English grammar,
physics, and calculus. 3 are first-turn drift (no frustration yet),
2 include frustration signals. Each captures: user asks specific
question, agents reply on adjacent topics, director is asked to pick
the next move.
Decision rule-judged into USER | TEACHER | OTHER_AGENT | END:
- USER → ✓ valid (cue user to clarify)
- TEACHER → ✓ valid (re-route to teacher to answer)
- OTHER_AGENT → ✗ wrong (peer-agent "variety" routing — the bug)
- END → ✗ wrong
A/B:
- baseline : current director template with rule 13 stripped
- with_rule : current director template as-shipped
Pass = with_rule.correctRate ≥ EVAL_PASS_THRESHOLD (default 0.7).
Pre-vs-post Δ is reported as informational only.
Note this commit only adds the eval; the rule it strips/restores in
the A/B is injected in the next commit.
* fix(orchestration): inject director rule for answering user questions
Adds rule 13 to lib/prompts/templates/director/system.md so the director
recognizes when the conversation contains an unanswered user question
and routes appropriately — to the teacher (literal role match) by
default, or to USER cue when the question is ambiguous or the user
has already complained multiple times.
This addresses the "答非所问" symptom: when agents drift onto adjacent
topics, the director keeps picking peer agents for "variety" rather
than routing back to the teacher to actually answer. The rule is
preventive (applies whenever the question is unanswered, including
the first turn after drift) rather than reactive-only (waiting for
the user to express frustration).
Rule wording uses constrained-output framing: literal `role: teacher`
match (not "best fit" / "highest priority" / "most knowledgeable"),
and an explicit list of forbidden choices (peer agents, END). Earlier
softer wordings let the model pick the assistant when it seemed more
topically relevant; the literal-match wording prevents that.
Validated by the question-answering eval added in the previous commit.
On google:gemini-3-flash-preview, 5 samples per variant across 5
synthesized scenarios:
| Scenario | baseline | with_rule |
|----------------------------------------------------|----------|-----------|
| math_quadratic_axis_drift_no_frustration | 100% | 100% |
| bio_dark_reaction_drift_no_frustration | 80% | 100% |
| english_team_isare_drift_no_frustration | 20% | 100% |
| physics_inertial_mass_escalated_frustration | 0% | 100% |
| calculus_product_rule_drift_no_frustration | 60% | 100% |
| **mean** | **52%** | **100%** |
3 of 5 scenarios reproduce the bug at baseline (correct rate ≤ 60%);
the rule lifts every scenario to 100%. The remaining 2 baselines are
already high — the rule doesn't regress them.
* fix(orchestration): teach agents to answer the user's literal question
Adds an "Answering the User's Question" section to the agent system
prompt. Before this, the agent SP had no rule for direct-answering — the
persona, role guideline, current-state context, and length guidelines all
pushed the agent to advance the lesson and inspire thought, with no
explicit directive to first answer the user's literal question.
The result observed in prod ("答非所问"): when the user asks a specific
question (a formula, a yes/no, a term), agents reply with adjacent
concepts, examples, or follow-up questions — but never the literal
answer.
The new section directs the agent, on every response where the user
message contains a question, to:
- lead with the concrete answer to the literal question,
- not pivot to an adjacent topic even when it seems pedagogically
richer,
- treat "inspire thought" and peer-differentiation guidance as
applying only AFTER the literal question is answered,
- say "I don't know" directly when uncertain rather than answering
a different question,
- on frustration signals from prior turns, look back at the original
question (before the frustration) and answer THAT specifically.
This is the agent-side counterpart to the director rule landed in the
previous commit. Director routes to the teacher when the question is
unanswered; the teacher now has explicit guidance for what to do when
dispatched.
* refactor+fix(orchestration): rip maxTurns end-to-end; distinguish errors from completion
Two coupled cleanups in one commit since the outcome-handler rewrite
naturally falls out of removing the client-side max-turns loop.
The director graph used `maxTurns: turnCount + 1` to force "one
director→agent cycle per request" — a hack expressing the single-round
contract as a turn limit. This commit removes the hack and the
surrounding user-facing setting, replacing implicit intent with explicit
graph topology.
Server:
- `director-graph.ts`: drop `maxTurns` from state, drop the hard-cap
check at the top of `directorNode`, change `agent_generate → director`
edge to `agent_generate → END`. Single-round contract is now expressed
by topology.
- `lib/types/chat.ts`: drop `maxTurns` and `currentTurn` from
`SessionConfig`; drop `maxTurns?` from `CreateSessionRequest.trigger`.
Client:
- `lib/store/settings.ts`: drop `maxTurns` state, `setMaxTurns` action,
the `'10'` default, and the localStorage migration path.
- `components/agent/agent-bar.tsx`: drop the entire compact-stepper UI
block plus the unused MessageSquare/Minus/Plus lucide imports.
- `components/settings/agent-settings.tsx`: drop the AgentSettings prop
and the multi-agent-only Input block.
- `lib/chat/agent-loop.ts`: drop `maxTurns` parameter, change
`while (turnCount < maxTurns)` to `while (true)` controlled by inner
exits, drop `'max_turns'` reason from `AgentLoopOutcome`.
- `components/chat/use-chat-sessions.ts`: drop the maxTurns lookup, drop
the trailing arg to runAgentLoop, drop vestigial `maxTurns: 0` /
`currentTurn: 0` from session creation.
- `eval/whiteboard-layout/runner.ts`: drop unused MAX_AGENT_TURNS plus
the trailing arg.
i18n: drop `settings.maxTurns` and `settings.maxTurnsDesc` from six
locales (en-US, zh-CN, zh-TW, ja-JP, ar-SA, ru-RU).
Motivation for removing the user-facing setting: the LLM director
controls round length via cue_user / END decisions, so a numeric
"max discussion turns" knob is redundant with rule-driven routing.
When the director prompt was buggy, the cap masked the bug by force-
ending; once director routing is fixed, the cap just removes user
agency.
Before, every non-`cue_user` outcome of the agent loop mapped to
`status: 'completed'` in `use-chat-sessions.ts`. That conflated four
distinct paths:
- `end`: director chose END (legitimate completion)
- `empty_turns`: two consecutive agents returned no text or actions
(parser failure, model timeout, etc.)
- `no_done`: SSE stream ended without a `done` event (infrastructure
failure)
- `aborted`: user pressed stop (handled elsewhere)
The new switch routes `end` to completed, `empty_turns` and `no_done`
through `clearLiveSessionAfterError` so the user sees an inline System
message describing what went wrong instead of a quiet "completed".
- `lib/types/chat.ts`: add `'error'` to `SessionStatus`.
- `components/chat/use-chat-sessions.ts`: rewrite the outcome handler
as a switch; have `clearLiveSessionAfterError` set `status: 'error'`
so all entry points (the runAgentLoop outcome AND existing throw
paths from fetchChat / SSE error events) converge on the same
visible state.
- `components/chat/session-list.tsx`: add `AlertCircle` red icon for
the new error status.
- i18n: add `chat.error.emptyAgentResponses` and
`chat.error.streamInterrupted` strings to all six locales.
This is the in-product fix that the prod investigation surfaced as
an independent root cause of the #511 "discussion ended unexpectedly"
symptom — separate from the director and agent prompt fixes in the
earlier commits of this PR.
* fix(chat): drop dead settingsState binding; wire eval:orchestration:answering script
Two minor CR follow-ups:
- components/chat/use-chat-sessions.ts: the `settingsState` lookup at the
top of runAgentLoopFn was the entry point for the maxTurns lookup
ripped in the previous commit; with maxTurns gone, the binding is
unused and ESLint flags it. Drop the line. (Other uses of
useSettingsStore.getState() elsewhere in this file are unaffected.)
- package.json: the answering eval runner had no convenience script,
while the premature-end runner did. Add eval:orchestration:answering
for symmetry and discoverability.
* fix(orchestration): route ambiguous/unaddressed user turns to teacher, never USER cue
Addresses reviewer issues 2 & 3 (ambiguous "帮我看一下这个" / "Can you take a
look at this?"): the director was cueing USER for vague input, which makes no
agent speak — the user faces dead air with nothing to react to ("nobody
responds"); and when an agent did speak it guessed a topic instead of asking
what the user meant.
Root cause was the previous rule 13, which offered `{"next_agent":"USER"}` as
a valid response to an ambiguous user turn. A USER cue is the wrong tool here:
it returns control to the user without any agent speaking. The right behavior
is to dispatch the teacher, who either answers or — if the message is too
vague — asks a clarifying question out loud.
Director (`director/system.md` rule 13):
- When the most recent user turn is unaddressed (clear question, vague
request, OR frustration signal), the only acceptable output is the literal
`role: teacher` agent id.
- USER cue is now explicitly forbidden as a response to a user turn (reserved
for rule 7, where an agent has just asked the user something).
- Tightened the END prohibition: a high turn count or a thoroughly-discussed
topic does NOT mean the specific question was answered. Added rule 5
(don't-drag-on) to the override list, since that pressure was driving the
director to END on well-discussed-but-unanswered turns.
Agent (`agent-system/system.md`):
- Added a clause to "Answering the User's Question": if the user's message is
too vague to answer (no clear referent), ask ONE short clarifying question
with a concrete option or two — do not guess a topic and lecture, do not
stay silent.
Eval (`answering-runner.ts` + `scenarios/answering.json`):
- Reclassified USER as wrong (dead air); only TEACHER counts as correct now
(teacher answers, or asks a clarifying question for vague input).
- Added two ambiguity scenarios: a bare "帮我看一下这个" opener, and a vague
"再帮我看看这个吧" follow-up after a real Q&A.
- Made the rule-13 strip regex match by rule number rather than heading text,
so it survives heading rewrites.
On google:gemini-3-flash-preview, 10 samples per variant, all 7 scenarios
PASS (with_rule mean ~97%, baseline ~57%). The two ambiguity scenarios:
ambiguous_look_at_this 100%/100%, ambiguous_vague_followup 20%→100%.
Manually verified the agent side: for "帮我看一下这个" the teacher now replies
"…你想讨论的是具体的代码逻辑,还是 Agent 的决策流程图呢?" (clarifying question +
concrete options) instead of guessing.
Note: reviewer issue 1 (follow-up not answered) could not be reproduced on
this branch across 7 configurations; it is consistent with the deployed
environment running older code and/or provider API flakiness, which the C7
error-vs-completion distinction in this PR already surfaces instead of a
silent 已结束.
* docs(eval): fix stale physics scenario description after USER→wrong reclassification
The physics_inertial_mass scenario description still said "USER-cue signal —
but TEACHER re-answer is also valid", which contradicts the new classifier
(USER is dead air = wrong; only TEACHER is correct). Cosmetic only — the
description field is printed into the report, never used in classification.