* fix(orchestration): use metadata-based human/agent discrimination in director
Closes#511
The original approach used AGENT_PREFIX_RE on converted message content to
distinguish human turns from agent turns. This was unreliable: message-converter.ts
applies a [senderName]: prefix to ALL role:'user' messages unconditionally, so
extractLastHumanMessage() matched every human message as an agent turn and always
returned null — the Open Student Question section never fired.
Fix per reviewer feedback (use real structured metadata, not display text):
- extractLastHumanMessage() now takes StatelessChatRequest['messages'] (pre-conversion)
and uses msg.metadata.originalRole === 'user' as the source-of-truth discriminator.
Text is extracted from msg.parts directly — no content parsing.
- summarizeConversation() uses msg.role for human/agent discrimination. In the director
path, convertMessagesToOpenAI() is called without currentAgentId, so peer agent
messages remain role:'assistant' and role:'user' reliably identifies human turns.
The [senderName]: prefix is stripped from summary output cosmetically only.
- buildDirectorPrompt() 10th param changed from OpenAIMessage[] to
StatelessChatRequest['messages']; director-graph.ts passes state.messages (already
in scope at line 164, no extra computation).
- Tests updated to use director-path message shapes: humanMsg() carries [You]: prefix,
agentMsg() uses role:'assistant'. extractLastHumanMessage tests use originalRole metadata.
* refactor(orchestration): move attribution fix to prompt layer; add eval fixtures
Replaces procedural extractLastHumanMessage + buildOpenStudentQuestionSection
wiring with three declarative rules in director/system.md — labels are
authoritative, END is blocked by an unresolved student question, and a
brief acknowledgment does not constitute resolution.
Removes: extractLastHumanMessage, buildOpenStudentQuestionSection,
{{openStudentQuestionSection}} placeholder, originalMessages 10th param.
Keeps: summarizeConversation role-based discrimination (still the core fix).
Adds eval/orchestration/ (runner.ts, judge.ts, reporter.ts, types.ts,
scenarios/premature-end.json) following the outline-language eval pattern,
with a deterministic premature-END scenario as the must-have regression guard.
* chore(orchestration): remove eval fixtures and stale tests per review
Per wyuc's review on PR #554: the eval/orchestration fixtures pass on
main before the fix, so they don't discriminate — pull them out so the
PR lands as a clean prompt-layer change. Track eval as a follow-up.
- Remove eval/orchestration/ (judge, runner, reporter, types, scenario)
- Remove eval:orchestration npm script from package.json
- Delete 3 stale tests from tests/prompts/templates.test.ts that
called buildDirectorPrompt with a 10th param removed in the refactor
- Fix prettier formatting in conversation-summary.test.ts and
templates.test.ts (was masking the TS break in CI)
All checks pass: prettier clean, 333 unit tests green, 0 lint errors,
tsc --noEmit clean.
---------
Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>