Ashutosh Rana e5148bece5
fix(orchestration): restore agent attribution in director summary (#554)
* fix(orchestration): use metadata-based human/agent discrimination in director

Closes #511

The original approach used AGENT_PREFIX_RE on converted message content to
distinguish human turns from agent turns. This was unreliable: message-converter.ts
applies a [senderName]: prefix to ALL role:'user' messages unconditionally, so
extractLastHumanMessage() matched every human message as an agent turn and always
returned null — the Open Student Question section never fired.

Fix per reviewer feedback (use real structured metadata, not display text):

- extractLastHumanMessage() now takes StatelessChatRequest['messages'] (pre-conversion)
  and uses msg.metadata.originalRole === 'user' as the source-of-truth discriminator.
  Text is extracted from msg.parts directly — no content parsing.

- summarizeConversation() uses msg.role for human/agent discrimination. In the director
  path, convertMessagesToOpenAI() is called without currentAgentId, so peer agent
  messages remain role:'assistant' and role:'user' reliably identifies human turns.
  The [senderName]: prefix is stripped from summary output cosmetically only.

- buildDirectorPrompt() 10th param changed from OpenAIMessage[] to
  StatelessChatRequest['messages']; director-graph.ts passes state.messages (already
  in scope at line 164, no extra computation).

- Tests updated to use director-path message shapes: humanMsg() carries [You]: prefix,
  agentMsg() uses role:'assistant'. extractLastHumanMessage tests use originalRole metadata.

* refactor(orchestration): move attribution fix to prompt layer; add eval fixtures

Replaces procedural extractLastHumanMessage + buildOpenStudentQuestionSection
wiring with three declarative rules in director/system.md — labels are
authoritative, END is blocked by an unresolved student question, and a
brief acknowledgment does not constitute resolution.

Removes: extractLastHumanMessage, buildOpenStudentQuestionSection,
{{openStudentQuestionSection}} placeholder, originalMessages 10th param.
Keeps: summarizeConversation role-based discrimination (still the core fix).

Adds eval/orchestration/ (runner.ts, judge.ts, reporter.ts, types.ts,
scenarios/premature-end.json) following the outline-language eval pattern,
with a deterministic premature-END scenario as the must-have regression guard.

* chore(orchestration): remove eval fixtures and stale tests per review

Per wyuc's review on PR #554: the eval/orchestration fixtures pass on
main before the fix, so they don't discriminate — pull them out so the
PR lands as a clean prompt-layer change. Track eval as a follow-up.

- Remove eval/orchestration/ (judge, runner, reporter, types, scenario)
- Remove eval:orchestration npm script from package.json
- Delete 3 stale tests from tests/prompts/templates.test.ts that
  called buildDirectorPrompt with a 10th param removed in the refactor
- Fix prettier formatting in conversation-summary.test.ts and
  templates.test.ts (was masking the TS break in CI)

All checks pass: prettier clean, 333 unit tests green, 0 lint errors,
tsc --noEmit clean.

---------

Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
2026-05-25 22:42:45 +08:00
..