wyuc dfebbcf33f
perf(classroom): speed up classroom loading (media hydration, sidebar thumbnails, media range requests) (#1276)
* perf(classroom): defer non-priority media blob hydration off the load path

Entering a classroom awaited the full mediaFiles restore — one object URL
per restored image/video blob — before the loading gate opened. For
video-heavy courses that is hundreds of MB of IndexedDB materialization
blocking first paint.

Split the restore into two phases:

- The awaited phase now builds metadata-complete task entries but only
  creates object URLs for failed rows (none needed) and for media
  referenced by the scene the classroom opens on (persisted cursor, else
  the first scene). buildRestoredMediaTasks gains an optional
  shouldHydrateBlob predicate, defaulting to eager hydration so existing
  callers are unchanged.
- Remaining blob-backed records hydrate in the background, chunked over
  requestIdleCallback (setTimeout fallback). Each deferred task stays
  'done' — so generation resume never re-runs it — but carries no
  objectUrl yet, which the media resolution state machine already renders
  as a pending skeleton until the URL lands.

Background hydration is guarded per record: a task that was replaced
(classroom switch, regeneration, retry) or that belongs to another stage
is skipped and its freshly minted URLs are revoked immediately.

* perf(classroom): lazy-render slide thumbnails in the playback sidebar

The playback scene sidebar mounted a full SlideCanvas for every slide
scene the moment the classroom opened (and every off-screen video element
opened a preload="metadata" fetch), because SlideThumbnail's existing
visible prop was never passed.

Extract the editor nav rail's near-viewport IntersectionObserver hook to
lib/hooks/use-near-viewport.ts and gate the sidebar's slide thumbnails
through it: only scenes within 200px of the viewport render the live
canvas; the rest show SlideThumbnail's existing placeholder until scrolled
near. The placeholder keeps the same box size, so gating never shifts
layout.

The shared hook now starts hidden instead of eager: with the previous
eager-initial state, opening a long deck still mounted every canvas for a
frame before the observer could flip off-screen items off. The observer's
guaranteed initial delivery flips near-viewport items within a frame; the
no-IntersectionObserver fallback (e.g. jsdom) defers to a microtask so
the effect never synchronously re-renders.

* perf(media): lazy image decoding and HTTP Range support for classroom media

Renderer: BaseImageElement's <img> now carries loading="lazy" and
decoding="async", so thumbnail-heavy surfaces (playback sidebar, editor
nav rail, course cards) no longer fetch and decode every slide image up
front. In-viewport images are unaffected — the browser fetches them
immediately. slideToPng forces eager loading inside its permanently
off-screen snapshot tree, where lazy images would otherwise never fetch
and exports would capture blank slides.

Server: GET /api/classroom-media/[classroomId]/[...path] now answers
single byte-range requests with 206 Partial Content (Content-Range,
Accept-Ranges, correct Content-Length), enabling progressive playback and
seeking for hosted video/audio instead of downloading whole files. Suffix
ranges are supported; unsatisfiable ranges get 416 with the full size;
unsupported units or multi-range sets fall back to the plain 200 full-body
response, which is always a legal answer. Existing caching headers are
kept on every response shape.

* chore(renderer): bump version to 0.1.4 for image lazy-loading change

* fix(classroom): guard deferred hydration against restarted tasks and bound idle waits

A deferred record is only ever 'done' without an objectUrl; a task that
regeneration or retry restarted passes through pending/generating, so skip
those instead of attaching stale persisted bytes. Also give the idle
scheduling a timeout so a busy main thread cannot starve hydration.

* fix(classroom): stop superseded deferred hydration before minting URLs

A restore epoch captured at apply time now gates each idle chunk: loading
another classroom or reloading the same one invalidates any older hydration
loop, so it neither keeps scheduling work for an abandoned classroom nor
attaches bytes read by an older load to a newer load's tasks.

* fix(classroom): keep 416s uncached and classify element-keyed media as priority

A 416 with public immutable caching can poison the media URL for later
valid requests, so range errors now send Cache-Control: no-store. Priority
classification also collects the opening scene's media element ids, since
task lookup binds records keyed stage:<elementId> even when the slide slot
carries a different opaque ref.

* fix(classroom): tie deferred hydration liveness to the classroom load token

The restore epoch only advanced when the next load reached apply, so an
abandoned classroom kept hydrating during the next load's storage/network
phase. Compose the epoch with the load's isCurrent (load token plus effect
cleanup) so navigation stops the loop at the next idle boundary.

* fix(classroom): classify legacy-recovered media before deferring it

Legacy singleton video recovery assigns a placeholderRef only at the end of
the task build, so a record keyed by an allocated id without placeholderRef
was deferred even when it backs the opening scene's gen_vid_* element. Run a
metadata-only build first to learn each record's effective ref and classify
against it, keeping the first visible page's legacy video eager.

* fix(action): wait for deferred video bytes before starting play_video

executePlayVideo treated status done as immediately playable, but a deferred
restore is done without an objectUrl: the renderer shows a skeleton, no
<video> exists, and the later hydration never retriggers play, leaving the
action stuck until the safety timeout. Readiness now follows the renderer's
contract (only done-with-bytes is playable) at the initial check, the
subscription exit, and the post-subscription recheck; the failed skip applies
whether or not a wait happened.

* fix(classroom): include the stage whiteboard in priority media refs

The stage-level whiteboard stays open across standalone classroom switches,
so its media can be visible before any scene is. Classifying it as deferred
left visible whiteboard media pending behind idle hydration chunks.
2026-08-30 03:48:04 +08:00

903 lines
29 KiB
TypeScript

/**
* ActionEngine — Unified execution layer for all agent actions.
*
* Replaces the 28 Vercel AI SDK tools in ai-tools.ts with a single engine
* that both online (streaming) and offline (playback) paths share.
*
* Two execution modes:
* - Fire-and-forget: spotlight, laser — dispatch and return immediately
* - Synchronous: speech, whiteboard, discussion — await completion
*/
import type { StageStore } from '@/lib/api/stage-api';
import { createStageAPI } from '@/lib/api/stage-api';
import { useCanvasStore } from '@/lib/store/canvas';
import { useWhiteboardHistoryStore } from '@/lib/store/whiteboard-history';
import { useMediaGenerationStore, type MediaTask } from '@/lib/store/media-generation';
import type { AudioPlayer } from '@/lib/utils/audio-player';
import type { LegacySpeechAction } from '@/lib/types/action';
import type {
Action,
SpotlightAction,
LaserAction,
SpeechAction,
PlayVideoAction,
WbDrawTextAction,
WbDrawShapeAction,
WbDrawChartAction,
WbDrawLatexAction,
WbDrawTableAction,
WbDeleteAction,
WbDrawLineAction,
WbDrawCodeAction,
WbEditCodeAction,
WidgetHighlightAction,
WidgetSetStateAction,
WidgetAnnotationAction,
WidgetRevealAction,
} from '@/lib/types/action';
import type { CodeLine, PPTVideoElement } from '@openmaic/dsl';
import {
resolveVideoMediaForElement,
type VideoMediaTaskResolution,
} from '@/lib/media/media-task-resolution';
import {
EFFECT_AUTO_CLEAR_MS,
MAX_VIDEO_WAIT_MS,
WB_OPEN_MS,
WB_DRAW_MS,
WB_EDIT_MS,
WB_DELETE_MS,
WB_CLOSE_MS,
WIDGET_MS,
wbDrawCodeMs,
wbClearMs,
} from '@/lib/choreography';
import katex from 'katex';
import { createLogger } from '@/lib/logger';
const log = createLogger('ActionEngine');
// ==================== SVG Paths for Shapes ====================
const SHAPE_PATHS: Record<string, string> = {
rectangle: 'M 0 0 L 1000 0 L 1000 1000 L 0 1000 Z',
circle: 'M 500 0 A 500 500 0 1 1 499 0 Z',
triangle: 'M 500 0 L 1000 1000 L 0 1000 Z',
};
// ==================== Helpers ====================
function delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
const COMMON_LATEX_COMMAND =
/\\(?:alpha|beta|cdot|delta|dfrac|frac|gamma|infty|int|lambda|left|lim|mu|neq|omega|pi|pm|prod|rightarrow|right|sigma|sqrt|sum|text|tfrac|theta|times)\b/;
function getDelimitedLatex(content: string): string | null {
const trimmed = content.trim();
if (trimmed.length > 4 && trimmed.startsWith('$$') && trimmed.endsWith('$$')) {
return trimmed.slice(2, -2).trim();
}
if (
trimmed.length > 2 &&
trimmed.startsWith('$') &&
trimmed.endsWith('$') &&
!trimmed.startsWith('$$') &&
!trimmed.endsWith('$$')
) {
return trimmed.slice(1, -1).trim();
}
return null;
}
function getLikelyLatexMath(content: string): string | null {
const trimmed = content.trim();
if (!trimmed || trimmed.startsWith('<')) return null;
const delimitedLatex = getDelimitedLatex(trimmed);
if (delimitedLatex !== null) return delimitedLatex;
if (/^[A-Za-z]:\\/.test(trimmed)) return null;
if (COMMON_LATEX_COMMAND.test(trimmed) || /[_^]\{/.test(trimmed)) return trimmed;
const commands = trimmed.match(/\\[A-Za-z]+/g) ?? [];
if (commands.length === 0) return null;
if (commands.length === 1) {
return /\\[A-Za-z]+\s*\{[^{}]*\}/.test(trimmed) ? trimmed : null;
}
if (!/[=+\-*/^_{}]/.test(trimmed)) return null;
const commandCharacters = commands.reduce((total, command) => total + command.length, 0);
return commandCharacters / trimmed.length >= 0.15 ? trimmed : null;
}
/** Convert raw code string to CodeLine array with unique IDs */
function codeToLines(code: string): CodeLine[] {
return code.split('\n').map((content, i) => ({
id: `L${i + 1}`,
content,
}));
}
let lineIdCounter = 0;
/** Generate unique line IDs for newly inserted lines */
function generateLineIds(count: number): string[] {
return Array.from({ length: count }, () => `L_${++lineIdCounter}_${Date.now().toString(36)}`);
}
/** Resolve the video element and its renderer-equivalent media binding for an action. */
export function resolveActionVideoMedia(
stageStore: StageStore,
tasks: Readonly<Record<string, MediaTask>>,
elementId: string,
): VideoMediaTaskResolution<MediaTask> | undefined {
const { stage, scenes, currentSceneId } = stageStore.getState();
const orderedScenes = currentSceneId
? [
scenes.find((scene) => scene.id === currentSceneId),
...scenes.filter((scene) => scene.id !== currentSceneId),
]
: scenes;
for (const scene of orderedScenes) {
if (!scene || scene.content.type !== 'slide') continue;
const element = scene.content.canvas.elements.find(
(candidate): candidate is PPTVideoElement =>
candidate.id === elementId && candidate.type === 'video',
);
if (element) return resolveVideoMediaForElement(tasks, element, stage?.id);
}
return undefined;
}
/**
* Renderer-aligned playability: a task is playable only once its bytes exist
* (`done` + objectUrl). A deferred restore is `done` without an objectUrl and
* the renderer still treats it as pending (`resolveMediaRef` maps it to
* 'pending'), so starting playback in that state would set
* playingVideoElementId while no <video> exists — and the later hydration
* would not retrigger play, leaving the action stuck until its timeout.
*/
function isPlayableVideoTask(task: MediaTask): boolean {
return task.status === 'done' && !!task.objectUrl;
}
// ==================== ActionEngine ====================
/** Callback for sending messages to widget iframe */
export type WidgetMessageCallback = (type: string, payload: Record<string, unknown>) => void;
export interface ActionExecutionOptions {
silent?: boolean;
signal?: AbortSignal;
}
export class ActionEngine {
private stageStore: StageStore;
private stageAPI: ReturnType<typeof createStageAPI>;
private audioPlayer: AudioPlayer | null;
private effectTimer: ReturnType<typeof setTimeout> | null = null;
private widgetMessageCallback: WidgetMessageCallback | null = null;
constructor(
stageStore: StageStore,
audioPlayer?: AudioPlayer | null,
widgetMessageCallback?: WidgetMessageCallback | null,
) {
this.stageStore = stageStore;
this.stageAPI = createStageAPI(stageStore);
this.audioPlayer = audioPlayer ?? null;
this.widgetMessageCallback = widgetMessageCallback ?? null;
}
/** Set callback for sending messages to widget iframe */
setWidgetMessageCallback(callback: WidgetMessageCallback | null): void {
this.widgetMessageCallback = callback;
}
/** Clean up timers when the engine is no longer needed */
dispose(): void {
if (this.effectTimer) {
clearTimeout(this.effectTimer);
this.effectTimer = null;
}
}
/**
* Execute a single action.
* Fire-and-forget actions return immediately.
* Synchronous actions return a Promise that resolves when the action is complete.
*/
async execute(action: Action, options: ActionExecutionOptions = {}): Promise<void> {
if (options.silent) {
if (action.type === 'speech' || action.type === 'spotlight' || action.type === 'laser') {
return;
}
if (action.type === 'discussion' || action.type === 'play_video') {
return;
}
if (action.type.startsWith('widget_')) {
return;
}
}
// Auto-open whiteboard if a draw/clear/delete action is attempted while it's closed
if (action.type.startsWith('wb_') && action.type !== 'wb_open' && action.type !== 'wb_close') {
await this.ensureWhiteboardOpen(options);
}
switch (action.type) {
// Fire-and-forget
case 'spotlight':
this.executeSpotlight(action);
return;
case 'laser':
this.executeLaser(action);
return;
// Synchronous — Video
case 'play_video':
return this.executePlayVideo(action as PlayVideoAction, options);
// Synchronous
case 'speech':
return this.executeSpeech(action);
case 'wb_open':
return this.executeWbOpen(options);
case 'wb_draw_text':
return this.executeWbDrawText(action, options);
case 'wb_draw_shape':
return this.executeWbDrawShape(action, options);
case 'wb_draw_chart':
return this.executeWbDrawChart(action, options);
case 'wb_draw_latex':
return this.executeWbDrawLatex(action, options);
case 'wb_draw_table':
return this.executeWbDrawTable(action, options);
case 'wb_draw_line':
return this.executeWbDrawLine(action as WbDrawLineAction, options);
case 'wb_draw_code':
return this.executeWbDrawCode(action as WbDrawCodeAction, options);
case 'wb_edit_code':
return this.executeWbEditCode(action as WbEditCodeAction, options);
case 'wb_clear':
return this.executeWbClear(options);
case 'wb_delete':
return this.executeWbDelete(action as WbDeleteAction, options);
case 'wb_close':
return this.executeWbClose(options);
case 'discussion':
// Discussion lifecycle is managed externally via engine callbacks
return;
// Widget actions — post message to iframe
case 'widget_highlight':
return this.executeWidgetHighlight(action as WidgetHighlightAction);
case 'widget_setState':
return this.executeWidgetSetState(action as WidgetSetStateAction);
case 'widget_annotation':
return this.executeWidgetAnnotation(action as WidgetAnnotationAction);
case 'widget_reveal':
return this.executeWidgetReveal(action as WidgetRevealAction);
}
}
/** Clear all active visual effects */
clearEffects(): void {
if (this.effectTimer) {
clearTimeout(this.effectTimer);
this.effectTimer = null;
}
useCanvasStore.getState().clearAllEffects();
}
resetPlaybackVisualState(): void {
this.clearEffects();
useCanvasStore.getState().pauseVideo();
useCanvasStore.getState().setWhiteboardOpen(false);
useCanvasStore.getState().setWhiteboardClearing(false);
const wb = this.stageAPI.whiteboard.get();
if (wb.success && wb.data) {
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
}
}
/** Schedule auto-clear for fire-and-forget effects */
private scheduleEffectClear(): void {
if (this.effectTimer) {
clearTimeout(this.effectTimer);
}
this.effectTimer = setTimeout(() => {
useCanvasStore.getState().clearAllEffects();
this.effectTimer = null;
}, EFFECT_AUTO_CLEAR_MS);
}
// ==================== Fire-and-forget ====================
private executeSpotlight(action: SpotlightAction): void {
useCanvasStore.getState().setSpotlight(action.elementId, {
dimness: action.dimOpacity ?? 0.5,
});
this.scheduleEffectClear();
}
private executeLaser(action: LaserAction): void {
useCanvasStore.getState().setLaser(action.elementId, {
color: action.color ?? '#ff0000',
});
this.scheduleEffectClear();
}
// ==================== Synchronous — Speech ====================
private async executeSpeech(action: SpeechAction): Promise<void> {
if (!this.audioPlayer) return;
return new Promise<void>((resolve) => {
this.audioPlayer!.onEnded(() => resolve());
// The legacy URL of an unconverted pair rides along as the fallback of
// last resort; converted documents carry no audioUrl.
this.audioPlayer!.play(action.audioId || '', (action as LegacySpeechAction).audioUrl)
.then((audioStarted) => {
if (!audioStarted) resolve();
})
.catch(() => resolve());
});
}
// ==================== Synchronous — Video ====================
private async executePlayVideo(
action: PlayVideoAction,
options: ActionExecutionOptions = {},
): Promise<void> {
if (options.signal?.aborted) return;
const resolveBinding = () =>
resolveActionVideoMedia(
this.stageStore,
useMediaGenerationStore.getState().tasks,
action.elementId,
);
const binding = resolveBinding();
if (binding?.task) {
// Wait while the task is not yet playable: pending/generating, or a
// deferred restore that is done but still byte-less.
if (!isPlayableVideoTask(binding.task) && binding.task.status !== 'failed') {
// Wait for media to be playable (or fail)
await new Promise<void>((resolve) => {
let unsubscribe = () => {};
const finish = () => {
unsubscribe();
options.signal?.removeEventListener('abort', finish);
resolve();
};
unsubscribe = useMediaGenerationStore.subscribe((state) => {
const t = resolveActionVideoMedia(this.stageStore, state.tasks, action.elementId)?.task;
if (!t || isPlayableVideoTask(t) || t.status === 'failed') {
finish();
}
});
options.signal?.addEventListener('abort', finish, { once: true });
// Check again in case it resolved between getState and subscribe
const current = resolveBinding()?.task;
if (!current || isPlayableVideoTask(current) || current.status === 'failed') {
finish();
}
});
if (options.signal?.aborted) return;
}
// If failed, skip playback
if (resolveBinding()?.task?.status === 'failed') {
return;
}
}
if (options.signal?.aborted) return;
useCanvasStore.getState().playVideo(action.elementId);
// Wait until the video finishes playing, with a safety timeout to prevent
// the playback engine from hanging indefinitely if the video element is
// invalid or the state change is missed.
return new Promise<void>((resolve) => {
let finished = false;
let unsubscribe = () => {};
const finish = () => {
if (finished) return;
finished = true;
clearTimeout(timeout);
unsubscribe();
options.signal?.removeEventListener('abort', abortPlayback);
resolve();
};
const abortPlayback = () => {
if (useCanvasStore.getState().playingVideoElementId === action.elementId) {
useCanvasStore.getState().pauseVideo();
}
finish();
};
const timeout = setTimeout(() => {
log.warn(`[playVideo] Timeout waiting for video ${action.elementId} to finish`);
finish();
}, MAX_VIDEO_WAIT_MS);
unsubscribe = useCanvasStore.subscribe((state) => {
if (state.playingVideoElementId !== action.elementId) {
finish();
}
});
options.signal?.addEventListener('abort', abortPlayback, { once: true });
if (useCanvasStore.getState().playingVideoElementId !== action.elementId) {
finish();
}
});
}
// ==================== Synchronous — Whiteboard ====================
/** Auto-open the whiteboard if it's not already open */
private async ensureWhiteboardOpen(options: ActionExecutionOptions = {}): Promise<void> {
if (!useCanvasStore.getState().whiteboardOpen) {
await this.executeWbOpen(options);
}
}
private async executeWbOpen(options: ActionExecutionOptions = {}): Promise<void> {
// Ensure a whiteboard exists
this.stageAPI.whiteboard.get();
useCanvasStore.getState().setWhiteboardOpen(true);
if (options.silent) return;
// Wait for open animation to complete (slow spring: stiffness 120, damping 18, mass 1.2)
await delay(WB_OPEN_MS);
}
private async executeWbDrawText(
action: WbDrawTextAction,
options: ActionExecutionOptions = {},
): Promise<void> {
let htmlContent = action.content ?? '';
if (!htmlContent) return; // nothing to draw
const latex = getLikelyLatexMath(htmlContent);
if (latex !== null) {
return this.executeWbDrawLatex(
{
...action,
type: 'wb_draw_latex',
latex,
},
options,
);
}
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
const fontSize = action.fontSize ?? 18;
if (!htmlContent.startsWith('<')) {
htmlContent = `<p style="font-size: ${fontSize}px;">${htmlContent}</p>`;
}
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'text',
content: htmlContent,
left: action.x,
top: action.y,
width: action.width ?? 400,
height: action.height ?? 100,
rotate: 0,
defaultFontName: 'Microsoft YaHei',
defaultColor: action.color ?? '#333333',
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) {
// Wait for element fade-in animation
await delay(WB_DRAW_MS);
}
}
private async executeWbDrawShape(
action: WbDrawShapeAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'shape',
viewBox: [1000, 1000] as [number, number],
path: SHAPE_PATHS[action.shape] ?? SHAPE_PATHS.rectangle,
left: action.x,
top: action.y,
width: action.width,
height: action.height,
rotate: 0,
fill: action.fillColor ?? '#5b9bd5',
fixedRatio: false,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) {
// Wait for element fade-in animation
await delay(WB_DRAW_MS);
}
}
private async executeWbDrawChart(
action: WbDrawChartAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'chart',
left: action.x,
top: action.y,
width: action.width,
height: action.height,
rotate: 0,
chartType: action.chartType,
data: action.data,
themeColors: action.themeColors ?? ['#5b9bd5', '#ed7d31', '#a5a5a5', '#ffc000', '#4472c4'],
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) await delay(WB_DRAW_MS);
}
private async executeWbDrawLatex(
action: WbDrawLatexAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
try {
const html = katex.renderToString(action.latex, {
throwOnError: false,
displayMode: true,
output: 'html',
});
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'latex',
left: action.x,
top: action.y,
width: action.width ?? 400,
height: action.height ?? 80,
rotate: 0,
latex: action.latex,
html,
color: action.color ?? '#000000',
fixedRatio: true,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
} catch (err) {
log.warn(`Failed to render latex "${action.latex}":`, err);
return;
}
if (!options.silent) await delay(WB_DRAW_MS);
}
private async executeWbDrawTable(
action: WbDrawTableAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
const rows = action.data.length;
const cols = rows > 0 ? action.data[0].length : 0;
if (rows === 0 || cols === 0) return;
// Build colWidths: equal distribution
const colWidths = Array(cols).fill(1 / cols);
// Build TableCell[][] from string[][]
let cellId = 0;
const tableData = action.data.map((row) =>
row.map((text) => ({
id: `cell_${cellId++}`,
colspan: 1,
rowspan: 1,
text,
})),
);
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'table',
left: action.x,
top: action.y,
width: action.width,
height: action.height,
rotate: 0,
colWidths,
cellMinHeight: 36,
data: tableData,
outline: action.outline ?? {
width: 2,
style: 'solid',
color: '#eeece1',
},
theme: action.theme
? {
color: action.theme.color,
rowHeader: true,
rowFooter: false,
colHeader: false,
colFooter: false,
}
: undefined,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) await delay(WB_DRAW_MS);
}
private async executeWbDrawLine(
action: WbDrawLineAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
// Calculate bounding box — left/top is the minimum of start/end coordinates
const left = Math.min(action.startX, action.endX);
const top = Math.min(action.startY, action.endY);
// Convert absolute coordinates to relative coordinates (relative to left/top)
const start: [number, number] = [action.startX - left, action.startY - top];
const end: [number, number] = [action.endX - left, action.endY - top];
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'line',
left,
top,
width: action.width ?? 2,
start,
end,
style: action.style ?? 'solid',
color: action.color ?? '#333333',
points: action.points ?? ['', ''],
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) {
// Wait for element fade-in animation
await delay(WB_DRAW_MS);
}
}
private async executeWbDrawCode(
action: WbDrawCodeAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
const lines = codeToLines(action.code);
const suppliedLineIds = (action as WbDrawCodeAction & { lineIds?: string[] }).lineIds;
if (suppliedLineIds?.length === lines.length) {
lines.forEach((line, index) => {
line.id = suppliedLineIds[index];
});
}
this.stageAPI.whiteboard.addElement(
{
id: action.elementId || '',
type: 'code',
language: action.language,
lines,
fileName: action.fileName,
showLineNumbers: true,
fontSize: 14,
left: action.x,
top: action.y,
width: action.width ?? 500,
height: action.height ?? 300,
rotate: 0,
// eslint-disable-next-line @typescript-eslint/no-explicit-any
} as any,
wb.data.id,
);
if (!options.silent) {
// Wait for typing animation (base 800ms + 50ms/line, capped at 3s)
const animMs = wbDrawCodeMs(lines.length);
await delay(animMs);
}
}
private async executeWbEditCode(
action: WbEditCodeAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
const elementResult = this.stageAPI.whiteboard.getElement(action.elementId, wb.data.id);
if (!elementResult.success || !elementResult.data) return;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const element = elementResult.data as any;
if (element.type !== 'code') return;
let lines: CodeLine[] = [...element.lines];
const newContentLines = action.content ? action.content.split('\n') : [];
const suppliedLineIds = (action as WbEditCodeAction & { newLineIds?: string[] }).newLineIds;
const newLineIds =
suppliedLineIds?.length === newContentLines.length
? suppliedLineIds
: generateLineIds(newContentLines.length);
switch (action.operation) {
case 'insert_after': {
const idx = lines.findIndex((l) => l.id === action.lineId);
if (idx === -1) return;
const newLines = newContentLines.map((content, i) => ({ id: newLineIds[i], content }));
lines.splice(idx + 1, 0, ...newLines);
break;
}
case 'insert_before': {
const idx = lines.findIndex((l) => l.id === action.lineId);
if (idx === -1) return;
const newLines = newContentLines.map((content, i) => ({ id: newLineIds[i], content }));
lines.splice(idx, 0, ...newLines);
break;
}
case 'delete_lines': {
if (!action.lineIds?.length) return;
const deleteSet = new Set(action.lineIds);
lines = lines.filter((l) => !deleteSet.has(l.id));
break;
}
case 'replace_lines': {
if (!action.lineIds?.length) return;
const replaceIds = action.lineIds;
const firstIdx = lines.findIndex((l) => l.id === replaceIds[0]);
if (firstIdx === -1) return;
const deleteSet = new Set(replaceIds);
lines = lines.filter((l) => !deleteSet.has(l.id));
const newLines = newContentLines.map((content, i) => ({
id: i < replaceIds.length ? replaceIds[i] : newLineIds[i],
content,
}));
lines.splice(firstIdx, 0, ...newLines);
break;
}
}
this.stageAPI.whiteboard.updateElement(
// eslint-disable-next-line @typescript-eslint/no-explicit-any
{ ...element, lines } as any,
wb.data.id,
);
if (!options.silent) {
// Wait for edit animation
await delay(WB_EDIT_MS);
}
}
private async executeWbDelete(
action: WbDeleteAction,
options: ActionExecutionOptions = {},
): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
this.stageAPI.whiteboard.deleteElement(action.elementId, wb.data.id);
if (!options.silent) await delay(WB_DELETE_MS);
}
private async executeWbClear(options: ActionExecutionOptions = {}): Promise<void> {
const wb = this.stageAPI.whiteboard.get();
if (!wb.success || !wb.data) return;
const elementCount = wb.data.elements?.length || 0;
if (elementCount === 0) return;
if (options.silent) {
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
useCanvasStore.getState().setWhiteboardClearing(false);
return;
}
// Save snapshot before AI clear (mirrors UI handleClear in index.tsx)
useWhiteboardHistoryStore.getState().pushSnapshot(wb.data.elements!);
// Trigger cascade exit animation
useCanvasStore.getState().setWhiteboardClearing(true);
// Wait for cascade (base 380ms + 55ms/element, capped at 1400ms)
const animMs = wbClearMs(elementCount);
await delay(animMs);
// Actually remove elements
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
useCanvasStore.getState().setWhiteboardClearing(false);
}
private async executeWbClose(options: ActionExecutionOptions = {}): Promise<void> {
useCanvasStore.getState().setWhiteboardOpen(false);
if (options.silent) return;
// Wait for close animation (500ms ease-out tween)
await delay(WB_CLOSE_MS);
}
// ==================== Widget Actions ====================
/** Send message to widget iframe */
private sendWidgetMessage(type: string, payload: Record<string, unknown>): void {
if (this.widgetMessageCallback) {
this.widgetMessageCallback(type, payload);
} else {
log.warn(`Widget message callback not set, cannot send: ${type}`);
}
}
/** Execute widget highlight action (quick visual change) */
private async executeWidgetHighlight(action: WidgetHighlightAction): Promise<void> {
this.sendWidgetMessage('HIGHLIGHT_ELEMENT', {
target: action.target,
content: action.content,
});
// Quick delay for visual effect
await delay(WIDGET_MS);
}
/** Execute widget setState action */
private async executeWidgetSetState(action: WidgetSetStateAction): Promise<void> {
this.sendWidgetMessage('SET_WIDGET_STATE', { state: action.state, content: action.content });
// Quick delay for state change to propagate
await delay(WIDGET_MS);
}
/** Execute widget annotation action */
private async executeWidgetAnnotation(action: WidgetAnnotationAction): Promise<void> {
this.sendWidgetMessage('ANNOTATE_ELEMENT', {
target: action.target,
content: action.content,
});
await delay(WIDGET_MS);
}
/** Execute widget reveal action */
private async executeWidgetReveal(action: WidgetRevealAction): Promise<void> {
this.sendWidgetMessage('REVEAL_ELEMENT', { target: action.target, content: action.content });
await delay(WIDGET_MS);
}
}