* perf(classroom): defer non-priority media blob hydration off the load path Entering a classroom awaited the full mediaFiles restore — one object URL per restored image/video blob — before the loading gate opened. For video-heavy courses that is hundreds of MB of IndexedDB materialization blocking first paint. Split the restore into two phases: - The awaited phase now builds metadata-complete task entries but only creates object URLs for failed rows (none needed) and for media referenced by the scene the classroom opens on (persisted cursor, else the first scene). buildRestoredMediaTasks gains an optional shouldHydrateBlob predicate, defaulting to eager hydration so existing callers are unchanged. - Remaining blob-backed records hydrate in the background, chunked over requestIdleCallback (setTimeout fallback). Each deferred task stays 'done' — so generation resume never re-runs it — but carries no objectUrl yet, which the media resolution state machine already renders as a pending skeleton until the URL lands. Background hydration is guarded per record: a task that was replaced (classroom switch, regeneration, retry) or that belongs to another stage is skipped and its freshly minted URLs are revoked immediately. * perf(classroom): lazy-render slide thumbnails in the playback sidebar The playback scene sidebar mounted a full SlideCanvas for every slide scene the moment the classroom opened (and every off-screen video element opened a preload="metadata" fetch), because SlideThumbnail's existing visible prop was never passed. Extract the editor nav rail's near-viewport IntersectionObserver hook to lib/hooks/use-near-viewport.ts and gate the sidebar's slide thumbnails through it: only scenes within 200px of the viewport render the live canvas; the rest show SlideThumbnail's existing placeholder until scrolled near. The placeholder keeps the same box size, so gating never shifts layout. The shared hook now starts hidden instead of eager: with the previous eager-initial state, opening a long deck still mounted every canvas for a frame before the observer could flip off-screen items off. The observer's guaranteed initial delivery flips near-viewport items within a frame; the no-IntersectionObserver fallback (e.g. jsdom) defers to a microtask so the effect never synchronously re-renders. * perf(media): lazy image decoding and HTTP Range support for classroom media Renderer: BaseImageElement's <img> now carries loading="lazy" and decoding="async", so thumbnail-heavy surfaces (playback sidebar, editor nav rail, course cards) no longer fetch and decode every slide image up front. In-viewport images are unaffected — the browser fetches them immediately. slideToPng forces eager loading inside its permanently off-screen snapshot tree, where lazy images would otherwise never fetch and exports would capture blank slides. Server: GET /api/classroom-media/[classroomId]/[...path] now answers single byte-range requests with 206 Partial Content (Content-Range, Accept-Ranges, correct Content-Length), enabling progressive playback and seeking for hosted video/audio instead of downloading whole files. Suffix ranges are supported; unsatisfiable ranges get 416 with the full size; unsupported units or multi-range sets fall back to the plain 200 full-body response, which is always a legal answer. Existing caching headers are kept on every response shape. * chore(renderer): bump version to 0.1.4 for image lazy-loading change * fix(classroom): guard deferred hydration against restarted tasks and bound idle waits A deferred record is only ever 'done' without an objectUrl; a task that regeneration or retry restarted passes through pending/generating, so skip those instead of attaching stale persisted bytes. Also give the idle scheduling a timeout so a busy main thread cannot starve hydration. * fix(classroom): stop superseded deferred hydration before minting URLs A restore epoch captured at apply time now gates each idle chunk: loading another classroom or reloading the same one invalidates any older hydration loop, so it neither keeps scheduling work for an abandoned classroom nor attaches bytes read by an older load to a newer load's tasks. * fix(classroom): keep 416s uncached and classify element-keyed media as priority A 416 with public immutable caching can poison the media URL for later valid requests, so range errors now send Cache-Control: no-store. Priority classification also collects the opening scene's media element ids, since task lookup binds records keyed stage:<elementId> even when the slide slot carries a different opaque ref. * fix(classroom): tie deferred hydration liveness to the classroom load token The restore epoch only advanced when the next load reached apply, so an abandoned classroom kept hydrating during the next load's storage/network phase. Compose the epoch with the load's isCurrent (load token plus effect cleanup) so navigation stops the loop at the next idle boundary. * fix(classroom): classify legacy-recovered media before deferring it Legacy singleton video recovery assigns a placeholderRef only at the end of the task build, so a record keyed by an allocated id without placeholderRef was deferred even when it backs the opening scene's gen_vid_* element. Run a metadata-only build first to learn each record's effective ref and classify against it, keeping the first visible page's legacy video eager. * fix(action): wait for deferred video bytes before starting play_video executePlayVideo treated status done as immediately playable, but a deferred restore is done without an objectUrl: the renderer shows a skeleton, no <video> exists, and the later hydration never retriggers play, leaving the action stuck until the safety timeout. Readiness now follows the renderer's contract (only done-with-bytes is playable) at the initial check, the subscription exit, and the post-subscription recheck; the failed skip applies whether or not a wait happened. * fix(classroom): include the stage whiteboard in priority media refs The stage-level whiteboard stays open across standalone classroom switches, so its media can be visible before any scene is. Classifying it as deferred left visible whiteboard media pending behind idle hydration chunks.
903 lines
29 KiB
TypeScript
903 lines
29 KiB
TypeScript
/**
|
|
* ActionEngine — Unified execution layer for all agent actions.
|
|
*
|
|
* Replaces the 28 Vercel AI SDK tools in ai-tools.ts with a single engine
|
|
* that both online (streaming) and offline (playback) paths share.
|
|
*
|
|
* Two execution modes:
|
|
* - Fire-and-forget: spotlight, laser — dispatch and return immediately
|
|
* - Synchronous: speech, whiteboard, discussion — await completion
|
|
*/
|
|
|
|
import type { StageStore } from '@/lib/api/stage-api';
|
|
import { createStageAPI } from '@/lib/api/stage-api';
|
|
import { useCanvasStore } from '@/lib/store/canvas';
|
|
import { useWhiteboardHistoryStore } from '@/lib/store/whiteboard-history';
|
|
import { useMediaGenerationStore, type MediaTask } from '@/lib/store/media-generation';
|
|
import type { AudioPlayer } from '@/lib/utils/audio-player';
|
|
import type { LegacySpeechAction } from '@/lib/types/action';
|
|
import type {
|
|
Action,
|
|
SpotlightAction,
|
|
LaserAction,
|
|
SpeechAction,
|
|
PlayVideoAction,
|
|
WbDrawTextAction,
|
|
WbDrawShapeAction,
|
|
WbDrawChartAction,
|
|
WbDrawLatexAction,
|
|
WbDrawTableAction,
|
|
WbDeleteAction,
|
|
WbDrawLineAction,
|
|
WbDrawCodeAction,
|
|
WbEditCodeAction,
|
|
WidgetHighlightAction,
|
|
WidgetSetStateAction,
|
|
WidgetAnnotationAction,
|
|
WidgetRevealAction,
|
|
} from '@/lib/types/action';
|
|
import type { CodeLine, PPTVideoElement } from '@openmaic/dsl';
|
|
import {
|
|
resolveVideoMediaForElement,
|
|
type VideoMediaTaskResolution,
|
|
} from '@/lib/media/media-task-resolution';
|
|
import {
|
|
EFFECT_AUTO_CLEAR_MS,
|
|
MAX_VIDEO_WAIT_MS,
|
|
WB_OPEN_MS,
|
|
WB_DRAW_MS,
|
|
WB_EDIT_MS,
|
|
WB_DELETE_MS,
|
|
WB_CLOSE_MS,
|
|
WIDGET_MS,
|
|
wbDrawCodeMs,
|
|
wbClearMs,
|
|
} from '@/lib/choreography';
|
|
import katex from 'katex';
|
|
import { createLogger } from '@/lib/logger';
|
|
|
|
const log = createLogger('ActionEngine');
|
|
|
|
// ==================== SVG Paths for Shapes ====================
|
|
|
|
const SHAPE_PATHS: Record<string, string> = {
|
|
rectangle: 'M 0 0 L 1000 0 L 1000 1000 L 0 1000 Z',
|
|
circle: 'M 500 0 A 500 500 0 1 1 499 0 Z',
|
|
triangle: 'M 500 0 L 1000 1000 L 0 1000 Z',
|
|
};
|
|
|
|
// ==================== Helpers ====================
|
|
|
|
function delay(ms: number): Promise<void> {
|
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
}
|
|
|
|
const COMMON_LATEX_COMMAND =
|
|
/\\(?:alpha|beta|cdot|delta|dfrac|frac|gamma|infty|int|lambda|left|lim|mu|neq|omega|pi|pm|prod|rightarrow|right|sigma|sqrt|sum|text|tfrac|theta|times)\b/;
|
|
|
|
function getDelimitedLatex(content: string): string | null {
|
|
const trimmed = content.trim();
|
|
|
|
if (trimmed.length > 4 && trimmed.startsWith('$$') && trimmed.endsWith('$$')) {
|
|
return trimmed.slice(2, -2).trim();
|
|
}
|
|
if (
|
|
trimmed.length > 2 &&
|
|
trimmed.startsWith('$') &&
|
|
trimmed.endsWith('$') &&
|
|
!trimmed.startsWith('$$') &&
|
|
!trimmed.endsWith('$$')
|
|
) {
|
|
return trimmed.slice(1, -1).trim();
|
|
}
|
|
|
|
return null;
|
|
}
|
|
|
|
function getLikelyLatexMath(content: string): string | null {
|
|
const trimmed = content.trim();
|
|
if (!trimmed || trimmed.startsWith('<')) return null;
|
|
|
|
const delimitedLatex = getDelimitedLatex(trimmed);
|
|
if (delimitedLatex !== null) return delimitedLatex;
|
|
if (/^[A-Za-z]:\\/.test(trimmed)) return null;
|
|
if (COMMON_LATEX_COMMAND.test(trimmed) || /[_^]\{/.test(trimmed)) return trimmed;
|
|
|
|
const commands = trimmed.match(/\\[A-Za-z]+/g) ?? [];
|
|
if (commands.length === 0) return null;
|
|
if (commands.length === 1) {
|
|
return /\\[A-Za-z]+\s*\{[^{}]*\}/.test(trimmed) ? trimmed : null;
|
|
}
|
|
if (!/[=+\-*/^_{}]/.test(trimmed)) return null;
|
|
|
|
const commandCharacters = commands.reduce((total, command) => total + command.length, 0);
|
|
return commandCharacters / trimmed.length >= 0.15 ? trimmed : null;
|
|
}
|
|
|
|
/** Convert raw code string to CodeLine array with unique IDs */
|
|
function codeToLines(code: string): CodeLine[] {
|
|
return code.split('\n').map((content, i) => ({
|
|
id: `L${i + 1}`,
|
|
content,
|
|
}));
|
|
}
|
|
|
|
let lineIdCounter = 0;
|
|
/** Generate unique line IDs for newly inserted lines */
|
|
function generateLineIds(count: number): string[] {
|
|
return Array.from({ length: count }, () => `L_${++lineIdCounter}_${Date.now().toString(36)}`);
|
|
}
|
|
|
|
/** Resolve the video element and its renderer-equivalent media binding for an action. */
|
|
export function resolveActionVideoMedia(
|
|
stageStore: StageStore,
|
|
tasks: Readonly<Record<string, MediaTask>>,
|
|
elementId: string,
|
|
): VideoMediaTaskResolution<MediaTask> | undefined {
|
|
const { stage, scenes, currentSceneId } = stageStore.getState();
|
|
const orderedScenes = currentSceneId
|
|
? [
|
|
scenes.find((scene) => scene.id === currentSceneId),
|
|
...scenes.filter((scene) => scene.id !== currentSceneId),
|
|
]
|
|
: scenes;
|
|
|
|
for (const scene of orderedScenes) {
|
|
if (!scene || scene.content.type !== 'slide') continue;
|
|
const element = scene.content.canvas.elements.find(
|
|
(candidate): candidate is PPTVideoElement =>
|
|
candidate.id === elementId && candidate.type === 'video',
|
|
);
|
|
if (element) return resolveVideoMediaForElement(tasks, element, stage?.id);
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
/**
|
|
* Renderer-aligned playability: a task is playable only once its bytes exist
|
|
* (`done` + objectUrl). A deferred restore is `done` without an objectUrl and
|
|
* the renderer still treats it as pending (`resolveMediaRef` maps it to
|
|
* 'pending'), so starting playback in that state would set
|
|
* playingVideoElementId while no <video> exists — and the later hydration
|
|
* would not retrigger play, leaving the action stuck until its timeout.
|
|
*/
|
|
function isPlayableVideoTask(task: MediaTask): boolean {
|
|
return task.status === 'done' && !!task.objectUrl;
|
|
}
|
|
|
|
// ==================== ActionEngine ====================
|
|
|
|
/** Callback for sending messages to widget iframe */
|
|
export type WidgetMessageCallback = (type: string, payload: Record<string, unknown>) => void;
|
|
|
|
export interface ActionExecutionOptions {
|
|
silent?: boolean;
|
|
signal?: AbortSignal;
|
|
}
|
|
|
|
export class ActionEngine {
|
|
private stageStore: StageStore;
|
|
private stageAPI: ReturnType<typeof createStageAPI>;
|
|
private audioPlayer: AudioPlayer | null;
|
|
private effectTimer: ReturnType<typeof setTimeout> | null = null;
|
|
private widgetMessageCallback: WidgetMessageCallback | null = null;
|
|
|
|
constructor(
|
|
stageStore: StageStore,
|
|
audioPlayer?: AudioPlayer | null,
|
|
widgetMessageCallback?: WidgetMessageCallback | null,
|
|
) {
|
|
this.stageStore = stageStore;
|
|
this.stageAPI = createStageAPI(stageStore);
|
|
this.audioPlayer = audioPlayer ?? null;
|
|
this.widgetMessageCallback = widgetMessageCallback ?? null;
|
|
}
|
|
|
|
/** Set callback for sending messages to widget iframe */
|
|
setWidgetMessageCallback(callback: WidgetMessageCallback | null): void {
|
|
this.widgetMessageCallback = callback;
|
|
}
|
|
|
|
/** Clean up timers when the engine is no longer needed */
|
|
dispose(): void {
|
|
if (this.effectTimer) {
|
|
clearTimeout(this.effectTimer);
|
|
this.effectTimer = null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Execute a single action.
|
|
* Fire-and-forget actions return immediately.
|
|
* Synchronous actions return a Promise that resolves when the action is complete.
|
|
*/
|
|
async execute(action: Action, options: ActionExecutionOptions = {}): Promise<void> {
|
|
if (options.silent) {
|
|
if (action.type === 'speech' || action.type === 'spotlight' || action.type === 'laser') {
|
|
return;
|
|
}
|
|
if (action.type === 'discussion' || action.type === 'play_video') {
|
|
return;
|
|
}
|
|
if (action.type.startsWith('widget_')) {
|
|
return;
|
|
}
|
|
}
|
|
|
|
// Auto-open whiteboard if a draw/clear/delete action is attempted while it's closed
|
|
if (action.type.startsWith('wb_') && action.type !== 'wb_open' && action.type !== 'wb_close') {
|
|
await this.ensureWhiteboardOpen(options);
|
|
}
|
|
|
|
switch (action.type) {
|
|
// Fire-and-forget
|
|
case 'spotlight':
|
|
this.executeSpotlight(action);
|
|
return;
|
|
case 'laser':
|
|
this.executeLaser(action);
|
|
return;
|
|
// Synchronous — Video
|
|
case 'play_video':
|
|
return this.executePlayVideo(action as PlayVideoAction, options);
|
|
|
|
// Synchronous
|
|
case 'speech':
|
|
return this.executeSpeech(action);
|
|
case 'wb_open':
|
|
return this.executeWbOpen(options);
|
|
case 'wb_draw_text':
|
|
return this.executeWbDrawText(action, options);
|
|
case 'wb_draw_shape':
|
|
return this.executeWbDrawShape(action, options);
|
|
case 'wb_draw_chart':
|
|
return this.executeWbDrawChart(action, options);
|
|
case 'wb_draw_latex':
|
|
return this.executeWbDrawLatex(action, options);
|
|
case 'wb_draw_table':
|
|
return this.executeWbDrawTable(action, options);
|
|
case 'wb_draw_line':
|
|
return this.executeWbDrawLine(action as WbDrawLineAction, options);
|
|
case 'wb_draw_code':
|
|
return this.executeWbDrawCode(action as WbDrawCodeAction, options);
|
|
case 'wb_edit_code':
|
|
return this.executeWbEditCode(action as WbEditCodeAction, options);
|
|
case 'wb_clear':
|
|
return this.executeWbClear(options);
|
|
case 'wb_delete':
|
|
return this.executeWbDelete(action as WbDeleteAction, options);
|
|
case 'wb_close':
|
|
return this.executeWbClose(options);
|
|
case 'discussion':
|
|
// Discussion lifecycle is managed externally via engine callbacks
|
|
return;
|
|
|
|
// Widget actions — post message to iframe
|
|
case 'widget_highlight':
|
|
return this.executeWidgetHighlight(action as WidgetHighlightAction);
|
|
case 'widget_setState':
|
|
return this.executeWidgetSetState(action as WidgetSetStateAction);
|
|
case 'widget_annotation':
|
|
return this.executeWidgetAnnotation(action as WidgetAnnotationAction);
|
|
case 'widget_reveal':
|
|
return this.executeWidgetReveal(action as WidgetRevealAction);
|
|
}
|
|
}
|
|
|
|
/** Clear all active visual effects */
|
|
clearEffects(): void {
|
|
if (this.effectTimer) {
|
|
clearTimeout(this.effectTimer);
|
|
this.effectTimer = null;
|
|
}
|
|
useCanvasStore.getState().clearAllEffects();
|
|
}
|
|
|
|
resetPlaybackVisualState(): void {
|
|
this.clearEffects();
|
|
useCanvasStore.getState().pauseVideo();
|
|
useCanvasStore.getState().setWhiteboardOpen(false);
|
|
useCanvasStore.getState().setWhiteboardClearing(false);
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (wb.success && wb.data) {
|
|
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
|
|
}
|
|
}
|
|
|
|
/** Schedule auto-clear for fire-and-forget effects */
|
|
private scheduleEffectClear(): void {
|
|
if (this.effectTimer) {
|
|
clearTimeout(this.effectTimer);
|
|
}
|
|
this.effectTimer = setTimeout(() => {
|
|
useCanvasStore.getState().clearAllEffects();
|
|
this.effectTimer = null;
|
|
}, EFFECT_AUTO_CLEAR_MS);
|
|
}
|
|
|
|
// ==================== Fire-and-forget ====================
|
|
|
|
private executeSpotlight(action: SpotlightAction): void {
|
|
useCanvasStore.getState().setSpotlight(action.elementId, {
|
|
dimness: action.dimOpacity ?? 0.5,
|
|
});
|
|
this.scheduleEffectClear();
|
|
}
|
|
|
|
private executeLaser(action: LaserAction): void {
|
|
useCanvasStore.getState().setLaser(action.elementId, {
|
|
color: action.color ?? '#ff0000',
|
|
});
|
|
this.scheduleEffectClear();
|
|
}
|
|
|
|
// ==================== Synchronous — Speech ====================
|
|
|
|
private async executeSpeech(action: SpeechAction): Promise<void> {
|
|
if (!this.audioPlayer) return;
|
|
|
|
return new Promise<void>((resolve) => {
|
|
this.audioPlayer!.onEnded(() => resolve());
|
|
// The legacy URL of an unconverted pair rides along as the fallback of
|
|
// last resort; converted documents carry no audioUrl.
|
|
this.audioPlayer!.play(action.audioId || '', (action as LegacySpeechAction).audioUrl)
|
|
.then((audioStarted) => {
|
|
if (!audioStarted) resolve();
|
|
})
|
|
.catch(() => resolve());
|
|
});
|
|
}
|
|
|
|
// ==================== Synchronous — Video ====================
|
|
|
|
private async executePlayVideo(
|
|
action: PlayVideoAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
if (options.signal?.aborted) return;
|
|
const resolveBinding = () =>
|
|
resolveActionVideoMedia(
|
|
this.stageStore,
|
|
useMediaGenerationStore.getState().tasks,
|
|
action.elementId,
|
|
);
|
|
const binding = resolveBinding();
|
|
|
|
if (binding?.task) {
|
|
// Wait while the task is not yet playable: pending/generating, or a
|
|
// deferred restore that is done but still byte-less.
|
|
if (!isPlayableVideoTask(binding.task) && binding.task.status !== 'failed') {
|
|
// Wait for media to be playable (or fail)
|
|
await new Promise<void>((resolve) => {
|
|
let unsubscribe = () => {};
|
|
const finish = () => {
|
|
unsubscribe();
|
|
options.signal?.removeEventListener('abort', finish);
|
|
resolve();
|
|
};
|
|
unsubscribe = useMediaGenerationStore.subscribe((state) => {
|
|
const t = resolveActionVideoMedia(this.stageStore, state.tasks, action.elementId)?.task;
|
|
if (!t || isPlayableVideoTask(t) || t.status === 'failed') {
|
|
finish();
|
|
}
|
|
});
|
|
options.signal?.addEventListener('abort', finish, { once: true });
|
|
// Check again in case it resolved between getState and subscribe
|
|
const current = resolveBinding()?.task;
|
|
if (!current || isPlayableVideoTask(current) || current.status === 'failed') {
|
|
finish();
|
|
}
|
|
});
|
|
|
|
if (options.signal?.aborted) return;
|
|
}
|
|
|
|
// If failed, skip playback
|
|
if (resolveBinding()?.task?.status === 'failed') {
|
|
return;
|
|
}
|
|
}
|
|
|
|
if (options.signal?.aborted) return;
|
|
useCanvasStore.getState().playVideo(action.elementId);
|
|
|
|
// Wait until the video finishes playing, with a safety timeout to prevent
|
|
// the playback engine from hanging indefinitely if the video element is
|
|
// invalid or the state change is missed.
|
|
return new Promise<void>((resolve) => {
|
|
let finished = false;
|
|
let unsubscribe = () => {};
|
|
const finish = () => {
|
|
if (finished) return;
|
|
finished = true;
|
|
clearTimeout(timeout);
|
|
unsubscribe();
|
|
options.signal?.removeEventListener('abort', abortPlayback);
|
|
resolve();
|
|
};
|
|
const abortPlayback = () => {
|
|
if (useCanvasStore.getState().playingVideoElementId === action.elementId) {
|
|
useCanvasStore.getState().pauseVideo();
|
|
}
|
|
finish();
|
|
};
|
|
const timeout = setTimeout(() => {
|
|
log.warn(`[playVideo] Timeout waiting for video ${action.elementId} to finish`);
|
|
finish();
|
|
}, MAX_VIDEO_WAIT_MS);
|
|
unsubscribe = useCanvasStore.subscribe((state) => {
|
|
if (state.playingVideoElementId !== action.elementId) {
|
|
finish();
|
|
}
|
|
});
|
|
options.signal?.addEventListener('abort', abortPlayback, { once: true });
|
|
if (useCanvasStore.getState().playingVideoElementId !== action.elementId) {
|
|
finish();
|
|
}
|
|
});
|
|
}
|
|
|
|
// ==================== Synchronous — Whiteboard ====================
|
|
|
|
/** Auto-open the whiteboard if it's not already open */
|
|
private async ensureWhiteboardOpen(options: ActionExecutionOptions = {}): Promise<void> {
|
|
if (!useCanvasStore.getState().whiteboardOpen) {
|
|
await this.executeWbOpen(options);
|
|
}
|
|
}
|
|
|
|
private async executeWbOpen(options: ActionExecutionOptions = {}): Promise<void> {
|
|
// Ensure a whiteboard exists
|
|
this.stageAPI.whiteboard.get();
|
|
useCanvasStore.getState().setWhiteboardOpen(true);
|
|
if (options.silent) return;
|
|
// Wait for open animation to complete (slow spring: stiffness 120, damping 18, mass 1.2)
|
|
await delay(WB_OPEN_MS);
|
|
}
|
|
|
|
private async executeWbDrawText(
|
|
action: WbDrawTextAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
let htmlContent = action.content ?? '';
|
|
if (!htmlContent) return; // nothing to draw
|
|
|
|
const latex = getLikelyLatexMath(htmlContent);
|
|
if (latex !== null) {
|
|
return this.executeWbDrawLatex(
|
|
{
|
|
...action,
|
|
type: 'wb_draw_latex',
|
|
latex,
|
|
},
|
|
options,
|
|
);
|
|
}
|
|
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
const fontSize = action.fontSize ?? 18;
|
|
if (!htmlContent.startsWith('<')) {
|
|
htmlContent = `<p style="font-size: ${fontSize}px;">${htmlContent}</p>`;
|
|
}
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'text',
|
|
content: htmlContent,
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width ?? 400,
|
|
height: action.height ?? 100,
|
|
rotate: 0,
|
|
defaultFontName: 'Microsoft YaHei',
|
|
defaultColor: action.color ?? '#333333',
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) {
|
|
// Wait for element fade-in animation
|
|
await delay(WB_DRAW_MS);
|
|
}
|
|
}
|
|
|
|
private async executeWbDrawShape(
|
|
action: WbDrawShapeAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'shape',
|
|
viewBox: [1000, 1000] as [number, number],
|
|
path: SHAPE_PATHS[action.shape] ?? SHAPE_PATHS.rectangle,
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width,
|
|
height: action.height,
|
|
rotate: 0,
|
|
fill: action.fillColor ?? '#5b9bd5',
|
|
fixedRatio: false,
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) {
|
|
// Wait for element fade-in animation
|
|
await delay(WB_DRAW_MS);
|
|
}
|
|
}
|
|
|
|
private async executeWbDrawChart(
|
|
action: WbDrawChartAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'chart',
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width,
|
|
height: action.height,
|
|
rotate: 0,
|
|
chartType: action.chartType,
|
|
data: action.data,
|
|
themeColors: action.themeColors ?? ['#5b9bd5', '#ed7d31', '#a5a5a5', '#ffc000', '#4472c4'],
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) await delay(WB_DRAW_MS);
|
|
}
|
|
|
|
private async executeWbDrawLatex(
|
|
action: WbDrawLatexAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
try {
|
|
const html = katex.renderToString(action.latex, {
|
|
throwOnError: false,
|
|
displayMode: true,
|
|
output: 'html',
|
|
});
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'latex',
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width ?? 400,
|
|
height: action.height ?? 80,
|
|
rotate: 0,
|
|
latex: action.latex,
|
|
html,
|
|
color: action.color ?? '#000000',
|
|
fixedRatio: true,
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
} catch (err) {
|
|
log.warn(`Failed to render latex "${action.latex}":`, err);
|
|
return;
|
|
}
|
|
|
|
if (!options.silent) await delay(WB_DRAW_MS);
|
|
}
|
|
|
|
private async executeWbDrawTable(
|
|
action: WbDrawTableAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
const rows = action.data.length;
|
|
const cols = rows > 0 ? action.data[0].length : 0;
|
|
if (rows === 0 || cols === 0) return;
|
|
|
|
// Build colWidths: equal distribution
|
|
const colWidths = Array(cols).fill(1 / cols);
|
|
|
|
// Build TableCell[][] from string[][]
|
|
let cellId = 0;
|
|
const tableData = action.data.map((row) =>
|
|
row.map((text) => ({
|
|
id: `cell_${cellId++}`,
|
|
colspan: 1,
|
|
rowspan: 1,
|
|
text,
|
|
})),
|
|
);
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'table',
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width,
|
|
height: action.height,
|
|
rotate: 0,
|
|
colWidths,
|
|
cellMinHeight: 36,
|
|
data: tableData,
|
|
outline: action.outline ?? {
|
|
width: 2,
|
|
style: 'solid',
|
|
color: '#eeece1',
|
|
},
|
|
theme: action.theme
|
|
? {
|
|
color: action.theme.color,
|
|
rowHeader: true,
|
|
rowFooter: false,
|
|
colHeader: false,
|
|
colFooter: false,
|
|
}
|
|
: undefined,
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) await delay(WB_DRAW_MS);
|
|
}
|
|
|
|
private async executeWbDrawLine(
|
|
action: WbDrawLineAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
// Calculate bounding box — left/top is the minimum of start/end coordinates
|
|
const left = Math.min(action.startX, action.endX);
|
|
const top = Math.min(action.startY, action.endY);
|
|
|
|
// Convert absolute coordinates to relative coordinates (relative to left/top)
|
|
const start: [number, number] = [action.startX - left, action.startY - top];
|
|
const end: [number, number] = [action.endX - left, action.endY - top];
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'line',
|
|
left,
|
|
top,
|
|
width: action.width ?? 2,
|
|
start,
|
|
end,
|
|
style: action.style ?? 'solid',
|
|
color: action.color ?? '#333333',
|
|
points: action.points ?? ['', ''],
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) {
|
|
// Wait for element fade-in animation
|
|
await delay(WB_DRAW_MS);
|
|
}
|
|
}
|
|
|
|
private async executeWbDrawCode(
|
|
action: WbDrawCodeAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
const lines = codeToLines(action.code);
|
|
const suppliedLineIds = (action as WbDrawCodeAction & { lineIds?: string[] }).lineIds;
|
|
if (suppliedLineIds?.length === lines.length) {
|
|
lines.forEach((line, index) => {
|
|
line.id = suppliedLineIds[index];
|
|
});
|
|
}
|
|
|
|
this.stageAPI.whiteboard.addElement(
|
|
{
|
|
id: action.elementId || '',
|
|
type: 'code',
|
|
language: action.language,
|
|
lines,
|
|
fileName: action.fileName,
|
|
showLineNumbers: true,
|
|
fontSize: 14,
|
|
left: action.x,
|
|
top: action.y,
|
|
width: action.width ?? 500,
|
|
height: action.height ?? 300,
|
|
rotate: 0,
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
} as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) {
|
|
// Wait for typing animation (base 800ms + 50ms/line, capped at 3s)
|
|
const animMs = wbDrawCodeMs(lines.length);
|
|
await delay(animMs);
|
|
}
|
|
}
|
|
|
|
private async executeWbEditCode(
|
|
action: WbEditCodeAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
const elementResult = this.stageAPI.whiteboard.getElement(action.elementId, wb.data.id);
|
|
if (!elementResult.success || !elementResult.data) return;
|
|
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
const element = elementResult.data as any;
|
|
if (element.type !== 'code') return;
|
|
|
|
let lines: CodeLine[] = [...element.lines];
|
|
const newContentLines = action.content ? action.content.split('\n') : [];
|
|
const suppliedLineIds = (action as WbEditCodeAction & { newLineIds?: string[] }).newLineIds;
|
|
const newLineIds =
|
|
suppliedLineIds?.length === newContentLines.length
|
|
? suppliedLineIds
|
|
: generateLineIds(newContentLines.length);
|
|
|
|
switch (action.operation) {
|
|
case 'insert_after': {
|
|
const idx = lines.findIndex((l) => l.id === action.lineId);
|
|
if (idx === -1) return;
|
|
const newLines = newContentLines.map((content, i) => ({ id: newLineIds[i], content }));
|
|
lines.splice(idx + 1, 0, ...newLines);
|
|
break;
|
|
}
|
|
case 'insert_before': {
|
|
const idx = lines.findIndex((l) => l.id === action.lineId);
|
|
if (idx === -1) return;
|
|
const newLines = newContentLines.map((content, i) => ({ id: newLineIds[i], content }));
|
|
lines.splice(idx, 0, ...newLines);
|
|
break;
|
|
}
|
|
case 'delete_lines': {
|
|
if (!action.lineIds?.length) return;
|
|
const deleteSet = new Set(action.lineIds);
|
|
lines = lines.filter((l) => !deleteSet.has(l.id));
|
|
break;
|
|
}
|
|
case 'replace_lines': {
|
|
if (!action.lineIds?.length) return;
|
|
const replaceIds = action.lineIds;
|
|
const firstIdx = lines.findIndex((l) => l.id === replaceIds[0]);
|
|
if (firstIdx === -1) return;
|
|
const deleteSet = new Set(replaceIds);
|
|
lines = lines.filter((l) => !deleteSet.has(l.id));
|
|
const newLines = newContentLines.map((content, i) => ({
|
|
id: i < replaceIds.length ? replaceIds[i] : newLineIds[i],
|
|
content,
|
|
}));
|
|
lines.splice(firstIdx, 0, ...newLines);
|
|
break;
|
|
}
|
|
}
|
|
|
|
this.stageAPI.whiteboard.updateElement(
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
{ ...element, lines } as any,
|
|
wb.data.id,
|
|
);
|
|
|
|
if (!options.silent) {
|
|
// Wait for edit animation
|
|
await delay(WB_EDIT_MS);
|
|
}
|
|
}
|
|
|
|
private async executeWbDelete(
|
|
action: WbDeleteAction,
|
|
options: ActionExecutionOptions = {},
|
|
): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
this.stageAPI.whiteboard.deleteElement(action.elementId, wb.data.id);
|
|
if (!options.silent) await delay(WB_DELETE_MS);
|
|
}
|
|
|
|
private async executeWbClear(options: ActionExecutionOptions = {}): Promise<void> {
|
|
const wb = this.stageAPI.whiteboard.get();
|
|
if (!wb.success || !wb.data) return;
|
|
|
|
const elementCount = wb.data.elements?.length || 0;
|
|
if (elementCount === 0) return;
|
|
|
|
if (options.silent) {
|
|
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
|
|
useCanvasStore.getState().setWhiteboardClearing(false);
|
|
return;
|
|
}
|
|
|
|
// Save snapshot before AI clear (mirrors UI handleClear in index.tsx)
|
|
useWhiteboardHistoryStore.getState().pushSnapshot(wb.data.elements!);
|
|
|
|
// Trigger cascade exit animation
|
|
useCanvasStore.getState().setWhiteboardClearing(true);
|
|
|
|
// Wait for cascade (base 380ms + 55ms/element, capped at 1400ms)
|
|
const animMs = wbClearMs(elementCount);
|
|
await delay(animMs);
|
|
|
|
// Actually remove elements
|
|
this.stageAPI.whiteboard.update({ elements: [] }, wb.data.id);
|
|
useCanvasStore.getState().setWhiteboardClearing(false);
|
|
}
|
|
|
|
private async executeWbClose(options: ActionExecutionOptions = {}): Promise<void> {
|
|
useCanvasStore.getState().setWhiteboardOpen(false);
|
|
if (options.silent) return;
|
|
// Wait for close animation (500ms ease-out tween)
|
|
await delay(WB_CLOSE_MS);
|
|
}
|
|
|
|
// ==================== Widget Actions ====================
|
|
|
|
/** Send message to widget iframe */
|
|
private sendWidgetMessage(type: string, payload: Record<string, unknown>): void {
|
|
if (this.widgetMessageCallback) {
|
|
this.widgetMessageCallback(type, payload);
|
|
} else {
|
|
log.warn(`Widget message callback not set, cannot send: ${type}`);
|
|
}
|
|
}
|
|
|
|
/** Execute widget highlight action (quick visual change) */
|
|
private async executeWidgetHighlight(action: WidgetHighlightAction): Promise<void> {
|
|
this.sendWidgetMessage('HIGHLIGHT_ELEMENT', {
|
|
target: action.target,
|
|
content: action.content,
|
|
});
|
|
// Quick delay for visual effect
|
|
await delay(WIDGET_MS);
|
|
}
|
|
|
|
/** Execute widget setState action */
|
|
private async executeWidgetSetState(action: WidgetSetStateAction): Promise<void> {
|
|
this.sendWidgetMessage('SET_WIDGET_STATE', { state: action.state, content: action.content });
|
|
// Quick delay for state change to propagate
|
|
await delay(WIDGET_MS);
|
|
}
|
|
|
|
/** Execute widget annotation action */
|
|
private async executeWidgetAnnotation(action: WidgetAnnotationAction): Promise<void> {
|
|
this.sendWidgetMessage('ANNOTATE_ELEMENT', {
|
|
target: action.target,
|
|
content: action.content,
|
|
});
|
|
await delay(WIDGET_MS);
|
|
}
|
|
|
|
/** Execute widget reveal action */
|
|
private async executeWidgetReveal(action: WidgetRevealAction): Promise<void> {
|
|
this.sendWidgetMessage('REVEAL_ELEMENT', { target: action.target, content: action.content });
|
|
await delay(WIDGET_MS);
|
|
}
|
|
}
|