openmaic-custom/lib/utils/concurrency.ts
ly-wang19 3ccd5da6da
feat(generation): opt-in parallel scene-content generation (#660)
* feat(generation): opt-in parallel scene-content generation

Scene generation was strictly serial — content -> actions -> TTS, one
outline at a time — so an N-scene classroom paid N x (content + actions
+ TTS) of wall-clock latency, which dominates the post-outline wait.

Add an opt-in hybrid two-phase path (#572):

- Phase 1 fetches scene *content* concurrently (bounded). Content is the
  only per-scene step independent of cross-scene state, so it is safe to
  parallelise; a content failure marks just that outline and does not
  pause the batch.
- Phase 2 keeps the existing in-order actions + TTS loop, so
  previousSpeeches threading, ordered addScene, the abort/epoch guards,
  and the pause-on-failure UX are all unchanged.

Gated by a server-side PARALLEL_SCENE_CONCURRENCY (default 0 = off,
clamped to 10), surfaced to the client through the existing
/api/server-providers response. With it unset the content map is null
and the loop is byte-for-byte the original serial path, so out-of-box
behaviour is unchanged.

The bounded worker pool is extracted to lib/utils/concurrency.ts
(mapWithConcurrency, with an early-stop hook for abort/epoch) and
unit-tested; getParallelSceneConcurrency env parsing/clamping is tested
too.

Closes #572

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(generation): pipeline content fetches instead of a barrier

Address @wyuc's review on #660. The previous version awaited the whole
bounded content pool before Phase 2, so for N > C scenes the user saw the
first scene, then a stall on a single page for the entire content phase,
then a burst — perceived as worse than serial despite the wall-clock win
(the gap scales with ceil((N-1)/C)).

Pipeline instead: lazyBoundedMap starts the content fetches (<= C in
flight) and returns one promise per outline immediately; the serial loop
awaits them in order, so each resolves as soon as its own content is ready
(usually already done, hidden behind the previous scene's actions/TTS). The
first scene now paints after content(1)+actions(1)+TTS(1) — same as serial
— with the full concurrency benefit and no stall.

Also from the review:
- a content fetch can no longer take its siblings down: an unexpected throw
  is caught and returned as a failure result, routed through the same
  mark-failed path as the serial loop (symmetric);
- drop the cosmetic Phase-1 setCurrentGeneratingOrder — the loop already
  sets it per outline;
- note the intentional belt-and-suspenders clamp;
- concurrency.test.ts asserts <= limit (not == limit), and gains
  lazyBoundedMap tests including the no-barrier property.

mapWithConcurrency is kept as a thin await-all wrapper over lazyBoundedMap.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn>
2026-06-08 07:44:33 -04:00

76 lines
2.6 KiB
TypeScript

/** A FIFO counting semaphore: at most `size` `run()` callbacks execute at once. */
function createSemaphore(size: number) {
const max = Math.max(1, Math.floor(size));
let active = 0;
const queue: Array<() => void> = [];
const pump = () => {
while (active < max && queue.length > 0) {
active += 1;
const start = queue.shift()!;
start();
}
};
return {
run<R>(fn: () => Promise<R>): Promise<R> {
return new Promise<R>((resolve, reject) => {
queue.push(() => {
fn()
.then(resolve, reject)
.finally(() => {
active -= 1;
pump();
});
});
pump();
});
},
};
}
/**
* Start `fn` over every item with at most `limit` calls in flight at once, and
* return one promise per item **immediately**, in input order — without awaiting
* them. Each item acquires a `limit`-sized semaphore slot before `fn` runs, so
* all the promises exist up front but only `limit` execute concurrently.
*
* This is the no-barrier primitive: the caller can `await` the promises in any
* order (e.g. sequentially) and each resolves as soon as *its* work is done,
* while later items keep running in the background. `shouldContinue` is checked
* when an item reaches the front of the queue; once it returns false, the
* remaining items resolve to `undefined` without running `fn`.
*
* `limit` is clamped to `[1, items.length]`, so a raw/too-large concurrency is
* safe to pass.
*/
export function lazyBoundedMap<T, R>(
items: readonly T[],
limit: number,
fn: (item: T, index: number) => Promise<R>,
options?: { shouldContinue?: () => boolean },
): Array<Promise<R | undefined>> {
const shouldContinue = options?.shouldContinue ?? (() => true);
const semaphore = createSemaphore(Math.min(Math.floor(limit), items.length || 1));
return items.map((item, index) =>
semaphore.run(async () => (shouldContinue() ? fn(item, index) : undefined)),
);
}
/**
* Run `fn` over `items` with at most `limit` calls in flight at once and await
* them all (a barrier). Results are returned in input order; a slot is
* `undefined` if its item was skipped because `shouldContinue` turned false.
*
* Prefer {@link lazyBoundedMap} when you can consume results incrementally —
* this wrapper waits for every item before returning.
*/
export async function mapWithConcurrency<T, R>(
items: readonly T[],
limit: number,
fn: (item: T, index: number) => Promise<R>,
options?: { shouldContinue?: () => boolean },
): Promise<Array<R | undefined>> {
return Promise.all(lazyBoundedMap(items, limit, fn, options));
}