* feat(model): optional per-stage LLM model routing (#745) Add a config-only stage -> model map consulted during model resolution, falling back to DEFAULT_MODEL when unset (zero behavior change unless opted in). - lib/server/model-routes.ts: parse/validate/cache MODEL_ROUTES env JSON; LLM_STAGES registry of routable stages; getStageModel(stage). - resolveModel: resolution order x-model > stage route > DEFAULT_MODEL > builtin; thread optional `stage` through resolveModelFromHeaders/FromRequest. - Wire each route's resolveModel call site to its stage; classroom-generation resolves generate-classroom and web-search-query-rewrite independently. - .env.example: document MODEL_ROUTES; unit tests for both modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model): address cross-review findings for per-stage routing (#745) - classroom-generation: resolve the web-search-query-rewrite model lazily, only when explicitly routed and inside the web-search branch, with try/catch fallback to the classroom model. Fixes a regression where a misconfigured optional route (keyless/invalid provider) threw and aborted ALL classroom generation even with web search disabled; also avoids wasted resolution on the common no-search path and preserves classroom-model inheritance when unrouted. - resolve-model: type the stage param as LlmStage across resolveModel/ resolveModelFromHeaders/resolveModelFromRequest so a mistyped stage literal is a compile error instead of silently falling through to DEFAULT_MODEL. - model-routes: simplify the cache to a process singleton (matching provider-config), dropping the raw-string-keyed cache and redundant STAGE_SET. - .env.example: clarify x-model precedence and that a route to an unconfigured provider fails at request time (no startup validation), like a bad DEFAULT_MODEL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model): stage route takes precedence over client x-model (#745) Cross-review (codex P2 + Claude tracer) found that the browser UI always sends its saved model as x-model, so the previous x-model > stage order made MODEL_ROUTES inert for normal UI traffic on exactly the heavy stages the RFC targets (scene-content, quiz, pbl-chat, chat). Flip resolution to stage route > x-model > DEFAULT_MODEL: a configured route is the operator's deliberate per-stage choice and wins, while unrouted stages still honor the client's x-model. Update docs and tests accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(model): prettier-format changed files for #745 CI Run the repo formatter (pnpm check / prettier) on the changed routes, the new model-routes module, and the new tests so `prettier . --check` (CI) passes; also correct a stale precedence comment in the resolve-model test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model): isolate routed model from client connection params (#745) When a stage route overrides the client's x-model and points at a different provider, the client-sent apiKey/baseUrl/providerType (for the client's model) must not bleed onto the routed provider — otherwise a routed Anthropic model would be constructed with the client's OpenAI providerType/key and fail. A routed model now resolves its connection params purely from server config, as if no x-model was sent. Unrouted stages still honor the client params. Adds tests covering both the drop (routed) and keep (unrouted) cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(model): correct graceful-degradation comment in classroom (#745) resolveModel does not throw on a missing key; clarify that a keyless route surfaces later in callLLM (degraded by the outer try/catch), while only a resolution-time failure is caught at the rewrite re-resolve. Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model): route scene-content per scene type (#745) Extend MODEL_ROUTES with composite keys scene-content:<type> for the four core scene types (slide/quiz/interactive/pbl). getStageModel now resolves composite `a:b` stages most-specific-first, trimming `:` segments, so scene-content:<type> falls back to the base scene-content route, then x-model, then DEFAULT_MODEL. The scene-content route derives its stage from outline.type. Backward compatible: with no composite key configured, behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model): per-stage thinking effort + drop hardcoded gpt fallback (#745) MODEL_ROUTES values may now be {model, effort} (not just a model string) to pin the thinking effort per stage. Arbitration mirrors model routing: a routed stage with effort set wins over the client's thinking (effort "none" disables); routed with no effort uses the model's default and drops client thinking; unrouted stages keep the client thinking. resolveModel is the single arbiter — body thinking is threaded in via resolveModelFromRequest. classroom-generation passes the resolved thinking into its callLLM calls so generate-classroom and web-search-query-rewrite honor route effort too. Also: removed the hardcoded `|| 'gpt-5.4-mini'` fallback in resolveModel — if no model resolves (no route / x-model / DEFAULT_MODEL) it now throws instead of silently picking a vendor default. model-routes: getStageRoute returns {model, effort?}; getStageModel delegates to it. Docs + tests updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(model): route value carries full ThinkingConfig (#745) Per-stage routing accepts the full unified ThinkingConfig in the route value ({model, thinking:{mode,effort,level,enabled,budgetTokens,excludeReasoningOutput}}) instead of just an effort string. The route's thinking is passed through resolveModel and normalized per the model's capability by callLLM, so budgetTokens (qwen), level (Gemini), enabled/mode, etc. all work. (qwen3.7-plus/max thinking capability now comes from upstream #753, so this no longer registers them.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(model): honor routed thinking for chat-adapter; doc fixes (#745) Cross-review (codex P2 + Claude) found chat-adapter was the only routable stage that ignored the route's thinking: it took the routed model but built its own thinkingConfig from the request body, so operator-pinned thinking never applied and client thinking still leaked onto a routed model. Now pass the client thinking into resolveModel and use the resolved thinkingConfig (route-pinned for a routed stage, client otherwise; defaults to disabled for low-latency chat). Also: clarify DEFAULT_MODEL is now required for server-side stages (the hardcoded gpt-5.4-mini fallback was removed → resolveModel throws if nothing resolves), and fix a stale {model,effort}→{model,thinking} doc comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
174 lines
6.8 KiB
TypeScript
174 lines
6.8 KiB
TypeScript
/**
|
|
* Shared model resolution utilities for API routes.
|
|
*
|
|
* Extracts the repeated parseModelString → resolveApiKey → resolveBaseUrl →
|
|
* resolveProxy → getModel boilerplate into a single call.
|
|
*/
|
|
|
|
import type { NextRequest } from 'next/server';
|
|
import { getModel, parseModelString, type ModelWithInfo } from '@/lib/ai/providers';
|
|
import type { ThinkingConfig } from '@/lib/types/provider';
|
|
import {
|
|
isServerConfiguredProvider,
|
|
resolveApiKey,
|
|
resolveBaseUrl,
|
|
resolveProxy,
|
|
} from '@/lib/server/provider-config';
|
|
import { validateUrlForSSRF } from '@/lib/server/ssrf-guard';
|
|
import { getStageRoute, type LlmStage } from '@/lib/server/model-routes';
|
|
|
|
export interface ResolvedModel extends ModelWithInfo {
|
|
/** Original model string (e.g. "openai/gpt-4o-mini") */
|
|
modelString: string;
|
|
/** Resolved provider ID (e.g. "openai", "ollama") */
|
|
providerId: string;
|
|
/** Resolved model ID (e.g. "gpt-4o-mini") */
|
|
modelId: string;
|
|
/** Effective API key after server-side fallback resolution */
|
|
apiKey: string;
|
|
/** Effective base URL after server/client resolution */
|
|
baseUrl?: string;
|
|
/** Optional per-request thinking configuration from the client. */
|
|
thinkingConfig?: ThinkingConfig;
|
|
}
|
|
|
|
/**
|
|
* Resolve a language model from explicit parameters.
|
|
*
|
|
* Use this when model config comes from the request body.
|
|
*/
|
|
export async function resolveModel(params: {
|
|
modelString?: string;
|
|
/**
|
|
* Optional generation stage (a `callLLM` source label, e.g. 'scene-content').
|
|
* When set and a route is configured via `MODEL_ROUTES`, the route wins for
|
|
* this call — even over a client-sent `modelString` (x-model). Unrouted
|
|
* stages fall back to `modelString` then `DEFAULT_MODEL`. See
|
|
* lib/server/model-routes.ts.
|
|
*/
|
|
stage?: LlmStage;
|
|
apiKey?: string;
|
|
baseUrl?: string;
|
|
providerType?: string;
|
|
thinkingConfig?: ThinkingConfig;
|
|
}): Promise<ResolvedModel> {
|
|
// Resolution order: stage route > x-model > DEFAULT_MODEL.
|
|
// A configured stage route is the operator's deliberate per-stage choice and
|
|
// wins even over a client-sent x-model (otherwise the browser UI, which always
|
|
// sends its saved model, would shadow every route). Unrouted stages fall back
|
|
// to the client x-model, then DEFAULT_MODEL. There is intentionally no hardcoded
|
|
// model fallback — if nothing resolves we fail loud rather than silently pick a
|
|
// vendor default.
|
|
const stageRoute = getStageRoute(params.stage);
|
|
const stageModel = stageRoute?.model;
|
|
const modelString = stageModel || params.modelString || process.env.DEFAULT_MODEL;
|
|
if (!modelString) {
|
|
throw new Error(
|
|
'No model could be resolved. Configure DEFAULT_MODEL (and/or a MODEL_ROUTES entry for this stage), or send a model via x-model.',
|
|
);
|
|
}
|
|
const { providerId, modelId } = parseModelString(modelString);
|
|
|
|
// When a stage route overrides the client's model, the client-sent connection
|
|
// params (apiKey/baseUrl/providerType) belong to the client's *other* model
|
|
// and must not bleed onto the routed provider — otherwise e.g. a routed
|
|
// Anthropic model would be built with the client's OpenAI providerType/key.
|
|
// A routed model resolves purely from server config, as if no x-model was sent.
|
|
const routed = Boolean(stageModel);
|
|
const clientApiKey = routed ? undefined : params.apiKey;
|
|
const clientProviderType = routed ? undefined : params.providerType;
|
|
const clientBaseUrlParam = routed ? undefined : params.baseUrl;
|
|
|
|
// Server-managed providers are admin-owned: the operator's key and base URL
|
|
// are authoritative and any client-sent override is ignored. SSRF validation
|
|
// therefore applies only to unmanaged providers, where the base URL really is
|
|
// client-supplied. (Server-configured URLs are trusted by the operator.)
|
|
const managed = isServerConfiguredProvider('providers', providerId);
|
|
const clientBaseUrl = managed ? undefined : clientBaseUrlParam || undefined;
|
|
if (clientBaseUrl && process.env.NODE_ENV === 'production') {
|
|
const ssrfError = await validateUrlForSSRF(clientBaseUrl);
|
|
if (ssrfError) {
|
|
throw new Error(ssrfError);
|
|
}
|
|
}
|
|
|
|
const apiKey = resolveApiKey(providerId, clientApiKey || '');
|
|
const baseUrl = resolveBaseUrl(providerId, clientBaseUrl);
|
|
const proxy = resolveProxy(providerId);
|
|
const { model, modelInfo } = getModel({
|
|
providerId,
|
|
modelId,
|
|
apiKey,
|
|
baseUrl,
|
|
proxy,
|
|
providerType: clientProviderType as 'openai' | 'anthropic' | 'google' | undefined,
|
|
});
|
|
|
|
// Thinking arbitration mirrors model routing — the route carries a full
|
|
// ThinkingConfig (mode/effort/level/enabled/budgetTokens/…) which callLLM
|
|
// normalizes against the model's capability:
|
|
// - routed + thinking set → the route's thinking wins (over client thinking).
|
|
// - routed + no thinking → routed model uses its own default; client thinking
|
|
// is dropped (it belonged to the client's other model).
|
|
// - unrouted → honor the client's thinking config.
|
|
const thinkingConfig: ThinkingConfig | undefined = routed
|
|
? stageRoute?.thinking
|
|
: params.thinkingConfig;
|
|
|
|
return {
|
|
model,
|
|
modelInfo,
|
|
modelString,
|
|
providerId,
|
|
modelId,
|
|
apiKey,
|
|
baseUrl,
|
|
thinkingConfig,
|
|
};
|
|
}
|
|
|
|
function getThinkingConfigFromBody(body: unknown): ThinkingConfig | undefined {
|
|
if (!body || typeof body !== 'object') return undefined;
|
|
const record = body as { thinkingConfig?: unknown; thinking?: unknown };
|
|
const config = record.thinkingConfig ?? record.thinking;
|
|
return config && typeof config === 'object' ? (config as ThinkingConfig) : undefined;
|
|
}
|
|
|
|
/**
|
|
* Resolve a language model from standard request headers.
|
|
*
|
|
* Reads: x-model, x-api-key, x-base-url, x-provider-type
|
|
* Note: requiresApiKey is derived server-side from the provider registry,
|
|
* never from client headers, to prevent auth bypass.
|
|
*/
|
|
export async function resolveModelFromHeaders(
|
|
req: NextRequest,
|
|
stage?: LlmStage,
|
|
thinkingConfig?: ThinkingConfig,
|
|
): Promise<ResolvedModel> {
|
|
return resolveModel({
|
|
modelString: req.headers.get('x-model') || undefined,
|
|
stage,
|
|
apiKey: req.headers.get('x-api-key') || undefined,
|
|
baseUrl: req.headers.get('x-base-url') || undefined,
|
|
providerType: req.headers.get('x-provider-type') || undefined,
|
|
thinkingConfig,
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Resolve a language model from standard request headers plus body fields.
|
|
*
|
|
* Reads model credentials from headers and per-request thinking config from
|
|
* the JSON body field `thinkingConfig` (or legacy/eval field `thinking`).
|
|
*/
|
|
export async function resolveModelFromRequest(
|
|
req: NextRequest,
|
|
body: unknown,
|
|
stage?: LlmStage,
|
|
): Promise<ResolvedModel> {
|
|
// Pass the client's body thinking into resolveModel so the single arbiter
|
|
// there decides (a routed stage may override or drop it). See resolveModel.
|
|
return resolveModelFromHeaders(req, stage, getThinkingConfigFromBody(body));
|
|
}
|