8 Commits

Author SHA1 Message Date
wyuc
0a10af11cf
feat(storage): opt-in indirect asset byte egress (#1007) (#1100)
* feat(storage): add opt-in indirect asset byte egress

A deployment can now opt an asset byte GET into a 302 to a short-lived
signed URL when the byte layer can sign. It is off by default, and
byte-for-byte unchanged when off. The byte layer gains an optional
signReadUrl capability; S3AssetByteStore implements it through the new
optional @aws-sdk/s3-request-presigner peer, resolved lazily exactly
like the client SDK, with a signer seam mirroring the commands seam so
tests can bind doubles. The signed URL pins the contract's response
headers -- the media type after the renderable allowlist, the fixed
disposition, and the no-store cache posture -- via S3 response-header
overrides. PgAssetStore mints the URL from the same ownership-checked
transactional read a direct resolve performs, so authorization still
runs per read before any URL exists, and a store whose byte layer
cannot sign falls back to direct bytes. The contract documents the
shape and the disclosure tradeoff: a Location into a hash-keyed byte
layer names the content hash, so deployments that need the
no-disclosure property keep direct egress.

Refs #1007

* feat(app): wire ASSET_BYTE_EGRESS into the persistence route

ASSET_BYTE_EGRESS=redirect opts asset byte GETs into the storage
package's indirect egress; unset, direct, or an unrecognized value
keeps the default byte-for-byte behavior, with a warning for the
unrecognized case. The lazy byte-store wrapper now forwards
signReadUrl and answers undefined when the resolved layer has no
signer, so the PostgreSQL byte column degrades to direct bytes and
the S3 layer signs without the wrapper knowing which it holds. The
collector path is untouched: it builds its byte store through
createAssetByteStore as before and only ever calls delete.

Refs #1007

* fix(storage): let the packaged client survive redirect egress

Cross-review round 1 on the indirect byte egress change found two
problems.

A byte response that followed a redirect carries the pinned content
type but no X-Asset-Revision, and HttpAssetStore required both from
the same response, so every cold resolve under redirect egress failed
MALFORMED_RESPONSE. The client now detects the shape once -- a
redirected byte response with no revision -- latches it, and probes
HEAD before each cold GET from then on, taking the label from the
probe. The probe predates the download on purpose: a replacement in
between labels newer bytes with an older revision, which the next
revalidation detects and corrects, while the reverse order could pin
stale bytes under a fresh revision with no signal to repair it. The
probe doubles as the miss check, so a redirect-mode miss costs no
download. The direct path is byte-for-byte unchanged; the one double
download per client lifetime on first contact is documented in the
contract, which gains the client-side paragraph this behavior
implements.

signedUrlTtlSeconds also accepted lifetimes beyond the seven-day
SigV4 presigning maximum, minting redirects the object store would
reject. The option is now capped at construction.

Refs #1007

* fix(storage): cap signed URL lifetime below the reclamation grace

A signed URL whose lifetime exceeds the collector grace period can
outlive its object: the last reference goes, the grace elapses, the
collector deletes the object, and the still-valid URL errors at the
object store. The previous ceiling -- the seven-day SigV4 maximum --
made that window days wide against the one-hour default grace. The
cap is now fifteen minutes, and the contract text states both the
ceiling and the rule for deployments that shorten their grace.

Refs #1007

* fix(storage): decline signing when the optional presigner is absent

An unresolvable @aws-sdk/s3-request-presigner made signReadUrl throw,
which the registry surfaced as a failed asset read -- a 500 for a
deployment whose only fault is a missing optional peer. The contract's
answer for a byte layer that cannot sign is the capability fallback:
return undefined and let the caller serve the bytes directly. Signing
errors from a resolved signer still fail loud; only the missing
capability declines.

Refs #1007

* fix(storage): answer redirect egress as a descriptor to asking clients

Following a 302 is not header-neutral: the platform fetch forwards the
original request's headers to the object store's origin, stripping only
Authorization, so a deployment whose credential travels in a custom
header would hand it to the object store -- and the preflight the
forwarded custom headers provoke commonly fails there besides. This
replaces the latch-and-probe client shape with an explicit one: the
client sends X-Asset-Egress: descriptor on every byte GET, a
redirect-egress server answers 200 with a JSON { url, revision } body
instead of a Location, and the client fetches the signed URL with no
deployment headers at all. The revision comes from the descriptor, so
the probe-first ordering and its one wasted download are gone too. The
302 remains the answer for consumers that did not ask; the descriptor
response is marked by its own header so a JSON-media asset can never
parse as one.

Refs #1007

* fix(storage): negotiate the descriptor through Accept, not a custom header

A custom X-Asset-Egress request header is not CORS-safelisted, so every
byte GET to a cross-origin server became a preflighted request -- a
regression for direct-egress deployments that never opted into
anything, against servers with no reason to allow the header. The
negotiation now rides Accept with a vendor media type, which is
safelisted and costs no preflight, and the descriptor answer is
identified by that media type as its Content-Type rather than by a
marker header.

Refs #1007

* fix(storage): sign outside the registry transaction and reserve the descriptor type

Two review findings on the indirect egress path.

resolveIndirect awaited the signer inside the coordinated read, but a
signer on refreshable credentials can wait on the network, and a
database connection plus the blob row's FOR SHARE lock would be held
across that delay. The read now closes first -- the lock still puts
the hash, label and revision in one snapshot -- and signing runs
afterwards, which cannot observe anything the read did not.

The descriptor media type is also reserved from renderableTypes: the
client identifies a descriptor answer by that exact Content-Type, and
an allowlisted asset served with it would be misread as a descriptor
and fail MALFORMED_RESPONSE. The handler now rejects the configuration
at construction, and the contract states the reservation.

Refs #1007

* fix(storage): ship the signing deps and negotiate without ambiguity

Three review findings on the indirect egress path.

The app now declares @aws-sdk/client-s3 and
@aws-sdk/s3-request-presigner as runtime dependencies, and the
standalone build force-includes them for the persistence route: both
are reached through deliberately untraced dynamic imports, so the
shipped image could not resolve them, and a deployment opting into S3
or redirect egress would have found the capability silently absent.

The descriptor negotiation matches exactly on both sides: the client
compares the Content-Type essence, so a longer media type that merely
begins with the reserved value still serves as bytes, and the server
parses Accept media ranges, so a descriptor range sent with q=0
selects the redirect instead of the descriptor it explicitly rejects.

Refs #1007

* fix(app): refuse redirect egress when the collection grace is shorter

A signed URL must never outlive its object: with the handler's
60-second default lifetime, a deployment that sets
ASSET_COLLECTION_GRACE_MS below ten minutes lets the collector delete
an object while a URL minted against it is still valid. The route now
refuses the combination at handler initialization, naming both
variables -- a deterministic misconfiguration, loud at startup rather
than silent at read time.

Refs #1007

* fix(app): ship the signing SDKs without loading them

Two findings on the standalone deployment of redirect egress.

The outputFileTracingIncludes globs did not follow pnpm's scoped
symlinks, so the standalone image carried neither AWS package and both
S3 mode and redirect egress could not resolve their SDK at runtime.
The packages are now server-external and referenced from literal --
but never called -- dynamic import thunks in the persistence byte-store
wiring, which gets them traced into the image while module resolution
still happens only on first S3 use. Verified against a local
standalone build: node_modules/@aws-sdk now contains both packages,
and the route tests that pin the never-resolve-unless-configured
discipline still pass.

Media type comparisons are also case-insensitive now, as HTTP
requires, on both the Accept parse and the descriptor recognition.

Refs #1007

* fix(app): treat an empty collection grace as unset

Number('') is 0, so a present-but-empty ASSET_COLLECTION_GRACE_MS
failed the redirect-egress coordination check and took persistence
down with it, while the collector's own parsing reads the same value
as the one-hour default. The check now trims and treats empty as
unset, matching durationEnv.

Refs #1007

* fix(storage): accept bytes too, omit signing on the known-PG wrapper, state the full tradeoff

Three review findings.

The client's descriptor request now advertises both representations --
the vendor descriptor type preferred, */* accepted -- so a strict
content-negotiating layer can never answer 406 to a client that does
in fact consume plain bytes.

The lazy byte-store wrapper no longer advertises signReadUrl when no
bucket is configured: the layer is known to be PostgreSQL at
construction, and carrying the method made resolveIndirect take its
ownership query and blob-row lock just to decline, then repeat them
in resolve, on every cold GET. With a bucket configured the lazy
validation is preserved and the wrapper still declines when the
resolved layer cannot sign.

And the contract's disclosure tradeoff now states the second edge:
object-store GETs carry ETag and Last-Modified, response overrides
cannot strip them, and because deduplicated PUTs rewrite the
hash-keyed object, Last-Modified moves when another principal
re-uploads the same bytes -- shared-object write timing, beyond byte
equality. Deployments for whom that signal is sensitive must keep
direct egress.

Refs #1007

* fix(app): degrade on bad grace, share the ttl invariant, cover the real boundary

The latest review round, four findings.

A misconfigured ASSET_COLLECTION_GRACE_MS now degrades redirect
egress to direct with a warning instead of failing the shared
handler's initialization -- the asset backend is optional, and its
misconfiguration must never take document and runtime traffic down.

The package exports assertSignedUrlTtlWithinGrace so a deployment
wiring the handler and the collector separately validates the
invariant once at its own boundary: the handler caps the lifetime,
but only the deployment knows both numbers.

The contract states that nosniff does not survive the redirect --
response-header overrides cannot set it -- and why the client-minted
blob makes that safe for conforming clients.

And the route gains a real-boundary test: the Fetch<->Node adapter,
the real createStorageHttpHandler, and the egress wiring, exercised
with an actual byte GET through the composed stack instead of mocks.

Refs #1007

* fix(storage): make the unsafe egress configuration unrepresentable

Three findings on the indirect egress path, all resolved by moving a
rule into a place where it cannot be bypassed rather than by adding a
check a consumer has to remember.

The grace/TTL invariant now lives in the option shape. byteEgress takes
'direct' or { mode: 'redirect', collectionGraceMs, signedUrlTtlSeconds? },
so enabling indirect egress without declaring the reclamation window it
lives inside no longer type-checks, and the handler validates both
numbers at construction. The flat signedUrlTtlSeconds option, its
cross-field "requires byteEgress redirect" check, and the standalone
900-second cap as the only guard all go away; the ceiling stays, but
alongside the ratio rather than in place of it. This is what the earlier
exported assertSignedUrlTtlWithinGrace could not do -- it protected only
consumers who called it -- and it costs nothing in compatibility because
the option is new in this branch. The app resolves the grace through one
shared parser with the collector, so the number the handler checks is the
number the collector runs with.

A signed URL whose object is gone is now a miss. The client maps a 404
from the byte fetch to ASSET_NOT_FOUND, matching the direct path: the
entry was owned and readable when the URL was minted, so a reclaimed
object is the same physical state the direct read reports as a miss, and
what a read means must not depend on the deployment's egress setting.
Signing still does not probe for the object -- that would restore the
round trip this feature removes and make the mint's price vary with prior
presence -- so the contract instead requires the byte layer to answer 404
for an absent object, and states that S3 needs s3:ListBucket to do so.

Route coverage now spans the whole egress matrix through the real
composed handler rather than only the direct case: 302 for a consumer
that did not ask, descriptor for one that did, direct bytes when the
byte layer declines to sign, and direct bytes when a short grace degraded
the configured mode.

Refs #1007

* feat(storage): forward byte egress options through reference server

* fix(storage): fail closed on redirects and unconfirmed 404s in indirect byte egress

A signed-fetch 404 now maps to a miss only when the object store's XML
body declares NoSuchKey; every other 404 from the signed fetch fails loud,
so a wrong bucket, access point, or endpoint can no longer make a live
asset read as absent. The descriptor byte GET is sent with redirect:
'manual' and any 3xx answer is treated as an error, so a server that
ignores the descriptor negotiation can never forward the deployment's
custom credential headers to a redirect target.

The ASSET_BYTE_EGRESS switch is now documented in .env.example and the
server-persistence deployment section, together with the object-store CORS
and s3:ListBucket prerequisites it requires, and the asset descriptor
media type is exported from the package root and the asset/http subpath.

---------

Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
2026-08-14 23:57:21 +08:00
wyuc
969574bee4
feat(storage): wire the asset backend into the app (#1007) (#1089)
* fix(persistence): stop corrupting binary response bodies in the route adapter

The embedded persistence route bridges a Node `RequestListener` to the
Fetch API by hand. Its response object is cast with `as unknown as
ServerResponse`, so the compiler checks none of that surface, and two
parts of it were wrong in ways only a byte-carrying handler would hit.

`end` decoded any `Uint8Array` chunk with `Buffer.from(chunk).toString()`,
which is UTF-8. `ServerResponse.end` accepts a `Uint8Array`, and those
bytes are not necessarily valid UTF-8, so every unpaired byte became
U+FFFD -- silent corruption with no error anywhere. The adapter now
buffers chunks as bytes and builds the response from them.

`write` was absent entirely, making any chunked handler a runtime
TypeError rather than a compile error. It is part of the surface this
object claims to implement, so it is now implemented.

Neither reaches a caller today: the document and runtime handlers only
ever end with JSON strings, which took the string branch. Both are traps
laid for the first handler that carries bytes.

Both cases are pinned by tests that fail before this change -- the
binary one on a byte comparison, the chunked one with a 500 from the
TypeError. The existing adapter round-trip test, whose comment already
called this the most bug-prone code in the route, covered only the
string path.

* feat(storage): wire the asset backend into the app (#1007)

The server asset registry shipped with #1007 but was reachable from
nothing: the persistence route never passed an `assetStore`, and no file
under `app/`, `lib/` or `components/` imported the asset server or client
modules. This connects it, additively -- no DSL change, no version bump,
and no behavioural change when server persistence is off.

The route now builds a `PgAssetStore` on the same pool and transaction as
the document store, with `PgAssetByteStore` by default and
`S3AssetByteStore` when `ASSET_S3_BUCKET` opts a deployment into it. The
AWS SDK stays an optional peer, reached only through a dynamic import on
that branch. The offline collector is deliberately not scheduled here,
and the route says so where a reader would look for it, because leaving
it unscheduled grows storage without bound and scheduling it is the
deployment's decision.

On the browser side the asset pool gains a single-shot configuration seam
modelled on the document store's, so that with server persistence enabled
the pool is an `HttpAssetStore` carrying the same auth headers as the
other stores.

Three browser-specific mechanisms needed decisions rather than
translation:

`clearAssetPool` deletes an IndexedDB database today. In server mode
there is nothing local to delete, and a "clear cache" action must not
remove server-side assets -- that would destroy user data from a button
promising to free space. It now revokes local object URLs and closes the
client, and deletes nothing remote.

The cross-tab replacement broadcast is unchanged. Its job is to make
another tab drop a warm object URL when the bytes behind an id change,
and that need is identical in server mode.

The exclusivity proof that decides whether an in-place `replace` is safe
reasons over this browser only: local documents, unflushed state, and a
cross-tab presence probe. In server mode those cover a strict subset of
the holders, since another device can reference the same id and no probe
can see it -- and asking the server who else references an id would be an
existence oracle over other principals. The proof therefore returns not
exclusive unconditionally in server mode, and regeneration takes the
existing fork path. That is the fail-closed direction and a graceful
degradation rather than a break.

The development authenticator had to grow the asset principal's required
`key`; without it every asset request would have been correctly denied.
It derives from the same learner partition, and a principal without one
still gets no asset access. The route now states plainly that this key
comes from a client-supplied header, so the cross-principal isolation the
asset contract describes is not in force under this authenticator -- that
warning previously lived only in the auth module's own header, where
nobody mounting the route would see it.

* fix(storage): close three defects in the asset app wiring (#1007)

**The S3 branch could not load, and isolated nothing.** The route
dynamically imported the storage package's S3 byte store, but that module
statically imported the AWS SDK, so merely resolving it pulled the SDK
into resolution and bundling. The route then separately performed an
ignored native import of the SDK from the app's own scope, which fails
with ERR_MODULE_NOT_FOUND because the SDK lives under the storage
workspace package -- so a deployment that set the bucket got a broken
store rather than an S3 one. There is now one lazy loader in one
resolution scope, the storage module no longer imports the SDK at module
load, and the bucket is validated at initialization rather than treated
as valid because it is non-empty. A test mocks the SDK to throw on
resolution and asserts that importing the byte-store module never
resolves it.

**A cross-tab replacement could leave a mounted lease stale.** The
broadcast handler re-resolves, and `HttpAssetStore.resolve` coalesces
concurrent calls for one id onto a single in-flight request -- correct on
its own, and required by the contract. Together they meant a resolve
triggered by the broadcast could merge into a GET that started before the
peer's replacement, return the old revision, and leave the consumer on
the old URL with nothing to dislodge it. The client gains an invalidation
that retires its cached snapshot and advances the per-id generation
without revoking any URL already issued; `release` would have been wrong
here, since the contract keeps an issued URL valid until its holder
releases it. Removing the invalidation call fails the new race test.

**Clearing a server-backed pool left the closed client installed.** In
server mode `clearAssetPool` closed the client and returned before
clearing the module singleton, so every later `getAssetPool()` handed
back a closed store until a page reload happened to intervene. Only the
IndexedDB deletion is skipped in server mode now; the singleton is always
cleared so the next call rebuilds through the configured factory.

Also adds the tests whose absence let these through: a route round-trip
carrying invalid UTF-8, the SDK-resolution assertion above, the broadcast
race, and reopening the pool after a clear. The earlier tests were not
wrong so much as aimed past the defect -- the route test used JSON text,
and the S3 test injected a loader rather than exercising the imports.

* chore(storage): bump package version to 0.3.0

* fix(persistence): make the route adapter faithful where it carries bytes

The embedded persistence route bridges a Node `RequestListener` to the
Fetch API by hand, and its response object is cast
`as unknown as ServerResponse`, so the compiler checks none of that
surface. Five divergences are closed here; the first two are the ones
that corrupt data, the last three were raised in review.

**`end` decoded bytes as text.** It ran `Buffer.from(chunk).toString()`
on a `Uint8Array` chunk, which is UTF-8, so every byte outside that
encoding became U+FFFD -- silent corruption with no error anywhere. The
adapter now buffers chunks as bytes.

**`write` was absent**, making any chunked handler a runtime `TypeError`
rather than a compile error.

**`write` callbacks ran inline.** Node invokes them after the chunk is
handed off. A handler that writes its next chunk from each callback
therefore recursed synchronously and could overflow the stack, and a
throwing callback threw in the wrong execution phase. They are now
deferred.

**The encoding argument was ignored.** `write(s, 'latin1')` emitted
`0xc3 0xa9` where Node emits `0xe9`, corrupting the body and potentially
invalidating a `Content-Length` the handler had set. Both `write` and
`end` now parse their full overload set and pass the encoding through.
Node's own behaviour for an invalid encoding was established by probing
`ServerResponse` on Node 22 rather than assumed: it throws
`ERR_UNKNOWN_ENCODING` synchronously, so the adapter validates with
`Buffer.isEncoding` and lets `Buffer.from` raise the native error.

**Bodyless statuses were not suppressed.** Node discards writes after
`writeHead(204)` or `writeHead(304)`; this passed the buffered bytes to
the Fetch `Response` constructor, which throws for those statuses, so the
request became a 500 instead of the intended 204 or 304. Buffered data is
now discarded for 204, 205 and 304, and for `HEAD`.

None of the five reaches a caller today -- the document and runtime
handlers only ever end with JSON strings. They are traps laid for the
first handler that carries bytes, which is why they surfaced while
wiring the asset server backend, whose byte routes hit the first two
immediately.

Each is pinned by a test that fails when its fix is reverted.

* fix(storage): keep { client, bucket } working for the S3 byte store

`S3AssetByteStoreOptions.commands` was required and the store's methods
called it directly, so the published `./asset/s3-bytes` export stopped
accepting the `{ client, bucket }` construction it had always taken.
Anyone not going through `loadS3AssetByteStore` either failed to compile
or hit an undefined command factory at runtime.

Make `commands` optional. Omitted, the store resolves the AWS SDK's own
command constructors through a dynamic import on its first `write` /
`read` / `delete` and caches them, so `{ client, bucket }` works again.
Importing the module and constructing a store still never reach the
optional peer dependency: a deployment that does not select S3 never
resolves it. When the SDK cannot be resolved the call rejects naming
`@aws-sdk/client-s3` rather than crashing on an undefined property, and
the resolution is left uncached so installing the dependency fixes an
already-constructed store.

The lazily bound store is run against the byte-store contract, and the
new isolation tests pin both that construction does not resolve the SDK
and that an unresolvable SDK is reported by name.

With the break gone this branch's change to the package is additive, so
the version becomes a patch above main (0.2.4 -> 0.2.5) rather than the
minor it was carrying.

Refs #1007

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(persistence): run the asset collector in the shipped deployment

`PgAssetStore.remove`, and a `replace` that changes content, only stamp
`asset_blobs.unreferenced_at`. `AssetCollector.collect` is the only path
that deletes anything, and the persistence route deliberately did not
schedule it, on the grounds that scheduling is the deployment's decision.

The deployment this repository ships is docker-compose.yml: the app and
PostgreSQL, and nothing else that could make that decision. "The
deployment decides" therefore meant "it never runs", and ordinary asset
churn retained PostgreSQL bytes or S3 objects forever.

Schedule it from instrumentation.ts, which Next runs once per server
process — a route module can be instantiated more than once and has no
shutdown hook, so a schedule started from one is really started per
instantiation. Next 15 and later pick the file up with no config, so
next.config.ts is unchanged.

Collection is on by default with a 15-minute interval and the package's
one-hour grace period, so the Compose stack is correct with no operator
action. ASSET_COLLECTION_INTERVAL_MS, ASSET_COLLECTION_GRACE_MS, and
ASSET_COLLECTION_ENABLED change or disable it, each documented where it
is read. Without DATABASE_URL nothing is scheduled at all, and a failed
pass is logged and retried on the next tick rather than ending the
schedule or the process.

Several instances may collect concurrently: each candidate blob row is
re-checked and locked FOR UPDATE inside its own transaction, so
collectors serialize on the row rather than racing. The comment says so,
so nobody adds a distributed lock that is not needed.

The byte-store selection moves to lib/persistence/asset-byte-store so the
collector reclaims through the same layer the request path wrote through.
A collector on the PostgreSQL byte store while the route wrote to S3
would drop the blob row and orphan the object permanently.

Refs #1007

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(storage): bound each collector pass and report whether it filled

An unbounded collect() is sized by however long a deployment ran before
collection was scheduled: one statement selecting every eligible blob,
then a transaction and a byte-layer delete each, in a loop nothing
interrupts. The first pass over the Compose deployment that grew a
backlog before collection was scheduled is exactly that pass.

A pass now takes at most batchSize blobs (default one thousand) and
returns. Ordinary churn between two scheduled passes is far below the
cap, so a healthy deployment behaves as it did when a pass was
unbounded; a capped pass costs the remainder one scheduling interval,
which is what the interval is for.

collect() still answers with the count, and the count alone cannot tell
an empty backlog from a full batch -- a re-referenced or concurrently
taken candidate is skipped, so even a full batch can collect less than
batchSize. collectPass() returns the count together with capped, which
is exactly "this batch was full, run again"; a caller draining a
backlog loops while it is true.

Candidates are taken oldest-unreferenced first, with the content hash
as tiebreaker within one timestamp. Ordering by the hash alone would
starve: digests are uniformly distributed, so a high-sorting blob
waits behind every lower digest stamped after it, and under steady
arrivals those keep coming. The tests pin the ordering against a
planner that happens to answer the right rows without an ORDER BY.

Refs #1007

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(persistence): keep asset backend faults off the shared handler

Two review findings on the app wiring.

A concrete asset pool instance is single-lifecycle, but
resolveConfiguredAssetPoolStore handed the same object out again after
clearAssetPool() had closed it, reinstalling a dead store as the live
pool. The second handout now refuses loudly and points at the factory
form, which rebuilds on every resolution.

createPersistenceHandler eagerly awaited the asset byte store, so an
invalid ASSET_S3_BUCKET or an unresolvable AWS SDK rejected the shared
persistence handler and took document and runtime traffic down with an
optional backend. Byte-store construction is now deferred to the first
asset byte operation, and a failed construction is not cached, so the
next asset request retries -- the route's own no-poisoned-singleton
rule.

Refs #1007

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 13:01:16 +08:00
wyuc
6825363de1
feat(storage): server asset registry over a pluggable byte layer (#1007) (#1077)
* docs(storage): specify the asset registry HTTP contract (#1007)

Adds the contract document and the server-side store, byte-layer, and
handler types for the asset layer, the only layer with no server backend.
Runtime behaviour is unchanged: the new modules are deliberately
unreferenced until the backends land.

Reclamation moves offline, and that decision carries the design. `remove`
deletes a registry entry and nothing else -- it never reads, counts, or
deletes bytes. A separate collector takes byte rows that have had no
references for a grace period. Three things follow. The hardest race in
the design disappears, because no request can delete bytes another
request is adopting, so no request path needs a lock on the byte row and
the write path never has to reconcile taking one with never branching on
whether the bytes already existed. A side channel closes, because a
`remove` that only deletes a row costs the same whether or not another
principal holds the same bytes. And nothing is given up: under global
deduplication `remove` could never promise the bytes were destroyed, so
deleting them synchronously bought far less than it appeared to. The cost
is that storage grows until the collector runs, which a deployment must
configure rather than inherit.

Offline reclamation is also what makes the byte layer pluggable, so this
keeps #1007's pluggable-backend model rather than narrowing it. With
collection out of every request path, the requirement on a byte layer is
an ordering rule rather than an atomicity one: bytes are written before
the row that references them, deleted after the last row referencing
them, and the reference count is serialized on the blob row that every
implementation keeps regardless of where bytes sit. Two implementations
are planned -- bytes in a column of the transactional store, and an
object store keyed by content hash -- which is why the interface exists;
an interface with one implementation is not a seam, and this package has
removed such a seam before. The browser backend keeps no such seam,
because it reclaims inline and its bytes therefore cannot leave its
transaction; four comments that stated this as a storage difference now
state it as a reclamation one.

Metadata crosses the wire as a multipart part. It is open-ended and
callers already fill it with generated-narration text and image prompts,
so a query parameter would put unbounded user content into the request
target and into every intermediary's logs. That also motivates a rule
the other layers do not need: metadata must never appear in a URL, a
response header, a log line, or an error message. Metadata gets a value
domain, and the document is explicit that this is a capability
reduction -- the browser backend's structured clone carries Date, Map,
Set, and cycles faithfully, so a deployment moving to a server backend
must audit its metadata rather than expect a quietly lossy path to
tighten.

Resolution yields a client-minted object URL, superseding the resolution
that a server deployment resolves to a self-hosted proxy path. A media
element sends only ambient credentials, so a proxy path works for
cookie-session deployments and silently fails for header-authenticated
ones; closing that gap means minting URL tokens, a second credential
form inside a storage library. Fetching bytes over the client's own
authenticated request needs none and removes a class of hash-disclosure
surface. The client obligations that make it preserve the browser
backend's semantics are specified, and split into the two the shared
suite already pins and the four an HTTP backend must pin itself. The
cost, no range requests and no progressive playback, is stated.

Channels that could carry a principal or a content hash are closed
structurally rather than by enumeration: the route table admits no
segment that could carry either, and no route takes a query string at
all. Headers are closed by not reading them -- the contract assigns no
meaning to any header beyond content type and length, and a handler must
not derive identity from one outside its authenticate hook. Rejecting
header names by pattern was tried and removed: no pattern broad enough
to catch X-Remote-User spares User-Agent, and the forwarded-identity
headers such a rule would reject are exactly what an authenticating
reverse proxy injects for that hook to read.

Ids that cannot be a path segment are resolved by the client as misses
rather than raised as errors, a deliberate divergence from the KV
contract. An unknown id is a miss by this layer's id-domain rule and the
shared suite pins the empty string as one, so a client that threw would
fail that suite and reintroduce the shape oracle the rule prevents.

Also specified: response values may not derive from the shared byte row
(Last-Modified off that row is an existence oracle), a collision-
resistant digest is required because writes are unconditional and an
object layer keys on that digest, byte responses carry the revision and
refuse Range, HEAD errors carry the code in a header since they have no
body, quota exhaustion has a status and a code, and relabelling a
non-renderable type is not refusal -- resolve still returns a URL.

* feat(storage): add pluggable server asset store (#1007)

* fix(storage): write asset bytes only after claiming the blob row (#1007)

The write path wrote bytes twice: once before the registry transaction
and once inside it, after the upsert that claims the blob row. Only the
second one carries any safety, and the first is what a large-media
deployment pays for on every upload.

The ordering that matters is claim, write, reference. Claiming the blob
row takes its lock, so a collector already holding that lock finishes
before the write proceeds; writing bytes before the claim instead lets
the collector delete them while the claim waits, leaving a fresh entry
that points at nothing. Writing them before the entry keeps every
surviving entry backed by bytes that were actually stored. Dropping the
first write preserves both ends of that order and removes a crash window
that produced orphans for no reason.

For a byte layer inside the transactional store this also means a failed
write now leaves nothing at all, rather than an orphan the collector had
to sweep later.

The four tests this changed were each pinned to the removed write:

- the statement-sequence test hardcoded the old four-statement shape; it
  now pins claim, write, entry, and still asserts the two existence paths
  are identical, which is the property that matters;
- the quota test asserted two writes per accepted put;
- the crash-window test asserted an orphan that a transactional byte
  layer no longer produces. It is now two tests: a transactional layer
  leaves nothing, and a non-transactional one strands an object with no
  blob row -- which the collector deliberately cannot see, since
  recovering it is deployment housekeeping rather than reference
  counting. It also no longer depends on state left behind by earlier
  tests in the file;
- the collector-race test drove its interleaving off the removed write.
  It was choreography rather than concurrency in any case: PGlite is
  single-connection, and the test serialized transactions, so no row lock
  was ever contended. It is replaced by the two invariants that actually
  make the race safe -- the byte write is unconditional, and bytes the
  collector has already removed are re-stored by an adopting put. The
  contended-lock case belongs to the real-PostgreSQL suite, where it can
  be tested rather than mimed.

Reverting the unconditional write fails two of them, so they pin it.

The contract said a failed write leaves no partially written bytes while
its byte-layer section permits exactly that orphan; the two are
reconciled, and the ordering rule now states the claim step and the
unconditional write it depends on, along with the cost that follows --
the byte write happens with the registry transaction open, so an object
store holds a row lock across a network upload.

* test(storage): fix the real-PostgreSQL asset tests against the new write order

Both were written against the removed pre-transaction byte write, and
both failed once it went away -- one of them by hanging, which also
timed out the next test's TRUNCATE.

The lock-contention test waited for the adopting put to start writing
bytes before releasing the collector. Bytes are now written after the
upsert that claims the blob row, and the collector holds that lock, so
the write could never start: a circular wait. It now waits for a backend
to actually appear in pg_stat_activity blocked on a lock, which is the
condition it meant to wait for, observed rather than signalled. Its final
assertion carries real weight now -- had the bytes been written before
the claim, the collector would have deleted them and the adopting entry
would resolve to nothing.

The orphan test asserted that a failed registry transaction strands
bytes. With this byte layer inside the registry's transaction it strands
nothing, so it now asserts that instead. Object storage does strand an
object, and that case is covered where it belongs.

All of this ran against PostgreSQL 16: 781 tests pass with 3 skipped,
and those 3 pass against a real S3-compatible server. Nothing in the
suite is now unverified for lack of infrastructure.

* feat(storage): add asset HTTP backend (#1007)

* docs(storage): separate trusting a header from reading one (#1007)

The header rule said a server reads exactly Content-Type and
Content-Length, while the size rules require reading Content-Encoding in
order to reject it -- the two cannot both be literal, and the
implementation had to pick one.

What the rule means is narrower: no request header may be trusted to say
who is asking, outside the authenticate hook. Reading a header to decide
how to frame or refuse a request is a different thing, and the transport
headers are read and acted on.

* chore(storage): bump to 0.2.3 for the asset server backend (#1007)

Additive: new asset registry, byte layers, collector, HTTP handler and
client, and their entry points. Nothing existing is removed or changed
incompatibly, so this is a patch under the pre-1.0 rule.

* fix(storage): close four defects found reviewing the asset backend (#1007)

**Concurrent writes could exceed a principal's logical quota.** The check
ran on the pool before the write transaction, so two concurrent puts both
read the old total and both passed; enough concurrency amplified it
arbitrarily. It now runs inside the write transaction behind a
transaction-scoped advisory lock on the principal, taken only when a quota
is configured -- a branch on deployment configuration, never on data.
Pinned by a real-PostgreSQL test: four concurrent six-byte writes against
a ten-byte quota, of which exactly one may be accepted. Removing the lock
accepts three.

**A part name smuggled inside a quoted filename was read as a real part
name.** `Content-Disposition` was scanned with a regex that does not
understand quoted strings, so `filename="x; name=meta; y"` parsed as a
`name` parameter here while an RFC-aware intermediary sees an unnamed
part. That is exactly the parser differential the contract requires be
rejected. Replaced with a tokenizer that honours quoted strings and
escapes and requires exactly one real `name`.

**Any S3 404 was reported as an absent object.** `NoSuchBucket`, a
misdirected endpoint, and a revoked access point all answer 404 while the
bytes still exist, so a storage outage surfaced as `404 ASSET_NOT_FOUND`
and a caller clearing its reference on that would turn it into real data
loss -- the same failure the contract spells out for `401`. Only
key-absent codes map to a miss now; everything else propagates to
`500 INTERNAL_ERROR`.

**Exceeding `maxParts` answered `400` rather than `413`.** It is one of
the declared multipart resource limits and now answers like the other
three.

Verified against PostgreSQL 16 and a real S3-compatible server:
840 tests, none skipped.

* fix(storage): repair two defects the previous fix round introduced (#1007)

Both were created by the fixes themselves rather than surviving them,
which is the failure mode a second review round exists to catch.

**An over-quota replace answered 500.** Moving the quota check inside the
write transaction put it under a catch that re-threw only
AssetNotFoundError, so AssetQuotaExceededError was collapsed into a
generic registry failure and the handler mapped it to INTERNAL_ERROR --
losing the status and code the contract gives that condition. Both typed
errors now survive the catch.

**The hand-written disposition tokenizer accepted malformed input and
trusted it.** The quoted-value loop never checked that it found a closing
quote, so `name="meta` ran to the end of the header and returned `meta`;
`name="meta"junk` returned `meta` as well; and an extended `name*` form
competing with a plain one was silently ignored. Each recreates the
parser differential the tokenizer was written to remove. A quoted value
now requires its closing quote, rejects a dangling escape, and permits
only whitespace and a separator after it, and an extended name form is
refused rather than resolved.

Also widened the quota lock key from `hashtext` to `hashtextextended`.
The 32-bit form collides -- two unrelated principals sharing a key would
block each other for the whole transaction, which spans a byte write that
may be a network upload.

Regression tests cover all three: an over-quota replace raises the quota
error and leaves the original bytes, and each of the three malformed
dispositions is refused.

Verified against PostgreSQL 16 and a real S3-compatible server:
842 tests, none skipped.

* fix(storage): narrow the multipart disposition surface and the error boundary (#1007)

Three review rounds have each found a defect in the part-disposition
parser: a regex that misread quoted strings, then a tokenizer that
accepted an unterminated quote, then RFC 2231 continuation forms
(`name*0*=UTF-8''bytes`) and malformed empty parameter slots. Patching it
a fourth time would have been the wrong move.

This contract needs exactly one parameter, so it now accepts exactly one:
a part disposition is `form-data` plus a single `name` whose value is
`meta` or `bytes`, and everything else is a validation failure. That
removes the continuation forms, the encoded forms, the empty slots, and
`filename` -- the parameter every one of these attacks travelled in --
without having to model RFC 2231 at all.

That rule turned out to bind our own client too: it emitted a fixed
`filename="asset"`. Nothing reads it, and the contract already forbids
deriving a response disposition filename from caller data, so the client
no longer sends one.

Separately, `instanceof` was being used as provenance. The byte layer is
pluggable, so a byte store raising a same-named `AssetQuotaExceededError`
or `AssetNotFoundError` inside the transaction was re-thrown as a logical
registry outcome, carrying the byte layer's own message to a direct
caller. The registry's checks now raise module-private sentinels, and
only those are converted -- at the boundary, into fresh public errors
with this package's fixed messages. Anything else, whatever its name,
collapses to the generic registry failure.

Also pinned the advisory-lock key width. Reverting `hashtextextended` to
the 32-bit `hashtext` left every test green, so the emitted lock
statement is now asserted. A SQL-text assertion is deliberate here: the
behavioural difference is a collision between two particular keys under a
database-internal hash, which is not a stable thing to assert against.

Verified against PostgreSQL 16 and a real S3-compatible server:
845 tests, none skipped.

* fix(storage): define multipart disposition grammar (#1007)

* fix(storage): delegate multipart parsing to Fetch (#1007)

* fix(storage): require both multipart parts to be files (#1007)

Delegating the framing to the platform parser cost two rules for a
metadata part sent without a filename: such a part comes back as a
string with its headers discarded, so its `application/json` type could
no longer be checked, and its bytes had already been replacement-decoded
-- `{"x":"<0xFF>"}" was stored as `{"x":"\uFFFD"}` instead of refused.

The bytes part already had to be a file for binary safety. Metadata is
now symmetric: the client sends a fixed filename on both, the server
requires both to be files, and with the part preserved the media type is
checked and the bytes are decoded fatally before parsing. Neither
filename is read anywhere.

The limits are also described honestly now. `maxParts`, `maxMetaBytes`
and `maxAssetBytes` are validated after the parser has materialized
every part, so they bound what is accepted rather than what is parsed,
and `maxRequestBytes` is what caps memory. The contract said `maxParts`
bounded parser work; it does not. Restoring that would mean adopting a
streaming parser with its own grammar, which is the disagreement with
intermediaries that delegating to the platform exists to remove -- so
the trade is stated rather than reversed.

Also corrected the media-type wording: the two branches are metadata
present versus absent, and retention of an untyped replacement's prior
type is a browser-backend behaviour the HTTP path cannot reproduce,
because a conforming parser supplies a default type for a file part
whose header omits one.

Verified against PostgreSQL 16 and a real S3-compatible server:
853 tests, none skipped.

* docs(storage): the asset server backend ships in this branch (#1007)

The contract still described the server backend as not yet shipped, which
this branch is what changes. The conformance server remains test-only.

* fix(storage): make asset HEAD bytes-free and pin reads (#1007)

* fix(storage): harden asset write boundaries (#1007)

* chore(storage): bump to 0.2.4 for the asset server backend (#1007)

main merged in a 0.2.3 of its own, so this branch's bump no longer
increased the version. Additive relative to the merged base -- new asset
registry, byte layers, collector, HTTP handler and client, with nothing
removed or narrowed -- so a patch bump under the pre-1.0 rule.

---------

Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
2026-08-10 12:30:06 +08:00
wyuc
091989c48b
feat(storage): promote KV HTTP contract + clients to main (Part A of #1000) (#1020)
* feat(storage): KV HTTP contract and clients (#1000 Part A) (#1002)

* feat(storage): KV + Asset HTTP contracts and clients

Adds the client half of the server backends for the two components in the
storage RFC that still had a browser backend only.

Asset contract: the ref is `sha256-<hex>` over the bytes, computed by one
shared rule both backends call, and it *is* the address bytes are written
to (`PUT /assets/{ref}`). The server re-hashes what it received and refuses
a mismatch, so content addressing is enforced end to end rather than
assumed, and uploads become idempotent. `resolve` returns a URL per call,
either an absolute signed URL passed through untouched or a root-relative
proxied path joined to the base origin — the document never stores a raw
URL either way.

KV contract: account scope only, with no scope path segment, query
parameter or body field anywhere on the wire. `HttpAccountKV` is the only
object that can reach it and none of its methods takes a scope, so a
device value is not a request the transport can express; `HttpKVStore`
routes `device` to a local backend it requires at construction. Tests
assert the invariant both ways — `@ts-expect-error` on every scope-passing
call (unused directives fail the build if a scope is ever added) and a
fetch spy proving device traffic does not exist.

Both clients pass the existing `runKVStoreContract` and
`runStorageProviderContract` suites against an in-memory conformance
server, the asset one in both the proxied and signed URL shapes.

* fix(storage): close the review findings on the KV + asset HTTP clients

Cross-review found the device-scope guarantee resting on its weakest layer,
and several places where the contract promised more than the code or the
conformance server could deliver.

deviceStore: the two type-level layers only documented intent — the barrier
that actually kept device values off the wire was the injected store, typed
as `KVStore`, which `HttpAccountKV` satisfies structurally (its methods are
the same methods minus an optional parameter). Injecting the transport as
the device backend type-checked and put device values on the network. The
device scope now requires a branded `LocalKVStore`, the transport declares
itself not to be one, and a runtime check refuses what a cast smuggles past.

resolve URLs: a media `src` carries cookies, never the client's headers
hook, so the proxied shape is only sound for cookie-authenticated
deployments — the contract now says so instead of claiming the content route
sits behind "the same authentication", and the conformance reader uses a
cookie rather than a header no browser could send. The client also validates
what it resolves: http(s) schemes only, and no `//host` or `/\host` slipping
through the relative branch to another origin.

Identifiers: refs and keys arrive percent-decoded, so `..%2F..%2F` was a
traversal waiting for the first backend to join it to a path. Both are now
constrained on both sides, and the contract states the MUST that keeps a
storage layer from being the first to look.

Also: the JSON gate runs before anything reads the value (a `JSON.stringify`
pre-flight ran caller code first and reclassified `{ toJSON: () => undefined }`
as a delete); scope routing fails closed; an upload's Content-Type can no
longer be relabelled by the headers hook; a mutation invalidates in-flight
resolution; unparseable 2xx bodies become typed errors rather than native
SyntaxErrors; signed tokens are validated rather than merely present; and the
contracts settle the per-principal claim model, the media-type allowlist, the
literal prefix rule, and the identifier charset that Part B would otherwise
have had to guess at.

* fix(storage): close the second review round on the KV + asset HTTP clients

The device-scope guard was in one place and needed to be in three. The
transport is exported and structurally a `KVStore`, so `kvPersistStorage(new
HttpAccountKV(...), 'device')` and a plain `KVStore`-typed reference both
compiled without a cast, dropped the scope argument on the floor, and put a
device value on the wire. The transport now takes the scope it was pretending
not to have — typed as `'account'`, refused at runtime for anything else, since
bivariant method parameters mean the type alone cannot hold — and the persist
adapter requires a `LocalKVStore` when paired with `'device'`.

The resolved-URL checks were text tests where only parsing works. The URL
parser strips ASCII tab, LF and CR before parsing, so `/<tab>/evil/x` walked
past the `//` prefix check and loaded cross-origin — in the app-mounted shape,
where a path-only base URL leaves that check as the only guard. The absolute
branch let through `https://` with no host and `https:host/x`, which names one
place standalone and another when a browser resolves it against a same-scheme
page. Both branches are now decided by the parser: control characters refused
outright, relative references validated against a probe origin, absolute ones
required to parse as `scheme://host` with an http(s) scheme — and still
returned byte-for-byte, so signatures survive.

The media-type MUST had no implementation and contradicted itself: the
conformance server stored and served `text/html` verbatim, and a metadata-less
upload became `application/octet-stream`, which cannot be on a renderable
allowlist. The server now implements the allowlist, serves everything else as
an opaque attachment, and the contract scopes its "usable as an img src"
promise to allowlisted types.

The KV key domain was HTTP-only, which made "the same suite proves the backends
equivalent" untrue: `a/b`, an empty key and a 600-byte key all passed in the
browser and would have become unreachable on a server. An audit of the app's
keys (learner key, storage generation, current-scene and migration markers,
persist names) found none in violation, so the rules are lifted to the
primitive and both backends enforce them, with the shared suite asserting it.

Also: the claim model the last round wrote into the contract is now implemented
and covered case by case; malformed percent-escapes are a 400 rather than a
500; the in-flight identity guard has a test that can fail; and the Content-Type
conflict raises a typed error.

* fix(storage): final review round — backend parity, URL probe, credentials

Four small things the final pass found, plus the contract patches Part B would
otherwise inherit.

The `toJSON` delete semantics were documented backwards. `BrowserKVStore.set`
was the trial-serialize the KV contract said no backend does, so
`{ toJSON: () => undefined }` was a silent delete there and a rejection over
HTTP — opposite answers to the same call. The browser backend now validates
scope and key before anything can run caller code, recognizes the delete case by
type, and refuses a value that still fails to serialize. Both cases are in the
shared suite, so the two backends cannot drift apart again.

The probe-origin check could not vouch for a URL naming the probe host itself:
`//asset-url-probe.invalid/x` resolves onto the probe origin and passed, then
resolves somewhere else entirely against a real base. Any authority-bearing
form is refused outright, so the check no longer depends on the probe host
being unguessable.

Cross-origin cookie deployments had no way to send a cookie: `fetch` omits
them cross-origin by default and the headers hook cannot help, since `Cookie`
is a forbidden header name. Both clients accept `credentials` and pass it to
`fetch`; the contract records the CORS obligations that come with it. Rather
than restricting cookie deployments to same-origin, which the asset contract
explicitly does not require.

The conformance server's dot-segment check was reading a path the URL parser
had already normalized, so `/assets/%2e%2e/x` arrived as `/x` and the documented
400 was really post-routing behaviour. It now validates the raw request target,
and the injected transport preserves it instead of handing over a normalized one.

Contract: the prefix rule names backslash (PostgreSQL's default LIKE escape,
and the one an implementer skips because a legal key cannot contain it, though
a prefix can), and acquiring a claim MUST leave the bytes present — a
de-duplicating server that inserts only the claim can race reclamation of the
last previous claim and leave a document resolving to a 404.

Also: base URLs are checked for an http(s) scheme at construction, the ref
domain moved next to the KV key domain and is enforced by both asset backends
with shared suite coverage, the KV client refuses a hook-supplied Content-Type
like the asset one, and CI runs the package typecheck that makes the
`@ts-expect-error` guards mean something.

* fix(storage): probe guard spelling, prefix validator, base url shape

Three one-line-level holes from the closing review.

The probe-host guard was spelled `startsWith('//')`, so the backslash forms
`/\host/x` and `/\/host/x` reached the origin comparison, passed it by naming
the probe host, and were stopped only by that host failing to resolve. It now
refuses any second character that can open an authority — which is also what
makes the comment true rather than aspirational: a path-absolute reference
always inherits the base's origin, so with no authority expressible the
comparison cannot be satisfied cross-origin, whatever the probe host is.

The conformance server validated `?prefix` with the path-segment validator,
rejecting `keys('.')` and `keys('..')` — legitimate prefixes of keys like
`.hidden`, which the client rightly allows. A prefix travels in the query
string and is never a path segment, so it now has its own validator: the key
rules minus the non-empty and dot-segment requirements. The contract says the
same, and the shared suite pins both backends to it.

Base URL validation moved into one shared helper both clients call, since two
copies of a security rule is how they drift. It now also refuses
`https:host/path` — which parses standalone to that host but is resolved
against the current document by `fetch`, sending requests and their credentials
somewhere the operator never wrote — and a base carrying a query or fragment,
which silently swallows the path concatenated onto it.

* fix(storage): close two holes in the shared base-url check

The path-only branch returned its input unexamined, so `//evil.example` and
`/\evil.example` were accepted as paths. A browser reads both as an authority,
which means every request built on such a base — and every credential the
headers hook attaches — leaves the origin. A base is now required to be a
single `/` followed by something that is not `/` or `\`, the same refusal the
resolved-URL check already makes, for the same reason. Control characters are
refused first: the parser strips or re-encodes them, and `/<tab>/evil.example`
becomes `//evil.example` after it does.

The query/fragment check read the parsed `search` and `hash`, which are empty
strings for a base ending in a bare `?` or `#` — so exactly the inputs that
swallow the concatenated path into a query or fragment were the ones that got
through. It reads the raw string now, which also makes the post-parse check
redundant rather than merely unreachable.

* fix(storage): bypass HTTP caches on reads, and require a 404 for not-found

Two mechanisms the clients were missing, on both sides.

Reads are marked `cache: 'no-store'`. The asset resolve request answers two
mutable questions — does this principal hold a claim, and which URL is valid
now — so a cached answer returns an expired signed URL or replays a negatively
cached 404 for an asset written since; the in-flight coalescing that would
otherwise cover this lives in the client and cannot reach the HTTP cache layer.
KV reads have the same problem for a different reason: `account` is by
definition the scope another device may have just written, and nothing here can
invalidate a cached copy when it does. The contracts now require `no-store` on
those responses. Asset content is left cacheable — bytes at a content-addressed
ref never change — and writes are unmarked, having nothing to read from a cache.

The not-found mappings now require the status as well as the code. A proxy or
gateway answering 401, 403 or 500 while echoing the body's error code was being
read as "this does not exist", and `null` is indistinguishable from a real
miss — so an outage reached the caller as data loss.

* fix(storage): reject unusable base urls, and make the harness meet the contract

Two configuration checks and two harness gaps.

A base URL carrying userinfo built a client that threw a native TypeError on
every operation, because Fetch refuses to construct a request from a URL with
embedded credentials — one legible error where the deployment is configured
beats a per-call failure with no explanation. A base with surrounding
whitespace was worse than useless: `new URL` strips it, so validation passed,
and then concatenation moved the space inside the URL where it is no longer
trailing and no longer stripped, breaking every request.

The conformance server now sends `Cache-Control: no-store` on the three reads
the contracts require it on. The requirement landed last round with nothing
asserting it, so the suite would have passed against exactly the implementation
it exists to catch. And `DELETE /kv/entries/{key}` refuses a request body
outright: the query and the header were already checked for a smuggled scope,
leaving the body as the one channel through which a delete could still describe
one.

* fix(storage): allow a device-routing composite to persist device-scoped state

cosarah found kvPersistStorage(httpKVStore, "device") wrongly rejected. An
HttpKVStore is a full dual-scope composite: it routes device operations to the
LocalKVStore it requires at construction and only reaches the network for
account, so device values never leave the machine. The guard tested
isLocalKVStore — "is the whole store local" — and so turned away a store that
is safe for device precisely because it is not fully local.

The guard now turns on the capability that actually matters: does the device
scope stay local. A new DeviceSafeKVStore brand (servesDeviceScopeLocally: true)
marks a store whose device writes never leave the device — held by a fully-local
LocalKVStore (which extends it) and by HttpKVStore, and declared false by
HttpAccountKV, the pure account transport with no local device backend. The
persist device overload requires DeviceSafeKVStore; the account overload is
unchanged.

The security boundary the red team opened this whole line on is untouched:
HttpAccountKV used for device is still doubly refused — its false brand is not
assignable to the true DeviceSafeKVStore requires (type), and the runtime check
rejects the cast that would erase it. The brand is structurally unforgeable, not
a flag the transport could set; and the injected device backend of a composite
still demands the stronger, fully-local LocalKVStore, so composites cannot nest
into a router loop with no local floor.

Regression test (cosarah's ask): kvPersistStorage(httpKVStore, "device")
set/get/remove reach the injected device backend, land device-scoped there, and
make zero HTTP requests. Mutation-checked: widening the brand so HttpAccountKV
is accepted, or neutering the guard, turns the device-out reject test red.

* refactor(storage): scope #1002 down to KV — remove the asset HTTP backend

Per the maintainer decision, the AssetProvider server design moves to the
global-resource-pool RFC (#1007), where AssetRef changes from content-addressed
to an allocated global id and the contract model itself changes. Continuing to
harden the content-addressed asset client here would be work on code that is
being replaced, so the asset half of #1002 is withdrawn and the PR is pure KV.

Removed (all introduced by #1002):
- docs/asset-http-contract.md
- src/asset/http.ts (HttpAssetProvider) and its index / package.json exports
- src/asset/content-ref.ts (assertAssetRef / computeAssetRef / the ref shape)
- test/http-asset-provider.test.ts and the asset half of the conformance server

Reverted to the #858 baseline, untouched:
- src/asset/browser.ts — back to opaque resolve/remove, no assertAssetRef guard.
  This directly removes cosarah's latest asset finding: an opaque ref such as
  `bucket/path` no longer throws, restoring the DSL AssetRef-opaque contract.
- test/asset-contract.ts, test/asset-browser.test.ts — 0-diff from origin/main.

Kept (all KV): kv-http-contract.md, src/kv/* (HttpKVStore / HttpAccountKV /
the DeviceSafeKVStore brand), the KV tests, and the KV conformance server —
renamed kv-conformance-server.ts and reduced to KV routing over the shared
harness framework (body caps, no-store, scope-channel and dot-segment guards,
auth hooks). base-url.ts stays; KV still uses it.

* fix(storage): treat KV keys as opaque; document account-sync prerequisites

cosarah found the key validation we added for traversal defense breaks
legitimate callers: DSL identifiers (stageId/sceneId/learnerKey) are
unconstrained strings — verified, the DSL only requires typeof === 'string' —
and app callers compose keys like `editor-current-scene:<stageId>`, so a valid
stage id `stage/one` now throws in BrowserKVStore before localStorage.

Fix, matching the asset-opaque precedent: a KV key is an opaque string again.
The `/` and `\` rejection is removed from assertKVKey / assertKVKeyPrefix and
from the conformance server. Traversal is defended by encoding, not rejection —
the HTTP client already percent-encodes the whole key into one path segment, so
a separator never opens a new segment on the wire, and the server stores the
decoded key as a plain map key (bound value), never a path. `a%2F..%2F..%2Fb`
decodes to the single opaque key `a/../../b`, which addresses one entry and
traverses nothing; a raw unencoded `..` path produces extra segments that match
no route (404). The rules that remain are only what encoding cannot cover and
the transport cannot carry (empty, whole-key `.`/`..`, NUL, lone surrogate,
length) — none producible by `prefix + id`.

Tests: the shared suite now round-trips keys with `/`, `\`, `%`, `:`, spaces
across both backends; a `stage/one` key round-trips through the real conformance
server; the traversal input is proven to store as one opaque key and the raw
path to 404. Mutation-checked: re-adding the `/` rejection reddens exactly the
round-trip cases.

Docs (kv-http-contract.md):
- "A key is an opaque string" replaces the identifier-shape section — keys may
  contain separators; traversal is an encoding + server-bound-value duty.
- Account settings sync now documents its security prerequisite next to the
  account scope: a deployment MUST provide real per-user isolation of stored
  rows and encryption at rest before enabling sync; the reference server is not
  that posture.
- kvPersistStorage(device) doc corrected from LocalKVStore to DeviceSafeKVStore
  (the device-safe brand accepts the HttpKVStore composite; HttpAccountKV stays
  double-rejected).

* fix(storage): close the scope/key/prefix contract gaps with a coverage matrix

Three completeness gaps, nailed by an executable matrix so an untested
channel/shape can't slip through the next pass rather than patched one by one.

1. Scope via a GET body was not rejected. `assertNoScopeChannel` covered the
   query and header, but the bodyless GET routes never read their body, so
   `GET /kv/keys` with `{"scope":"device"}` returned 200 []. Bodyless methods
   (GET, DELETE) now reject any request body up front — that is the channel a
   scope would hide in — and only `set` reads a body, checking it for a scope
   field. The DELETE route's own body check folds into the shared one.

2. The 512-byte key bound wrongly rejected legitimate keys. A DSL id is an
   unconstrained string (verified: the DSL requires only typeof === 'string'),
   and `editor-current-scene:` + a 500-char stage id is 521 bytes, which
   BrowserKVStore rejected before localStorage. Keys are length-unconstrained;
   the bound is raised to a generous 8 KiB DoS ceiling and documented as a
   resource guard, not part of the key domain — not a derived/hashed key, which
   would break the literal-prefix semantics of keys(prefix). Both backends and
   the conformance server move in lockstep.

3. The prefix contract text contradicted itself: it claimed a legal key can
   never contain a backslash, while the key section (correctly) makes keys
   opaque, backslash included. Reworded — a prefix is opaque like a key — and
   the shared literal-prefix matrix now covers `\` (PostgreSQL's LIKE escape),
   which it had skipped, alongside `%`, `_`, and combinations.

Coverage matrix (CI-executable):
- scope-rejection: {get,set,remove,keys} × {query,header,body}. query/header
  run in-process; the body cells use a raw HTTP client (fetch cannot put a body
  on a GET — exactly the non-conforming client the finding described) against a
  loopback listener, skipped where the sandbox forbids binding.
- key-validity: `/ \ % _ : .` space, BMP + astral Unicode, `%2F`, a 500-char
  composed key, and all-at-once — each round-trips (set/get, keys(prefix)
  match, remove) on both browser and HTTP backends via the shared suite.
- prefix-charset: `%`, `_`, `\`, and combinations match literally, both backends.
- Boundaries still rejected: whole-key `.`/`..`, NUL, lone surrogate, empty,
  and the DoS ceiling.

Mutation-checked: removing the bodyless-body rejection reddens the three GET/
DELETE body cells; lowering the ceiling back to 512 reddens the long-key
round-trip on both backends.

Bumps @openmaic/storage to 0.1.2 (src/kv/* changed).

* fix(storage): bound the KV key DoS ceiling by encoded size, not UTF-8 bytes

Follow-up to the 8 KiB ceiling: measuring it in UTF-8 bytes ignored
percent-encoding expansion and reopened a backend-parity break. A non-ASCII key
is small in bytes but large on the wire — `€` is 3 UTF-8 bytes and `%E2%82%AC`,
9, encoded — so `'€'.repeat(2730)` (8190 bytes, under the old cap) encoded to
~24.6 KB, which the Node HTTP server rejects with 431 before routing while
`BrowserKVStore`, which never builds a URL, accepted it.

The ceiling is now measured on `encodeURIComponent(key).length` — the size the
key occupies as a request target — at 4096, far above any real id (a 500-char id
under `editor-current-scene:` encodes to ~523) and well below Node's default
16 KiB `maxHeaderSize` (empirically the 431 trigger on Node 22), so no key that
passes validation can reach a 431. Both backends and the conformance server use
the same encoded-size limit, so a key is accepted iff its encoded form fits,
whichever backend it reaches — parity restored. `MAX_KV_KEY_BYTES` becomes
`MAX_KV_KEY_ENCODED_LENGTH`.

Tests, against a real node:http server for the 431 boundary:
- the max-legal non-ASCII key (455×€, encoded 4095) round-trips over real HTTP
  with no 431, and the browser accepts the same key — parity on accept;
- one code point over (456×€, encoded 4104) is rejected by both backends at
  validation, before any request — the exact parity case a byte bound missed
  (1368 bytes, would have passed, then 431'd over HTTP);
- a raw client bypassing client validation with an over-ceiling (under 16 KiB)
  encoded key gets a clean 400, not a 431.
Shared suite's key-domain matrix now bounds by encoded size on both backends.

Mutation-checked: reverting the ceiling to UTF-8 bytes reddens the three
encoded-length parity tests. Bumps @openmaic/storage to 0.1.3 (src/kv/* changed).

* fix(storage): make the KV key domain truly opaque; keep transport limits out

The recurring key-limit findings (512 → 8 KiB → 4096, each patch exposing the
next) shared one root cause: transport-driven rules kept being added to the key
domain — the browser primitive included — breaking the "opaque unconstrained
string key" contract. This is the structural fix, per cosarah: "keep HTTP
transport limits out of the browser primitive and preserve the existing string
key domain."

Approach (a), the minimal one: a KV key is any string, validated nowhere. The
browser primitive is a Map/Storage that stores any key; the HTTP backend carries
the key as a URL path segment and inherits that transport's limits, documented
as HTTP-deployment concerns that never constrain the key domain. Every real key
(a `prefix:id` from an unconstrained DSL string) round-trips on both backends;
only pathological keys hit a transport limit, and none of those are keys a caller
produces. (b) — keys in the request body for accept-parity — was unnecessary:
verified that no real key needs it.

Removed from the key domain (browser + HTTP + conformance server):
- the encoded-size / length ceiling entirely — there is no key-length limit;
- the NUL exclusion — NUL percent-encodes to %00, is preserved end to end, and
  is a legitimate Map/Storage key (a false "cannot represent" premise);
- the `.`/`..`/empty/`/`/`\` rejections — all opaque, all round-trip.
`assertKVKey` / `assertKVKeyPrefix` / `MAX_KV_KEY_*` are gone.

Kept, at the transport layer only (HttpAccountKV), never reaching the primitive:
- an unpaired UTF-16 surrogate genuinely has no percent-encoding, so the HTTP
  client surfaces a clear transport error (KEY_NOT_ENCODABLE) while the browser
  stores the same key — verified: encodeURIComponent throws on it.

Scope completeness (P2): the "reject scope in every channel" guarantee had two
holes — a scope path segment (`/kv/device/keys`) 404'd instead of 400, and only
`x-scope` was checked among header spellings. The server now rejects a scope
path segment and the whole prohibited header set (scope, x-scope, kv-scope,
x-kv-scope). The coverage matrix is {get,set,remove,keys} × {path, query, each
header spelling, body}.

Coverage matrix reworked to verify arbitrary opaque keys (empty, NUL, control
chars, separators, long, Unicode) round-trip on both backends — not to verify a
limit. A real node:http test proves NUL and long keys round-trip end to end, and
that a lone surrogate is a transport limit (browser accepts, HTTP client errors).
Mutation-checked: re-adding a NUL rejection, removing the path-segment check, or
dropping a header spelling each reddens the matching cells.

Cleanup: package.json description updated (KV HTTP delivered, not "later");
network-server bind/skip/close lifecycle deduped into one helper. Bumps
@openmaic/storage to 0.1.4 (src/kv/* changed).

* fix(storage): reject whole-key "."/".." in the HTTP client — they alias entries

Follow-up to making the key domain opaque: removing the `.`/`..` key-domain
rejection was right (the browser stores them fine), but the HTTP client then let
them reach `fetch`, where URL path normalization silently rewrites the target
before it is sent. `encodeURIComponent('.')` is `.`, so `/kv/entries/.` collapses
to `/kv/entries/` (the empty key) and `/kv/entries/..` walks up a level — so
`get`/`set`/`remove` on a whole-key `.` or `..` silently read, overwrite, or
delete a *different* entry, with no error. A data-correctness bug.

Fix, same pattern as the unpaired-surrogate limit already handled: a whole-key
`.`/`..` is something the URL-path transport structurally cannot carry, so the
client's path-segment encoder refuses it with `KEY_NOT_ENCODABLE` before a
request is built. Split the encoder in two: `encodeKeyPathSegment` (key → path
segment, rejects whole-key `.`/`..`) and `encodeComponent` (the `keys()` prefix →
query value, where `.`/`..` are legitimate and untouched). A key that merely
*contains* a dot (`a.b`, `prefix:id`) is unaffected and round-trips. The browser
primitive still stores `.`/`..` opaquely.

The HTTP transport's limits are now complete — three cases, all fail-loud at the
transport layer, all storable on the browser: (1) an unpaired surrogate (no
percent-encoding), (2) a whole-key `.`/`..` (URL normalization aliases it), and
(3) an over-long key (deployment request-target ceiling). The contract lists all
three as HTTP-deployment limits, not key-domain constraints.

Tests: whole-key `.`/`..` — browser stores and round-trips (distinct from the
empty key); HTTP client throws `KEY_NOT_ENCODABLE` and builds no request; and a
real-server test asserts the empty-key entry is never read, overwritten, or
deleted by a `.`/`..` operation. A dot-containing key (`a.b`) round-trips.
Mutation-checked: dropping the whole-key check reddens the alias tests.

Bumps @openmaic/storage to 0.1.5 (src/kv/* changed).

* fix(storage): reconcile the settings/profile persist consumer with the KV API

The integration merge (#1002 KV into main) surfaced two things the merge itself
could not resolve mechanically:

- `lib/store/kv-persist.ts` (added on main by #1001) called
  `kvPersistStorage(kv, scope)` with a *variable* scope, which #1002's overloads
  deliberately reject — `'device'` must be paired with a device-safe store, so
  the overloads take the scope as a literal. Branch on the scope: `'account'`
  takes any `KVStore`; `'device'` narrows the backend to `DeviceSafeKVStore` via
  its brand (the adapter re-checks at runtime) and otherwise fails loud. The app
  wires only `'account'` today; the `'device'` arm keeps the generic helper
  honest. `ControllableKV` in the persist test wraps a local `BrowserKVStore`,
  so it is genuinely device-safe and now declares the brand.

- `packages/@openmaic/storage` version bumped 0.1.5 → 0.1.6: the merge changed
  storage inputs (README + a pg-schema test carried from main) relative to the
  previous integration tip, so the version-bump guard requires an increase above
  #1002's 0.1.5. Still ahead of main's 0.1.1.

Verified: root `tsc --noEmit` clean, the persist test green (40 passed), and the
storage package suite green on Node 22.
2026-07-30 22:31:17 -04:00
wyuc
e39c64cd9e
feat(persistence): one-command server-backed stack (compose profile + embedded API + docs) (#982)
* feat(persistence): one-command server-backed stack — embedded API + compose postgres profile

docker compose --profile server-persistence up --build runs exactly two
containers: the app (persistence HTTP server embedded at /api/persistence,
enabled only when DATABASE_URL is set — unset keeps today's browser-only
behavior byte-for-byte) and PostgreSQL 16. Client bootstrap configures both
storage seams against the same-origin route when NEXT_PUBLIC_PERSISTENCE=1,
riding the existing lazy migrations so prior browser data reaches the
server per-course on first open. Dev auth (bearer token + client-asserted
learner header) lives in one overridable module and is loudly documented
as development-grade. Verified end-to-end against the real stack:
document PUT/GET and runtime session create land in Postgres; bad tokens 401.

Co-authored-by: Codex <codex@openai.com>

* fix(persistence): review round — structural bootstrap ordering, retryable init, honest failure surfaces

The bootstrap side-effect import moves into the two storage seam entry
modules, so any module graph that can resolve a store necessarily
evaluates it first — no client-entry ordering luck, and the two configure
calls are all-or-nothing. A failed server init clears its cached promise
(next request retries instead of permanent 500s). listStages rethrows
backend failures and the dashboard surfaces a persistence-unavailable
error instead of rendering an empty list that reads as data loss. Docs
state the build-time nature of NEXT_PUBLIC_PERSISTENCE, the real token
boundary (any visitor can extract it and impersonate any learner —
localhost/trusted-network only), and the correct postgres password
rotation. Dev-token compare is timing-safe; the handler singleton lives
on globalThis so HMR cannot leak pools.

Co-authored-by: Codex <codex@openai.com>

* fix(persistence): review response — no-confidentiality wording + adapter round-trip coverage

The token docs now state plainly that the NEXT_PUBLIC token is compiled
into the public bundle and provides no confidentiality or user isolation
(its only role is keeping unrelated scanners off a trusted network). The
Fetch/Node adapter — the route's most bug-prone code — gains a round-trip
integration test: PUT body streaming, 201 with multi-value headers, 204
empty body, and encoded path segments reaching the handler un-decoded.

---------

Co-authored-by: Codex <codex@openai.com>
2026-07-23 16:28:03 +08:00
wyuc
5cef056d17
feat(storage): server-backed documents — HTTP contract, Postgres backend, one reference server for both stores (#979)
* ci: whitelist the document-server-backend integration trunk

* feat(storage): DocumentStore HTTP contract + client + reference-server routes (1B PR1) (#975)

Mirrors the runtime line's server-backed delivery for the document store:
a documented /documents HTTP contract (validation/not-found/FUTURE_VERSION
taxonomy; authorization stays a deployment concern via injected hooks with
an any-authenticated default), an HttpDocumentStore that runs the same
validators and migrate-on-read semantics as the browser backend before and
after the wire, and document routes on the shared reference server — one
server, both stores. The backend-agnostic document contract suite now runs
against HTTP → reference server → browser store.

Co-authored-by: Codex <codex@openai.com>

* ci: whitelist the trunk in the storage PG contract workflow too

* feat(storage): PgDocumentStore — Postgres document backend (1B PR2) (#976)

Implements the full DocumentStore contract over document_stages /
document_scenes / document_outlines, reusing the pure adapter split so
every backend shares one aggregate semantics: saveDocument replaces the
aggregate in one transaction under the stage row's FOR UPDATE lock
(vanished scenes deleted, outline row added/removed — omission means
deletion), incremental putStage/putScene/deleteScene take the same
current-version gate as the browser backend, loads read FOR SHARE and
migrate without writing back, listings never touch content, and deletes
cascade by FK. The document contract suite is PG-gated the same way the
runtime one is; local coverage comes from stubbed-queryable unit tests.

Co-authored-by: Codex <codex@openai.com>

* feat(document-store): sealed bootstrap injection seam for the document backend (1B PR3) (#977)

configureDocumentStorage mirrors the runtime seam: module-level bootstrap
injection sealed after first resolution, a factory that receives the app
validators so a server-backed store cannot silently skip them, and
reset-for-tests wiring. A deployment swaps the browser document backend
for HttpDocumentStore without touching app code. Storage README roadmap
updated — the document HTTP/PG backend items are done.

Co-authored-by: Codex <codex@openai.com>

* fix(storage): 1B final-review hardening — error classification, JSON-safety, body limits, parity coverage (#978)

Unversioned legacy incremental writes classify as 400 instead of leaking a
500; document/scene reads assert JSON safety before serialization so a
structured-clone value fails loud (500 NOT_JSON_SAFE naming the path)
instead of silently converting; a shared streaming body reader bounds
request bodies (default 32 MiB, 413 PAYLOAD_TOO_LARGE, option exposed
through every handler factory); the putStage stale/future guards and
corrupt-stamp listing tolerance move into the shared contract suite so all
four backends prove them; both HTTP store errors retain structured details.

Co-authored-by: Codex <codex@openai.com>

* refactor(storage): typed document version/not-found errors replace prose classification (#980)

DocumentVersionError (future | not-current, storedVersion) and
DocumentNotFoundError now carry the classification the server previously
regex-derived from message prose — instanceof matching mirrors the runtime
sibling's RuntimeAppendConflictError pattern, so a wording change can no
longer silently break status mapping. The prose fallback remains, after
the typed checks, solely for third-party DocumentStore implementations.
The HTTP client rematerializes 409s as DocumentVersionError for symmetry,
and http.ts's private validator fields are renamed to match the public
option names all three backends already share.

Co-authored-by: Codex <codex@openai.com>

---------

Co-authored-by: Codex <codex@openai.com>
2026-07-23 12:56:26 +08:00
wyuc
3eea9dc542
feat(runtime): complete the #869 learner-data cutover — quiz + playback onto RuntimeStore (#955)
* feat(runtime): persist quiz attempts in RuntimeStore

* fix(runtime): coalesce quiz draft snapshots

* fix(runtime): recover concurrent quiz attempts

* fix(runtime): handle quiz completion races

* fix(runtime): commit quiz review atomically

* fix(runtime): dedupe concurrent quiz writes

* fix(runtime): drop stale quiz drafts

* fix: harden quiz runtime review recovery

* fix: serialize quiz attempt identity

* feat: read quiz state from runtime store

* fix: persist quiz retries before resetting

* fix: preserve authoritative quiz outcomes

* fix: preserve legacy quiz retries during cutover

* fix: reconcile legacy quiz snapshots safely

* fix: drain rollover quiz write queues

* fix: drain completed quiz retry queues

* fix: reuse concurrent quiz retries

* fix(quiz): preserve drafts across abrupt reloads

* fix(quiz): recover empty retry sessions

* fix(chat): abort stalled runtime state reads

* fix(quiz): expose queued phases to readers

* fix(quiz): retain concurrent writer tails

* fix(quiz): canonicalize retry branches

* fix(quiz): keep retry rollovers monotonic

* fix(quiz): validate skipped retry siblings

* fix(quiz): close read cutover races

* test(classroom): cover legacy quiz summaries

* fix(quiz): reset async consumers on scene changes

* fix(quiz): close scene transition windows

* test(pbl): cover launch freshness guards

* fix(quiz): close cutover concurrency gaps

* fix(quiz): harden retry and context freshness

* fix(quiz): reject malformed legacy answers

* fix(quiz): validate legacy answer values

* test(quiz): cover legacy multi-answer migration

* fix(merge): retire dead ChatRequestTemplate.storeState after quiz read cutover

Main's three call sites built static storeState blocks that runAgentLoopFn
never consumed (it always rebuilds fresh state via getStoreState); the quiz
read cutover replaced that callback with the async two-phase RuntimeStore
read, leaving the template field with zero consumers. Drop it.

* feat(storage): conform HTTP/PG backends and reference server to RuntimeAppendOptions

The quiz write path's expectedLastSeq / sessionTransition / RuntimeAppendConflictError
semantics existed only in the browser backend; server-backed deployments would
silently accept conflicting appends and leave completed sessions active. Forward
the options over the wire, detect conflicts atomically under the PG transaction,
map them to HTTP 409 RUNTIME_APPEND_CONFLICT, and rematerialize the typed error
client-side so quiz retry logic works across every backend.

Co-authored-by: Codex <codex@openai.com>

* fix(chat): Pi single requests build storeState via the async runtime quiz read

Pi bypasses runAgentLoop's per-iteration getStoreState and serializes the
request template straight to /api/chat/pi, which rejects bodies without
storeState. Extract the fresh-snapshot builder (async RuntimeStore quiz read
with the scene-transition guard) and call it on the Pi path too.

* ci: whitelist the runtime-data-cutover integration trunk for PR checks

* feat(runtime): playback cutover — cursor in KV, discussion facts in RuntimeStore (#956)

* feat(runtime): cut playback over to the runtime layer — cursor in KV, facts in RuntimeStore (#869)

The fourth and last runtime family. Consumed-discussion facts become
append-only 'playback' records folded into a set at read (at-least-once
appends, no conflict machinery); the resume cursor is device-scoped
last-write-wins KV per the amended #779/#869 split. sessionStorage keeps
same-tab priority; KV takes over on fresh tabs/reloads. The dead Dexie
playbackState machinery is retired, with a one-time lazy migration of any
legacy row (cursor half + facts half) before deletion, and stage deletion
now clears both the KV cursor and any unmigrated legacy row.

Co-authored-by: Codex <codex@openai.com>

* test(runtime): include playbackState in the stage-delete db mock

---------

Co-authored-by: Codex <codex@openai.com>

* fix(playback): persist discussion facts on every consumption path (#957)

* fix(playback): persist discussion facts on every consumption path (final-review P0+P1s)

- The engine now publishes a progress snapshot the moment a discussion is
  consumed (join / skip / unselected-agent auto-skip). onProgress otherwise
  fires before the discussion action executes and a discussion is the scene's
  last action, so the fact never reached persistence.
- Reads fold records across ALL playback sessions in the learner partition
  (mergeLearner deliberately preserves same-kind sessions from both keys).
- Legacy migration appends only not-yet-durable facts, so an interrupted
  migration resumes instead of dropping the tail.
- recordConsumedDiscussion reports durability; the component drops failed ids
  from its observed set so a later progress tick retries (at-least-once).

* test(e2e): live verification of the playback persistence chain

Seeds a deterministic stage straight into the Dexie DB, starts the lecture
via the canvas overlay, and asserts the full chain: discussion auto-skip
appends a discussionConsumed record to maic-runtime, the device cursor lands
in KV, and both survive a fresh browsing context (empty sessionStorage).

* refactor(playback): consumed-discussion state is volatile by decision — cursor-only persistence (#959)

Product ruling on #869's fourth family: playback learner state is front-end
ephemeral UX, not learner data. A re-shown proactive card auto-skips, joined
discussions' content already lives in chat runtime records, and no replay
export / analytics consumer exists — so durable facts bought nothing over
in-memory + same-tab sessionStorage. Drop lib/playback/runtime.ts and the
RuntimeStore facts wiring; keep the device-scoped KV resume cursor (the half
with real UX value), the engine's consumption-time progress snapshot (cursor
freshness), and the legacy Dexie retirement (cursor half migrates, row
deletes, consumed ids are dropped).

* fix(review): P3 pair from cross-review — scene-id boundary + sessionTransition 400 (#966)

* fix(review): scene-id boundary for quiz context + 4xx for malformed sessionTransition (P3 pair)

Review findings on #955: didActiveSceneRemainUnchanged compared the active
scene by object identity, so a store update reallocating the scene during
the async quiz read dropped the learner's graded answers from that turn's
request — the scene id is the real boundary. The records route now
classifies a malformed sessionTransition as a validation failure instead of
letting the store's throw surface as a 500.

* fix(playback): superseded-engine cursor guard + migration write-window recheck

Second-vendor review of the #959 shrink (requested after the cross-review
noted it had single-vendor coverage) found: an engine orphaned by a scene
switch during async lecture resume could pass the idle-only recheck, be
resurrected, and publish its old scene's progress over the new scene's
debounced cursor — the resume continuation now requires identity with the
installed engine, and onProgress drops snapshots from superseded engines.
The legacy cursor migration also rechecks KV immediately before its write
so a concurrent tab's newer cursor cannot be overwritten and orphaned by
the legacy-row delete.

Co-authored-by: Codex <codex@openai.com>

---------

Co-authored-by: Codex <codex@openai.com>

* fix(review): approval follow-up P3 nits (#967)

* release: v0.3.1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): approval P3 nits — ISO gate on sessionTransition, dead mocks, corrupt-timestamp guard

- The records route's sessionTransition guard now requires an ISO
  updatedAt (isIsoTimestamp), matching the sibling PATCH /status route
- Dead vi.mock factories for the deleted playback-storage module dropped
- A corrupt legacy playback timestamp falls back to 'now' instead of
  wedging migration into a permanent re-throw that disabled resume

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 15:28:04 +08:00
wyuc
34448beb6c
feat(storage): server-backed runtime — pluggable backend seam, HTTP contract, Postgres backend, reference server (#939) (#946)
* ci: run PR checks against the runtime-server-backend integration branch (#939)

* feat(storage): RuntimeStore HTTP contract + HttpRuntimeStore client (#939 Part B) (#940)

* feat(storage): RuntimeStore HTTP contract + HttpRuntimeStore client (#939 Part B)

- documented JSON HTTP contract for every RuntimeStore operation
  (server-assigned seq, learnerKey derived from auth — never trusted
  from the request, machine-readable error codes, idempotency notes)
- HttpRuntimeStore client with injected fetch + auth headers hook;
  session reads (including createSession responses) migrate forward
  on the runtime line so an older server cannot leak stale envelopes
- conformance test server bridging the contract onto the browser
  backend; full runRuntimeStoreContract green over HTTP plus error
  mapping cases

* fix(storage): harden HTTP runtime backend after cross-review (#939 Part B)

Cross-review fixes:
- appendRecord gates payloads through assertJsonValue: fail loud on
  values JSON cannot represent (Map, Date, NaN, nested undefined, NUL
  strings) instead of silently mangling them in transit; payload
  domain documented in the contract
- appendRecord/listRecords responses validate via validateRuntimeRecord
  (server-assigned seq is no longer trusted verbatim) and listRecords
  sorts by seq like listSessions already sorted defensively
- request headers no longer require an ambient Headers constructor
  when fetch is injected
- conformance server classifies errors structurally (existence checks,
  DSL validators, version stamps) instead of regex-sniffing messages
  that interpolate caller-controlled ids
- router preserves empty path segments so empty-key calls keep browser
  no-op semantics instead of shifting onto other routes
- stray runtimeDslVersion/seq in request bodies are ignored (store-
  assigned wins), matching the browser reference; docs updated
- loopback integration test exercises the real listening server
- docs state the conformance server is test-only; auth-derived
  learnerKey is Part D's reference server

* fix(storage): round-2 cross-review hardening for the HTTP backend (#939 Part B)

- json-value guard rewritten: additionally rejects -0, symbol-keyed and
  non-enumerable own properties, and non-index own properties on arrays;
  U+2028/U+2029 are accepted again (they round-trip through JSON per
  RFC 8259 — rejecting them lost legitimate pasted text)
- conformance server classifies racing duplicate creates as 409 via a
  structural post-check, gates payloads and merge learner keys as 400
- segment() rejects '.'/'..' ids the URL layer would fold away
- list responses must be arrays and mergeLearner's moved must be a
  finite number, else a typed MALFORMED_RESPONSE error

* fix(storage): round-3 cross-review hardening for the HTTP backend (#939 Part B)

- json-value guard: NUL is rejected in object keys too, and unpaired
  UTF-16 surrogates are rejected in string values and keys (jsonb
  refuses both; other JSON stacks corrupt lone surrogates to U+FFFD)
- typed storage errors survive non-object 200 responses instead of
  degrading into TypeErrors on .id access
- mergeLearner's moved must be a non-negative integer
- conformance server classifies missing/non-object request bodies as
  400 VALIDATION_FAILED
- createSession/appendRecord gate the full init envelope, not only the
  payload (stray Date/undefined properties, NUL in ids)

* fix(storage): round-4 convergence fixes for the HTTP backend (#939 Part B)

- envelope JSON gate tolerates explicitly-undefined optional anchors
  (sceneId: undefined behaves like omission, matching the browser)
- body-carried identifiers reject '.'/'..' so nothing persists that
  the URL layer cannot later address; conformance server mirrors it
- mergeLearner keys pass the JSON-domain gate on client and server
- json-value guard v4: rejects enumerable accessors (validation/serialize
  TOCTOU), Array subclasses and null-proto arrays, prototype-supplied
  array indices; sparse-array test now constructs a genuine hole

* fix(storage): round-5 convergence fixes for the HTTP backend (#939 Part B)

- undefined-stripping is limited to the DSL-declared optional anchors
  (sceneId/actionIndex/subAnchor); any other undefined member fails the
  JSON gate loud instead of being dropped silently
- json-value guard: array and object prototype checks are realm-agnostic
  (chain-shape instead of identity), so ordinary values from another
  realm are accepted while subclasses stay rejected; shared
  isLosslessJsonString predicate exported for SQL key guards
- conformance server merge route validates target-key addressability

* fix(storage): close the toJSON prototype channel in the JSON guard (#939)

toJSON is the single channel through which a prototype can alter JSON
output — prototype properties never serialize and own accessors are
already rejected — so refusing any value with a callable toJSON closes
prototype influence on serialization entirely, including prototypes
crafted to pass the realm-agnostic chain-shape check.

* fix(storage): probe toJSON via descriptor walk, not property read (#939)

Reading value.toJSON would execute an inherited accessor, letting a
stateful getter hide from validation and reappear at stringify time.
The probe now walks own descriptors up the prototype chain without
invoking any user code. Post-validation mutation of the caller's object
graph is documented as out of scope — it is equally unpreventable for
every other validated property.

* fix(storage): toJSON probe mirrors JSON.stringify semantics exactly (#939)

The own-most descriptor decides: absent is safe, a non-callable data
value is an ordinary shadowing member and is safe, a callable data
value is rejected, and an accessor is rejected because it cannot be
inspected without invoking it. No daylight remains between the guard
and the serializer on this property.

* feat(storage): PgRuntimeStore — Postgres runtime backend (#939 Part C) (#941)

* feat(storage): PgRuntimeStore — Postgres runtime backend over an injected queryable (#939 Part C)

- PgRuntimeStore implements RuntimeStore over a minimal injected
  Queryable (node-postgres and PGlite both satisfy it) — the package
  keeps zero runtime dependencies beyond @openmaic/dsl
- runtime_sessions / runtime_records schema exported as
  RUNTIME_PG_SCHEMA with idempotent ensureSchema()
- appends serialize per session via a session-row lock; MAX(seq)+1
  and the insert share one transaction, UNIQUE(session_id, seq) plus
  bounded retry backstop non-cooperating writers
- envelope semantics mirror the browser backend: version stamping,
  validation gates, migrate-on-read, fail-loud on future-stamped rows
- full runRuntimeStoreContract green on PGlite plus PG-specific cases
  (concurrent-append seq atomicity, ensureSchema and mergeLearner
  idempotence)

* fix(storage): require pinned transactions + JSON payload gate in PG backend (#939 Part C)

Cross-review fixes:
- withTransaction is now required; the BEGIN/COMMIT fallback is removed
  (on a pg Pool it spread BEGIN/body/COMMIT across different connections
  — no real transaction, leaked idle-in-transaction clients, broken
  mergeLearner atomicity; on a shared pinned client concurrent calls
  interleaved transactions)
- single-statement deletes no longer wrap in a transaction hook call
- appendRecord gates payloads through assertJsonValue: fail loud on
  values JSON cannot represent (Map, Date, NaN, nested undefined, NUL
  strings) instead of silently persisting something different
- append retry also covers 40001/40P01; READ COMMITTED assumption
  documented at the retry loop
- ensureSchema documented as create-only; redundant stage index dropped
- deterministic interleaving test proves the 23505 retry path; real
  PostgreSQL contract lane added (postgres:16 service workflow, pg
  driver suite skipped locally without PG_CONTRACT_URL)

* fix(storage): round-2 cross-review hardening for the PG backend (#939 Part C)

- json-value guard rewritten (shared with the HTTP backend): additionally
  rejects -0, symbol-keyed and non-enumerable own properties, and
  non-index own properties on arrays; accepts U+2028/U+2029
- storage-pg-contract workflow now triggers for the runtime-server-backend
  integration branch and fails loud (STORAGE_PG_CONTRACT_REQUIRED=1)
  when PG_CONTRACT_URL is missing instead of silently skipping
- loadSession distinguishes corrupt non-object rows from absent rows so
  getSession fails loud instead of reporting the session missing

* fix(storage): round-3 cross-review hardening for the PG backend (#939 Part C)

- json-value guard (shared): NUL rejected in object keys, unpaired
  UTF-16 surrogates rejected in values and keys
- createSession/appendRecord gate the full persisted envelope through
  assertJsonValue, not only the payload — stray Date/undefined
  properties and NUL in ids fail loud instead of silently diverging
  from the returned record or leaking raw 22P05
- real-PG lane covers a genuine 23505 conflict from a second
  connection and recovery after an aborted transaction
- retry-set asymmetry and mergeLearner's unbounded lock set documented

* fix(storage): round-4 convergence fixes for the PG backend (#939 Part C)

- NUL/lone-surrogate lookup and delete keys resolve to absent/no-op
  instead of leaking 22021 driver errors; setSessionStatus reports the
  session missing; mergeLearner from-key moves 0
- mergeLearner destination key passes the JSON-domain gate fail-loud
- envelope JSON gate tolerates explicitly-undefined optional anchors,
  matching the browser backend
- json-value guard v4 (shared) + tests for accessors, Array subclasses,
  prototype-supplied indices

* fix(storage): round-5 convergence fixes for the PG backend (#939 Part C)

- appendRecord pre-checks the session key like every other lookup path,
  so a NUL/lone-surrogate sessionId reports 'no session' instead of
  leaking a 22021 driver error
- the queryable-key predicate now structurally reuses the shared
  isLosslessJsonString export instead of restating the string rule
- undefined-stripping limited to the DSL-declared optional anchors
- json-value guard: realm-agnostic prototype checks (shared)

* fix(storage): close the toJSON prototype channel in the JSON guard (#939)

Shared guard change with the HTTP backend branch; adds the crafted
null-proto-prototype regression test. Both final-gate reviewers
independently converged on this same channel.

* fix(storage): probe toJSON via descriptor walk, not property read (#939)

Reading value.toJSON would execute an inherited accessor, letting a
stateful getter hide from validation and reappear at stringify time.
The probe now walks own descriptors up the prototype chain without
invoking any user code. Post-validation mutation of the caller's object
graph is documented as out of scope — it is equally unpreventable for
every other validated property.

* fix(storage): toJSON probe mirrors JSON.stringify semantics exactly (#939)

The own-most descriptor decides: absent is safe, a non-callable data
value is an ordinary shadowing member and is safe, a callable data
value is rejected, and an accessor is rejected because it cannot be
inspected without invoking it. No daylight remains between the guard
and the serializer on this property.

* fix(storage): conform HTTP/PG backends to the post-#926 RuntimeStore interface (#943)

The chat cutover (#926) added deleteAllRuntime() to the RuntimeStore
contract while the HTTP and Postgres backends were developed in
parallel against the pre-#926 interface. Implements the method on both
backends (single-statement wipe on PG via the FK cascade; DELETE
/runtime on the HTTP contract, documented as an operator-gated
administrative endpoint), restoring a green typecheck and contract
suite on the integration branch.

* feat(runtime): injectable RuntimeStore backend + learner-key provider (#939 Part A) (#944)

* wip(runtime): backend injection seam — pending gate verification

* fix(runtime): cross-review hardening for the storage injection seam

- stage-deletion IndexedDB probe applies only to the default browser
  backend; an injected store always receives deleteStageRuntime
- explicit kv argument takes precedence over the configured learner-key
  provider, matching the store seam's explicit-beats-global rule
- configured learner-key resolution is latched with in-flight dedup,
  mirroring the store singleton; identity changes require app-level
  handling
- factory retry semantics documented; seal errors explain the
  module-level bootstrap requirement; isRuntimeStorageConfigured()
  probe and a test-only reset added
- client-bootstrap-only contract documented (SSR/HMR caveats)

* fix(runtime): snapshot configuration and make the test reset complete

- configureRuntimeStorage copies the option fields so mutating the
  caller's object after configuring cannot swap the sealed backend or
  identity provider
- resetRuntimeStorageForTests now clears every latched consumer cache
  (store singleton, learner-key in-flight promise) via a reset-hook
  registry, so a reset-then-reconfigure test actually gets the new
  backend

* fix(runtime): reset also clears the default learner-key caches

resetRuntimeStorageForTests left defaultInFlight/defaultKv latched, so
a default-path test could leak its anonymous key or KV store into the
next test despite the documented full-reset promise.

* feat(storage): runtime reference server — auth-derived learnerKey (#939 Part D) (#945)

* wip(storage): reference server — pending deleteAllRuntime route + gates

* feat(storage): runtime reference server — auth-derived learnerKey enforcement (#939 Part D)

- createRuntimeHttpHandler(store, options) wires the documented HTTP
  contract onto any injected RuntimeStore over node http
- authenticate is required; every learner-scoped operation verifies the
  path/body learnerKey against the authenticated principal (403
  FORBIDDEN_LEARNER) — the client-supplied value is never trusted
- mergeLearner requires an explicit authorizeMerge grant and admin
  surfaces (stage cascade, DELETE /runtime) require authorizeAdmin;
  both default-deny
- runnable reference entry demonstrates a pg Pool withTransaction and
  bearer-token authentication, marked as demo-only
- contract suite green through HttpRuntimeStore -> listening reference
  handler -> PgRuntimeStore(pglite); security matrix tested (401/403
  paths, admin default-deny); threat model documented

* fix(storage): cross-review hardening for the reference server (#939 Part D)

- principal learnerKey is optional: admin/merge-only credentials no
  longer fabricate learner identity; learner-scoped routes 403 without
  ownership
- full session/record envelopes pass the JSON-domain gate at the
  handler, so NUL/lone-surrogate identifiers map to 400 instead of 500
- payload validation follows the injected store's validator map
  (options.payloadValidators) instead of imposing DSL defaults
- reference factory accepts authenticate/authorizeMerge/authorizeAdmin/
  payloadValidators overrides; docs no longer claim the factory binds
  to localhost; demo-impersonation warning hardened
- 500 responses carry a generic message; details go to the server log
- ownership checks precede version checks and unowned sessions read as
  404, closing existence/version oracles; cross-learner denial matrix
  tested per route
- merge/delete concurrency documented as linearizable-equivalent with
  the ownership re-check narrowed to the delete call

* fix(storage): align reference-server semantics with the store contract

- future-stamped sessions read and delete through unchanged; 409
  FUTURE_VERSION applies only to guarded writes (status, append, merge)
- check-then-write races reclassify structurally via a post-failure
  re-fetch: missing session 404, non-active session 400, never a
  message-sniffed or generic 500

* fix(storage): close the reference CLI's pg pool on startup failure

ensureSchema opens connections inside createReferenceRuntimeServer, so
a failed schema init or occupied port left the pool holding database
resources until its idle timeout.

* fix(storage): close the records-route existence oracle

cosarah's review point on #946: the records list answered 200 [] for an
absent session but 404 for another learner's, so the 404 leaked that an
id exists. Absent and foreign sessions now answer identically (404
SESSION_NOT_FOUND) and the HTTP client maps that code back to an empty
list, preserving the store contract's absent-lists-as-empty semantics.
Contract doc records the server MAY/SHOULD and the client MUST.
2026-07-17 16:15:57 +08:00