58 lines
9.3 KiB
Markdown
58 lines
9.3 KiB
Markdown
# Durable annotation processing
|
||
|
||
## Compatibility boundary
|
||
|
||
No change to dictionary selection, role/supplier ownership, semantic blanks, explicit invalid intervals, discard evidence, acceptance rules, exact frame boundaries or delivery directory layout. Materialization manifests remain v2/v3. Existing generation files remain readable. New media objects are independently owned by each generation; no symlinks or references into a generation that retention can delete.
|
||
|
||
## Performance paths
|
||
|
||
1. New full-video annotation tasks enqueue index preparation. The independent processor builds a source-version-keyed timeline and stores a compressed, checksummed document in `annotationTimelineIndexes`. API restarts and memory-cache eviction no longer require rebuilding an existing index. Source ETag/version changes invalidate it. Legacy/open tasks can be backfilled without creating Runs or changing assignments.
|
||
2. When only names/descriptions/disposition change and all frame windows still match, the metadata-only path validates the source version and prior manifest, copies verified media/PTS/IMU objects server-side in BOS, and rewrites only small annotation/synchronization manifests locally. It does not reread original IMU/PTS or decode video. A native-filesystem installation retains checked file copies as a compatibility fallback. Old BOS generations without pinned object ETags receive normal verification once.
|
||
3. New/changed slices share input decoding in groups of up to eight outputs. Every output still uses half-open original-frame trims and decoded frame-count checks. Tests compare decoded frame hashes against the single-output implementation.
|
||
4. Generation publication uses at most four concurrent BOS transfers and publishes its manifest last. Final delivery also uses bounded BOS concurrency, retaining conditional ownership checks, digest verification, retry semantics and manifest-last ordering.
|
||
5. The API validates a submission once and atomically stores the durable outbox in the HumanItem CAS. It returns processing, not success. The processor rechecks current actor/project/role/source ownership and uses an expiring, fenced claim. A stale worker cannot publish after takeover. Worker Run snapshots preserve API-published materializations.
|
||
6. Task-pool cards use an authorized task-summary view instead of loading each task's full history. Processing polls use the existing lightweight status view. Histories are loaded for editing, not every progress tick.
|
||
7. The workbench requests a lossless compact timeline. Integer deltas and IEEE-754 correction bits preserve the original arrays exactly; the full JSON response remains available for older clients. Decompression has strict size/integer bounds. Source downloads use ETag-pinned BOS streaming into local scratch, rather than repeated small reads through the mount; full frame verification uses bounded decoder threads.
|
||
|
||
## Operation
|
||
|
||
- Default `ANNOTATION_PROCESSOR_MODE=inprocess` preserves existing deployments until explicitly switched.
|
||
- The API uses `ANNOTATION_PROCESSOR_MODE=external` after the independent processor is healthy.
|
||
- The processor runs the same API image/binary with `ANNOTATION_PROCESSOR_MODE=processor`. It has no public task-mutation endpoint; `/ready` on port 3003 is for health checks only. It does not need a Kubernetes service-account token.
|
||
- `ANNOTATION_PROCESSOR_CONCURRENCY=2` is the default (allowed 1–4 per process). Multiple replicas claim independently from the database. Local `/tmp` needs bounded scratch capacity; the supplied deployment uses a 20 GiB emptyDir.
|
||
- `ANNOTATION_INDEX_BACKFILL=1` queues existing pending full-video annotation tasks at startup. Index work has a separate bounded loop. Precomputation may take time for a backlog; it is not a claim that every newly discovered file is immediately ready.
|
||
- Processing leases last two minutes and renew every ten seconds. Interrupted work is requeued on graceful shutdown, or reclaimed after lease expiry. Repeated interruptions stop after three attempts with the saved workspace intact. Legacy API-only reservations can be adopted without changing the visible business version.
|
||
- BOS workspace mapping uses `ANNOTATION_WORKSPACE_CONNECTION_ID` / `ANNOTATION_WORKSPACE_BUCKET`, with the prior project-deletion workspace mapping as a fallback. Native local files use filesystem versions.
|
||
- Manifest rendering: set `ANNOTATION_PROCESSOR_ENABLED=true` plus the workspace mapping to render `k8s/annotation-processor.template.yaml` and switch API mode. Keep the processor on the same code/image release as the API when deploying future changes. Roll out the processor before enabling external mode; inspect existing legacy in-flight submissions before changing modes.
|
||
- Project/workflow deletion removes index/cache bookkeeping with the associated scope. Media retention still removes only superseded private generation paths, never original inputs.
|
||
|
||
## Verification
|
||
|
||
Focused tests cover durable outbox BSON/JSON privacy, concurrent exclusive claims, expired-owner fencing, process replacement, source changes, persistent index recovery/corruption, proactive indexing, semantic-empty submission, metadata-only generation, decoded frame equivalence and delivery ownership/retry behavior. Live load tests must use a separate database/processor and private Run prefixes; production source objects are read-only. Report the tested concurrency and actual video duration rather than extrapolating unlimited capacity.
|
||
|
||
## Fanyu verification — 2026-09-17
|
||
|
||
Deployed implementation: `b57980fd`. API and independent annotation processor use API digest `sha256:886d7ed7d4f53180243d780bc5a95b241babfd703357e9507b0c2d4db70792dc`; Worker digest `sha256:8c3897e1a77ca173c341ff8dee838abe10925596d7b1fc6b73479e6d04f7ec8b`. API mode is `external`, processor concurrency is two. Algorithm and package operator images remain unchanged.
|
||
|
||
Load validation used five separate accounts and five independent tasks referencing the same real 842.966667-second source Capture. It used a separate database and API/processor deployments in Fanyu, not production staff tasks. This measures a five-client repeated-sample workload, not arbitrary fleet capacity.
|
||
|
||
| Operation | Median | P95 | Samples |
|
||
| --- | ---: | ---: | ---: |
|
||
| First timeline response after precomputation | 547 ms | 549 ms | 5 |
|
||
| Repeated timeline response | 747 ms | 1,492 ms | 50 |
|
||
| Confirm slice request accepted | 240 ms | 250 ms | 5 |
|
||
| Status read during processing | 72 ms | 128 ms | 50 |
|
||
| Save description | 219 ms | 252 ms | 5 |
|
||
| Submit description request accepted | 102 ms | 116 ms | 5 |
|
||
|
||
- Before the compact wire format, the same five-client test still took 10–14 seconds to transfer full timeline responses, despite a fast cached helper. This led to the lossless wire optimization; no server-only timing is reported as browser performance.
|
||
- A 27,001-point fixture encoded from 1,736,701 to 15,511 bytes. Go-to-JavaScript and actual Chrome decoding restored every integer and IEEE-754 time exactly; Chrome decode took about 13 ms. No duration/frame resampling was introduced.
|
||
- The isolated processor Pod was deleted during active long-video processing. Jobs resumed on the replacement processor, with affected attempts incrementing to two; all five slice tasks and metadata-only submissions completed. The initial comparison run also completed recovery. Sample readback verified 21 files and 50,573 decoded stereo frames against checksums/PTS after each run.
|
||
- Long-video generation logs under two active jobs changed from approximately 375–388 seconds per job to about 224 seconds in the optimized input/decoder run. Physical generation is still background work; request-acceptance timings above do not claim that videos are generated in milliseconds.
|
||
- Production functional fixture: `project-discard-verify-20260917allqa1`. Mixed/rework/invalid-only/discard path produced child `run-human-release-61c4cb89cece43f47843391dbc7cf868`, 2 delivered slices / 40 seconds / 23 verified files / 320 stereo frames. All-blank path produced `run-human-release-2b386cb57fb91d8dc83e83226b67efbf`, 3 slices / 60 seconds / 30 files / 480 frames / zero invented annotations. Both reached `delivered`; discarded evidence remained available in QA/acceptance but not in delivery.
|
||
- Actual production browser submission used summary reads, one compact timeline request, and one lightweight status poll during the measured interaction. It displayed processing after 202 and returned to the pool only on authoritative quality handoff. Role-release prohibitions and dictionary constraints were retested.
|
||
- Existing full-suite failures remain unchanged: 60 API tests and 9 frontend tests. The expanded frontend suite has 545 tests, 536 passing. Targeted concurrency/race, persistence, recovery, decoded-frame equality and browser checks passed.
|
||
- Performance deployments were removed after the runs; synthetic accounts were disabled. Test data and evidence were retained, but copied sandbox storage credentials were cleared. Original production files were read-only.
|
||
|
||
Remaining limits: an input whose index is still being prepared can still wait; queues still wait for finite CPU/I/O capacity. Native local filesystem media-copy fallback remains for non-BOS deployments. The measured concurrency was five accounts; higher concurrency needs its own sizing/validation. Chrome was exercised directly; the codec uses standard APIs available in modern Chromium-based Edge, but a separate Windows Edge session was not available for this run.
|