# Annotation operation latency ## Changes - Cache validated hardware timelines per organization and source identity. For BOS-mounted workspaces use authoritative object HEAD ETag/version/size; do not use the FUSE mount's synthetic, unstable mtime. Read the five object versions concurrently. Native local files use size/mtime. Check identity on every request, including before/after cold validation. Coalesce concurrent requests; retain at most 32 entries / 32 MiB of serialized grids for one hour. No IMU samples or drafts in this cache. Errors are not cached. - Workspace object mapping uses `ANNOTATION_WORKSPACE_CONNECTION_ID` / `ANNOTATION_WORKSPACE_BUCKET`, falling back to the existing `PROJECT_DELETION_WORKSPACE_CONNECTION_ID` / `PROJECT_DELETION_WORKSPACE_BUCKET` mapping when unset. These reads are scoped to the task organization. Fanyu already has this workspace mapping; no new manual toggle is required. - Stream IMU validation with cancellation and constant-space accel/gyro pair tracking. Timeline requests retain only coverage endpoints. Actual slicing still loads samples once; slice lookup uses binary search instead of rescanning the entire sample list for each slice. - `GET /api/plugins/annotation-qa/tasks/:id?view=status` returns only scoped progress/status, not histories, media or the annotation workspace. It uses the same access checks as task detail and does not mutate/recover state. An interrupted job requests one full-detail refresh for existing recovery behavior. - Poll status every 2–3 seconds; load full detail on a transition. Do not repeatedly abort/rebuild the timeline while a background operation is running. A 202 response means processing, not quality handoff; keep the editor locked and preserve data until the authoritative stage transition. - Read the linked human item using a scoped legacy-task filter with a nonunique compound index; no role or supplier changes. - Run physical materialization on a temporary local directory with one concurrent materialization per API process and two encoder threads per camera. Stage each source video once, then process/verify/hash locally and copy completed outputs into the existing run directory. Temporary files are removed on success/failure. Local scratch capacity is still needed for source and output videos. - Reuse previously verified slice-video bytes when source identity, encoder contract and both cameras' exact frame bounds match. Names, descriptions and disposition are not media identity. Verify manifest and video SHA-256 before reuse; corruption falls back to encoding. Regenerate annotation/PTS/IMU/review manifests in the new generation. Copies are independent: no hard links or symlinks into generations removed by retention. - Show actual queued/validation/slicing/publication stages and completed-slice counts, not invented percentages or ETAs. Log queue and total processing time plus actual reuse count. QA/acceptance no longer keep the successful action spinner waiting for a full pool/statistics refresh. ## October 15: navigation, pool loading and failed-slice recovery Owned annotation tasks load independently of claim-count. Failed counts remain unknown instead of blocking or emptying the owned pool; account/project/load changes fence late responses. Blind annotator visibility is pushed into storage before pagination, preserving author history and exceptional legacy stage roles. Final server permission checks remain authoritative. Claim-count groups compact candidate counters in the database and applies current batch rules through the existing projected access reader, preserving supplier and membership predicates. Review opinion time navigation selects the owning slice and seeks together. Rapid navigation back to the current render's slice also replaces an earlier pending target. Ordinary timeline selection and B playback retain their behavior. “保留已保存切片重试” uses the existing slice-confirmations endpoint with `retryFailed: true` and the current version. It validates the server workspace, including the legacy draft fallback, records the preceding failure as `retryOf`, and queues a new fenced durable job. Retained semantic work is frozen in the private outbox; client-supplied snapshots are ignored. Acceptance keeps the task occupied and does not report completion. Missing/malformed snapshots or invalid source/frame/dictionary data fail without clearing content. Draft writes during slicing are rejected. Explicit first-pass reset/cancellation remains separate; cancelled or stale workers cannot publish over a newer request. Local restart/reclaim, stale-owner, validation, duplicate retry and late legacy draft tests cover this path. An isolated 12k-item/12k-batch Mongo fixture examined 12k documents with the original list query, 100 for a 100-row page with sorted indexes, and zero documents/keys for the blind actor query. Full scoped listing including stage classification took 38ms; scalar count took 885ms. These are local observations, not production or concurrent P95 results. The three prepared indexes were created in Fanyu on October 15 under the user's separately approved index plan; existing indexes were retained. Further index changes require another approved plan. Retry-capable API/frontend require the matching processor: update and verify the processor first, then the API/frontend. Older processors do not understand retained-workspace retry data. For rollback, disable the new API entry first and finish or reconcile outstanding retry jobs before replacing the processor with an older version. This is release compatibility guidance; it does not authorize further releases or customer retries. ### Production count regression and follow-up The first d8a8530c rollout passed infrastructure and three supplied-role login checks, but actual claim-count HTTP timed out at 45 seconds versus a prior 4.179-second observation. Owned tasks remained usable at 261ms; available-list read took 7.954 seconds. The first release is not business-accepted. Its local 885ms aggregate result above did not predict production behavior. The follow-up removes correlated per-batch counting and full-history metadata classification. MongoDB groups current candidates by batch/stage with separate legacy and supplier-pool membership counts. The store reuses ListHumanBatchAccessRecords for only those groups and the authoritative Go stage and workflow classifiers to return one count. Blind lists also restrict access metadata to candidate batch IDs before pagination; source batches outside the item's project remain supported. No permission cache, new index, task migration or business write is added. Separate request-scoped reads are not a transactional snapshot; claim mutation still checks current authority and candidate state. A sparse 1,200-batch fixture reproduced the historical metadata scan and now budgets candidate and metadata reads independently. Dense 12k and legacy/pool/ revoked-drain/role/duplicate-stage scenarios run against isolated local Mongo7/8. Actual hotfix HTTP latency must be measured after its separately approved rollout. Direct method probes through the diagnostic kubectl TCP tunnel are not equivalent: the old projected-read control also timed out during bulk getMore over that path. ## Unchanged contracts Frame boundaries, original-frame stepping, coverage and cross-camera checks remain. Semantic blanks, explicit invalid annotations, discard evidence, all-discard delivery filtering, dictionary-only slice names and all-role release prohibition are unchanged. Package schemas and delivery directories are unchanged. Existing artifacts without reusable-source identity take the normal validated generation path. ## Evidence and limits - Same real 842.966667-second Capture (task `annotation-task-7d738b9fc2bd9e9e778308029a7d6a50`): baseline timeline reconstruction 26.034 / 16.989 seconds; optimized cold / warm helper 27.163 seconds / 41.55 ms. This read-only probe used deployed source files and the new code; it is not a measurement of the whole save request. Cold BOS reads are still slow. - Synthetic real stereo video checks cover keep, mixed discard, all discard, blank and partial annotation revisions; verify 80 frames per camera per slice, unchanged video hashes, corrupt-cache fallback, changed boundary invalidation and old-generation removal. - Status endpoint tests check response size, authorization and no mutation. Cache tests cover coalescing, caller-owned arrays, expiry, changed identity and failure/cancellation. Run race detection as well as package compatibility tests. - Existing full-suite baseline failures must be reported separately from the focused optimization checks. - One process-local cache is not a persistent precomputed index; restart/eviction requires a cold read. This change does not move jobs into an external durable queue or change legacy restart recovery. Original-video prefix decoding can still cost CPU for many slices; no approximate seek or stream-copy shortcut was introduced. ## Fanyu acceptance, 2026-09-17 - Final API and Worker implementation release: `6deb36c2`. API image digest `sha256:ecb0dbc3a0f2a0fd716500138ba76f6f5a3340394f3882ade31ce91d8106a4e2`; Worker `sha256:628130e26164cedb7ab3bd4e70137457889816a8ff733388aca6d7045e3e4253`. Algorithm/package images were not changed. - During the first rollout the new mtime check detected unstable BOS-FUSE timestamps. The API was rolled back, object-version validation was added and tested against the same files, then the corrected build was deployed. Genuine object ETag changes still invalidate the cache, including same-size overwrites. - Final read-only probe of the same 842.966667-second production input: cold timeline 23.653 seconds, cached timeline 43.954 ms. The cold read remains storage-bound. These are helper timings, not whole HTTP save latencies. - Isolated project `project-discard-verify-20260917latency3`: first 60-second three-slice materialization 5.500 seconds; next unchanged-video revision reused all three slices (5.261 seconds including file publication). Small fixtures remain dominated by storage round trips, not encoding. - Actual Chrome submission showed processing immediately after 202 and returned to the pool only on real quality handoff. Its request trace contained one timeline request and two lightweight status polls during the measured interaction. - Mixed/invalid-only/rework/discard path: child `run-human-release-ca420a50e0f0c6948be176a0d7ee2d1c`, 2 delivered slices / 40 seconds / 23 verified files / 320 stereo frames. The discarded middle slice stayed visible in QA/acceptance but was absent from final delivery. - Entirely blank semantic path: child `run-human-release-4378ecffb7c6b9ff6a4126c01122641c`, 3 delivered slices / 60 seconds / 30 verified files / 480 stereo frames / zero invented annotations. Both parent/child pairs completed; both Captures reached business stage `delivered`. All object sizes/SHA-256 and per-video decoded frame counts matched the manifests. - All four synthetic accounts per attempted fixture were disabled after testing. Test projects/evidence remain for audit; original customer input was not edited. - Focused Go tests with race detection and 14 package/delivery compatibility tests passed. Full API comparison retains exactly the same 60 baseline failures; no new API failures.