forked from longtao.wu/droboticflow
32 lines
5.4 KiB
Markdown
32 lines
5.4 KiB
Markdown
# Durable annotation processing
|
||
|
||
## Compatibility boundary
|
||
|
||
No change to dictionary selection, role/supplier ownership, semantic blanks, explicit invalid intervals, discard evidence, acceptance rules, exact frame boundaries or delivery directory layout. Materialization manifests remain v2/v3. Existing generation files remain readable. New media objects are independently owned by each generation; no symlinks or references into a generation that retention can delete.
|
||
|
||
## Performance paths
|
||
|
||
1. New full-video annotation tasks enqueue index preparation. The independent processor builds a source-version-keyed timeline and stores a compressed, checksummed document in `annotationTimelineIndexes`. API restarts and memory-cache eviction no longer require rebuilding an existing index. Source ETag/version changes invalidate it. Legacy/open tasks can be backfilled without creating Runs or changing assignments.
|
||
2. When only names/descriptions/disposition change and all frame windows still match, the metadata-only path validates the source version and prior manifest, copies verified media/PTS/IMU objects server-side in BOS, and rewrites only small annotation/synchronization manifests locally. It does not reread original IMU/PTS or decode video. A native-filesystem installation retains checked file copies as a compatibility fallback. Old BOS generations without pinned object ETags receive normal verification once.
|
||
3. New/changed slices share input decoding in groups of up to eight outputs. Every output still uses half-open original-frame trims and decoded frame-count checks. Tests compare decoded frame hashes against the single-output implementation.
|
||
4. Generation publication uses at most four concurrent BOS transfers and publishes its manifest last. Final delivery also uses bounded BOS concurrency, retaining conditional ownership checks, digest verification, retry semantics and manifest-last ordering.
|
||
5. The API validates a submission once and atomically stores the durable outbox in the HumanItem CAS. It returns processing, not success. The processor rechecks current actor/project/role/source ownership and uses an expiring, fenced claim. A stale worker cannot publish after takeover. Worker Run snapshots preserve API-published materializations.
|
||
6. Task-pool cards use an authorized task-summary view instead of loading each task's full history. Processing polls use the existing lightweight status view. Histories are loaded for editing, not every progress tick.
|
||
7. The workbench requests a lossless compact timeline. Integer deltas and IEEE-754 correction bits preserve the original arrays exactly; the full JSON response remains available for older clients. Decompression has strict size/integer bounds. Source downloads use ETag-pinned BOS streaming into local scratch, rather than repeated small reads through the mount; full frame verification uses bounded decoder threads.
|
||
|
||
## Operation
|
||
|
||
- Default `ANNOTATION_PROCESSOR_MODE=inprocess` preserves existing deployments until explicitly switched.
|
||
- The API uses `ANNOTATION_PROCESSOR_MODE=external` after the independent processor is healthy.
|
||
- The processor runs the same API image/binary with `ANNOTATION_PROCESSOR_MODE=processor`. It has no public task-mutation endpoint; `/ready` on port 3003 is for health checks only. It does not need a Kubernetes service-account token.
|
||
- `ANNOTATION_PROCESSOR_CONCURRENCY=2` is the default (allowed 1–4 per process). Multiple replicas claim independently from the database. Local `/tmp` needs bounded scratch capacity; the supplied deployment uses a 20 GiB emptyDir.
|
||
- `ANNOTATION_INDEX_BACKFILL=1` queues existing pending full-video annotation tasks at startup. Index work has a separate bounded loop. Precomputation may take time for a backlog; it is not a claim that every newly discovered file is immediately ready.
|
||
- Processing leases last two minutes and renew every ten seconds. Interrupted work is requeued on graceful shutdown, or reclaimed after lease expiry. Repeated interruptions stop after three attempts with the saved workspace intact. Legacy API-only reservations can be adopted without changing the visible business version.
|
||
- BOS workspace mapping uses `ANNOTATION_WORKSPACE_CONNECTION_ID` / `ANNOTATION_WORKSPACE_BUCKET`, with the prior project-deletion workspace mapping as a fallback. Native local files use filesystem versions.
|
||
- Manifest rendering: set `ANNOTATION_PROCESSOR_ENABLED=true` plus the workspace mapping to render `k8s/annotation-processor.template.yaml` and switch API mode. Keep the processor on the same code/image release as the API when deploying future changes. Roll out the processor before enabling external mode; inspect existing legacy in-flight submissions before changing modes.
|
||
- Project/workflow deletion removes index/cache bookkeeping with the associated scope. Media retention still removes only superseded private generation paths, never original inputs.
|
||
|
||
## Verification
|
||
|
||
Focused tests cover durable outbox BSON/JSON privacy, concurrent exclusive claims, expired-owner fencing, process replacement, source changes, persistent index recovery/corruption, proactive indexing, semantic-empty submission, metadata-only generation, decoded frame equivalence and delivery ownership/retry behavior. Live load tests must use a separate database/processor and private Run prefixes; production source objects are read-only. Report the tested concurrency and actual video duration rather than extrapolating unlimited capacity.
|