droboticflow/docs/annotation-processing.md

5.0 KiB
Raw Blame History

Durable annotation processing

Compatibility boundary

No change to dictionary selection, role/supplier ownership, semantic blanks, explicit invalid intervals, discard evidence, acceptance rules, exact frame boundaries or delivery directory layout. Materialization manifests remain v2/v3. Existing generation files remain readable. New media objects are independently owned by each generation; no symlinks or references into a generation that retention can delete.

Performance paths

  1. New full-video annotation tasks enqueue index preparation. The independent processor builds a source-version-keyed timeline and stores a compressed, checksummed document in annotationTimelineIndexes. API restarts and memory-cache eviction no longer require rebuilding an existing index. Source ETag/version changes invalidate it. Legacy/open tasks can be backfilled without creating Runs or changing assignments.
  2. When only names/descriptions/disposition change and all frame windows still match, the metadata-only path validates the source version and prior manifest, copies verified media/PTS/IMU objects server-side in BOS, and rewrites only small annotation/synchronization manifests locally. It does not reread original IMU/PTS or decode video. A native-filesystem installation retains checked file copies as a compatibility fallback. Old BOS generations without pinned object ETags receive normal verification once.
  3. New/changed slices share input decoding in groups of up to eight outputs. Every output still uses half-open original-frame trims and decoded frame-count checks. Tests compare decoded frame hashes against the single-output implementation.
  4. Generation publication uses at most four concurrent BOS transfers and publishes its manifest last. Final delivery also uses bounded BOS concurrency, retaining conditional ownership checks, digest verification, retry semantics and manifest-last ordering.
  5. The API validates a submission once and atomically stores the durable outbox in the HumanItem CAS. It returns processing, not success. The processor rechecks current actor/project/role/source ownership and uses an expiring, fenced claim. A stale worker cannot publish after takeover. Worker Run snapshots preserve API-published materializations.
  6. Task-pool cards use an authorized task-summary view instead of loading each task's full history. Processing polls use the existing lightweight status view. Histories are loaded for editing, not every progress tick.

Operation

  • Default ANNOTATION_PROCESSOR_MODE=inprocess preserves existing deployments until explicitly switched.
  • The API uses ANNOTATION_PROCESSOR_MODE=external after the independent processor is healthy.
  • The processor runs the same API image/binary with ANNOTATION_PROCESSOR_MODE=processor. It has no public task-mutation endpoint; /ready on port 3003 is for health checks only. It does not need a Kubernetes service-account token.
  • ANNOTATION_PROCESSOR_CONCURRENCY=2 is the default (allowed 1–4 per process). Multiple replicas claim independently from the database. Local /tmp needs bounded scratch capacity; the supplied deployment uses a 20 GiB emptyDir.
  • ANNOTATION_INDEX_BACKFILL=1 queues existing pending full-video annotation tasks at startup. Index work has a separate bounded loop. Precomputation may take time for a backlog; it is not a claim that every newly discovered file is immediately ready.
  • Processing leases last two minutes and renew every ten seconds. Interrupted work is requeued on graceful shutdown, or reclaimed after lease expiry. Repeated interruptions stop after three attempts with the saved workspace intact. Legacy API-only reservations can be adopted without changing the visible business version.
  • BOS workspace mapping uses ANNOTATION_WORKSPACE_CONNECTION_ID / ANNOTATION_WORKSPACE_BUCKET, with the prior project-deletion workspace mapping as a fallback. Native local files use filesystem versions.
  • Manifest rendering: set ANNOTATION_PROCESSOR_ENABLED=true plus the workspace mapping to render k8s/annotation-processor.template.yaml and switch API mode. Keep the processor on the same code/image release as the API when deploying future changes. Roll out the processor before enabling external mode; inspect existing legacy in-flight submissions before changing modes.
  • Project/workflow deletion removes index/cache bookkeeping with the associated scope. Media retention still removes only superseded private generation paths, never original inputs.

Verification

Focused tests cover durable outbox BSON/JSON privacy, concurrent exclusive claims, expired-owner fencing, process replacement, source changes, persistent index recovery/corruption, proactive indexing, semantic-empty submission, metadata-only generation, decoded frame equivalence and delivery ownership/retry behavior. Live load tests must use a separate database/processor and private Run prefixes; production source objects are read-only. Report the tested concurrency and actual video duration rather than extrapolating unlimited capacity.