droboticflow-new/docs/capture-delivery-directory.md

6.2 KiB

Configurable final delivery directory

The continuous-processing settings now include a required delivery root in the same connection/bucket as intake. The UI previews:

<delivery root>/<project name>__<project ID>/<capture name>/<part name>/

clips/, calibration/ and package/evidence manifests are direct children of each part directory. Every part creates an independent Run using the same workflow instance, with its own annotation, quality review and acceptance sampling. A single part also retains its directory. Flat legacy sources omit the part layer. Run IDs are not added to the delivery path.

Part-index delivery mode

The package node selects the delivery shape with the PART_INDEX_DELIVERY hyperparameter (default false).

  • false — the directory form above: sliced media per kept slice, calibration copies, package/evidence manifests.
  • true — one JSON document per part. The part directory holds part.json plus the shared completion manifest.json. No sliced media, PTS, IMU, pose or calibration bytes are delivered; downstream packaging reads the index and resolves media from the pinned materialization revision.

The index carries both split levels in one document: the task-level split (slices[].taskSplit) and the action-level split (slices[].segments). Every reviewed interval is classified exactly once — kept intervals in slices[], discarded intervals in discardedSlices[] with their discardEvidence. Discarded intervals are now visible to the customer by design; earlier directory deliveries omitted discard reasons entirely.

clips/ is absent in this mode, so consumers must branch on the completion manifest kind (part-index versus directory) instead of assuming a directory layout. The index carries the per-slice poseDelivery include/exclude decision, but never pose frame data itself — frames stay in the pinned materialization revision for downstream resolution.

Migrating an already delivered part

An already published directory delivery keeps its clips/, calibration and evidence files. scripts/part_index_migration.py produces the candidate part.json for such a part from a local copy of the delivered directory plus the accepted materialization revision. It reuses the package operator's own builder, so a migrated index is byte-identical to what an online part-index package would have produced for the same evidence — apart from a lineage.migratedFrom provenance marker.

The tool never writes object storage, and the worker publication path cannot publish that candidate into a destination whose completion manifest already exists: rewriting a customer's completion marker in place is refused (a same-owner, same-bytes retry of an existing part-index manifest is the one accepted idempotent case). A part that was already delivered in the directory form therefore gains the index through an explicit re-delivery of that part, never through an in-place patch of the delivered directory. If the part already has a part.json with different bytes, the candidate is refused as well.

Safety and persistence

  • API validates disjoint input/output prefixes (including other workflow listeners), rejects reserved system prefixes and probes write/delete permission before enabling intake. The project directory name is server-owned and reused across subsequent configuration updates.
  • New admissions freeze connection, bucket and complete target prefix on the Capture record. Existing snapshots are not changed by settings edits or project renames.
  • Older tasks with a frozen no-part target are not silently moved. Publication rejects ancestor/descendant overlap with another task's frozen delivery location; resolve the old binding explicitly before publishing into an overlapping location.
  • An old task without a delivery snapshot fails publication explicitly. It is not silently migrated and cannot fall back to the Run directory. No existing output is moved automatically.
  • The package operator still prepares and validates files in its internal workspace. The worker publishes the delivery.directory.index artifact, or the part.json artifact in part-index mode, into the frozen destination. Other intermediate artifacts keep their existing storage policy.
  • .delivery-owner.json reserves the destination using conditional creation. Every member also uses conditional creation and metadata SHA-256/size verification. Retrying identical publication is idempotent; changed content or a different owner is rejected, not overwritten. Both delivery modes share one owner marker per destination, so a part cannot be delivered twice by different attempts.
  • A part-index publication refuses to adopt a destination that already holds files without an owner marker. Additive migration into an existing legacy directory therefore requires that directory to carry its owner marker; otherwise the destination is rejected rather than claimed.
  • Files can be visible during publication, but manifest.json is written last and is the authoritative completion marker. The business task cannot complete until publication succeeds. This is not a claim of atomic BOS directory rename.
  • packages-manifest.json is published after data members. manifest.json indexes every published member with size and SHA-256. The directory form additionally publishes package and evidence manifests; the part-index form publishes the index alone. Neither output is a ZIP.
  • Temporary upload objects stay under system/runs/<run>/delivery-staging/ and successful uploads remove them. Interrupted staging is covered by normal Run cleanup.
  • Project and workflow deletion protect configured/frozen final delivery destinations. Deleting source business records is not permission to delete delivered data.
  • Capture records show the frozen destination and an “open delivery directory” link after delivery.

Release boundary

API and worker must be deployed together. The earlier part-layout change additionally needs both updated directory/normalizer operator images. Do not deploy only the frontend or start a new scanner with old operators that ignore its explicit file selection. Part-index mode additionally requires the worker that understands the part.json routing block: an older worker would upload the index as an ordinary run artifact instead of honoring the frozen destination.