✅ Enforce persistent POC operator evidence

This commit is contained in:
eust-w 2026-06-22 19:38:44 +08:00
parent 562e1f1ea4
commit 08fe07bcac
6 changed files with 44 additions and 2 deletions

View File

@ -119,7 +119,7 @@ For first-time POC cluster setup, configure the shared workspace before deployin
DEPLOY_ENV=poc ./scripts/configure-poc-environment.sh
```
This installs the BOS CSI node plugin and creates the `droboticflow-workspace` PV/PVC. `WORKSPACE_BOS_BUCKET` has no repository default; it must come from the local `poc` secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
This installs the BOS CSI node plugin and creates the `droboticflow-workspace` PV/PVC. The BOS PV mount options include `meta_expires=0`, `allow_other`, and `mount_umask=000` so the non-root API, worker, and operator-runner containers can traverse and write the shared workspace. `WORKSPACE_BOS_BUCKET` has no repository default; it must come from the local `poc` secrets file or the current shell. Do not commit BOS keys, bucket names used for private data, or Mongo credentials.
The POC environment also creates a public `droboticflow-api-public` LoadBalancer service. The standard `droboticflow-api` service defaults to NodePort for compatibility, but can be rendered as ClusterIP with `API_SERVICE_TYPE=ClusterIP` when exposure is handled by the LoadBalancer or an ingress. The public access URL should use the LoadBalancer external IP. When `API_PUBLIC_LOAD_BALANCER=true`, `scripts/deploy-k8s.sh` requires `DROBOTICFLOW_API_TOKEN`, requires `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true`, and defaults `DROBOTICFLOW_SESSION_COOKIE_SECURE=true`; if the token is omitted locally, the script can reuse an existing `droboticflow-auth-secret` from the target namespace. Boolean aliases such as `yes`, `on`, and `1` are treated as true by both manifest rendering and deployment flow control.

View File

@ -194,7 +194,7 @@ Run the deployed POC operator-runner full builtin-operator acceptance Job:
npm run acceptance:poc-operators
```
`scripts/poc-operator-full-acceptance.sh` loads the `poc` deployment environment on `dev`, reads the current `OPERATOR_RUNNER_IMAGE`, workspace PVC, Mongo secret, and BOS workspace secret references from Kubernetes, mounts `scripts/poc_operator_full_acceptance_job.py` into a temporary Job, and prints one `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` result. The Job executes builtin modules from the deployed runner image, writes evidence under `/workspace/acceptance/<run-id>/` when the PVC permits the runner user to write there, and otherwise falls back to `/tmp/droboticflow-acceptance/<run-id>/` while reporting `workspaceFallbackReason`. By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, `MONGO_URI`, or a D-Cloud FAT token/accepted response is reported as an acceptance failure instead of being treated as a pass. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_ENDPOINT` and `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose those values. D-Cloud FAT tokens are read from `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` (default `droboticflow-dcloud-secret`) keys `DCLOUD_FAT_AUTH_TOKEN` or `DCLOUD_API_TOKEN`; do not pass tokens through dry-run output.
`scripts/poc-operator-full-acceptance.sh` loads the `poc` deployment environment on `dev`, reads the current `OPERATOR_RUNNER_IMAGE`, workspace PVC, Mongo secret, and BOS workspace secret references from Kubernetes, mounts `scripts/poc_operator_full_acceptance_job.py` into a temporary Job, and prints one `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` result. The Job executes builtin modules from the deployed runner image and writes evidence under `/workspace/acceptance/<run-id>/`; if the PVC is misconfigured and the runner cannot write there, the run fails by default instead of silently accepting temporary evidence. Set `POC_ACCEPTANCE_ALLOW_TMP_FALLBACK=true` only for diagnostic runs that intentionally write to `/tmp/droboticflow-acceptance/<run-id>/` and report `workspaceFallbackReason`. By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, `MONGO_URI`, or a D-Cloud FAT token/accepted response is reported as an acceptance failure instead of being treated as a pass. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_ENDPOINT` and `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose those values. D-Cloud FAT tokens are read from `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` (default `droboticflow-dcloud-secret`) keys `DCLOUD_FAT_AUTH_TOKEN` or `DCLOUD_API_TOKEN`; do not pass tokens through dry-run output.
Run X-platform operator-chain acceptance against an explicit real dataset:

View File

@ -249,6 +249,8 @@ spec:
volumeHandle: ${WORKSPACE_BOS_BUCKET}
mountOptions:
- -o meta_expires=0
- -o allow_other
- -o mount_umask=000
---
apiVersion: v1
kind: PersistentVolumeClaim

View File

@ -14,6 +14,7 @@ POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT="${POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT:-18
POC_ACCEPTANCE_REQUIRED_EXTERNALS="${POC_ACCEPTANCE_REQUIRED_EXTERNALS:-bos,dcloud,mongo}"
POC_OPERATOR_ACCEPTANCE_CLEANUP_JOB="${POC_OPERATOR_ACCEPTANCE_CLEANUP_JOB:-false}"
POC_OPERATOR_ACCEPTANCE_DRY_RUN="${POC_OPERATOR_ACCEPTANCE_DRY_RUN:-false}"
POC_ACCEPTANCE_ALLOW_TMP_FALLBACK="${POC_ACCEPTANCE_ALLOW_TMP_FALLBACK:-false}"
POC_ACCEPTANCE_DCLOUD_SECRET_NAME="${POC_ACCEPTANCE_DCLOUD_SECRET_NAME:-droboticflow-dcloud-secret}"
SCRIPT_PATH="${POC_OPERATOR_ACCEPTANCE_SCRIPT_PATH:-$ROOT_DIR/scripts/poc_operator_full_acceptance_job.py}"
@ -44,6 +45,20 @@ validate_go_duration() {
fi
}
validate_bool() {
local name="$1"
local value="$2"
local normalized
normalized="$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]')"
case "$normalized" in
true|false|1|0|yes|no|on|off) ;;
*)
echo "ERROR: $name must be a boolean value" >&2
exit 1
;;
esac
}
duration_seconds() {
python3 - "$1" <<'PY'
import re
@ -97,6 +112,7 @@ fi
validate_dns_label K8S_NAMESPACE "$K8S_NAMESPACE"
validate_dns_label POC_OPERATOR_ACCEPTANCE_JOB_NAME "$POC_OPERATOR_ACCEPTANCE_JOB_NAME"
validate_go_duration POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT "$POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT"
validate_bool POC_ACCEPTANCE_ALLOW_TMP_FALLBACK "$POC_ACCEPTANCE_ALLOW_TMP_FALLBACK"
OPERATOR_RUNNER_IMAGE="${OPERATOR_RUNNER_IMAGE:-$(config_value OPERATOR_RUNNER_IMAGE)}"
WORKSPACE_PVC_NAME="${WORKSPACE_PVC_NAME:-$(config_value WORKSPACE_PVC_NAME)}"
@ -122,6 +138,7 @@ image_json="$(json_string "$OPERATOR_RUNNER_IMAGE")"
workspace_mount_json="$(json_string "$WORKSPACE_MOUNT_PATH")"
workspace_pvc_json="$(json_string "$WORKSPACE_PVC_NAME")"
required_externals_json="$(json_string "$POC_ACCEPTANCE_REQUIRED_EXTERNALS")"
allow_tmp_fallback_json="$(json_string "$POC_ACCEPTANCE_ALLOW_TMP_FALLBACK")"
bos_endpoint_json="$(json_string "$BOS_ENDPOINT")"
bos_bucket_json="$(json_string "$POC_ACCEPTANCE_BOS_BUCKET")"
dcloud_base_json="$(json_string "${DCLOUD_FAT_BASE_URL:-https://cloud-fat.d-robotics.cc}")"
@ -170,6 +187,8 @@ $image_pull_secrets_yaml
value: $workspace_mount_json
- name: POC_ACCEPTANCE_REQUIRED_EXTERNALS
value: $required_externals_json
- name: POC_ACCEPTANCE_ALLOW_TMP_FALLBACK
value: $allow_tmp_fallback_json
- name: POC_ACCEPTANCE_BOS_BUCKET
value: $bos_bucket_json
- name: POC_ACCEPTANCE_BOS_ENDPOINT

View File

@ -154,6 +154,18 @@ def required_externals() -> set[str]:
return {item.strip().lower() for item in raw.split(",") if item.strip()}
def env_bool(name: str, default: bool = False) -> bool:
raw = os.getenv(name)
if raw is None or not raw.strip():
return default
normalized = raw.strip().lower()
if normalized in {"1", "true", "yes", "on"}:
return True
if normalized in {"0", "false", "no", "off"}:
return False
raise AcceptanceFailure(f"{name} must be a boolean value")
def create_video(path: pathlib.Path, seconds: int = 3) -> dict[str, Any]:
path.parent.mkdir(parents=True, exist_ok=True)
cmd = [
@ -370,6 +382,7 @@ def run_acceptance() -> dict[str, Any]:
run_id = os.getenv("POC_ACCEPTANCE_RUN_ID", f"poc-operator-acceptance-{now_id()}")
workspace_root = pathlib.Path(os.getenv("WORKSPACE_MOUNT_PATH", "/workspace"))
base_dir = pathlib.Path(os.getenv("POC_ACCEPTANCE_WORKDIR", str(workspace_root / "acceptance" / run_id)))
allow_tmp_fallback = env_bool("POC_ACCEPTANCE_ALLOW_TMP_FALLBACK", False)
workspace_fallback_reason = ""
try:
if base_dir.exists():
@ -377,6 +390,11 @@ def run_acceptance() -> dict[str, Any]:
base_dir.mkdir(parents=True)
except OSError as error:
workspace_fallback_reason = f"{type(error).__name__}: {error}"
if not allow_tmp_fallback:
raise AcceptanceFailure(
f"workspace evidence directory is not writable: {base_dir}; {workspace_fallback_reason}. "
"Set POC_ACCEPTANCE_ALLOW_TMP_FALLBACK=true only for diagnostic runs."
) from error
base_dir = pathlib.Path("/tmp/droboticflow-acceptance") / run_id
if base_dir.exists():
shutil.rmtree(base_dir)

View File

@ -24,6 +24,7 @@ def run_script(extra_env: dict[str, str]) -> subprocess.CompletedProcess:
"POC_ACCEPTANCE_BOS_BUCKET",
"POC_ACCEPTANCE_BOS_ENDPOINT",
"POC_ACCEPTANCE_REQUIRED_EXTERNALS",
"POC_ACCEPTANCE_ALLOW_TMP_FALLBACK",
"POC_OPERATOR_ACCEPTANCE_DRY_RUN",
"POC_OPERATOR_ACCEPTANCE_JOB_NAME",
"POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT",
@ -44,6 +45,7 @@ def run_script(extra_env: dict[str, str]) -> subprocess.CompletedProcess:
"POC_OPERATOR_ACCEPTANCE_DRY_RUN": "true",
"POC_OPERATOR_ACCEPTANCE_JOB_NAME": "droboticflow-operator-acceptance-test",
"POC_OPERATOR_ACCEPTANCE_WAIT_TIMEOUT": "300s",
"POC_ACCEPTANCE_ALLOW_TMP_FALLBACK": "false",
"POC_ACCEPTANCE_DCLOUD_SECRET_NAME": "droboticflow-dcloud-secret",
}
)
@ -78,6 +80,7 @@ def test_poc_operator_acceptance_renders_strict_poc_job_manifest():
assert 'namespace: "droboticflow"' in result.stdout
assert 'image: "registry.example.com/dataflow/droboticflow-operator-runner:test"' in result.stdout
assert 'value: "bos,dcloud,mongo"' in result.stdout
assert 'value: "false"' in result.stdout
assert 'value: "poc-bucket"' in result.stdout
assert 'value: "https://bj.bcebos.com"' in result.stdout
assert "name: droboticflow-bos-workspace" in result.stdout