diff --git a/docs/operations.md b/docs/operations.md index 77866539..93a72369 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -267,7 +267,7 @@ The legacy explicit variables remain supported and take precedence over the mani By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, `MONGO_URI`, or accepted D-Cloud FAT responses is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with `POC_ACCEPTANCE_BOS_ENDPOINT` only when validating a different bucket region. When `dcloud` is required, the wrapper checks that `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` exists and contains `DCLOUD_FAT_BASE_URL`, a usable auth path, `DCLOUD_ORG_SYMBOL`, and `DCLOUD_SOURCEAPP` before creating the Kubernetes Job; missing or incomplete D-Cloud context fails fast without printing secret values. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose the bucket name. D-Cloud FAT context is read from `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` (default `droboticflow-dcloud-secret`): provide `DCLOUD_FAT_BASE_URL`, `DCLOUD_FAT_AUTH_TOKEN` or `DCLOUD_API_TOKEN`, `DCLOUD_FAT_COOKIE` or `DCLOUD_COOKIE`, or `DCLOUD_FAT_USERNAME`/`DCLOUD_FAT_PASSWORD` for runtime `/api/signin`; also provide `DCLOUD_ORG_SYMBOL` and `DCLOUD_SOURCEAPP`. `DCLOUD_D_USER_ID` is optional on JWT gateway paths because the gateway maps the token claim to `d-user-id`; keep it only for direct-service diagnostics. Do not pass tokens, cookies, passwords, or secret values through dry-run output. -The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators. +The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `externalVerificationPassed`, `externalVerificationMissing`, `externalVerificationPassedCount`, `externalVerificationRequiredCount`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every required external system is verified, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, Mongo requires a write followed by an independent count/drop verification, and D-Cloud requires both FAT API operators to pass. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators. When the Job emits a `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=` marker, the wrapper extracts it from the Job logs even if the Job exits failed. If `POC_ACCEPTANCE_IMPORT_API_BASE_URL`, `API_BASE_URL`, or `BASE_URL` is set, it posts that result to `/api/acceptance/poc-operator-result` and records it as a normal platform run under the `POC 算子验收` project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, or the shared `DROBOTICFLOW_API_TOKEN` for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a `poc-operator-full-acceptance-result.json` artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Set `POC_ACCEPTANCE_IMPORT_RESULT=false` for one-off diagnostics that should not create or update platform acceptance history. @@ -275,7 +275,7 @@ If strict acceptance cannot even create the Kubernetes Job because required read Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH` or all explicit real fixture inputs, including `POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH`, `POC_ACCEPTANCE_CSV_INPUT_PATH`, `POC_ACCEPTANCE_JSON_INPUT_PATH`, `POC_ACCEPTANCE_LEROBOT_SOURCE_PATH`, `POC_ACCEPTANCE_ROSBAG_SOURCE_PATH`, `POC_ACCEPTANCE_MODEL_JSON_PATH`, and `POC_ACCEPTANCE_SHELL_COMMAND`. Manifest and explicit fixture paths must point under `WORKSPACE_MOUNT_PATH`; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with `fixture-readiness` coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later. -The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance. +The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, `requiredExternals` containing `bos,dcloud,mongo`, `externalVerificationPassed` containing all three systems, empty `externalVerificationMissing`, `externalVerificationPassedCount>=3`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance. Configure or refresh the POC real fixture manifest from local fixture files and directories: diff --git a/internal/api/acceptance_import.go b/internal/api/acceptance_import.go index 12a57522..5165680d 100644 --- a/internal/api/acceptance_import.go +++ b/internal/api/acceptance_import.go @@ -76,6 +76,15 @@ func stringSliceField(payload map[string]any, key string) []string { return items } +func stringSetContains(items []string, value string) bool { + for _, item := range items { + if item == value { + return true + } + } + return false +} + func boolFieldPresent(payload map[string]any, key string) (bool, bool) { value, ok := payload[key].(bool) return value, ok @@ -136,6 +145,30 @@ func validatePocAcceptanceFullConsistency(payload map[string]any) []validationIs if stringField(payload, "acceptanceScope") != "full" { issues = append(issues, validationIssue{Field: "result.acceptanceScope", Message: "acceptanceScope must be full when fullExternalAcceptance is true"}) } + requiredExternals := stringSliceField(payload, "requiredExternals") + for _, external := range []string{"bos", "dcloud", "mongo"} { + if !stringSetContains(requiredExternals, external) { + issues = append(issues, validationIssue{Field: "result.requiredExternals", Message: "requiredExternals must include bos, dcloud, and mongo when fullExternalAcceptance is true"}) + break + } + } + externalVerificationPassed := stringSliceField(payload, "externalVerificationPassed") + for _, external := range []string{"bos", "dcloud", "mongo"} { + if !stringSetContains(externalVerificationPassed, external) { + issues = append(issues, validationIssue{Field: "result.externalVerificationPassed", Message: "externalVerificationPassed must include bos, dcloud, and mongo when fullExternalAcceptance is true"}) + break + } + } + if externalVerificationMissing := stringSliceField(payload, "externalVerificationMissing"); len(externalVerificationMissing) > 0 { + issues = append(issues, validationIssue{Field: "result.externalVerificationMissing", Message: "externalVerificationMissing must be empty when fullExternalAcceptance is true"}) + } + externalVerificationPassedCount, ok := numericFieldPresent(payload, "externalVerificationPassedCount") + if !ok || externalVerificationPassedCount < 3 { + issues = append(issues, validationIssue{Field: "result.externalVerificationPassedCount", Message: "externalVerificationPassedCount must be at least 3 when fullExternalAcceptance is true"}) + } + if externalVerificationRequiredCount, ok := numericFieldPresent(payload, "externalVerificationRequiredCount"); !ok || externalVerificationRequiredCount < 3 { + issues = append(issues, validationIssue{Field: "result.externalVerificationRequiredCount", Message: "externalVerificationRequiredCount must be at least 3 when fullExternalAcceptance is true"}) + } operatorCount, ok := numericFieldPresent(payload, "operatorCount") if !ok || operatorCount <= 0 { issues = append(issues, validationIssue{Field: "result.operatorCount", Message: "operatorCount must be a positive integer"}) diff --git a/internal/api/server_test.go b/internal/api/server_test.go index b6756ac4..dccdfbd9 100644 --- a/internal/api/server_test.go +++ b/internal/api/server_test.go @@ -2567,25 +2567,28 @@ func TestPocAcceptanceImportCreatesPlatformRunArtifact(t *testing.T) { handler := NewHandler(dataStore, nil) payload := map[string]any{ "result": map[string]any{ - "runId": "poc-operator-acceptance-20260623010101", - "status": "passed", - "workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101", - "acceptanceScope": "dependency-isolation", - "fullExternalAcceptance": false, - "operatorCount": 28, - "operatorPassedCount": 26, - "coveredOperatorCount": 26, - "allOperatorCoverageComplete": false, - "blockingOperatorCoverageComplete": true, - "missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"}, - "blockingMissingCoverage": []string{}, - "requiredExternals": []string{"bos", "mongo"}, - "statusCounts": map[string]any{"passed": 28, "skipped": 2}, - "failedCount": 0, - "skippedCount": 2, - "optionalSkippedCount": 2, - "requiredSkippedCount": 0, - "externalVerificationPassedCount": 1, + "runId": "poc-operator-acceptance-20260623010101", + "status": "passed", + "workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101", + "acceptanceScope": "dependency-isolation", + "fullExternalAcceptance": false, + "operatorCount": 28, + "operatorPassedCount": 26, + "coveredOperatorCount": 26, + "allOperatorCoverageComplete": false, + "blockingOperatorCoverageComplete": true, + "missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"}, + "blockingMissingCoverage": []string{}, + "requiredExternals": []string{"bos", "mongo"}, + "statusCounts": map[string]any{"passed": 28, "skipped": 2}, + "failedCount": 0, + "skippedCount": 2, + "optionalSkippedCount": 2, + "requiredSkippedCount": 0, + "externalVerificationPassed": []string{"bos", "mongo"}, + "externalVerificationMissing": []string{}, + "externalVerificationPassedCount": 2, + "externalVerificationRequiredCount": 2, }, "source": map[string]any{ "environment": "poc", @@ -2776,6 +2779,10 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) { body := recorder.Body.String() for _, field := range []string{ "result.acceptanceScope", + "result.requiredExternals", + "result.externalVerificationPassed", + "result.externalVerificationPassedCount", + "result.externalVerificationRequiredCount", "result.operatorPassedCount", "result.coveredOperatorCount", "result.allOperatorCoverageComplete", @@ -2787,27 +2794,89 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) { } } +func TestPocAcceptanceImportRejectsIncompleteExternalVerification(t *testing.T) { + dataStore := store.NewMemoryStore() + handler := NewHandler(dataStore, nil) + payload := map[string]any{ + "result": map[string]any{ + "runId": "poc-operator-acceptance-missing-external-proof", + "status": "passed", + "acceptanceScope": "full", + "fullExternalAcceptance": true, + "operatorCount": 28, + "operatorPassedCount": 28, + "coveredOperatorCount": 28, + "allOperatorCoverageComplete": true, + "blockingOperatorCoverageComplete": true, + "missingCoverage": []string{}, + "blockingMissingCoverage": []string{}, + "requiredExternals": []string{"bos", "dcloud", "mongo"}, + "externalVerificationPassed": []string{"bos", "mongo"}, + "externalVerificationMissing": []string{"dcloud"}, + "externalVerificationPassedCount": 2, + "externalVerificationRequiredCount": 3, + "failedCount": 0, + "requiredSkippedCount": 0, + "generatedFixtureCount": 0, + "realFixtureCount": 6, + "allFixturesReal": true, + "fixtureSources": map[string]any{ + "allowGeneratedFixtures": false, + "items": map[string]any{ + "xplatform": map[string]any{"path": "/workspace/fixtures/xplatform", "generated": false}, + "csv": map[string]any{"path": "/workspace/fixtures/records.csv", "generated": false}, + }, + }, + }, + } + rawPayload, err := json.Marshal(payload) + if err != nil { + t.Fatalf("marshal payload: %v", err) + } + + recorder := httptest.NewRecorder() + handler.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, "/api/acceptance/poc-operator-result", bytes.NewReader(rawPayload))) + if recorder.Code != http.StatusBadRequest { + t.Fatalf("status: %d body=%s", recorder.Code, recorder.Body.String()) + } + body := recorder.Body.String() + for _, field := range []string{ + "result.externalVerificationPassed", + "result.externalVerificationMissing", + "result.externalVerificationPassedCount", + } { + if !strings.Contains(body, field) { + t.Fatalf("expected validation field %s in %s", field, body) + } + } +} + func TestPocAcceptanceImportRejectsGeneratedFixtureFullAcceptance(t *testing.T) { dataStore := store.NewMemoryStore() handler := NewHandler(dataStore, nil) payload := map[string]any{ "result": map[string]any{ - "runId": "poc-operator-acceptance-generated-fixtures", - "status": "passed", - "acceptanceScope": "full", - "fullExternalAcceptance": true, - "operatorCount": 28, - "operatorPassedCount": 28, - "coveredOperatorCount": 28, - "allOperatorCoverageComplete": true, - "blockingOperatorCoverageComplete": true, - "missingCoverage": []string{}, - "blockingMissingCoverage": []string{}, - "failedCount": 0, - "requiredSkippedCount": 0, - "generatedFixtureCount": 1, - "realFixtureCount": 5, - "allFixturesReal": false, + "runId": "poc-operator-acceptance-generated-fixtures", + "status": "passed", + "acceptanceScope": "full", + "fullExternalAcceptance": true, + "operatorCount": 28, + "operatorPassedCount": 28, + "coveredOperatorCount": 28, + "allOperatorCoverageComplete": true, + "blockingOperatorCoverageComplete": true, + "missingCoverage": []string{}, + "blockingMissingCoverage": []string{}, + "requiredExternals": []string{"bos", "dcloud", "mongo"}, + "externalVerificationPassed": []string{"bos", "dcloud", "mongo"}, + "externalVerificationMissing": []string{}, + "externalVerificationPassedCount": 3, + "externalVerificationRequiredCount": 3, + "failedCount": 0, + "requiredSkippedCount": 0, + "generatedFixtureCount": 1, + "realFixtureCount": 5, + "allFixturesReal": false, "fixtureSources": map[string]any{ "allowGeneratedFixtures": true, "items": map[string]any{ diff --git a/scripts/poc-operator-full-acceptance.sh b/scripts/poc-operator-full-acceptance.sh index 11d1809c..125bd55c 100755 --- a/scripts/poc-operator-full-acceptance.sh +++ b/scripts/poc-operator-full-acceptance.sh @@ -520,6 +520,7 @@ for stage, reason in zip(stages, reasons, strict=True): unique_stages = list(dict.fromkeys(stages)) stage = unique_stages[0] if len(unique_stages) == 1 else "preflight-readiness" missing_coverage = sorted(missing_coverage_set) +required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required))) payload = { "runId": run_id, "workspace": f"{workspace_mount}/acceptance/{run_id}", @@ -538,7 +539,10 @@ payload = { "requiredExternals": required, "statusCounts": {"failed": len(records)}, "passedCount": 0, + "externalVerificationPassed": [], + "externalVerificationMissing": required_external_systems, "externalVerificationPassedCount": 0, + "externalVerificationRequiredCount": len(required_external_systems), "failedCount": len(records), "skippedCount": 0, "requiredSkippedCount": 0, diff --git a/scripts/poc_operator_full_acceptance_job.py b/scripts/poc_operator_full_acceptance_job.py index 99e8040f..5cb31b8b 100755 --- a/scripts/poc_operator_full_acceptance_job.py +++ b/scripts/poc_operator_full_acceptance_job.py @@ -235,6 +235,31 @@ def acceptance_scope(required: set[str]) -> str: return "dependency-isolation" +def has_passed_record(records: list[dict[str, Any]], operator: str) -> bool: + return any(item.get("operator") == operator and item.get("status") == "passed" for item in records) + + +def external_verification_summary(records: list[dict[str, Any]], required: set[str]) -> dict[str, Any]: + required_external_systems = sorted(DEFAULT_REQUIRED_EXTERNALS.intersection(required)) + passed: list[str] = [] + if "bos" in required and has_passed_record(records, "upload_bos_external") and has_passed_record(records, "bos_incremental_scan"): + passed.append("bos") + if "mongo" in required and has_passed_record(records, "mongodb_write_external_verification"): + passed.append("mongo") + if ( + "dcloud" in required + and has_passed_record(records, "dcloud_embodied_annotation") + and has_passed_record(records, "dcloud_data_generation") + ): + passed.append("dcloud") + return { + "externalVerificationPassed": sorted(passed), + "externalVerificationMissing": sorted(set(required_external_systems) - set(passed)), + "externalVerificationPassedCount": len(passed), + "externalVerificationRequiredCount": len(required_external_systems), + } + + def env_bool(name: str, default: bool = False) -> bool: raw = os.getenv(name) if raw is None or not raw.strip(): @@ -1520,6 +1545,7 @@ def run_acceptance() -> dict[str, Any]: failures.append(f"operator failed: {item.get('operator')} ({item.get('error')})") scope = acceptance_scope(required) all_operator_coverage_complete = not missing_coverage + external_summary = external_verification_summary(records, required) summary = { "runId": run_id, @@ -1537,9 +1563,7 @@ def run_acceptance() -> dict[str, Any]: "requiredExternals": sorted(required), "statusCounts": status_counts, "passedCount": len(passed), - "externalVerificationPassedCount": len( - [item for item in passed if str(item.get("operator", "")).endswith("_external_verification")] - ), + **external_summary, "failedCount": len(failed), "skippedCount": len(required_skips) + len(optional_skips), "requiredSkippedCount": len(required_skips), diff --git a/scripts/poc_strict_preflight_result.py b/scripts/poc_strict_preflight_result.py index c314d0b9..dfd38360 100644 --- a/scripts/poc_strict_preflight_result.py +++ b/scripts/poc_strict_preflight_result.py @@ -212,6 +212,7 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]: unique_stages = list(dict.fromkeys(stages)) stage = unique_stages[0] if len(unique_stages) == 1 else "strict-preflight-readiness" missing_coverage = sorted(missing_coverage_set) + required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required))) return { "runId": run_id, "workspace": f"{workspace_mount}/acceptance/{run_id}", @@ -230,7 +231,10 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]: "requiredExternals": required, "statusCounts": {"failed": len(records)}, "passedCount": 0, + "externalVerificationPassed": [], + "externalVerificationMissing": required_external_systems, "externalVerificationPassedCount": 0, + "externalVerificationRequiredCount": len(required_external_systems), "failedCount": len(records), "skippedCount": 0, "requiredSkippedCount": 0, diff --git a/src/components/PocAcceptanceSummaryCard.tsx b/src/components/PocAcceptanceSummaryCard.tsx index 0e1f36a5..564dce67 100644 --- a/src/components/PocAcceptanceSummaryCard.tsx +++ b/src/components/PocAcceptanceSummaryCard.tsx @@ -133,6 +133,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor .join(" / "); const coverage = `${summary.operatorPassedCount}/${summary.operatorCount}`; const covered = `${summary.coveredOperatorCount}/${summary.operatorCount}`; + const externalVerification = `${summary.externalVerificationPassedCount}/${summary.externalVerificationRequiredCount || "--"}`; return (
@@ -157,7 +158,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor - + @@ -230,6 +231,8 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
+ +
diff --git a/src/lib/poc-acceptance-summary.js b/src/lib/poc-acceptance-summary.js index 5f0437df..eaf74d0d 100644 --- a/src/lib/poc-acceptance-summary.js +++ b/src/lib/poc-acceptance-summary.js @@ -117,6 +117,8 @@ export function summarizePocAcceptanceArtifact(artifact) { const optionalSkippedCount = integerOrZero(payload.optionalSkippedCount); const requiredSkippedCount = integerOrZero(payload.requiredSkippedCount); const externalVerificationPassedCount = integerOrZero(payload.externalVerificationPassedCount); + const externalVerificationPassed = stringList(payload.externalVerificationPassed); + const externalVerificationMissing = stringList(payload.externalVerificationMissing); const fullExternalAcceptance = payload.fullExternalAcceptance === true; const allOperatorCoverageComplete = payload.allOperatorCoverageComplete === true; const blockingOperatorCoverageComplete = payload.blockingOperatorCoverageComplete === true; @@ -153,6 +155,12 @@ export function summarizePocAcceptanceArtifact(artifact) { const strictExternalScope = acceptanceScope === "full" && ["bos", "dcloud", "mongo"].every((external) => requiredExternals.includes(external)); + const externalVerificationRequiredCount = + integerOrNull(payload.externalVerificationRequiredCount) ?? requiredExternals.length; + const externalVerificationComplete = + externalVerificationPassedCount >= Math.max(externalVerificationRequiredCount, requiredExternals.length) && + externalVerificationMissing.length === 0 && + requiredExternals.every((external) => externalVerificationPassed.includes(external)); const strictOperatorCoverage = operatorCount > 0 && integerOrNull(payload.operatorPassedCount) === operatorCount && @@ -168,6 +176,7 @@ export function summarizePocAcceptanceArtifact(artifact) { status === "passed" && fullExternalAcceptance && strictExternalScope && + externalVerificationComplete && strictOperatorCoverage && fixtureEvidenceComplete; const incompleteFullFixtureEvidence = status === "passed" && fullExternalAcceptance && !allFixturesReal; @@ -204,6 +213,9 @@ export function summarizePocAcceptanceArtifact(artifact) { optionalSkippedCount, requiredSkippedCount, externalVerificationPassedCount, + externalVerificationRequiredCount, + externalVerificationPassed, + externalVerificationMissing, workspace: cleanString(payload.workspace), workspaceFallbackReason: cleanString(payload.workspaceFallbackReason), preflightFailure, diff --git a/tests/backend/poc_operator_acceptance_job_test.py b/tests/backend/poc_operator_acceptance_job_test.py index d96227cd..8e147f70 100644 --- a/tests/backend/poc_operator_acceptance_job_test.py +++ b/tests/backend/poc_operator_acceptance_job_test.py @@ -25,6 +25,53 @@ def test_acceptance_scope_requires_all_strict_externals(): assert module.acceptance_scope(set()) == "dependency-isolation" +def test_external_verification_summary_counts_required_systems(): + module = load_job_module() + records = [ + {"operator": "upload_bos_external", "status": "passed"}, + {"operator": "bos_incremental_scan", "status": "passed"}, + {"operator": "mongodb_write_external_verification", "status": "passed"}, + {"operator": "dcloud_embodied_annotation", "status": "skipped"}, + {"operator": "dcloud_data_generation", "status": "skipped"}, + ] + + summary = module.external_verification_summary(records, {"bos", "mongo"}) + + assert summary == { + "externalVerificationPassed": ["bos", "mongo"], + "externalVerificationMissing": [], + "externalVerificationPassedCount": 2, + "externalVerificationRequiredCount": 2, + } + + +def test_external_verification_summary_requires_all_dcloud_operators(): + module = load_job_module() + records = [ + {"operator": "upload_bos_external", "status": "passed"}, + {"operator": "bos_incremental_scan", "status": "passed"}, + {"operator": "mongodb_write_external_verification", "status": "passed"}, + {"operator": "dcloud_embodied_annotation", "status": "passed"}, + {"operator": "dcloud_data_generation", "status": "failed"}, + ] + + summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"}) + + assert summary == { + "externalVerificationPassed": ["bos", "mongo"], + "externalVerificationMissing": ["dcloud"], + "externalVerificationPassedCount": 2, + "externalVerificationRequiredCount": 3, + } + + records[-1]["status"] = "passed" + summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"}) + assert summary["externalVerificationPassed"] == ["bos", "dcloud", "mongo"] + assert summary["externalVerificationMissing"] == [] + assert summary["externalVerificationPassedCount"] == 3 + assert summary["externalVerificationRequiredCount"] == 3 + + def test_dcloud_acceptance_probe_is_loaded_from_operator_metadata(): module = load_job_module() diff --git a/tests/frontend/poc-acceptance-summary.test.mjs b/tests/frontend/poc-acceptance-summary.test.mjs index 2ed03e10..2bdce704 100644 --- a/tests/frontend/poc-acceptance-summary.test.mjs +++ b/tests/frontend/poc-acceptance-summary.test.mjs @@ -27,6 +27,9 @@ const fullPayload = { optionalSkippedCount: 0, requiredSkippedCount: 0, externalVerificationPassedCount: 3, + externalVerificationRequiredCount: 3, + externalVerificationPassed: ["bos", "dcloud", "mongo"], + externalVerificationMissing: [], generatedFixtureCount: 0, realFixtureCount: 2, allFixturesReal: true, @@ -55,6 +58,10 @@ test("summarizes strict full POC operator acceptance artifacts", () => { assert.equal(summary.fullExternalAcceptance, true); assert.equal(summary.operatorPassedCount, 28); assert.deepEqual(summary.requiredExternals, ["bos", "dcloud", "mongo"]); + assert.equal(summary.externalVerificationPassedCount, 3); + assert.equal(summary.externalVerificationRequiredCount, 3); + assert.deepEqual(summary.externalVerificationPassed, ["bos", "dcloud", "mongo"]); + assert.deepEqual(summary.externalVerificationMissing, []); assert.equal(summary.generatedFixtureCount, 0); assert.equal(summary.realFixtureCount, 2); assert.equal(summary.allFixturesReal, true); @@ -123,6 +130,14 @@ test("requires full scope and complete operator coverage for strict pass", () => name: "required-skipped", patch: { requiredSkippedCount: 1 }, }, + { + name: "missing-external-verification", + patch: { + externalVerificationPassedCount: 2, + externalVerificationPassed: ["bos", "mongo"], + externalVerificationMissing: ["dcloud"], + }, + }, ]; for (const item of cases) { @@ -157,6 +172,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => { requiredExternals: ["bos", "mongo"], skippedCount: 2, optionalSkippedCount: 2, + externalVerificationPassedCount: 2, + externalVerificationRequiredCount: 2, + externalVerificationPassed: ["bos", "mongo"], + externalVerificationMissing: [], fixtureSources: { allowGeneratedFixtures: false, fixtureManifestPath: "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json", @@ -176,6 +195,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => { assert.deepEqual(summary.missingCoverage, ["dcloud_data_generation", "dcloud_embodied_annotation"]); assert.equal(summary.fixtureManifestPath, "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json"); assert.equal(summary.generatedFixtureCount, 0); + assert.equal(summary.externalVerificationPassedCount, 2); + assert.equal(summary.externalVerificationRequiredCount, 2); + assert.deepEqual(summary.externalVerificationPassed, ["bos", "mongo"]); + assert.deepEqual(summary.externalVerificationMissing, []); assert.equal(summary.realFixtureCount, 2); assert.equal(summary.allFixturesReal, true); assert.deepEqual(summary.fixtureSources, [