✅ Count external POC verification systems

This commit is contained in:
eust-w 2026-06-23 17:57:32 +08:00
parent cdfde23562
commit 7cfb0da433
10 changed files with 260 additions and 41 deletions

View File

@ -267,7 +267,7 @@ The legacy explicit variables remain supported and take precedence over the mani
By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, `MONGO_URI`, or accepted D-Cloud FAT responses is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with `POC_ACCEPTANCE_BOS_ENDPOINT` only when validating a different bucket region. When `dcloud` is required, the wrapper checks that `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` exists and contains `DCLOUD_FAT_BASE_URL`, a usable auth path, `DCLOUD_ORG_SYMBOL`, and `DCLOUD_SOURCEAPP` before creating the Kubernetes Job; missing or incomplete D-Cloud context fails fast without printing secret values. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose the bucket name. D-Cloud FAT context is read from `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` (default `droboticflow-dcloud-secret`): provide `DCLOUD_FAT_BASE_URL`, `DCLOUD_FAT_AUTH_TOKEN` or `DCLOUD_API_TOKEN`, `DCLOUD_FAT_COOKIE` or `DCLOUD_COOKIE`, or `DCLOUD_FAT_USERNAME`/`DCLOUD_FAT_PASSWORD` for runtime `/api/signin`; also provide `DCLOUD_ORG_SYMBOL` and `DCLOUD_SOURCEAPP`. `DCLOUD_D_USER_ID` is optional on JWT gateway paths because the gateway maps the token claim to `d-user-id`; keep it only for direct-service diagnostics. Do not pass tokens, cookies, passwords, or secret values through dry-run output.
The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators.
The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `externalVerificationPassed`, `externalVerificationMissing`, `externalVerificationPassedCount`, `externalVerificationRequiredCount`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every required external system is verified, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, Mongo requires a write followed by an independent count/drop verification, and D-Cloud requires both FAT API operators to pass. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators.
When the Job emits a `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` marker, the wrapper extracts it from the Job logs even if the Job exits failed. If `POC_ACCEPTANCE_IMPORT_API_BASE_URL`, `API_BASE_URL`, or `BASE_URL` is set, it posts that result to `/api/acceptance/poc-operator-result` and records it as a normal platform run under the `POC 算子验收` project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, or the shared `DROBOTICFLOW_API_TOKEN` for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a `poc-operator-full-acceptance-result.json` artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Set `POC_ACCEPTANCE_IMPORT_RESULT=false` for one-off diagnostics that should not create or update platform acceptance history.
@ -275,7 +275,7 @@ If strict acceptance cannot even create the Kubernetes Job because required read
Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH` or all explicit real fixture inputs, including `POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH`, `POC_ACCEPTANCE_CSV_INPUT_PATH`, `POC_ACCEPTANCE_JSON_INPUT_PATH`, `POC_ACCEPTANCE_LEROBOT_SOURCE_PATH`, `POC_ACCEPTANCE_ROSBAG_SOURCE_PATH`, `POC_ACCEPTANCE_MODEL_JSON_PATH`, and `POC_ACCEPTANCE_SHELL_COMMAND`. Manifest and explicit fixture paths must point under `WORKSPACE_MOUNT_PATH`; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with `fixture-readiness` coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later.
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, `requiredExternals` containing `bos,dcloud,mongo`, `externalVerificationPassed` containing all three systems, empty `externalVerificationMissing`, `externalVerificationPassedCount>=3`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
Configure or refresh the POC real fixture manifest from local fixture files and directories:

View File

@ -76,6 +76,15 @@ func stringSliceField(payload map[string]any, key string) []string {
return items
}
func stringSetContains(items []string, value string) bool {
for _, item := range items {
if item == value {
return true
}
}
return false
}
func boolFieldPresent(payload map[string]any, key string) (bool, bool) {
value, ok := payload[key].(bool)
return value, ok
@ -136,6 +145,30 @@ func validatePocAcceptanceFullConsistency(payload map[string]any) []validationIs
if stringField(payload, "acceptanceScope") != "full" {
issues = append(issues, validationIssue{Field: "result.acceptanceScope", Message: "acceptanceScope must be full when fullExternalAcceptance is true"})
}
requiredExternals := stringSliceField(payload, "requiredExternals")
for _, external := range []string{"bos", "dcloud", "mongo"} {
if !stringSetContains(requiredExternals, external) {
issues = append(issues, validationIssue{Field: "result.requiredExternals", Message: "requiredExternals must include bos, dcloud, and mongo when fullExternalAcceptance is true"})
break
}
}
externalVerificationPassed := stringSliceField(payload, "externalVerificationPassed")
for _, external := range []string{"bos", "dcloud", "mongo"} {
if !stringSetContains(externalVerificationPassed, external) {
issues = append(issues, validationIssue{Field: "result.externalVerificationPassed", Message: "externalVerificationPassed must include bos, dcloud, and mongo when fullExternalAcceptance is true"})
break
}
}
if externalVerificationMissing := stringSliceField(payload, "externalVerificationMissing"); len(externalVerificationMissing) > 0 {
issues = append(issues, validationIssue{Field: "result.externalVerificationMissing", Message: "externalVerificationMissing must be empty when fullExternalAcceptance is true"})
}
externalVerificationPassedCount, ok := numericFieldPresent(payload, "externalVerificationPassedCount")
if !ok || externalVerificationPassedCount < 3 {
issues = append(issues, validationIssue{Field: "result.externalVerificationPassedCount", Message: "externalVerificationPassedCount must be at least 3 when fullExternalAcceptance is true"})
}
if externalVerificationRequiredCount, ok := numericFieldPresent(payload, "externalVerificationRequiredCount"); !ok || externalVerificationRequiredCount < 3 {
issues = append(issues, validationIssue{Field: "result.externalVerificationRequiredCount", Message: "externalVerificationRequiredCount must be at least 3 when fullExternalAcceptance is true"})
}
operatorCount, ok := numericFieldPresent(payload, "operatorCount")
if !ok || operatorCount <= 0 {
issues = append(issues, validationIssue{Field: "result.operatorCount", Message: "operatorCount must be a positive integer"})

View File

@ -2567,25 +2567,28 @@ func TestPocAcceptanceImportCreatesPlatformRunArtifact(t *testing.T) {
handler := NewHandler(dataStore, nil)
payload := map[string]any{
"result": map[string]any{
"runId": "poc-operator-acceptance-20260623010101",
"status": "passed",
"workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101",
"acceptanceScope": "dependency-isolation",
"fullExternalAcceptance": false,
"operatorCount": 28,
"operatorPassedCount": 26,
"coveredOperatorCount": 26,
"allOperatorCoverageComplete": false,
"blockingOperatorCoverageComplete": true,
"missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"},
"blockingMissingCoverage": []string{},
"requiredExternals": []string{"bos", "mongo"},
"statusCounts": map[string]any{"passed": 28, "skipped": 2},
"failedCount": 0,
"skippedCount": 2,
"optionalSkippedCount": 2,
"requiredSkippedCount": 0,
"externalVerificationPassedCount": 1,
"runId": "poc-operator-acceptance-20260623010101",
"status": "passed",
"workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101",
"acceptanceScope": "dependency-isolation",
"fullExternalAcceptance": false,
"operatorCount": 28,
"operatorPassedCount": 26,
"coveredOperatorCount": 26,
"allOperatorCoverageComplete": false,
"blockingOperatorCoverageComplete": true,
"missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"},
"blockingMissingCoverage": []string{},
"requiredExternals": []string{"bos", "mongo"},
"statusCounts": map[string]any{"passed": 28, "skipped": 2},
"failedCount": 0,
"skippedCount": 2,
"optionalSkippedCount": 2,
"requiredSkippedCount": 0,
"externalVerificationPassed": []string{"bos", "mongo"},
"externalVerificationMissing": []string{},
"externalVerificationPassedCount": 2,
"externalVerificationRequiredCount": 2,
},
"source": map[string]any{
"environment": "poc",
@ -2776,6 +2779,10 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) {
body := recorder.Body.String()
for _, field := range []string{
"result.acceptanceScope",
"result.requiredExternals",
"result.externalVerificationPassed",
"result.externalVerificationPassedCount",
"result.externalVerificationRequiredCount",
"result.operatorPassedCount",
"result.coveredOperatorCount",
"result.allOperatorCoverageComplete",
@ -2787,27 +2794,89 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) {
}
}
func TestPocAcceptanceImportRejectsIncompleteExternalVerification(t *testing.T) {
dataStore := store.NewMemoryStore()
handler := NewHandler(dataStore, nil)
payload := map[string]any{
"result": map[string]any{
"runId": "poc-operator-acceptance-missing-external-proof",
"status": "passed",
"acceptanceScope": "full",
"fullExternalAcceptance": true,
"operatorCount": 28,
"operatorPassedCount": 28,
"coveredOperatorCount": 28,
"allOperatorCoverageComplete": true,
"blockingOperatorCoverageComplete": true,
"missingCoverage": []string{},
"blockingMissingCoverage": []string{},
"requiredExternals": []string{"bos", "dcloud", "mongo"},
"externalVerificationPassed": []string{"bos", "mongo"},
"externalVerificationMissing": []string{"dcloud"},
"externalVerificationPassedCount": 2,
"externalVerificationRequiredCount": 3,
"failedCount": 0,
"requiredSkippedCount": 0,
"generatedFixtureCount": 0,
"realFixtureCount": 6,
"allFixturesReal": true,
"fixtureSources": map[string]any{
"allowGeneratedFixtures": false,
"items": map[string]any{
"xplatform": map[string]any{"path": "/workspace/fixtures/xplatform", "generated": false},
"csv": map[string]any{"path": "/workspace/fixtures/records.csv", "generated": false},
},
},
},
}
rawPayload, err := json.Marshal(payload)
if err != nil {
t.Fatalf("marshal payload: %v", err)
}
recorder := httptest.NewRecorder()
handler.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, "/api/acceptance/poc-operator-result", bytes.NewReader(rawPayload)))
if recorder.Code != http.StatusBadRequest {
t.Fatalf("status: %d body=%s", recorder.Code, recorder.Body.String())
}
body := recorder.Body.String()
for _, field := range []string{
"result.externalVerificationPassed",
"result.externalVerificationMissing",
"result.externalVerificationPassedCount",
} {
if !strings.Contains(body, field) {
t.Fatalf("expected validation field %s in %s", field, body)
}
}
}
func TestPocAcceptanceImportRejectsGeneratedFixtureFullAcceptance(t *testing.T) {
dataStore := store.NewMemoryStore()
handler := NewHandler(dataStore, nil)
payload := map[string]any{
"result": map[string]any{
"runId": "poc-operator-acceptance-generated-fixtures",
"status": "passed",
"acceptanceScope": "full",
"fullExternalAcceptance": true,
"operatorCount": 28,
"operatorPassedCount": 28,
"coveredOperatorCount": 28,
"allOperatorCoverageComplete": true,
"blockingOperatorCoverageComplete": true,
"missingCoverage": []string{},
"blockingMissingCoverage": []string{},
"failedCount": 0,
"requiredSkippedCount": 0,
"generatedFixtureCount": 1,
"realFixtureCount": 5,
"allFixturesReal": false,
"runId": "poc-operator-acceptance-generated-fixtures",
"status": "passed",
"acceptanceScope": "full",
"fullExternalAcceptance": true,
"operatorCount": 28,
"operatorPassedCount": 28,
"coveredOperatorCount": 28,
"allOperatorCoverageComplete": true,
"blockingOperatorCoverageComplete": true,
"missingCoverage": []string{},
"blockingMissingCoverage": []string{},
"requiredExternals": []string{"bos", "dcloud", "mongo"},
"externalVerificationPassed": []string{"bos", "dcloud", "mongo"},
"externalVerificationMissing": []string{},
"externalVerificationPassedCount": 3,
"externalVerificationRequiredCount": 3,
"failedCount": 0,
"requiredSkippedCount": 0,
"generatedFixtureCount": 1,
"realFixtureCount": 5,
"allFixturesReal": false,
"fixtureSources": map[string]any{
"allowGeneratedFixtures": true,
"items": map[string]any{

View File

@ -520,6 +520,7 @@ for stage, reason in zip(stages, reasons, strict=True):
unique_stages = list(dict.fromkeys(stages))
stage = unique_stages[0] if len(unique_stages) == 1 else "preflight-readiness"
missing_coverage = sorted(missing_coverage_set)
required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required)))
payload = {
"runId": run_id,
"workspace": f"{workspace_mount}/acceptance/{run_id}",
@ -538,7 +539,10 @@ payload = {
"requiredExternals": required,
"statusCounts": {"failed": len(records)},
"passedCount": 0,
"externalVerificationPassed": [],
"externalVerificationMissing": required_external_systems,
"externalVerificationPassedCount": 0,
"externalVerificationRequiredCount": len(required_external_systems),
"failedCount": len(records),
"skippedCount": 0,
"requiredSkippedCount": 0,

View File

@ -235,6 +235,31 @@ def acceptance_scope(required: set[str]) -> str:
return "dependency-isolation"
def has_passed_record(records: list[dict[str, Any]], operator: str) -> bool:
return any(item.get("operator") == operator and item.get("status") == "passed" for item in records)
def external_verification_summary(records: list[dict[str, Any]], required: set[str]) -> dict[str, Any]:
required_external_systems = sorted(DEFAULT_REQUIRED_EXTERNALS.intersection(required))
passed: list[str] = []
if "bos" in required and has_passed_record(records, "upload_bos_external") and has_passed_record(records, "bos_incremental_scan"):
passed.append("bos")
if "mongo" in required and has_passed_record(records, "mongodb_write_external_verification"):
passed.append("mongo")
if (
"dcloud" in required
and has_passed_record(records, "dcloud_embodied_annotation")
and has_passed_record(records, "dcloud_data_generation")
):
passed.append("dcloud")
return {
"externalVerificationPassed": sorted(passed),
"externalVerificationMissing": sorted(set(required_external_systems) - set(passed)),
"externalVerificationPassedCount": len(passed),
"externalVerificationRequiredCount": len(required_external_systems),
}
def env_bool(name: str, default: bool = False) -> bool:
raw = os.getenv(name)
if raw is None or not raw.strip():
@ -1520,6 +1545,7 @@ def run_acceptance() -> dict[str, Any]:
failures.append(f"operator failed: {item.get('operator')} ({item.get('error')})")
scope = acceptance_scope(required)
all_operator_coverage_complete = not missing_coverage
external_summary = external_verification_summary(records, required)
summary = {
"runId": run_id,
@ -1537,9 +1563,7 @@ def run_acceptance() -> dict[str, Any]:
"requiredExternals": sorted(required),
"statusCounts": status_counts,
"passedCount": len(passed),
"externalVerificationPassedCount": len(
[item for item in passed if str(item.get("operator", "")).endswith("_external_verification")]
),
**external_summary,
"failedCount": len(failed),
"skippedCount": len(required_skips) + len(optional_skips),
"requiredSkippedCount": len(required_skips),

View File

@ -212,6 +212,7 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]:
unique_stages = list(dict.fromkeys(stages))
stage = unique_stages[0] if len(unique_stages) == 1 else "strict-preflight-readiness"
missing_coverage = sorted(missing_coverage_set)
required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required)))
return {
"runId": run_id,
"workspace": f"{workspace_mount}/acceptance/{run_id}",
@ -230,7 +231,10 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]:
"requiredExternals": required,
"statusCounts": {"failed": len(records)},
"passedCount": 0,
"externalVerificationPassed": [],
"externalVerificationMissing": required_external_systems,
"externalVerificationPassedCount": 0,
"externalVerificationRequiredCount": len(required_external_systems),
"failedCount": len(records),
"skippedCount": 0,
"requiredSkippedCount": 0,

View File

@ -133,6 +133,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
.join(" / ");
const coverage = `${summary.operatorPassedCount}/${summary.operatorCount}`;
const covered = `${summary.coveredOperatorCount}/${summary.operatorCount}`;
const externalVerification = `${summary.externalVerificationPassedCount}/${summary.externalVerificationRequiredCount || "--"}`;
return (
<div className="rounded-lg border border-[#2a2b2f] bg-[#0f1012] p-4">
@ -157,7 +158,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
<Metric label="完整外部验收" value={summary.fullExternalAcceptance ? "true" : "false"} />
<Metric label="通过算子" value={coverage} />
<Metric label="覆盖算子" value={covered} />
<Metric label="外部校验" value={summary.externalVerificationPassedCount} />
<Metric label="外部校验" value={externalVerification} />
<Metric label="失败记录" value={summary.failedCount} />
<Metric label="跳过记录" value={`${summary.skippedCount} (${summary.optionalSkippedCount} optional)`} />
<Metric label="必需跳过" value={summary.requiredSkippedCount} />
@ -230,6 +231,8 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
<div className="mt-4 space-y-3">
<ChipList label="阻塞缺口" items={summary.blockingMissingCoverage} tone="danger" />
<ChipList label="未覆盖算子" items={summary.missingCoverage} tone="warning" />
<ChipList label="缺失外部校验" items={summary.externalVerificationMissing} tone="danger" />
<ChipList label="已通过外部校验" items={summary.externalVerificationPassed} tone="warning" />
<FixtureSourceList sources={summary.fixtureSources} />
<RecordList records={summary.records} />
</div>

View File

@ -117,6 +117,8 @@ export function summarizePocAcceptanceArtifact(artifact) {
const optionalSkippedCount = integerOrZero(payload.optionalSkippedCount);
const requiredSkippedCount = integerOrZero(payload.requiredSkippedCount);
const externalVerificationPassedCount = integerOrZero(payload.externalVerificationPassedCount);
const externalVerificationPassed = stringList(payload.externalVerificationPassed);
const externalVerificationMissing = stringList(payload.externalVerificationMissing);
const fullExternalAcceptance = payload.fullExternalAcceptance === true;
const allOperatorCoverageComplete = payload.allOperatorCoverageComplete === true;
const blockingOperatorCoverageComplete = payload.blockingOperatorCoverageComplete === true;
@ -153,6 +155,12 @@ export function summarizePocAcceptanceArtifact(artifact) {
const strictExternalScope =
acceptanceScope === "full" &&
["bos", "dcloud", "mongo"].every((external) => requiredExternals.includes(external));
const externalVerificationRequiredCount =
integerOrNull(payload.externalVerificationRequiredCount) ?? requiredExternals.length;
const externalVerificationComplete =
externalVerificationPassedCount >= Math.max(externalVerificationRequiredCount, requiredExternals.length) &&
externalVerificationMissing.length === 0 &&
requiredExternals.every((external) => externalVerificationPassed.includes(external));
const strictOperatorCoverage =
operatorCount > 0 &&
integerOrNull(payload.operatorPassedCount) === operatorCount &&
@ -168,6 +176,7 @@ export function summarizePocAcceptanceArtifact(artifact) {
status === "passed" &&
fullExternalAcceptance &&
strictExternalScope &&
externalVerificationComplete &&
strictOperatorCoverage &&
fixtureEvidenceComplete;
const incompleteFullFixtureEvidence = status === "passed" && fullExternalAcceptance && !allFixturesReal;
@ -204,6 +213,9 @@ export function summarizePocAcceptanceArtifact(artifact) {
optionalSkippedCount,
requiredSkippedCount,
externalVerificationPassedCount,
externalVerificationRequiredCount,
externalVerificationPassed,
externalVerificationMissing,
workspace: cleanString(payload.workspace),
workspaceFallbackReason: cleanString(payload.workspaceFallbackReason),
preflightFailure,

View File

@ -25,6 +25,53 @@ def test_acceptance_scope_requires_all_strict_externals():
assert module.acceptance_scope(set()) == "dependency-isolation"
def test_external_verification_summary_counts_required_systems():
module = load_job_module()
records = [
{"operator": "upload_bos_external", "status": "passed"},
{"operator": "bos_incremental_scan", "status": "passed"},
{"operator": "mongodb_write_external_verification", "status": "passed"},
{"operator": "dcloud_embodied_annotation", "status": "skipped"},
{"operator": "dcloud_data_generation", "status": "skipped"},
]
summary = module.external_verification_summary(records, {"bos", "mongo"})
assert summary == {
"externalVerificationPassed": ["bos", "mongo"],
"externalVerificationMissing": [],
"externalVerificationPassedCount": 2,
"externalVerificationRequiredCount": 2,
}
def test_external_verification_summary_requires_all_dcloud_operators():
module = load_job_module()
records = [
{"operator": "upload_bos_external", "status": "passed"},
{"operator": "bos_incremental_scan", "status": "passed"},
{"operator": "mongodb_write_external_verification", "status": "passed"},
{"operator": "dcloud_embodied_annotation", "status": "passed"},
{"operator": "dcloud_data_generation", "status": "failed"},
]
summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"})
assert summary == {
"externalVerificationPassed": ["bos", "mongo"],
"externalVerificationMissing": ["dcloud"],
"externalVerificationPassedCount": 2,
"externalVerificationRequiredCount": 3,
}
records[-1]["status"] = "passed"
summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"})
assert summary["externalVerificationPassed"] == ["bos", "dcloud", "mongo"]
assert summary["externalVerificationMissing"] == []
assert summary["externalVerificationPassedCount"] == 3
assert summary["externalVerificationRequiredCount"] == 3
def test_dcloud_acceptance_probe_is_loaded_from_operator_metadata():
module = load_job_module()

View File

@ -27,6 +27,9 @@ const fullPayload = {
optionalSkippedCount: 0,
requiredSkippedCount: 0,
externalVerificationPassedCount: 3,
externalVerificationRequiredCount: 3,
externalVerificationPassed: ["bos", "dcloud", "mongo"],
externalVerificationMissing: [],
generatedFixtureCount: 0,
realFixtureCount: 2,
allFixturesReal: true,
@ -55,6 +58,10 @@ test("summarizes strict full POC operator acceptance artifacts", () => {
assert.equal(summary.fullExternalAcceptance, true);
assert.equal(summary.operatorPassedCount, 28);
assert.deepEqual(summary.requiredExternals, ["bos", "dcloud", "mongo"]);
assert.equal(summary.externalVerificationPassedCount, 3);
assert.equal(summary.externalVerificationRequiredCount, 3);
assert.deepEqual(summary.externalVerificationPassed, ["bos", "dcloud", "mongo"]);
assert.deepEqual(summary.externalVerificationMissing, []);
assert.equal(summary.generatedFixtureCount, 0);
assert.equal(summary.realFixtureCount, 2);
assert.equal(summary.allFixturesReal, true);
@ -123,6 +130,14 @@ test("requires full scope and complete operator coverage for strict pass", () =>
name: "required-skipped",
patch: { requiredSkippedCount: 1 },
},
{
name: "missing-external-verification",
patch: {
externalVerificationPassedCount: 2,
externalVerificationPassed: ["bos", "mongo"],
externalVerificationMissing: ["dcloud"],
},
},
];
for (const item of cases) {
@ -157,6 +172,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => {
requiredExternals: ["bos", "mongo"],
skippedCount: 2,
optionalSkippedCount: 2,
externalVerificationPassedCount: 2,
externalVerificationRequiredCount: 2,
externalVerificationPassed: ["bos", "mongo"],
externalVerificationMissing: [],
fixtureSources: {
allowGeneratedFixtures: false,
fixtureManifestPath: "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json",
@ -176,6 +195,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => {
assert.deepEqual(summary.missingCoverage, ["dcloud_data_generation", "dcloud_embodied_annotation"]);
assert.equal(summary.fixtureManifestPath, "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json");
assert.equal(summary.generatedFixtureCount, 0);
assert.equal(summary.externalVerificationPassedCount, 2);
assert.equal(summary.externalVerificationRequiredCount, 2);
assert.deepEqual(summary.externalVerificationPassed, ["bos", "mongo"]);
assert.deepEqual(summary.externalVerificationMissing, []);
assert.equal(summary.realFixtureCount, 2);
assert.equal(summary.allFixturesReal, true);
assert.deepEqual(summary.fixtureSources, [