forked from longtao.wu/droboticflow
✅ Count external POC verification systems
This commit is contained in:
parent
cdfde23562
commit
7cfb0da433
@ -267,7 +267,7 @@ The legacy explicit variables remain supported and take precedence over the mani
|
||||
|
||||
By default `POC_ACCEPTANCE_REQUIRED_EXTERNALS=bos,dcloud,mongo`, so missing `BOS_ENDPOINT`, BOS bucket/AK/SK, `MONGO_URI`, or accepted D-Cloud FAT responses is reported as an acceptance failure instead of being treated as a pass. The POC preset supplies the current Beijing BOS endpoint; override it with `POC_ACCEPTANCE_BOS_ENDPOINT` only when validating a different bucket region. When `dcloud` is required, the wrapper checks that `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` exists and contains `DCLOUD_FAT_BASE_URL`, a usable auth path, `DCLOUD_ORG_SYMBOL`, and `DCLOUD_SOURCEAPP` before creating the Kubernetes Job; missing or incomplete D-Cloud context fails fast without printing secret values. For dependency isolation runs, explicitly narrow that list, for example `POC_ACCEPTANCE_REQUIRED_EXTERNALS=mongo npm run acceptance:poc-operators`; such a run is useful diagnostics, but it is not full external acceptance evidence. Use `POC_ACCEPTANCE_BOS_BUCKET` when the cluster defaults do not expose the bucket name. D-Cloud FAT context is read from `POC_ACCEPTANCE_DCLOUD_SECRET_NAME` (default `droboticflow-dcloud-secret`): provide `DCLOUD_FAT_BASE_URL`, `DCLOUD_FAT_AUTH_TOKEN` or `DCLOUD_API_TOKEN`, `DCLOUD_FAT_COOKIE` or `DCLOUD_COOKIE`, or `DCLOUD_FAT_USERNAME`/`DCLOUD_FAT_PASSWORD` for runtime `/api/signin`; also provide `DCLOUD_ORG_SYMBOL` and `DCLOUD_SOURCEAPP`. `DCLOUD_D_USER_ID` is optional on JWT gateway paths because the gateway maps the token claim to `d-user-id`; keep it only for direct-service diagnostics. Do not pass tokens, cookies, passwords, or secret values through dry-run output.
|
||||
|
||||
The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators.
|
||||
The result JSON exposes `acceptanceScope`, `fullExternalAcceptance`, `fixtureSources`, `generatedFixtureCount`, `realFixtureCount`, `allFixturesReal`, `operatorPassedCount`, `coveredOperatorCount`, `allOperatorCoverageComplete`, `blockingOperatorCoverageComplete`, `externalVerificationPassed`, `externalVerificationMissing`, `externalVerificationPassedCount`, `externalVerificationRequiredCount`, `statusCounts`, and skipped-count fields. Treat `fullExternalAcceptance=true` plus `status=passed` as the strict full-acceptance signal only when the scope is `full`, required externals include `bos,dcloud,mongo`, every required external system is verified, every operator is covered and passed, no failed/required-skipped/missing-coverage entries exist, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. External verification is counted by system: BOS requires both real upload and BOS incremental scan records, Mongo requires a write followed by an independent count/drop verification, and D-Cloud requires both FAT API operators to pass. The Monitoring summary card applies the same strict display rules; a `dependency-isolation` result can prove the configured dependencies but not the omitted external operators.
|
||||
|
||||
When the Job emits a `DROBOTICFLOW_POC_OPERATOR_ACCEPTANCE=<json>` marker, the wrapper extracts it from the Job logs even if the Job exits failed. If `POC_ACCEPTANCE_IMPORT_API_BASE_URL`, `API_BASE_URL`, or `BASE_URL` is set, it posts that result to `/api/acceptance/poc-operator-result` and records it as a normal platform run under the `POC 算子验收` project. In session-auth POC deployments, the acceptance-import endpoint accepts either an admin/pipeline/operator session through `DROBOTICFLOW_ACCEPTANCE_EMAIL` and `DROBOTICFLOW_ACCEPTANCE_PASSWORD`, or the shared `DROBOTICFLOW_API_TOKEN` for this fixed POC-result import route only; regular business writes still require a user session. The imported run contains a `poc-operator-full-acceptance-result.json` artifact, so the Monitoring page can show the same strict-vs-dependency-isolation summary that appears in the raw Job log. Set `POC_ACCEPTANCE_IMPORT_RESULT=false` for one-off diagnostics that should not create or update platform acceptance history.
|
||||
|
||||
@ -275,7 +275,7 @@ If strict acceptance cannot even create the Kubernetes Job because required read
|
||||
|
||||
Generated fixtures are disabled by default, so the wrapper refuses to create the Job when no real fixture contract is configured. Provide either `POC_ACCEPTANCE_FIXTURE_MANIFEST_PATH` or all explicit real fixture inputs, including `POC_ACCEPTANCE_XPLATFORM_SOURCE_PATH`, `POC_ACCEPTANCE_CSV_INPUT_PATH`, `POC_ACCEPTANCE_JSON_INPUT_PATH`, `POC_ACCEPTANCE_LEROBOT_SOURCE_PATH`, `POC_ACCEPTANCE_ROSBAG_SOURCE_PATH`, `POC_ACCEPTANCE_MODEL_JSON_PATH`, and `POC_ACCEPTANCE_SHELL_COMMAND`. Manifest and explicit fixture paths must point under `WORKSPACE_MOUNT_PATH`; before creating the acceptance Job, the wrapper verifies those files or directories through the deployed API Pod's shared workspace PVC mount. Missing, unreadable, empty file, or off-workspace fixture paths are imported with `fixture-readiness` coverage for the full builtin operator set, so the platform history distinguishes missing external credentials from missing real acceptance data without launching a Job that can only fail later.
|
||||
|
||||
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
|
||||
The import endpoint validates full-acceptance consistency before it creates or updates a platform run. A payload with `fullExternalAcceptance=true` must also have `status=passed`, `acceptanceScope=full`, `requiredExternals` containing `bos,dcloud,mongo`, `externalVerificationPassed` containing all three systems, empty `externalVerificationMissing`, `externalVerificationPassedCount>=3`, complete operator coverage, zero failed or required-skipped operators, empty missing-coverage lists, `allFixturesReal=true`, `generatedFixtureCount=0`, and `realFixtureCount>0`. Dependency-isolation evidence can still be imported, but it cannot be mislabeled as full external acceptance.
|
||||
|
||||
Configure or refresh the POC real fixture manifest from local fixture files and directories:
|
||||
|
||||
|
||||
@ -76,6 +76,15 @@ func stringSliceField(payload map[string]any, key string) []string {
|
||||
return items
|
||||
}
|
||||
|
||||
func stringSetContains(items []string, value string) bool {
|
||||
for _, item := range items {
|
||||
if item == value {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func boolFieldPresent(payload map[string]any, key string) (bool, bool) {
|
||||
value, ok := payload[key].(bool)
|
||||
return value, ok
|
||||
@ -136,6 +145,30 @@ func validatePocAcceptanceFullConsistency(payload map[string]any) []validationIs
|
||||
if stringField(payload, "acceptanceScope") != "full" {
|
||||
issues = append(issues, validationIssue{Field: "result.acceptanceScope", Message: "acceptanceScope must be full when fullExternalAcceptance is true"})
|
||||
}
|
||||
requiredExternals := stringSliceField(payload, "requiredExternals")
|
||||
for _, external := range []string{"bos", "dcloud", "mongo"} {
|
||||
if !stringSetContains(requiredExternals, external) {
|
||||
issues = append(issues, validationIssue{Field: "result.requiredExternals", Message: "requiredExternals must include bos, dcloud, and mongo when fullExternalAcceptance is true"})
|
||||
break
|
||||
}
|
||||
}
|
||||
externalVerificationPassed := stringSliceField(payload, "externalVerificationPassed")
|
||||
for _, external := range []string{"bos", "dcloud", "mongo"} {
|
||||
if !stringSetContains(externalVerificationPassed, external) {
|
||||
issues = append(issues, validationIssue{Field: "result.externalVerificationPassed", Message: "externalVerificationPassed must include bos, dcloud, and mongo when fullExternalAcceptance is true"})
|
||||
break
|
||||
}
|
||||
}
|
||||
if externalVerificationMissing := stringSliceField(payload, "externalVerificationMissing"); len(externalVerificationMissing) > 0 {
|
||||
issues = append(issues, validationIssue{Field: "result.externalVerificationMissing", Message: "externalVerificationMissing must be empty when fullExternalAcceptance is true"})
|
||||
}
|
||||
externalVerificationPassedCount, ok := numericFieldPresent(payload, "externalVerificationPassedCount")
|
||||
if !ok || externalVerificationPassedCount < 3 {
|
||||
issues = append(issues, validationIssue{Field: "result.externalVerificationPassedCount", Message: "externalVerificationPassedCount must be at least 3 when fullExternalAcceptance is true"})
|
||||
}
|
||||
if externalVerificationRequiredCount, ok := numericFieldPresent(payload, "externalVerificationRequiredCount"); !ok || externalVerificationRequiredCount < 3 {
|
||||
issues = append(issues, validationIssue{Field: "result.externalVerificationRequiredCount", Message: "externalVerificationRequiredCount must be at least 3 when fullExternalAcceptance is true"})
|
||||
}
|
||||
operatorCount, ok := numericFieldPresent(payload, "operatorCount")
|
||||
if !ok || operatorCount <= 0 {
|
||||
issues = append(issues, validationIssue{Field: "result.operatorCount", Message: "operatorCount must be a positive integer"})
|
||||
|
||||
@ -2567,25 +2567,28 @@ func TestPocAcceptanceImportCreatesPlatformRunArtifact(t *testing.T) {
|
||||
handler := NewHandler(dataStore, nil)
|
||||
payload := map[string]any{
|
||||
"result": map[string]any{
|
||||
"runId": "poc-operator-acceptance-20260623010101",
|
||||
"status": "passed",
|
||||
"workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101",
|
||||
"acceptanceScope": "dependency-isolation",
|
||||
"fullExternalAcceptance": false,
|
||||
"operatorCount": 28,
|
||||
"operatorPassedCount": 26,
|
||||
"coveredOperatorCount": 26,
|
||||
"allOperatorCoverageComplete": false,
|
||||
"blockingOperatorCoverageComplete": true,
|
||||
"missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"},
|
||||
"blockingMissingCoverage": []string{},
|
||||
"requiredExternals": []string{"bos", "mongo"},
|
||||
"statusCounts": map[string]any{"passed": 28, "skipped": 2},
|
||||
"failedCount": 0,
|
||||
"skippedCount": 2,
|
||||
"optionalSkippedCount": 2,
|
||||
"requiredSkippedCount": 0,
|
||||
"externalVerificationPassedCount": 1,
|
||||
"runId": "poc-operator-acceptance-20260623010101",
|
||||
"status": "passed",
|
||||
"workspace": "/workspace/acceptance/poc-operator-acceptance-20260623010101",
|
||||
"acceptanceScope": "dependency-isolation",
|
||||
"fullExternalAcceptance": false,
|
||||
"operatorCount": 28,
|
||||
"operatorPassedCount": 26,
|
||||
"coveredOperatorCount": 26,
|
||||
"allOperatorCoverageComplete": false,
|
||||
"blockingOperatorCoverageComplete": true,
|
||||
"missingCoverage": []string{"dcloud_data_generation", "dcloud_embodied_annotation"},
|
||||
"blockingMissingCoverage": []string{},
|
||||
"requiredExternals": []string{"bos", "mongo"},
|
||||
"statusCounts": map[string]any{"passed": 28, "skipped": 2},
|
||||
"failedCount": 0,
|
||||
"skippedCount": 2,
|
||||
"optionalSkippedCount": 2,
|
||||
"requiredSkippedCount": 0,
|
||||
"externalVerificationPassed": []string{"bos", "mongo"},
|
||||
"externalVerificationMissing": []string{},
|
||||
"externalVerificationPassedCount": 2,
|
||||
"externalVerificationRequiredCount": 2,
|
||||
},
|
||||
"source": map[string]any{
|
||||
"environment": "poc",
|
||||
@ -2776,6 +2779,10 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) {
|
||||
body := recorder.Body.String()
|
||||
for _, field := range []string{
|
||||
"result.acceptanceScope",
|
||||
"result.requiredExternals",
|
||||
"result.externalVerificationPassed",
|
||||
"result.externalVerificationPassedCount",
|
||||
"result.externalVerificationRequiredCount",
|
||||
"result.operatorPassedCount",
|
||||
"result.coveredOperatorCount",
|
||||
"result.allOperatorCoverageComplete",
|
||||
@ -2787,27 +2794,89 @@ func TestPocAcceptanceImportRejectsInconsistentFullAcceptance(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPocAcceptanceImportRejectsIncompleteExternalVerification(t *testing.T) {
|
||||
dataStore := store.NewMemoryStore()
|
||||
handler := NewHandler(dataStore, nil)
|
||||
payload := map[string]any{
|
||||
"result": map[string]any{
|
||||
"runId": "poc-operator-acceptance-missing-external-proof",
|
||||
"status": "passed",
|
||||
"acceptanceScope": "full",
|
||||
"fullExternalAcceptance": true,
|
||||
"operatorCount": 28,
|
||||
"operatorPassedCount": 28,
|
||||
"coveredOperatorCount": 28,
|
||||
"allOperatorCoverageComplete": true,
|
||||
"blockingOperatorCoverageComplete": true,
|
||||
"missingCoverage": []string{},
|
||||
"blockingMissingCoverage": []string{},
|
||||
"requiredExternals": []string{"bos", "dcloud", "mongo"},
|
||||
"externalVerificationPassed": []string{"bos", "mongo"},
|
||||
"externalVerificationMissing": []string{"dcloud"},
|
||||
"externalVerificationPassedCount": 2,
|
||||
"externalVerificationRequiredCount": 3,
|
||||
"failedCount": 0,
|
||||
"requiredSkippedCount": 0,
|
||||
"generatedFixtureCount": 0,
|
||||
"realFixtureCount": 6,
|
||||
"allFixturesReal": true,
|
||||
"fixtureSources": map[string]any{
|
||||
"allowGeneratedFixtures": false,
|
||||
"items": map[string]any{
|
||||
"xplatform": map[string]any{"path": "/workspace/fixtures/xplatform", "generated": false},
|
||||
"csv": map[string]any{"path": "/workspace/fixtures/records.csv", "generated": false},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
rawPayload, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal payload: %v", err)
|
||||
}
|
||||
|
||||
recorder := httptest.NewRecorder()
|
||||
handler.ServeHTTP(recorder, httptest.NewRequest(http.MethodPost, "/api/acceptance/poc-operator-result", bytes.NewReader(rawPayload)))
|
||||
if recorder.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status: %d body=%s", recorder.Code, recorder.Body.String())
|
||||
}
|
||||
body := recorder.Body.String()
|
||||
for _, field := range []string{
|
||||
"result.externalVerificationPassed",
|
||||
"result.externalVerificationMissing",
|
||||
"result.externalVerificationPassedCount",
|
||||
} {
|
||||
if !strings.Contains(body, field) {
|
||||
t.Fatalf("expected validation field %s in %s", field, body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPocAcceptanceImportRejectsGeneratedFixtureFullAcceptance(t *testing.T) {
|
||||
dataStore := store.NewMemoryStore()
|
||||
handler := NewHandler(dataStore, nil)
|
||||
payload := map[string]any{
|
||||
"result": map[string]any{
|
||||
"runId": "poc-operator-acceptance-generated-fixtures",
|
||||
"status": "passed",
|
||||
"acceptanceScope": "full",
|
||||
"fullExternalAcceptance": true,
|
||||
"operatorCount": 28,
|
||||
"operatorPassedCount": 28,
|
||||
"coveredOperatorCount": 28,
|
||||
"allOperatorCoverageComplete": true,
|
||||
"blockingOperatorCoverageComplete": true,
|
||||
"missingCoverage": []string{},
|
||||
"blockingMissingCoverage": []string{},
|
||||
"failedCount": 0,
|
||||
"requiredSkippedCount": 0,
|
||||
"generatedFixtureCount": 1,
|
||||
"realFixtureCount": 5,
|
||||
"allFixturesReal": false,
|
||||
"runId": "poc-operator-acceptance-generated-fixtures",
|
||||
"status": "passed",
|
||||
"acceptanceScope": "full",
|
||||
"fullExternalAcceptance": true,
|
||||
"operatorCount": 28,
|
||||
"operatorPassedCount": 28,
|
||||
"coveredOperatorCount": 28,
|
||||
"allOperatorCoverageComplete": true,
|
||||
"blockingOperatorCoverageComplete": true,
|
||||
"missingCoverage": []string{},
|
||||
"blockingMissingCoverage": []string{},
|
||||
"requiredExternals": []string{"bos", "dcloud", "mongo"},
|
||||
"externalVerificationPassed": []string{"bos", "dcloud", "mongo"},
|
||||
"externalVerificationMissing": []string{},
|
||||
"externalVerificationPassedCount": 3,
|
||||
"externalVerificationRequiredCount": 3,
|
||||
"failedCount": 0,
|
||||
"requiredSkippedCount": 0,
|
||||
"generatedFixtureCount": 1,
|
||||
"realFixtureCount": 5,
|
||||
"allFixturesReal": false,
|
||||
"fixtureSources": map[string]any{
|
||||
"allowGeneratedFixtures": true,
|
||||
"items": map[string]any{
|
||||
|
||||
@ -520,6 +520,7 @@ for stage, reason in zip(stages, reasons, strict=True):
|
||||
unique_stages = list(dict.fromkeys(stages))
|
||||
stage = unique_stages[0] if len(unique_stages) == 1 else "preflight-readiness"
|
||||
missing_coverage = sorted(missing_coverage_set)
|
||||
required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required)))
|
||||
payload = {
|
||||
"runId": run_id,
|
||||
"workspace": f"{workspace_mount}/acceptance/{run_id}",
|
||||
@ -538,7 +539,10 @@ payload = {
|
||||
"requiredExternals": required,
|
||||
"statusCounts": {"failed": len(records)},
|
||||
"passedCount": 0,
|
||||
"externalVerificationPassed": [],
|
||||
"externalVerificationMissing": required_external_systems,
|
||||
"externalVerificationPassedCount": 0,
|
||||
"externalVerificationRequiredCount": len(required_external_systems),
|
||||
"failedCount": len(records),
|
||||
"skippedCount": 0,
|
||||
"requiredSkippedCount": 0,
|
||||
|
||||
@ -235,6 +235,31 @@ def acceptance_scope(required: set[str]) -> str:
|
||||
return "dependency-isolation"
|
||||
|
||||
|
||||
def has_passed_record(records: list[dict[str, Any]], operator: str) -> bool:
|
||||
return any(item.get("operator") == operator and item.get("status") == "passed" for item in records)
|
||||
|
||||
|
||||
def external_verification_summary(records: list[dict[str, Any]], required: set[str]) -> dict[str, Any]:
|
||||
required_external_systems = sorted(DEFAULT_REQUIRED_EXTERNALS.intersection(required))
|
||||
passed: list[str] = []
|
||||
if "bos" in required and has_passed_record(records, "upload_bos_external") and has_passed_record(records, "bos_incremental_scan"):
|
||||
passed.append("bos")
|
||||
if "mongo" in required and has_passed_record(records, "mongodb_write_external_verification"):
|
||||
passed.append("mongo")
|
||||
if (
|
||||
"dcloud" in required
|
||||
and has_passed_record(records, "dcloud_embodied_annotation")
|
||||
and has_passed_record(records, "dcloud_data_generation")
|
||||
):
|
||||
passed.append("dcloud")
|
||||
return {
|
||||
"externalVerificationPassed": sorted(passed),
|
||||
"externalVerificationMissing": sorted(set(required_external_systems) - set(passed)),
|
||||
"externalVerificationPassedCount": len(passed),
|
||||
"externalVerificationRequiredCount": len(required_external_systems),
|
||||
}
|
||||
|
||||
|
||||
def env_bool(name: str, default: bool = False) -> bool:
|
||||
raw = os.getenv(name)
|
||||
if raw is None or not raw.strip():
|
||||
@ -1520,6 +1545,7 @@ def run_acceptance() -> dict[str, Any]:
|
||||
failures.append(f"operator failed: {item.get('operator')} ({item.get('error')})")
|
||||
scope = acceptance_scope(required)
|
||||
all_operator_coverage_complete = not missing_coverage
|
||||
external_summary = external_verification_summary(records, required)
|
||||
|
||||
summary = {
|
||||
"runId": run_id,
|
||||
@ -1537,9 +1563,7 @@ def run_acceptance() -> dict[str, Any]:
|
||||
"requiredExternals": sorted(required),
|
||||
"statusCounts": status_counts,
|
||||
"passedCount": len(passed),
|
||||
"externalVerificationPassedCount": len(
|
||||
[item for item in passed if str(item.get("operator", "")).endswith("_external_verification")]
|
||||
),
|
||||
**external_summary,
|
||||
"failedCount": len(failed),
|
||||
"skippedCount": len(required_skips) + len(optional_skips),
|
||||
"requiredSkippedCount": len(required_skips),
|
||||
|
||||
@ -212,6 +212,7 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]:
|
||||
unique_stages = list(dict.fromkeys(stages))
|
||||
stage = unique_stages[0] if len(unique_stages) == 1 else "strict-preflight-readiness"
|
||||
missing_coverage = sorted(missing_coverage_set)
|
||||
required_external_systems = sorted({"bos", "dcloud", "mongo"}.intersection(set(required)))
|
||||
return {
|
||||
"runId": run_id,
|
||||
"workspace": f"{workspace_mount}/acceptance/{run_id}",
|
||||
@ -230,7 +231,10 @@ def build_result(args: argparse.Namespace) -> dict[str, Any]:
|
||||
"requiredExternals": required,
|
||||
"statusCounts": {"failed": len(records)},
|
||||
"passedCount": 0,
|
||||
"externalVerificationPassed": [],
|
||||
"externalVerificationMissing": required_external_systems,
|
||||
"externalVerificationPassedCount": 0,
|
||||
"externalVerificationRequiredCount": len(required_external_systems),
|
||||
"failedCount": len(records),
|
||||
"skippedCount": 0,
|
||||
"requiredSkippedCount": 0,
|
||||
|
||||
@ -133,6 +133,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
|
||||
.join(" / ");
|
||||
const coverage = `${summary.operatorPassedCount}/${summary.operatorCount}`;
|
||||
const covered = `${summary.coveredOperatorCount}/${summary.operatorCount}`;
|
||||
const externalVerification = `${summary.externalVerificationPassedCount}/${summary.externalVerificationRequiredCount || "--"}`;
|
||||
|
||||
return (
|
||||
<div className="rounded-lg border border-[#2a2b2f] bg-[#0f1012] p-4">
|
||||
@ -157,7 +158,7 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
|
||||
<Metric label="完整外部验收" value={summary.fullExternalAcceptance ? "true" : "false"} />
|
||||
<Metric label="通过算子" value={coverage} />
|
||||
<Metric label="覆盖算子" value={covered} />
|
||||
<Metric label="外部校验" value={summary.externalVerificationPassedCount} />
|
||||
<Metric label="外部校验" value={externalVerification} />
|
||||
<Metric label="失败记录" value={summary.failedCount} />
|
||||
<Metric label="跳过记录" value={`${summary.skippedCount} (${summary.optionalSkippedCount} optional)`} />
|
||||
<Metric label="必需跳过" value={summary.requiredSkippedCount} />
|
||||
@ -230,6 +231,8 @@ export function PocAcceptanceSummaryCard({ item }: { item: ExecutionData | Recor
|
||||
<div className="mt-4 space-y-3">
|
||||
<ChipList label="阻塞缺口" items={summary.blockingMissingCoverage} tone="danger" />
|
||||
<ChipList label="未覆盖算子" items={summary.missingCoverage} tone="warning" />
|
||||
<ChipList label="缺失外部校验" items={summary.externalVerificationMissing} tone="danger" />
|
||||
<ChipList label="已通过外部校验" items={summary.externalVerificationPassed} tone="warning" />
|
||||
<FixtureSourceList sources={summary.fixtureSources} />
|
||||
<RecordList records={summary.records} />
|
||||
</div>
|
||||
|
||||
@ -117,6 +117,8 @@ export function summarizePocAcceptanceArtifact(artifact) {
|
||||
const optionalSkippedCount = integerOrZero(payload.optionalSkippedCount);
|
||||
const requiredSkippedCount = integerOrZero(payload.requiredSkippedCount);
|
||||
const externalVerificationPassedCount = integerOrZero(payload.externalVerificationPassedCount);
|
||||
const externalVerificationPassed = stringList(payload.externalVerificationPassed);
|
||||
const externalVerificationMissing = stringList(payload.externalVerificationMissing);
|
||||
const fullExternalAcceptance = payload.fullExternalAcceptance === true;
|
||||
const allOperatorCoverageComplete = payload.allOperatorCoverageComplete === true;
|
||||
const blockingOperatorCoverageComplete = payload.blockingOperatorCoverageComplete === true;
|
||||
@ -153,6 +155,12 @@ export function summarizePocAcceptanceArtifact(artifact) {
|
||||
const strictExternalScope =
|
||||
acceptanceScope === "full" &&
|
||||
["bos", "dcloud", "mongo"].every((external) => requiredExternals.includes(external));
|
||||
const externalVerificationRequiredCount =
|
||||
integerOrNull(payload.externalVerificationRequiredCount) ?? requiredExternals.length;
|
||||
const externalVerificationComplete =
|
||||
externalVerificationPassedCount >= Math.max(externalVerificationRequiredCount, requiredExternals.length) &&
|
||||
externalVerificationMissing.length === 0 &&
|
||||
requiredExternals.every((external) => externalVerificationPassed.includes(external));
|
||||
const strictOperatorCoverage =
|
||||
operatorCount > 0 &&
|
||||
integerOrNull(payload.operatorPassedCount) === operatorCount &&
|
||||
@ -168,6 +176,7 @@ export function summarizePocAcceptanceArtifact(artifact) {
|
||||
status === "passed" &&
|
||||
fullExternalAcceptance &&
|
||||
strictExternalScope &&
|
||||
externalVerificationComplete &&
|
||||
strictOperatorCoverage &&
|
||||
fixtureEvidenceComplete;
|
||||
const incompleteFullFixtureEvidence = status === "passed" && fullExternalAcceptance && !allFixturesReal;
|
||||
@ -204,6 +213,9 @@ export function summarizePocAcceptanceArtifact(artifact) {
|
||||
optionalSkippedCount,
|
||||
requiredSkippedCount,
|
||||
externalVerificationPassedCount,
|
||||
externalVerificationRequiredCount,
|
||||
externalVerificationPassed,
|
||||
externalVerificationMissing,
|
||||
workspace: cleanString(payload.workspace),
|
||||
workspaceFallbackReason: cleanString(payload.workspaceFallbackReason),
|
||||
preflightFailure,
|
||||
|
||||
@ -25,6 +25,53 @@ def test_acceptance_scope_requires_all_strict_externals():
|
||||
assert module.acceptance_scope(set()) == "dependency-isolation"
|
||||
|
||||
|
||||
def test_external_verification_summary_counts_required_systems():
|
||||
module = load_job_module()
|
||||
records = [
|
||||
{"operator": "upload_bos_external", "status": "passed"},
|
||||
{"operator": "bos_incremental_scan", "status": "passed"},
|
||||
{"operator": "mongodb_write_external_verification", "status": "passed"},
|
||||
{"operator": "dcloud_embodied_annotation", "status": "skipped"},
|
||||
{"operator": "dcloud_data_generation", "status": "skipped"},
|
||||
]
|
||||
|
||||
summary = module.external_verification_summary(records, {"bos", "mongo"})
|
||||
|
||||
assert summary == {
|
||||
"externalVerificationPassed": ["bos", "mongo"],
|
||||
"externalVerificationMissing": [],
|
||||
"externalVerificationPassedCount": 2,
|
||||
"externalVerificationRequiredCount": 2,
|
||||
}
|
||||
|
||||
|
||||
def test_external_verification_summary_requires_all_dcloud_operators():
|
||||
module = load_job_module()
|
||||
records = [
|
||||
{"operator": "upload_bos_external", "status": "passed"},
|
||||
{"operator": "bos_incremental_scan", "status": "passed"},
|
||||
{"operator": "mongodb_write_external_verification", "status": "passed"},
|
||||
{"operator": "dcloud_embodied_annotation", "status": "passed"},
|
||||
{"operator": "dcloud_data_generation", "status": "failed"},
|
||||
]
|
||||
|
||||
summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"})
|
||||
|
||||
assert summary == {
|
||||
"externalVerificationPassed": ["bos", "mongo"],
|
||||
"externalVerificationMissing": ["dcloud"],
|
||||
"externalVerificationPassedCount": 2,
|
||||
"externalVerificationRequiredCount": 3,
|
||||
}
|
||||
|
||||
records[-1]["status"] = "passed"
|
||||
summary = module.external_verification_summary(records, {"bos", "mongo", "dcloud"})
|
||||
assert summary["externalVerificationPassed"] == ["bos", "dcloud", "mongo"]
|
||||
assert summary["externalVerificationMissing"] == []
|
||||
assert summary["externalVerificationPassedCount"] == 3
|
||||
assert summary["externalVerificationRequiredCount"] == 3
|
||||
|
||||
|
||||
def test_dcloud_acceptance_probe_is_loaded_from_operator_metadata():
|
||||
module = load_job_module()
|
||||
|
||||
|
||||
@ -27,6 +27,9 @@ const fullPayload = {
|
||||
optionalSkippedCount: 0,
|
||||
requiredSkippedCount: 0,
|
||||
externalVerificationPassedCount: 3,
|
||||
externalVerificationRequiredCount: 3,
|
||||
externalVerificationPassed: ["bos", "dcloud", "mongo"],
|
||||
externalVerificationMissing: [],
|
||||
generatedFixtureCount: 0,
|
||||
realFixtureCount: 2,
|
||||
allFixturesReal: true,
|
||||
@ -55,6 +58,10 @@ test("summarizes strict full POC operator acceptance artifacts", () => {
|
||||
assert.equal(summary.fullExternalAcceptance, true);
|
||||
assert.equal(summary.operatorPassedCount, 28);
|
||||
assert.deepEqual(summary.requiredExternals, ["bos", "dcloud", "mongo"]);
|
||||
assert.equal(summary.externalVerificationPassedCount, 3);
|
||||
assert.equal(summary.externalVerificationRequiredCount, 3);
|
||||
assert.deepEqual(summary.externalVerificationPassed, ["bos", "dcloud", "mongo"]);
|
||||
assert.deepEqual(summary.externalVerificationMissing, []);
|
||||
assert.equal(summary.generatedFixtureCount, 0);
|
||||
assert.equal(summary.realFixtureCount, 2);
|
||||
assert.equal(summary.allFixturesReal, true);
|
||||
@ -123,6 +130,14 @@ test("requires full scope and complete operator coverage for strict pass", () =>
|
||||
name: "required-skipped",
|
||||
patch: { requiredSkippedCount: 1 },
|
||||
},
|
||||
{
|
||||
name: "missing-external-verification",
|
||||
patch: {
|
||||
externalVerificationPassedCount: 2,
|
||||
externalVerificationPassed: ["bos", "mongo"],
|
||||
externalVerificationMissing: ["dcloud"],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
for (const item of cases) {
|
||||
@ -157,6 +172,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => {
|
||||
requiredExternals: ["bos", "mongo"],
|
||||
skippedCount: 2,
|
||||
optionalSkippedCount: 2,
|
||||
externalVerificationPassedCount: 2,
|
||||
externalVerificationRequiredCount: 2,
|
||||
externalVerificationPassed: ["bos", "mongo"],
|
||||
externalVerificationMissing: [],
|
||||
fixtureSources: {
|
||||
allowGeneratedFixtures: false,
|
||||
fixtureManifestPath: "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json",
|
||||
@ -176,6 +195,10 @@ test("marks dependency-isolation acceptance as limited evidence", () => {
|
||||
assert.deepEqual(summary.missingCoverage, ["dcloud_data_generation", "dcloud_embodied_annotation"]);
|
||||
assert.equal(summary.fixtureManifestPath, "/workspace/fixtures/poc-operator-acceptance/poc-operator-acceptance.json");
|
||||
assert.equal(summary.generatedFixtureCount, 0);
|
||||
assert.equal(summary.externalVerificationPassedCount, 2);
|
||||
assert.equal(summary.externalVerificationRequiredCount, 2);
|
||||
assert.deepEqual(summary.externalVerificationPassed, ["bos", "mongo"]);
|
||||
assert.deepEqual(summary.externalVerificationMissing, []);
|
||||
assert.equal(summary.realFixtureCount, 2);
|
||||
assert.equal(summary.allFixturesReal, true);
|
||||
assert.deepEqual(summary.fixtureSources, [
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user