forked from longtao.wu/droboticflow
🧾 Make operator doc source explicit
This commit is contained in:
parent
f30294b167
commit
934527d746
@ -73,7 +73,7 @@ Drobodata is a workflow orchestration platform for embodied AI and vision pipeli
|
||||
- `npm run acceptance:xplatform`: X-platform operator-chain acceptance against an explicit real dataset; requires `XPLATFORM_SOURCE_PATH`, explicit `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, quality/slicing thresholds, and a storage connection selector when the source or upload path is `bos://...`
|
||||
- `python3 scripts/wjzh_operator_regression.py --preflight-only`: preflight the production-style WJZH real-data operator regression against a live API; requires explicit `SOURCE_PATH`, `UPLOAD_ROOT_PREFIX`, storage connection selector, target video filenames, route/filter values, real Robotics dataset paths, incremental seed metadata, quality thresholds, video timing, and `SAMPLING_RULES_JSON`
|
||||
|
||||
Builtin operator metadata is declared once in `internal/api/builtin_operators_catalog.json`. The API embeds and validates that catalog before seeding `/api/operators`, while `scripts/generate_operator_docs.py` reads the same file for runtime handler hints. Runtime deployment fields such as the active operator-runner image are injected at startup or reconciliation time.
|
||||
Builtin operator metadata is declared once in `internal/api/builtin_operators_catalog.json`. The API embeds and validates that catalog before seeding `/api/operators`, while `scripts/generate_operator_docs.py` defaults to that same local catalog and only contacts a live API when `--api-base-url` is provided. Runtime deployment fields such as the active operator-runner image are injected at startup or reconciliation time.
|
||||
|
||||
## Runtime Configuration
|
||||
|
||||
|
||||
@ -67,6 +67,8 @@ python3 scripts/generate_operator_docs.py --input-json /path/to/generated-operat
|
||||
- 纯数组:`[ {...}, {...} ]`
|
||||
- 单个 operator 对象:`{ ... }`
|
||||
|
||||
不传参数时,脚本只读取仓库内的 `internal/api/builtin_operators_catalog.json`;需要连接真实 API 时必须显式传 `--api-base-url http://<api-host>:30080`。
|
||||
|
||||
对于单个 `custom` operator,生成结果会出现在:
|
||||
|
||||
- `docs/custom-operators/instances/`
|
||||
|
||||
@ -38,6 +38,12 @@
|
||||
|
||||
生成命令示例:
|
||||
|
||||
```bash
|
||||
python3 scripts/generate_operator_docs.py
|
||||
```
|
||||
|
||||
默认输入是仓库内的 `internal/api/builtin_operators_catalog.json`,不会隐式连接本机 API。如果要从某个真实环境导出自定义算子实例,显式传入目标 API:
|
||||
|
||||
```bash
|
||||
python3 scripts/generate_operator_docs.py --api-base-url http://<api-host>:30080
|
||||
```
|
||||
|
||||
@ -245,7 +245,7 @@ export DROBOTICFLOW_SMOKE_PASSWORD=<admin-password>
|
||||
|
||||
`scripts/reconcile-system-data.sh` is only the environment wrapper. The Python implementation reads the current builtin operator catalog from `/api/operators` and updates those live records to the requested `OPERATOR_RUNNER_IMAGE`; it does not carry a second static builtin-operator list. The cleanup step skips builtin operators even when their image tag is `:latest`, so old custom/demo operators can be removed without tombstoning platform-provided builtins.
|
||||
|
||||
The source catalog for platform-provided builtin operators is `internal/api/builtin_operators_catalog.json`. The API embeds this file, validates required fields and duplicate IDs/names at startup, and injects only runtime deployment fields such as the current operator-runner image and timestamps. `scripts/generate_operator_docs.py` reads the same file for runtime handler hints, so adding or changing a builtin operator should update the catalog first instead of editing separate Go and Python maps.
|
||||
The source catalog for platform-provided builtin operators is `internal/api/builtin_operators_catalog.json`. The API embeds this file, validates required fields and duplicate IDs/names at startup, and injects only runtime deployment fields such as the current operator-runner image and timestamps. `scripts/generate_operator_docs.py` defaults to the same local file and contacts a live API only when `--api-base-url` is provided, so adding or changing a builtin operator should update the catalog first instead of editing separate Go and Python maps.
|
||||
|
||||
## Kubernetes Verification
|
||||
|
||||
|
||||
@ -116,7 +116,7 @@ K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http:/
|
||||
|
||||
The reconcile wrapper delegates to `scripts/reconcile_system_data.py`. It removes legacy demo records, updates cluster settings through the real API, and derives builtin operators from the live `/api/operators` catalog before refreshing their runner image; it no longer maintains a separate hardcoded builtin-operator catalog.
|
||||
|
||||
Builtin operator metadata is declared in `internal/api/builtin_operators_catalog.json` and embedded by the API. Startup validation rejects malformed or duplicate catalog entries before they can drift into the database, and documentation generation reads the same catalog for runtime handler hints.
|
||||
Builtin operator metadata is declared in `internal/api/builtin_operators_catalog.json` and embedded by the API. Startup validation rejects malformed or duplicate catalog entries before they can drift into the database, and documentation generation defaults to the same local catalog; live API export requires an explicit `--api-base-url`.
|
||||
|
||||
Run GPU smoke on a real GPU cluster:
|
||||
|
||||
|
||||
@ -10,7 +10,6 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
DEFAULT_API_BASE_URL = "http://127.0.0.1:3001"
|
||||
DEFAULT_BUILTIN_OUTPUT_DIR = "docs/operators"
|
||||
DEFAULT_CUSTOM_OUTPUT_DIR = "docs/custom-operators/instances"
|
||||
DEFAULT_BUILTIN_CATALOG_PATH = "internal/api/builtin_operators_catalog.json"
|
||||
@ -55,7 +54,7 @@ TRANSMISSION_HINTS = {
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Generate detailed markdown manuals for builtin and custom operators")
|
||||
parser.add_argument("--api-base-url", default=DEFAULT_API_BASE_URL)
|
||||
parser.add_argument("--api-base-url", default="", help="Fetch operator payload from a specific live API /api/operators endpoint")
|
||||
parser.add_argument("--input-json", default="", help="Read operator payload from a local JSON file instead of calling /api/operators")
|
||||
parser.add_argument("--builtin-output-dir", default=DEFAULT_BUILTIN_OUTPUT_DIR)
|
||||
parser.add_argument("--custom-output-dir", default=DEFAULT_CUSTOM_OUTPUT_DIR)
|
||||
@ -67,12 +66,25 @@ def request_json(url: str) -> Any:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
|
||||
|
||||
def load_operator_items(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
def load_operator_payload(args: argparse.Namespace) -> tuple[Any, str]:
|
||||
if args.input_json and args.api_base_url:
|
||||
raise SystemExit("set only one of --input-json or --api-base-url")
|
||||
if args.input_json:
|
||||
payload = json.loads(Path(args.input_json).read_text(encoding="utf-8"))
|
||||
else:
|
||||
input_path = Path(args.input_json)
|
||||
return json.loads(input_path.read_text(encoding="utf-8")), str(input_path)
|
||||
if args.api_base_url:
|
||||
api_base_url = args.api_base_url.rstrip("/")
|
||||
payload = request_json(f"{api_base_url}/api/operators")
|
||||
return request_json(f"{api_base_url}/api/operators"), f"{api_base_url}/api/operators"
|
||||
catalog_path = Path(DEFAULT_BUILTIN_CATALOG_PATH)
|
||||
return json.loads(catalog_path.read_text(encoding="utf-8")), str(catalog_path)
|
||||
|
||||
|
||||
def load_operator_items(args: argparse.Namespace) -> list[dict[str, Any]]:
|
||||
payload, _ = load_operator_payload(args)
|
||||
return operator_items_from_payload(payload)
|
||||
|
||||
|
||||
def operator_items_from_payload(payload: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(payload, dict) and isinstance(payload.get("items"), list):
|
||||
items = payload.get("items", [])
|
||||
elif isinstance(payload, list):
|
||||
@ -584,7 +596,8 @@ def write_custom_docs(items: list[dict[str, Any]], output_dir: Path) -> int:
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
items = load_operator_items(args)
|
||||
payload, source = load_operator_payload(args)
|
||||
items = operator_items_from_payload(payload)
|
||||
builtin_items = choose_builtin_records(items)
|
||||
custom_items = choose_custom_records(items)
|
||||
builtin_count = write_builtin_docs(builtin_items, Path(args.builtin_output_dir))
|
||||
@ -597,6 +610,7 @@ def main() -> int:
|
||||
"customCount": custom_count,
|
||||
"builtinOutputDir": args.builtin_output_dir,
|
||||
"customOutputDir": args.custom_output_dir,
|
||||
"source": source,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
@ -2,6 +2,7 @@ import json
|
||||
import pathlib
|
||||
import sys
|
||||
import unittest
|
||||
from argparse import Namespace
|
||||
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
@ -24,6 +25,21 @@ class OperatorDocsGenerationTest(unittest.TestCase):
|
||||
)
|
||||
self.assertEqual(sorted(hints), sorted(builtin_ids))
|
||||
|
||||
def test_default_operator_payload_comes_from_local_catalog(self):
|
||||
args = Namespace(input_json="", api_base_url="")
|
||||
payload, source = generate_operator_docs.load_operator_payload(args)
|
||||
items = generate_operator_docs.operator_items_from_payload(payload)
|
||||
|
||||
self.assertEqual(source, "internal/api/builtin_operators_catalog.json")
|
||||
self.assertGreater(len(items), 20)
|
||||
self.assertTrue(any(item.get("name") == "视频质检" for item in items))
|
||||
|
||||
def test_operator_payload_source_must_be_explicitly_unique(self):
|
||||
args = Namespace(input_json="operators.json", api_base_url="http://api.example.test")
|
||||
|
||||
with self.assertRaisesRegex(SystemExit, "set only one"):
|
||||
generate_operator_docs.load_operator_payload(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user