🧠 add operator-level shared memory settings
This commit is contained in:
parent
1678e33d8f
commit
ea401191e5
@ -136,7 +136,7 @@ The settings page can update K8s access details, the worker defaults, the global
|
||||
|
||||
The API now includes a cookie-session account foundation for organization-based access. Auth endpoints cover bootstrap, login, logout, current session, and super-admin organization/user management. Existing local deployments remain compatible by default; public LoadBalancer deploys require `DROBOTICFLOW_REQUIRE_SESSION_AUTH=true` so browser business APIs keep role checks. Session-authenticated unsafe requests, including logout, use a double-submit `droboticflow_csrf` cookie plus `X-Droboticflow-CSRF-Token` header. Browser JSON, object-storage preview, and multipart object upload requests share the same credentials and CSRF header path; object-storage preview byte-count headers are accepted only when they are exact safe non-negative integers. Worker control endpoints under `/worker-api` require an admin, pipeline engineer, data operator, or legacy operator session, or an API token when session auth is not enforced. Bundled acceptance and reconcile scripts can log in with `DROBOTICFLOW_ACCEPTANCE_EMAIL` / `DROBOTICFLOW_ACCEPTANCE_PASSWORD` and will send CSRF headers on writes. Do not embed `DROBOTICFLOW_API_TOKEN` into frontend builds; it is intended for machine-to-machine calls only.
|
||||
|
||||
The Kubernetes runtime now assumes a shared workspace PVC. Worker writes per-node `input-artifacts.json` files into the shared workspace, operator-runner Jobs read those files through `INPUT_ARTIFACTS_PATH`, then persist `result.json` and `stage-manifest.json` back into the same PVC. Large artifact lists no longer need to travel through giant env vars or stdout payloads. Node CPU and memory requests/limits are applied to Kubernetes Job container resources and must be valid positive Kubernetes resource quantities; malformed explicit values fail before a Job is created instead of being omitted. GPU requests use `nvidia.com/gpu` when the node or default resource config sets a positive GPU count. GPU scheduling requires a cluster with GPU nodes and the NVIDIA device plugin/runtime installed. Optional GPU node selector, tolerations, and runtime class settings are only injected into Jobs that request a positive GPU count. Invalid GPU selector or tolerations configuration fails worker startup instead of being ignored. Operator-runner Jobs also receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, and the cleanup script can remove old completed Jobs that were created before TTL was enabled. Run `npm run acceptance:gpu` against a real GPU cluster to validate allocatable GPU resources, scheduling, runtime, and `nvidia-smi` execution; the acceptance script rejects invalid namespace, image, timeout, GPU count, selector, toleration, resource-name, Job-name, and runtime-class inputs before it touches the cluster, and renders dynamic manifest values as quoted YAML scalars. Runs that execute on NVIDIA hardware report GPU utilization, GPU memory, GPU count, and GPU model in the resource samples shown on the monitoring page.
|
||||
The Kubernetes runtime now assumes a shared workspace PVC. Worker writes per-node `input-artifacts.json` files into the shared workspace, operator-runner Jobs read those files through `INPUT_ARTIFACTS_PATH`, then persist `result.json` and `stage-manifest.json` back into the same PVC. Large artifact lists no longer need to travel through giant env vars or stdout payloads. Node CPU and memory requests/limits are applied to Kubernetes Job container resources and must be valid positive Kubernetes resource quantities; malformed explicit values fail before a Job is created instead of being omitted. Operator defaults and workflow-node overrides can also set `sharedMemorySize` for a writable `/dev/shm` memory volume; the effective order is node override, operator default, then `DROBOTICFLOW_SHARED_MEMORY_SIZE` (POC default `8Gi`). GPU requests use `nvidia.com/gpu` when the node or default resource config sets a positive GPU count. GPU scheduling requires a cluster with GPU nodes and the NVIDIA device plugin/runtime installed. Optional GPU node selector, tolerations, and runtime class settings are only injected into Jobs that request a positive GPU count. Invalid GPU selector or tolerations configuration fails worker startup instead of being ignored. Operator-runner Jobs also receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, and the cleanup script can remove old completed Jobs that were created before TTL was enabled. Run `npm run acceptance:gpu` against a real GPU cluster to validate allocatable GPU resources, scheduling, runtime, and `nvidia-smi` execution; the acceptance script rejects invalid namespace, image, timeout, GPU count, selector, toleration, resource-name, Job-name, and runtime-class inputs before it touches the cluster, and renders dynamic manifest values as quoted YAML scalars. Runs that execute on NVIDIA hardware report GPU utilization, GPU memory, GPU count, and GPU model in the resource samples shown on the monitoring page.
|
||||
|
||||
Explicit `COMMAND` / `SCRIPT` execution is disabled by default. Deployments that need the shell operator must set `OPERATOR_ALLOW_SHELL_COMMANDS=true` and should constrain commands through allowlist and denylist regular expressions. These policy variables are protected system env vars: workflow and node hyperparams cannot override the worker-provided values.
|
||||
|
||||
|
||||
@ -123,7 +123,7 @@ Project names are trimmed and required on create and update. Project status defa
|
||||
}
|
||||
```
|
||||
|
||||
Workflow names are trimmed and required on create and update. Workflow `type` accepts only `batch` or `streaming`, `kind` accepts only `standard` or `scan`, and workflow `status` updates accept only `draft`, `ready`, `running`, `completed`, or `failed`.
|
||||
Workflow names are trimmed and required on create and update. Workflow `type` accepts only `batch` or `streaming`, `kind` accepts only `standard` or `scan`, and workflow `status` updates accept only `draft`, `ready`, `running`, `completed`, or `failed`. Each operator node may optionally set `data.sharedMemorySize`; when omitted, the worker uses the linked operator resource default and then the platform `DROBOTICFLOW_SHARED_MEMORY_SIZE` fallback.
|
||||
|
||||
`POST /api/workflows/:id/run`
|
||||
|
||||
@ -226,7 +226,7 @@ The built-in catalog includes the original workflow primitives plus:
|
||||
}
|
||||
```
|
||||
|
||||
`POST /api/operators` registers custom operators only. Operator names, categories, descriptions, versions, images, image IDs, skill definitions, port IDs, and port names are trimmed before persistence. Operator `version` defaults to `v1` when omitted and must match `[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}`; use it as the business/runtime version visible to workflow authors. `imageId` is the digest-pinned image reference when available and defaults to `image` for directly registered image references. Operator `type` is immutable after creation and accepts only `builtin` or `custom`; registration requires `custom`. Port `type` accepts only `json`, `image`, `video`, `tensor`, or `file`, and non-empty `transmission` accepts only `bus`, `oss`, `shm`, or `stream`. Operator CPU/MEM resource values must be positive Kubernetes resource quantities, and GPU must be a positive integer when present.
|
||||
`POST /api/operators` registers custom operators only. Operator names, categories, descriptions, versions, images, image IDs, skill definitions, port IDs, and port names are trimmed before persistence. Operator `version` defaults to `v1` when omitted and must match `[A-Za-z0-9_][A-Za-z0-9_.-]{0,127}`; use it as the business/runtime version visible to workflow authors. `imageId` is the digest-pinned image reference when available and defaults to `image` for directly registered image references. Operator `type` is immutable after creation and accepts only `builtin` or `custom`; registration requires `custom`. Port `type` accepts only `json`, `image`, `video`, `tensor`, or `file`, and non-empty `transmission` accepts only `bus`, `oss`, `shm`, or `stream`. Operator CPU/MEM/`sharedMemorySize` resource values must be positive Kubernetes resource quantities when present, and GPU must be a positive integer when present. `sharedMemorySize` controls the writable `/dev/shm` memory volume for newly created Jobs; a workflow node value overrides the operator default, and an empty value inherits the platform `DROBOTICFLOW_SHARED_MEMORY_SIZE` default.
|
||||
|
||||
`POST /api/operators/image-archives/import`
|
||||
|
||||
|
||||
@ -84,6 +84,10 @@ worker 在执行节点前,会构造一组环境变量注入到算子进程或
|
||||
- GPU 算子镜像需要自行包含 CUDA/NVIDIA 运行依赖,集群也必须已安装 NVIDIA device plugin/runtime
|
||||
- 平台级 GPU nodeSelector、tolerations、runtimeClass 只会注入到 `GPU_LIMIT` 为正数的 Job
|
||||
- Worker 会在创建 GPU Job 前检查 Kubernetes Node allocatable `nvidia.com/gpu`;当前集群无可调度 GPU 时会直接失败并返回 `gpu request cannot be scheduled`,不会创建长时间 Pending 的 Job
|
||||
- `sharedMemorySize` / `/dev/shm`
|
||||
- 可在算子默认资源和工作流节点资源中配置,例如 `8Gi` 或 `16Gi`
|
||||
- 生效优先级为“节点覆盖值 > 算子默认值 > 平台 `DROBOTICFLOW_SHARED_MEMORY_SIZE`”,空值表示继承
|
||||
- Kubernetes Job 使用可读写的内存型 `emptyDir` 挂载到 `/dev/shm`;容量是上限,实际使用仍会占用 Pod/节点内存
|
||||
- `OPERATOR_ALLOW_SHELL_COMMANDS` / `OPERATOR_COMMAND_ALLOWLIST_REGEX` / `OPERATOR_COMMAND_DENYLIST_REGEX`
|
||||
- 平台级 `COMMAND/SCRIPT` 执行策略,由 worker 注入到 operator-runner Job
|
||||
- workflow 或节点 hyperparams 不能覆盖这些值
|
||||
|
||||
@ -494,7 +494,7 @@ Design a production-ready external container operator contract, not just a card
|
||||
Use oss transmission for video/file artifacts, stream for tensors, bus only for compact JSON.
|
||||
Leave image and imageId empty unless the user explicitly provided a registry image or digest; never invent a built image.
|
||||
Add enough typed hyperparams and resource defaults for a real K8s Job. Hyperparam value must always be a string.
|
||||
Schema: {"name":"short operator name","category":"AI|IO|CV|MATH|HUMAN|FLOW","description":"1 sentence","version":"v1.0.0","image":"optional existing registry image","imageId":"optional digest image","containerCommand":["python","/app/src/operator.py"],"containerArgs":[],"resources":{"cpuRequest":"500m","cpuLimit":"2","memoryRequest":"1Gi","memoryLimit":"4Gi","gpuType":"","gpuLimit":""},"runtimeContract":{"protocol":"droboticflow.external-operator/v1","implementationStatus":"requires-build-and-registry-push"},"inputs":[{"name":"input","type":"json|image|video|tensor|file","transmission":"bus|oss|shm|stream","description":"optional"}],"outputs":[{"name":"result","type":"json|image|video|tensor|file","transmission":"bus|oss|shm|stream","description":"optional"}],"hyperparams":[{"key":"ENV_KEY","value":"default","type":"string|number|integer|boolean|path|json","description":"what it controls"}],"skillDefinition":"detailed usage, inputs, outputs, limits, failure modes, acceptance"}`)
|
||||
Schema: {"name":"short operator name","category":"AI|IO|CV|MATH|HUMAN|FLOW","description":"1 sentence","version":"v1.0.0","image":"optional existing registry image","imageId":"optional digest image","containerCommand":["python","/app/src/operator.py"],"containerArgs":[],"resources":{"cpuRequest":"500m","cpuLimit":"2","memoryRequest":"1Gi","memoryLimit":"4Gi","gpuType":"","gpuLimit":"","sharedMemorySize":"optional Kubernetes quantity such as 8Gi"},"runtimeContract":{"protocol":"droboticflow.external-operator/v1","implementationStatus":"requires-build-and-registry-push"},"inputs":[{"name":"input","type":"json|image|video|tensor|file","transmission":"bus|oss|shm|stream","description":"optional"}],"outputs":[{"name":"result","type":"json|image|video|tensor|file","transmission":"bus|oss|shm|stream","description":"optional"}],"hyperparams":[{"key":"ENV_KEY","value":"default","type":"string|number|integer|boolean|path|json","description":"what it controls"}],"skillDefinition":"detailed usage, inputs, outputs, limits, failure modes, acceptance"}`)
|
||||
|
||||
var generated GeneratedOperator
|
||||
if err := client.generateJSON(ctx, systemPrompt, strings.TrimSpace(description), operatorMaxTokens, &generated); err != nil {
|
||||
@ -551,12 +551,13 @@ func normalizeGeneratedOperator(generated *GeneratedOperator, description string
|
||||
generated.Resources = inferGeneratedResources(*generated, description)
|
||||
} else {
|
||||
generated.Resources = &domain.OperatorResources{
|
||||
CPURequest: strings.TrimSpace(generated.Resources.CPURequest),
|
||||
CPULimit: strings.TrimSpace(generated.Resources.CPULimit),
|
||||
MemoryRequest: strings.TrimSpace(generated.Resources.MemoryRequest),
|
||||
MemoryLimit: strings.TrimSpace(generated.Resources.MemoryLimit),
|
||||
GPUType: strings.TrimSpace(generated.Resources.GPUType),
|
||||
GPULimit: strings.TrimSpace(generated.Resources.GPULimit),
|
||||
CPURequest: strings.TrimSpace(generated.Resources.CPURequest),
|
||||
CPULimit: strings.TrimSpace(generated.Resources.CPULimit),
|
||||
MemoryRequest: strings.TrimSpace(generated.Resources.MemoryRequest),
|
||||
MemoryLimit: strings.TrimSpace(generated.Resources.MemoryLimit),
|
||||
GPUType: strings.TrimSpace(generated.Resources.GPUType),
|
||||
GPULimit: strings.TrimSpace(generated.Resources.GPULimit),
|
||||
SharedMemorySize: strings.TrimSpace(generated.Resources.SharedMemorySize),
|
||||
}
|
||||
}
|
||||
if generated.RuntimeContract == nil {
|
||||
|
||||
@ -287,6 +287,18 @@ func upsertBuiltinOperator(ctx context.Context, server *server, items []domain.O
|
||||
updated.SkillDefinition = builtin.SkillDefinition
|
||||
updated.Hyperparams = builtin.Hyperparams
|
||||
updated.Resources = builtin.Resources
|
||||
// Built-in metadata is reconciled from the source catalog at startup. Keep
|
||||
// an explicit user SHM override so an API restart does not silently reset a
|
||||
// per-operator /dev/shm setting; an empty value still inherits the catalog
|
||||
// or platform default.
|
||||
if existing.Resources != nil && strings.TrimSpace(existing.Resources.SharedMemorySize) != "" {
|
||||
resources := domain.OperatorResources{}
|
||||
if updated.Resources != nil {
|
||||
resources = *updated.Resources
|
||||
}
|
||||
resources.SharedMemorySize = strings.TrimSpace(existing.Resources.SharedMemorySize)
|
||||
updated.Resources = &resources
|
||||
}
|
||||
updated.UpdatedAt = builtin.UpdatedAt
|
||||
if strings.TrimSpace(updated.CreatedAt) == "" {
|
||||
updated.CreatedAt = builtin.CreatedAt
|
||||
|
||||
36
internal/api/builtin_operators_test.go
Normal file
36
internal/api/builtin_operators_test.go
Normal file
@ -0,0 +1,36 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"droboticflow/internal/domain"
|
||||
"droboticflow/internal/store"
|
||||
)
|
||||
|
||||
func TestUpsertBuiltinOperatorPreservesSharedMemoryOverride(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
server := &server{store: store.NewMemoryStore()}
|
||||
existing := domain.OperatorRecord{
|
||||
ID: "operator-builtin-shm",
|
||||
Name: "SHM operator",
|
||||
Type: "builtin",
|
||||
Resources: &domain.OperatorResources{
|
||||
CPULimit: "2",
|
||||
SharedMemorySize: "16Gi",
|
||||
},
|
||||
}
|
||||
builtin := existing
|
||||
builtin.Resources = &domain.OperatorResources{CPULimit: "4"}
|
||||
|
||||
items, err := upsertBuiltinOperator(ctx, server, []domain.OperatorRecord{existing}, builtin)
|
||||
if err != nil {
|
||||
t.Fatalf("upsert builtin operator: %v", err)
|
||||
}
|
||||
if len(items) != 1 || items[0].Resources == nil {
|
||||
t.Fatalf("unexpected upserted items: %#v", items)
|
||||
}
|
||||
if items[0].Resources.CPULimit != "4" || items[0].Resources.SharedMemorySize != "16Gi" {
|
||||
t.Fatalf("expected catalog CPU and user SHM override, got %#v", items[0].Resources)
|
||||
}
|
||||
}
|
||||
@ -3467,11 +3467,23 @@ func validateWorkflowGraphTopology(nodes []domain.WorkflowNode, edges []domain.W
|
||||
}
|
||||
|
||||
func validateWorkflowTopology(nodes []domain.WorkflowNode, edges []domain.WorkflowEdge) []validationIssue {
|
||||
issues := validateInputPortConnectionTopology(nodes, edges)
|
||||
issues := validateWorkflowNodeResources(nodes)
|
||||
issues = append(issues, validateInputPortConnectionTopology(nodes, edges)...)
|
||||
issues = append(issues, validateWorkflowGraphTopology(nodes, edges)...)
|
||||
return issues
|
||||
}
|
||||
|
||||
func validateWorkflowNodeResources(nodes []domain.WorkflowNode) []validationIssue {
|
||||
issues := make([]validationIssue, 0)
|
||||
for index, node := range nodes {
|
||||
field := "nodes[" + strconv.Itoa(index) + "].data.sharedMemorySize"
|
||||
if err := k8s.ValidateResourceQuantity(field, node.Data.SharedMemorySize); err != nil {
|
||||
issues = append(issues, validationIssue{Field: field, Message: err.Error()})
|
||||
}
|
||||
}
|
||||
return issues
|
||||
}
|
||||
|
||||
func (server *server) validateWorkflowOperatorPublication(ctx context.Context, nodes []domain.WorkflowNode, scope domain.WorkflowExecutionScope) []validationIssue {
|
||||
if normalizeWorkflowExecutionScope(scope) == domain.WorkflowExecutionScopeSandbox {
|
||||
return nil
|
||||
@ -4531,12 +4543,13 @@ func normalizeOperatorResources(resources *domain.OperatorResources) *domain.Ope
|
||||
gpuType = "NVIDIA GPU"
|
||||
}
|
||||
return &domain.OperatorResources{
|
||||
CPURequest: strings.TrimSpace(resources.CPURequest),
|
||||
CPULimit: strings.TrimSpace(resources.CPULimit),
|
||||
MemoryRequest: strings.TrimSpace(resources.MemoryRequest),
|
||||
MemoryLimit: strings.TrimSpace(resources.MemoryLimit),
|
||||
GPUType: gpuType,
|
||||
GPULimit: gpuLimit,
|
||||
CPURequest: strings.TrimSpace(resources.CPURequest),
|
||||
CPULimit: strings.TrimSpace(resources.CPULimit),
|
||||
MemoryRequest: strings.TrimSpace(resources.MemoryRequest),
|
||||
MemoryLimit: strings.TrimSpace(resources.MemoryLimit),
|
||||
GPUType: gpuType,
|
||||
GPULimit: gpuLimit,
|
||||
SharedMemorySize: strings.TrimSpace(resources.SharedMemorySize),
|
||||
}
|
||||
}
|
||||
|
||||
@ -4637,6 +4650,7 @@ func validateOperatorResources(resources *domain.OperatorResources) []validation
|
||||
{field: "resources.cpuLimit", value: resources.CPULimit},
|
||||
{field: "resources.memoryRequest", value: resources.MemoryRequest},
|
||||
{field: "resources.memoryLimit", value: resources.MemoryLimit},
|
||||
{field: "resources.sharedMemorySize", value: resources.SharedMemorySize},
|
||||
} {
|
||||
if err := k8s.ValidateResourceQuantity(item.field, item.value); err != nil {
|
||||
return []validationIssue{{Field: item.field, Message: err.Error()}}
|
||||
|
||||
@ -6027,6 +6027,19 @@ func TestWorkflowExecutionConfigValidationAndPersistence(t *testing.T) {
|
||||
if invalidUpdateRecorder.Code != http.StatusBadRequest {
|
||||
t.Fatalf("invalid update status: %d body=%s", invalidUpdateRecorder.Code, invalidUpdateRecorder.Body.String())
|
||||
}
|
||||
|
||||
invalidSHMCreateRecorder := httptest.NewRecorder()
|
||||
invalidSHMCreatePayload := []byte(`{
|
||||
"name":"非法共享内存工作流",
|
||||
"description":"workflow",
|
||||
"type":"batch",
|
||||
"nodes":[{"id":"node-source","type":"operator","data":{"label":"source","inputs":[],"outputs":[],"sharedMemorySize":"0"},"position":{"x":0,"y":0}}],
|
||||
"edges":[]
|
||||
}`)
|
||||
handler.ServeHTTP(invalidSHMCreateRecorder, httptest.NewRequest(http.MethodPost, "/api/projects/"+project.ID+"/workflows", bytes.NewReader(invalidSHMCreatePayload)))
|
||||
if invalidSHMCreateRecorder.Code != http.StatusBadRequest || !strings.Contains(invalidSHMCreateRecorder.Body.String(), "nodes[0].data.sharedMemorySize") {
|
||||
t.Fatalf("invalid shared memory workflow status/body: %d %s", invalidSHMCreateRecorder.Code, invalidSHMCreateRecorder.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkflowAndTemplateMetadataValidationAndNormalization(t *testing.T) {
|
||||
@ -6205,7 +6218,7 @@ func TestOperatorMetadataValidationAndNormalization(t *testing.T) {
|
||||
"image":" registry.example.com/custom/operator:v1 ",
|
||||
"inputs":[{"id":" in-1 ","name":" input ","type":" JSON ","description":" source ","transmission":" BUS "}],
|
||||
"outputs":[{"id":"out-1","name":" result ","type":" FILE ","transmission":" OSS "}],
|
||||
"resources":{"cpuLimit":" 500m ","memoryLimit":" 1Gi ","gpuType":" NVIDIA A800 80GB ","gpuLimit":" 1 "}
|
||||
"resources":{"cpuLimit":" 500m ","memoryLimit":" 1Gi ","gpuType":" NVIDIA A800 80GB ","gpuLimit":" 1 ","sharedMemorySize":" 16Gi "}
|
||||
}`))),
|
||||
)
|
||||
if createRecorder.Code != http.StatusCreated {
|
||||
@ -6234,7 +6247,7 @@ func TestOperatorMetadataValidationAndNormalization(t *testing.T) {
|
||||
if len(created.Outputs) != 1 || created.Outputs[0].Type != domain.PortTypeFile || created.Outputs[0].Transmission != "oss" {
|
||||
t.Fatalf("expected normalized output port, got %#v", created.Outputs)
|
||||
}
|
||||
if created.Resources == nil || created.Resources.CPULimit != "500m" || created.Resources.MemoryLimit != "1Gi" || created.Resources.GPUType != "NVIDIA A800 80GB" || created.Resources.GPULimit != "1" {
|
||||
if created.Resources == nil || created.Resources.CPULimit != "500m" || created.Resources.MemoryLimit != "1Gi" || created.Resources.GPUType != "NVIDIA A800 80GB" || created.Resources.GPULimit != "1" || created.Resources.SharedMemorySize != "16Gi" {
|
||||
t.Fatalf("expected normalized resources, got %#v", created.Resources)
|
||||
}
|
||||
|
||||
@ -6302,6 +6315,7 @@ func TestOperatorMetadataValidationAndNormalization(t *testing.T) {
|
||||
{name: "invalid output transmission", payload: `{"outputs":[{"id":"out-1","name":"result","type":"json","transmission":"mailbox"}]}`},
|
||||
{name: "duplicate output ids", payload: `{"outputs":[{"id":"dup","name":"one","type":"json"},{"id":"dup","name":"two","type":"json"}]}`},
|
||||
{name: "invalid resources", payload: `{"resources":{"gpuLimit":"half"}}`},
|
||||
{name: "invalid shared memory", payload: `{"resources":{"sharedMemorySize":"0"}}`},
|
||||
{name: "invalid version", payload: `{"version":"bad version"}`},
|
||||
} {
|
||||
recorder := httptest.NewRecorder()
|
||||
@ -6351,7 +6365,7 @@ func TestOperatorMetadataValidationAndNormalization(t *testing.T) {
|
||||
if len(updated.Outputs) != 1 || updated.Outputs[0].ID != "out-2" || updated.Outputs[0].Name != "tensor result" || updated.Outputs[0].Type != domain.PortTypeTensor || updated.Outputs[0].Transmission != "stream" {
|
||||
t.Fatalf("expected normalized updated output port, got %#v", updated.Outputs)
|
||||
}
|
||||
if updated.Resources == nil || updated.Resources.CPULimit != "750m" || updated.Resources.MemoryLimit != "2Gi" || updated.Resources.GPULimit != "" {
|
||||
if updated.Resources == nil || updated.Resources.CPULimit != "750m" || updated.Resources.MemoryLimit != "2Gi" || updated.Resources.GPULimit != "" || updated.Resources.SharedMemorySize != "" {
|
||||
t.Fatalf("expected normalized updated resources, got %#v", updated.Resources)
|
||||
}
|
||||
}
|
||||
|
||||
@ -182,6 +182,7 @@ type WorkflowNodeData struct {
|
||||
CPU string `json:"cpu,omitempty" bson:"cpu,omitempty"`
|
||||
Memory string `json:"memory,omitempty" bson:"memory,omitempty"`
|
||||
GPU string `json:"gpu,omitempty" bson:"gpu,omitempty"`
|
||||
SharedMemorySize string `json:"sharedMemorySize,omitempty" bson:"sharedMemorySize,omitempty"`
|
||||
}
|
||||
|
||||
type WorkflowNode struct {
|
||||
@ -222,12 +223,13 @@ type WorkflowStatusCounts struct {
|
||||
}
|
||||
|
||||
type OperatorResources struct {
|
||||
CPURequest string `json:"cpuRequest,omitempty" bson:"cpuRequest,omitempty"`
|
||||
CPULimit string `json:"cpuLimit,omitempty" bson:"cpuLimit,omitempty"`
|
||||
MemoryRequest string `json:"memoryRequest,omitempty" bson:"memoryRequest,omitempty"`
|
||||
MemoryLimit string `json:"memoryLimit,omitempty" bson:"memoryLimit,omitempty"`
|
||||
GPUType string `json:"gpuType,omitempty" bson:"gpuType,omitempty"`
|
||||
GPULimit string `json:"gpuLimit,omitempty" bson:"gpuLimit,omitempty"`
|
||||
CPURequest string `json:"cpuRequest,omitempty" bson:"cpuRequest,omitempty"`
|
||||
CPULimit string `json:"cpuLimit,omitempty" bson:"cpuLimit,omitempty"`
|
||||
MemoryRequest string `json:"memoryRequest,omitempty" bson:"memoryRequest,omitempty"`
|
||||
MemoryLimit string `json:"memoryLimit,omitempty" bson:"memoryLimit,omitempty"`
|
||||
GPUType string `json:"gpuType,omitempty" bson:"gpuType,omitempty"`
|
||||
GPULimit string `json:"gpuLimit,omitempty" bson:"gpuLimit,omitempty"`
|
||||
SharedMemorySize string `json:"sharedMemorySize,omitempty" bson:"sharedMemorySize,omitempty"`
|
||||
}
|
||||
|
||||
type OperatorPublicationStatus string
|
||||
|
||||
@ -358,11 +358,12 @@ func buildNodeEnv(run *domain.RunRecord, workflow *domain.WorkflowRecord, node d
|
||||
}
|
||||
|
||||
type effectiveNodeResources struct {
|
||||
CPURequest string
|
||||
CPULimit string
|
||||
MemoryRequest string
|
||||
MemoryLimit string
|
||||
GPULimit string
|
||||
CPURequest string
|
||||
CPULimit string
|
||||
MemoryRequest string
|
||||
MemoryLimit string
|
||||
GPULimit string
|
||||
SharedMemorySize string
|
||||
}
|
||||
|
||||
func buildNodeEnvWithResources(run *domain.RunRecord, workflow *domain.WorkflowRecord, node domain.WorkflowNode, inputArtifacts []domain.ExecutionArtifact, workspace nodeWorkspace, resources effectiveNodeResources) (map[string]string, error) {
|
||||
@ -513,10 +514,12 @@ func resolveEffectiveNodeResources(node domain.WorkflowNode, settings *domain.Se
|
||||
resources.MemoryRequest = firstNonEmpty(operatorResources.MemoryRequest, resources.MemoryRequest)
|
||||
resources.MemoryLimit = firstNonEmpty(operatorResources.MemoryLimit, resources.MemoryLimit)
|
||||
resources.GPULimit = firstNonEmpty(operatorResources.GPULimit, resources.GPULimit)
|
||||
resources.SharedMemorySize = firstNonEmpty(operatorResources.SharedMemorySize, resources.SharedMemorySize)
|
||||
}
|
||||
resources.CPULimit = firstNonEmpty(node.Data.CPU, resources.CPULimit)
|
||||
resources.MemoryLimit = firstNonEmpty(node.Data.Memory, resources.MemoryLimit)
|
||||
resources.GPULimit = firstNonEmpty(node.Data.GPU, resources.GPULimit)
|
||||
resources.SharedMemorySize = firstNonEmpty(node.Data.SharedMemorySize, resources.SharedMemorySize)
|
||||
return resources
|
||||
}
|
||||
|
||||
@ -654,6 +657,7 @@ func (runtime *Runtime) executeNode(ctx context.Context, run *domain.RunRecord,
|
||||
MemoryRequest: resources.MemoryRequest,
|
||||
MemoryLimit: resources.MemoryLimit,
|
||||
GPULimit: resources.GPULimit,
|
||||
SharedMemorySize: resources.SharedMemorySize,
|
||||
LongRunning: runtimeContractLongRunning(node.Data.RuntimeContract),
|
||||
CodeSubPath: codeSubPath,
|
||||
CodeMountPath: codeMountPath,
|
||||
|
||||
@ -393,11 +393,12 @@ func TestExecuteNodeFallsBackToOperatorResourceDefaults(t *testing.T) {
|
||||
Description: "gpu operator",
|
||||
Outputs: []domain.PortDefinition{{ID: "out-1", Name: "model", Type: domain.PortTypeFile}},
|
||||
Resources: &domain.OperatorResources{
|
||||
CPURequest: "8",
|
||||
CPULimit: "24",
|
||||
MemoryRequest: "32Gi",
|
||||
MemoryLimit: "128Gi",
|
||||
GPULimit: "1",
|
||||
CPURequest: "8",
|
||||
CPULimit: "24",
|
||||
MemoryRequest: "32Gi",
|
||||
MemoryLimit: "128Gi",
|
||||
GPULimit: "1",
|
||||
SharedMemorySize: "12Gi",
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
@ -460,6 +461,29 @@ func TestExecuteNodeFallsBackToOperatorResourceDefaults(t *testing.T) {
|
||||
if captured.CPURequest != "8" || captured.CPULimit != "24" || captured.MemoryRequest != "32Gi" || captured.MemoryLimit != "128Gi" {
|
||||
t.Fatalf("expected operator resource defaults, got %#v", captured)
|
||||
}
|
||||
if captured.SharedMemorySize != "12Gi" {
|
||||
t.Fatalf("expected operator shared memory default, got %q", captured.SharedMemorySize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveEffectiveNodeSharedMemoryPrecedence(t *testing.T) {
|
||||
operatorResources := &domain.OperatorResources{SharedMemorySize: "8Gi"}
|
||||
operatorDefault := resolveEffectiveNodeResources(domain.WorkflowNode{}, nil, operatorResources)
|
||||
if operatorDefault.SharedMemorySize != "8Gi" {
|
||||
t.Fatalf("operator shared memory default = %q, want 8Gi", operatorDefault.SharedMemorySize)
|
||||
}
|
||||
|
||||
nodeOverride := resolveEffectiveNodeResources(domain.WorkflowNode{
|
||||
Data: domain.WorkflowNodeData{SharedMemorySize: "16Gi"},
|
||||
}, nil, operatorResources)
|
||||
if nodeOverride.SharedMemorySize != "16Gi" {
|
||||
t.Fatalf("node shared memory override = %q, want 16Gi", nodeOverride.SharedMemorySize)
|
||||
}
|
||||
|
||||
platformFallback := resolveEffectiveNodeResources(domain.WorkflowNode{}, nil, nil)
|
||||
if platformFallback.SharedMemorySize != "" {
|
||||
t.Fatalf("empty shared memory should be left for launcher platform fallback, got %q", platformFallback.SharedMemorySize)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessRunFailsInvalidWorkflowWithDuplicateInputEdges(t *testing.T) {
|
||||
|
||||
@ -2,6 +2,7 @@ const operatorTypes = new Set(["builtin", "custom", "lightweight"]);
|
||||
const portTypes = new Set(["json", "image", "video", "tensor", "file"]);
|
||||
const portTransmissions = new Set(["", "bus", "oss", "shm", "stream"]);
|
||||
const positiveInteger = /^[1-9]\d*$/;
|
||||
const positiveResourceQuantity = /^(?:[1-9]\d*(?:\.\d+)?|0\.\d+)(?:(?:[eE][+-]?\d+)|(?:[numkKMGTP](?:i)?))?$/;
|
||||
|
||||
function trimString(value) {
|
||||
return typeof value === "string" ? value.trim() : "";
|
||||
@ -31,6 +32,22 @@ function normalizePorts(items) {
|
||||
: [];
|
||||
}
|
||||
|
||||
function normalizeResources(resources) {
|
||||
if (!resources || typeof resources !== "object") {
|
||||
return resources;
|
||||
}
|
||||
return {
|
||||
...resources,
|
||||
cpuRequest: trimString(resources.cpuRequest),
|
||||
cpuLimit: trimString(resources.cpuLimit),
|
||||
memoryRequest: trimString(resources.memoryRequest),
|
||||
memoryLimit: trimString(resources.memoryLimit),
|
||||
gpuType: trimString(resources.gpuType),
|
||||
gpuLimit: trimString(resources.gpuLimit),
|
||||
sharedMemorySize: trimString(resources.sharedMemorySize),
|
||||
};
|
||||
}
|
||||
|
||||
export function buildOperatorPayload(source = {}, overrides = {}) {
|
||||
const type = normalizeOperatorType(overrides.type ?? source.type ?? "custom") || "custom";
|
||||
const category = trimString(overrides.category ?? source.category) || "CV";
|
||||
@ -55,7 +72,7 @@ export function buildOperatorPayload(source = {}, overrides = {}) {
|
||||
generationSource: trimString(overrides.generationSource ?? source.generationSource),
|
||||
publicationEvidence: deepClone(overrides.publicationEvidence ?? source.publicationEvidence),
|
||||
hyperparams: deepClone(overrides.hyperparams ?? source.hyperparams ?? []),
|
||||
resources: deepClone(overrides.resources ?? source.resources),
|
||||
resources: normalizeResources(deepClone(overrides.resources ?? source.resources)),
|
||||
};
|
||||
}
|
||||
|
||||
@ -86,12 +103,16 @@ function validatePorts(field, items) {
|
||||
function validateResources(resources = {}) {
|
||||
const gpuLimit = trimString(resources.gpuLimit);
|
||||
const gpuType = trimString(resources.gpuType);
|
||||
const sharedMemorySize = trimString(resources.sharedMemorySize);
|
||||
if (gpuLimit && !positiveInteger.test(gpuLimit)) {
|
||||
return "GPU 数量必须是正整数";
|
||||
}
|
||||
if (gpuLimit && !gpuType) {
|
||||
return "需要 GPU 时必须选择 GPU 型号";
|
||||
}
|
||||
if (sharedMemorySize && !positiveResourceQuantity.test(sharedMemorySize)) {
|
||||
return "共享内存必须是大于 0 的 Kubernetes resource quantity,例如 8Gi";
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
|
||||
@ -1394,6 +1394,11 @@ export default function Operators() {
|
||||
GPU {gpuRequirement.gpuType} x{gpuRequirement.gpuLimit}
|
||||
</span>
|
||||
)}
|
||||
{trimResource(op.resources?.sharedMemorySize) && (
|
||||
<span className="text-[10px] px-2 py-0.5 rounded-full font-mono uppercase tracking-wider bg-violet-500/10 text-violet-300 border border-violet-500/30">
|
||||
SHM {trimResource(op.resources?.sharedMemorySize)}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@ -1989,7 +1994,7 @@ export default function Operators() {
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="grid grid-cols-1 gap-3 md:grid-cols-4">
|
||||
<div className="grid grid-cols-1 gap-3 md:grid-cols-5">
|
||||
<div className="space-y-1.5">
|
||||
<label className="text-[10px] font-medium uppercase tracking-wider text-gray-500">
|
||||
CPU Request
|
||||
@ -2038,6 +2043,27 @@ export default function Operators() {
|
||||
onChange={(event) => updateResources({ memoryLimit: event.target.value })}
|
||||
/>
|
||||
</div>
|
||||
<div className="space-y-1.5">
|
||||
<label className="text-[10px] font-medium uppercase tracking-wider text-gray-500">
|
||||
Shared Memory (/dev/shm)
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
list="droboticflow-shared-memory-sizes"
|
||||
placeholder="留空继承平台默认"
|
||||
className="w-full bg-[#151619] border border-[#2a2b2f] rounded-md px-3 py-2 text-xs focus:outline-none focus:border-blue-500"
|
||||
value={newOp.resources?.sharedMemorySize || ""}
|
||||
onChange={(event) => updateResources({ sharedMemorySize: event.target.value })}
|
||||
/>
|
||||
<datalist id="droboticflow-shared-memory-sizes">
|
||||
{["1Gi", "2Gi", "4Gi", "8Gi", "16Gi", "32Gi"].map((size) => (
|
||||
<option key={size} value={size} />
|
||||
))}
|
||||
</datalist>
|
||||
<p className="text-[10px] leading-snug text-gray-600">
|
||||
空值继承平台默认;实际使用会占用 Pod/节点内存。
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{getGPURequirement(newOp).configured && (
|
||||
|
||||
@ -216,6 +216,7 @@ interface OperatorNodeData {
|
||||
cpu?: string;
|
||||
memory?: string;
|
||||
gpu?: string;
|
||||
sharedMemorySize?: string;
|
||||
executionState?: OperatorExecutionState;
|
||||
}
|
||||
|
||||
@ -785,6 +786,12 @@ export default function WorkflowEditor() {
|
||||
});
|
||||
const isReadOnly = editorLockState.locked;
|
||||
const gpuStatus = useMemo(() => gpuCapacityStatus(workerCapacity?.gpu), [workerCapacity?.gpu]);
|
||||
const selectedNodeOperator = useMemo(() => {
|
||||
const operatorID = String((selectedNode?.data as OperatorNodeData | undefined)?.operatorId || "").trim();
|
||||
return operatorID ? operators.find((operator) => operator.id === operatorID) || null : null;
|
||||
}, [operators, selectedNode?.data]);
|
||||
const inheritedSharedMemorySize = String(selectedNodeOperator?.resources?.sharedMemorySize || "").trim();
|
||||
const sharedMemoryPlaceholder = inheritedSharedMemorySize ? `继承算子 ${inheritedSharedMemorySize}` : "继承平台默认";
|
||||
const selectedNodeRequestsGPU = hasPositiveGPURequest((selectedNode?.data as OperatorNodeData | undefined)?.gpu);
|
||||
const selectedNodeGPUBlocked = selectedNodeRequestsGPU && workerCapacity?.gpu && !workerCapacity.gpu.available;
|
||||
const workflowGPUCapacityBlocker = useMemo(
|
||||
@ -2455,6 +2462,7 @@ export default function WorkflowEditor() {
|
||||
cpu: op.resources?.cpuLimit || "",
|
||||
memory: op.resources?.memoryLimit || "",
|
||||
gpu: op.resources?.gpuLimit || "",
|
||||
sharedMemorySize: "",
|
||||
},
|
||||
};
|
||||
|
||||
@ -6519,7 +6527,7 @@ export default function WorkflowEditor() {
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div className={cn("grid gap-2", isConfigPanelExpanded ? "grid-cols-3" : "grid-cols-2")}>
|
||||
<div className={cn("grid gap-2", isConfigPanelExpanded ? "grid-cols-4" : "grid-cols-2")}>
|
||||
<div className="space-y-1">
|
||||
<label className="text-[10px] text-gray-500 uppercase font-mono">资源限制 (CPU)</label>
|
||||
<input
|
||||
@ -6564,6 +6572,27 @@ export default function WorkflowEditor() {
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
<div className="space-y-1">
|
||||
<label className="text-[10px] text-gray-500 uppercase font-mono">
|
||||
共享内存 (/dev/shm)
|
||||
</label>
|
||||
<input
|
||||
type="text"
|
||||
list="droboticflow-workflow-shared-memory-sizes"
|
||||
className="w-full bg-[#0a0a0a] border border-[#2a2b2f] rounded px-2 py-1 text-xs focus:outline-none focus:border-blue-500 disabled:opacity-50"
|
||||
value={selectedNode.data.sharedMemorySize || ""}
|
||||
placeholder={sharedMemoryPlaceholder}
|
||||
onChange={(e) => updateNodeData({ sharedMemorySize: e.target.value })}
|
||||
readOnly={isReadOnly}
|
||||
disabled={isReadOnly}
|
||||
/>
|
||||
<datalist id="droboticflow-workflow-shared-memory-sizes">
|
||||
{["1Gi", "2Gi", "4Gi", "8Gi", "16Gi", "32Gi"].map((size) => (
|
||||
<option key={size} value={size} />
|
||||
))}
|
||||
</datalist>
|
||||
<p className="text-[10px] leading-snug text-gray-600">留空按算子默认或平台默认执行。</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="space-y-3">
|
||||
|
||||
@ -58,6 +58,7 @@ export interface GeneratedOperator {
|
||||
memoryLimit?: string;
|
||||
gpuType?: string;
|
||||
gpuLimit?: string;
|
||||
sharedMemorySize?: string;
|
||||
};
|
||||
inputs: { name: string; type: PortType; transmission: PortTransmission; description?: string }[];
|
||||
outputs: { name: string; type: PortType; transmission: PortTransmission; description?: string }[];
|
||||
|
||||
@ -259,6 +259,7 @@ export interface Operator {
|
||||
memoryLimit?: string;
|
||||
gpuType?: string;
|
||||
gpuLimit?: string;
|
||||
sharedMemorySize?: string;
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@ -103,3 +103,19 @@ test("operator payload validation rejects incomplete gpu resources", () => {
|
||||
assert.equal(getOperatorPayloadIssue({ ...base, resources: { gpuLimit: "1" } }), "需要 GPU 时必须选择 GPU 型号");
|
||||
assert.equal(getOperatorPayloadIssue({ ...base, resources: { gpuType: "NVIDIA A800 80GB", gpuLimit: "1" } }), "");
|
||||
});
|
||||
|
||||
test("operator payload preserves and validates shared memory resources", () => {
|
||||
const base = {
|
||||
name: "HoloBrain 训练",
|
||||
type: "custom",
|
||||
category: "AI",
|
||||
description: "训练算子",
|
||||
inputs: [],
|
||||
outputs: [],
|
||||
resources: { sharedMemorySize: " 16Gi " },
|
||||
};
|
||||
const payload = buildOperatorPayload(base);
|
||||
assert.equal(payload.resources.sharedMemorySize, "16Gi");
|
||||
assert.equal(getOperatorPayloadIssue(base), "");
|
||||
assert.match(getOperatorPayloadIssue({ ...base, resources: { sharedMemorySize: "0" } }), /共享内存必须是大于 0/);
|
||||
});
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user