DRoboticFlow

DRoboticFlow is a workflow orchestration platform for embodied AI and vision pipelines. This repository now contains:

  • a React/Vite frontend
  • a Go-based MongoDB-backed REST API
  • a Go-based worker service that executes workflow DAGs
  • a Python-based Kubernetes Job operator runner
  • deployment assets for MongoDB, Kubernetes, and Baidu Cloud Container Registry

Architecture

  • cmd/api: Go API entrypoint and SPA host
  • cmd/worker: Go worker entrypoint and worker control API
  • internal/api: REST handlers and request validation
  • internal/worker: run scheduler, DAG executor, audit pause/resume, worker control API
  • internal/store: Mongo-backed repositories plus in-memory test store
  • internal/k8s: Kubernetes Job launcher for operator nodes
  • backend/operator-runner: Python entrypoint used by Kubernetes Jobs
  • src: existing frontend, now wired to real APIs
  • k8s: Kubernetes manifest template
  • scripts: build, render, deploy, and smoke test scripts
  • docs: API, deployment, operations, and design documents

Local Development

Prerequisites

  • Node.js 22+
  • Go 1.24+
  • Python 3.9+
  • MongoDB 7+ reachable at MONGO_URI

Setup

  1. Install dependencies:

    npm install
    
  2. Copy the example environment file and adjust values:

    cp .env.example .env.local
    
  3. Start the Go API and frontend:

    npm run dev
    
  4. Start the full stack including Go worker:

    npm run dev:full
    

Useful Scripts

  • npm run lint: type-check frontend and verify Go formatting
  • npm test: Go unit and integration tests
  • npm run build: build frontend assets and Go binaries
  • npm run verify: lint + test + build
  • npm run smoke: local smoke test against the running API and worker
  • npm run reconcile:data -- <api-base-url>: remove legacy demo data, reset settings, and bootstrap built-in operators through the real API

Runtime Configuration

Key environment variables:

  • PORT: API port, default 3001
  • WORKER_PORT: worker port, default 3002
  • MONGO_URI: MongoDB connection string
  • MONGO_DB: Mongo database name
  • MONGO_CONNECT_RETRIES: startup retry count for API and worker Mongo connections
  • MONGO_CONNECT_RETRY_DELAY_MS: delay between Mongo connection retries
  • WORKER_POLL_INTERVAL_MS: worker polling interval
  • K8S_ENABLED: enable Kubernetes Job execution in the worker
  • K8S_NAMESPACE: namespace used for Jobs
  • K8S_IMAGE_PULL_SECRET_NAME: image pull secret injected into operator-runner Jobs
  • OPERATOR_RUNNER_IMAGE: container image used for node execution Jobs
  • AI_API_BASE_URL: OpenAI-compatible DashScope endpoint used by server-side AI generation, default https://coding.dashscope.aliyuncs.com/v1
  • AI_MODEL: server-side model name, default glm-5
  • AI_API_KEY: server-side DashScope-compatible API key used for workflow/operator generation
  • VITE_API_BASE_URL: optional frontend API base URL override

The frontend never reads the AI key directly. Workflow generation and operator generation are proxied through the Go API so secrets stay on the server side. The settings page can update K8s access details, the worker defaults, the global timezone, provider, baseURL, model, and the server-side AI key. GET /api/settings only returns whether secrets exist via k8s.hasKubeconfig / ai.hasAPIKey; it does not echo plaintext kubeconfig or API keys back to the browser. Timestamps in the UI are rendered in the configured IANA timezone with compact minute-level formatting for cards, lists, and summaries, while logs keep finer granularity when needed. OSS-related bucket and path settings belong on OSS operators in workflows, not in the global system settings page. The OSS browser APIs also now require the current node SOURCE_PATH and only list content inside that node-scoped root.

Workflow execution concurrency now has two levels:

  • system default: settings.worker.maxConcurrentRuns
  • workflow override: workflow.maxConcurrentRuns

The worker enforces both. A workflow value of 0 means "inherit the global default". Dispatch only starts queued runs while the global active run count is below the system limit and the active count for that workflow is below its effective limit.

Workflow-level execution safeguards are configured alongside the other top-bar workflow settings in the editor:

  • workflow.nodeTimeoutSeconds
    • 0: disable workflow-level per-node timeout
    • 1+: fail an operator execution once a single node exceeds the configured seconds
  • workflow.runTimeoutSeconds
    • 0: disable workflow-level timeout
    • 1+: fail the current run once total execution time exceeds the configured seconds
  • workflow.maxRetryAttempts
    • 0: disable automatic retry
    • 1+: automatically enqueue a new retry run after a failure, up to the configured count
  • workflow.retryBackoffSeconds
    • 0: retry immediately
    • 1+: keep the retry run queued until its backoff window expires

Automatic retries create real follow-up runs with trigger=retry, attempt=2+, parentRunId pointing at the failed attempt, and availableAt marking the first dispatchable time after backoff. Monitoring and history stay auditable because the original run is never mutated into the retry attempt.

Workflow templates now carry the same workflow-level defaults as project workflows. When you save a workflow as a template, duplicate a template, edit a template, or instantiate a template into a project, the workflow type, schedule, and execution-safeguard fields move across with it instead of being dropped or reset.

Artifact previews are persisted with the run artifacts themselves. JSON artifacts keep their structured payload in artifact.data, while non-JSON artifacts can attach a compact artifact.data.preview block for text, CSV/TSV, JSONL, images, audio/video metadata, PDF/document hints, SQLite/rosbag summaries, and LeRobot dataset summaries. Artifacts also carry portId / portName, which lets the worker route multi-output operators by edge handle instead of mixing all outputs together.

The built-in operator catalog now includes real data-processing and robotics helpers on top of the original source/merge/upload/audit flow: 目录扫描, CSV 转 JSON, JSON 转 CSV, JSON 字段提取, JSON 过滤, JSON 映射重命名, 文本模板渲染, 条件分流, LeRobot 数据集概览, ROSBag 数据概览, and Shell 命令执行.

Deployment

The repository includes three container images:

  • droboticflow-api
  • droboticflow-worker
  • droboticflow-operator-runner

The API image builds the Vite frontend and a Go binary into one container. The worker image contains a Go binary. The operator-runner image is Python-only and is launched by the worker as a Kubernetes Job.

Build and push images:

./scripts/build-and-push-images.sh

If the target cluster cannot pull the upstream mongo:7 image from Docker Hub, mirror it into the same Baidu registry and set MONGO_IMAGE during deployment.

Render Kubernetes manifests:

./scripts/render-manifests.sh

Deploy to Kubernetes:

./scripts/deploy-k8s.sh

Reconcile persistent API data after deployment:

./scripts/reconcile-system-data.sh http://<node-ip>:30080

Run a smoke test locally:

npm run smoke

Run a smoke test against a deployed cluster:

./scripts/smoke-test.sh http://<node-ip>:30080

See the detailed guides:

Description
No description provided
Readme 80 MiB
Languages
Python 48.8%
Go 29.8%
TypeScript 10.3%
JavaScript 7.9%
Shell 1.3%
Other 1.7%