feat: deploy the web workbench on the local k3s cluster
The local cluster runs the workbench next to the API. It is a development build of the web, from the image's new WEB_BUILD=development argument: - Next.js inlines NODE_ENV into production builds. - Development sign-in therefore exists only in development builds, and production images never carry it. `node scripts/k3s-local.mjs web` forwards it to http://localhost:3000. The web refuses changes from any origin but its WF_PUBLIC_URL, so that address is set at deploy time (WF_K3S_WEB_PORT). `build` now also takes the images to rebuild, for example `build web`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
b4ece6057c
commit
1beee7e593
@ -5,7 +5,12 @@ const config: NextConfig = {
|
||||
poweredByHeader: false,
|
||||
agentRules: false,
|
||||
devIndicators: false,
|
||||
experimental: { optimizePackageImports: ['lucide-react'] },
|
||||
experimental: {
|
||||
optimizePackageImports: ['lucide-react'],
|
||||
// Local development clusters run a development build (the image's WEB_BUILD=development):
|
||||
// development sign-in exists only there, since production builds inline NODE_ENV=production.
|
||||
...(process.env.WF_WEB_DEVELOPMENT_BUILD === 'true' ? { allowDevelopmentBuild: true as const } : {}),
|
||||
},
|
||||
async headers() {
|
||||
return [
|
||||
{
|
||||
|
||||
@ -10,7 +10,7 @@
|
||||
| 沙箱 | 普通 runc 容器,`WF_RUN_SANDBOX_MODE=host`,没有 Kata 与 bubblewrap | Kata 虚拟机 + bubblewrap |
|
||||
| Codex 的命令沙箱 | 起不来(runc Pod 不允许 bubblewrap 创建用户命名空间),Codex 不带它运行,时间线的「Executor started」会注明 | 必须可用,否则运行停在 `infrastructure` 等待 |
|
||||
| 服务之间 | 集群内 HTTP(`WF_ALLOW_INSECURE_HTTP=true`) | HTTPS 与 mTLS |
|
||||
| 登录 | 开发令牌,没有 OIDC | OIDC |
|
||||
| 登录 | 开发令牌,没有 OIDC;Web 是 Next.js 的开发构建 | OIDC,Web 是生产构建 |
|
||||
| 镜像 | 本地构建,推到虚拟机里的 `localhost:5000` | 私有仓库中经审阅的镜像 |
|
||||
|
||||
每次运行的 NetworkPolicy 照常生效(k3s 自带网络策略控制器):沙箱只能访问 DNS、API(含模型代理)、Temporal 与平台 Git。模型调用经集群内 API 的运行级代理转发到你配置的网关。
|
||||
@ -35,7 +35,7 @@
|
||||
node scripts/k3s-local.mjs up
|
||||
```
|
||||
|
||||
构建 API、worker、平台 Git 与 provisioner 镜像并推送(第一次需要几分钟):
|
||||
构建 API、worker、平台 Git、provisioner 与 Web 镜像并推送(第一次需要几分钟):
|
||||
|
||||
```bash
|
||||
node scripts/k3s-local.mjs build
|
||||
@ -73,12 +73,16 @@ node scripts/k3s-local.mjs down
|
||||
|
||||
## 工作台
|
||||
|
||||
本地 k3s 里没有部署 Web。把 API 转发到本机,再用 `npm run dev:web` 启动工作台:在 `apps/web/.env.local` 中把 `WF_API_URL` 设为 `http://127.0.0.1:4000`,`WF_DEV_AUTH_TOKEN` 设为 `.wishforge/k3s/secrets.json` 中的同名值。
|
||||
Web 工作台随 `deploy` 一起部署在集群里,由它在集群内调用 API。把它转发到本机(保持运行,Ctrl-C 停止转发):
|
||||
|
||||
```bash
|
||||
kubectl --context colima-wishforge --namespace wishforge-control port-forward service/api 4000:4000
|
||||
node scripts/k3s-local.mjs web
|
||||
```
|
||||
|
||||
然后打开 <http://localhost:3000>,地址要完全一致:工作台只接受来自部署时登记的地址(`WF_PUBLIC_URL`)的修改请求,换成 `127.0.0.1` 打开,提交愿望等操作会被拒绝。端口被占用时,用 `WF_K3S_WEB_PORT=<端口>` 重新执行 `deploy`,`web` 会转发到新端口。每次 `deploy` 都会换掉 Web 的 Pod,端口转发随之断开,需要再执行一次 `web`。
|
||||
|
||||
集群里的 Web 是 Next.js 的开发构建(镜像构建参数 `WEB_BUILD=development`)。本地没有 OIDC,每个访问者都以开发令牌的身份登录。这种登录方式只存在于开发构建中,生产镜像是生产构建,不带它。
|
||||
|
||||
Temporal 控制台:
|
||||
|
||||
```bash
|
||||
@ -91,4 +95,4 @@ kubectl --context colima-wishforge --namespace wishforge-control port-forward se
|
||||
- 沙箱 Job 在 `wishforge-tenant-demo` 命名空间,名称形如 `wf-<哈希>-executor` / `-verifier`。运行进入等待或结束后,Job 会被释放,要看它的日志得趁运行还在进行。
|
||||
- 运行时间线(`smoke` 的输出,或 `GET /v1/runs/<id>/events`)记录 Agent 执行的每条命令及其退出码,模型请求经代理记账。
|
||||
- `up` 在下载 k3s 时失败(GitHub 的下载偶尔中断):直接重新执行 `up`,它会先停下装了一半的虚拟机再重新启动。
|
||||
- 改了代码后重新执行 `build` 与 `deploy`;新的运行会使用新的 worker 镜像。
|
||||
- 改了代码后重新执行 `build` 与 `deploy`;新的运行会使用新的 worker 镜像。只改了个别服务时可以只重建它们,例如 `node scripts/k3s-local.mjs build web`(可选 `runtime`、`worker`、`git-storage`、`provisioner`、`web`),其余镜像沿用上次的 digest。
|
||||
|
||||
@ -8,7 +8,7 @@
|
||||
| `worker` | coordinator、deployer、executor、verifier(含 Git、Python、bubblewrap 与 Docker CLI) | `DOCKER_CLI_IMAGE` |
|
||||
| `runner` | 部署 Runner | `DOCKER_CLI_IMAGE` |
|
||||
| `git-storage` | 平台 Git 服务 | — |
|
||||
| `web` | Web 工作台(Next.js 生产构建) | — |
|
||||
| `web` | Web 工作台(Next.js 生产构建) | 可选 `WEB_BUILD=development`:本地开发集群用的开发构建,只有它带开发令牌登录 |
|
||||
| `provisioner` | provisioner(Kubernetes 后端,含 kubectl) | `KUBECTL_IMAGE`,可选 `KUBECTL_PATH` |
|
||||
|
||||
所有目标都需要 `NODE_IMAGE`(Node 22 或以上、基于 Debian)。运维提供经过审阅的不可变镜像引用,仓库中没有可变的默认标签;工具镜像参数默认为 `scratch`,构建需要它们的目标时没有提供就会失败。
|
||||
|
||||
@ -46,10 +46,18 @@ FROM runtime AS git-storage
|
||||
CMD ["node", "dist/git-service/main.mjs"]
|
||||
|
||||
FROM runtime AS web
|
||||
# development is for local development clusters only (scripts/k3s-local.mjs): the web's development
|
||||
# sign-in exists only in a development build, so production images never contain it.
|
||||
ARG WEB_BUILD=production
|
||||
USER root
|
||||
RUN npm run build -w @wishforge/web && chown -R node:node apps/web/.next
|
||||
RUN case "$WEB_BUILD" in \
|
||||
production) ;; \
|
||||
development) export NODE_ENV=development WF_WEB_DEVELOPMENT_BUILD=true ;; \
|
||||
*) echo "WEB_BUILD must be production or development" >&2; exit 1 ;; \
|
||||
esac \
|
||||
&& npm run build -w @wishforge/web && chown -R node:node apps/web/.next
|
||||
USER node
|
||||
ENV NODE_ENV=production
|
||||
ENV NODE_ENV=${WEB_BUILD}
|
||||
EXPOSE 3000
|
||||
CMD ["node", "node_modules/next/dist/bin/next", "start", "apps/web", "--hostname", "0.0.0.0", "--port", "3000"]
|
||||
|
||||
|
||||
@ -264,6 +264,45 @@ spec:
|
||||
selector: { app: api }
|
||||
ports: [{ port: 4000, targetPort: 4000 }]
|
||||
---
|
||||
# The workbench, a development build of the web (node scripts/k3s-local.mjs web forwards it to the
|
||||
# host). It signs every visitor in with the development token and calls the API in the cluster.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: web
|
||||
namespace: wishforge-control
|
||||
spec:
|
||||
replicas: 1
|
||||
selector: { matchLabels: { app: web } }
|
||||
template:
|
||||
metadata:
|
||||
labels: { app: web, wishforge.io/component: web }
|
||||
spec:
|
||||
containers:
|
||||
- name: web
|
||||
image: '{{WEB_IMAGE}}'
|
||||
env:
|
||||
- { name: WF_DEPLOYMENT_MODE, value: development }
|
||||
- { name: NODE_ENV, value: development }
|
||||
- { name: WF_API_URL, value: http://api.wishforge-control.svc.cluster.local:4000 }
|
||||
# The address the browser opens; the web refuses changes from any other origin.
|
||||
- { name: WF_PUBLIC_URL, value: '{{WEB_PUBLIC_URL}}' }
|
||||
- name: WF_DEV_AUTH_TOKEN
|
||||
valueFrom: { secretKeyRef: { name: wishforge-local, key: WF_DEV_AUTH_TOKEN } }
|
||||
ports: [{ containerPort: 3000 }]
|
||||
readinessProbe:
|
||||
httpGet: { path: /api/session, port: 3000 }
|
||||
periodSeconds: 5
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: web
|
||||
namespace: wishforge-control
|
||||
spec:
|
||||
selector: { app: web }
|
||||
ports: [{ port: 3000, targetPort: 3000 }]
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
|
||||
@ -3,9 +3,11 @@
|
||||
//
|
||||
// node scripts/k3s-local.mjs up start the colima-wishforge VM with k3s and a local registry
|
||||
// node scripts/k3s-local.mjs build build the service images and push them to the local registry
|
||||
// (`build web worker` rebuilds only those)
|
||||
// node scripts/k3s-local.mjs deploy render infra/k8s/local and apply it (needs WF_CODEX_DEFAULT_MODEL)
|
||||
// node scripts/k3s-local.mjs model store WF_TEST_MODEL_API_KEY as the tenant's Codex key
|
||||
// node scripts/k3s-local.mjs smoke create a project and run one wish to the end
|
||||
// node scripts/k3s-local.mjs web forward the workbench to the host (http://localhost:3000)
|
||||
// node scripts/k3s-local.mjs status show the platform and sandbox pods
|
||||
// node scripts/k3s-local.mjs down stop the VM (data stays; `up` resumes it)
|
||||
//
|
||||
@ -22,7 +24,9 @@ const CONTEXT = `colima-${PROFILE}`;
|
||||
const REGISTRY = 'localhost:5000';
|
||||
const STATE = '.wishforge/k3s';
|
||||
const DOCKER_CONFIG = path.resolve(STATE, 'docker');
|
||||
const TARGETS = ['runtime', 'worker', 'git-storage', 'provisioner'];
|
||||
const TARGETS = ['runtime', 'worker', 'git-storage', 'provisioner', 'web'];
|
||||
/** Development sign-in exists only in a development build of the web. */
|
||||
const TARGET_ARGS = { web: { WEB_BUILD: 'development' } };
|
||||
const BASE_IMAGES = {
|
||||
NODE_IMAGE: process.env.WF_K3S_NODE_IMAGE ?? 'node:22-bookworm',
|
||||
DOCKER_CLI_IMAGE: process.env.WF_K3S_DOCKER_CLI_IMAGE ?? 'docker:28-cli',
|
||||
@ -166,9 +170,13 @@ async function up() {
|
||||
|
||||
async function build() {
|
||||
requireCluster();
|
||||
const selected = process.argv.slice(3);
|
||||
const unknown = selected.filter(target => !TARGETS.includes(target));
|
||||
if (unknown.length) throw new Error(`Unknown images ${unknown.join(', ')}; choose from ${TARGETS.join(', ')}`);
|
||||
await prepareDocker();
|
||||
const images = {};
|
||||
for (const target of TARGETS) {
|
||||
// Images not rebuilt keep the digests they were deployed with.
|
||||
const images = await readJson('images.json');
|
||||
for (const target of selected.length ? selected : TARGETS) {
|
||||
const tag = `${REGISTRY}/wishforge/${target}:dev`;
|
||||
docker([
|
||||
'build',
|
||||
@ -176,7 +184,10 @@ async function build() {
|
||||
'infra/images/runtime.Dockerfile',
|
||||
'--target',
|
||||
target,
|
||||
...Object.entries(BASE_IMAGES).flatMap(([name, value]) => ['--build-arg', `${name}=${value}`]),
|
||||
...Object.entries({ ...BASE_IMAGES, ...TARGET_ARGS[target] }).flatMap(([name, value]) => [
|
||||
'--build-arg',
|
||||
`${name}=${value}`,
|
||||
]),
|
||||
'--tag',
|
||||
tag,
|
||||
'.',
|
||||
@ -204,6 +215,8 @@ async function deploy() {
|
||||
throw new Error('Build the images first: node scripts/k3s-local.mjs build');
|
||||
const model = process.env.WF_CODEX_DEFAULT_MODEL;
|
||||
if (!model) throw new Error('Set WF_CODEX_DEFAULT_MODEL to a model your gateway serves (GET <gateway>/v1/models)');
|
||||
const webPort = process.env.WF_K3S_WEB_PORT ?? '3000';
|
||||
if (!/^\d{2,5}$/.test(webPort)) throw new Error('WF_K3S_WEB_PORT must be a port number');
|
||||
// Generated once and kept outside Git, so redeploying keeps the database password and tokens.
|
||||
const secrets = await readJson('secrets.json');
|
||||
const secret = () => randomBytes(32).toString('hex');
|
||||
@ -229,6 +242,9 @@ async function deploy() {
|
||||
WORKER_IMAGE: images.worker,
|
||||
GIT_STORAGE_IMAGE: images['git-storage'],
|
||||
PROVISIONER_IMAGE: images.provisioner,
|
||||
WEB_IMAGE: images.web,
|
||||
// `web` forwards this port; the workbench accepts changes only from this origin.
|
||||
WEB_PUBLIC_URL: `http://localhost:${webPort}`,
|
||||
MODEL: model,
|
||||
OPENAI_BASE_URL: process.env.WF_OPENAI_BASE_URL ?? '',
|
||||
};
|
||||
@ -249,9 +265,10 @@ async function deploy() {
|
||||
'provisioner',
|
||||
'worker-coordinator',
|
||||
'worker-deployer',
|
||||
'web',
|
||||
])
|
||||
kubectl(['--namespace', 'wishforge-control', 'rollout', 'status', `deployment/${name}`, '--timeout=600s']);
|
||||
console.log('WishForge is running. Next: node scripts/k3s-local.mjs model, then smoke.');
|
||||
console.log('WishForge is running. Next: node scripts/k3s-local.mjs model, then smoke; `web` opens the workbench.');
|
||||
}
|
||||
|
||||
/** Runs a small Node program inside the API pod, against the API on its own loopback. */
|
||||
@ -341,6 +358,25 @@ console.log('Timed out after 20 minutes');
|
||||
process.exit(1);`);
|
||||
}
|
||||
|
||||
/** Forwards the workbench to the port it was deployed for: it refuses changes from other origins. */
|
||||
async function web() {
|
||||
requireCluster();
|
||||
const deployed = capture('kubectl', [
|
||||
'--context',
|
||||
CONTEXT,
|
||||
'--namespace',
|
||||
'wishforge-control',
|
||||
'get',
|
||||
'deployment/web',
|
||||
'--output',
|
||||
'jsonpath={.spec.template.spec.containers[0].env[?(@.name=="WF_PUBLIC_URL")].value}',
|
||||
]);
|
||||
if (!deployed.ok || !deployed.stdout.trim()) throw new Error('Deploy first: node scripts/k3s-local.mjs deploy');
|
||||
const url = new URL(deployed.stdout.trim());
|
||||
console.log(`Open ${url.origin} (exactly this address). Ctrl-C stops forwarding.`);
|
||||
kubectl(['--namespace', 'wishforge-control', 'port-forward', 'service/web', `${url.port}:3000`]);
|
||||
}
|
||||
|
||||
async function status() {
|
||||
requireCluster();
|
||||
kubectl(['get', 'pods,jobs,pvc', '--namespace', 'wishforge-control']);
|
||||
@ -351,7 +387,7 @@ async function down() {
|
||||
run('colima', ['stop', '--profile', PROFILE]);
|
||||
}
|
||||
|
||||
const commands = { up, build, deploy, model, smoke, status, down };
|
||||
const commands = { up, build, deploy, model, smoke, web, status, down };
|
||||
const command = commands[process.argv[2]];
|
||||
if (!command) {
|
||||
console.error(`Usage: node scripts/k3s-local.mjs ${Object.keys(commands).join('|')}`);
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user