droboticflow/docs/production-readiness.md
2026-07-14 16:14:39 +08:00

174 lines
25 KiB
Markdown

# Production Readiness Notes
This note tracks the production-facing safeguards that must stay true for Drobodata deployments.
## Runtime Boundaries
- Production execution uses the Go worker and Python operator-runner Jobs. The API does not execute workflow nodes in-process.
- The API proxies `/worker-api/*` through `WORKER_API_BASE_URL`; the default is the in-cluster worker Service.
- Worker proxy health, stats, metrics, and retry calls use short upstream timeouts. Manual `process-once` uses a long timeout because it can execute real queued work.
- Explicit shell operators are disabled by default through `OPERATOR_ALLOW_SHELL_COMMANDS=false`; allowlist and denylist regexes are platform-controlled and cannot be overridden by workflow hyperparams.
- Environment-specific values such as `K8S_ENDPOINT`, registry host/namespace, registry username, deploy image references, and workspace BOS bucket have no private repository defaults; provide them from local deploy secrets or CI secrets.
- Manifest rendering, image build, and deployment wrappers reject invalid Kubernetes namespace/Secret/PVC/Service names, non-absolute workspace mount paths, malformed worker/AI/BOS service URLs, and malformed image references before emitting manifests, building images, or calling `kubectl`. Multi-line ConfigMap block values are indented before manifest output so formatted GPU tolerations JSON and multi-line shell command policy regexes remain valid YAML.
- Local development and acceptance endpoints are resolved through shared helpers instead of per-script localhost literals. Vite dev proxy targets can be overridden with `VITE_DEV_API_BASE_URL` and `VITE_DEV_WORKER_BASE_URL`; browser-oriented acceptance scripts accept `BASE_URL` or `API_BASE_URL`, while reconcile/deploy wrappers keep `API_BASE_URL` as the primary operational variable. Explicit endpoint values must be absolute `http://` or `https://` URLs and fail before Vite, curl, Playwright, or urllib requests are attempted. Browser-side `VITE_API_BASE_URL` follows the same absolute http(s) rule, with the empty value reserved for same-origin API calls. Deployed workflow acceptance should set `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` so an omitted endpoint cannot fall back to localhost.
- Data reconciliation uses real API calls and supports protected deployments through `DROBOTICFLOW_API_TOKEN` or session login credentials plus CSRF headers. Public session-auth environments must use an organization admin account for reconcile because it updates settings and management resources.
- Session-auth management surfaces are admin-only on both read and write paths. `/api/users`, `/api/settings`, database browser routes, object-storage management routes, Mongo connection tests, and external business-DB token management require an organization admin; session-auth user creation and password updates require a non-blank password so new accounts are immediately login-capable and blank password updates cannot appear successful. User names cannot be blank, and user status accepts only `active` or `disabled`. Project mutations reject blank names and unknown statuses; project status accepts only `active` or `archived` so dashboard counters and project-list filters cannot drift. Workflow and template mutations reject blank names and unknown metadata enum values so UI filters, worker dispatch, and stored records stay aligned. Operator mutations reject blank metadata, unknown operator/port enum values, duplicate port IDs, and invalid resource defaults so the operator library cannot persist records the workflow editor or Kubernetes launcher cannot interpret. Reusable storage/Mongo connection list/get routes and workflow storage browse helpers require admin/operator role, not auditor access. `/api/public/settings` exposes only the runtime timezone so non-admin pages can render timestamps without receiving K8s, AI, Mongo, or token metadata. Users update their own sidebar profile through `PUT /api/users/me`, which accepts only `name` and `email`; user storage-path favorites are self-or-admin scoped. Model-backed AI generation routes require admin/operator role rather than auditor access.
- Python live-acceptance and WJZH regression scripts use the shared `DroboticflowClient` helper for API token headers, session-login cookies, CSRF propagation, JSON calls, and binary object-storage calls. WJZH quality thresholds are caller-supplied through `QUALITY_MIN_CLARITY`, `QUALITY_MAX_SHAKE`, and `QUALITY_MAX_HAND_OCCLUSION`; window, slicing, and annotation timing are caller-supplied through the documented `*_SECONDS` values; sampling thresholds are caller-supplied through `SAMPLING_RULES_JSON`. The scripts do not carry built-in audit rules. New Python acceptance scripts should reuse that helper instead of carrying their own auth/request stack.
- Python WJZH and X-platform acceptance scripts validate quality thresholds through the shared normalized-threshold gate. `QUALITY_MIN_CLARITY`, `QUALITY_MAX_SHAKE`, `QUALITY_MAX_HAND_OCCLUSION`, `XPLATFORM_MIN_CLARITY`, `XPLATFORM_MAX_SHAKE`, and `XPLATFORM_MAX_HAND_OCCLUSION` must be finite values in `[0, 1]`; invalid values fail before live workflows are created.
- Python WJZH and X-platform acceptance scripts also share the same sampling-rule validator. `SAMPLING_RULES_JSON` and `XPLATFORM_RULES_JSON` must be non-empty rule arrays with unique `id` values, supported operators, required `field`/`value` fields, and finite numeric `samplePercent` values in `(0, 100]`; boolean `samplePercent` values are rejected before creating live workflows instead of being coerced to `1` or `0`.
- Python live-acceptance positive integer gates are exact and safe. WJZH timing values, X-platform `XPLATFORM_SEGMENT_DURATION_SECONDS`, every `XPLATFORM_MIN_*` count, and wait-time knobs reject signs, leading zeros, decimals, units, unsafe integers, and wait values below 60 seconds instead of coercing or clamping them.
- Node/Playwright acceptance scripts use the shared `createAcceptanceAPIClient` helper for bearer token headers, Playwright context CSRF propagation, JSON calls, retries, exact integer and boolean env parsing, and optional session login. Functional and operator-preview checks also pass `DROBOTICFLOW_API_TOKEN` through browser contexts so protected UI requests exercise the same token-authenticated path as API setup calls.
- Browser API calls use a shared URL builder and request-header helper. JSON calls, object-storage text previews, and multipart object uploads all send `credentials: include`; unsafe methods, including logout, attach the double-submit CSRF token, while FormData uploads avoid forcing a JSON `Content-Type`.
- Object-storage text preview metadata accepts byte-count response headers only as exact safe non-negative integers. Corrupt values such as decimals, units, signs, leading-zero strings, or unsafe integers are ignored instead of being truncated into plausible sizes.
- Data-asset aggregation treats local `artifact.data.dataRef.payloadPath` values as authoritative external payloads. If the referenced local payload is unreadable, malformed JSON, or not a JSON object, source collections and package previews omit it instead of falling back to stale inline data. Data-asset summary integers such as counts, byte sizes, and frame counts must be exact integer values; lossy floats, `NaN`, invalid `json.Number`, and overflowing unsigned values are ignored rather than truncated.
- Incremental scans require scan-workflow `DATA_SOURCE` to be `BOS`, `SCAN_MODE` to be either `FULL` or `INCREMENTAL`, `INCREMENTAL_UNIT` to be either `OBJECT` or `DIRECTORY`, and `INCREMENTAL_DIRECTORY_DEPTH` to be a positive integer. The workflow editor, API, worker, and operator-runner all reject lossy or invalid values instead of truncating or silently falling back; invalid source-node hyperparams are rejected before workflow creation/update and fail the worker run as a normal node error if stale persisted data reaches execution.
- Incremental source checkpoints are committed only from authoritative, readable checkpoint payloads. If `source-checkpoint.json` is unreadable, malformed, empty, missing `sourceFingerprint`, or contains lossy `directoryDepth` / `listedCount` / `newCount` values, worker checkpoint commit fails instead of falling back to stale inline artifact data or truncating counts.
- Video quality and data-filter runtime fallbacks match the builtin operator catalog defaults: `MIN_CLARITY=0.8`, `MAX_SHAKE=0.1`, and `MAX_HAND_OCCLUSION=0.1`. Missing node hyperparams must not silently widen the filter into `0.0/1.0/1.0`, and explicit threshold hyperparams must be finite numeric `[0, 1]` values or the node fails.
- Video quality, slicing, and annotation runtime defaults match the builtin operator catalog for `CLARITY_SCALE=35`, `SEGMENT_DURATION_SECONDS=300`, and `FRAME_INTERVAL_SECONDS=1.0`. Explicit timing and scale overrides must be finite positive values; `SEGMENT_DURATION_SECONDS` is an exact safe positive integer and rejects signs, leading zeros, decimals, units, and unsafe integers, so bad workflow hyperparams fail instead of being silently clamped.
- Builtin operator hyperparams fail fast when callers provide malformed structured values. `EXPECTED_JSON`, `FIELD_MAP_JSON`, and `RULES_JSON` must be valid JSON; field maps must be JSON objects with non-empty string source/destination fields and no output-field collisions; sampling rules must be JSON arrays whose entries include `id`, `field`, `operator`, `value`, and finite numeric `(0, 100]` `samplePercent` values; comparison operators must be supported; source `DATA_SOURCE` values must be `INLINE`, `LOCAL`, `BOS`, or `UPSTREAM`; boolean `samplePercent` values are not numeric percentages; boolean hyperparams such as `REQUIRE_INTRINSICS` accept only `true/false`, `1/0`, `yes/no`, or `on/off` spellings.
- Workflow schedule and execution-guard numeric fields are integer-gated in the editor before save. Schedule intervals must be positive while active, execution guard values such as concurrency, timeouts, retry count, and retry backoff must be non-negative, and lossy values like `60s`, `1.5`, or `2days` are rejected instead of being coerced into defaults.
- System settings worker numeric fields are positive-integer gated in the Settings UI, the shared settings payload builder, and `PUT /api/settings`. Worker poll interval and default max concurrency reject blank strings, zero, decimals, units, unsafe integers, and other lossy values instead of ignoring invalid explicit updates. Default CPU/MEM settings are also validated server-side as positive Kubernetes resource quantities, and default GPU must be a positive integer when present.
- Database document browsing validates explicit `limit` and `skip` query values in both the UI and API before it reaches platform or business Mongo stores. `limit` must be a positive integer, `skip` must be a non-negative integer, and oversized values are capped to the documented bounds.
- Explicit startup environment values fail fast when malformed. `MONGO_URI` is required for API/worker startup and must be a concrete `mongodb://` or `mongodb+srv://` URI without whitespace or unreplaced placeholder markers; there is no hidden localhost runtime fallback. API/worker ports must be valid TCP ports, Mongo retry count and delay must be positive integers, worker polling must be positive, boolean auth flags must use supported boolean spellings, and manifest/deploy preflight rejects invalid replicas, NodePort, and Job TTL values before Kubernetes dry-run or cluster writes. String startup values, including worker, AI, BOS, dist, token, proxy, GPU, and workspace settings, are trimmed before use; whitespace-only optional values are treated as unset rather than silently entering runtime config. AI generation has no hardcoded provider/model fallback: without `AI_API_KEY` it stays disabled, and with `AI_API_KEY` both `AI_API_BASE_URL` and `AI_MODEL` must be explicit. The API command passes the validated config into the HTTP handler, so worker proxy routing and the optional global BOS endpoint are bound at startup and embedded/test handler construction reports invalid config through a controlled error response instead of a process panic. API and worker use the validated `BOS_ENDPOINT` startup value only when a scan config or BOS storage connection omits endpoint; if both are empty, real BOS operations fail with an explicit missing-endpoint error rather than using a hardcoded region.
## Kubernetes Jobs
- Operator-runner Jobs receive CPU and memory requests/limits from system defaults and node overrides. Explicit CPU and memory values must be valid positive Kubernetes resource quantities; malformed values fail before Job creation instead of being silently omitted.
- A positive GPU value emits `nvidia.com/gpu` in both requests and limits.
- GPU node selector, tolerations, and runtime class are only injected for Jobs that request GPU.
- Finished Jobs receive `ttlSecondsAfterFinished` from `OPERATOR_JOB_TTL_SECONDS`, default `86400`.
- Use `DRY_RUN=true npm run k8s:cleanup-jobs` before deleting old finished Jobs that predate TTL.
## Data Plane Limits
- Object-storage JSON object creation is capped at 4 MiB.
- Browser multipart uploads stream to BOS and are capped at 2 GiB.
- Directory uploads keep browser `webkitRelativePath` values, prepare at most 5000 files/placeholders per batch, and upload four files concurrently through the same presigned direct/multipart path. Conflict policy is explicit (`overwrite`, `skip`, or stop with `error`), failed files can be retried independently, and explicitly listed empty directories are preserved as `.keep` objects because browser directory inputs do not expose truly empty directories.
- Large inline object reads use preview or presigned URL behavior instead of loading full objects into API memory.
- Browser video previews are lazy-loaded in monitoring, artifact preview, and object-storage browsing surfaces. The UI does not attach the `<video>` source until the user starts the preview, so large MP4 artifacts do not trigger Range requests and decoder work during initial page render or artifact switching.
- Run logs, resource samples, and artifacts support strict `limit` and `offset` / `skip` pagination. Empty values use defaults, explicit `limit` must be positive, explicit offsets must be non-negative, and invalid values return validation errors instead of silently falling back to default pages. Logs and resources default to the latest 1000 records for large runs.
- Data-asset aggregation uses artifact-focused run projections so list endpoints do not read complete logs and execution traces.
## GPU Observability
- Operator-runner resource samples include CPU seconds, process memory, GPU utilization, GPU memory used/total, GPU count, and GPU model when `nvidia-smi` is available.
- The monitoring page displays GPU utilization, GPU memory, and GPU device information.
- `GET /worker-api/capacity` exposes the worker's current Kubernetes Node allocatable GPU snapshot, including matched nodes, schedulable allocatable count, max single-node GPU count, and `available`. The Settings page displays this snapshot, workflow node GPU inputs surface a warning when the current cluster has no schedulable GPU, and the workflow run action blocks GPU-requesting workflows when the loaded worker capacity proves they cannot be scheduled.
- `npm run acceptance:gpu` loads the deployment environment resolver first, so `dev` runs against POC by default when `DEPLOY_ENV` is unset. It remains the real acceptance check for allocatable GPU resources, scheduling, runtime, and `nvidia-smi`. It rejects invalid namespace, image, timeout, GPU count, selector, tolerations, resource-name, Job-name, and runtime-class inputs before it calls `kubectl`, verifies that at least one selector-matched and non-unschedulable node has the requested allocatable GPU count before creating a Job, and quotes dynamic manifest values before rendering the temporary Job. A cluster with no matching allocatable GPU resources is GPU-not-accepted, not a skipped success.
- `npm run acceptance:poc-gpu-capacity` validates the deployed POC worker capacity endpoint from inside the cluster with the currently configured operator-runner image. The default accepted result is CPU-only capacity (`gpu.available=false`); set `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true` after POC has a real GPU node pool.
- `npm run acceptance:poc-readiness` includes deployed worker GPU capacity evidence through a default `kubectl port-forward` probe and verifies the platform `gpuWorkflowPreflight` gate through `GET /api/acceptance/poc-readiness`, so every POC readiness report states both whether the current hardware expectation is CPU-only or GPU-available and whether the latest real GPU-requesting workflow matches that expectation. The API check uses `DROBOTICFLOW_API_TOKEN` from the environment or `droboticflow-auth-secret`; set `POC_PLATFORM_READINESS_ENABLED=false` only for isolated script tests that intentionally bypass platform evidence.
- `npm run acceptance:poc-gpu-workflow` verifies the worker GPU path through a real POC workflow run. It uses a session-authenticated acceptance user when provided; otherwise it uses `DROBOTICFLOW_API_TOKEN` from the environment or `droboticflow-auth-secret` only for the bounded POC GPU route plus run reads, then triggers the deployed worker through a local port-forward. It expects `gpu request cannot be scheduled` while POC is CPU-only; with `POC_GPU_WORKFLOW_EXPECT_AVAILABLE=true` or `POC_GPU_CAPACITY_EXPECT_AVAILABLE=true`, it expects a completed run with artifacts. The latest proof is exposed in POC readiness as `gpuWorkflowPreflight`.
## Acceptance
Run local verification before merging:
```bash
npm run verify:local
npm run verify:security
```
The local gate includes shell syntax checks for operational scripts, so GPU acceptance, deployment, cleanup, and reconciliation wrappers are parsed before a change is accepted. The security gate runs `npm audit --audit-level=moderate` through the retry wrapper; `npm run verify` runs both gates in order.
Run deployed live workflow acceptance after rollout:
```bash
WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true \
WORKFLOW_ACCEPTANCE_SOURCE_PATH=<path-visible-to-operator-runner> \
./scripts/live-workflow-acceptance.sh http://<node-ip>:30080
```
The workflow acceptance source path may be local/PVC-backed or `bos://bucket/prefix`. The source node now fails fast when no real input source is configured, so a misconfigured workflow cannot pass by producing an empty source manifest. `WORKFLOW_ACCEPTANCE_REQUIRE_EXPLICIT_ENDPOINT=true` keeps deployed acceptance from silently using local defaults.
Run browser object-storage acceptance after rollout:
```bash
BASE_URL=http://<node-ip>:30080 \
OBJECT_STORAGE_CONNECTION_ID=<storage-connection-id> \
OBJECT_STORAGE_BUCKET=<bucket> \
OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE=/path/to/real-object-payload.json \
npm run acceptance:object-storage
```
`OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` is required and must point to a non-empty real UTF-8 text or JSON payload. The script requires an explicit storage connection selector, uses the supplied payload for create/upload/preview checks, and will not modify connection metadata or fall back to a built-in object body to make the check pass. Set `OBJECT_STORAGE_ACCEPTANCE_CONTENT_TYPE` when the file extension is ambiguous. Set `OBJECT_STORAGE_VIDEO_KEY` to validate browser playback for a known existing video object.
Run operator preview acceptance against supplied payloads and real source paths:
```bash
BASE_URL=http://<node-ip>:30080 \
PREVIEW_CHECK_CSV_PAYLOAD_FILE=/path/to/real-table.csv \
PREVIEW_CHECK_TEXT_PAYLOAD_FILE=/path/to/real-notes.txt \
PREVIEW_CHECK_JSON_PAYLOAD_FILE=/path/to/real-data-ops.json \
PREVIEW_CHECK_JSON_EXTRACT_FIELD=<field.path> \
PREVIEW_CHECK_JSON_FILTER_FIELD=<items-field> \
PREVIEW_CHECK_JSON_FIELD_MAP_JSON='{"sourceField":"renamedField"}' \
PREVIEW_CHECK_SCAN_SOURCE_PATH=/workspace/datasets/<real-directory> \
PREVIEW_CHECK_ROSBAG_SOURCE_PATH=/workspace/datasets/<real-rosbag-path> \
PREVIEW_CHECK_LEROBOT_SOURCE_PATH=/workspace/datasets/<real-lerobot-root> \
npm run acceptance:operators-preview
```
The operator preview script has no built-in CSV/text/JSON/ROSBag/LeRobot dataset fallback. Payload files are caller supplied; scan, ROSBag, and LeRobot paths must be readable from operator-runner Jobs. Set optional `PREVIEW_CHECK_JSON_FILTER_EXPECTED_JSON`, `PREVIEW_CHECK_SCAN_MIN_FILES`, `PREVIEW_CHECK_ROSBAG_TOPIC`, and `PREVIEW_CHECK_LEROBOT_ROBOT_TYPE` when the target dataset needs stricter assertions. `PREVIEW_CHECK_JSON_FIELD_MAP_JSON` uses the same strict field-map rules as the runtime operator, and `PREVIEW_CHECK_SCAN_MIN_FILES` must be an exact safe positive integer.
Run X-platform operator-chain acceptance against a real dataset:
```bash
BASE_URL=http://<node-ip>:30080 \
XPLATFORM_SOURCE_PATH=bos://<bucket>/<input-prefix> \
XPLATFORM_STORAGE_CONNECTION_ID=<storage-connection-id> \
XPLATFORM_MIN_CLARITY=<threshold> \
XPLATFORM_MAX_SHAKE=<threshold> \
XPLATFORM_MAX_HAND_OCCLUSION=<threshold> \
XPLATFORM_CLARITY_SCALE=<scale> \
XPLATFORM_SEGMENT_DURATION_SECONDS=<seconds> \
XPLATFORM_FRAME_INTERVAL_SECONDS=<seconds> \
XPLATFORM_RULES_JSON='<sampling-rules-json-array>' \
XPLATFORM_MIN_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VALID_COLLECTION_COUNT=<count> \
XPLATFORM_MIN_VIDEO_COUNT=<count> \
XPLATFORM_MIN_FILTERED_COUNT=<count> \
XPLATFORM_MIN_CLIP_COUNT=<count> \
XPLATFORM_MIN_ANNOTATION_COUNT=<count> \
XPLATFORM_MIN_PACKAGE_COUNT=<count> \
XPLATFORM_MIN_SAMPLED_COUNT=<count> \
npm run acceptance:xplatform
```
The script has no synthetic dataset fallback and no built-in acceptance criteria. For `bos://` input or upload paths, provide an existing storage connection selector. Set `XPLATFORM_RULES_JSON`, every `XPLATFORM_MIN_*` count, quality thresholds, and slice/annotation timing explicitly to match the target dataset before treating the run as production acceptance evidence. The X-platform quality thresholds must be finite normalized `[0, 1]` values, `XPLATFORM_CLARITY_SCALE` and `XPLATFORM_FRAME_INTERVAL_SECONDS` must be finite positive numbers, and `XPLATFORM_SEGMENT_DURATION_SECONDS` plus every `XPLATFORM_MIN_*` count must be exact safe positive integers. Python acceptance and reconciliation scripts use the shared protected API client for token/session auth, CSRF propagation, JSON requests, and optional transient-error retries.
Reconcile persistent system data after rollout:
```bash
K8S_ENDPOINT=https://<api-server>:6443 ./scripts/reconcile-system-data.sh http://<node-ip>:30080
```
The reconcile wrapper delegates to `scripts/reconcile_system_data.py`. It removes legacy demo records, updates cluster settings through the real API, and derives builtin operators from the live `/api/operators` catalog before refreshing each builtin to the digest-pinned image declared in `operator-repos/builtin/<slug>/release.json` for the target environment. Registry digest lookup is used only as a guardrail: a mismatch between the committed release manifest and the live registry digest blocks reconciliation instead of becoming an implicit production truth.
Builtin operator source lives in `operator-repos/builtin/<slug>/operator.json` and the paired `release.json`. `internal/api/builtin_operators_catalog.json` is generated from those source and release files before being embedded by the API. Startup validation rejects malformed or duplicate generated catalog entries before they can drift into the database, and documentation generation defaults to the local generated catalog; live API export requires an explicit `--api-base-url`.
Run GPU acceptance on a real GPU cluster:
```bash
npm run acceptance:gpu
```
Run the POC workflow-level GPU preflight check with a real acceptance user:
```bash
DROBOTICFLOW_ACCEPTANCE_EMAIL=<operator@example.com> \
DROBOTICFLOW_ACCEPTANCE_PASSWORD=<password> \
npm run acceptance:poc-gpu-workflow
```
Live workflow acceptance, GPU acceptance, object-storage acceptance, operator preview acceptance, and X-platform real-data acceptance can run against a deployed environment from a local shell or external CI. Provide `WORKFLOW_ACCEPTANCE_SOURCE_PATH` for workflow acceptance, configure a kubeconfig when GPU acceptance targets a non-default environment, and provide explicit live data paths plus storage connection selectors when storage-backed acceptance is enabled. For object-storage acceptance, provide `OBJECT_STORAGE_ACCEPTANCE_PAYLOAD_FILE` as a non-empty real UTF-8 payload file before invoking the browser check. For operator preview acceptance, provide `PREVIEW_CHECK_CSV_PAYLOAD_FILE`, `PREVIEW_CHECK_TEXT_PAYLOAD_FILE`, `PREVIEW_CHECK_JSON_PAYLOAD_FILE`, the JSON field settings, and the real scan/ROSBag/LeRobot source paths. For X-platform acceptance, provide the real source path plus `XPLATFORM_RULES_JSON`, all `XPLATFORM_MIN_*` counts, quality thresholds, and slice/annotation timing through environment variables or external CI secrets.
## Remaining Cluster Responsibilities
Drobodata now exposes the necessary application controls, but production readiness still depends on the target cluster:
- TLS ingress or trusted private network exposure
- network policies around MongoDB, API, worker, and operator-runner Jobs
- MongoDB backup, restore, and credential rotation
- registry credential rotation
- Prometheus scraping and alerting for worker metrics and failed Jobs