6.5 KiB

Node4 synthetic isolation smoke test

This is a diagnostic workload, not a deployment of the query application or a connection to production Mongo. Company-cluster execution requires explicit authorization for the synthetic node test. Rendering a plan is read-only.

The original retained-mount implementation FAILED across Pod lifetimes. The corrected initializer normally unmounts/rebinds only the exact-owned, idle synthetic filesystem before admitting a new Pod; it never reformats retained data or force-unmounts. The small recovery suite has since passed. Historical failures, contrasting inode measurements and current results are recorded in the node report.

python3 scripts/local/node4-query-pilot/pilot.py \
  --kubeconfig <local-kubeconfig> --output <private-evidence-directory>

Review the rendered objects and SHA256 before supplying --execute --confirm-plan <sha256>. This runner is fixed to the verified node and production namespace identities, an already cached immutable Mongo image, a 512Mi backing file and a new dedicated namespace. It creates no Service/PVC and exposes no port. The Mongo process binds loopback and uses only node4_synthetic_pilot.

The privileged initializer sets limits on the exact Pod UID's Guaranteed cgroup before creating or mounting the backing file. It does not change root or other Pod limits. Retaining the old mounted filesystem can retain inode writeback ownership from a deleted cgroup; merely setting limits on the new Pod does not repair it. Each bootstrap verifies the backing inode, normally unmounts/rebinds only that filesystem and sets both loop/backing-device limits. EBUSY fails closed. The running probe has UID999, no capabilities, a read-only container root and read-only host cgroup access. It rejects missing mounts or limits instead of writing to an unbounded fallback directory.

The test performs direct/buffered/fsync IO probes, journals a synthetic marker, deletes and recreates only its own Pod, checks the new UID/limits and preserved marker, then exhausts its own filesystem with a filler file. Recovery removes only the filler/reserve files, preserves database and journal files, restarts Mongo and verifies the previously acknowledged marker plus a new journaled write. A filesystem ENOSPC alone does not prove that Mongo rejected a write: inspect the captured MONGO_FULL_WRITE_EXIT and output.

During the test, read-only monitoring samples the existing API readiness endpoint, production Pod UID/restart counts, node pressure, free disk, disk counters and IO PSI. It protects existing workloads using their individual cgroup pressure plus physical-device latency; global PSI includes the intentionally throttled probe and is retained as a separate observation. Sustained readiness/existing-workload IO degradation or a production Pod change stops the synthetic Pod. A short readiness sample cannot establish complete business latency or long-term capacity. It is not a production write acceptance test.

Creation errors and timeouts are not retried automatically. Resume requires inspection of recorded state, an empty test namespace, an exact namespace UID, unchanged scripts/network policy/quota and a freshly confirmed plan. The runner does not overwrite existing objects. Preserve admission failures and partial results.

The normal smoke run stops only the test Pod on exit and retains the backing file/mount and namespace for review. It does not delete host directories, unmount arbitrary loop devices or delete Mongo files.

The explicit --query25 --acceptance-file <small-result.json> mode requires a passing small recovery result, binds its hash into a new plan, and uses a different25Gi backing file and ConfigMap. Rendering is still offline; execution still requires a matching plan and user authorization. This mode creates only a loopback synthetic single-member replica set, verifies a majority+journaled marker through another Pod replacement, and retains a successful test Pod. It uses1CPU/1Gi, dual IO limits, a continuous75Gi host-free-space guard and a six-hour Pod deadline. Data remains after the deadline; it is not production HA, a production sync consumer or a capacity acceptance result. There is no public Service or production credential.

Local safety checks:

python3 -m unittest discover -s scripts/local/node4-query-pilot -p 'test_*.py' -v

Authorized query25 resource stages

For an existing query25 filesystem, use scripts/local/query-lab-resource-stage.py, not the initial namespace provisioning flow. The authoritative finite profiles are in scripts/local/query_lab_profiles.py: baseline, compute, balanced and read-headroom. Rendering updates the Guaranteed Pod resources, quota, WiredTiger cache, initializer and runtime checks for both loop and backing block devices together. The verifier checks the running WiredTiger cache as well as the actual cgroup caps. No profile changes disk capacity or durability.

The resource-stage runner first saves exact original objects, desired objects and rollback instructions in a new evidence directory. --execute is allowed only within separately authorized lab changes. It requires the exact current query Pod UID and an otherwise empty test namespace; it never replaces a concurrent fixture. It uses UID deletion and resourceVersion/value preconditions for ConfigMap/quota updates. A recreated query Pod retains its data and has a 3600-second deadline; ordinary bootstrap provisions the older six-hour pilot. The runtime host-free-space floor remains75Gi. The original smoke profiles and IO probe thresholds remain unchanged.

Example (the current UID must come from a fresh read-only preflight):

python3 scripts/local/query-lab-resource-stage.py \
  --kubeconfig <local-kubeconfig> --output <new-private-evidence-directory> \
  --profile compute --expected-pod-uid <current-query25-uid> --execute

Before any resource change, check node/production health and current occupation. Before increasing the next stage, UID-clean the owned source fixture/policies and PID/executable-check the bridge, restoring its recorded query-only quota. Use the comparison monitor throughout synthetic load. On protection failure, stop load first and restore the original stage using --restore <original.json> with the current UID and a new output directory. Restoration is not blocked by an unhealthy readiness check; it only restores saved lab objects. A CAS conflict requires inspection, never a force overwrite. Neither rebuild nor rollback reformats data or performs a global cache drop.