Node4 synthetic isolation smoke test
This is a diagnostic workload, not a deployment of the query application or a connection to production Mongo. Company-cluster execution requires explicit authorization for the synthetic node test. Rendering a plan is read-only.
The original retained-mount implementation FAILED across Pod lifetimes. The corrected initializer normally unmounts/rebinds only the exact-owned, idle synthetic filesystem before admitting a new Pod; it never reformats retained data or force-unmounts. The small recovery suite has since passed. Historical failures, contrasting inode measurements and current results are recorded in the node report.
python3 scripts/local/node4-query-pilot/pilot.py \
--kubeconfig <local-kubeconfig> --output <private-evidence-directory>
Review the rendered objects and SHA256 before supplying --execute --confirm-plan <sha256>. This runner is fixed to the verified node and production namespace identities, an already cached immutable Mongo image, a 512Mi backing file and a new dedicated namespace. It creates no Service/PVC and exposes no port. The Mongo process binds loopback and uses only node4_synthetic_pilot.
The privileged initializer sets limits on the exact Pod UID's Guaranteed cgroup before creating or mounting the backing file. It does not change root or other Pod limits. Retaining the old mounted filesystem can retain inode writeback ownership from a deleted cgroup; merely setting limits on the new Pod does not repair it. Each bootstrap verifies the backing inode, normally unmounts/rebinds only that filesystem and sets both loop/backing-device limits. EBUSY fails closed. The running probe has UID999, no capabilities, a read-only container root and read-only host cgroup access. It rejects missing mounts or limits instead of writing to an unbounded fallback directory.
The test performs direct/buffered/fsync IO probes, journals a synthetic marker, deletes and recreates only its own Pod, checks the new UID/limits and preserved marker, then exhausts its own filesystem with a filler file. Recovery removes only the filler/reserve files, preserves database and journal files, restarts Mongo and verifies the previously acknowledged marker plus a new journaled write. A filesystem ENOSPC alone does not prove that Mongo rejected a write: inspect the captured MONGO_FULL_WRITE_EXIT and output.
During the test, read-only monitoring samples the existing API readiness endpoint, production Pod UID/restart counts, node pressure, free disk, disk counters and IO PSI. It protects existing workloads using their individual cgroup pressure plus physical-device latency; global PSI includes the intentionally throttled probe and is retained as a separate observation. Sustained readiness/existing-workload IO degradation or a production Pod change stops the synthetic Pod. A short readiness sample cannot establish complete business latency or long-term capacity. It is not a production write acceptance test.
Creation errors and timeouts are not retried automatically. Resume requires inspection of recorded state, an empty test namespace, an exact namespace UID, unchanged scripts/network policy/quota and a freshly confirmed plan. The runner does not overwrite existing objects. Preserve admission failures and partial results.
The normal smoke run stops only the test Pod on exit and retains the backing file/mount and namespace for review. It does not delete host directories, unmount arbitrary loop devices or delete Mongo files.
The explicit --query25 --acceptance-file <small-result.json> mode requires a passing small recovery result, binds its hash into a new plan, and uses a different25Gi backing file and ConfigMap. Rendering is still offline; execution still requires a matching plan and user authorization. This mode creates only a loopback synthetic single-member replica set, verifies a majority+journaled marker through another Pod replacement, and retains a successful test Pod. It uses1CPU/1Gi, dual IO limits, a continuous75Gi host-free-space guard and a six-hour Pod deadline. Data remains after the deadline; it is not production HA, a production sync consumer or a capacity acceptance result. There is no public Service or production credential.
Local safety checks:
python3 -m unittest discover -s scripts/local/node4-query-pilot -p 'test_*.py' -v