test: fix hostile-decoy env race flaking the setup-matrix and e2e suites#135
Open
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
Open
test: fix hostile-decoy env race flaking the setup-matrix and e2e suites#135Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
Conversation
CI flake (seen twice: setup_matrix_pypi
host_guard::poetry_check_recognizes_structural_hook_host, exit 2 with a
clap --help hint): several test binaries set hostile ambient env decoys
PROCESS-WIDE (std::env::set_var) to prove the child-env scrub in their
run helpers works, while sibling tests run on parallel threads. The
scrub snapshots std::env::vars_os() and then spawns; a set_var landing
between a sibling's snapshot and its spawn reaches the child un-scrubbed
and (for SOCKET_STRICT=banana etc.) aborts its arg parse with exit 2.
Mechanism proven RED/GREEN by making the schedule deterministic: a 10ms
delay before the decoy set (simulating the late thread start CI hits)
plus a 25ms window between snapshot and spawn fails 2/3 host_guard
tests on EVERY run without the fix, and 0/5 runs with it (probes not
committed).
Fix, applied to every binary that mutates process env with >1 test
(setup_matrix_{pypi,npm,deno,maven,nuget}, e2e_safety_cow,
e2e_vendor_pypi_build):
- #[serial_test::serial] on every test in the binary — serial only
excludes other #[serial] tests, so annotating just the mutator would
fix nothing
- smc::DecoyGuard, an RAII setter that removes the decoys on drop, so a
panicking assertion can't leave the process env poisoned for the tests
that run after it (npm and nuget previously never removed their decoys
at all)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flake
Seen twice in CI (both times on
setup_matrix_pypi'shost_guard::poetry_check_recognizes_structural_hook_host):setup --checkexits 2 with a clap--helphint instead of 0/1. Passed on rerun with zero code change.Root cause
Several test binaries deliberately set hostile ambient env decoys process-wide (
std::env::set_var("SOCKET_STRICT", "banana"), …) to prove theirrun()helpers' child-env scrub works — while their sibling tests run on parallel threads in the same process. The scrub snapshotsstd::env::vars_os()and then spawns; aset_varlanding between a sibling's snapshot and its spawn reaches the child un-scrubbed, andSOCKET_STRICT=bananaaborts every CLI arg parse with exit 2. The in-code comment claiming the decoys are "safe to set process-wide because every other test scrubs" was exactly the racy assumption.Mechanism proven RED/GREEN
Made the CI schedule deterministic locally: a 10ms delay before the decoy set (simulating the late thread start CI runners hit) plus a 25ms window between scrub snapshot and spawn → 2/3 host_guard tests fail on every run with the exact CI assertion; with the fix in place and the same probes, 5/5 runs green. (Probes not committed.)
The fix
Applied to every test binary that mutates process env and has parallel siblings (
setup_matrix_{pypi,npm,deno,maven,nuget},e2e_safety_cow,e2e_vendor_pypi_build— found by repo-wide sweep):#[serial_test::serial]on every test in those binaries — serial only excludes other#[serial]tests, so annotating just the mutator fixes nothing.smc::DecoyGuard: RAII decoy setter that removes the pairs on drop, so a panicking assertion can't leave the process env poisoned for later tests. (npm and nuget previously never removed their decoys at all.)Single-test binaries with
set_var(setup_matrix_env_guard,e2e_golang_build) have no intra-process race and are untouched.Validation
All touched binaries run locally: pypi/deno/maven/nuget/safety_cow/vendor_pypi_build green (docker-gated cases skip/fail identically on an untouched checkout — pre-existing local-env, e.g. the known BASELINE GAP cases). The
test-releasererun on #133 that motivated this went green with no code change, consistent with the diagnosis.🤖 Generated with Claude Code