fix(replication): drain priority sync queue and recover from routing-event lag

[mickvandijke]

Summary

This PR hardens replication repair under churn, load, queue pressure, event-stream failure, and shutdown. It expands the original neighbor-sync drain fix into a broader set of paid-list repair, verification, bootstrap accounting, audit concurrency, fresh-offer dispatch, admission gating, and async lifecycle fixes so legitimate repair work is not silently dropped, converted into false audit failures, delayed behind stale peers, left spinning on closed broadcasts, serialized behind a saturated worker pool and dropped through broadcast lag, used to conscript out-of-range nodes into storing arbitrary keys, downloaded and stored under responsibility decisions that topology churn had already invalidated, left holding LMDB/P2P resources — including detached LMDB blocking transactions — after engine shutdown, or wedged in permanent bootstrap by a peer-removal race that orphans capacity-rejection accounting.

The branch contains 54 commits (plus one merge of main), listed oldest first:

• 124e8f8 - fix(replication): harden paid-list verification repair
• 72c1815 - fix(replication): tolerate paid-list edge churn
• 3479a75 - fix(replication): drain priority sync queue and recover from routing-event lag
• 819b4e2 - fix(replication): preserve verification retry capacity
• c61d432 - fix(replication): eliminate false audit challenge timeouts
• 4099f6c - fix(replication): preserve dequeued retry reservations
• 712b514 - chore(replication): fix audit admission clippy
• 0b721f9 - fix(replication): release cancelled async work
• 6cd9c83 - fix(replication): silence no-logging audit label warnings
• da79e46 - fix(replication): unblock bootstrap when rejected peer leaves
• 67cc46a - refactor(replication): prioritize source-aware hints
• 17b26ef - refactor(replication): aggregate bootstrap hint batches
• d507303 - fix(replication): aggregate verification per peer
• 61ebee8 - fix(replication): drain fresh offers through LMDB writes
• 8c298e1 - fix(replication): track detached audit work
• 09e44b4 - fix(replication): stop message handler when event streams close
• 473b288 - fix(replication): prune departed peers during DHT lag recovery
• 4ed6c45 - fix(replication): penalize rejected singleton replica hints
• 7a86d1e - fix(replication): keep fresh offers off the serial message loop
• ff58baa - fix(replication): gate replica downloads on storage responsibility
• e521ed5 - refactor(replication): admit hints through a single relevance gate
• e3cb539 - perf(replication): merge duplicate hints without rebuilding the fetch heap
• 5467dba - fix(replication): drain detached LMDB blocking ops on shutdown
• 49093a0 - fix(replication): expire orphaned capacity-rejection records to unstall bootstrap
• a3dfc9b - fix(replication): recheck storage responsibility at the point of download
• 3a7bdba - docs(adr): record replication repair hardening decisions (PR #165)
• aa68bb0 - fix(replication): address deep review findings
• 79aaebb - fix(replication): resolve all-target clippy failures
• e5ec1ac - fix(docs): avoid private rustdoc link
• 6b0e4ce - fix(storage): guard raw LMDB reads against concurrent map resize
• d061762 - fix(replication): fairly share verification capacity
• 6b4f168 - fix(replication): generalize bounded responder admission
• 2c58d94 - fix(replication): isolate fetch responders
• ddb1071 - fix(replication): isolate verification responders
• 349a1df - fix(replication): isolate neighbor sync responders
• e6b15f7 - fix(replication): drop serial queue overflow
• 5b85d1f - fix(replication): count responder load shedding
• 0f9e3b1 - feat(replication): trace audit request origins and latency
• a5f60f3 - feat(replication): log possession proof failure reasons
• 7e8a068 - fix(replication): align admission and expiry tests
• ab0a313 - fix: resolve clippy and rustdoc warnings
• 4c9c681 - fix: support no-logging builds
• 20d3f88 - docs(adr): define scalable prune backlog draining
• 0abe74a - fix(replication): drain mature prune backlogs
• 54fea24 - fix(replication): grade prune-audit size rejections as failures
• d3a1d56 - fix(replication): bound bootstrap capacity-rejection debt by first sighting
• a84c42b - fix(replication): bound paid-notify and fresh-offer admission per source
• 809f509 - fix(replication): bound per-key hint sources and charge retry reservations
• 1094543 - fix(replication): scale the paid-list flexible edge with the group
• 0a03c71 - refactor(replication): delete unused prune-pass and audit-tick wrappers
• cca744c - docs(replication): record the remove_batch cancellation window
• eeb068b - test(e2e): make the hint-label parity test discriminate
• 1632ba7 - docs(adr): record the removed per-peer cap and revised admission bounds
• d79752e - fix(replication): track the first-audit launch on the detached tracker

Problems Addressed

1. Paid-list repair could become terminal too early. Duplicate replica/paid hints were deduplicated before their actual admission outcome was known, and verified repair work could lose its retry context after transient no-holder or no-source rounds.

2. Paid-list edge peers were too strict under churn. Boundary disagreement could reject a valid majority formed by the stable core of the paid close group.

3. Neighbor sync could stall after topology bursts. Notify coalescing allowed queued priority peers to wait for later 10-20 minute periodic ticks, while lagged DHT broadcasts could hide entrants entirely or leave departed peers ahead of current neighbors, each consuming a request timeout.

4. Verification retry capacity could be stolen. Promoted verified keys stopped counting against pending capacity, allowing unrelated hints to consume the capacity required to requeue them.

5. Verification work lacked source-aware prioritisation. Duplicate hints discarded useful corroborating-source information, and large ready queues could schedule weak singleton claims ahead of better-supported repair work.

6. Bootstrap hint batches were published incrementally. Verification could race a partially admitted neighbor-sync batch and miss the complete source picture.

7. Verification fan-out was unnecessarily fragmented. Per-peer key sets were split into many requests despite already having a bounded verification cycle.

8. Bootstrap could remain blocked by departed peers. A peer that caused an admission-capacity rejection could leave while its rejection marker continued preventing drain.

9. Local audit bursts could look like honest-peer timeouts. Independent audit issuers could exceed the responder's per-source admission limit and interpret the resulting drops as remote failure.

10. Cancelled fresh offers could detach LMDB writes. Dropping an async spawn_blocking waiter does not cancel the blocking transaction, so shutdown could report drained while LMDB still owned the environment.

11. Detached responder/audit work was outside engine lifecycle tracking. Digest, subtree, byte, possession-check, and audit-launch tasks could outlive engine shutdown while retaining storage or P2P state.

12. Closed event streams could spin the replication loop. Closed Tokio broadcast receivers remain immediately ready forever; continuing after RecvError::Closed could consume a core, flood P2P warnings, and prevent the replication pipeline from shutting down cleanly.

13. Sole-source replica hints could avoid trust penalties. A definitive close-group rejection was only penalized when the advertising peer also explicitly denied possession, allowing unsupported free-replication claims to escape punishment.

14. Fresh-offer dispatch could serialize on the non-audit loop. Once all four worker permits were held, dispatch handled the next offer — an on-chain payment verification plus a multi-MiB LMDB write — inline on the serial message loop. Per-key shard locks (key[0] % 64) collapsed the close-prefix accepted-key set onto a single shard, making the inline path the steady state, backing up the 256-slot inbound queue, and dropping replication messages wholesale once the P2P event loop lagged its broadcast receiver.

15. Replica hints could conscript out-of-range nodes into storage. HintPipeline::Replica skipped the is_responsible(storage_admission_width) gate that paid keys had to pass, and a stored pipeline tag let a second replica message escalate a queued paid entry through the already_pending fast path, so a peer could name any key and force nodes ranked 10-20 for it to fetch and store it.

16. Duplicate hint merges rebuilt the whole fetch heap. Merging a re-advertised key's advertiser only touches a field the heap never orders on, but the old path took the whole heap, scanned it linearly, and re-heapified — O(n) per duplicate and O(m·n) for a batch under the global queue write lock, which a neighbor could trigger deliberately by re-hinting queued keys.

17. Shutdown could return while detached LMDB transactions were still running. Several paths race a select! on the shutdown token against futures awaiting a spawn_blocking LMDB transaction — the fetch worker's storage.put, the prune pass's storage.delete/paid_list.remove_batch, the verification worker's paid_list.insert. Dropping the losing future does not cancel the blocking closure, which keeps running with a cloned Env; per-fetch tasks were also bare-spawned outside both trackers, so a dropped in_flight set could leak Arc<LmdbStorage> past shutdown(). Reopening the same environment afterwards was undefined behavior.

18. Bootstrap drain could stall permanently on a peer-removal race. A capacity-rejection record for source P was cleared only by P's next clean admission cycle or by P's PeerRemoved cleanup — but the note sites and the removal handler run on different tokio tasks, with await points between the last "P is live" observation and the insert. A removal fully processed inside that window made its clear a no-op, and the subsequent note recorded an entry no future event could retire: check_bootstrap_drained returned false forever, audits stayed disabled (Invariant 19), and the node advertised bootstrapping: true indefinitely, drawing network-wide bootstrap-claim trust penalties — reachable deliberately by overflowing pending_verify during a victim's bootstrap and disconnecting. Adjacent liveness gap: every drain check was event-driven and a clean-cycle clear never re-checked, so a quiet node could satisfy the drain condition with nothing left to observe it.

19. Fetch decisions could go stale between promotion and download. Storage responsibility was checked once, at verification-completion time, and never again before the chunk was downloaded and stored. The fetch queue holds up to 131,072 entries and dequeues nearest-first, so a far candidate can wait unboundedly long while closer keys jump ahead, and every per-source retry reused the same stale answer — topology churn after promotion still ended in a download, a disk write, and fetch→store→prune churn, violating the contract documented in types.rs and admission.rs that responsibility is decided against live routing state at the point of download.

20. Raw LMDB reads could race concurrent map resize. all_keys() and get_raw() opened read transactions without the shared environment lock, allowing an exclusive env.resize() to change the memory map while those transactions were active.

21. One routing peer could monopolize verification capacity. The global-only 131,072-entry admission bound let one peer fill the queue and continuously consume the 8,192-key verification cycle, rejecting honest hints before source-count prioritization could help; restoring a fixed 8,192-per-peer quota would instead truncate legitimate bootstrap batches containing tens of thousands of hints.

22. Bulk replication responders could monopolize the serial message loop. Fetch responses perform large LMDB reads and uploads, while verification and neighbor-sync responses perform synchronous point-lookups and queue publication; handling them directly let one source delay unrelated protocol traffic.

23. Serial-queue backpressure could defeat its own bound. When the bounded handoff filled or closed, the fallback executed the rejected handler inline on the P2P event receiver, allowing overload to propagate into broadcast lag and wider message loss.

24. Responder load shedding and audit latency lacked actionable attribution. Admission drops, stale dequeues, logical audit issuers, remote sources, and stage timings were not available as bounded counters and structured summaries.

25. Responsive possession-proof failures lost their precise cause. Validation branches collapsed into a generic failed verdict, making malformed proofs, missing records, and commitment mismatches indistinguishable in logs even though trust behavior correctly remained the same.

26. Mature prune backlogs could not drain at scale. The prune-confirmation audit budget was counted per candidate-to-peer edge, so a close group of 7 capped a pass at roughly nine records regardless of how many mature out-of-range records existed; far-out copies (ranked beyond the width-20 admission neighbourhood) were retained indefinitely because every deletion required a remote possession round. A first attempt to raise audit throughput then string-parsed a deployed responder's oversized-batch rejection to split-and-retry in-band — brittle wire-wording coupling whose failure mode was a trust penalty against an innocent capacity-limited peer.

Changes

Paid-list verification and repair

• Allows a duplicate paid hint to proceed when the duplicate replica path was rejected, while preserving replica precedence when that path was admitted.
• Keeps locally paid repair keys alive through inconclusive, no-holder, and exhausted-source rounds by deferring and re-verifying them.
• Carries verification retry metadata through pending_verify, fetch queue, and in-flight fetch state.
• Reserves global and per-sender pending capacity for retryable verified work until completion, discard, or restoration to verification.
• Uses by-value dequeue APIs so retry reservations cannot be orphaned.
• Includes valid replica-hint senders as fallback fetch sources and allows verified paid-only repair when this node is in storage range.
• Caches responder-side storage and paid-list lookups per key and validates malformed paid-index input defensively.

Paid-list edge churn

• Adds a four-peer flexible edge for paid close groups that have reached the runtime-configured full width.
• Negative or missing edge votes do not enlarge the denominator; positive edge votes still count and expand it.
• Keeps undersized groups on strict-majority rules and preserves inconclusive outcomes when unresolved votes could still change the result.

Neighbor sync and bootstrap lifecycle

• Drains priority peers back-to-back and parks only when the durable priority queue is empty.
• Recovers from lagged DHT events by resnapshotting close neighbors, pruning queued peers that have departed, and queueing current members for priority sync.
• Clears a departed peer's outstanding capacity-rejection marker and immediately rechecks bootstrap drain.
• Updates bootstrap drain accounting when stale pending verification entries are evicted.
• Runs each bootstrap neighbor batch concurrently, admits the completed batch under one queue lock, and keeps the batch outstanding until hints and drain accounting are fully published.
• Prevents verification from selecting a partially published bootstrap batch.
• Records each capacity-rejected source’s most recent rejection time (HashMap<PeerId, Instant> instead of a bare set) and expires records only after a runtime-derived full-cycle window covering every neighbor batch, per-peer request deadlines, cooldown, and one slow-cadence interval of slack (125 minutes with defaults), forfeiting abandoned/departed-source debt consistently with peer-removal cleanup.
• Runs expiry plus a drain re-check on every verification worker tick until bootstrap drains, ahead of the pending_peer_requests early-returns: pending requests legitimately block the drain check itself but no longer block expiry, and a drain condition that became true without a triggering event is now observed within one tick.

Source-aware bounded verification

• Retains all live hint sources per key, including the subset that explicitly claimed replica possession.
• Assigns every admitted key a capacity owner independently of its evidence sources, choosing the least-loaded live advertiser.
• Allows a sender to borrow all unused space below the global 131,072-entry limit, so a sole bootstrap source can contribute tens of thousands of genuine hints without a fixed per-sender cutoff.
• On contention, computes elastic max-min allocations and reclaims a low-priority borrowed entry from an over-allocation before admitting a fairly entitled source; displaced bootstrap hints remain owed for redelivery.
• Services ready work with persistent round-robin fairness across owners, redistributing idle shares while prioritising protected retries and better-corroborated work within each owner.
• Bounds one verification cycle to 8,192 keys and caps simultaneous verification exchanges at 32.
• Aggregates each peer's keys into one verification request per cycle instead of splitting into 1,024-key fragments.
• Accepts one full-cycle incoming request and rejects oversized requests with a bounded, wire-compatible empty response.
• Removes the timing-based singleton aggregation delay; atomic bootstrap batch publication now supplies the complete source set deterministically.
• Reports bounded trust failures for a sole replica advertiser when either the close group definitively rejects the key or that advertiser explicitly reports it absent; inconclusive, paid-only, and corroborated hints remain neutral.

Fresh-offer dispatch, admission gating, and fetch-queue merges

• Removes the inline fresh-offer fallback: dispatch only claims the key, takes an admission permit, and spawns; the worker permit is awaited inside the task, so handle_fresh_offer has a single caller and no inline verification/LMDB path on the serial loop.
• Bounds outstanding admitted offers with a 16-permit admission semaphore (a 64 MiB payload ceiling) and refuses offers past the bound rather than queueing or handling them inline; a refused offer is penalized as absent by the delayed possession check.
• Replaces the key[0] % 64 shard locks with an exact per-key in-flight set behind an RAII guard, so unrelated keys never contend and concurrent duplicates collapse onto the first claimant instead of repeating its verification.
• Derives the replica/paid pipeline from live replica_hint_sources instead of storing the tag, deleting the paid→replica escalation and both demotion sites.
• Gates replica downloads on is_responsible(storage_admission_width) at both fetch sites (local paid-list fast path and post-verification path), matching pruning width; fresh PUTs keep their wider accept window.
• Routes both replica and paid hints through a single relevance gate at paid_list_close_group_size (20), so mislabelling a hint gains nothing and the two-gate rescue dance (rejected_replica rescued by admitted_paid) is removed; the admissible key set is unchanged.
• Splits FetchCandidate into FetchOrder (key + distance, the only fields the Ord impl reads) held in the heap and FetchPayload (sources + retry metadata) held in a key-indexed map that also serves as the membership index, making a duplicate-source merge an O(1) map lookup and rebuilding the heap only when a departed peer actually orphans a candidate.
• Rechecks is_responsible(storage_admission_width) on every fetch attempt inside execute_single_fetch — before spending bandwidth (per-source retries re-enter there, so they are covered) and once more before storage.put, so bytes arriving after responsibility lapsed mid-round-trip are not written. A lapsed attempt resolves as FetchResult::NoLongerResponsible, which shares Stored's terminal path (retry-slot release plus bootstrap accounting) and reports no trust event; the verification-time check remains as a cheap pre-filter that keeps never-responsible keys out of the fetch queue.
• Extracts the fetch worker's result handling into apply_fetch_result, so each FetchResult variant's queue transition is unit-testable without a live network.

Audit concurrency and observability

• Adds a shared per-target AuditChallengeCoordinator across responsible, prune-confirmation, and possession audits.
• Limits local concurrency to the deployed responder admission capacity and starts response deadlines only after local admission.
• Makes coordinator reference accounting cancellation-safe with RAII cleanup.
• Separates timeout, unreachable, and send-failure observability while retaining wire-compatible evidence semantics.
• Records responder admission drops and digest dispatch latency, with logging-feature-safe metric labels.

Bounded responder isolation and diagnostics

• Extracts global and per-source responder admission into a reusable RAII guard; cancellation and guard drops release both forms of capacity.
• Serves inbound fetch requests on three bandwidth-limited workers, and verification and neighbor-sync requests on two workers each, with flood-fair per-source admission.
• Allows only one outstanding verification or neighbor-sync batch per source while preserving concurrency across distinct peers.
• Drops expired bulk work at worker dequeue against the existing requester deadlines, avoiding stale LMDB reads and uploads.
• Drops and classifies a serial message when the bounded handoff is full or closed instead of running its handler inline on the P2P receiver.
• Counts admission drops, stale-request sheds, and serial-queue drops independently by responder or wire-message class.
• Traces audit logical issuers, responder source peers, admission decisions, stage timings, and minute-window top-origin summaries.
• Carries responsive possession-validation failures into a stable possession_failure_reason field without changing evidence, trust, or penalty semantics.

Storage resize safety

• Acquires env_lock.read() inside the tracked blocking closures for all_keys() and get_raw(), matching the other raw storage operations.
• Keeps every LMDB read transaction mutually exclusive with the unsafe exclusive env.resize() operation while preserving cancellation and shutdown tracking semantics.

Event-stream and detached-task lifecycle

• Treats closed P2P and DHT broadcast streams as terminal for the replication message loop instead of repeatedly selecting an event source that can never recover.
• Breaking the loop drops the replication sender and cascades a clean shutdown to the serial handler.
• Tracks detached fresh-offer workers and waits for started handlers to complete LMDB writes instead of cancelling their async waiters.
• Removes the best-effort timeout from the detached-task drain so shutdown cannot claim LMDB-safe completion while blocking storage work remains active.
• Expands the shared tracker to digest, subtree, and byte responders; delayed possession checks; first-audit launches; and gossip-triggered audits.
• Stops all producer loops before closing and draining the shared tracker, preventing late registration races.
• Ensures engine shutdown does not return while tracked detached work still retains LMDB or P2P state.
• Tracks LMDB blocking tasks at the storage layer: LmdbStorage and PaidList route every spawn_blocking through per-instance TaskTrackers and expose wait_idle(), so a dropped async awaiter no longer untracks a live transaction (constructor-time opens stay untracked — they cannot outlive the constructor).
• shutdown() awaits timed-out long-lived tasks after requesting abort, then awaits wait_idle() on both environments after the detached-task drain; when it returns, producer tasks have actually exited, no LMDB blocking operation is still running, and no engine-spawned task holds Arc<LmdbStorage>/Arc<PaidList>.
• Spawns per-fetch tasks (initial and retry) on the detached task tracker instead of bare tokio::spawn; the in_flight plumbing and prompt network-I/O cancellation are unchanged — only the bounded in-flight LMDB transaction is awaited.

Prune backlog draining

• Adds a fast-delete path: once a record is mature (out of width-9 for the full three-day hysteresis) and a complete self-inclusive width-20 lookup excludes self, it is deleted without a remote possession audit, after an immediate bootstrap/width-9/width-20/hysteresis/commitment revalidation. This makes the width-20 admission boundary an eventual hard storage boundary; the accepted trade-off (a distance statement is not a possession proof) is documented in ADR-0005.
• Replaces the per-candidate-to-peer-edge audit budget with independent candidate and request caps, batching each strict-close peer's keys into multi-key challenges chunked at the existing sqrt(local_stored_keys) sender limit, so a pass is bounded by actual batched requests and complete candidate proof sets rather than ~9 records.
• Selects only whole candidates whose complete proof set fits the pass request budget; no partial candidate audit is carried across passes, and a rotating cursor gives every candidate a turn.
• Treats a responder's oversized-batch Rejected as an attributable failure (logged size_reject) rather than renegotiating it in-band. The responder's max_incoming_audit_keys already carries a 2x margin, so an honestly-sized peer only rejects past a 4x close-group store spread — double the assumed ~2x — and a Rejected is never a possession signal (a peer lacking a key answers Digests with an absent marker). This removed the earlier string-parsing split-and-retry, its request-budget atomics, and the innocent-peer trust-penalty risk. No wire-format change; the deployed multi-key challenge/response representation is unchanged.

Latest Commit Details

54fea24 - grade prune-audit size rejections as failures

Drops the wire-wording split-and-retry that 0abe74a used to recover from an oversized prune-audit batch. The responder rejects a batch only when its max_incoming_audit_keys(stored_chunks) = 2 * responsible_audit_key_limit(stored_chunks) is below the batch size; because that already carries a 2x margin over the challenger's own sqrt-scaled sender size, an honestly-sized peer rejects only once the close-group store spread exceeds 4x (sqrt(cc) > 2*sqrt(rc)). Under the assumed ~2x spread — half the threshold — a size rejection is attributable misbehaviour, so a Rejected response (any reason) now grades to a distinct non-recovering PruneAuditStatus::Rejected, flows through the normal failure path (fresh-responsibility-confirmation gate, per-peer-per-pass dedup) under a size_reject metric label, and never parses the reason text. A Rejected is never a possession signal — a peer lacking a key answers Digests with ABSENT_KEY_DIGEST — and nobody gains by dodging a prune audit (it only makes the challenger retain its own copy), so the label carries no in-band recovery. This removes the string parser, the split-request-budget/request-attempt atomics, and the retry loop (net −51 lines), and eliminates the trust-penalty-against-an-innocent-peer risk that a wire-wording drift would have created. Per-peer multi-key batching and request-bounded candidate selection are unchanged; no wire-format change. ADR-0005 and REPLICATION_DESIGN record the store-spread assumption and add a size_reject re-open trigger. Validation: all replication library tests pass, no-default-feature build compiles, all-feature Clippy clean under the denied panic/unwrap/expect gate.

0abe74a - drain mature prune backlogs

Adds the width-20 fast-delete path and request-bounded audit batching described under "Prune backlog draining", replacing the candidate-to-peer-edge budget that capped a pass at roughly nine records. Fast-path and audited deletions both re-run a TOCTOU revalidation against the current routing table immediately before deleting; a key that moves inside width 9 clears hysteresis and is retained, and a key that moves outside a complete width-20 group during an audit may take the fast path after fast-path revalidation.

20d3f88 - define scalable prune backlog draining

Adds ADR-0005 section 11 and updates REPLICATION_DESIGN: the three-day width-20 fast-delete path (with its accepted far-copy durability trade-off), request-bounded multi-key audit batching, and rolling compatibility with deployed nodes.

e0c0385 - address deep review findings

Closes four review findings before merge. Bootstrap capacity-rejection expiry is now derived from the runtime neighbor scope, batch width, slowest cadence, per-peer request deadline, cooldown, and one cadence interval of slack; with defaults the window is 125 minutes rather than 70, so a live source near the end of a five-batch cycle is not forfeited before its next legitimate re-hint. shutdown() now awaits a long-lived task after requesting abort, ensuring synchronous sections have actually exited and dropped storage/P2P clones before detached work and LMDB trackers are declared quiescent. Paid-list flexible-edge activation now uses ReplicationConfig::paid_list_close_group_size, preserving strict majority for an undersized 20/24 group and enabling the intended four-edge tolerance for a full non-default 16-peer group. The commit also fixes the all-target/all-feature Clippy gate and adds focused regression coverage. Validation: 738 all-feature library tests, 434 no-default-feature replication tests, the shutdown-LMDB PoC, formatting, diff checks, and the exact CI Clippy command all pass.

14dfd46 - record replication repair hardening decisions

Adds ADR-0005, documenting the repair-hardening decisions, invariants, operational trade-offs, and lifecycle guarantees implemented by this PR.

4c8f821 - recheck storage responsibility at the point of download

0cda93b gated replica downloads on is_responsible(storage_admission_width), but only at verification-completion time — the answer was never re-asked before the chunk was actually downloaded and stored. The fetch queue is nearest-first and up to 131,072 entries deep, so a far candidate can wait unboundedly long while closer keys jump ahead, and the per-source retry path reused the same stale decision for every fallback source. Topology churn between promotion and download therefore still ended in a download, a disk write, and later fetch→store→prune churn once pruning evicted the record at the same width — while types.rs and admission.rs both documented that responsibility is decided against live routing state at the point of download. Responsibility is now rechecked per fetch attempt inside execute_single_fetch, where no queue lock is held: once at the top before spending bandwidth (retries re-enter there), and once more before storage.put so bytes that arrive after responsibility lapsed mid-round-trip are not written; edge flapping is dampened by the margin storage_admission_width already adds over close_group_size. A lapsed attempt resolves as the new FetchResult::NoLongerResponsible, which deliberately shares Stored's terminal path — complete_fetch releases the verification retry-slot reservation and the worker's terminal handling shrinks the bootstrap pending set, so a declined key cannot stall bootstrap drain — and reports no trust event, since the source did nothing wrong. The promotion-time check stays as a cheap pre-filter; event-driven purging of the fetch queue on routing events was considered and rejected (O(queue) scans per churn event, racy, strictly more complex than the lazy recheck). The worker's result handling is extracted into apply_fetch_result for direct unit coverage, and a new e2e drives a live 12-node network through a test-only seam that enqueues a fetch candidate for a key the target is not responsible for — the exact state a stale promotion leaves behind, since live topology cannot be shifted deterministically inside the millisecond promotion→dequeue window — asserting the chunk is never stored and the key exits the pipeline terminally, with an in-responsibility positive control through the same seam and holder. With the rechecks disabled the e2e fails by storing the stale chunk, confirming it discriminates.

9cef69f - expire orphaned capacity-rejection records to unstall bootstrap

37f8e34 made PeerRemoved retire a departed source's capacity-rejection record, but the record's lifecycle was still purely event-driven and the note/removal paths run on different tokio tasks. Each note path has await points (LMDB reads, lock acquisitions) between its last "source is live" observation and the insert, so a removal fully processed inside that window cleared nothing and the subsequent note recorded an entry that no later admission cycle or removal event could ever retire — permanently blocking drain, disabling all audits (Invariant 19), and leaving the node claiming bootstrap network-wide, where other peers' bootstrap-claim abuse logic trust-penalizes it. An attacker could reach this deliberately by overflowing a victim's pending_verify during bootstrap and disconnecting immediately. The record now carries its most recent rejection time and expires after a runtime-derived full neighbor-cycle window (all configured batches and request deadlines, the cooldown floor, plus one slow-cadence interval of slack; 125 minutes with defaults), with expiry and a drain re-check run from the verification worker tick ahead of the pending-request early-returns. That same tick closes the adjacent liveness gap where a clean-cycle clear satisfied the drain condition with no event left to observe it. Expiry forfeits the keys the departed source still owed — consistent with update_bootstrap_after_peer_removed, which already forfeits a departed source's owed work wholesale; post-bootstrap neighbor sync and audit/repair recover them. Deliberately not a routing-table membership check at the note sites (only shrinks the TOCTOU window) and not a global bootstrap deadline (would change Invariant 19 semantics for genuinely busy bootstraps). Regression coverage reproduces the exact race ordering: removal cleanup first as a no-op, the rejection recorded after, drain asserted blocked, then TTL expiry drains it.

44bd01d - keep fresh offers off the serial message loop

Once all four worker permits were held, dispatch_fresh_offer handled the next offer — an on-chain payment verification and a multi-MiB LMDB write — inline on the serial non-audit loop, backing up the 256-slot inbound queue until the P2P event loop also handled messages inline and its broadcast receiver lagged and dropped replication messages wholesale. The per-key shard locks (key[0] % 64) made that the steady state rather than the exception: a node only receives fresh offers for keys close to its own ID, so the accepted key set shares a long prefix and nearly every offer landed on one shard. The ordering those locks preserved has no meaning here, since the key is the content address of the data. Dispatch now only claims the key, takes an admission permit, and spawns; the worker permit is awaited inside the task so handle_fresh_offer has a single caller and no inline fallback exists. A 16-permit admission semaphore bounds outstanding offers at a 64 MiB ceiling, and an exact per-key in-flight set behind an RAII guard replaces the shard locks so unrelated keys never contend and concurrent duplicates collapse onto the first claimant. Past the bound an offer is refused — not stored, and therefore penalized as absent by the delayed possession check — rather than queued or handled inline.

0cda93b - gate replica downloads on storage responsibility

A replica hint is a possession claim, but the receiver treated it as authorization: HintPipeline::Replica skipped the is_responsible(storage_admission_width) check that PaidOnly keys had to pass, so whoever labelled the hint decided what we store. Two messages were enough to exploit it — a paid hint queued key K as PaidOnly, then a replica re-advertisement hit the already_pending fast path in admit_hints and escalated the live entry to Replica, and both fetch paths downloaded without ever asking whether this node was in the top-9 for K. The tag is now derived from replica_hint_sources instead of stored (the tag is "did any peer claim to hold this"), which deletes the escalation and both demotion sites, and downloads are gated on live routing state at both diverging fetch sites. This narrows replica repair from the sender's view to our own — a transiently skewed routing table now declines to repair a key it ranks outside the top-9, matching pruning, which already evicts at the same width — while fresh PUTs keep their wider accept window.

92339a1 - admit hints through a single relevance gate

Admission previously asked two questions depending on the sender's label — replica hints gated at storage_admission_width (9), paid hints at paid_list_close_group_size (20) — letting the sender choose its own gate. Admission now decides only relevance (should we learn this key exists and is paid for) at the 20-wide paid close group, since that is the width across which nodes track payment validity; storage responsibility stays at download time against live routing state. Mislabelling a hint gains an attacker nothing because both labels reach the same gate; the admissible key set is unchanged (a paid hint for any key a replica hint could carry was already admissible at 20), and the two-gate rescue dance is removed. It does recover information the old gate discarded: a replica hint for a key we rank 10-20 for was previously rejected outright even though that band exists precisely to track payment validity.

b5c0367 - merge duplicate hints without rebuilding the fetch heap

A hint for a key already in the fetch queue only needs its advertiser merged into that candidate's source set — a field the heap never orders on — but the old path took the whole heap, scanned it linearly for the key, and re-heapified, costing O(n) per duplicate; under source aggregation a batch of m duplicates ran O(m·n) while holding the global queue write lock, which a neighbor could trigger by re-hinting queued keys. FetchCandidate is split into FetchOrder (key + distance, the only fields the Ord impl reads) held in the heap and FetchPayload (sources + retry metadata) held in a key-indexed map that also serves as the membership index; merging a source is now an O(1) map lookup that never touches the heap, and the departed-peer path edits payloads in place and rebuilds only when a peer actually orphaned a candidate. Measured per-key merge cost goes from 302µs at a 50k-deep queue to a flat ~400ns independent of depth.

3a00fef - drain detached LMDB blocking ops on shutdown

ReplicationEngine::shutdown() promises that when it returns no background work still holds the LMDB environment, but several paths race a select! on the shutdown token against futures awaiting a spawn_blocking LMDB transaction — the fetch worker's storage.put, the neighbor-sync prune pass's storage.delete and paid_list.remove_batch (the PaidList is a second LMDB environment with the same pattern), and the verification worker's paid_list.insert. Dropping the losing future does not cancel spawn_blocking: the closure keeps running on the blocking pool with a cloned Env, so reopening the same environment after shutdown was undefined behavior. Rather than shielding every call site, the blocking tasks are now tracked at the storage layer: LmdbStorage and PaidList route every spawn_blocking through their own TaskTracker and expose wait_idle(), which shutdown() awaits after its existing detached-task drain. Per-fetch tasks are additionally spawned on the detached tracker, so an aborted worker's dropped in_flight set can no longer leak Arc<LmdbStorage> past shutdown. Cancellation semantics are unchanged — network I/O still aborts promptly, and shutdown waits only for the bounded in-flight transaction (milliseconds). Regression coverage parks a write inside its blocking closure, drops the awaiter, and proves wait_idle()/shutdown() block until it commits and both environments reopen cleanly.

Preceding Follow-up Commit Details

ab848a2 - stop the message handler when event streams close

Makes P2P lag remain recoverable but treats a closed P2P broadcast as terminal, matching the new terminal handling for a closed DHT broadcast. The replication loop now exits instead of spinning indefinitely on an immediately ready closed receiver; dropping its sender also lets the serial handler finish. The commit additionally hoists imports required by the Rust conventions hook.

3e8860c - prune departed peers during DHT lag recovery

Brings lag recovery in line with the normal KClosestPeersChanged path by retaining only the current close-peer set before requeueing its members. Stale peers from missed departure events no longer sit ahead of genuine entrants and burn one request timeout each.

fc9dff0 - penalize rejected singleton replica hints

Penalizes a sole replica advertiser when either the close group definitively rejects the key or that advertiser explicitly denies possessing it. This closes the free-replication abuse path while keeping inconclusive rounds without a direct contradiction, paid-only advertisements, and corroborated replica hints non-penalizing. Trust reports remain bounded per peer and verification cycle.

Earlier Follow-up Commit Details

37f8e34 - unblock bootstrap when a rejected peer leaves

Retires capacity-rejection debt on PeerRemoved and immediately reruns bootstrap drain detection, so progress no longer depends on an unrelated later pipeline event.

8f38d4c - prioritise source-aware hints

Replaces the single hint sender with live source sets, preserves replica claimants separately, prioritises corroborated work, bounds verification cycles and network concurrency, and simplifies stale proof-of-concept tests around the production queue implementation.

94705cf - aggregate bootstrap hint batches

Makes a bootstrap neighbor batch an atomic source-aggregation unit: sync requests run concurrently, response metadata is processed first, admitted hints are queued together, and verification waits until batch publication and drain accounting are complete.

f9263de - aggregate verification per peer

Sends one bounded request per target peer for the cycle, aligns the incoming limit with the cycle bound, and rejects oversized batches rather than processing a misleading prefix.

8d3834b - drain fresh offers through LMDB writes

Removes cancellation around a started fresh-offer handler and makes its tracker drain unconditional, preventing a dropped async waiter from detaching a live blocking LMDB transaction.

b577e8e - track detached audit work

Generalises the fresh-offer tracker into a shared detached-task lifecycle barrier and applies it to storage/P2P-capable audit responders, delayed possession checks, and audit launches.

Latest Follow-up Commit Details

172e97f - guard raw LMDB reads against concurrent map resize

all_keys() and get_raw() now acquire the shared LMDB environment lock inside their tracked blocking closures before opening a read transaction. This restores the invariant assumed by try_resize(): its exclusive lock means no transaction can still reference the old memory map when env.resize() runs. A deterministic regression parks an in-flight raw read and proves resize waits for it to release.

86c4910 - fairly share verification capacity

Replaces the removed hard per-peer admission quota with elastic max-min accounting under the unchanged global cap. A peer may use all otherwise idle capacity, preserving large legitimate bootstrap batches, but under contention an under-served sender can displace a low-priority borrowed entry held by an over-served sender. Verification service is independently fair across capacity owners via persistent round-robin scheduling, with slack redistribution and source-count/retry priority within each owner. Capacity ownership survives fetch retries, transfers when a source departs, and is separate from evidence provenance; bootstrap explicitly records displaced hints for redelivery. The wire protocol and serialized/public compatibility surface are unchanged.

Adversarial coverage includes a paid-only attacker filling all 131,072 slots before an honest source submits 10,000 hints, the reverse arrival order, a sole legitimate 50,000-hint source, full-cycle fairness, retry protection, owner transfer, duplicate-bookkeeping bounds, and bootstrap displacement accounting. Validation passes all 766 library tests, the seven bounded-queue PoCs, formatting, diff checks, and all-target/all-feature Clippy with warnings denied.

Newest Follow-up Commit Details

a40da7b - generalize bounded responder admission

Extracts the audit responder's global permit and per-peer slot into a reusable admission primitive. A provisional peer slot is RAII-owned while waiting for global capacity, so both normal guard drops and cancelled waits restore capacity.

ec8eb6a, 864a5b4, and 57873d0 - isolate bulk responders

Moves fetch, verification, and neighbor-sync response work off the serial replication lane and onto bounded worker pools. Fetch uses three bandwidth-limited workers; verification and neighbor sync use two each and permit one outstanding batch per source. Requests that have already exceeded the caller's existing deadline are shed at dequeue, before stale LMDB or upload work begins.

7b5ee55 and 48db2d5 - make overload bounded and visible

Removes the inline execution fallback when the serial handoff is full or closed, preserving responsiveness of the P2P event receiver. Relaxed atomic counters record admission drops, stale dequeues, and per-variant serial drops without adding contention to the overloaded paths.

4ef753b and 339f44b - add audit and possession diagnostics

Adds structured audit issuer/source attribution, capacity decisions, stage latency, and bounded minute summaries. Responsive possession failures now retain a stable reason label for every existing validation branch; the wire protocol and trust decisions are unchanged.

Coverage Added Or Updated

• Bounded responder guards release global and per-source slots on ordinary drop and cancelled admission waits.

• Neighbor-sync admission serializes one source while allowing a different source to progress concurrently.

• Expired responder work is shed at dequeue while fresh requests are served.

• A full serial handoff drops the message instead of invoking its handler inline.

• Bulk-responder counters remain separated by class and admission-versus-staleness outcome.

• Audit summaries order top logical origins and preserve per-stage timings.

• Possession validation branches map to stable structured failure labels.

• LMDB resize waits for an in-flight get_raw() transaction holding the shared environment lock.

• Elastic max-min admission under attacker-first and honest-first arrival orders, including large sole-source bootstrap batches and the unchanged global capacity bound.

• Fair per-owner verification service with slack redistribution, retry protection, peer-removal ownership transfer, bounded duplicate bookkeeping, and displaced-bootstrap redelivery accounting.

• Paid-hint admission after duplicate replica rejection.

• Verification retry, reservation transfer, discard, exhaustion, and per-sender capacity behavior.

• Paid-list edge vote behavior for negative, positive, unresolved, self-inclusive, undersized, and non-default runtime-configured group widths.

• Neighbor-sync priority drain/termination and lag recovery.

• Bootstrap completion after a capacity-rejected peer departs.

• Capacity-rejection TTL derivation across the full configured sync cycle and runtime settings, plus semantics: a within-TTL rejection still blocks drain, an expired record unblocks it, expiry is per-source (a stale source's expiry does not forfeit a fresh source's owed re-delivery), and a repeat rejection refreshes the timestamp.

• The peer-removal race ordering itself: removal cleanup runs first as a no-op, the rejection is recorded after for the now-departed peer, drain is blocked, then TTL expiry retires the orphaned record and drain completes.

• The verification-tick self-heal helper: an orphaned record survives a within-TTL tick and a past-TTL tick expires it and completes bootstrap.

• Duplicate hint source aggregation and source-aware scheduling.

• Singleton replica-hint penalties for definitive close-group rejection and explicit source denial, including neutral inconclusive, paid-only, and corroborated cases.

• Atomic bootstrap batch publication and full-cycle verification request bounds.

• Fresh-offer admission bounding, per-key in-flight collapse of concurrent duplicates, and the refusal-past-bound possession penalty.

• Replica-download responsibility gating at both fetch sites, including rejection of out-of-range keys and the removed paid→replica escalation path.

• Single-gate hint admission parity across replica and paid labels for the unchanged admissible key set.

• O(1) duplicate-source merge and heap rebuild only on genuine candidate orphaning.

• Audit coordinator per-target serialization, cross-peer parallelism, and cancellation cleanup.

• Closed P2P event handling now explicitly verifies terminal control flow; lagged P2P event handling still verifies continuation and metric accounting.

• E2E paid-list majority repair below storage quorum.

• Storage- and paid-list-level wait_idle regression tests: a write parked inside its blocking closure with a dropped awaiter keeps wait_idle blocked, commits after release, and leaves the store usable.

• Engine-level shutdown drain (poc_shutdown_lmdb_drain): shutdown() blocks until a detached LMDB write commits, after which both environments reopen cleanly.

• Per-attempt responsibility recheck: worker disposition of NoLongerResponsible (terminal exit, retry-slot release, no verification requeue), terminal-path parity with Stored, and preserved source-failure retry/requeue transitions.

• E2E stale-fetch-candidate driver on a live 12-node network: a seam-enqueued candidate for an out-of-responsibility key is never stored and exits the pipeline terminally, with an in-responsibility positive control through the same seam and holder; the test fails when the rechecks are disabled.

• Existing prune, fetch-retry, and replication tests updated for shared coordination and reservation-aware APIs.

• Prune backlog draining: width-9 hysteresis classification; complete-width-20 fast deletion versus incomplete-width-20 audited fallback; fast-path and audited TOCTOU revalidation; dynamic square-root peer batches; request-budget admission of complete candidate proof sets; rotating fairness; and every Rejected reason grading to the non-recovering Rejected status without parsing its wording (a challenge-id mismatch stays a plain failure).

Expected Effect

• Fetch, verification, and neighbor-sync floods cannot monopolize the serial replication lane; bounded, per-source-fair workers isolate their LMDB, queue, and upload costs.

• Serial-queue saturation remains local to the rejected message instead of propagating inline work and lag into the P2P receiver.

• Operators can distinguish admission pressure, expired queued work, audit-origin hotspots, slow response stages, and exact responsive possession-proof failures.

• Raw LMDB reads cannot overlap an unsafe memory-map resize, closing the remaining resize-safety gap for audit, pruning, and commitment scans.

• A routing peer can use idle verification capacity but cannot monopolize admission or service once honest peers contend; large genuine bootstrap batches remain admissible when capacity is available.

• Partition heals and mass joins drain priority neighbor-sync work in seconds-scale rounds instead of waiting through periodic ticks or stale-peer timeouts.

• Repair remains live through transient routing disagreement, no-holder/no-source rounds, queue pressure, and fetch exhaustion.

• Better-corroborated hints are verified first without allowing verification rounds or request fan-out to grow without bound.

• Sole peers cannot advertise unacknowledged replicas to offload storage for free without incurring bounded trust penalties.

• Fresh-offer verification and storage never run inline on the serial loop, so worker saturation no longer backs up the inbound queue or drops replication messages through broadcast lag.

• A peer cannot conscript out-of-range nodes into fetching and storing arbitrary keys by labelling hints as replicas.

• Duplicate-hint bursts no longer scale the fetch-queue write-lock hold time with queue depth.

• Topology churn between fetch promotion and download no longer costs a download, a disk write, and a prune cycle — stale candidates are declined at download time, terminally and without stalling bootstrap drain or penalizing an innocent source.

• Bootstrap cannot be held indefinitely by partial batch publication or departed capacity-rejected peers — including one whose removal races the rejection record — and a drain condition that becomes true without a triggering event is observed within one worker tick, so audits cannot be disabled permanently nor the node trust-penalized for a perpetual bootstrap claim.

• Local audit concurrency no longer manufactures false remote timeout verdicts.

• Mature out-of-range records drain at scale: far-out copies beyond a complete width-20 view are fast-deleted in bounded local batches without remote traffic, and the audited rank-10-20 band is denser (multi-key peer batches bounded by actual requests) instead of ~9 records per pass. An oversized-batch rejection is graded as an attributable failure rather than parsed and retried, so no honest capacity-limited peer is trust-penalized under the assumed store spread.

• Closed replication event streams terminate cleanly without CPU spin or repeated warnings.

• Graceful shutdown does not return while tracked detached storage or P2P work is still active.

• When shutdown() returns, timed-out long-lived tasks have been aborted and joined, no LMDB blocking operation is still running on either environment, and no engine-spawned task holds the storage, so the same LMDB files can be reopened safely.

SemVer

Minor.

Review Response (grumbach, 2026-07-27)

Investigated all thirteen findings from the review at 4c9c681. Twelve
confirmed; one did not hold (see below). Nine were fixed, three deliberately
deferred with the reasoning recorded in ADR-0005 rather than left implicit.

Must fix

• Bootstrap capacity-rejection debt never expired (d3a1d56). The record was
  stamped with the most recent rejection and aged from that stamp, so a source
  that keeps overflowing the queue kept its own record permanently fresh:
  check_bootstrap_drained never returned true, the node advertised
  bootstrapping: true indefinitely, and auditing stayed off for the whole
  duration of the pressure (Invariant 19). No attacker needed — an ordinarily
  busy peer wedges the node open. Now recorded first-seen and never refreshed.
  A fairness displacement also no longer stamps its victim; charging our own
  reclaim decision to the displaced owner let a flooder block our drain through
  an unrelated honest peer.
• PaidNotify verified inline on the serial loop (a84c42b). Not a bounded
  local computation: on the merkle path it performs an iterative Kademlia lookup
  capped only by the verifier's CLOSENESS_LOOKUP_TIMEOUT (240s). One message
  could park every other non-audit message for minutes while the bounded inbound
  queue behind it overflowed and dropped unrelated replication traffic wholesale.
  Moved onto the detached-responder pattern.
• Fresh offers had no per-source admission share (a84c42b), unlike every
  other responder class. One peer could hold all sixteen slots, and since a
  refusal is later read as absence by the sender's delayed possession check, the
  resulting refusals land as audit-severity penalties on the refuser — making
  an unbounded global pool a targeted eviction primitive. Structural checks
  (missing proof, oversized payload) now run before the permit, so malformed work
  cannot occupy a slot for the length of a verification.

Should fix

• shutdown() could hang indefinitely (a84c42b, d79752e). The detached
  drain is deliberately unbounded — a timeout could return with an
  Arc<LmdbStorage> still held — so it was made finite instead: worker
  semaphores are closed so queued work takes the (previously unreachable) error
  arm and exits, superseded fresh offers are shed at dequeue, and payment
  verification races the shutdown token. The cancellation boundary is drawn at
  network I/O only; storage.put awaits spawn_blocking and is never cancelled.
• Per-key hint sources grew without bound (809f509). MAX_PENDING_VERIFY
  counts keys, not the peers remembered against each one, and re-advertising an
  already-pending key is the cheapest path through admission. Capped at eight,
  with replica claimants displacing paid-only advertisers.
• Retry reservations were billed to nobody (809f509), so an owner with many
  in-flight retries read as under-loaded and won more than its share.
• The paid-list flexible edge was disproportionate at small widths
  (1094543). A fixed four-peer edge is a fifth of the production width but four
  fifths of a configured width of five, leaving a voting core of one and
  authorizing on a single Confirmed vote. Now scaled to the group with an
  absolute confirmation floor. Production behaviour at width 20 is unchanged.
• The removed MAX_PENDING_VERIFY_PER_PEER was undocumented (1632ba7,
  809f509). ADR-0005 described the replacement but never recorded that it
  removed a named defence, or that one sender's worst case rises 8,192 → 131,072.
  Now stated with the adversarial numbers, plus a test pinning the convergence
  property that replaced the hard cap.

Smaller

• Cancelled paid-list prune window documented rather than "fixed" (cca744c) —
  including why clearing timestamps before the commit would be strictly worse.
• Two unused wrappers deleted (0a03c71); each built a throwaway
  AuditChallengeCoordinator that would have defeated the shared one.
• The far-key parity e2e could not detect what it claimed (eeb068b): it used a
  key out of range for both gates, so every version of the code passes it, and
  had no positive control. Rewritten against a key in the symmetric difference of
  the two gates, with a control proving the harness observes a store at all.

Not fixed, and why

• scheduling.rs requeue_fetch_for_verification leaving a stale
  capacity_owner_by_key entry — this one does not hold. The no-owner branch
  is only reachable when the lookup already returned None (the .or_else runs
  only on None), so there is no entry to leave behind.
• A capacity refusal still costs the refuser trust. Fixing it properly needs
  the sender to read the Rejected response — the receiver already returns a
  precise reason, it is simply discarded — and to grade the possession outcome
  accordingly. The per-source share bounds how large the effect can be made
  deliberately, but does not remove it. Recorded in ADR-0005 Trade-offs.
• Inbound queue byte budget and audit-metrics by_source top-k deferred.

Two regressions caught and fixed during this work

Both were found by the e2e suite and confirmed against a stashed baseline rather
than assumed:

• Scaling the paid-list edge in summarize_paid_list_votes without scaling edge
  membership in build_verification_targets let core + confirmed_edge exceed
  the real group size and demand more confirmations than existed.
• Sizing PaidNotify like a request/response responder (8 outstanding / 2 per
  peer / 15s shed) discards durable paid-list evidence rather than shedding load,
  because the message is one-way with no retry. A joining node silently lost
  entries for a fraction of its responsible keys. Now sized as a memory ceiling.

Integration

Merged origin/main rather than rebasing (d86e3f1). The branch and main
diverged independently inside the same functions — main has none of this
branch's detached-task tracking and replaced the first-audit scheduler wholesale
— so replaying fifty-three commits would have meant roughly forty semantic
re-applications onto rewritten code. The merge resolved ten conflicts once. Its
commit message records each resolution, notably taking main's rewritten
first-audit scheduler over this branch's superseded LruCache implementation,
and keeping both the cumulative and windowed audit-drop counters, which are
complementary rather than duplicates.

d79752e then re-tracks main's new first-audit launch, the one detached task
its rewrite spawned bare.

Verification

On the merged tree: 828 lib tests, 92 e2e (3 pre-existing ignored), 38 PoC tests,
clippy clean at -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used,
rustfmt clean, release and --no-default-features builds green.

[Copilot]

Pull request overview

This PR improves replication convergence under churn by (a) ensuring neighbor sync drains queued priority peers without waiting for periodic ticks and (b) recovering from lagged DHT routing-table broadcast events. It also expands verification/fetch pipeline behavior (batching caps, retry/defer semantics, paid-list edge-voter quorum handling) and adds an e2e scenario covering paid-list-authorized repair below storage quorum.

Changes:

• Neighbor-sync loop now drains the priority queue back-to-back and resyncs close peers on RecvError::Lagged.
• Verification/fetch pipeline enhancements: bounded verification batches, fetch→verification retry metadata, and deferred re-verification scheduling.
• Paid-list quorum evaluation updated with “edge voter” handling; adds a deterministic paid-list repair e2e scenario and bumps crate version.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/poc_d1_bounded_queues.rs	Updates test VerificationEntry construction for new timing fields.
tests/poc_bootstrap_stall.rs	Updates test VerificationEntry construction for new timing fields.
tests/e2e/replication.rs	Adds deterministic e2e scenario for paid-list-majority repair under low storage quorum.
src/replication/types.rs	Adds next_verify_at, fetch retry metadata, and NeighborSyncState::has_priority_peers + test.
src/replication/scheduling.rs	Adds deferred pending scheduling, fetch retry→verification requeue, and returns evicted keys.
src/replication/quorum.rs	Adds edge-aware paid-list vote summary and splits verification requests into capped batches.
src/replication/mod.rs	Implements lag recovery, neighbor-sync drain-before-park, verification request caps, and retry/defer integration.
src/replication/config.rs	Introduces PAID_LIST_FLEX_EDGE_COUNT and MAX_VERIFICATION_KEYS_PER_REQUEST.
src/replication/bootstrap.rs	Updates test VerificationEntry construction for new timing fields.
src/replication/admission.rs	Adjusts cross-set dedup/admission so paid hints can survive replica rejection under churn.
Cargo.toml	Bumps version to 0.14.3.
Cargo.lock	Updates locked crate version to 0.14.3.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 19 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/replication/storage_commitment_audit.rs:161

• run_subtree_audit’s docstring still says “gossip-triggered” and “the peer just gossiped”, but this wrapper now always calls run_subtree_audit_with_origin(..., SubtreeAuditOrigin::Manual). That makes the public docs misleading about how/when this API is used and what context it carries.

/// Run one gossip-triggered subtree audit against `challenged_peer`, pinned to
/// the commitment hash the peer just gossiped (`expected_commitment_hash`).
///

src/replication/admission.rs:283

• This test describes a scenario where a replica hint is rejected by a storage-close-group gate but the paid hint for the same key must still be admitted. With the new unified is_relevant admission gate (replica and paid hints both go through the same relevance check), that scenario should no longer be possible, so this test’s commentary and intent are likely outdated/misleading.

    fn paid_hint_survives_duplicate_replica_rejection() {
        // With churned local views, a sender may believe we are in the storage
        // close group while our own view rejects the replica hint. If the same
        // key also arrives as a paid hint, paid-list admission must still run.
        let key = xor_name_from_byte(0xEE);

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/replication/storage_commitment_audit.rs:161

• run_subtree_audit is documented as “gossip-triggered”, but it always calls run_subtree_audit_with_origin(..., SubtreeAuditOrigin::Manual) and is used by the test-only audit_peer_now path. This makes the docstring misleading and can confuse log interpretation when reading this API.

/// Run one gossip-triggered subtree audit against `challenged_peer`, pinned to
/// the commitment hash the peer just gossiped (`expected_commitment_hash`).
///

src/replication/pruning.rs:1568

• If AuditChallengeCoordinator::acquire() returns None here, that’s a local lifecycle/coordination condition (the semaphore was closed), not a malformed remote response. Returning MalformedResponse flows into receive_prune_audit_response and ultimately reports a prune-audit failure/trust penalty for the peer even though no request was sent.

    let Some(_slot) = audit_challenge_coordinator.acquire(*peer).await else {
        warn!(
            "Prune audit challenge with {key_count} keys against {peer} could not acquire coordinator slot"
        );
        return PruneAuditChallengeResult::MalformedResponse;

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/replication/storage_commitment_audit.rs:160

• The doc comment for run_subtree_audit still says “gossip-triggered”, but this wrapper now hard-codes SubtreeAuditOrigin::Manual and is used by the test/manual audit_peer_now path. This makes the documentation misleading (and could confuse log attribution expectations).

/// Run one gossip-triggered subtree audit against `challenged_peer`, pinned to
/// the commitment hash the peer just gossiped (`expected_commitment_hash`).

[jacderida]

Testnet comparison — #165 vs current RC

Ran this branch through a 4h staging testnet and compared it against the current RC staging run.

Setup

Both runs used the same profile, staging-short @ 47165ba — identical ~990-node topology (66 node VMs + 7 bootstraps, same DO/Vultr/OVH/OVH-3az split, same 30% NAT sim and clock-skew mix). Only the node binary differs:

	node binary
STG-01 (RC)	release v0.15.0-rc.1
DEV-01 (this PR)	branch fix/neighbor-sync-drain-window

File operations — steady 3h window, both runs

Both runs: 100% success, zero failures, every file-size bucket, uploads and downloads. Durations:

Uploads

size	RC — n / p50 / mean	#165 — n / p50 / mean
21 MB	787 / 30.3s / 35.3s	738 / 33.3s / 37.6s
105 MB	176 / 51.0s / 58.8s	164 / 55.8s / 62.9s
524 MB	24 / 258s / 289s	16 / 179s / 205s
1049 MB	62 / 247s / 307s	65 / 240s / 275s

Downloads — near-identical across all buckets (1049 MB: p50 98.0s vs 98.8s; mean 112s vs 110s).

Statistically indistinguishable throughput and latency. #165 is marginally faster on the largest uploads (1049 MB mean 275s vs 307s), marginally slower on the smallest — all within run-to-run noise. No regression on the transfer path.

Node host metrics — whole-run aggregate

Fleet mean of per-host averages:

metric	RC	#165	Δ
Node CPU avg %	53.6	44.8	−8.8
Node CPU peak %	100.0	100.0	~0
Node mem avg %	32.9	33.9	+1.0
Node mem peak %	63.3	67.3	+4.0
Node net out MB/s	15.09	16.64	+1.55
Node-storage write MB/s	2.23	2.52	+0.29
Bootstrap CPU avg %	28.2	21.7	−6.5

#165 runs ~9 CPU points cheaper fleet-wide, with bootstraps moving the same direction at −6.5. The RC run carried the sustained ant_node::replication audit-timeout storm (~70–110k lines/30min across 72 hosts, all reason=Timeout), which began at T0+45m and persisted through its steady window. A storm of that shape across the fleet is precisely the load that presents as broad node-tier CPU, and it is reduced by ~9 points on this branch — the expected signature of the drain-window fix.

Memory is marginally higher (+1 avg / +4 peak), well within headroom. Net and node-storage write are slightly up, consistent with nodes spending cycles on productive work rather than retry churn.

Bottom line

• Reliability: a tie — both 100% success, 0 failures, matched steady 3h windows.
• Transfer performance: unchanged — the fix costs nothing on upload or download.
• Node CPU: fix(replication): drain priority sync queue and recover from routing-event lag #165 is ~9 points cheaper, consistent with it partially eliminating the audit-timeout storm it was written to address.

[grumbach]

Reviewed at 4c9c681. Read the full diff against eb74b56, plus ADR-0005.

Great work, and the parts that land are genuinely strong. The responsibility recheck, the derived pipeline(), and the LMDB task tracking all do what the ADR says. My concern is size and blast radius: 42 commits and +9.6k across 25 files means the good fixes are hostage to the risky ones. I found three new remotely reachable resource paths and one removed defence that the ADR does not mention. I would like to see this split so items 1-3 under "Suggested split" can go in this week.

Must fix before merge

1. The new inbound queue is bounded by message count, not bytes

INBOUND_REPLICATION_SERIAL_QUEUE_CAPACITY = 256 (src/replication/mod.rs:197) holds fully decoded ReplicationMessage values, and the transport ceiling is 10 MiB (src/replication/config.rs:306, wired at src/config.rs:388). Worst case retention is around 2.5 GiB. Every body except AuditChallenge is queued (mod.rs:1648), including unsolicited response variants that only become no-ops after dequeue (mod.rs:3651).

The stall primitive is in the same file: handle_paid_notify still runs on-chain payment verification inline on the serial loop (mod.rs:3131). That is the pattern decision 2 of the ADR removes, but only for fresh offers. One PaidNotify parks the consumer on an EVM round trip and 256 near-limit messages pile up behind it. The 16-permit fresh-offer bound does not help because the permit is taken after dequeue (mod.rs:3706).

Suggested: byte budget before decode, drop unsolicited response variants on the one-way path, and move PaidNotify verification onto the same detached responder pattern you already built for neighbor sync and verification.

Small related thing: the doc comment on the constant says a full queue is handled inline rather than dropped, but mod.rs:1669 drops and warns. Worth correcting either way.

2. Fresh-offer refusal turns into an audit-severity penalty on the refuser, with no per-peer cap

• 16 global admission permits, 4 workers (mod.rs:205, mod.rs:216), and no per-source share. Every other responder class in this PR has one (neighbor sync 1/peer, verification 1/peer, fetch 2/peer).
• The permit is taken before any validation, so a peer holds one through the whole payment verification even with a junk proof, and a failed proof carries no trust penalty (mod.rs:3956).
• The sender never reads the Rejected response. fresh.rs:120 uses send_message, so the only signal is the delayed possession check, which sees a missing key and reports AUDIT_FAILURE_TRUST_WEIGHT.

So one peer can hold all 16 permits with cheap invalid offers, every honest offer in that window is refused, and the honest receiver gets penalised at audit severity by honest senders. That feeds the eviction path. The ADR lists refusal as a trade-off but does not note it doubles as a targeted eviction primitive. A per-peer share plus validating cheaply before taking the permit would close most of it.

3. The bootstrap capacity-rejection TTL never expires under sustained pressure

publish_bootstrap_admission_outcomes re-stamps capacity_rejected_sources.insert(source, now) on every admission unit (mod.rs:5237), and expire_capacity_rejected only fires on age. A peer that keeps overflowing the queue keeps its own record fresh forever, so check_bootstrap_drained never returns true, audits stay off, and the node keeps advertising bootstrapping: true.

mod.rs:5221 also re-stamps the displaced owner, so this can be pointed at an unrelated honest peer. Decision 6 closes the departed-peer race, which is real, but the live-peer case is still open. A first-seen timestamp that a repeat rejection does not refresh, or a cap on total refreshes, would fix it.

Should fix

4. shutdown() can hang

detached_task_tracker.wait() has no timeout (mod.rs:1024), which I agree is the right call for LMDB. But fresh-offer workers on that tracker never check the cancellation token and have no staleness shed, unlike the neighbor-sync and verification responders. They await verify_payment, whose HTTP call has no timeout at our level. A hung RPC hangs shutdown. The storage-layer trackers this PR adds are exactly what makes cancelling the outer async waiter safe now, so this should be cheap. Side note: every worker_semaphore.acquire_owned() error arm is dead, since none of those semaphores are ever closed.

5. The per-peer pending_verify cap is gone and the ADR does not say so

Base scheduling.rs:66 had MAX_PENDING_VERIFY_PER_PEER = 8_192, documented as the primary flood defence. There is no equivalent now. "A sole sender may borrow unused capacity" means one routing-table peer can hold all 131,072 slots instead of 6.25% of them, a 16x jump in single-peer memory. The max-min convergence is arguably better than the old hard cap, so this may well be the right call, but it needs to be in the ADR with the adversarial numbers rather than read as pure hardening.

6. Duplicate hints grow source associations with no budget

MAX_PENDING_VERIFY caps unique keys, but duplicates merge new sources into two per-key HashSets plus the reverse index at no capacity cost (scheduling.rs:254), and pending keys short-circuit the relevance check (admission.rs:80). Twenty routing-table peers re-advertising 131k keys gives millions of associations. This is new, since the base stored one hint_sender. A per-key source cap would bound it.

7. The four-peer flexible edge lowers the paid-list bar more than intended

At width 20, four silent or negative edge voters take the threshold from 11/20 to 9/16, and the new unit test pins that (quorum.rs:1271). At smaller configured widths it degrades badly: the guard is configured_group_size > PAID_LIST_FLEX_EDGE_COUNT, so a configured group of 5 leaves a core of 1 and a single Confirmed vote authorizes the key. validate() accepts any size at or above 1 (config.rs:587). Since PaidListVerified writes the paid list and enables repair fetches, this is worth tightening: scale the edge with the group, add an absolute confirmation floor, or reject unvalidated widths in validate().

Smaller

• audit_metrics.rs:186: record_received inserts into by_source before admission and is unbounded until the 60s snapshot, which then sorts the whole map. A bounded top-k plus an overflow counter would do.
• paid_list.rs:400: remove_batch commits in the blocking closure but clears the hysteresis timestamps only after the await resumes, so a cancelled prune leaves the key deleted with stale timestamps.
• tests/e2e/verify_storage_gate.rs:295: the far-key parity test picks a key that both pre-PR gates rejected, so it does not discriminate, and "a sync response came back" does not prove admission. Needs a key in the symmetric difference and a positive control.
• scheduling.rs:1138: requeue_fetch_for_verification leaves a capacity_owner_by_key entry behind on the no-owner path.
• Retry reservations are subtracted globally but never charged to an owner in reclaim_borrowed_slot's demand map, so an owner with many in-flight retries looks under-loaded and gets a larger share than it should.
• pruning.rs:359 and audit.rs:108 are unused wrappers that each build a throwaway AuditChallengeCoordinator, which would quietly defeat the shared coordinator if anyone called them. No callers today, so probably just delete.

What I liked

• The responsibility recheck is the strongest part. Two live-routing checks per attempt, before spending bandwidth and again before storage.put (mod.rs:6031, mod.rs:6158), sharing the terminal path with Stored so drain cannot stall. The e2e has a real positive control and a conclusive negative, so it actually discriminates.
• Deriving pipeline() from replica_hint_sources instead of storing it genuinely kills the two-message conscription path. Nice simplification on top of a security fix.
• Putting the LMDB tracker at the storage layer rather than at each call site is the right choice, and it picked up a real pre-existing bug on the way: all_keys() was iterating without the env_lock, so a concurrent env.resize() could run during an open read txn.
• AuditChallengeCoordinator reference accounting is cancellation-safe including for parked acquires. I went looking for a leak and could not find one.
• The titular fix is correct. Cooldown is still honoured for priority peers (neighbor_sync.rs:286), queue_priority_peers dedupes, and the drain provably terminates.
• Moving neighbor-sync and verification responders off the serial loop with per-peer admission and staleness shedding is a good pattern. It just needs to reach fresh offers and PaidNotify too.

Suggested split

Ship these first, they look close to ready:

1. Event-stream terminal and lagged handling, priority-sync drain, bootstrap liveness
2. Fetch responsibility recheck and queue transition accounting
3. LMDB lifetime tracking, with a bounded shutdown folded in (item 4)

Hold these for a redesign pass:

4. Inbound queue byte budget and PaidNotify off the serial loop (item 1)
5. Fresh-offer per-peer admission and the refusal to penalty path (item 2)
6. Source association budgets and the paid-quorum edge policy (items 6 and 7)

Plus an ADR update to record the per-peer cap removal (item 5) and correct the TTL claim (item 3).

Needs a rebase either way, the branch conflicts with main. The Windows CI failure is a runner flake (Failed to create transport during node setup), not a code problem.