ACM

BAM vs. kcompactd: how we fixed small blocks with two sysctls

How we traced BAM small blocks from the leader window down to a memory-compaction daemon on the Linux host
Authors: BAM Performance Team
Summary
In late September, about one in every 800 blocks built by BAM (Block Assembly Marketplace) came in under 7 million compute units. We call these small blocks (this is not a Solana-wide definition). The small block rate was low, but every small block is revenue a validator didn’t earn and blockspace the network didn’t use.
Over two weeks we found and fixed time losses in five layers: block replay in the BAM node, the BAM scheduler, the memory allocator, Linux TCP congestion control, and the Linux kernel on the physical hosts that run our TEE VMs. All of these changes made a measurable difference. One change explains most of the drop in small blocks, and it was a kernel scan on the Linux host that slowed our TEE VMs for about 48 seconds out of every 80.
This technical post covers the leads we chased, what each fix did and didn’t change, and the unexpected culprit.
We want to give a special thanks to Alessandro Decina (@alessandrod) from Anza. Slowlana was a critical observability tool to identify where a small block lost useful leader time (replay/bank readiness, scheduling, or Turbine propagation) and validate fixes against normal slots to deliver the results presented in this post.
Background: How Jito BAM builds a block
Jito BAM runs inside an AMD SEV-SNP [1] confidential Trusted Execution Environment (TEE) Virtual Machine (VM), which we call the TEE VM. Each TEE VM has about 736 GiB of RAM and 119 vCPUs, and runs alone on a dedicated physical Linux machine, the host. The BAM node follows the chain itself: it replays every block to keep its own bank current, and it stays connected to each validator it serves.
A leader gets four consecutive slots. During this investigation each slot was 250 ms long [2, 11]. For each slot, BAM works through three steps:

Bank handoff. The parent block’s shreds arrive, BAM’s replay executes and freezes the block, the next bank is created, and the bank for the leader’s slot becomes usable.
Build. BAM runs an auction over pending transactions and bundles, then simulates the winners against speculative banks.
Forward. BAM streams batches to the leader validator, which executes them while its slot is still open.
All three steps share the same 250 ms. Time lost before the handoff and time lost during the build look identical from outside: a light block, or batches the validator rejects as OUTSIDE_LEADER_SLOT because they arrived after the targeted slot finished. A small block tells you that time ran out, but not where it went.
Problem deep dive: late banks
So we started at the bank handoff. We joined BAM’s build and forward logs, validator feedback, replay metrics and leader schedules into a per-slot timeline, and reconstructed the first four small blocks one by one.
All four had the same shape. BAM made its first build 3.8–6.5 ms after the bank became usable and forwarded 5.1–14.1 ms after that. Building was fast in every case. The bank had arrived late.

Slow replay of the parent bank explained the delay. On our Frankfurt BAM node, we saw slot replay times of 415–602 ms, more than a full slot. In Singapore, one 5.9-second replay (4.8 s of it in program-cache work) created a backlog. The next leader rotation began while BAM was still catching up, so its first two slots were lost and the last two recovered.
One fix we tried was more replay parallelism. Raising replay worker threads from 8 to 16 to 24 gave no improvement; the mainnet transaction load was long dependency chains. Replay had enough threads, but each one was running slow. That raised the question the rest of this post answers: what was slowing them down?
Time Stealing Leads
Profiling the BAM node turned up four types of stall. The first slowed replay directly. The other three cost time after the handoff. Each was a real cost worth fixing, so we’ll cover them briefly.
1. False sharing that slowed replay
We questioned why replay inside the BAM node was slower than it should have been, even on capable hardware. Something else in the node was getting in its way.
We found it in BAM's scheduling thread. While waiting for its leader slot, this thread checked the working bank about 3.1 million times a second, taking two read locks each time. A read lock looks harmless, but taking one still writes to memory, because it updates a counter inside the lock. Checking the bank's memory layout showed that those counters sat on the same cache lines as bank data replay reads constantly. Every time the scheduling thread took a lock, the CPU had to pull those cache lines away from the replay cores, often from a different CCX (a group of AMD EPYC cores that share an L3 cache). Replay then had to wait to pull them back.
This is called false sharing: the two threads never touched the same data, but sharing cache lines was enough to slow replay down. We moved the lock reads to only the ticks that need them, and the slowdown went away.

2. Bank destruction on the hot path
Our speculative banks store the speculative account state for the leader’s block. On slot transitions, we retire the speculative bank, freeing up tens of megabytes. Jemalloc, the memory allocated used by BAM, then returned those pages to the kernel with madvise, synchronously, on the main state machine thread. Inside a TEE VM, every discard can trigger TLB shootdowns across all vCPUs. One capture showed 366 MADV_DONTNEED calls costing 38.6 ms.
We moved destruction to a bounded reaper thread, upgraded to jemalloc 5.3.1, and enabled background_thread:true [6] so allocator threads do the purging.
3. Auction teardown and pacing
Wavefront, our Directed Acyclic Graph (DAG) based simulator, could wait on a straggling simulation long after its round stopped being useful. This could cause auction rounds to extend more than the intended time period and reduce the throughput. Teardown is now fenced and bounded to 5ms.
Separately, a pacing latch checked only the input queue and not work already admitted to the DAG, so bundle work could resume while ordinary transactions were still unfinished. In one slot, 179 transactions sat in the DAG with only 9 executed, and the latch still reported that the round was on pace.
4. Linux BBR at the start of the leader window
BAM streams batches to the validator over a TCP connection that uses Linux BBR (Bottleneck Bandwidth and Round-trip propagation time) congestion control [3, 4]. BBR paces sending from its own estimates of bandwidth and minimum round-trip time instead of reacting to packet loss. The connection is nearly idle between leader windows, and we found two kernel paths that could start the next window throttled:
A sub-MSS ACK after idle could push BBR into PROBE_RTT, capping the congestion window at four packets for about 200 ms.
A sub-MSS sent into an empty pipe could make BBR conclude that bandwidth had collapsed.
We shipped a userspace workaround (padding the first batch to 3,000 bytes) and two patches to net/ipv4/tcp_bbr.c [5] in our TEE VM’s kernel. The first keeps BBR’s idle-restart guard in place until a fresh min-RTT sample arrives:

The second treats a send of less than one MSS into an empty pipe as app-limited, so the small first write isn’t mistaken for a bandwidth sample:


What fixing all four changed
Every one of the four stall types was real, and every fix worked. When the scheduler fixes reached production on October 2, the longest Wavefront cleanup dropped from 65.7 ms to 5.9 ms, and main loop gaps over 5 ms went from 54 to zero.

But the fleet’s small-block rate barely moved. We had removed real stalls, and something else was still slowing every thread.
The mysterious TEE VM slow hash rate
Having followed the other leads to the end, we came across a pattern in a metric we couldn’t explain.
TEE VMs emit Agave’s poh-service metric, the time the Proof of History (PoH) hash loop takes per hash. This metric, being CPU-bound, is expected to be flat. On the contrary, every TEE VM showed a square wave, alternating between a fast mode and a mode 7–17% slower. The slow mode typically lasted 45–55 seconds, then the fast mode about 32 seconds, and the cycle repeated around the clock. Nothing in the TEE VM’s workload ran on that cycle.

That surprised us. Our hosts pin every CPU to the performance frequency governor specifically to avoid throughput swings from frequency changes, so a 7–17% swing in vCPU speed shouldn’t have been there.
Turns out the answer was outside the TEE VM, in the Linux kernel on the host side. QEMU backs the TEE VM’s RAM with a memfd, and the host kernel’s kcompactd daemon periodically tries to build contiguous free memory by isolating and migrating pages [7]. We had locked the TEE VM’s memory, and locked pages land on the unevictable LRU [8]. But unevictable doesn’t mean unmovable: with the default vm.compact_unevictable_allowed=1, the host kernel still tries to migrate them.
So on every pass, the host’s kcompactd scanned about 191 million pages, essentially the whole 736 GiB TEE VM. It worked through them for 45–55 seconds, never completed a successful compaction, and backed off for 500 ms << 6 = 32 s before starting again. Each attempt went through the KVM MMU notifier and invalidated the TEE VM’s memory mappings. The TEE VM paid the full cost every cycle, and the host never got any contiguous memory out of it.
The numbers lined up closely. We compared each host’s page-isolation rate with its own TEE VM’s slow-mode fraction over October 2–5, in 30-second intervals. On all ten hosts with PoH data, the two correlated at r = 0.94–0.99. Pairing a host with a different host’s TEE VM gave r ≤ 0.11. TEE VM memory size divided by the host’s average scan rate also predicted the cycle period on 28 of 30 hosts, with an average error of 1.8% and a worst case of 4.4%.

TEE VM memory was already locked with QEMU -overcommit mem-lock=on-fault [9], so the fix was two runtime sysctls on the host side [10]:
vm.compact_unevictable_allowed=0, so locked pages are never isolated for compaction
vm.compaction_proactiveness=0, so there is no background compaction pass
On October 5, between 21:44 and 21:52 UTC, we applied the fix to all 15 warm-spare hosts. The 15 serving hosts stayed unfixed, which gave us a control group.

This canary proved the host’s scan was throttling the TEE VM’s vCPUs. The fixed hosts were spares, though, and weren’t producing blocks, so the canary couldn’t show that the throttling caused small blocks. For that we needed a different experiment.
Results
Proving causation on hosts that were producing blocks
The slow mode is periodic, so it works as a natural experiment. On any unfixed host that was serving traffic, each block was produced during either a slow phase or a fast phase. Everything else is the same between the two: the host, the build, the validators connected to it, the time of day. If the slow mode causes small blocks, small blocks should pile up in the slow phase.
We took the six unfixed hosts that served the most traffic from October 2 to 5. For each 5-second window, we classified the TEE VM as slow or fast using the midpoint between that host’s 10th and 90th percentile hash time. Then we placed each of 293,000 produced BAM blocks on that timeline.

The slow phase covered about 62% of the time and 63% of all blocks. It also held 89% of the small blocks.

A block built during a compaction pass was 4.8 times as likely to be small (95% CI: 3.4–6.9×). The effect showed up on every host individually, ranging from 2.3× on the mildest to 17× on the most affected.
Fleet impact
On October 6 we moved serving traffic to the warm spares, which had received the host-side fix the day before. Before the fix, their TEE VMs had run in the slow mode with a 9–12% PoH spread. After the fix that spread was about 1.2%, and hashing was about 9% faster overall.


The small-block rate fell by about 77%, from about one block in 800 to one in 3,600.
The phase analysis predicts almost exactly that. Before the fix, fast-phase blocks were small 0.031% of the time. After the fix, everything runs in what used to be the fast phase, and the fleet came in at 0.028%. Removing the slow mode accounts for nearly the whole drop.
What the other fixes did:
Scheduler and allocator fixes: removed real tail latency and stopped BAM from slowing replay. They may have lowered small blocks too, but they shipped alongside host changes, so we can’t measure their share.
Linux BBR fixes: closed a real throttling path, confirmed by a reproducer, with no visible change in the fleet small-block rate.
Late-forward rejections (OUTSIDE_LEADER_SLOT): fell from 6.3–7.9% of validator responses to 4.4–4.6% from October 7, so more of each leader window is now usable.
We started out hunting for a scheduler bug, and the largest single win turned out to be two sysctls on the host kernel.
Comparison across clients
With these improvements in place, we compared BAM’s block-packing performance against other clients, measured by the share of small blocks (under 7M CU) produced. We looked at the period before the initial fixes (Sep 16–29) and after they were finalized (Oct 9–20).
In aggregate, the share of small blocks produced by BAM leaders fell from 0.1044% to 0.0325%, a 69% improvement. Over the same period, the rest of the cluster moved in the opposite direction: the non-BAM small-block share rose 73.7%, from 0.2911% to 0.5056%.
As a result, the gap widened significantly. In September, non-BAM leaders produced small blocks at about 2.8x the rate of BAM leaders; by October, that rate was 15.6x.

Conclusion
Every fix in this post started the same way: a per-slot timeline or a profile, a controlled test, then a before-and-after measurement in production. Four of those fixes removed real stalls inside BAM, but we couldn’t tie any of them to a measurable drop in small blocks. The change that mattered most was two lines of host config, outside the TEE VM entirely. We found it only because we kept measuring after fixing the obvious suspects, when the small-block rate still hadn’t moved much.
There is still more work to do. About one BAM block in 3,600 still comes in small. Since writing this, we’ve been working on faster ingress, simulation, bundle filtering, and other performance improvements.
What do you want us to write about next? Let us know by tagging @buffalu__ on x or in #validator-general in Discord.
References
AMD, “AMD SEV-SNP: Strengthening VM Isolation with Integrity Protection and More,� white paper. https://www.amd.com/content/dam/amd/en/documents/epyc-business-docs/white-papers/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf
Anza, “Leader Rotation,� Agave validator documentation. https://docs.anza.xyz/consensus/leader-rotation
N. Cardwell, Y. Cheng, C. S. Gunn, S. Hassas Yeganeh and V. Jacobson, “BBR: Congestion-Based Congestion Control,� ACM Queue 14(5), 2016. https://queue.acm.org/doi/full/10.1145/3012426.3022184
IETF CCWG, “BBR Congestion Control,� Internet-Draft draft-ietf-ccwg-bbr. https://datatracker.ietf.org/doc/draft-ietf-ccwg-bbr/
Linux kernel source, net/ipv4/tcp_bbr.c. https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_bbr.c
jemalloc manual, background_thread option. https://jemalloc.net/jemalloc.3.html
Linux kernel documentation, “Page migration.� https://docs.kernel.org/mm/page_migration.html
Linux kernel documentation, “Unevictable LRU Infrastructure.� https://docs.kernel.org/mm/unevictable-lru.html
QEMU documentation, -overcommit mem-lock=on|off|on-fault. https://www.qemu.org/docs/master/system/invocation.html
Linux kernel documentation, /proc/sys/vm: compact_unevictable_allowed and compaction_proactiveness. https://docs.kernel.org/admin-guide/sysctl/vm.html
Solana, “Reduced Slot Times� (SIMD-0525). https://solana.com/upgrades/reduced-slot-times

Methodology: small block = heartbeat-attributed, BAM-produced block below 7,000,000 cost units, from bam.enriched_block_stats. Phase classification uses poh-service total_hash_time_us / hashes in 5-second windows. One validator that produced only small blocks (68 of 68) from October 2 to 5 is excluded as a validator-side outlier. Confidence intervals are Poisson.(Historical earnings: For 2026Q2 (period ended 2026-06-27), AMD reported basic EPS of 2.26, diluted EPS of 2.22, net income of USD 3.680 billion, and revenue of USD 21.789 billion. Consensus expectations: For 2026Q3, consensus expects EPS of 1.9346 and revenue of USD 13.078 billion.)