vmm: Pin and parallelize memory prefault by saravan2 · Pull Request #8785 · cloud-hypervisor/cloud-hypervisor

saravan2 · 2026-08-25T07:20:47Z

Summary

Memory prefault workers inherit the process affinity, so on multi socket
hosts some of them zero guest memory across the interconnect. Memory zones
are also prefaulted one after another, so a guest with one zone per NUMA
node zeroes each node while the other idles.

This series pins each prefault worker, best effort, to the CPUs of its
zone's host_numa_node, raises the per region worker cap from 16 to 32,
and prefaults all regions concurrently. Snapshot restore prefaults its
regions the same way.

Measurements

On a dual socket HGX B300 host, a guest VM with 240 vCPUs and ~3 TiB of
memory, one 1477 GiB zone per NUMA node on 1 GiB hugepages, prefaulted in
51.8s at 57 GiB/s before this change and in 7.4s after, sustaining
400 GiB/s across the two sockets.

Notes

One barrier spans every region's workers. A thread stack allocation
takes the mmap lock as a writer while each populate call holds it as a
reader for its whole range, so spawning must finish before any populate
starts or the pools convoy down to a few running threads.
Pinning is strictly best effort. A missing sysfs cpulist, a memory only
node, or a cpuset that rejects the affinity call all fall back to the
current unpinned behavior. Pinning failures never fail the boot.
Under landlock, zones with a host_numa_node add a read rule for
/sys/devices/system/node so the prefault workers can read the node
cpulist.
The one exception is a region backing a virtio-mem hotplug area,
which still prefaults inline when it is created at boot.
The RAM region mapping log line now prints before population and a new
prefaulted N memory regions line reports once all regions are
populated.

Tests

Unit tests cover the cpulist to cpu_set_t conversion and the
IntegerList parser.

cargo test -p vmm --features kvm cpulist
cargo test -p option_parser

Integration tests. test_memory_prefault boots two prefault zones, one
pinned via host_numa_node=0 and one unpinned, and asserts the guest
memory is fully resident. test_snapshot_restore_prefault restores with
prefault=on and asserts the completion log line.

scripts/dev_cli.sh tests --integration -- --test-filter test_memory_prefault
scripts/dev_cli.sh tests --integration -- --test-filter test_snapshot_restore_prefault

Commits

vmm: Pin and parallelize memory prefault carries the change and unit
tests for the cpulist to cpu_set_t conversion.
tests: Extend memory prefault coverage pins one zone of the memory
prefault test so the pinned and unpinned worker paths both run in CI, and
adds a snapshot restore variant that restores with prefault=on and
asserts the completion log line.

rbradford

Why are you prefaulting?. Is it because you need huge pages and your configuration predates reserve=on ?

saravan2 · 2026-08-26T19:02:37Z

Why are you prefaulting?. Is it because you need huge pages and your configuration predates reserve=on ?

The usecase is a guest with identity-mapped (no viommu) VFIO devices.
Cloud-Hypervisor has to map all of guest RAM into the IOMMU at boot,
and pinning forces population, so every page gets populated and pinned
during VM boot either way. The only question is which code path pays for it.
Without prefault it is GUP inside VFIO_IOMMU_MAP_DMA, taking the hugetlb
faults serially on the vmm thread. With prefault the same population
runs just before that as a parallel NUMA local phase, 16.6s for the
~3 TiB guest in the commit message, and the DMA map then only pins
already present pages.

The boot sequence for that guest:

device 0:     open device
              container created (first VFIO device needs one)
              ioctl VFIO_IOMMU_MAP_DMA  region 1 (3 GiB low)        one per RAM region,
              ioctl VFIO_IOMMU_MAP_DMA  region 2 (1474 GiB, node 0) the loop in
              ioctl VFIO_IOMMU_MAP_DMA  region 3 (1477 GiB, node 1) add_vfio_device,
                                                                    GUP pins each range
              region info ioctls, allocate BAR GPAs
              mmap BARs, p2p MAP_DMA of its BARs, kvm memslots

device 1..N:  open device (container exists, no RAM ioctls)
              region info, allocate BAR GPAs, mmap BARs, p2p MAP_DMA, memslots

Without prefault the three RAM ioctls serially absorb the ~3 TiB of
hugetlb fault work on the vmm thread. With prefault they find every
page present and only pin.

reserve=on solves a different problem. It guarantees the hugetlb pool
reservation so a later fault cannot fail, but it does not populate, so the
fault cost has to land somewhere else.

yamahata · 2026-08-26T21:50:49Z

+                handles.push(handle);
+            }
+
+            for handle in handles {


We don't have to wait here. This join can be deferred until vCPU is about to run first time, and let the main thread go.

Wow. This is awesome.

This change requires more code than what it appears.

With current implementation :

t=0 workers start populating boot thread WAITS at the join ──────── 7.4s t=7.4 device 0: MAP_DMA of RAM → pages all present → pin only, fast BARs, devices 1..N, memslots ... ~24s t≈32 vCPUs run

When we defer the join to happen before the vcpu start phase

t=0 workers start populating; boot thread proceeds immediately t≈0.3 device 0: **MAP_DMA of all RAM** regions issued on the vmm thread → the ioctl must pin EVERY page of the region → pinning a page requires the page to exist → most pages don't exist yet, the workers are still zeroing → so the vmm thread sits INSIDE the ioctl until population finishes (GUP either faults missing pages itself or blocks on the fault mutex of whichever worker owns them) t≈7.4+ MAP_DMA finally returns BARs, devices 1..N ... ~24s t≈32+ vCPUs run ← same as today

As you see we wont shave off ~7 seconds by deferring the join
when MAP_DMA of RAM regions will convoy with our prefault workers
by contending the mmap lock. The prefault workers can hold the
read lock for a long time as each thread can fault >= 1 hugepage.

The proper way to shave off the ~7 seconds would require us
to change the current code and defer RAM MAP_DMA after
our join, before the vcpus run.

t=0 workers start populating boot thread proceeds: region info, BAR GPAs, BAR mmaps, p2p maps, memslots for all 15 devices ... real work, genuinely overlapping the 7.4s of zeroing t=7.4 populate done (join, now placed late — before vCPUs run) t≈7.4+ RAM MAP_DMA — moved here, pages all present, pin only, fast t≈25 vCPUs run ← the ~7s win

I also think we would require our prefault workers to fault 1 hugepage
at a time instead of the slice range we have

# current madvise(slice.addr, slice.len, MADV_POPULATE_WRITE); # proposed for chunk in slice.split(1 GiB) { madvise(chunk.addr, chunk.len, MADV_POPULATE_WRITE); }

I believe we should do this in a new PR.
Let me know your thoughts.

Prefault workers inherit the process affinity, so on multi socket hosts some of them zero guest memory across the interconnect, and memory zones are prefaulted one after another, so a guest with one zone per NUMA node zeroes each node while the other idles. Pin each worker, best effort, to the CPUs of its zone's host NUMA node, drop the fixed cap of 16 workers in favor of the host's available parallelism, and prefault all regions concurrently. Snapshot restore prefaults its regions the same way. The one exception is a region backing a virtio-mem hotplug area, which still prefaults inline when it is created at boot. Under landlock, zones with a host_numa_node add a read rule for /sys/devices/system/node so the cpulist stays readable. One barrier spans all workers across regions, since a thread stack allocation takes the mmap lock as a writer during another region's populate read holds and a queued writer blocks later readers, which would convoy every pool down to a few running threads. The RAM region mapping log line now prints before population and a new line reports once all regions are prefaulted. On a dual socket HGX B300 host, a large guest VM with 240 vCPUs and ~3 TiB of memory, one 1477 GiB zone per NUMA node on 1 GiB hugepages, prefaulted in 51.8s at 57 GiB/s before this change and in 7.4s after, sustaining 400 GiB/s across the two sockets. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Saravanan D <saravanand@crusoe.ai>

The memory prefault test now gives one zone a host_numa_node while the other zone keeps none, so the pinned and the default unpinned worker paths both run in the same boot. A new snapshot restore variant restores with prefault enabled and asserts the completion log line, covering the deferred restore prefault path. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Saravanan D <saravanand@crusoe.ai>

saravan2 force-pushed the inc-prealloc-thread branch 2 times, most recently from d5fddac to 0d80801 Compare August 26, 2026 07:53

saravan2 marked this pull request as ready for review August 26, 2026 07:54

saravan2 requested a review from a team as a code owner August 26, 2026 07:54

saravan2 self-assigned this Aug 26, 2026

rbradford requested changes Aug 26, 2026

View reviewed changes

Comment thread vmm/src/memory_manager.rs Outdated

Comment thread vmm/src/memory_manager.rs