vmm: hand off the guest-RAM userfaultfd to an external manager - #8657
vmm: hand off the guest-RAM userfaultfd to an external manager#8657kiryl wants to merge 9 commits into
Conversation
|
@kiryl You should rebase to drop the merge commit |
Add the kernel UAPI constants needed by the upcoming uffd-handoff support: - UFFDIO_REGISTER_MODE_WP write-protect register mode (was unbound) - UFFD_FEATURE_WP_HUGETLBFS_SHMEM enable WP on shmem/hugetlbfs backings - UFFD_FEATURE_WP_UNPOPULATED WP fires on unmapped pages too - UFFD_FEATURE_WP_ASYNC kernel-side async WP fault resolution Generalise uffd::register() to take the desired register mode as a parameter instead of hardcoding UFFDIO_REGISTER_MODE_MISSING; update the snapshot-restore caller (the only existing one) to pass MISSING explicitly. No functional change. Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:Opus-4.8
A `UffdHandoffSpec` value carries the parsed bits of a `|`-separated
set of UFFD-mode tokens, e.g. `"MISSING|WP|WP_ASYNC"`. Two kernel
surfaces are deliberately conflated into one user-facing string:
- register modes (MISSING, WP)
→ `UFFDIO_REGISTER` `mode` field
- features (WP_UNPOPULATED, WP_ASYNC)
→ `UFFDIO_API` `features` field
They live in different ioctls and the async features are also
toggleable later via `UFFDIO_SET_MODE`, but for the handoff use case
they describe one decision ("how do I want this uffd configured?"),
the token namespaces are disjoint, and the only features we accept
here are the ones tightly coupled to the WP register mode — so a
single spec is the ergonomic choice.
The struct itself stores the parsed bits, but FromStr / Display /
serde via `try_from = "String"` / `into = "String"` mean the
on-the-wire and on-the-CLI form is the original token string.
Tokens are case-sensitive (matching the kernel UAPI macro names).
All validation surfaces at deserialise time, never at handoff:
- unknown tokens
- no register-mode token (need at least one of MISSING/WP)
- feature token without its matching register mode
(WP_UNPOPULATED / WP_ASYNC require WP)
UffdHandoffSpec is consumed by the boot-time and runtime
uffd-handoff entry points landed in the next patches: a single typed
object propagates from config / HTTP body all the way down to the
kernel call.
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:Opus-4.8
Add an external memory-control plane. A manager process listens on a
Unix socket; during MemoryManager construction the VMM dials it,
creates a userfaultfd registered over all guest RAM as configured, and
hands the fd off via SCM_RIGHTS with a JSON descriptor of the
registered regions. The manager ACKs with one byte once it is polling,
which closes the boot-before-handoff race: by the time
MemoryManager::new returns, every guest access faults into the manager.
Config is one optional MemoryConfig sub-struct, both fields required:
uffd_handoff = { socket = "...", mode = "MISSING|WP|WP_ASYNC" }
The CLI flattens it to uffd_handoff_socket= / uffd_handoff_mode= (which
must appear together). There is no default mode; the caller picks what
to register. mode deserialises through UffdHandoffSpec, so bad tokens
fail at parse time.
UFFD_API features are derived from the register modes and the backing:
MISSING on shmem/hugetlbfs -> MISSING_{SHMEM,HUGETLBFS}; WP on
shmem/hugetlbfs -> WP_HUGETLBFS_SHMEM.
Wire protocol: each message is <u32 LE len><JSON body> in one sendmsg
so the SCM_RIGHTS fd attaches to the right message; the body is a
"type"-tagged enum. The Handoff message carries the protocol version,
the VMM pid, the register mode, and the region list (guest_phys_addr,
host_virt_addr, size, backing_file_offset) sorted by guest_phys_addr.
Only the uffd is passed in SCM_RIGHTS.
A manager that needs direct backing-store access (e.g. eviction) opens
/proc/<vmm_pid>/map_files/<host_va_start>-<host_va_end> rather than
receiving a backing fd, which avoids the TOCTOU in passing an fd or a
path. This needs the same CAP_SYS_PTRACE-equivalent already required to
read /proc/<vmm_pid>/pagemap, and requires the manager to share the
VMM's PID namespace (see UffdRegionInfo).
The handshake runs synchronously on the VMM thread (the
runtime-attach path also holds the MemoryManager mutex across it) and
each send/recv is bounded by UFFD_HANDOFF_TIMEOUT, so a manager that
stalls mid-handshake can't wedge the VMM for long. Moving the handshake
off the VMM thread (async attach) is left as future work.
CH keeps its own dup of the uffd for the VM's lifetime, so the
registrations survive a manager exit: faulting vCPUs block (CH never
resolves faults itself) rather than losing data.
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:Opus-4.8
After the initial handoff, keep the manager connection, the uffd, and the negotiated register mode so ACPI memory hotplug can extend the handoff. On hotplug_ram_region the new VMA is registered against the same uffd and an `add_region` notification is sent to the manager (a new variant of the handoff wire enum, with no SCM_RIGHTS payload — the manager already holds the uffd). Any I/O failure drops the manager connection and fails the hotplug, rather than silently leaving the new region invisible to the manager. virtio-pmem is out of scope: its VMAs live outside self.memory_zones (DeviceManager builds them directly), so they are covered by neither the initial handoff nor add-region; a TODO in collect_uffd_regions notes this. Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:Opus-4.8
Allow an external manager to attach to an already-running VM (no
boot-time config) via a new PUT endpoint:
/api/v1/vm.uffd-attach
{ "handoff_socket": "/path/to/sock", "mode": "WP|WP_ASYNC" }
Same handoff protocol as the boot-time path: CH dials the manager,
sends the framed Handoff message via SCM_RIGHTS, and waits for the ACK.
mode is the same typed UffdHandoffSpec used in config, so unknown
tokens are rejected at deserialise time (400, not a 5xx mid-handoff).
The endpoint requires the VM to be Running; attach during
Created/Paused/Shutdown/BreakPoint is rejected (Shutdown can race the
teardown of the VMAs being registered).
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:Opus-4.8
The external manager may crash and restart. CH already keeps its dup of the uffd for the VM's lifetime, so the registrations survive and faults block (no data loss) while it is gone. Make do_uffd_handoff idempotent: a re-attach with the same mode re-sends the existing uffd fd and current region set (resume) rather than erroring. The manager side is identical to a first attach, so it need not know it is resuming; on reconnect it drains the queued fault backlog and wakes the stalled vCPUs. No new uffd is created and no VMA is re-registered. A re-attach with a *different* mode is rejected (changing the mode would be a re-key, not implemented). Exactly one live manager at a time is the orchestrator's invariant: once the fd is sent via SCM_RIGHTS, CH cannot revoke a manager's dup, and two readers racing the uffd queue corrupts fault handling. Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:Opus-4.8
Wire the vm.uffd-attach HTTP endpoint into ch-remote:
ch-remote --api-socket ... uffd-attach --socket <path> --mode <modes>
It builds the { handoff_socket, mode } body and PUTs it via the generic
simple_api_command helper, mirroring the other vm.* subcommands.
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:Opus-4.8
Document the userfaultfd handoff feature: the --memory uffd_handoff_* boot options and the vm.uffd-attach runtime endpoint (plus its OpenAPI schema reference), including the same-PID-namespace requirement for the manager's /proc/<vmm_pid> access. Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:Opus-4.8
Integration tests for the external userfaultfd handoff, with the shared
test_infra helpers they need (framed SCM_RIGHTS recv, PAGEMAP_SCAN
wrappers, a minimal fault handler).
- test_uffd_handoff drives the boot path end to end: the handshake, the
regions JSON, and a MISSING fault resolved with UFFDIO_ZEROPAGE.
- test_uffd_writeset_{anon,shmem,anon_thp,shmem_thp} attach at runtime
with mode=WP|WP_ASYNC and exercise the WP-async write-set interface a
manager uses for tiering/eviction: PAGEMAP_SCAN classifies guest RAM
into the hot set (PAGE_IS_WRITTEN) and the reclaimable complement,
plus non-resident holes; PM_SCAN_WP_MATCHING re-arms tracking and the
kernel resolves writes silently. The THP variants assert guest RAM is
PMD-backed (PAGE_IS_HUGE) and survives uffd-WP registration.
- test_uffd_resume attaches a manager, simulates its death (closes the
manager's uffd dup and the socket), resumes over a fresh socket, and
verifies the registration is still live via a WP-async write-set cycle.
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:Opus-4.8
sboeuf
left a comment
There was a problem hiding this comment.
One general comment, AI is way too verbose on this patchset, could you please make the comments shorter and more concise? And remove a bunch of them when the code is self explanatory :)
| /// at handoff time. | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)] | ||
| #[serde(into = "String", try_from = "String")] | ||
| pub struct UffdHandoffSpec { |
There was a problem hiding this comment.
Could we call it UffdConfig or UffdMode instead? It's more a configuration than a spec, and also it's not specific to the handoff case.
| #[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize, serde::Deserialize)] | ||
| #[serde(into = "String", try_from = "String")] | ||
| pub struct UffdHandoffSpec { | ||
| pub register: u64, |
There was a problem hiding this comment.
I'd call it mode or register_mode, but I find register slightly misleading.
| "MISSING" => register |= UFFDIO_REGISTER_MODE_MISSING, | ||
| "WP" => register |= UFFDIO_REGISTER_MODE_WP, | ||
| "WP_UNPOPULATED" => features |= UFFD_FEATURE_WP_UNPOPULATED, | ||
| "WP_ASYNC" => features |= UFFD_FEATURE_WP_ASYNC, |
There was a problem hiding this comment.
@rbradford should we accept lower case notation as well? or actually lower case only?
There was a problem hiding this comment.
can do either. your call.
if you want it to accept case-insensitive, what case do we want when we emit? I went with upper case because it is how kernel API defines it.
There was a problem hiding this comment.
Let's wait for @rbradford's feedback on this one, as I'm not sure what is best.
| .add("thp"); | ||
| .add("thp") | ||
| .add("uffd_handoff_socket") | ||
| .add("uffd_handoff_mode"); |
There was a problem hiding this comment.
I think we should call this option uffd_mode as it applies to the memory no matter if we perform a uffd handoff or not.
The other option uffd_handoff_socket should keep this name since it is specific to the handoff feature.
There was a problem hiding this comment.
ack, uffd_handoff_mode -> uffd_mode
|
|
||
| pub type MemoryZones = HashMap<String, MemoryZone>; | ||
|
|
||
| /// Metadata about a memory region registered for the uffd handoff, |
There was a problem hiding this comment.
There are massive comments added by the AI here and through this file, could you please trim it down.
| if let Some(handoff) = &config.uffd_handoff { | ||
| memory_manager.do_uffd_handoff(&handoff.socket, handoff.mode)?; | ||
| } |
There was a problem hiding this comment.
It's kinda weird to call into this after we've created the MemoryManager. I mean either we call memory_manager.do_uffd_handoff() from vm.rs, or we call this before we construct the memory manager and we don't make this function a method of the MemoryManager.
| #[serde(tag = "type", rename_all = "snake_case")] | ||
| enum UffdHandoffMessage<'a> { | ||
| Handoff { | ||
| version: u16, |
There was a problem hiding this comment.
Do we want to start taking care of version compatibility?
There was a problem hiding this comment.
it's just my attempt at future-proofing the wire format, following the CURRENT_PROTOCOL_VERSION idiom in vm-migration::protocol — declare a version, keep changes within it additive, bump it if the shape or the meaning of a field changes.
The difference from migration is that there CH is on the receiving end, so it can validate: Request decodes the sender version and rejects anything outside supported_protocol_versions(). Here CH is the sender, so there's nothing for it to check — the field only lets the manager refuse a message it doesn't understand, and that check lives in the manager rather than in CH.
So it buys us the option of evolving the format without buying any code today. Happy to drop it if you'd rather not carry a field nothing acts on.
| enum UffdHandoffMessage<'a> { | ||
| Handoff { | ||
| version: u16, | ||
| vmm_pid: u32, |
There was a problem hiding this comment.
Just curious, what is this PID used for? A unique ID?
There was a problem hiding this comment.
Not an ID — the manager needs it to reach the VMM through /proc. Two things depend on it: /proc/<vmm_pid>/pagemap for PAGEMAP_SCAN, which is how a WP/RWP manager actually samples the write-set or access-set (the uffd lets it arm tracking, pagemap is how it reads the result back), and /proc/<vmm_pid>/map_files/ to get an fd for the backing memfd so it can read pages out and punch holes. It's also what makes the same-PID-namespace requirement in the docs necessary.
There was a problem hiding this comment.
and /proc/<vmm_pid>/map_files/ to get an fd for the backing memfd so it can read pages out and punch holes
Oh... I don't think we should do something like this. We usually share the memfds backing the guest RAM from the VMM to a daemon by sending these fds through ancillary data. And I think similarly for the pagemap, you should be able to convey that information through the protocol between the VMM and the daemon.
| /// Register-mode bits the active uffd was configured with — used | ||
| /// when issuing `UFFDIO_REGISTER` against newly hot-added VMAs so | ||
| /// they match the rest of the registered set. | ||
| uffd_register_mode: Option<u64>, |
There was a problem hiding this comment.
Can we group all these uffd related fields inside a single one?
There was a problem hiding this comment.
Ack. Folding uffd_attached, uffd_handoff_sock, uffd_handoff_fd and uffd_spec into a single Option<...>.
|
|
||
| /vm.uffd-attach: | ||
| put: | ||
| summary: Attach an external userfaultfd manager to a running VM. |
There was a problem hiding this comment.
Do you actually need this for your use case. The way I was thinking about the uffd handoff case, I assumed the caller/user would know before starting the VM if he wants to run a uffd handoff daemon or not. That means the daemon should be running before the VM is being started, and by passing the uffd_handoff_socket through the VM's config, the VM would automatically try to connect to the daemon. I believe this would simplify a lot the logic and would not require a dedicated HTTP endpoint.
/cc @rbradford
There was a problem hiding this comment.
Agreed that boot-time config is the primary path and it's what we use today. The endpoint is for attaching once the guest reaches steady state, and for handing the VM back to a restarted manager — upgrades as much as crashes. Full reasoning in #8657 (comment).
Fair enough, will do. Let me comment on other feedback first. |
|
Four of the review comments are really one question — why CH retains any uffd state, and why there's a runtime attach endpoint — so answering them together here rather than four times inline. Why CH keeps its own dup of the uffd Because dropping it turns a manager crash into silent guest data corruption. The registration only lives as long as some reference to the uffd does. If CH hands the fd over and keeps nothing, the manager's exit closes the last one and the memory stops being registered with userfaultfd at all: any vCPU already blocked on a fault is released, and later accesses are handled by the kernel as ordinary page faults rather than being reported to anyone. If the manager had already reclaimed a page, having written it out and punched a hole in the shmem file, that ordinary fault refills the hole with newly allocated zeroed memory. The guest reads zeros over its own data and nothing reports a problem. That's data corruption. With CH holding a dup, the registration outlives the manager and a faulting vCPU simply blocks. The VM stalls until a manager comes back, but nothing is lost. CH deliberately never resolves a fault itself — it has no idea what the page should contain, and guessing is exactly the corruption above. So what is the re-attach path for Not reboot. It's manager restart. Since the registrations are still live, a replacement manager doesn't need a new uffd or a re-registration — it needs the same fd back, plus the current region set. That's what makes the retained state necessary: calling And why the HTTP endpoint You're right that the first attach is knowable before boot, and that is how we use it today — the daemon is already listening, and CH dials it and blocks on the handshake. Happy to treat that as the primary path. Two things still want an attach after boot, though. The first is that working-set tracking is much more useful once the guest has settled. During boot the guest touches almost all of its memory — init, services, page cache warm-up — so everything looks hot, there is nothing cold to reclaim yet, and you pay registration and fault overhead through the most memory-intensive phase of the VM's life to collect samples you would throw away. Attaching once the workload reaches steady state is the natural way to run this, rather than attaching at boot and discarding the first several intervals. The second is that the manager need not be tied to a single VM. One daemon can drive several VMs on a host, so its lifecycle isn't 1:1 with any of them: VMs come and go underneath it, and it may need to restart — to pick up a new version at least as often as it crashes — at a moment that has nothing to do with any particular VM. If the only way to hand over the fd is |
Thanks that's much clearer indeed.
I wanted to say that we could avoid the dedicated attach endpoint by having the VMM trying to reconnect to the handoff daemon after a lost connection, but that wouldn't encompass your first point (more useful tracking after the guest has booted). @rbradford I'd like your opinion on this, but the logic behind it is sound. |
Hand off Cloud Hypervisor's guest-RAM userfaultfd to an external process, so demand paging and memory-management policy can live outside the VMM. CH creates the uffd, registers guest RAM in a caller-specified mode, sends the fd plus a description of the regions over a Unix socket, and then resolves no faults and makes no policy decisions for the rest of the VM's life.
Discussed and agreed in #8644 (and #8554 before it). Supersedes #8636, which was closed pending that process.
My own consumer is out-of-process guest memory reclaim: a manager tracks the working set with uffd RWP, writes cold pages to tiered storage, punches them out of the backing file, and services the resulting MISSING faults itself. Nothing in this PR depends on RWP, though — it needs only
MISSING,WP,WP_UNPOPULATEDandWP_ASYNC, all available on shipping kernels. RWP support is a small follow-up (one commit, onuffd-rwp) that I'll post once the kernel side reaches a release.What's here
vmm: extend userfaultfd bindings for WP register modes— UAPI constants and theUffdHandoffSpecmode/feature split.vmm: add UffdHandoffSpec parser for UFFD-mode token strings—MISSING|WP|WP_ASYNC-style specs, validated at the config/API boundary so a bad token fails there rather than mid-handshake.vmm: add boot-time userfaultfd handoff over Unix socket—--memory ...,uffd_handoff_socket=<path>,uffd_handoff_mode=<tokens>, the wire protocol, and the handshake.vmm: notify the uffd manager of hot-added memory regions— ACPI hotplug extends the handoff via anadd_regionmessage.vmm: add vm.uffd-attach HTTP API for runtime uffd handoff— attach to an already-running VM with no boot-time config.vmm: resume uffd handoff after manager restart— make the handoff idempotent so a restarted manager can re-attach.ch-remote: add vm.uffd-attach subcommanddocs: document external userfaultfd handofftests: add uffd handoff integration testsEach step builds and is independently reviewable. Happy to split this into a smaller series if you'd rather review it in pieces — the natural cut is 1–3 (mechanism), 4–7 (runtime surface), 8–9.
Protocol
Length-prefixed JSON, one
sendmsgwith the uffd attached viaSCM_RIGHTS, then a one-byte ACK from the manager — the ACK closes the boot-before-handoff race, so returning from the handshake means the manager is polling. The message carries a version, the VMM pid, the negotiated mode, and the region list: guest physical address, host virtual address, size, backing-file offset. It deliberately says where the pages are and nothing about what should happen to them.The manager reads
/proc/<vmm_pid>/pagemapforPAGEMAP_SCAN, so it must share a PID namespace with the VMM. That is also how it reaches the backing file, viamap_files— I opted not to pass the memfds overSCM_RIGHTSfor now, but it's an easy change if reviewers prefer it.CH keeps its own dup of the uffd for the life of the VM. So kernel-side registrations survive a manager crash, and while the manager is gone faulting vCPUs simply block — CH never resolves faults itself, so there is no zero-fill and no data loss. Exactly one live manager at a time remains the orchestrator's invariant: once the fd is sent CH cannot revoke it, and two readers racing the uffd queue corrupts fault handling. CH does a best-effort liveness probe before allowing a resume, as a guard rail rather than enforcement.
On converging with the offload daemon
@sboeuf — on your point about avoiding duplicate designs, and possibly having the live migration protocol carry the uffd so
PageFaultisn't needed: I think that's the right direction and I've tried to shape this so it stays open rather than settling it here.What already points that way: the region description carries guest physical address and backing-file offset, not just host VA, so a restore policy can map a fault to snapshot content without any protocol addition. Register mode and features are negotiated explicitly rather than implied, so a
MISSING-only restore consumer and aWP/RWPtracking consumer use the same handshake. And the handshake has a version field that only grows additively.What's genuinely missing for convergence is a manager-to-CH signal. A restore has a completion point — once the guest is fully resident, CH needs to know so the
restoring()gating from #8525 can clear and migration/snapshot become allowed again. CH cannot derive that; only the manager knows. There's also no detach path today, because a VM-lifetime handoff never needs one while a restore does. Both are small additions and I'd rather add them with a real consumer than speculatively.So my suggestion would be: land the mechanism here without touching
UffdMemorySource,PageFault, or the offload daemon, then do the convergence as a follow-up that adds the completion/detach signal and movesSocketUffdMemorySourceonto the handoff. That keeps this PR reviewable and doesn't pre-judge the design of the migration side. Happy to take it in the other order if you'd prefer the converged shape up front.Worth noting the convergence should be a straight win on the fault path: today
SocketUffdMemorySourcehas CH relaying — read the fault, request bytes,UFFDIO_COPY— where a handoff has the daemon resolve directly, one less process hop per fault.Scope
No existing behaviour changes.
UffdMemorySource, thePageFaultprotocol, the offload daemon and postcopy paths are untouched; a handoff and an internal on-demand restore are mutually exclusive on the same VM. virtio-pmem regions aren't covered, since DeviceManager builds those VMAs outsidememory_zones— there's aTODOatcollect_uffd_regions. Device backends that reach guest RAM outside CH's own mapping need separate treatment, and the reclaim policy in particular constrains which of them can be used; I left the detail out of #8644 but can go into it if that's useful.Testing
cargo testplus integration tests that need no special CI hardware — just/dev/userfaultfd(or the syscall) andPAGEMAP_SCAN:test_uffd_handoff— boot path end to end: handshake, region JSON, a MISSING fault resolved withUFFDIO_ZEROPAGE.test_uffd_writeset_{anon,shmem,anon_thp,shmem_thp}— runtime attach withWP|WP_ASYNC, exercising the write-set interface a manager actually uses:PAGEMAP_SCANsplits guest RAM into the hot set and the reclaimable complement,PM_SCAN_WP_MATCHINGre-arms, and the THP variants assert guest RAM stays PMD-backed across uffd-WP registration.test_uffd_resume— attach, simulate manager death, resume over a fresh socket, confirm the registration is still live.LLM assistance is disclosed with
Assisted-by:trailers on each commit, per CONTRIBUTING.md.