UVM in the Guest

UVM in the Guest

Scope. What UVM (nvidia_uvm.ko) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: ../docs/03-architecture/04-uvm.md; where this page and the docs tree disagree, the docs tree wins.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] observed with the stock guest driver; [wip] partially working, software bridge in charge; [gated] implemented default-off; [target] planned end state.

1. What UVM is

UVM provides unified memory and GPU-driven page-fault handling. In practice it is a second GPU driver layered on the RM, owning GPU address spaces and running its own memory-management work on the GPU.

Component	Role
VA spaces (uvm_va_space)	per-process GPU address spaces
Channel managers	pools of GPU channels (CE, SEC2, GR as needed) for UVM's work
Push buffers / GPFIFO	UVM builds and submits its own pushes
Semaphores and trackers	UVM waits on GPU-written semaphores to track completion
Page trees	GPU page tables for its VA spaces
Fault handling	fault buffer plus notification; UVM resolves and replays/cancels faults

For CUDA, UVM is entered early: cuInit → driver init → UVM_REGISTER_GPU. If UVM cannot register the GPU, CUDA does not start — which is why "UVM advances" is the project's primary correctness metric on the way to CUDA.

2. The registration path (traced end to end)

libcuda: UVM_REGISTER_GPU ioctl (0x25)
  uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
    nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
      nvUvmInterfaceDeviceCreate          (RM object creation)
      uvm_hal_init_gpu / get_gpu_caps     (CE caps)
      uvm_ats_add_gpu / uvm_gpu_init_isr  (fault/interrupt plumbing)
    init_channel_manager
      create semaphore pool (sysmem)
      create channels per CE (GPFIFO + pushbuffer + semaphores)
      channel_init() -> first push -> semaphore release -> wait
    uvm_page_tree_init                      (page tables for the VA space)

Each stage has a distinct failure signature — see Driver-Lifecycle-Field-Guide.md.

3. What UVM requires from the "GPU"

Requirement	Detail	Status
GPU identity	must match the UUID from RM	[live]
CE capabilities	per-engine support bits; UVM requires supported && !grce && sysmem for CPU_TO_GPU / GPU_TO_CPU channels	[live] caps accepted; [gated] copies forwarded to the physical copy engine
Channel creation	RM channel objects with GPFIFO/userd/instance parameters	[live]
Pushes executed	first action on a new channel is an init push plus a semaphore-tracked wait	[wip] software bridge executes memsets, copies and releases; [gated] CE-class copies forwarded
Page trees	UVM initializes page trees with its own pushes; server-owned ranges must be populated by us	[wip]
Fault delivery	fault buffer plus notification; replay/cancel semantics	[gated] backend-side observation/relay; guest contract [target]
Completion semantics	tracker values must advance exactly	[wip]

Current frontier. Context creation and the full CUDA driver-API smoke pass (ALL OK, copies matched) run end to end. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored releases and a fault relay. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed; context validity at dispatch is ruled out (Execution-Frontier.md). The remaining UVM work is completion translation and the guest-side fault replay/cancel contract.

4. The discovery that defines the work: how UVM waits

UVM's tracker is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps its state:

uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
    releasing value 1

Mechanically, UVM built a push ending in a semaphore release — a method sequence (CE channel class, e.g. TURING_DMA_COPY_A) that sets an address and a payload and launches a DMA operation to write the payload to memory — then submitted it and waits for the payload to appear. On real hardware the copy engine writes it; on our device nobody does unless the emulation does. The push we found in guest RAM decodes as:

20010000 0000c5b5   set-object (TURING_DMA_COPY_A)
2001a000 0000c076   set-object (GP100_UVM_SW)
20038090 00000001   incrementing-method header: count 3, addr 0x240
21000000 00000001   lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c   LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD

So the emulation's job for this push is: write payload to the memory at the semaphore's GPU VA, which requires a GPU-VA → guest-PA translation. The host does this through the learned VA→PA map: while the channel's GPFIFO has pending entries it reads each push, decodes the method stream and runs the work (page-table memsets, copies, semaphore releases). When a push page cannot be resolved — or a channel consumes entries without any release landing — it falls back to content: guest RAM is scanned for release groups and pending payloads are applied forward-only. A release is applied only when payload > current, so recycled push buffers and stale copies are harmless. One pitfall observed live: a semaphore-page VA can flap between two physical pages across map rebuilds; pinning the validated translation fixes the stalled wait.

5. The memory model UVM assumes

• UVM's semaphore pool is system memory (UVM_APERTURE_SYS), one 4 KiB page per channel manager, addressed as 4-byte slots (1024 per page).
• Each page has a GPU VA (chosen by the client's RM) and a CPU VA (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
• Both addresses are recorded in UVM's structures in guest memory — the key that makes translation possible without extra help from the guest.
• The CPU VA is a direct-map address, so PA = cpu_va - page_offset_base (verified live: writing the payload there completes the push).
• Values are per channel and monotonic (1, 2, 3 …); UVM's own check (UVM_GPU_SEMAPHORE_MAX_JUMP) detects and reports wrong values.
• The client's page tables for server-owned ranges are empty chains: the client hands us the physical addresses of its page-table levels and expects the server to own the lower levels (GSP-and-GSP-RM.md).

6. Page trees and faults

uvm_page_tree_init initializes a VA space's page tables with pushes on UVM's own channels (memops and page-table writes; the copy engine executes them on real hardware). Fault handling is event-driven: UVM programs a fault buffer (replayable and non-replayable; sizes from INTERNAL_GMMU_GET_STATIC_INFO, answered by the emulation) and, on a fault, reads the entry, populates the page table and replays or cancels the fault with a MEM_OP push. Reconciling the guest's memory model with the real GPU's here is the deepest remaining work.

7. How to observe UVM

In the guest, UVM prints nothing by default:

echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints

Messages then appear with the nvidia-uvm: prefix. Known-good markers, in order: uvm_gpu_retain_by_uuid → channel manager created → uvm_page_tree_init pushes → UVM_REGISTER_GPU returns and CUDA proceeds. Failure signatures characterized live:

Message	Meaning
Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED	CE caps wrong or missing
stuck waiting plus Tracker entry ... channel_init()	the push's semaphore was never written
Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101	a wrong payload was written; UVM caught it

8. Why UVM is the perfect correctness oracle

UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling and error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU.

9. Related documentation

• ../docs/03-architecture/04-uvm.md — canonical chapter (this page's source).
• Guest-Stack-and-CUDASW.md, GSP-and-GSP-RM.md — the guest stack and the RM objects UVM consumes.
• CUDA-Userspace-Driver.md — the userspace half of the contract; Execution-Plane.md, Execution-Frontier.md — how UVM's pushes execute, and the current wall.
• ../docs/04-implementation/01-known-knowns-and-unknowns.md — server-owned page tables and the fault contract.