UVM in the Guest

Scope. What UVM (nvidia_uvm.ko) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: ../docs/03-architecture/04-uvm.md; where this page and the docs tree disagree, the docs tree wins.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] observed with the stock guest driver; [wip] partially working, software bridge in charge; [gated] implemented default-off; [target] planned end state.

1. What UVM is

UVM provides unified memory and GPU-driven page-fault handling. In practice it is a second GPU driver layered on the RM, owning GPU address spaces and running its own memory-management work on the GPU.

Component	Role
VA spaces (`uvm_va_space`)	per-process GPU address spaces
Channel managers	pools of GPU channels (CE, SEC2, GR as needed) for UVM's work
Push buffers / GPFIFO	UVM builds and submits its own pushes
Semaphores and trackers	UVM waits on GPU-written semaphores to track completion
Page trees	GPU page tables for its VA spaces
Fault handling	fault buffer plus notification; UVM resolves and replays/cancels faults

For CUDA, UVM is entered early: cuInit → driver init → UVM_REGISTER_GPU. If UVM cannot register the GPU, CUDA does not start — which is why "UVM advances" is the project's primary correctness metric on the way to CUDA.

2. The registration path (traced end to end)

libcuda: UVM_REGISTER_GPU ioctl (0x25)
  uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
    nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
      nvUvmInterfaceDeviceCreate          (RM object creation)
      uvm_hal_init_gpu / get_gpu_caps     (CE caps)
      uvm_ats_add_gpu / uvm_gpu_init_isr  (fault/interrupt plumbing)
    init_channel_manager
      create semaphore pool (sysmem)
      create channels per CE (GPFIFO + pushbuffer + semaphores)
      channel_init() -> first push -> semaphore release -> wait
    uvm_page_tree_init                      (page tables for the VA space)

Each stage has a distinct failure signature — see Driver-Lifecycle-Field-Guide.md.

3. What UVM requires from the "GPU"

Requirement	Detail	Status
GPU identity	must match the UUID from RM	`[live]`
CE capabilities	per-engine support bits; UVM requires `supported && !grce && sysmem` for CPU_TO_GPU / GPU_TO_CPU channels	`[live]` caps accepted; `[gated]` copies forwarded to the physical copy engine
Channel creation	RM channel objects with GPFIFO/userd/instance parameters	`[live]`
Pushes executed	first action on a new channel is an init push plus a semaphore-tracked wait	`[wip]` software bridge executes memsets, copies and releases; `[gated]` CE-class copies forwarded
Page trees	UVM initializes page trees with its own pushes; server-owned ranges must be populated by us	`[wip]`
Fault delivery	fault buffer plus notification; replay/cancel semantics	`[gated]` backend-side observation/relay; guest contract `[target]`
Completion semantics	tracker values must advance exactly	`[wip]`

Current frontier. Context creation and the full CUDA driver-API smoke pass (ALL OK, copies matched) run end to end. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored releases and a fault relay. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed; context validity at dispatch is ruled out (Execution-Frontier.md). The remaining UVM work is completion translation and the guest-side fault replay/cancel contract.

4. The discovery that defines the work: how UVM waits

UVM's tracker is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps its state:

uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
    releasing value 1

Mechanically, UVM built a push ending in a semaphore release — a method sequence (CE channel class, e.g. TURING_DMA_COPY_A) that sets an address and a payload and launches a DMA operation to write the payload to memory — then submitted it and waits for the payload to appear. On real hardware the copy engine writes it; on our device nobody does unless the emulation does. The push we found in guest RAM decodes as:

20010000 0000c5b5   set-object (TURING_DMA_COPY_A)
2001a000 0000c076   set-object (GP100_UVM_SW)
20038090 00000001   incrementing-method header: count 3, addr 0x240
21000000 00000001   lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c   LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD

So the emulation's job for this push is: write payload to the memory at the semaphore's GPU VA, which requires a GPU-VA → guest-PA translation. The host does this through the learned VA→PA map: while the channel's GPFIFO has pending entries it reads each push, decodes the method stream and runs the work (page-table memsets, copies, semaphore releases). When a push page cannot be resolved — or a channel consumes entries without any release landing — it falls back to content: guest RAM is scanned for release groups and pending payloads are applied forward-only. A release is applied only when payload > current, so recycled push buffers and stale copies are harmless. One pitfall observed live: a semaphore-page VA can flap between two physical pages across map rebuilds; pinning the validated translation fixes the stalled wait.

5. The memory model UVM assumes

UVM's semaphore pool is system memory (UVM_APERTURE_SYS), one 4 KiB page per channel manager, addressed as 4-byte slots (1024 per page).
Each page has a GPU VA (chosen by the client's RM) and a CPU VA (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
Both addresses are recorded in UVM's structures in guest memory — the key that makes translation possible without extra help from the guest.
The CPU VA is a direct-map address, so PA = cpu_va - page_offset_base (verified live: writing the payload there completes the push).
Values are per channel and monotonic (1, 2, 3 …); UVM's own check (UVM_GPU_SEMAPHORE_MAX_JUMP) detects and reports wrong values.
The client's page tables for server-owned ranges are empty chains: the client hands us the physical addresses of its page-table levels and expects the server to own the lower levels (GSP-and-GSP-RM.md).

6. Page trees and faults

uvm_page_tree_init initializes a VA space's page tables with pushes on UVM's own channels (memops and page-table writes; the copy engine executes them on real hardware). Fault handling is event-driven: UVM programs a fault buffer (replayable and non-replayable; sizes from INTERNAL_GMMU_GET_STATIC_INFO, answered by the emulation) and, on a fault, reads the entry, populates the page table and replays or cancels the fault with a MEM_OP push. Reconciling the guest's memory model with the real GPU's here is the deepest remaining work.

7. How to observe UVM

In the guest, UVM prints nothing by default:

echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints

Messages then appear with the nvidia-uvm: prefix. Known-good markers, in order: uvm_gpu_retain_by_uuid → channel manager created → uvm_page_tree_init pushes → UVM_REGISTER_GPU returns and CUDA proceeds. Failure signatures characterized live:

Message	Meaning
`Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED`	CE caps wrong or missing
`stuck waiting` plus `Tracker entry ... channel_init()`	the push's semaphore was never written
`Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101`	a wrong payload was written; UVM caught it

8. Why UVM is the perfect correctness oracle

UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling and error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU.

9. Related documentation

../docs/03-architecture/04-uvm.md — canonical chapter (this page's source).
Guest-Stack-and-CUDASW.md, GSP-and-GSP-RM.md — the guest stack and the RM objects UVM consumes.
CUDA-Userspace-Driver.md — the userspace half of the contract; Execution-Plane.md, Execution-Frontier.md — how UVM's pushes execute, and the current wall.
../docs/04-implementation/01-known-knowns-and-unknowns.md — server-owned page tables and the fault contract.

UVM in the Guest

Contents

UVM in the Guest

1. What UVM is

2. The registration path (traced end to end)

3. What UVM requires from the "GPU"

4. The discovery that defines the work: how UVM waits

5. The memory model UVM assumes

6. Page trees and faults

7. How to observe UVM

8. Why UVM is the perfect correctness oracle

9. Related documentation

Navigation menu

UVM in the Guest

UVM in the Guest

1. What UVM is

2. The registration path (traced end to end)

3. What UVM requires from the "GPU"

4. The discovery that defines the work: how UVM waits

5. The memory model UVM assumes

6. Page trees and faults

7. How to observe UVM

8. Why UVM is the perfect correctness oracle

9. Related documentation

Navigation menu

Search