WikiBot: Fix repository links: point to the cgit tree URLs

2026-10-12T04:41:02Z

Fix repository links: point to the cgit tree URLs

← Older revision		Revision as of 04:41, 12 October 2026
Line 3:		Line 3:
	= UVM in the Guest =		= UVM in the Guest =

	<blockquote>'''Scope.''' What UVM (<code>nvidia_uvm.ko</code>) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>]; where this page and the docs tree disagree, the docs tree wins.</blockquote>		<blockquote>'''Scope.''' What UVM (<code>nvidia_uvm.ko</code>) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>]; where this page and the docs tree disagree, the docs tree wins.</blockquote>

	Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> partially working, software bridge in charge; <code>[gated]</code> implemented default-off; <code>[target]</code> planned end state.		Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> partially working, software bridge in charge; <code>[gated]</code> implemented default-off; <code>[target]</code> planned end state.
Line 152:		Line 152:
	== 9. Related documentation ==		== 9. Related documentation ==

	* [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>] — canonical chapter (this page's source).		* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>] — canonical chapter (this page's source).
	* [[The Guest Stack and CUDASW\|<code>Guest-Stack-and-CUDASW.md</code>]], [[GSP and GSP-RM\|<code>GSP-and-GSP-RM.md</code>]] — the guest stack and the RM objects UVM consumes.		* [[The Guest Stack and CUDASW\|<code>Guest-Stack-and-CUDASW.md</code>]], [[GSP and GSP-RM\|<code>GSP-and-GSP-RM.md</code>]] — the guest stack and the RM objects UVM consumes.
	* [[CUDA Userspace Driver\|<code>CUDA-Userspace-Driver.md</code>]] — the userspace half of the contract; [[The vGSP Execution Plane (vEP)\|<code>Execution-Plane.md</code>]], [[The Execution Frontier\|<code>Execution-Frontier.md</code>]] — how UVM's pushes execute, and the current wall.		* [[CUDA Userspace Driver\|<code>CUDA-Userspace-Driver.md</code>]] — the userspace half of the contract; [[The vGSP Execution Plane (vEP)\|<code>Execution-Plane.md</code>]], [[The Execution Frontier\|<code>Execution-Frontier.md</code>]] — how UVM's pushes execute, and the current wall.
	* [https://git.vgpu.io/open-iov/~~blob/master~~/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the fault contract.		* [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the fault contract.

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

2026-10-12T04:29:53Z

Carry the site-wide About/licensing message on every page (2026-10-12)

← Older revision		Revision as of 04:29, 12 October 2026
Line 1:		Line 1:
			{{About message}}

	= UVM in the Guest =		= UVM in the Guest =

WikiBot: Add page from tracked wiki source (2026 IA update)

2026-10-12T04:22:16Z

Add page from tracked wiki source (2026 IA update)

New page

= UVM in the Guest =

<blockquote>'''Scope.''' What UVM (<code>nvidia_uvm.ko</code>) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>]; where this page and the docs tree disagree, the docs tree wins.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> partially working, software bridge in charge; <code>[gated]</code> implemented default-off; <code>[target]</code> planned end state.

== 1. What UVM is ==

UVM provides unified memory and GPU-driven page-fault handling. In practice it is a '''second GPU driver layered on the RM''', owning GPU address spaces and running its own memory-management work on the GPU.

{| class="wikitable"
|-
! Component
! Role
|-
| VA spaces (<code>uvm_va_space</code>)
| per-process GPU address spaces
|-
| Channel managers
| pools of GPU channels (CE, SEC2, GR as needed) for UVM's work
|-
| Push buffers / GPFIFO
| UVM builds and submits its own pushes
|-
| Semaphores and trackers
| UVM waits on GPU-written semaphores to track completion
|-
| Page trees
| GPU page tables for its VA spaces
|-
| Fault handling
| fault buffer plus notification; UVM resolves and replays/cancels faults
|}

For CUDA, UVM is entered early: <code>cuInit</code> → driver init → <code>UVM_REGISTER_GPU</code>. If UVM cannot register the GPU, CUDA does not start — which is why "UVM advances" is the project's primary correctness metric on the way to CUDA.

== 2. The registration path (traced end to end) ==

<pre>libcuda: UVM_REGISTER_GPU ioctl (0x25)
uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
nvUvmInterfaceDeviceCreate (RM object creation)
uvm_hal_init_gpu / get_gpu_caps (CE caps)
uvm_ats_add_gpu / uvm_gpu_init_isr (fault/interrupt plumbing)
init_channel_manager
create semaphore pool (sysmem)
create channels per CE (GPFIFO + pushbuffer + semaphores)
channel_init() -> first push -> semaphore release -> wait
uvm_page_tree_init (page tables for the VA space)</pre>

Each stage has a distinct failure signature — see [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]].

== 3. What UVM requires from the "GPU" ==

{| class="wikitable"
|-
! Requirement
! Detail
! Status
|-
| GPU identity
| must match the UUID from RM
| <code>[live]</code>
|-
| CE capabilities
| per-engine support bits; UVM requires <code>supported && !grce && sysmem</code> for CPU_TO_GPU / GPU_TO_CPU channels
| <code>[live]</code> caps accepted; <code>[gated]</code> copies forwarded to the physical copy engine
|-
| Channel creation
| RM channel objects with GPFIFO/userd/instance parameters
| <code>[live]</code>
|-
| Pushes executed
| first action on a new channel is an init push plus a semaphore-tracked wait
| <code>[wip]</code> software bridge executes memsets, copies and releases; <code>[gated]</code> CE-class copies forwarded
|-
| Page trees
| UVM initializes page trees with its own pushes; server-owned ranges must be populated by us
| <code>[wip]</code>
|-
| Fault delivery
| fault buffer plus notification; replay/cancel semantics
| <code>[gated]</code> backend-side observation/relay; guest contract <code>[target]</code>
|-
| Completion semantics
| tracker values must advance exactly
| <code>[wip]</code>
|}

'''Current frontier.''' Context creation and the full CUDA driver-API smoke pass (<code>ALL OK</code>, copies matched) run end to end. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored releases and a fault relay. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed; context validity at dispatch is ruled out ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]). The remaining UVM work is completion translation and the guest-side fault replay/cancel contract.

== 4. The discovery that defines the work: how UVM waits ==

UVM's tracker is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps its state:

<pre>uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
releasing value 1</pre>

Mechanically, UVM built a push ending in a '''semaphore release''' — a method sequence (CE channel class, e.g. <code>TURING_DMA_COPY_A</code>) that sets an address and a payload and launches a DMA operation to write the payload to memory — then submitted it and waits for the payload to appear. On real hardware the copy engine writes it; on our device nobody does unless the emulation does. The push we found in guest RAM decodes as:

<pre>20010000 0000c5b5 set-object (TURING_DMA_COPY_A)
2001a000 0000c076 set-object (GP100_UVM_SW)
20038090 00000001 incrementing-method header: count 3, addr 0x240
21000000 00000001 lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD</pre>

So the emulation's job for this push is: '''write <code>payload</code> to the memory at the semaphore's GPU VA''', which requires a GPU-VA → guest-PA translation. The host does this through the learned VA→PA map: while the channel's GPFIFO has pending entries it reads each push, decodes the method stream and runs the work (page-table memsets, copies, semaphore releases). When a push page cannot be resolved — or a channel consumes entries without any release landing — it falls back to content: guest RAM is scanned for release groups and pending payloads are applied forward-only. A release is applied only when <code>payload > current</code>, so recycled push buffers and stale copies are harmless. One pitfall observed live: a semaphore-page VA can flap between two physical pages across map rebuilds; pinning the validated translation fixes the stalled wait.

== 5. The memory model UVM assumes ==

* UVM's semaphore pool is '''system memory''' (<code>UVM_APERTURE_SYS</code>), one 4 KiB page per channel manager, addressed as 4-byte slots (1024 per page).
* Each page has a '''GPU VA''' (chosen by the client's RM) and a '''CPU VA''' (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
* Both addresses are recorded in UVM's structures in guest memory — the key that makes translation possible without extra help from the guest.
* The CPU VA is a direct-map address, so <code>PA = cpu_va - page_offset_base</code> (verified live: writing the payload there completes the push).
* Values are per channel and monotonic (1, 2, 3 …); UVM's own check (<code>UVM_GPU_SEMAPHORE_MAX_JUMP</code>) detects and reports wrong values.
* The client's page tables for '''server-owned ranges are empty chains''': the client hands us the physical addresses of its page-table levels and expects the server to own the lower levels ([[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]]).

== 6. Page trees and faults ==

<code>uvm_page_tree_init</code> initializes a VA space's page tables with pushes on UVM's own channels (memops and page-table writes; the copy engine executes them on real hardware). Fault handling is event-driven: UVM programs a fault buffer (replayable and non-replayable; sizes from <code>INTERNAL_GMMU_GET_STATIC_INFO</code>, answered by the emulation) and, on a fault, reads the entry, populates the page table and replays or cancels the fault with a <code>MEM_OP</code> push. Reconciling the guest's memory model with the real GPU's here is the deepest remaining work.

== 7. How to observe UVM ==

In the guest, UVM prints nothing by default:

<pre>echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints</pre>

Messages then appear with the <code>nvidia-uvm:</code> prefix. Known-good markers, in order: <code>uvm_gpu_retain_by_uuid</code> → channel manager created → <code>uvm_page_tree_init</code> pushes → <code>UVM_REGISTER_GPU</code> returns and CUDA proceeds. Failure signatures characterized live:

{| class="wikitable"
|-
! Message
! Meaning
|-
| <code>Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED</code>
| CE caps wrong or missing
|-
| <code>stuck waiting</code> plus <code>Tracker entry ... channel_init()</code>
| the push's semaphore was never written
|-
| <code>Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101</code>
| a wrong payload was written; UVM caught it
|}

== 8. Why UVM is the perfect correctness oracle ==

UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling and error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU.

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>] — canonical chapter (this page's source).
* [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the guest stack and the RM objects UVM consumes.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the userspace half of the contract; [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — how UVM's pushes execute, and the current wall.
* [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the fault contract.

UVM in the Guest - Revision history

WikiBot: Fix repository links: point to the cgit tree URLs

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

WikiBot: Add page from tracked wiki source (2026 IA update)