WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

2026-10-12T04:40:06Z

Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

← Older revision		Revision as of 04:40, 12 October 2026
Line 3:		Line 3:
	= Guest Driver Lifecycle: A Field Guide =		= Guest Driver Lifecycle: A Field Guide =

	<blockquote>'''Scope.''' A walking tour of what happens between "guest boots" and "CUDA initializes", using message, function and control IDs observed from an unmodified 570.144 guest. Canonical source: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>		<blockquote>'''Scope.''' A walking tour of what happens between "guest boots" and "CUDA initializes", using message, function and control IDs observed from an unmodified 570.144 guest. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>

	<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[design]</code> stages documented ahead of arrival.</blockquote>		<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[design]</code> stages documented ahead of arrival.</blockquote>
Line 200:		Line 200:
	== 11. Related documentation ==		== 11. Related documentation ==

	* [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>] — canonical field guide.		* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>] — canonical field guide.
	* [[GSP and GSP-RM\|<code>GSP-and-GSP-RM.md</code>]] — GSP boot, message queues and RM architecture.		* [[GSP and GSP-RM\|<code>GSP-and-GSP-RM.md</code>]] — GSP boot, message queues and RM architecture.
	* [[GSP RPC Emulation\|<code>GSP-RPC-Emulation.md</code>]] — control/alloc emulation on the host side.		* [[GSP RPC Emulation\|<code>GSP-RPC-Emulation.md</code>]] — control/alloc emulation on the host side.

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

2026-10-12T04:29:42Z

Carry the site-wide About/licensing message on every page (2026-10-12)

← Older revision		Revision as of 04:29, 12 October 2026
Line 1:		Line 1:
			{{About message}}

	= Guest Driver Lifecycle: A Field Guide =		= Guest Driver Lifecycle: A Field Guide =

WikiBot: Add page from tracked wiki source (2026 IA update)

2026-10-12T04:22:06Z

Add page from tracked wiki source (2026 IA update)

New page

= Guest Driver Lifecycle: A Field Guide =

<blockquote>'''Scope.''' A walking tour of what happens between "guest boots" and "CUDA initializes", using message, function and control IDs observed from an unmodified 570.144 guest. Canonical source: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[design]</code> stages documented ahead of arrival.</blockquote>

== 1. How to read this page ==

Each stage lists what the '''guest''' does, what the '''device/firmware''' must answer, what host emulation does today, and the log signature to look for (host-side <code>dmesg</code> with prefixes <code>vgsp:</code> / <code>vgsp-rm:</code>; guest-side <code>dmesg</code> in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the ''stages'' are stable.

== 2. Stage map ==

<pre>guest boots
│
├─ 1. PCI probe & config space ........ lspci is the reference
├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core
├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65)
├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic
├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160
├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60
└─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier</pre>

== 3. PCI probe and config space ==

'''Guest:''' reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. '''Device must:''' report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. '''Emulation:''' config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. '''Log:''' guest-side <code>lspci -vvv</code> is the reference.

== 4. GSP boot handshake ==

'''Guest:''' writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". '''Device must:''' complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. '''Emulation:''' the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate.

<pre>vgsp: GSP-BOOT: ... trigger auto-cleared ...</pre>

== 5. Static configuration ==

'''Guest:''' <code>GET_GSP_STATIC_INFO</code> (65) → a large <code>GspStaticConfigInfo</code> structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. '''Device must:''' provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. '''Emulation:''' the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values).

<pre>vgsp-rm: GET_GSP_STATIC_INFO → ...</pre>

== 6. Message queues ==

'''Guest:''' sends <code>SET_GUEST_SYSTEM_INFO</code> (1) with libos regions, a queue buffer description and a page table (the buffer is <code>NV_MEMORY_NONCONTIGUOUS</code>); also <code>GSP_SET_SYSTEM_INFO</code> (72), <code>SET_REGISTRY</code> (73) and <code>CONTINUATION_RECORD</code> (71) as init traffic. '''Device must:''' consume them silently (async) and consume the command queue exactly in the guest's write order. '''Emulation:''' parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — '''element-based, not byte-stream-based''': one message is one element and may span pages.

<pre>vgsp: libos region[i] id8=... pa=... size=...
vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=...</pre>

== 7. The RM control battery ==

'''Guest:''' sends <code>GSP_RM_CONTROL</code> (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic <code>SUCCESS</code> and zeroed params (the fallback).

{| class="wikitable"
|-
! Control
! Meaning
! What matters
|-
| <code>0x20800a40</code> family
| internal device info table
| per-index values, entry count
|-
| <code>0x20801112</code>
| FIFO device info table
| engine inventory; count must match engine queries
|-
| <code>0x20800a5c</code>
| interrupt kernel table
| vector topology the driver maps
|-
| <code>0x20800a59</code>
| GMMU static info
| fault buffer sizes; a wrong value breaks UVM early
|-
| <code>0x20800a01</code>
| display static info
| display count; headless values are fine
|-
| <code>0x20801823/-a/-b</code>
| BUS/FB info V2
| sysmem connection type and PCIe link caps feed bandwidth math
|-
| <code>0x20802a07</code>
| CE physical caps
| per-engine bits; UVM refuses channels without <code>sysmem</code> support
|-
| <code>0x20802a0a/0b</code>
| all-CE caps
| caps table for all engines
|-
| <code>0x20802a02</code>
| CE PCE mask
| one bit per engine
|-
| <code>0x20802a08</code>
| CE fault method buffer size
| method buffer sizing
|-
| <code>0x90f10106</code>
| vaspace reserved-PDE contract
| carries page-table physical addresses (see [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]])
|-
| <code>0x20800123</code> / <code>0x20800170</code>
| <code>GPU_GET_ENGINES</code> / <code>GET_ENGINES_V2</code>
| engine inventory; the guest must see the full list, including async copy (<code>COPY14</code>)
|-
| <code>0x800201</code>
| <code>GPU_GET_CLASSLIST</code>
| class list; must agree with engine list and CE caps
|-
| <code>0x2080012b</code>
| <code>GPU_PROMOTE_CTX</code> (560-byte params)
| load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe
|-
| <code>0x20803039</code>
| NVLink masks/VBIOS info
| fallback: generic <code>SUCCESS</code> with zeroed params
|-
| <code>0x20800a70</code>
| BUS flush with sysmembar
| fallback: generic <code>SUCCESS</code> with zeroed params; UVM issues these often
|}

<pre>vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...</pre>

== 8. Object creation ==

'''Guest:''' <code>GSP_RM_ALLOC</code> (103) creates the object tree. Typical order for a CUDA process:

<pre>0x0 NV01_ROOT (client root, 120-byte params)
0x80 NV01_DEVICE_0 (56 bytes)
0x2080 NV20_SUBDEVICE_0 (4 bytes)
0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space
0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)</pre>

A hard-won rule from the <code>TURING_COMPUTE_A</code> (0xc5c0) alloc: the caller may issue it with <code>paramsSize=0</code> against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must '''echo the request params back in the alloc reply''', or the user-mode client segfaults. Channel parameters decode into <code>gpFifoOffset</code> (a GPU VA), <code>gpFifoEntries</code>, <code>engineType</code> (e.g. <code>COPY0 = 0x9</code>), and <code>NV_MEMORY_DESC_PARAMS</code> for instance/userd/ramfc/method buffers with '''physical''' addresses and apertures (<code>ADDR_SYSMEM = 1</code>, <code>ADDR_FBMEM = 2</code>).

<pre>vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...</pre>

== 9. UVM registration and the execution frontier ==

'''Guest:''' <code>cuInit</code> → <code>UVM_REGISTER_GPU</code> → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. '''Emulation today:''' the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet.

'''Progress:''' context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted <code>CTXNOTVALID</code>, and a following <code>GPU_PROMOTE_CTX</code> wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [[The Execution Frontier|<code>Execution-Frontier.md</code>]]. The roadmap from here is:

<pre>UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)</pre>

Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions.

<pre>nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N</pre>

== 10. Quick reference: function IDs from a stock 570.144 guest ==

{| class="wikitable"
|-
! Function
! Name
! Frequency per CUDA run (approx.)
|-
| 1
| SET_GUEST_SYSTEM_INFO
| once
|-
| 10
| FREE
| ~14
|-
| 47
| UNLOADING_GUEST_DRIVER
| on unload
|-
| 65
| GET_GSP_STATIC_INFO
| once
|-
| 70
| UPDATE_BAR_PDE
| a few
|-
| 71/72/73
| CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY
| once each
|-
| 76
| GSP_RM_CONTROL
| ~160
|-
| 103
| GSP_RM_ALLOC
| ~60
|}

If a new driver version produces '''new function numbers''', do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic.

== 11. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>] — canonical field guide.
* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — GSP boot, message queues and RM architecture.
* [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]] — control/alloc emulation on the host side.
* [[RPC Message Format|<code>RPC-Message-Format.md</code>]] — wire format of the function 76/103 traffic.
* [[Address Reference|<code>Address-Reference.md</code>]] and [[GSP 570.144W Firmware Analysis|<code>GSP-570.144W-Firmware-Analysis.md</code>]] — ID reference and the firmware they belong to.
* [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — registration, the frontier, the host stages and the forwarding backend.

Guest Driver Lifecycle: A Field Guide - Revision history

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

WikiBot: Add page from tracked wiki source (2026 IA update)