Guest Driver Lifecycle: A Field Guide

Guest Driver Lifecycle: A Field Guide

Scope. A walking tour of what happens between "guest boots" and "CUDA initializes", using message, function and control IDs observed from an unmodified 570.144 guest. Canonical source: ../docs/03-architecture/07-driver-lifecycle-field-guide.md; if this page and the docs tree disagree, the docs tree wins.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] observed with the stock guest driver; [design] stages documented ahead of arrival.

1. How to read this page

Each stage lists what the guest does, what the device/firmware must answer, what host emulation does today, and the log signature to look for (host-side dmesg with prefixes vgsp: / vgsp-rm:; guest-side dmesg in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the stages are stable.

2. Stage map

guest boots
  │
  ├─ 1. PCI probe & config space ........ lspci is the reference
  ├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core
  ├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65)
  ├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic
  ├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160
  ├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60
  └─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier

3. PCI probe and config space

Guest: reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. Device must: report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. Emulation: config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. Log: guest-side lspci -vvv is the reference.

4. GSP boot handshake

Guest: writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". Device must: complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. Emulation: the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate.

vgsp: GSP-BOOT: ... trigger auto-cleared ...

5. Static configuration

Guest: GET_GSP_STATIC_INFO (65) → a large GspStaticConfigInfo structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. Device must: provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. Emulation: the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values).

vgsp-rm: GET_GSP_STATIC_INFO → ...

6. Message queues

Guest: sends SET_GUEST_SYSTEM_INFO (1) with libos regions, a queue buffer description and a page table (the buffer is NV_MEMORY_NONCONTIGUOUS); also GSP_SET_SYSTEM_INFO (72), SET_REGISTRY (73) and CONTINUATION_RECORD (71) as init traffic. Device must: consume them silently (async) and consume the command queue exactly in the guest's write order. Emulation: parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — element-based, not byte-stream-based: one message is one element and may span pages.

vgsp: libos region[i] id8=... pa=... size=...
vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=...

7. The RM control battery

Guest: sends GSP_RM_CONTROL (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic SUCCESS and zeroed params (the fallback).

Control	Meaning	What matters
0x20800a40 family	internal device info table	per-index values, entry count
0x20801112	FIFO device info table	engine inventory; count must match engine queries
0x20800a5c	interrupt kernel table	vector topology the driver maps
0x20800a59	GMMU static info	fault buffer sizes; a wrong value breaks UVM early
0x20800a01	display static info	display count; headless values are fine
0x20801823/-a/-b	BUS/FB info V2	sysmem connection type and PCIe link caps feed bandwidth math
0x20802a07	CE physical caps	per-engine bits; UVM refuses channels without sysmem support
0x20802a0a/0b	all-CE caps	caps table for all engines
0x20802a02	CE PCE mask	one bit per engine
0x20802a08	CE fault method buffer size	method buffer sizing
0x90f10106	vaspace reserved-PDE contract	carries page-table physical addresses (see GSP-RPC-Emulation.md)
0x20800123 / 0x20800170	GPU_GET_ENGINES / GET_ENGINES_V2	engine inventory; the guest must see the full list, including async copy (COPY14)
0x800201	GPU_GET_CLASSLIST	class list; must agree with engine list and CE caps
0x2080012b	GPU_PROMOTE_CTX (560-byte params)	load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe
0x20803039	NVLink masks/VBIOS info	fallback: generic SUCCESS with zeroed params
0x20800a70	BUS flush with sysmembar	fallback: generic SUCCESS with zeroed params; UVM issues these often

vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...

8. Object creation

Guest: GSP_RM_ALLOC (103) creates the object tree. Typical order for a CUDA process:

0x0        NV01_ROOT              (client root, 120-byte params)
0x80       NV01_DEVICE_0          (56 bytes)
0x2080     NV20_SUBDEVICE_0       (4 bytes)
0x90f1     FERMI_VASPACE_A        (48 bytes)  ← the GPU address space
0xa06c     KEPLER_CHANNEL_GROUP_A (20 bytes)  ← CE channel group
0xc46f     TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076     GP100_UVM_SW           (0 bytes)
0xc5b5     TURING_DMA_COPY_A      (8 bytes)   ← CE object
0xc5c0     TURING_COMPUTE_A       (16 bytes)  ← compute object (engine registration path)

A hard-won rule from the TURING_COMPUTE_A (0xc5c0) alloc: the caller may issue it with paramsSize=0 against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must echo the request params back in the alloc reply, or the user-mode client segfaults. Channel parameters decode into gpFifoOffset (a GPU VA), gpFifoEntries, engineType (e.g. COPY0 = 0x9), and NV_MEMORY_DESC_PARAMS for instance/userd/ramfc/method buffers with physical addresses and apertures (ADDR_SYSMEM = 1, ADDR_FBMEM = 2).

vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...

9. UVM registration and the execution frontier

Guest: cuInit → UVM_REGISTER_GPU → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. Emulation today: the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet.

Progress: context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted CTXNOTVALID, and a following GPU_PROMOTE_CTX wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See Execution-Frontier.md. The roadmap from here is:

UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)

Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions.

nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N

10. Quick reference: function IDs from a stock 570.144 guest

Function	Name	Frequency per CUDA run (approx.)
1	SET_GUEST_SYSTEM_INFO	once
10	FREE	~14
47	UNLOADING_GUEST_DRIVER	on unload
65	GET_GSP_STATIC_INFO	once
70	UPDATE_BAR_PDE	a few
71/72/73	CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY	once each
76	GSP_RM_CONTROL	~160
103	GSP_RM_ALLOC	~60

If a new driver version produces new function numbers, do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic.

11. Related documentation

• ../docs/03-architecture/07-driver-lifecycle-field-guide.md — canonical field guide.
• GSP-and-GSP-RM.md — GSP boot, message queues and RM architecture.
• GSP-RPC-Emulation.md — control/alloc emulation on the host side.
• RPC-Message-Format.md — wire format of the function 76/103 traffic.
• Address-Reference.md and GSP-570.144W-Firmware-Analysis.md — ID reference and the firmware they belong to.
• UVM-in-the-Guest.md, Execution-Frontier.md, Host-Module.md, Physical-GPU-Backend.md — registration, the frontier, the host stages and the forwarding backend.