The Guest Stack and CUDASW

The Guest Stack and CUDASW

Scope. What runs inside the guest, how the NVIDIA software layers sit on top of the device, and what the unmodified guest driver expects from the emulated GPU. Canonical source: ../docs/03-architecture/02-guest-stack-and-cudasw.md; the docs tree wins on disagreement with this page.

Status: 2026-10-30, tree master @ b4930168. Tags: [live] observed with the stock guest driver; [gated] implemented default-off behind supervised gates; [wall] known blocking limit; [target] designed, not built; [planned] roadmap.

1. Vocabulary: what "CUDASW" means here

CUDASW is the family of NVIDIA software components that sit between an application and the GPU driver. The project's compatibility target is the whole stack, not one library:

application
   │  CUDA API (runtime)
libcudart.so ─────────────────────────────┐  runtime: device/stream/event
   │  CUDA driver API                    │  management, kernel launch
libcuda.so ───────────────────────────────┤  driver: contexts, modules,
   │  ioctls                             │  memory, channel submission
/dev/nvidia* ─────────────────────────────┘
nvidia.ko (RM + UVM + modeset)

Adjacent members of the family: libnvidia-ptxjitcompiler (PTX→SASS JIT — usually "just works" once the driver initializes), libnvidia-fatbinaryloader / nvrtc, UVM (nvidia_uvm.ko — UVM-in-the-Guest.md), the graphics stack (libGLX_nvidia, the driver's GL core, the Vulkan ICD, nvidia_modeset/nvidia_drm — partially live on the emulated vGPU: Xorg/GLX run, the engine executes GL method streams, and the Vulkan pipeline runs end to end; rendered pixels are the frontier — Graphics-Execution.md) and the video stack (NVENC/NVDEC) — roadmap.

2. Layering rules the emulation must respect

1. The kernel driver is the contract. Everything above it talks to nvidia.ko through stable ioctls, memory mappings and /dev/nvidia* nodes. If the kernel driver initializes and behaves, user space follows.
2. User space is opaque to us. We never intercept user-space APIs. All hooks are in the device model (host side) or in the hardware interface the kernel driver sees (guest side).
3. Version skew is normal. Driver version, CUDA version and firmware version are three independent axes. The compatibility target is "the driver family that matches the emulated firmware profile", with configuration-driven adaptation for others.

3. Kernel-driver anatomy (what the guest brings)

Component	Role	Why it matters here
RM core	object model, handles, locks, device lifecycle	drives the GSP boot and the whole RPC/control surface
Resource server (resserv)	allocates/tracks objects, memory, channels	the client half of the client/server RM split
Memory manager (memmgr)	VRAM/sysmem objects, mappings, page tables	the guest expects the GSP to own page-table leaves for its VA spaces
FIFO/runlist	channels, TSGs, GPFIFO, USERD, doorbells	where the guest's work is submitted
GSP client	message queues, RPC transport, event handling	the interface we emulate
UVM	unified memory, page faults, channel manager	starts early in CUDA init, extremely stateful
nvidia_modeset / nvidia_drm	display / DRM	needed later for graphics
CE / GR HALs	engine programming	needed to execute forwarded work

4. Driver initialization vs runtime

Understanding this split is the key to debugging.

Initialization (boot/probe) — deterministic, mostly one-shot; where compatibility is won or lost [live]:

PCI probe → identify GPU from config space and BAR0 registers
   → boot the GSP (our emulated handshake)
   → read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
   → create root client/device/subdevice objects
   → initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
   → expose /dev/nvidia* and register with the kernel

The actual message/control IDs observed at each step are listed in Driver-Lifecycle-Field-Guide.md.

Runtime (per-process, per-context) — driven by user-space workloads; where execution is won or lost:

open /dev/nvidia* → cuInit → UVM registration (per process)
   → cuCtxCreate → object/allocations → memory maps
   → channel creation + push buffers + semaphores
   → kernel launches (GR) / copies (CE) / memops
   → faults, completions, events

Context creation and the full CUDA driver-API smoke pass (ALL OK, copies matched) [live], and guest kernel launches now execute on the physical engine with engine-authored completions relayed to the guest [live]. The backend is a module-owned GPU running its real GSP firmware as the resource-manager server (Physical-GPU-Backend.md). The leading edge is the staged-transfer class, the every-boot acceptance campaign and per-guest isolation (Execution-Frontier.md). Graphics work reuses the same submission machinery on its own frontier: the X server and GLX run, the engine executes GL method streams, and the Vulkan pipeline runs end to end; rendered pixels are the wall (Graphics-Execution.md).

5. What the guest expects from the device, per phase

Phase	Guest action	Device/firmware expectation
Probe	config space reads, BAR sizing, MSI-X capability	a plausible TU104-family PCIe GPU
Boot	BAR0 register writes, WPR setup, firmware boot sequence	GSP boots, booter completes, "GSP init done"
Config	RPC GET_GSP_STATIC_INFO, device tables, VBIOS reads	consistent, self-describing tables
RM init	GSP_RM_CONTROL battery (FB/BUS/CE/INTR/GMMU/...)	correct shapes and plausible values
Objects	GSP_RM_ALLOC (root, device, subdevice, vaspace, channel, CE objects)	success + correct handles/params
Runtime	allocs/controls for maps and channels; pushes	executed work, semaphores, faults

The RPC surface is described in GSP-and-GSP-RM.md and RPC-Message-Format.md; the boot handshake in WPR-Management.md.

6. Why user-space-only interception does not work

Two hard-won facts from this project:

1. libcuda issues raw syscall instructions, so an LD_PRELOAD shim around ioctl() sees nothing. Observing CUDA's kernel calls requires a ptrace-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the project's diagnostics do.
2. **The CUDA driver talks to the kernel driver, not to the device.** All device interaction happens in nvidia.ko. This is good news: the emulation boundary is a small set of kernel-side interfaces (MMIO, DMA, interrupts, the GSP protocol) rather than a large user-space API surface.

7. Windows guests

Windows brings its own kernel-mode driver, nvlddmkm.sys, which uses the same underlying GSP device/firmware model but different host interfaces (WDDM, MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the firmware interface, forward the work. Windows support is a roadmap item (Hardware-Software-Matrix.md) [planned], and the design goal is that no Windows-specific code lives in the guest. The Windows/DirectX path — WDDM, the Direct3D user-/kernel-mode drivers, and what a bring-up requires — is described on DirectX.md.

8. Compatibility notes for versions

• The guest's driver and the emulated firmware profile must agree on the RPC message versions. The guest negotiates these at start-up (SET_GUEST_SYSTEM_INFO carries the VGX/RPC version); the emulation answers with a profile that matches the driver family in use.
• CUDA versions ride on driver versions; the test matrix tracks (driver, CUDA, firmware) triples (../docs/05-operations/01-performance-correctness-security.md).

9. Related documentation

• ../docs/03-architecture/02-guest-stack-and-cudasw.md — canonical chapter.
• Host-Module.md — the other side of the boundary.
• UVM-in-the-Guest.md, CUDA-Userspace-Driver.md — user-space driver internals.
• Driver-Lifecycle-Field-Guide.md, GSP-and-GSP-RM.md — the observed protocol reality.
• Physical-GPU-Backend.md, Execution-Frontier.md — where guest work is headed.