The Host Module (vgsp.ko)

The Host Module (vgsp.ko)

Scope. The host-side kernel module that presents one virtual Turing-class GPU per mdev instance: the virtual device, the emulation engine, the guest-memory model and the work-forwarding design. Canonical source: ../docs/03-architecture/06-host-module-and-hybrid-model.md; the docs tree wins on disagreement with this page.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] observed with the stock guest driver; [wip] demonstrated live, being productized; [gated] implemented default-off behind supervised gates; [wall] known blocking limit; [target] designed, not built; [planned] roadmap.

1. Role and position

guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor
   (KVM/QEMU or Xen: VFIO mdev transport, IOMMU)
   │
vgsp.ko: virtual device · emulation core · memory model · work path
   │
physical GPU + GSP firmware, driven by this module (host driver unmodified)

The module registers as an mdev parent on the physical GPU, exposing one device type per supported profile; each mdev is an independent virtual GPU (System-Overview.md, Execution-Plane.md).

2. The virtual device

The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability list). Some fields exist only for validation: PCIe link capabilities (the hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows.

BAR	Size	Backing	Contents
BAR0	16 MB	host vzalloc shadow	boot, doorbells, VF registers, PRAMIN window, VBIOS
BAR1	256 MB	host vzalloc vidmem buffer	guest-visible VRAM aperture
BAR3	—	instance window	instance/PRAMIN behaviour

MMIO is served by VFIO region callbacks or directly from the shadow when mapped into the guest. BAR1 is host RAM today — correctness first: the real GPU cannot read it, so forwarding must later migrate objects to real VRAM or expose DMA-mapped system memory [target] (Physical-GPU-Backend.md).

3. The emulation core

A kernel thread drives the device's "firmware" behaviour [live]:

BAR0 doorbell / VF-register write → read msgq element from guest memory
   → dispatch RPC → write reply → inject MSI-X (eventfd → irqfd)

Guest memory is reached via vfio_dma_rw() from kernel-thread context. The RPC philosophy: correct shape first (or the driver walks off into uninitialized memory), plausible values second (values must be self-consistent), never invent input (in/out parameter blocks are echoed; zeroing the GSP_RM_ALLOC reply once faulted libcuda). Handlers cover boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries, reserved-PDE contracts, channel/CE allocation and an unknown-control fallback. Push buffers are decoded as method streams and executed in software — memsets, copies, semaphore releases — a correctness bridge, not the data plane [wip]; emulation-mode CUDA is verified end to end (vectorAdd over 1M elements, bad=0) [live]. Physical-GSP boot is default-off [gated]; the firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are not yet on the live path [target].

4. The guest-memory model

What the client gives us: channel memory descriptors (NV_MEMORY_DESC_PARAMS) with physical addresses (instance memory, method buffer, userd, ramfc, error notifiers) and the page-table contract for server-owned VA ranges (COPY_SERVER_RESERVED_PDES — table-level PAs, page shifts, VA range, empty leaves). It gives us none of its own sysmem maps.

What we derive instead [live] — the module rebuilds the missing mappings from the guest's own bookkeeping, with no cooperation from the guest driver:

1. VA↔PA pairs live in guest memory: UVM and RM record both the GPU VA and the kernel CPU VA per object (uvm_rm_mem_t.vas[CPU]/vas[GPU] +0x30/+0x38).
2. CPU VAs are direct-map addresses: PA = cpu_va - page_offset_base, the base recovered from guest memory or /proc/kallsyms during development.
3. Exception: control-device mmap() pages have no leaf PTEs; verified pairs arrive via sema_map_add until a durable join exists. The page-granular cache is fed by a trusted GMMU walk (Introduction-to-GMMU.md).

Load-bearing copy invariants: (1) a copy runs at most once per GPFIFO entry — replays are suppressed by entry-word identity plus an executed-operation bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were contiguous in one observed 1 MB range); (3) a consumed slot's work never runs later — the per-put-epoch mark is authoritative until a put wrap.

5. The forwarding path (design)

guest push submitted
   → detect (doorbell / GPPUT / push-buffer scan), decode the method stream
   → EMULATE state/notification (release, membar, fence) or
     FORWARD real engine work (GR launch, CE copy, memop)
   → FORWARD: submit equivalent work on a real channel bound to the physical GPU
   → relay completions (semaphore / fault / interrupt) back into guest memory

Proxying through the distribution host driver has no callable entry point from kernel context, so the module owns the device and runs its real GSP firmware as the resource-manager server, with emulation as the control plane. CE copies, release and fault relay are implemented default-off and verified live [gated]; GR execution is the open wall [wall] — a forwarded launch reaches the front-end; the inline launch-descriptor install is unobserved (Execution-Frontier.md).

6. Debug parameters

/sys/module/vgsp/parameters/* makes guest memory and the emulated device inspectable without rebuilding:

Parameter	Purpose
scan_guest_ram=<v> / scan_vidmem=<v>	find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow
scan_seq=<v1,...> / scan_pt=<pa> / scan_masked=<v>,<mask>	find a 32-bit sequence, entries pointing at a PA, or (x & mask) == v
dump_vidmem=<off>,<len> / fill_vidmem=<v>	dump/scan or fill the VRAM shadow
poke_guest=<gpa>,<v> / poke_fill=<gpa>,<v>,<n>	write a 32-bit value / fill a range in guest physical memory
sema_probe=<gpu_va>[,count] / sema_map_add=<va>,<pa>	probe a semaphore VA / teach one verified VA→PA pair
dump_guest=<gpa>,<len>	dump guest physical memory (capped)

The physical-backend path adds a default-off set of backend controls (documented in the repository; workflows in ../docs/05-operations/02-debugging-and-maintenance-workflows.md).

7. Build and symbol contracts

• vgsp.ko exports the nvidia_vgpu_vfio_* interface (probe/remove, ops get/set, mdev parent registration) to pair the device with OpenRM host modules.
• OpenRM modules built from source need KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers and must build after this module (missing symbols and stale build caches are common otherwise).
• On single-GPU developer systems the distribution driver owns the physical GPU and the module runs standalone; the OpenRM path is only for native-vGPU management, not QEMU.

8. Implemented vs designed

Area	Status
mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live]	implemented
Message queues + RPC + init controls (stock 570.144) [live]	implemented
Guest sysmem access + VA↔PA discovery [wip]	debug-grade, to be productized
Push decoding + semaphore completion [wip]	demonstrated live, being productized
Emulation-mode CUDA end-to-end [live]	repeatable 1M-element vectorAdd, bad=0
Guest-RAM physical path + server-owned page-table population [gated]	default-off; GPA→HPA via shared-file page cache verified live on the CE
GPU work forwarding: CE copy, release and fault relay [gated]	default-off, verified live; GR open [wall]
Execution backend (module-owned GPU, direct real-GSP session) [wall]	boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried)
Real VRAM backing [target]; Windows, Xen, aarch64 [planned]	designed, not implemented

9. Related documentation

• Canonical chapter: ../docs/03-architecture/06-host-module-and-hybrid-model.md — the source of this condensation.
• GSP-RPC-Emulation.md, BAR-Emulation-Specification.md, WPR-Management.md; Physical-GPU-Backend.md, Execution-Frontier.md, Guest-Stack-and-CUDASW.md.