Physical GPU Backend and Guest Wiring

Physical GPU Backend and Guest Wiring

Scope. The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: ../docs/03-architecture/08-physical-gpu-integration.md and ../docs/03-architecture/11-physical-backend-guest-wiring.md; the docs tree wins on disagreement.

Status: 2026-10-20, tree master @ 16291c72. Tags: [live] proven with the stock guest driver; [gated] implemented default-off; [wip] partially built; [wall] current blocker; [target] planned end state.

1. The problem and its constraints

The guest's real work (copies, memory operations, grid launches) must execute on the physical GPU. The constraints: one DKMS kernel module on the host, no daemon and no new host userspace; host and guest drivers unmodified, with no paravirtual enlightenment in the guest; all emulation and translation logic inside the module; and real execution — compute is not emulated in software.

2. Why the host driver is not the path

Interface	What it offers	Reachable from our module?
Host driver (distribution build)	resource management, scheduling, memory	No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface
Host driver peer-memory exports	pin/map memory for peer devices	Partially — memory only, no submission
Vendor vGPU host stack	mediated work submission and scheduling	Requires the licensed vGPU host branch and a vGPU-aware guest — excluded
SR-IOV virtual functions	hardware-partitioned device	Not available on this GPU generation
The GPU's own GSP firmware	the resource manager server itself	Yes — once the module owns the device; this is the path used

With an unmodified host driver there is no callable submission interface into it, so the module must own the device and talk to the hardware itself.

3. Mechanism options and the decision

 (A) Proxy through the host driver: module -> host driver's RM -> real channel
     -> GPU. Needs a modified host driver or a vendor interface — not implementable.
 (B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
     tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
 (C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
 (D) Own the GPU; use its real GSP firmware as the server      <- chosen
     guest push -> module -> real GSP-RM session -> real channel -> GPU
     The module becomes the host-side resource-manager client; scheduling,
     page tables and context management stay the firmware's job.
 (E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.

Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.

4. The chosen mechanism: bring-up ladder

The module binds the physical GPU as its driver while the distribution driver is unbound (dedicated-GPU deployment, §9). Each stage is reversible and independently verified:

1. Ownership — the board's power domain is held active across the window (the platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. [live]
2. Firmware staging — the matching GSP image is fetched and validated (ELF64 RISC-V; loadable section, version, hardware signature); no device state touched. [live]
3. Boot — protected region in video memory, secure-verification stage, booter, RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. [live]
4. Host RM client — the module speaks the already-mapped RPC protocol to the real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". [live]
5. Guest mediation — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. [wip]
6. Work path — decoded guest pushes become real push streams; resolved host pages map through module-built scratch page tables; the firmware schedules the work. [live] observed end to end for the verified guest kernel clients.
7. Completion relay — hardware-completed semaphores land in guest-polled memory; events and faults relay through the existing MSI-X injection path. [live] engine-authored completions relay to the guest and its waits resolve.

5. The two planes and the wiring

The emulation plane serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The physical backend plane owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible work — the scratch-mapped forwarding path, parameterised and fail-open to software.

GUEST (QEMU)    HOST: vgsp.ko (module-owned GPU)                  PHYSICAL GPU
-----------------------------------------------------------------------------
INIT  preflight: distribution driver green; no vgsp/mdev/QEMU          [live]
      claim: power domain held -> distribution driver released -> module
         binds the device -> liveness check                             [live]
      boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init        [live]
      mdev + ladder: backend joins the live session; real objects,
         channels, first real CE copy                                   [live]
      guest: nvidia.ko probes; emulation serves handshake, static config,
         RM battery, UVM registration                                   [live]
WORK  guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
      host pump decodes; eligible CE copies forwarded via module-built scratch
         page tables; GR: launches execute on the engine, completions relay
         to the guest                                                   [live]
DONE  engine authors the release in guest memory; host relays via MSI-X   [live]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
      -> protected region down -> distribution driver rebound            [live]
FAIL  a failed mutating operation parks the device in a locked fail-safe;
      never auto-retried; recovery is supervised                          [live]

6. Proven capabilities

Capability	Status
Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X)	done [live]
Driver-API baseline (init, devices, context, allocation, copies, free, destroy)	done — 64/64 payload round-trip [live]
Software bridge for decoded memory/state operations	in progress [wip]
Device ownership, firmware staging, board-liveness checks	verified live [live]
Power domain held active across the ownership window	verified across supervised cycles [live]
Real GSP boot by the module, through init-done and clean release	done, no reboot on clean cycles [live]
Host RM client and first real engine work (static config, real objects, 4 KiB CE copy)	done [live]
Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay	verified live (seams parameterised and fail-open); the large-buffer staged-transfer class is under repair [live]
CUDA kernel execution (vectorAdd and further clients)	done [live] — stock guest CUDA clients (vector-add, matrix-multiply, transformer) execute on the physical engine with verified results and engine-authored completions relayed to the guest

7. The execution frontier

The RM client ladder reaches the compute engine (a GR0 channel plus a compute-class object). Forwarded guest CUDA launches now execute on the engine: the inline launch descriptor installs and is read, the grid dispatches, the engine authors the completion, the host relays it, and the guest's synchronisation wait resolves — verified on fresh boots with the vector-add, matrix-multiply and transformer clients. The leading edge is the large-buffer staged-transfer class (under repair), reproducibility across fresh boots, and per-guest execution isolation — see Execution-Frontier.md. A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step.

8. Safety and platform discipline

• Dedicated GPU deployment. One GPU serves the vGPU service; the host uses another GPU or integrated graphics. The distribution driver stays unmodified and is unbound from that device only while the service is active.
• The board's power domain belongs to the platform. Releasing the last driver can power-gate the board and the power-on sequence can fail; the ownership path holds the whole board in a held-active state across the window, verifies config-space liveness, and aborts on any power-loss signature without retry.
• Supervised cycles. The device is driven by the module under supervised cycles with a written recovery plan. A failed mutating operation parks the device in a locked fail-safe and is never auto-retried; any reboot requires explicit human approval with a written reason and post-reboot plan first.
• Fail closed. No forwarding seam programs guest-derived page tables into the real MMU until the population path is proven end to end; the current forwarded path submits engine work only through module-built scratch page tables mapping already-resolved host pages.

9. Sharing policy and isolation

• Dedicated GPU (chosen): one GPU serves the vGPU service; a multi-GPU host gives the distribution driver its own GPU.
• Shared single GPU with the host driver bound: no conforming mechanism exists on this generation without vendor interfaces; it would require relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
• Isolation is one real GSP client per mdev with per-client page tables; host access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top (GMMU-Multi-Tenant-Protection.md).
• Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress.

10. Open questions

• Firmware/host driver ABI match for the supported family on the development GPU.
• GPU unbind/reset lifecycle and recovery after guest faults.
• Which guest allocations need real VRAM and which can stay in system memory.
• Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.

11. Related documentation

• ../docs/03-architecture/08-physical-gpu-integration.md — canonical: mechanism choice and device bring-up.
• ../docs/03-architecture/11-physical-backend-guest-wiring.md — canonical: the guest-wiring design.
• Execution-Frontier.md, Execution-Plane.md — the execution frontier and the software execution plane.
• Host-Module.md — the module that owns both planes.
• Introduction-to-GMMU.md — page-table seams and scratch mappings.
• Hardware-Software-Matrix.md, Project-Status.md — support scope and the honest status picture.