WikiBot: Fix repository links: point to the cgit tree URLs

2026-10-12T04:40:59Z

Fix repository links: point to the cgit tree URLs

← Older revision		Revision as of 04:40, 12 October 2026
Line 3:		Line 3:
	= Physical GPU Backend and Guest Wiring =		= Physical GPU Backend and Guest Wiring =

	<blockquote>'''Scope.''' The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] and [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>]; the docs tree wins on disagreement.</blockquote>		<blockquote>'''Scope.''' The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>]; the docs tree wins on disagreement.</blockquote>

	Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> planned end state.		Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> planned end state.
Line 153:		Line 153:
	== 11. Related documentation ==		== 11. Related documentation ==

	* [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] — canonical: mechanism choice and device bring-up.		* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] — canonical: mechanism choice and device bring-up.
	* [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — canonical: the guest-wiring design.		* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — canonical: the guest-wiring design.
	* [[The Execution Frontier\|<code>Execution-Frontier.md</code>]], [[The vGSP Execution Plane (vEP)\|<code>Execution-Plane.md</code>]] — the GR wall and the software execution plane.		* [[The Execution Frontier\|<code>Execution-Frontier.md</code>]], [[The vGSP Execution Plane (vEP)\|<code>Execution-Plane.md</code>]] — the GR wall and the software execution plane.
	* [[The Host Module (vgsp.ko)\|<code>Host-Module.md</code>]] — the module that owns both planes.		* [[The Host Module (vgsp.ko)\|<code>Host-Module.md</code>]] — the module that owns both planes.
	* [[Introduction to GMMU\|<code>Introduction-to-GMMU.md</code>]] — page-table seams and scratch mappings.		* [[Introduction to GMMU\|<code>Introduction-to-GMMU.md</code>]] — page-table seams and scratch mappings.
	* [[Hardware and Software Matrix\|<code>Hardware-Software-Matrix.md</code>]], [[Project Status\|<code>Project-Status.md</code>]] — support scope and the honest status picture.		* [[Hardware and Software Matrix\|<code>Hardware-Software-Matrix.md</code>]], [[Project Status\|<code>Project-Status.md</code>]] — support scope and the honest status picture.

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

2026-10-12T04:29:50Z

Carry the site-wide About/licensing message on every page (2026-10-12)

← Older revision		Revision as of 04:29, 12 October 2026
Line 1:		Line 1:
			{{About message}}

	= Physical GPU Backend and Guest Wiring =		= Physical GPU Backend and Guest Wiring =

WikiBot: Add page from tracked wiki source (2026 IA update)

2026-10-12T04:22:13Z

Add page from tracked wiki source (2026 IA update)

New page

= Physical GPU Backend and Guest Wiring =

<blockquote>'''Scope.''' The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] and [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> planned end state.

== 1. The problem and its constraints ==

The guest's real work (copies, memory operations, grid launches) must execute on the physical GPU. The constraints: one DKMS kernel module on the host, no daemon and no new host userspace; host and guest drivers unmodified, with no paravirtual enlightenment in the guest; all emulation and translation logic inside the module; and real execution — compute is not emulated in software.

== 2. Why the host driver is not the path ==

{| class="wikitable"
|-
! Interface
! What it offers
! Reachable from our module?
|-
| Host driver (distribution build)
| resource management, scheduling, memory
| No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface
|-
| Host driver peer-memory exports
| pin/map memory for peer devices
| Partially — memory only, no submission
|-
| Vendor vGPU host stack
| mediated work submission and scheduling
| Requires the licensed vGPU host branch and a vGPU-aware guest — excluded
|-
| SR-IOV virtual functions
| hardware-partitioned device
| Not available on this GPU generation
|-
| The GPU's own GSP firmware
| the resource manager server itself
| Yes — once the module owns the device; this is the path used
|}

With an unmodified host driver there is no callable submission interface into it, so the module must own the device and talk to the hardware itself.

== 3. Mechanism options and the decision ==

<pre> (A) Proxy through the host driver: module -> host driver's RM -> real channel
-> GPU. Needs a modified host driver or a vendor interface — not implementable.
(B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
(C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client; scheduling,
page tables and context management stay the firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.</pre>

Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.

== 4. The chosen mechanism: bring-up ladder ==

The module binds the physical GPU as its driver while the distribution driver is unbound (dedicated-GPU deployment, §9). Each stage is reversible and independently verified:

# '''Ownership''' — the board's power domain is held active across the window (the platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. <code>[live]</code>
# '''Firmware staging''' — the matching GSP image is fetched and validated (ELF64 RISC-V; loadable section, version, hardware signature); no device state touched. <code>[live]</code>
# '''Boot''' — protected region in video memory, secure-verification stage, booter, RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. <code>[live]</code>
# '''Host RM client''' — the module speaks the already-mapped RPC protocol to the real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". <code>[live]</code>
# '''Guest mediation''' — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. <code>[wip]</code>
# '''Work path''' — decoded guest pushes become real push streams; resolved host pages map through module-built scratch page tables; the firmware schedules the work. <code>[gated]</code>
# '''Completion relay''' — hardware-completed semaphores land in guest-polled memory; events and faults relay through the existing MSI-X injection path. <code>[gated]</code>

== 5. The two planes and the wiring ==

The '''emulation plane''' serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The '''physical backend plane''' owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.

<pre>GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
-----------------------------------------------------------------------------
INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
claim: power domain held -> distribution driver released -> module
binds the device -> liveness check [live]
boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
mdev + ladder: backend joins the live session; real objects,
channels, first real CE copy [live]
guest: nvidia.ko probes; emulation serves handshake, static config,
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
page tables; GR: launch reaches the front-end, descriptor not seen [wall]
DONE engine authors the release in guest memory; host relays via MSI-X [gated]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
never auto-retried; recovery is supervised [live]</pre>

== 6. Proven capabilities ==

{| class="wikitable"
|-
! Capability
! Status
|-
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X)
| done <code>[live]</code>
|-
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy)
| done — 64/64 payload round-trip <code>[live]</code>
|-
| Software bridge for decoded memory/state operations
| in progress <code>[wip]</code>
|-
| Device ownership, firmware staging, board-liveness checks
| verified live <code>[live]</code>
|-
| Power domain held active across the ownership window
| verified across supervised cycles <code>[live]</code>
|-
| Real GSP boot by the module, through init-done and clean release
| done, no reboot on clean cycles <code>[live]</code>
|-
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy)
| done <code>[live]</code>
|-
| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay
| default-off, verified live; one relayed-copy mismatch under triage <code>[gated]</code>
|-
| CUDA kernel execution (<code>vectorAdd</code>)
| open <code>[wall]</code> — the launch-descriptor install path; correct results are the <code>[target]</code>
|}

== 7. The execution frontier ==

The RM client ladder reaches the compute engine (a GR0 channel plus a compute-class object). A forwarded guest CUDA launch now reaches the front-end (ring entry fetched, methods executed) but the inline launch-descriptor install has not been observed and no grid has dispatched; context validity at dispatch is ruled out. The wall is the form of the forwarded launch method stream — see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]. A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step.

== 8. Safety and platform discipline ==

* '''Dedicated GPU deployment.''' One GPU serves the vGPU service; the host uses another GPU or integrated graphics. The distribution driver stays unmodified and is unbound from that device only while the service is active.
* '''The board's power domain belongs to the platform.''' Releasing the last driver can power-gate the board and the power-on sequence can fail; the ownership path holds the whole board in a held-active state across the window, verifies config-space liveness, and aborts on any power-loss signature without retry.
* '''Supervised cycles.''' The device is driven by the module under supervised cycles with a written recovery plan. A failed mutating operation parks the device in a locked fail-safe and is never auto-retried; any reboot requires explicit human approval with a written reason and post-reboot plan first.
* '''Fail closed.''' No forwarding seam programs guest-derived page tables into the real MMU until the population path is proven end to end; the current forwarded path submits engine work only through module-built scratch page tables mapping already-resolved host pages.

== 9. Sharing policy and isolation ==

* '''Dedicated GPU (chosen):''' one GPU serves the vGPU service; a multi-GPU host gives the distribution driver its own GPU.
* '''Shared single GPU with the host driver bound:''' no conforming mechanism exists on this generation without vendor interfaces; it would require relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
* Isolation is one real GSP client per mdev with per-client page tables; host access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]]).

== 10. Open questions ==

* Firmware/host driver ABI match for the supported family on the development GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.

== 11. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] — canonical: mechanism choice and device bring-up.
* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — canonical: the guest-wiring design.
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — the GR wall and the software execution plane.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the module that owns both planes.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]] — page-table seams and scratch mappings.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]], [[Project Status|<code>Project-Status.md</code>]] — support scope and the honest status picture.

Physical GPU Backend and Guest Wiring - Revision history

WikiBot: Fix repository links: point to the cgit tree URLs

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

WikiBot: Add page from tracked wiki source (2026 IA update)