WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

2026-10-12T04:40:08Z

Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

← Older revision		Revision as of 04:40, 12 October 2026
Line 3:		Line 3:
	= The vGSP Execution Plane (vEP) =		= The vGSP Execution Plane (vEP) =

	<blockquote>'''Canonical, maintained version of this page:''' [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>]. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/README.md <code>../docs/README.md</code>].</blockquote>		<blockquote>'''Canonical, maintained version of this page:''' [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>]. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

	Status: 2026-10-08, tree <code>master @ 1f571dda</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.		Status: 2026-10-08, tree <code>master @ 1f571dda</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.
Line 80:		Line 80:
	----		----

	''Canonical document: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/13-vexecution-plane.md <code>docs/03-architecture/13-vexecution-plane.md</code>]. See also: [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/09-full-system-picture.md full system picture], [https://git.vgpu.io/open-iov/~~blob/master~~/docs/03-architecture/11-physical-backend-guest-wiring.md physical backend guest wiring], [https://git.vgpu.io/open-iov/~~blob/master~~/docs/04-implementation/03-execution-frontier.md the execution frontier].''		''Canonical document: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>docs/03-architecture/13-vexecution-plane.md</code>]. See also: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/09-full-system-picture.md full system picture], [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md physical backend guest wiring], [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md the execution frontier].''

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

2026-10-12T04:29:44Z

Carry the site-wide About/licensing message on every page (2026-10-12)

← Older revision		Revision as of 04:29, 12 October 2026
Line 1:		Line 1:
			{{About message}}

	= The vGSP Execution Plane (vEP) =		= The vGSP Execution Plane (vEP) =

WikiBot: Re-sync page from tracked wiki source: fix cross-document links and diagram; refresh status

2026-10-12T04:10:01Z

Re-sync page from tracked wiki source: fix cross-document links and diagram; refresh status

Show changes

WikiBot: Remove unpaired blockquote tags left by reference cleanup

2026-10-12T03:58:28Z

Remove unpaired blockquote tags left by reference cleanup

New page

= The vGSP Execution Plane (vEP) =

Status: 2026-10-08, tree <code>master @ 1f571dda</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.

== 1. What it is ==

'''vGSP (virtual GSP)''' is our host-side reimplementation of the GPU System Processor interface: the virtual device plus the emulated GSP-RM service the stock guest driver talks to, implemented by <code>vgsp.ko</code>. The '''vGSP Execution Plane (vEP)''' is the half of vGSP that turns guest-authored GPU work into physical execution and guest-visible completion: channel ring and USERD discovery, GPU VA-to-PA truth, semaphore-release application, completion counters and notifiers, QMD handling for forwarded launches, and submission to the physical GPU's engines under the module-owned backend session.
The other half — the control plane (boot handshake, message queues, RPC, RM objects, BAR shadows) — makes the guest driver ''run''. The vEP makes the guest's work ''run''.
'''Four work domains''' organize the subsystem:

{| class="wikitable"
|-
! Domain
! Question
! Implemented by
|-
| Scheduler
| which channel's work runs, at what pace
| doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request
|-
| Allocator
| where rings, USERDs, counters, notifiers and scratch windows live
| ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow
|-
| Control
| which RM objects/controls shape execution
| channel and event registration, <code>SET_PAGE_DIRECTORY</code>, <code>PROMOTE_CTX</code>, engine topology, the mode matrix
|-
| Share
| how clients/mdevs share the one GPU
| mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned
|}


== 2. The shape of the plane ==

<pre class="mermaid">flowchart TB
subgraph GUEST["Guest VM - stock software"]
APP["CUDA application -> libcuda -> nvidia.ko push + QMD + GPFIFO + doorbell"]
end
subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
DB["doorbell steering"] --> PUMP["channel pump"]
PUMP --> DISC["ring/USERD discovery + pins"]
DISC --> TRUTH["VA-to-PA truth walk, rescue, pins"]
TRUTH --> DEC["push decode + QMD capture"]
DEC --> REL["release apply + counters + notifiers"]
DEC --> WQ["ordered workqueue one in-flight request"]
WQ --> SUB["ladder submission mode matrix"]
end
subgraph PHY["Physical backend"]
LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
GSP --> ENG["GR / CE engines"]
end
APP -->|"MMIO + DMA"| DB
REL -->|"release, counter, event"| APP
SUB --> LAD
ENG -->|"engine-authored release"| SUB</pre>
ASCII fallback:

<syntaxhighlight lang="text">guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
-> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
-> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
-> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
-> engine-authored release -> poll -> completion publication</syntaxhighlight>

== 3. One CUDA kernel launch, end to end ==

# '''Launch build (guest).''' <code>libcuda</code>/<code>nvidia.ko</code> build the GR push: context methods, the inline launch descriptor (QMD), SASS and parameter pointers, and a release target. They append a GPFIFO entry and ring the BAR0 doorbell. <code>[live]</code> observed with the stock driver.
# '''Steering and truth (host).''' The doorbell wakes the service thread; the pump resolves the channel's ring VA and USERD, reads <code>GP_PUT</code>, fetches the GPFIFO entry and decodes the method stream. Translations are read-back proven from driver-published page directories, pins, or content-rescue. <code>[live]</code> for CE; <code>[wip]</code> for GR (discovery now succeeds, see §4).
# '''Capture.''' The decoder accumulates the QMD window and captures the launch candidate; the sema service queues one request on the single ordered workqueue. <code>[live]</code> observed (ring consumed, methods execute).
# '''Physical submit.''' The backend relocates the QMD, copies program/param pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to the ladder channel's GPFIFO and rings the real doorbell. <code>[gated]</code> default-off; latched on a proven real GR submission.
# '''Engine.''' The front-end fetches the entry and executes methods; then the '''inline-QMD install''' must materialise the descriptor for the engine to read, currently '''<code>[wall]</code>''' — not observed. Context validity, destination mapping and fetch/alignment are exonerated; the recovered-method-stream form is the investigation.
# '''Completion.''' The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X wakes the guest. CE work is live in this shape <code>[gated]</code>; for GR the completion-counter publication is the current second face of the wall.

== 4. How work is found: discovery, truth, completion ==

* '''Ring/USERD discovery''' has three cooperating sources: injected pins, a kernel-native trusted walk of driver-published page directories, and content-keyed discovery when neither names the ring. Content discovery scans '''both apertures''' (emulated vidmem and guest RAM), runs the FB scan first on a 2-second cadence, pairs a ring with the rank's staged ctx-init push, and validates the pair before pinning. Since the 2026-10-08 re-arm work (<code>7abd8f87</code>, <code>66388560</code>) the discovery no longer retires permanently at a pass cap.
* '''The semaphore service''' decodes releases, translates their target pages through the trusted resolver plus a driver-published-VAS pool resolver, defers untranslatable releases with retry telemetry, and applies each release exactly once — in software, or by letting the engine author it under the relay (default off).
* '''Completion counters''' use a fixed 20-slot pool (a partial live channel count poisoned slots before the fix); each decoded release lands in the channel's exact slot. '''Notifiers''' are serviced by the pump, and family events are delivered as RM events + MSI-X through the guest's registered event objects.

== 5. Challenges, honestly ==

* '''GR dispatch wall.''' The forward path runs front-end methods, the context is valid and the destination window is accepted — and the descriptor install is still not observed. The mode matrix carries dozens of single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory block, lane controls, shadow windows, pointer relocation).
* '''Completion publication.''' A boot where GR rings were discovered and releases applied still saw the guest re-queue the same push and give up. A later <code>sema_dbg=Y</code> run settled it: the release was written to the exact slot, the notifier cleared and the event posted, while the completion counter publication never wrote because the fixed-total autoreg mapping makes it a structural no-op. The leading reading is the counter-slot publication; the GP_GET write-back is the secondary, still-open candidate. Both are debug-gated today.
* '''Reproducibility (ISSUE-0001).''' The green/stall split is per-boot. The first fix wave landed and the earlier CE freeze is gone; the stall moved to GR ring discovery, which the latest discovery work addresses. Acceptance is 10 cold boots green, pending.
* '''Multiple clients (ISSUE-0002).''' A second CUDA process wedges in guest UVM because host per-client sema/GR state is not reset when the guest re-runs its per-client boot handshake. Per-client state containers and reset are the designed path.

== 6. Portability ==

The '''architecture''' ports: observe pushes, resolve truth, decode, execute/forward, publish completion. So does the '''RPC/RM object model''' and the mode-matrix practice. What does '''not''' port is address-level fact: ring conventions, VA windows, QMD offsets, context-image layout, PTE encodings, engine inventories and scratch layouts must be re-derived per driver version and GPU architecture by measurement, never guessed. A new version enters as a profile plus a measured delta; driver 610 additionally sits behind an explicit human approval gate.

== 7. Future ==

The planned next steps are reproducibility acceptance, the descriptor-install and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger where possible), the arbitrary-kernel acceptance suite, then the near-native performance program. The unsolved future classes — multi-tenant fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction. 3D graphics reuses this pipeline once compute dispatch is proven; it adds engine breadth (video/image), more channels and synchronisation volume, and a much larger method vocabulary, but not a new plane.
-----

The vGSP Execution Plane (vEP) - Revision history

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

WikiBot: Re-sync page from tracked wiki source: fix cross-document links and diagram; refresh status

WikiBot: Remove unpaired blockquote tags left by reference cleanup