Difference between revisions of "The vGSP Execution Plane (vEP)"

From Open-IOV
Jump to navigation Jump to search
m (2026-11-12 status refresh: X/GLX fully green, GPU-rendered captures, direct-GL command-stream delivery wall)
m (2026-11-20 late status refresh: corrected beef-lane diagnosis (PBDMA fetch proven; host-side forensic spin/no-write retire was the floor); r73 beef election + r75 shadow-VTG; v1.0.0-rc1 retracted; latest acceptance red on known flake/race)
 
(12 intermediate revisions by the same user not shown)
Line 5: Line 5:
<blockquote>'''Canonical, maintained version of this page:''' [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>]. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>
<blockquote>'''Canonical, maintained version of this page:''' [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>]. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>


Status: 2026-11-12, tree <code>master @ 800c108f</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.
Status: 2026-11-20, tree <code>master @ 3f34d58a</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.


== 1. What it is ==
== 1. What it is ==
Line 65: Line 65:
== 5. Challenges, honestly ==
== 5. Challenges, honestly ==


* '''The CUDA battery gate is closed (2026-11-19).''' The page-tree-init push and its successors were resolved by the round-32..59 sequence: CE rings resolved from content (<code>36cbfe66</code>), the channel-VAS tracker wrap (<code>47c849a0</code>), GR ring pairing (<code>3be286ac</code>), the CE pool layout (<code>d7d2164e</code>), GR ctx-init semaphores (<code>e462ab74</code>), CE pool ordinal pairing (<code>5369dbf8</code>), the payload-2 pushes (<code>443dc9c8</code>, <code>eb386166</code>), dropping the host-path SW methods and the <code>SET_VALID_SPAN_OVERFLOW_AREA</code> trio that reset FECS (<code>41c97e61</code>), the inline re-rooter program page (<code>951790a3</code>), the pa=0 fill/relocate class (<code>f384a1dc</code>), the compute-lane retarget gate (<code>335c05b0</code>) and the launch-lottery retry (<code>8566a61f</code>). Two consecutive clean-boot accepts (<code>VADD bad=0</code>, <code>STATUS OK</code>) and a green live <code>t/03</code> closed the gate (<code>EXPECT_CUDA_BATTERY=closed</code>), re-proven in r63 and later rounds. The TSG engine-inheritance checkpoint was falsified and reverted (<code>b26452fb</code>).
* '''The staged-transfer class (compute).''' Some large multi-chunk CUDA clients still mis-verify; the failure modes are proven (a release applied before the op it owes, a ledger-vs-writer page divergence, a readback racing a writeback) and the fixes are in flight.
* '''The staged-transfer class (compute).''' Some large multi-chunk CUDA clients still mis-verify; the failure modes are proven (a release applied before the op it owes, a ledger-vs-writer page divergence, a readback racing a writeback) and the fixes are in flight.
* '''Every-boot reproducibility.''' The boot-poisoning defect (wrong status-queue consumption cursor under the driver's swap mode) and the post-reboot ring-block ownership defect were root-caused and fixed; green clients reproduce across fresh boots, and the formal every-boot acceptance campaign is the remaining work.
* '''Every-boot reproducibility.''' The boot-poisoning defect (wrong status-queue consumption cursor under the driver's swap mode) and the post-reboot ring-block ownership defect were root-caused and fixed; green clients reproduce across fresh boots, and the formal every-boot acceptance campaign is the remaining work.
* '''Multiple clients and per-guest isolation.''' Two guests with different emulated VRAM sizes coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress.
* '''Multiple clients and per-guest isolation.''' Two guests with different emulated VRAM sizes coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress.
* '''The graphics pixel frontier.''' Graphics work rides this plane in three shapes (verbatim 3D method-stream replay, synthesized launches, and the direct GL client's own GR/CE channels). The display/KMS path is served by the emulated display engine (modeset push buffers plus the notifier/<code>UPDATE</code> completion, class-aware arm), so NvKms registers, X settles and the stock NVIDIA GLX path is fully green (<code>glxinfo -B</code> rc 0, OpenGL 4.6.0 direct, <code>glxgears</code> ~520 FPS); the X DDX waits are fixed, and the guest's real GL draws now forward and execute on the physical GPU (<code>backend_ok=5-7</code>, <code>backend_timeout=0</code>); the DDX 3D replay executes, the 2D path is green, the direct-GL ring/VAS and USERD machinery is hardened, and the first non-black GPU-rendered frames are proven (500/500 off-screen frames per run plus captured 45.4 s and 50 s videos). No non-black '''window/screen''' frame has been produced through the NVIDIA path yet: the direct-GL client's own 59,556-byte setup push does not yet resolve/read host-side under its page-directory churn (so its draw sections never open), glxgears' window is black, and X screen-init is a per-boot lottery, so those are the named blockers. The lanes and the ladder are on [[Graphics Execution|<code>Graphics-Execution.md</code>]].
* '''The graphics pixel frontier.''' Graphics work rides this plane in three shapes (verbatim 3D method-stream replay, synthesized launches, and the direct GL client's own GR/CE channels). The display/KMS path is served by the emulated display engine (modeset push buffers plus the notifier/<code>UPDATE</code> completion, class-aware arm), so NvKms registers, X settles and the stock NVIDIA GLX path brings up on a good boot (<code>glxinfo -B</code> OpenGL 4.6.0 direct; the current direct-GL stack measures 0.255 FPS in <code>glxgears</code>, r64-r72); the X DDX waits are fixed (the idle-DMA/ScreenInit, USERD-latch, re-probe, kernel-mode-spin and flip-event walls are closed, and X has stayed green &gt;10 minutes), and the guest's real GL draws now forward and execute on the physical GPU (<code>backend_ok=5-7</code>, <code>backend_timeout=0</code>); captured GL sections execute on the engine (<code>GR LAUNCH submitted</code> / <code>GUEST GR LAUNCH OK</code>) with the data window populated, the DDX 3D replay executes, the 2D path is green, the direct-GL ring/VAS and USERD machinery is hardened, and the first non-black GPU-rendered frames are proven (500/500 off-screen frames per run plus captured 45.4 s and 50 s videos). The direct-GL submission is fully decoded (no method-grammar draw header; the work is macro-driven — data-block blobs plus small state pushes invoking MME macros), the recycled pushes are recovered, and the client's draw-state push opens, captures and reaches the engine (<code>GR LAUNCH submitted</code> + OK, golden MME overlay accepted). The 11-16 wave then proved the '''host-side chain deterministic end to end''': the real frame pushes are admitted (persisted colour-target bind, draw markers inside counted runs, a 14.8-thousand-word window replayed as 40 chunks, a 128-slot candidate ring), the beef draw lane latches a '''clean DS''' (<code>DS-FAST2 ... ds_after=80000000</code>, c963) at the live-derived GS attribute map, capture-less boots get a '''golden 2799-word MME pack''', and <code>MME-CALL-HOLD -&gt; MME-CARRY-SAVE -&gt; MME-CARRY-FIRE -&gt; MME-PACK-REARM -&gt; MME-SA-REARM</code> fires with the prelude latch kept across the channel recycle. The '''late-11-16 wave crossed the guest submission gap''': an offscreen EGL/GBM shim gives GZDoom a real NVIDIA GL context without Xorg/DDX, and the real op=4 <code>CALL(0)</code> <code>0x3800</code> draw is admitted and replayed; <code>CALL(0)</code> executes its captured macro without trapping (the immediate resolves through client SA[0] then golden, and not-resident calls are elided), and c1027 reached the first DS with real stage data. The '''GL pixel gate then closed on 2026-11-18''' (<code>b5a68b9a</code>) without that per-draw macro body: the first clear/CALL(0)-bearing window is archived and replayed, the engine clear stores red on the guest surface, the readback fence is delivered with the copy (11.03-11.05 s measured down to 2.3-7.2 ms), and the CE D2H copy's source VA is adopted as the readback surface. Two consecutive boots read <code>GL_FRAME nonblack=291840 px0=(255,0,0)</code>, live <code>t/04</code> was green on that proof, and <code>EXPECT_GL_PIXELS=closed</code> remains set (re-confirmed by the r62b two-boot proof; the latest full run is red on the readback pin/copy race). What remains is the '''window/screen present''': <code>glxgears</code> measures 0.255 FPS on the direct-GL stack (r64-r72). r72b corrected the earlier diagnosis: the PBDMA fetches the direct-GL client-ring launch (<code>h=0xbeef0100</code>, <code>push=0x400108c</code>, 59580 B) in ~1 ms and the engine writes the channel release; the floor was host-side (a 2.8 s forensic MME spin enabled by debug params, plus a ~2.45 s no-write retirement wait when the window stores 0 pages). r73's beef read-only election cuts the retire wait (1337 -&gt; 12 ms) and r75 mirrors the VTG program image into the win0 shadow (<code>3f34d58a</code>); the remaining blockers are the draw's zero stores plus the engine <code>Illegal Instruction Encoding</code> fault, and the X/GLX bring-up lottery (a hung GLX client blocks in <code>poll()</code> on an unsignalled dma-fence <code>sync_file</code>; a DDX channel-idle timeout kills DMA init). A GZDoom video was separately delivered through the guest's '''software GL''' path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim. Those are the named blockers. The lanes and the ladder are on [[Graphics Execution|<code>Graphics-Execution.md</code>]].


== 6. Portability ==
== 6. Portability ==

Latest revision as of 20:42, 20 November 2026

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

The vGSP Execution Plane (vEP)

Canonical, maintained version of this page: ../docs/03-architecture/13-vexecution-plane.md. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: ../docs/README.md.

Status: 2026-11-20, tree master @ 3f34d58a. Tags: [live] proven with the stock guest driver; [gated] implemented, default-off or parameter-gated; [wip] partial; [wall] current blocker; [target] named success latch; [planned] designed, not built.

1. What it is

vGSP (virtual GSP) is our host-side reimplementation of the GPU System Processor interface: the virtual device plus the emulated GSP-RM service the stock guest driver talks to, implemented by vgsp.ko. The vGSP Execution Plane (vEP) is the half of vGSP that turns guest-authored GPU work into physical execution and guest-visible completion: channel ring and USERD discovery, GPU VA-to-PA truth, semaphore-release application, completion counters and notifiers, QMD handling for forwarded launches, and submission to the physical GPU's engines under the module-owned backend session.

The other half — the control plane (boot handshake, message queues, RPC, RM objects, BAR shadows) — makes the guest driver run. The vEP makes the guest's work run. The canonical document defines the name and rejects the alternates ("work plane", "submission plane", "forwarding plane", ...) that would collide with existing project vocabulary.

Four work domains organize the subsystem:

Domain Question Implemented by
Scheduler which channel's work runs, at what pace doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request
Allocator where rings, USERDs, counters, notifiers and scratch windows live ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow
Control which RM objects/controls shape execution channel and event registration, SET_PAGE_DIRECTORY, PROMOTE_CTX, engine topology, the mode matrix
Share how clients/mdevs share the one GPU mdev backend refcount, one real session under a single lock, one workqueue; two guests with different emulated VRAM sizes coexist at the device level, per-guest execution isolation is in progress

2. The shape of the plane

The vGSP execution plane (vEP): one CUDA launch, end to end.
guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
  -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
  -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
  -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
  -> engine-authored release -> poll -> completion publication

3. One CUDA kernel launch, end to end

  1. Launch build (guest). libcuda/nvidia.ko build the GR push: context methods, the inline launch descriptor (QMD), SASS and parameter pointers, and a release target. They append a GPFIFO entry and ring the BAR0 doorbell. [live] observed with the stock driver.
  2. Steering and truth (host). The doorbell wakes the service thread; the pump resolves the channel's ring VA and USERD, reads GP_PUT, fetches the GPFIFO entry and decodes the method stream. Translations are read-back proven from driver-published page directories, pins, or content-rescue. [live] for CE; [wip] for GR (discovery now succeeds, see §4).
  3. Capture. The decoder accumulates the QMD window and captures the launch candidate; the sema service queues one request on the single ordered workqueue. [live] observed (ring consumed, methods execute).
  4. Physical submit. The backend relocates the QMD, copies program/param pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to the ladder channel's GPFIFO and rings the real doorbell. [live] — exercised on every verified guest launch (default-off while the seams are validated).
  5. Engine. The front-end fetches the entry and executes methods; the inline launch descriptor installs and the engine reads it, the grid dispatches to the SMs, and the engine authors the release. [live] — context validity, destination mapping and fetch/alignment were exonerated on the way.
  6. Completion. The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X wakes the guest. [live] for CE and GR: engine-authored completions relay to the guest and its synchronisation waits resolve.

4. How work is found: discovery, truth, completion

  • Ring/USERD discovery has three cooperating sources: injected pins, a kernel-native trusted walk of driver-published page directories, and content-keyed discovery when neither names the ring. Content discovery scans both apertures (emulated vidmem and guest RAM), runs the FB scan first on a 2-second cadence, pairs a ring with the rank's staged ctx-init push, and validates the pair before pinning. Since the 2026-10-08 re-arm work (7abd8f87, 66388560) the discovery no longer retires permanently at a pass cap.
  • The semaphore service decodes releases, translates their target pages through the trusted resolver plus a driver-published-VAS pool resolver, defers untranslatable releases with retry telemetry, and applies each release exactly once — in software, or by letting the engine author it under the relay (default off).
  • Completion counters use a fixed 20-slot pool (a partial live channel count poisoned slots before the fix); each decoded release lands in the channel's exact slot. Notifiers are serviced by the pump, and family events are delivered as RM events + MSI-X through the guest's registered event objects.

5. Challenges, honestly

  • The CUDA battery gate is closed (2026-11-19). The page-tree-init push and its successors were resolved by the round-32..59 sequence: CE rings resolved from content (36cbfe66), the channel-VAS tracker wrap (47c849a0), GR ring pairing (3be286ac), the CE pool layout (d7d2164e), GR ctx-init semaphores (e462ab74), CE pool ordinal pairing (5369dbf8), the payload-2 pushes (443dc9c8, eb386166), dropping the host-path SW methods and the SET_VALID_SPAN_OVERFLOW_AREA trio that reset FECS (41c97e61), the inline re-rooter program page (951790a3), the pa=0 fill/relocate class (f384a1dc), the compute-lane retarget gate (335c05b0) and the launch-lottery retry (8566a61f). Two consecutive clean-boot accepts (VADD bad=0, STATUS OK) and a green live t/03 closed the gate (EXPECT_CUDA_BATTERY=closed), re-proven in r63 and later rounds. The TSG engine-inheritance checkpoint was falsified and reverted (b26452fb).
  • The staged-transfer class (compute). Some large multi-chunk CUDA clients still mis-verify; the failure modes are proven (a release applied before the op it owes, a ledger-vs-writer page divergence, a readback racing a writeback) and the fixes are in flight.
  • Every-boot reproducibility. The boot-poisoning defect (wrong status-queue consumption cursor under the driver's swap mode) and the post-reboot ring-block ownership defect were root-caused and fixed; green clients reproduce across fresh boots, and the formal every-boot acceptance campaign is the remaining work.
  • Multiple clients and per-guest isolation. Two guests with different emulated VRAM sizes coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress.
  • The graphics pixel frontier. Graphics work rides this plane in three shapes (verbatim 3D method-stream replay, synthesized launches, and the direct GL client's own GR/CE channels). The display/KMS path is served by the emulated display engine (modeset push buffers plus the notifier/UPDATE completion, class-aware arm), so NvKms registers, X settles and the stock NVIDIA GLX path brings up on a good boot (glxinfo -B OpenGL 4.6.0 direct; the current direct-GL stack measures 0.255 FPS in glxgears, r64-r72); the X DDX waits are fixed (the idle-DMA/ScreenInit, USERD-latch, re-probe, kernel-mode-spin and flip-event walls are closed, and X has stayed green >10 minutes), and the guest's real GL draws now forward and execute on the physical GPU (backend_ok=5-7, backend_timeout=0); captured GL sections execute on the engine (GR LAUNCH submitted / GUEST GR LAUNCH OK) with the data window populated, the DDX 3D replay executes, the 2D path is green, the direct-GL ring/VAS and USERD machinery is hardened, and the first non-black GPU-rendered frames are proven (500/500 off-screen frames per run plus captured 45.4 s and 50 s videos). The direct-GL submission is fully decoded (no method-grammar draw header; the work is macro-driven — data-block blobs plus small state pushes invoking MME macros), the recycled pushes are recovered, and the client's draw-state push opens, captures and reaches the engine (GR LAUNCH submitted + OK, golden MME overlay accepted). The 11-16 wave then proved the host-side chain deterministic end to end: the real frame pushes are admitted (persisted colour-target bind, draw markers inside counted runs, a 14.8-thousand-word window replayed as 40 chunks, a 128-slot candidate ring), the beef draw lane latches a clean DS (DS-FAST2 ... ds_after=80000000, c963) at the live-derived GS attribute map, capture-less boots get a golden 2799-word MME pack, and MME-CALL-HOLD -> MME-CARRY-SAVE -> MME-CARRY-FIRE -> MME-PACK-REARM -> MME-SA-REARM fires with the prelude latch kept across the channel recycle. The late-11-16 wave crossed the guest submission gap: an offscreen EGL/GBM shim gives GZDoom a real NVIDIA GL context without Xorg/DDX, and the real op=4 CALL(0) 0x3800 draw is admitted and replayed; CALL(0) executes its captured macro without trapping (the immediate resolves through client SA[0] then golden, and not-resident calls are elided), and c1027 reached the first DS with real stage data. The GL pixel gate then closed on 2026-11-18 (b5a68b9a) without that per-draw macro body: the first clear/CALL(0)-bearing window is archived and replayed, the engine clear stores red on the guest surface, the readback fence is delivered with the copy (11.03-11.05 s measured down to 2.3-7.2 ms), and the CE D2H copy's source VA is adopted as the readback surface. Two consecutive boots read GL_FRAME nonblack=291840 px0=(255,0,0), live t/04 was green on that proof, and EXPECT_GL_PIXELS=closed remains set (re-confirmed by the r62b two-boot proof; the latest full run is red on the readback pin/copy race). What remains is the window/screen present: glxgears measures 0.255 FPS on the direct-GL stack (r64-r72). r72b corrected the earlier diagnosis: the PBDMA fetches the direct-GL client-ring launch (h=0xbeef0100, push=0x400108c, 59580 B) in ~1 ms and the engine writes the channel release; the floor was host-side (a 2.8 s forensic MME spin enabled by debug params, plus a ~2.45 s no-write retirement wait when the window stores 0 pages). r73's beef read-only election cuts the retire wait (1337 -> 12 ms) and r75 mirrors the VTG program image into the win0 shadow (3f34d58a); the remaining blockers are the draw's zero stores plus the engine Illegal Instruction Encoding fault, and the X/GLX bring-up lottery (a hung GLX client blocks in poll() on an unsignalled dma-fence sync_file; a DDX channel-idle timeout kills DMA init). A GZDoom video was separately delivered through the guest's software GL path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim. Those are the named blockers. The lanes and the ladder are on Graphics-Execution.md.

6. Portability

The architecture ports: observe pushes, resolve truth, decode, execute/forward, publish completion. So does the RPC/RM object model and the mode-matrix practice. What does not port is address-level fact: ring conventions, VA windows, QMD offsets, context-image layout, PTE encodings, engine inventories and scratch layouts must be re-derived per driver version and GPU architecture by measurement, never guessed. A new version enters as a profile plus a measured delta; driver 610 additionally sits behind an explicit human approval gate.

7. Future

The planned next steps are the staged-transfer fixes, the every-boot acceptance campaign, per-guest isolation, and the graphics pixel frontier (offline-first with the RISC-V GSP-RM debugger where possible), then the near-native performance program. The unsolved future classes — multi-tenant fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction in the canonical document. 3D graphics reuses this pipeline and now executes across three shapes, including the direct GL client's own GR/CE channels (Graphics-Execution.md); it adds engine breadth (video/image), more channels and synchronisation volume, and a much larger method vocabulary, but not a new plane.


Canonical document: docs/03-architecture/13-vexecution-plane.md. See also: full system picture, physical backend guest wiring, the execution frontier.