Difference between revisions of "The Execution Frontier"

From Open-IOV
Jump to navigation Jump to search
m (2026-11-10 afternoon refresh: first real GPU readback (pbuffer red pixel); direct-GL draw capture and window/screen present are the walls)
m (2026-11-11 refresh: first non-black GPU frames + 45.4 s video; direct-GL GR ring resolves; CE-ring/window walls)
Line 5: Line 5:
<blockquote>'''What this page is.''' The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A '''parallel graphics frontier''' (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — [[Graphics Execution|<code>Graphics-Execution.md</code>]], [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]]. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>]; the docs tree wins on disagreement.</blockquote>
<blockquote>'''What this page is.''' The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A '''parallel graphics frontier''' (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — [[Graphics Execution|<code>Graphics-Execution.md</code>]], [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]]. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>]; the docs tree wins on disagreement.</blockquote>


Status: 2026-11-09, tree <code>master @ df6e71bb</code>. Tags: <code>[live]</code> observed; <code>[wip]</code> partial; <code>[wall]</code> current blocker; theories are labelled as such.
Status: 2026-11-11, tree <code>master @ 4acac093</code>. Tags: <code>[live]</code> observed; <code>[wip]</code> partial; <code>[wall]</code> current blocker; theories are labelled as such.


== 1. The chain that should fire ==
== 1. The chain that should fire ==
Line 89: Line 89:
=== A parallel frontier: 3D graphics ===
=== A parallel frontier: 3D graphics ===


Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (<code>/dev/dri</code>), X settles, <code>glxinfo</code> reports OpenGL 4.6.0 with direct rendering, and the X DDX waits are fixed (class-aware display arm, <code>0x8000 -&gt; 0</code> notification completion, decoded GL release applied); the direct GL client's GR and CE channels submit and retire, and the X DDX kick channel is decoded. GZDoom reached <code>MAP01</code> once the DDX push-buffer loop was unblocked, but on Mesa llvmpipe — not the emulated vGPU. What is missing is '''screen pixels through the NVIDIA path''': the 2D path is green, the DDX 3D replay executes, and the first real GPU readback landed (pbuffer <code>pixel=ff0000ff</code>), but the direct-GL client still emits only setup/notify kicks (<code>ops_runs=0</code>) and a synchronous window draw wedges X. The direct-GL draw capture and the window/screen present are the named walls. The guest Vulkan pipeline's inline-to-memory upload arm is its gap. The full picture lives on [[Graphics Execution|<code>Graphics-Execution.md</code>]].
Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (<code>/dev/dri</code>), X settles, <code>glxinfo</code> reports OpenGL 4.6.0 with direct rendering, and the X DDX waits are fixed (class-aware display arm, <code>0x8000 -&gt; 0</code> notification completion, decoded GL release applied); the direct GL client's GR and CE channels submit and retire, and the X DDX kick channel is decoded. The '''first non-black GPU-rendered output''' landed: a pbuffer probe returns <code>pixel=ff0000ff</code>, the off-screen path produced 500/500 animated 640x480 frames, and a 45.4 s video was captured. The direct-GL '''GR ring <code>0xbeef0100</code> now resolves''' and its releases advance; GZDoom reaches <code>V_Init: allocate screen</code> with its GL 4.6.0 context whenever the shared nvClip <code>modeNum</code> is even. What is missing is '''window/ screen pixels through the NVIDIA path''': glxgears' window is black, the direct-GL '''CE ring <code>0xbeef0101</code> never resolves''' (the guest wedges ~90 s after <code>V_Init</code>), and X window creation wedges on some boots. The guest Vulkan pipeline's inline-to-memory upload arm is its gap. The full picture lives on [[Graphics Execution|<code>Graphics-Execution.md</code>]].


No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned.
No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned.
Line 147: Line 147:
'''Issue C — per-guest execution isolation.''' Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress.
'''Issue C — per-guest execution isolation.''' Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress.


'''Issue D — the graphics pixel frontier.''' The display/KMS stack and the GL client now run end to end (NvKms registers, X settles, GL 4.6, the DDX 3D replay executes, and the first real GPU readback returns a red pbuffer pixel); but the direct-GL client still emits only setup/notify kicks (<code>ops_runs=0</code>), a window draw wedges X, and no non-black screen frame has been produced. The direct-GL draw capture and the window/screen present are the named blockers ([[Graphics Execution|<code>Graphics-Execution.md</code>]]).
'''Issue D — the graphics pixel frontier.''' The display/KMS stack and the GL client now run end to end (NvKms registers, X settles, GL 4.6, the DDX 3D replay executes, the first non-black GPU-rendered frames and a 45.4 s video are captured off-screen, and the direct-GL GR ring resolves with advancing releases); but the direct-GL CE ring <code>0xbeef0101</code> does not resolve, a window draw wedges X and glxgears' window is black, and no non-black window/ screen frame has been produced. The CE ring, the window/screen present and X stability are the named blockers ([[Graphics Execution|<code>Graphics-Execution.md</code>]]).


== 6. Lines of investigation (live) ==
== 6. Lines of investigation (live) ==
Line 155: Line 155:
# '''Per-guest execution isolation''' — build from the device-level coexistence to isolation for simultaneous CUDA guests.
# '''Per-guest execution isolation''' — build from the device-level coexistence to isolation for simultaneous CUDA guests.
# '''Hold the acceptance battery''' — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (<code>bad=0</code> + <code>STATUS OK</code>).
# '''Hold the acceptance battery''' — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (<code>bad=0</code> + <code>STATUS OK</code>).
# '''The graphics pixel frontier''' — materialise the driver's program uploads and map the program VA so draws execute with real state; the lanes, the walls and the milestone ladder are on [[Graphics Execution|<code>Graphics-Execution.md</code>]].
# '''The graphics pixel frontier''' — resolve the direct-GL CE ring <code>0xbeef0101</code>, make the window/screen present land, and stabilise X across boots so the proven off-screen frames become window pixels; the lanes, the walls and the milestone ladder are on [[Graphics Execution|<code>Graphics-Execution.md</code>]].


== 7. Theories, resolved ==
== 7. Theories, resolved ==

Revision as of 15:48, 11 November 2026

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

The Execution Frontier

What this page is. The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A parallel graphics frontier (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — Graphics-Execution.md, OpenGL.md, Vulkan.md. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from ../docs/04-implementation/03-execution-frontier.md; the docs tree wins on disagreement.

Status: 2026-11-11, tree master @ 4acac093. Tags: [live] observed; [wip] partial; [wall] current blocker; theories are labelled as such.

1. The chain that should fire

A guest CUDA launch becomes a method stream in a push buffer, referenced by a GPFIFO ring entry, announced by a doorbell. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware front-end fetches and executes the methods, the launch descriptor (QMD) is materialised and read by the GR engine, the grid is dispatched to the SMs, and completion is published as semaphores the guest waits on.

guest: builds launch + method stream
       GPFIFO entry + doorbell
   -> host: doorbell trap, decode, rebuild forward stream,
            prewrite QMD, ring/USERD, doorbell
   -> front-end: fetch GPFIFO entry (ring cursor advances)
   -> front-end: execute methods (object bind, cache ops, report sem)
   -> front-end: install inline QMD into the descriptor buffer
   -> GR engine: read QMD through the channel's page tables
   -> context switch loads the channel context (FECS/ucode)
   -> GR: dispatch grid to the SMs
   -> engine authors release semaphore + channel semaphore
   -> host polls release, relays to guest; CUDA wakes
# Stage State
1 Guest builds launch + method stream [live] observed (stock driver, unmodified)
2 GPFIFO entry + doorbell [live] observed
3 Host decode + forward stream + QMD prewrite + ring/USERD [live] observed
4 Front-end fetches the ring entry [live] observed (ring cursor advances)
5 Front-end executes methods [live] observed (object bind, cache ops, report semaphore)
6 Inline-QMD install (descriptor materialised) [live] observed
7 Engine reads the QMD [live] observed
8 Context switch loads the channel context [live] context validity at dispatch is ruled in
9 GR dispatches the grid to the SMs [live] observed (the guest's results verify)
10 Engine authors release + channel semaphore [live] engine-authored releases relay to the guest
11 Host relays completion; guest wakes [live] the guest's synchronisation waits resolve

One-line summary: the whole chain is live; the leading edge is now breadth and hardening — the large-buffer staged-transfer class, reproducibility across boots, and per-guest execution isolation.

2. Where the leading edge is

A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (victory_mail) prints VADD n=32768 bad=0 + STATUS OK, the matrix-multiply client (victory_matmul) prints MATMUL n=256 bad=0 + STATUS OK, and a transformer kernel prints dh=256 seq=64 bad=0, FFN4 layers=4 bad=0 + STATUS OK. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve.

The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it:

  • The staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven (a release applied before the op it owes, a ledger-vs-writer page divergence, and a readback racing a writeback).
  • Reproducibility across boots. Two defects landed and were fixed: the boot-poisoning defect (under the driver's status-queue mode the emulation read the wrong queue cursor, so under load the reply ring could desynchronise and the first CUDA client would fail) and a ring-block ownership defect that could stall the first client after a reboot. Green clients reproduce across fresh boots; a formal every-boot campaign remains.
  • Per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level (two guests demonstrated); per-guest execution isolation for multiple simultaneous CUDA guests is in progress.

A parallel frontier: 3D graphics

Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (/dev/dri), X settles, glxinfo reports OpenGL 4.6.0 with direct rendering, and the X DDX waits are fixed (class-aware display arm, 0x8000 -> 0 notification completion, decoded GL release applied); the direct GL client's GR and CE channels submit and retire, and the X DDX kick channel is decoded. The first non-black GPU-rendered output landed: a pbuffer probe returns pixel=ff0000ff, the off-screen path produced 500/500 animated 640x480 frames, and a 45.4 s video was captured. The direct-GL GR ring 0xbeef0100 now resolves and its releases advance; GZDoom reaches V_Init: allocate screen with its GL 4.6.0 context whenever the shared nvClip modeNum is even. What is missing is window/ screen pixels through the NVIDIA path: glxgears' window is black, the direct-GL CE ring 0xbeef0101 never resolves (the guest wedges ~90 s after V_Init), and X window creation wedges on some boots. The guest Vulkan pipeline's inline-to-memory upload arm is its gap. The full picture lives on Graphics-Execution.md.

No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned.

3. How we know (observables and their limits)

Observable What it proves What it does not prove
Client verification output (bad=0 + STATUS OK) the guest executed its kernel and verified its results by itself, which engine produced the results
Engine-authored release relayed to the guest the engine executed the launch and the relay works nothing about other launches
The guest's synchronisation wait resolving completion reached the guest through the relay nothing about other launches
Ring entry consumed; ring cursor advances the front-end is alive and fetching not engine execution — front-end signals are necessary but not sufficient
Reproduction across fresh boots the path is not a lucky boot no "every boot" claim yet while re-measurement is in progress
No GR fault recorded the engine did not raise this error silence is not health: a skipped launch is silent by construction

Consequence: acceptance is verified guest results plus the engine-authored release; a green run counts as progress only when it reproduces across fresh boots.

4. What is exonerated (do not relitigate)

  • Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set).
  • The descriptor destination window mapping — installed in the engine's table view at submit.
  • Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0.
  • The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta.
  • Parenting/TSG for the fetch — both variants fetch identically.
  • The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is separate.
  • The descriptor-gate line of investigation (the old T1–T4 theories) — the inline-descriptor install, the engine read, the dispatch, the engine-authored release and the guest wake-up are all observed; closed by observation.
  • The old boot-poisoning defect — root-caused to the status-queue consumption cursor and fixed.

5. The clear issues (ranked)

Issue A — the staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven and named.

Issue B — every-boot reproducibility (acceptance campaign). The boot-poisoning defect and the post-reboot ring-block ownership defect were root-caused and fixed, and green clients reproduce on fresh boots; the formal every-boot acceptance campaign is the remaining work.

Issue C — per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress.

Issue D — the graphics pixel frontier. The display/KMS stack and the GL client now run end to end (NvKms registers, X settles, GL 4.6, the DDX 3D replay executes, the first non-black GPU-rendered frames and a 45.4 s video are captured off-screen, and the direct-GL GR ring resolves with advancing releases); but the direct-GL CE ring 0xbeef0101 does not resolve, a window draw wedges X and glxgears' window is black, and no non-black window/ screen frame has been produced. The CE ring, the window/screen present and X stability are the named blockers (Graphics-Execution.md).

6. Lines of investigation (live)

  1. Repair the staged-transfer class — the green clients stay the regression battery and must remain green throughout.
  2. Complete the every-boot acceptance campaign — demonstrate the fixed boot path across a defined run of fresh boots before claiming boot-to-boot reproducibility.
  3. Per-guest execution isolation — build from the device-level coexistence to isolation for simultaneous CUDA guests.
  4. Hold the acceptance battery — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (bad=0 + STATUS OK).
  5. The graphics pixel frontier — resolve the direct-GL CE ring 0xbeef0101, make the window/screen present land, and stabilise X across boots so the proven off-screen frames become window pixels; the lanes, the walls and the milestone ladder are on Graphics-Execution.md.

7. Theories, resolved

The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is engineering (staged transfers, an every-boot acceptance campaign, per-guest isolation, the graphics pixel frontier), each with an acceptance signal named before the work.

8. Claim discipline

  • Engine-authored release is the acceptance signal for execution — now observed end to end; the guest's own verification (bad=0 + STATUS OK) is the guest-side acceptance signal.
  • Front-end signals are necessary but not sufficient. Ring cursor and front-end method execution can be front-end-authored; they never substitute for the engine-authored release.
  • Silence is not health. No fault recorded does not mean no fault occurred.
  • Emulator verdicts are instruction/memory-level. They bound behaviour, never timing or device environment.
  • A green run counts only when it reproduces across fresh boots.
  • A wall is closed only by an acceptance signal, named before the experiment.

9. Related pages