Difference between revisions of "The Execution Frontier"
m (2026-11-17 status refresh: real NVIDIA GPU-GL draw admitted and replayed (CALL(0) 0x3800), CALL(0) executes without wedging; per-draw MME macro body is the remaining gate; software-GL video remains non-vGPU) |
m (2026-11-20 late status refresh: corrected beef-lane diagnosis (PBDMA fetch proven; host-side forensic spin/no-write retire was the floor); r73 beef election + r75 shadow-VTG; v1.0.0-rc1 retracted; latest acceptance red on known flake/race) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 5: | Line 5: | ||
<blockquote>'''What this page is.''' The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A '''parallel graphics frontier''' (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — [[Graphics Execution|<code>Graphics-Execution.md</code>]], [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]]. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>]; the docs tree wins on disagreement.</blockquote> | <blockquote>'''What this page is.''' The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A '''parallel graphics frontier''' (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — [[Graphics Execution|<code>Graphics-Execution.md</code>]], [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]]. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>]; the docs tree wins on disagreement.</blockquote> | ||
Status: 2026-11- | Status: 2026-11-20, tree <code>master @ 3f34d58a</code>. Tags: <code>[live]</code> observed; <code>[wip]</code> partial; <code>[wall]</code> current blocker; theories are labelled as such. | ||
== 1. The chain that should fire == | == 1. The chain that should fire == | ||
| Line 81: | Line 81: | ||
A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (<code>victory_mail</code>) prints <code>VADD n=32768 bad=0</code> + <code>STATUS OK</code>, the matrix-multiply client (<code>victory_matmul</code>) prints <code>MATMUL n=256 bad=0</code> + <code>STATUS OK</code>, and a transformer kernel prints <code>dh=256 seq=64 bad=0</code>, <code>FFN4 layers=4 bad=0</code> + <code>STATUS OK</code>. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve. | A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (<code>victory_mail</code>) prints <code>VADD n=32768 bad=0</code> + <code>STATUS OK</code>, the matrix-multiply client (<code>victory_matmul</code>) prints <code>MATMUL n=256 bad=0</code> + <code>STATUS OK</code>, and a transformer kernel prints <code>dh=256 seq=64 bad=0</code>, <code>FFN4 layers=4 bad=0</code> + <code>STATUS OK</code>. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve. | ||
The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it: | On 2026-11-19 the acceptance battery gate '''closed''' (round 59): two consecutive clean-boot <code>guest-accept --cuda</code> runs printed <code>MAIL</code> correct, <code>VADD n=32768 bad=0</code> and <code>STATUS OK</code> (r59d/e), and the live <code>t/03</code> lane is green (<code>artifacts/acceptance/20261119T225057Z</code>: <code># cuda: 4 assertions, 0 failed</code>, <code>GUEST_ACCEPT_RESULT pass</code>; <code>EXPECT_CUDA_BATTERY=closed</code>). The gate was re-proven in r63 (two consecutive clean-boot t/03 greens) and in later rounds (r67d, r68b-t03). The latest full acceptance run (2026-11-20 19:49Z) is red on the known UVM/GR-wait lottery (<code>victory_mail rc=1</code>, 205 s), so the gate is not yet every-boot green. The wall was crossed by a sequence of bounded root causes, not one change: the CE rings resolved from content (<code>36cbfe66</code>), the channel-VAS tracker wrap (<code>47c849a0</code>), GR ring pairing (<code>3be286ac</code>), the CE pool layout (<code>d7d2164e</code>), the GR ctx-init semaphores (<code>e462ab74</code>), CE pool ordinal pairing (<code>5369dbf8</code>), the client's payload-2 CE push (<code>443dc9c8</code>), the GR ctx-init payload 2 (<code>eb386166</code>), dropping the host-path SW methods from the init preamble after the <code>SET_VALID_SPAN_OVERFLOW_AREA</code> trio was found to reset FECS (<code>41c97e61</code>), the inline re-rooter program-page fix (<code>951790a3</code>), the pa=0 fill/relocate fix (<code>f384a1dc</code>), the compute-lane inline-retarget gate (<code>335c05b0</code>), and a launch-lottery retry plus a CE-init UBSAN fix (<code>8566a61f</code>). The TSG engine-inheritance checkpoint (<code>5b23a983</code>) was falsified by two clean boots and reverted (<code>b26452fb</code>). | ||
The leading edge has moved from the launch chain itself — it fires and its battery gate is closed — to breadth and hardening around it: | |||
* '''The staged-transfer class.''' Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven (a release applied before the op it owes, a ledger-vs-writer page divergence, and a readback racing a writeback). | * '''The staged-transfer class.''' Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven (a release applied before the op it owes, a ledger-vs-writer page divergence, and a readback racing a writeback). | ||
| Line 89: | Line 91: | ||
=== A parallel frontier: 3D graphics === | === A parallel frontier: 3D graphics === | ||
Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (<code>/dev/dri</code>), X settles, and the stock NVIDIA GLX path | Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (<code>/dev/dri</code>), X settles, and the stock NVIDIA GLX path brings up on a good boot (<code>glxinfo -B</code> OpenGL 4.6.0 direct; the current direct-GL stack measures 0.255 FPS in <code>glxgears</code>, r64-r72). The X DDX waits are fixed and its 3D replay executes; the '''first non-black GPU-rendered output''' landed: a pbuffer probe returns <code>pixel=ff0000ff</code>, the off-screen path produced 500/500 non-black 640x480 frames per run and captured videos (45.4 s on 2026-11-11, a 50 s capture on 2026-11-12), and GZDoom reaches <code>V_Init: allocate screen</code> with its GL 4.6.0 context when the shared nvClip <code>modeNum</code> is even. The X side of the path was hardened in the 11-12/11-13 wave (DDX idle-DMA/ScreenInit, USERD-latch, re-probe, kernel-mode spin and flip-event walls closed; X green for more than 10 minutes), captured GL sections now execute on the engine, and the direct-GL submission is fully decoded: no client push carries a method-grammar draw header — the work is macro-driven (data-block blobs plus small state pushes invoking MME macros) — and the recycled pushes are now recovered, so the client's draw-state push opens a <code>C597</code> section, captures, and reaches the engine with the golden MME overlay. The MME chain then advanced from the multi-part gate to '''draw-macro execution''': multi-frame keep-windows land, the part-launch poison is eliminated (the client's own <code>SET_REPORT_SEMAPHORE</code>/<code>SET_NOTIFY</code> writes to remapped client VAs were the real fault; they are redirected to a mapped scratch slot), the 2799-word pack completes (2799/2799), the first <code>MME-PACK-REARM</code> fires, the fixed-CB region is aliased, and <code>DRAW_IAB</code> (<code>mac=0x22</code>) runs with the SPH-BIND prelude intact (84 words) and the DS stage VAs populated. The '''DS stage-0/1 image is now authored and live''': a VTG SPH at <code>family+0x900</code> (<code>SPH-AUTHOR</code>, first word <code>020a0481</code>) is what both vertex slots fetch (<code>DS-IMAGE st0=st1=020a0481 code=ff007321</code>), the window's own later slot-1 bind is rewritten to it, and the DS value fell <code>0x86040626 -> 0x84000420</code> (reproduced on two boots). The release path is now host-deterministic when the pack completes: the saved-window release carrier snapshots the held window and re-submits it with the calls released once the pack is resident (c772 end to end: save 240/68 -> fire -> synthetic <code>GR LAUNCH 0x6d6d6500</code> -> <code>MME-PACK-REARM</code> -> DS <code>0x84000420</code>), and the held-CALL census plus the SA[34] golden fill prove the <code>DRAW_IAB</code> call and its target are present (<code>MME-CALL-TARGET slot=34 method=0x3910 client=0x265 golden=0x265 used=0x265</code>). What is missing is '''window/screen pixels through the NVIDIA path'''. The draw macro executes end to end (c788 ran <code>DRAW_IAB</code> (<code>mac=0x22</code>) on a released held window after the <code>MME-DRAW-MOVE</code> reorder; TCS-off, <code>45dbcdcd</code>, clears the DS latch; a synthetic full-RT viewport, <code>dab4a02c</code>, and a chained release, <code>3481e62d</code>, complete the replay; the body decodes with the public Mesa MME toolchain as 8 draws of 3 vertices), and the '''11-16 wave proved the host-side chain deterministic end to end''': the real frame pushes are admitted (persisted colour-target bind, draw markers inside counted runs, a 14.8-thousand-word window replayed as 40 chunks, a 128-slot candidate ring), the beef draw lane latches a '''clean DS''' (<code>DS-FAST2 ... ds_after=80000000</code>, c963) at the live-derived GS attribute map, capture-less boots get a '''golden 2799-word MME pack''', and <code>MME-CALL-HOLD -> MME-CARRY-SAVE -> MME-CARRY-FIRE -> MME-PACK-REARM -> MME-SA-REARM</code> fires with the prelude latch kept across the channel recycle. The '''late-11-16 wave crossed the guest submission gap''': an offscreen SDL EGL/GBM shim gives GZDoom a real NVIDIA GL context without Xorg/DDX (<code>GL_RENDERER: NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2</code>, <code>GL_VERSION: 4.6.0 NVIDIA 570.144</code>; no llvmpipe), and the real draw window surfaced — 12068 B at push <code>0x402f9d8</code>, an op=4 IMMD <code>CALL(0)</code> to method <code>0x3800</code> (<code>imm=0xd48</code>) plus slots 44/57/58 (<code>0x3960/0x39c8/0x39d0</code>), with no <code>0x3910/0x3914</code> and no colour-target bind of its own. Recognition covers the real marker, admission happens on the marker even without a bind, the CALL-bearing chunk is forced first and every chunk replays verbatim. <code>CALL(0)</code> no longer wedges: the immediate is resolved as an IR address through client SA[0] (then golden) and a not-resident call is elided to a NOP; c1026/c1027 entered the captured macro without trapping and c1027 produced the first DS with real stage data (<code>DS-FAST2 ds_after=83044206 stage=015e0900,015e0a00,015c0400</code>). The '''GL pixel gate closed on 2026-11-18''' (<code>b5a68b9a</code>) without that per-draw macro body: the first clear/CALL(0)-bearing window is archived and replayed, the engine's clear stores red on the guest surface, and the client's CE readback follows the copy's source VA with a generation-pinned snapshot and a fence delivered with the copy (the measured ~11 s wait fell to 2.3-7.2 ms). Two consecutive boots read <code>GL_FRAME nonblack=291840 px0=(255,0,0)</code>, live <code>t/04</code> was green on that proof, and <code>EXPECT_GL_PIXELS=closed</code> remains set (re-confirmed by the r62b two-boot proof; the latest full run is red on the readback pin/copy race); the proof frame is <code>r23j-artifact/</code>. What remains is the '''X/GLX frame rate''' (0.255 FPS; the PBDMA fetch and release are proven, the draw still stores 0 pages and faults <code>Illegal Instruction Encoding</code>, and X/GLX bring-up is a lottery). A GZDoom video was separately delivered through the guest's '''software GL''' path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim. The guest Vulkan pipeline's inline-to-memory upload arm is its gap. The full picture lives on [[Graphics Execution|<code>Graphics-Execution.md</code>]]. | ||
No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned. | No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned. | ||
| Line 140: | Line 142: | ||
== 5. The clear issues (ranked) == | == 5. The clear issues (ranked) == | ||
'''Current wall (2026-11-20) — the X/GLX speed gate.''' The compute battery gate is closed; the open gate is graphics. <code>glxgears</code> measures 0.255 FPS (r64-r72) on the direct-GL stack. r72b corrected the earlier r71 diagnosis: the PBDMA fetches the direct-GL client-ring launch (<code>h=0xbeef0100</code>, <code>push=0x400108c</code>, 59580 B) in ~1 ms and the engine writes the channel release; the floor was host-side — the debug-only forensic MME spin (2.8 s) plus the host no-write retirement wait (~2.45 s when the window stores 0 of 1024 tracked pages). r73's beef read-only election cuts the retire wait (1337 -> 12 ms, post 1118 -> 52 ms), and r75 mirrors the VTG program image into the win0 shadow (<code>3f34d58a</code>); the remaining blockers are the draw's zero stores plus the engine <code>Illegal Instruction Encoding</code> fault. The X/GLX bring-up also remains a lottery: a hung GLX client blocks in <code>poll()</code> on an unsignalled dma-fence <code>sync_file</code>, and a DDX channel-idle timeout leads to <code>Failed to initialize DMA</code> and Xorg exit. | |||
'''Issue A — the staged-transfer class.''' Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven and named. | '''Issue A — the staged-transfer class.''' Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven and named. | ||
| Line 147: | Line 151: | ||
'''Issue C — per-guest execution isolation.''' Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress. | '''Issue C — per-guest execution isolation.''' Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress. | ||
'''Issue D — the graphics | '''Issue D — the graphics speed frontier.''' The guest GL readback pixel gate closed on 2026-11-18, and the host-side draw chain is proven; the current wall is the '''X/GLX frame rate''': <code>glxgears</code> measures 0.255 FPS (r64-r72). r72b corrected the r71 diagnosis — the PBDMA fetches the direct-GL client-ring launch (<code>h=0xbeef0100</code>, <code>push=0x400108c</code>, 59580 B) in ~1 ms and the engine writes the channel release; the floor was the debug-only forensic MME spin (2.8 s) plus the host no-write retirement wait (~2.45 s when the window stores 0 of 1024 pages). r73's beef read-only election cuts the retire wait (1337 -> 12 ms) and r75 mirrors the VTG program image into the win0 shadow (<code>3f34d58a</code>); the draw still stores 0 pages and faults <code>Illegal Instruction Encoding</code>. A parallel lottery remains in X/GLX bring-up: a hung GLX client blocks in <code>poll()</code> on an unsignalled dma-fence <code>sync_file</code>, and a DDX channel-idle timeout kills DMA init and Xorg. A GZDoom video was separately delivered through the guest's '''software GL''' path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim ([[Graphics Execution|<code>Graphics-Execution.md</code>]]). | ||
== 6. Lines of investigation (live) == | == 6. Lines of investigation (live) == | ||
'''Current work unit (2026-11-20):''' make the beef draw write pixels: the PBDMA fetch and the channel release are proven (r72b), the zero-write retire wait is bounded (r73), and the program shadow now carries the VTG SASS (r75); the remaining blockers are the draw's zero stores plus the <code>Illegal Instruction Encoding</code> fault, and the X/GLX bring-up lottery. The draw-synth/RT0 admission, the RB-SNAP replay-on-arm ordering and the CUDA <code>victory_mail</code> UVM/GR-wait flake are the parallel closure campaigns. <code>glxgears</code> must exceed 30 FPS on two consecutive clean boots before <code>EXPECT_X_DDX</code> flips. Keep the closed CUDA and GL gates green on the same builds. | |||
# '''Repair the staged-transfer class''' — the green clients stay the regression battery and must remain green throughout. | # '''Repair the staged-transfer class''' — the green clients stay the regression battery and must remain green throughout. | ||
| Line 155: | Line 161: | ||
# '''Per-guest execution isolation''' — build from the device-level coexistence to isolation for simultaneous CUDA guests. | # '''Per-guest execution isolation''' — build from the device-level coexistence to isolation for simultaneous CUDA guests. | ||
# '''Hold the acceptance battery''' — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (<code>bad=0</code> + <code>STATUS OK</code>). | # '''Hold the acceptance battery''' — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (<code>bad=0</code> + <code>STATUS OK</code>). | ||
# '''The graphics | # '''The graphics speed frontier''' — the guest GL readback pixel gate is closed (2026-11-18, re-confirmed r62b/r63c), and the compute battery gate is closed (2026-11-19); the latest full acceptance run is red on the known flake/race classes, so every-boot green is still the bar. Next: make the beef draw store pixels (the PBDMA fetch and release are proven; the zero-write retire wait is bounded; the program shadow carries the VTG SASS), reduce the X/GLX bring-up lottery, and keep both closed gates green; the lanes, the walls and the milestone ladder are on [[Graphics Execution|<code>Graphics-Execution.md</code>]]. | ||
== 7. Theories, resolved == | == 7. Theories, resolved == | ||
Latest revision as of 20:42, 20 November 2026
An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.
This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.
Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.
The Execution Frontier
What this page is. The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A parallel graphics frontier (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer —
Graphics-Execution.md,OpenGL.md,Vulkan.md. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from../docs/04-implementation/03-execution-frontier.md; the docs tree wins on disagreement.
Status: 2026-11-20, tree master @ 3f34d58a. Tags: [live] observed; [wip] partial; [wall] current blocker; theories are labelled as such.
1. The chain that should fire
A guest CUDA launch becomes a method stream in a push buffer, referenced by a GPFIFO ring entry, announced by a doorbell. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware front-end fetches and executes the methods, the launch descriptor (QMD) is materialised and read by the GR engine, the grid is dispatched to the SMs, and completion is published as semaphores the guest waits on.
guest: builds launch + method stream
GPFIFO entry + doorbell
-> host: doorbell trap, decode, rebuild forward stream,
prewrite QMD, ring/USERD, doorbell
-> front-end: fetch GPFIFO entry (ring cursor advances)
-> front-end: execute methods (object bind, cache ops, report sem)
-> front-end: install inline QMD into the descriptor buffer
-> GR engine: read QMD through the channel's page tables
-> context switch loads the channel context (FECS/ucode)
-> GR: dispatch grid to the SMs
-> engine authors release semaphore + channel semaphore
-> host polls release, relays to guest; CUDA wakes
| # | Stage | State |
|---|---|---|
| 1 | Guest builds launch + method stream | [live] observed (stock driver, unmodified)
|
| 2 | GPFIFO entry + doorbell | [live] observed
|
| 3 | Host decode + forward stream + QMD prewrite + ring/USERD | [live] observed
|
| 4 | Front-end fetches the ring entry | [live] observed (ring cursor advances)
|
| 5 | Front-end executes methods | [live] observed (object bind, cache ops, report semaphore)
|
| 6 | Inline-QMD install (descriptor materialised) | [live] observed
|
| 7 | Engine reads the QMD | [live] observed
|
| 8 | Context switch loads the channel context | [live] context validity at dispatch is ruled in
|
| 9 | GR dispatches the grid to the SMs | [live] observed (the guest's results verify)
|
| 10 | Engine authors release + channel semaphore | [live] engine-authored releases relay to the guest
|
| 11 | Host relays completion; guest wakes | [live] the guest's synchronisation waits resolve
|
One-line summary: the whole chain is live; the leading edge is now breadth and hardening — the large-buffer staged-transfer class, reproducibility across boots, and per-guest execution isolation.
2. Where the leading edge is
A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (victory_mail) prints VADD n=32768 bad=0 + STATUS OK, the matrix-multiply client (victory_matmul) prints MATMUL n=256 bad=0 + STATUS OK, and a transformer kernel prints dh=256 seq=64 bad=0, FFN4 layers=4 bad=0 + STATUS OK. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve.
On 2026-11-19 the acceptance battery gate closed (round 59): two consecutive clean-boot guest-accept --cuda runs printed MAIL correct, VADD n=32768 bad=0 and STATUS OK (r59d/e), and the live t/03 lane is green (artifacts/acceptance/20261119T225057Z: # cuda: 4 assertions, 0 failed, GUEST_ACCEPT_RESULT pass; EXPECT_CUDA_BATTERY=closed). The gate was re-proven in r63 (two consecutive clean-boot t/03 greens) and in later rounds (r67d, r68b-t03). The latest full acceptance run (2026-11-20 19:49Z) is red on the known UVM/GR-wait lottery (victory_mail rc=1, 205 s), so the gate is not yet every-boot green. The wall was crossed by a sequence of bounded root causes, not one change: the CE rings resolved from content (36cbfe66), the channel-VAS tracker wrap (47c849a0), GR ring pairing (3be286ac), the CE pool layout (d7d2164e), the GR ctx-init semaphores (e462ab74), CE pool ordinal pairing (5369dbf8), the client's payload-2 CE push (443dc9c8), the GR ctx-init payload 2 (eb386166), dropping the host-path SW methods from the init preamble after the SET_VALID_SPAN_OVERFLOW_AREA trio was found to reset FECS (41c97e61), the inline re-rooter program-page fix (951790a3), the pa=0 fill/relocate fix (f384a1dc), the compute-lane inline-retarget gate (335c05b0), and a launch-lottery retry plus a CE-init UBSAN fix (8566a61f). The TSG engine-inheritance checkpoint (5b23a983) was falsified by two clean boots and reverted (b26452fb).
The leading edge has moved from the launch chain itself — it fires and its battery gate is closed — to breadth and hardening around it:
- The staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven (a release applied before the op it owes, a ledger-vs-writer page divergence, and a readback racing a writeback).
- Reproducibility across boots. Two defects landed and were fixed: the boot-poisoning defect (under the driver's status-queue mode the emulation read the wrong queue cursor, so under load the reply ring could desynchronise and the first CUDA client would fail) and a ring-block ownership defect that could stall the first client after a reboot. Green clients reproduce across fresh boots; a formal every-boot campaign remains.
- Per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level (two guests demonstrated); per-guest execution isolation for multiple simultaneous CUDA guests is in progress.
A parallel frontier: 3D graphics
Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The display and GL stacks now run end to end in the guest: the emulated display engine serves the modeset push buffers, nvidia-modeset/NvKms registers (/dev/dri), X settles, and the stock NVIDIA GLX path brings up on a good boot (glxinfo -B OpenGL 4.6.0 direct; the current direct-GL stack measures 0.255 FPS in glxgears, r64-r72). The X DDX waits are fixed and its 3D replay executes; the first non-black GPU-rendered output landed: a pbuffer probe returns pixel=ff0000ff, the off-screen path produced 500/500 non-black 640x480 frames per run and captured videos (45.4 s on 2026-11-11, a 50 s capture on 2026-11-12), and GZDoom reaches V_Init: allocate screen with its GL 4.6.0 context when the shared nvClip modeNum is even. The X side of the path was hardened in the 11-12/11-13 wave (DDX idle-DMA/ScreenInit, USERD-latch, re-probe, kernel-mode spin and flip-event walls closed; X green for more than 10 minutes), captured GL sections now execute on the engine, and the direct-GL submission is fully decoded: no client push carries a method-grammar draw header — the work is macro-driven (data-block blobs plus small state pushes invoking MME macros) — and the recycled pushes are now recovered, so the client's draw-state push opens a C597 section, captures, and reaches the engine with the golden MME overlay. The MME chain then advanced from the multi-part gate to draw-macro execution: multi-frame keep-windows land, the part-launch poison is eliminated (the client's own SET_REPORT_SEMAPHORE/SET_NOTIFY writes to remapped client VAs were the real fault; they are redirected to a mapped scratch slot), the 2799-word pack completes (2799/2799), the first MME-PACK-REARM fires, the fixed-CB region is aliased, and DRAW_IAB (mac=0x22) runs with the SPH-BIND prelude intact (84 words) and the DS stage VAs populated. The DS stage-0/1 image is now authored and live: a VTG SPH at family+0x900 (SPH-AUTHOR, first word 020a0481) is what both vertex slots fetch (DS-IMAGE st0=st1=020a0481 code=ff007321), the window's own later slot-1 bind is rewritten to it, and the DS value fell 0x86040626 -> 0x84000420 (reproduced on two boots). The release path is now host-deterministic when the pack completes: the saved-window release carrier snapshots the held window and re-submits it with the calls released once the pack is resident (c772 end to end: save 240/68 -> fire -> synthetic GR LAUNCH 0x6d6d6500 -> MME-PACK-REARM -> DS 0x84000420), and the held-CALL census plus the SA[34] golden fill prove the DRAW_IAB call and its target are present (MME-CALL-TARGET slot=34 method=0x3910 client=0x265 golden=0x265 used=0x265). What is missing is window/screen pixels through the NVIDIA path. The draw macro executes end to end (c788 ran DRAW_IAB (mac=0x22) on a released held window after the MME-DRAW-MOVE reorder; TCS-off, 45dbcdcd, clears the DS latch; a synthetic full-RT viewport, dab4a02c, and a chained release, 3481e62d, complete the replay; the body decodes with the public Mesa MME toolchain as 8 draws of 3 vertices), and the 11-16 wave proved the host-side chain deterministic end to end: the real frame pushes are admitted (persisted colour-target bind, draw markers inside counted runs, a 14.8-thousand-word window replayed as 40 chunks, a 128-slot candidate ring), the beef draw lane latches a clean DS (DS-FAST2 ... ds_after=80000000, c963) at the live-derived GS attribute map, capture-less boots get a golden 2799-word MME pack, and MME-CALL-HOLD -> MME-CARRY-SAVE -> MME-CARRY-FIRE -> MME-PACK-REARM -> MME-SA-REARM fires with the prelude latch kept across the channel recycle. The late-11-16 wave crossed the guest submission gap: an offscreen SDL EGL/GBM shim gives GZDoom a real NVIDIA GL context without Xorg/DDX (GL_RENDERER: NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2, GL_VERSION: 4.6.0 NVIDIA 570.144; no llvmpipe), and the real draw window surfaced — 12068 B at push 0x402f9d8, an op=4 IMMD CALL(0) to method 0x3800 (imm=0xd48) plus slots 44/57/58 (0x3960/0x39c8/0x39d0), with no 0x3910/0x3914 and no colour-target bind of its own. Recognition covers the real marker, admission happens on the marker even without a bind, the CALL-bearing chunk is forced first and every chunk replays verbatim. CALL(0) no longer wedges: the immediate is resolved as an IR address through client SA[0] (then golden) and a not-resident call is elided to a NOP; c1026/c1027 entered the captured macro without trapping and c1027 produced the first DS with real stage data (DS-FAST2 ds_after=83044206 stage=015e0900,015e0a00,015c0400). The GL pixel gate closed on 2026-11-18 (b5a68b9a) without that per-draw macro body: the first clear/CALL(0)-bearing window is archived and replayed, the engine's clear stores red on the guest surface, and the client's CE readback follows the copy's source VA with a generation-pinned snapshot and a fence delivered with the copy (the measured ~11 s wait fell to 2.3-7.2 ms). Two consecutive boots read GL_FRAME nonblack=291840 px0=(255,0,0), live t/04 was green on that proof, and EXPECT_GL_PIXELS=closed remains set (re-confirmed by the r62b two-boot proof; the latest full run is red on the readback pin/copy race); the proof frame is r23j-artifact/. What remains is the X/GLX frame rate (0.255 FPS; the PBDMA fetch and release are proven, the draw still stores 0 pages and faults Illegal Instruction Encoding, and X/GLX bring-up is a lottery). A GZDoom video was separately delivered through the guest's software GL path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim. The guest Vulkan pipeline's inline-to-memory upload arm is its gap. The full picture lives on Graphics-Execution.md.
No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned.
3. How we know (observables and their limits)
| Observable | What it proves | What it does not prove |
|---|---|---|
Client verification output (bad=0 + STATUS OK)
|
the guest executed its kernel and verified its results | by itself, which engine produced the results |
| Engine-authored release relayed to the guest | the engine executed the launch and the relay works | nothing about other launches |
| The guest's synchronisation wait resolving | completion reached the guest through the relay | nothing about other launches |
| Ring entry consumed; ring cursor advances | the front-end is alive and fetching | not engine execution — front-end signals are necessary but not sufficient |
| Reproduction across fresh boots | the path is not a lucky boot | no "every boot" claim yet while re-measurement is in progress |
| No GR fault recorded | the engine did not raise this error | silence is not health: a skipped launch is silent by construction |
Consequence: acceptance is verified guest results plus the engine-authored release; a green run counts as progress only when it reproduces across fresh boots.
4. What is exonerated (do not relitigate)
- Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set).
- The descriptor destination window mapping — installed in the engine's table view at submit.
- Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0.
- The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta.
- Parenting/TSG for the fetch — both variants fetch identically.
- The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is separate.
- The descriptor-gate line of investigation (the old T1–T4 theories) — the inline-descriptor install, the engine read, the dispatch, the engine-authored release and the guest wake-up are all observed; closed by observation.
- The old boot-poisoning defect — root-caused to the status-queue consumption cursor and fixed.
5. The clear issues (ranked)
Current wall (2026-11-20) — the X/GLX speed gate. The compute battery gate is closed; the open gate is graphics. glxgears measures 0.255 FPS (r64-r72) on the direct-GL stack. r72b corrected the earlier r71 diagnosis: the PBDMA fetches the direct-GL client-ring launch (h=0xbeef0100, push=0x400108c, 59580 B) in ~1 ms and the engine writes the channel release; the floor was host-side — the debug-only forensic MME spin (2.8 s) plus the host no-write retirement wait (~2.45 s when the window stores 0 of 1024 tracked pages). r73's beef read-only election cuts the retire wait (1337 -> 12 ms, post 1118 -> 52 ms), and r75 mirrors the VTG program image into the win0 shadow (3f34d58a); the remaining blockers are the draw's zero stores plus the engine Illegal Instruction Encoding fault. The X/GLX bring-up also remains a lottery: a hung GLX client blocks in poll() on an unsignalled dma-fence sync_file, and a DDX channel-idle timeout leads to Failed to initialize DMA and Xorg exit.
Issue A — the staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven and named.
Issue B — every-boot reproducibility (acceptance campaign). The boot-poisoning defect and the post-reboot ring-block ownership defect were root-caused and fixed, and green clients reproduce on fresh boots; the formal every-boot acceptance campaign is the remaining work.
Issue C — per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress.
Issue D — the graphics speed frontier. The guest GL readback pixel gate closed on 2026-11-18, and the host-side draw chain is proven; the current wall is the X/GLX frame rate: glxgears measures 0.255 FPS (r64-r72). r72b corrected the r71 diagnosis — the PBDMA fetches the direct-GL client-ring launch (h=0xbeef0100, push=0x400108c, 59580 B) in ~1 ms and the engine writes the channel release; the floor was the debug-only forensic MME spin (2.8 s) plus the host no-write retirement wait (~2.45 s when the window stores 0 of 1024 pages). r73's beef read-only election cuts the retire wait (1337 -> 12 ms) and r75 mirrors the VTG program image into the win0 shadow (3f34d58a); the draw still stores 0 pages and faults Illegal Instruction Encoding. A parallel lottery remains in X/GLX bring-up: a hung GLX client blocks in poll() on an unsignalled dma-fence sync_file, and a DDX channel-idle timeout kills DMA init and Xorg. A GZDoom video was separately delivered through the guest's software GL path (Mesa llvmpipe) as a guaranteed-quality deliverable; it is not a vGPU rendering claim (Graphics-Execution.md).
6. Lines of investigation (live)
Current work unit (2026-11-20): make the beef draw write pixels: the PBDMA fetch and the channel release are proven (r72b), the zero-write retire wait is bounded (r73), and the program shadow now carries the VTG SASS (r75); the remaining blockers are the draw's zero stores plus the Illegal Instruction Encoding fault, and the X/GLX bring-up lottery. The draw-synth/RT0 admission, the RB-SNAP replay-on-arm ordering and the CUDA victory_mail UVM/GR-wait flake are the parallel closure campaigns. glxgears must exceed 30 FPS on two consecutive clean boots before EXPECT_X_DDX flips. Keep the closed CUDA and GL gates green on the same builds.
- Repair the staged-transfer class — the green clients stay the regression battery and must remain green throughout.
- Complete the every-boot acceptance campaign — demonstrate the fixed boot path across a defined run of fresh boots before claiming boot-to-boot reproducibility.
- Per-guest execution isolation — build from the device-level coexistence to isolation for simultaneous CUDA guests.
- Hold the acceptance battery — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (
bad=0+STATUS OK). - The graphics speed frontier — the guest GL readback pixel gate is closed (2026-11-18, re-confirmed r62b/r63c), and the compute battery gate is closed (2026-11-19); the latest full acceptance run is red on the known flake/race classes, so every-boot green is still the bar. Next: make the beef draw store pixels (the PBDMA fetch and release are proven; the zero-write retire wait is bounded; the program shadow carries the VTG SASS), reduce the X/GLX bring-up lottery, and keep both closed gates green; the lanes, the walls and the milestone ladder are on
Graphics-Execution.md.
7. Theories, resolved
The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is engineering (staged transfers, an every-boot acceptance campaign, per-guest isolation, the graphics pixel frontier), each with an acceptance signal named before the work.
8. Claim discipline
- Engine-authored release is the acceptance signal for execution — now observed end to end; the guest's own verification (
bad=0+STATUS OK) is the guest-side acceptance signal. - Front-end signals are necessary but not sufficient. Ring cursor and front-end method execution can be front-end-authored; they never substitute for the engine-authored release.
- Silence is not health. No fault recorded does not mean no fault occurred.
- Emulator verdicts are instruction/memory-level. They bound behaviour, never timing or device environment.
- A green run counts only when it reproduces across fresh boots.
- A wall is closed only by an acceptance signal, named before the experiment.
9. Related pages
Project-Status.md— the project-level ladder.CUDA-Userspace-Driver.md— the guest-side client path.Physical-GPU-Backend.md— the backend the work is forwarded to.Graphics-Execution.md— the parallel graphics frontier (OpenGL/Vulkan), sharing this machinery.Execution-Plane.md— the execution layer's design.Introduction-to-GMMU.md— page tables, QMDs and the engines.../docs/04-implementation/03-execution-frontier.md— the canonical, continuously-updated version with the full evidence tables.