The Execution Frontier

The Execution Frontier

What this page is. The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. A parallel graphics frontier (OpenGL/Vulkan execution) shares the same submission machinery at a different execution layer — Graphics-Execution.md, OpenGL.md, Vulkan.md. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from ../docs/04-implementation/03-execution-frontier.md; the docs tree wins on disagreement.

Status: 2026-10-30, tree master @ b4930168. Tags: [live] observed; [wip] partial; [wall] current blocker; theories are labelled as such.

1. The chain that should fire

A guest CUDA launch becomes a method stream in a push buffer, referenced by a GPFIFO ring entry, announced by a doorbell. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware front-end fetches and executes the methods, the launch descriptor (QMD) is materialised and read by the GR engine, the grid is dispatched to the SMs, and completion is published as semaphores the guest waits on.

guest: builds launch + method stream
       GPFIFO entry + doorbell
   -> host: doorbell trap, decode, rebuild forward stream,
            prewrite QMD, ring/USERD, doorbell
   -> front-end: fetch GPFIFO entry (ring cursor advances)
   -> front-end: execute methods (object bind, cache ops, report sem)
   -> front-end: install inline QMD into the descriptor buffer
   -> GR engine: read QMD through the channel's page tables
   -> context switch loads the channel context (FECS/ucode)
   -> GR: dispatch grid to the SMs
   -> engine authors release semaphore + channel semaphore
   -> host polls release, relays to guest; CUDA wakes

#	Stage	State
1	Guest builds launch + method stream	[live] observed (stock driver, unmodified)
2	GPFIFO entry + doorbell	[live] observed
3	Host decode + forward stream + QMD prewrite + ring/USERD	[live] observed
4	Front-end fetches the ring entry	[live] observed (ring cursor advances)
5	Front-end executes methods	[live] observed (object bind, cache ops, report semaphore)
6	Inline-QMD install (descriptor materialised)	[live] observed
7	Engine reads the QMD	[live] observed
8	Context switch loads the channel context	[live] context validity at dispatch is ruled in
9	GR dispatches the grid to the SMs	[live] observed (the guest's results verify)
10	Engine authors release + channel semaphore	[live] engine-authored releases relay to the guest
11	Host relays completion; guest wakes	[live] the guest's synchronisation waits resolve

One-line summary: the whole chain is live; the leading edge is now breadth and hardening — the large-buffer staged-transfer class, reproducibility across boots, and per-guest execution isolation.

2. Where the leading edge is

A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (victory_mail) prints VADD n=32768 bad=0 + STATUS OK, the matrix-multiply client (victory_matmul) prints MATMUL n=256 bad=0 + STATUS OK, and a transformer kernel prints dh=256 seq=64 bad=0, FFN4 layers=4 bad=0 + STATUS OK. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve.

The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it:

• The staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven (a release applied before the op it owes, a ledger-vs-writer page divergence, and a readback racing a writeback).
• Reproducibility across boots. Two defects landed and were fixed: the boot-poisoning defect (under the driver's status-queue mode the emulation read the wrong queue cursor, so under load the reply ring could desynchronise and the first CUDA client would fail) and a ring-block ownership defect that could stall the first client after a reboot. Green clients reproduce across fresh boots; a formal every-boot campaign remains.
• Per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level (two guests demonstrated); per-guest execution isolation for multiple simultaneous CUDA guests is in progress.

A parallel frontier: 3D graphics

Graphics execution shares this page's submission and completion machinery but runs a different engine workload. The story so far: the X server and GLX run on the emulated vGPU; the emulation captures the X server's graphics-class stream and the engine executes it — both as a verbatim 3D method-stream replay and as a synthesized launch that retires real sm_75 SASS — and the guest Vulkan pipeline runs end to end (device → command buffer → fence → readback). What is missing is rendered pixels: every frame is still black, and the program-delivery/program-VA-mapping pair is the working frontier. The full picture lives on Graphics-Execution.md.

No breadth beyond these is claimed: Windows guests, migration and performance numbers remain planned.

3. How we know (observables and their limits)

Observable	What it proves	What it does not prove
Client verification output (bad=0 + STATUS OK)	the guest executed its kernel and verified its results	by itself, which engine produced the results
Engine-authored release relayed to the guest	the engine executed the launch and the relay works	nothing about other launches
The guest's synchronisation wait resolving	completion reached the guest through the relay	nothing about other launches
Ring entry consumed; ring cursor advances	the front-end is alive and fetching	not engine execution — front-end signals are necessary but not sufficient
Reproduction across fresh boots	the path is not a lucky boot	no "every boot" claim yet while re-measurement is in progress
No GR fault recorded	the engine did not raise this error	silence is not health: a skipped launch is silent by construction

Consequence: acceptance is verified guest results plus the engine-authored release; a green run counts as progress only when it reproduces across fresh boots.

4. What is exonerated (do not relitigate)

• Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set).
• The descriptor destination window mapping — installed in the engine's table view at submit.
• Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0.
• The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta.
• Parenting/TSG for the fetch — both variants fetch identically.
• The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is separate.
• The descriptor-gate line of investigation (the old T1–T4 theories) — the inline-descriptor install, the engine read, the dispatch, the engine-authored release and the guest wake-up are all observed; closed by observation.
• The old boot-poisoning defect — root-caused to the status-queue consumption cursor and fixed.

5. The clear issues (ranked)

Issue A — the staged-transfer class. Multi-megabyte transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify, with the failure modes proven and named.

Issue B — every-boot reproducibility (acceptance campaign). The boot-poisoning defect and the post-reboot ring-block ownership defect were root-caused and fixed, and green clients reproduce on fresh boots; the formal every-boot acceptance campaign is the remaining work.

Issue C — per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level (two guests demonstrated); per-guest execution isolation for simultaneous CUDA guests is in progress.

Issue D — the graphics pixel frontier. The engine executes graphics method streams, but no rendered (non-black) frame has been produced yet; program delivery and program-VA mapping are the named blockers (Graphics-Execution.md).

6. Lines of investigation (live)

1. Repair the staged-transfer class — the green clients stay the regression battery and must remain green throughout.
2. Complete the every-boot acceptance campaign — demonstrate the fixed boot path across a defined run of fresh boots before claiming boot-to-boot reproducibility.
3. Per-guest execution isolation — build from the device-level coexistence to isolation for simultaneous CUDA guests.
4. Hold the acceptance battery — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (bad=0 + STATUS OK).
5. The graphics pixel frontier — materialise the driver's program uploads and map the program VA so draws execute with real state; the lanes, the walls and the milestone ladder are on Graphics-Execution.md.

7. Theories, resolved

The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is engineering (staged transfers, an every-boot acceptance campaign, per-guest isolation, the graphics pixel frontier), each with an acceptance signal named before the work.

8. Claim discipline

• Engine-authored release is the acceptance signal for execution — now observed end to end; the guest's own verification (bad=0 + STATUS OK) is the guest-side acceptance signal.
• Front-end signals are necessary but not sufficient. Ring cursor and front-end method execution can be front-end-authored; they never substitute for the engine-authored release.
• Silence is not health. No fault recorded does not mean no fault occurred.
• Emulator verdicts are instruction/memory-level. They bound behaviour, never timing or device environment.
• A green run counts only when it reproduces across fresh boots.
• A wall is closed only by an acceptance signal, named before the experiment.

9. Related pages

• Project-Status.md — the project-level ladder.
• CUDA-Userspace-Driver.md — the guest-side client path.
• Physical-GPU-Backend.md — the backend the work is forwarded to.
• Graphics-Execution.md — the parallel graphics frontier (OpenGL/Vulkan), sharing this machinery.
• Execution-Plane.md — the execution layer's design.
• Introduction-to-GMMU.md — page tables, QMDs and the engines.
• ../docs/04-implementation/03-execution-frontier.md — the canonical, continuously-updated version with the full evidence tables.