The Execution Frontier

The Execution Frontier

What this page is. The state of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, how we know, and where the leading edge is now. The chain fires end to end: verified guest CUDA clients execute on the physical GPU and wake up correctly; the leading edge is now breadth and hardening. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from ../docs/04-implementation/03-execution-frontier.md; the docs tree wins on disagreement.

Status: 2026-10-20, tree master @ 16291c72. Tags: [live] observed; [wip] partial; [wall] current blocker; theories are labelled as such.

1. The chain that should fire

A guest CUDA launch becomes a method stream in a push buffer, referenced by a GPFIFO ring entry, announced by a doorbell. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware front-end fetches and executes the methods, the launch descriptor (QMD) is materialised and read by the GR engine, the grid is dispatched to the SMs, and completion is published as semaphores the guest waits on.

guest: builds launch + method stream
       GPFIFO entry + doorbell
   -> host: doorbell trap, decode, rebuild forward stream,
            prewrite QMD, ring/USERD, doorbell
   -> front-end: fetch GPFIFO entry (ring cursor advances)
   -> front-end: execute methods (object bind, cache ops, report sem)
   -> front-end: install inline QMD into the descriptor buffer
   -> GR engine: read QMD through the channel's page tables
   -> context switch loads the channel context (FECS/ucode)
   -> GR: dispatch grid to the SMs
   -> engine authors release semaphore + channel semaphore
   -> host polls release, relays to guest; CUDA wakes

#	Stage	State
1	Guest builds launch + method stream	[live] observed (stock driver, unmodified)
2	GPFIFO entry + doorbell	[live] observed
3	Host decode + forward stream + QMD prewrite + ring/USERD	[live] observed
4	Front-end fetches the ring entry	[live] observed (ring cursor advances)
5	Front-end executes methods	[live] observed (object bind, cache ops, report semaphore)
6	Inline-QMD install (descriptor materialised)	[live] observed
7	Engine reads the QMD	[live] observed
8	Context switch loads the channel context	[live] context validity at dispatch is ruled in
9	GR dispatches the grid to the SMs	[live] observed (the guest's results verify)
10	Engine authors release + channel semaphore	[live] engine-authored releases relay to the guest
11	Host relays completion; guest wakes	[live] the guest's synchronisation waits resolve

One-line summary: the whole chain is live; the leading edge is now breadth and hardening — the large-buffer staged-transfer class, reproducibility across boots, and per-guest execution isolation.

2. Where the leading edge is

A stock, unmodified guest CUDA program executes its kernels on the physical GPU and verifies its results. On fresh boots the vector-add client (victory_mail) prints VADD n=32768 bad=0 + STATUS OK, the matrix-multiply client (victory_matmul) prints MATMUL n=256 bad=0 + STATUS OK, and a transformer kernel prints dh=256 seq=64 bad=0, FFN4 layers=4 bad=0 + STATUS OK. The executing engine is the physical GPU under the module's own real GSP-RM session; the host CUDA bridge is not part of this path. Engine-authored completions relay to the guest and its synchronisation waits resolve.

The leading edge has moved from the launch chain itself — it fires — to breadth and hardening around it:

• The large-buffer staged-transfer class. Multi-megabyte staged transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify.
• Reproducibility across boots. A boot-poisoning defect was root-caused and fixed: under the driver's status-queue mode the emulation read the wrong queue cursor, so under load the reply ring could desynchronise and the first CUDA client would fail (the user-visible STATUS FAIL-then-freeze report). Re-measurement across fresh boots is in progress.
• Per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests already coexist at the device level; per-guest execution isolation for multiple simultaneous CUDA guests is in progress.

No breadth beyond these is claimed: OpenGL/Vulkan, Windows guests, migration and performance numbers remain planned.

3. How we know (observables and their limits)

Observable	What it proves	What it does not prove
Client verification output (bad=0 + STATUS OK)	the guest executed its kernel and verified its results	by itself, which engine produced the results
Engine-authored release relayed to the guest	the engine executed the launch and the relay works	nothing about other launches
The guest's synchronisation wait resolving	completion reached the guest through the relay	nothing about other launches
Ring entry consumed; ring cursor advances	the front-end is alive and fetching	not engine execution — front-end signals are necessary but not sufficient
Reproduction across fresh boots	the path is not a lucky boot	no "every boot" claim yet while re-measurement is in progress
No GR fault recorded	the engine did not raise this error	silence is not health: a skipped launch is silent by construction

Consequence: acceptance is verified guest results plus the engine-authored release; a green run counts as progress only when it reproduces across fresh boots.

4. What is exonerated (do not relitigate)

• Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set).
• The descriptor destination window mapping — installed in the engine's table view at submit.
• Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0.
• The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta.
• Parenting/TSG for the fetch — both variants fetch identically.
• The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is separate.
• The descriptor-gate line of investigation (the old T1–T4 theories) — the inline-descriptor install, the engine read, the dispatch, the engine-authored release and the guest wake-up are all observed; closed by observation.
• The old boot-poisoning defect — root-caused to the status-queue consumption cursor and fixed.

5. The clear issues (ranked)

Issue A — the large-buffer staged-transfer class. Multi-megabyte staged transfers between guest memory and device memory are under repair: some large multi-chunk clients still mis-verify.

Issue B — reproducibility across boots (re-measurement). The boot-poisoning defect (wrong status-queue consumption cursor under the driver's status-queue mode; the reply ring could desynchronise under load, the first client would fail) was root-caused and fixed. Re-measurement across fresh boots is in progress; no "every boot" claim is made until it completes.

Issue C — per-guest execution isolation. Multiple mediated devices with different emulated VRAM sizes and multiple guests coexist at the device level; per-guest execution isolation for simultaneous CUDA guests is in progress.

6. Lines of investigation (live)

1. Repair the large-buffer staged-transfer class — the three green clients stay the regression battery and must remain green throughout.
2. Complete the fresh-boot re-measurement — demonstrate the fixed boot path across fresh boots before claiming boot-to-boot reproducibility.
3. Per-guest execution isolation — build from the device-level coexistence to isolation for simultaneous CUDA guests.
4. Hold the acceptance battery — vector-add, matrix-multiply and the transformer kernel; each entry is accepted only on its own verification output (bad=0 + STATUS OK).

7. Theories, resolved

The descriptor-gate theories (T1–T4) are resolved and are not kept as open: the inline-descriptor install, the engine's QMD read, the grid dispatch, the engine-authored release and the guest wake-up are all directly observed, so the launch chain they questioned is proven end to end. The remaining work is engineering (large-buffer transfers, boot-to-boot reproducibility, per-guest isolation), each with an acceptance signal named before the work.

8. Claim discipline

• Engine-authored release is the acceptance signal for execution — now observed end to end; the guest's own verification (bad=0 + STATUS OK) is the guest-side acceptance signal.
• Front-end signals are necessary but not sufficient. Ring cursor and front-end method execution can be front-end-authored; they never substitute for the engine-authored release.
• Silence is not health. No fault recorded does not mean no fault occurred.
• Emulator verdicts are instruction/memory-level. They bound behaviour, never timing or device environment.
• A green run counts only when it reproduces across fresh boots.
• A wall is closed only by an acceptance signal, named before the experiment.

9. Related pages

• Project-Status.md — the project-level ladder.
• CUDA-Userspace-Driver.md — the guest-side client path.
• Physical-GPU-Backend.md — the backend the work is forwarded to.
• Execution-Plane.md — the execution layer's design.
• Introduction-to-GMMU.md — page tables, QMDs and the engines.
• ../docs/04-implementation/03-execution-frontier.md — the canonical, continuously-updated version with the full evidence tables.