The vGSP Execution Plane (vEP)

The vGSP Execution Plane (vEP)

Canonical, maintained version of this page: ../docs/03-architecture/13-vexecution-plane.md. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: ../docs/README.md.

Status: 2026-10-08, tree master @ 1f571dda. Tags: [live] proven with the stock guest driver; [gated] implemented, default-off or parameter-gated; [wip] partial; [wall] current blocker; [target] named success latch; [planned] designed, not built.

1. What it is

vGSP (virtual GSP) is our host-side reimplementation of the GPU System Processor interface: the virtual device plus the emulated GSP-RM service the stock guest driver talks to, implemented by vgsp.ko. The vGSP Execution Plane (vEP) is the half of vGSP that turns guest-authored GPU work into physical execution and guest-visible completion: channel ring and USERD discovery, GPU VA-to-PA truth, semaphore-release application, completion counters and notifiers, QMD handling for forwarded launches, and submission to the physical GPU's engines under the module-owned backend session.

The other half — the control plane (boot handshake, message queues, RPC, RM objects, BAR shadows) — makes the guest driver run. The vEP makes the guest's work run. The canonical document defines the name and rejects the alternates ("work plane", "submission plane", "forwarding plane", ...) that would collide with existing project vocabulary.

Four work domains organize the subsystem:

Domain	Question	Implemented by
Scheduler	which channel's work runs, at what pace	doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request
Allocator	where rings, USERDs, counters, notifiers and scratch windows live	ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow
Control	which RM objects/controls shape execution	channel and event registration, SET_PAGE_DIRECTORY, PROMOTE_CTX, engine topology, the mode matrix
Share	how clients/mdevs share the one GPU	mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned

2. The shape of the plane

guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
  -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
  -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
  -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
  -> engine-authored release -> poll -> completion publication

3. One CUDA kernel launch, end to end

1. Launch build (guest). libcuda/nvidia.ko build the GR push: context methods, the inline launch descriptor (QMD), SASS and parameter pointers, and a release target. They append a GPFIFO entry and ring the BAR0 doorbell. [live] observed with the stock driver.
2. Steering and truth (host). The doorbell wakes the service thread; the pump resolves the channel's ring VA and USERD, reads GP_PUT, fetches the GPFIFO entry and decodes the method stream. Translations are read-back proven from driver-published page directories, pins, or content-rescue. [live] for CE; [wip] for GR (discovery now succeeds, see §4).
3. Capture. The decoder accumulates the QMD window and captures the launch candidate; the sema service queues one request on the single ordered workqueue. [live] observed (ring consumed, methods execute).
4. Physical submit. The backend relocates the QMD, copies program/param pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to the ladder channel's GPFIFO and rings the real doorbell. [gated] default-off; latched on a proven real GR submission.
5. Engine. The front-end fetches the entry and executes methods; then the inline-QMD install must materialise the descriptor for the engine to read, currently [wall] — not observed. Context validity, destination mapping and fetch/alignment are exonerated; the recovered-method-stream form is the investigation.
6. Completion. The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X wakes the guest. CE work is live in this shape [gated]; for GR the completion-counter publication is the current second face of the wall.

4. How work is found: discovery, truth, completion

• Ring/USERD discovery has three cooperating sources: injected pins, a kernel-native trusted walk of driver-published page directories, and content-keyed discovery when neither names the ring. Content discovery scans both apertures (emulated vidmem and guest RAM), runs the FB scan first on a 2-second cadence, pairs a ring with the rank's staged ctx-init push, and validates the pair before pinning. Since the 2026-10-08 re-arm work (7abd8f87, 66388560) the discovery no longer retires permanently at a pass cap.
• The semaphore service decodes releases, translates their target pages through the trusted resolver plus a driver-published-VAS pool resolver, defers untranslatable releases with retry telemetry, and applies each release exactly once — in software, or by letting the engine author it under the relay (default off).
• Completion counters use a fixed 20-slot pool (a partial live channel count poisoned slots before the fix); each decoded release lands in the channel's exact slot. Notifiers are serviced by the pump, and family events are delivered as RM events + MSI-X through the guest's registered event objects.

5. Challenges, honestly

• GR dispatch wall. The forward path runs front-end methods, the context is valid and the destination window is accepted — and the descriptor install is still not observed. The mode matrix carries dozens of single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory block, lane controls, shadow windows, pointer relocation).
• Completion publication. A boot where GR rings were discovered and releases applied still saw the guest re-queue the same push and give up. A later sema_dbg=Y run settled it: the release was written to the exact slot, the notifier cleared and the event posted, while the completion counter publication never wrote because the fixed-total autoreg mapping makes it a structural no-op. The leading reading is the counter-slot publication; the GP_GET write-back is the secondary, still-open candidate. Both are debug-gated today.
• Reproducibility (ISSUE-0001). The green/stall split is per-boot. The first fix wave landed and the earlier CE freeze is gone; the stall moved to GR ring discovery, which the latest discovery work addresses. Acceptance is 10 cold boots green, pending.
• Multiple clients (ISSUE-0002). A second CUDA process wedges in guest UVM because host per-client sema/GR state is not reset when the guest re-runs its per-client boot handshake. Per-client state containers and reset are the designed path.

6. Portability

The architecture ports: observe pushes, resolve truth, decode, execute/forward, publish completion. So does the RPC/RM object model and the mode-matrix practice. What does not port is address-level fact: ring conventions, VA windows, QMD offsets, context-image layout, PTE encodings, engine inventories and scratch layouts must be re-derived per driver version and GPU architecture by measurement, never guessed. A new version enters as a profile plus a measured delta; driver 610 additionally sits behind an explicit human approval gate.

7. Future

The planned next steps are reproducibility acceptance, the descriptor-install and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger where possible), the arbitrary-kernel acceptance suite, then the near-native performance program. The unsolved future classes — multi-tenant fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction in the canonical document. 3D graphics reuses this pipeline once compute dispatch is proven; it adds engine breadth (video/image), more channels and synchronisation volume, and a much larger method vocabulary, but not a new plane.

---

Canonical document: docs/03-architecture/13-vexecution-plane.md. See also: full system picture, physical backend guest wiring, the execution frontier.