The vGSP Execution Plane (vEP)

The vGSP Execution Plane (vEP)

Status: 2026-10-08, tree master @ 1f571dda. Tags: [live] proven with the stock guest driver; [gated] implemented, default-off or parameter-gated; [wip] partial; [wall] current blocker; [target] named success latch; [planned] designed, not built.

1. What it is

vGSP (virtual GSP) is our host-side reimplementation of the GPU System Processor interface: the virtual device plus the emulated GSP-RM service the stock guest driver talks to, implemented by vgsp.ko. The vGSP Execution Plane (vEP) is the half of vGSP that turns guest-authored GPU work into physical execution and guest-visible completion: channel ring and USERD discovery, GPU VA-to-PA truth, semaphore-release application, completion counters and notifiers, QMD handling for forwarded launches, and submission to the physical GPU's engines under the module-owned backend session. The other half — the control plane (boot handshake, message queues, RPC, RM objects, BAR shadows) — makes the guest driver run. The vEP makes the guest's work run. Four work domains organize the subsystem:

Domain	Question	Implemented by
Scheduler	which channel's work runs, at what pace	doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request
Allocator	where rings, USERDs, counters, notifiers and scratch windows live	ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow
Control	which RM objects/controls shape execution	channel and event registration, SET_PAGE_DIRECTORY, PROMOTE_CTX, engine topology, the mode matrix
Share	how clients/mdevs share the one GPU	mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned

2. The shape of the plane

flowchart TB
  subgraph GUEST["Guest VM - stock software"]
    APP["CUDA application -> libcuda -> nvidia.ko<br/>push + QMD + GPFIFO + doorbell"]
  end
  subgraph VEP["vGSP Execution Plane inside vgsp.ko"]
    DB["doorbell steering"] --> PUMP["channel pump"]
    PUMP --> DISC["ring/USERD discovery + pins"]
    DISC --> TRUTH["VA-to-PA truth<br/>walk, rescue, pins"]
    TRUTH --> DEC["push decode + QMD capture"]
    DEC --> REL["release apply + counters + notifiers"]
    DEC --> WQ["ordered workqueue<br/>one in-flight request"]
    WQ --> SUB["ladder submission<br/>mode matrix"]
  end
  subgraph PHY["Physical backend"]
    LAD["module-owned ladder channel"] --> GSP["real GSP-RM server"]
    GSP --> ENG["GR / CE engines"]
  end
  APP -->|"MMIO + DMA"| DB
  REL -->|"release, counter, event"| APP
  SUB --> LAD
  ENG -->|"engine-authored release"| SUB

ASCII fallback:

<syntaxhighlight lang="text">guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)

 -> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
 -> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
 -> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
 -> engine-authored release -> poll -> completion publication</syntaxhighlight>

3. One CUDA kernel launch, end to end

1. Launch build (guest). libcuda/nvidia.ko build the GR push: context methods, the inline launch descriptor (QMD), SASS and parameter pointers, and a release target. They append a GPFIFO entry and ring the BAR0 doorbell. [live] observed with the stock driver.
2. Steering and truth (host). The doorbell wakes the service thread; the pump resolves the channel's ring VA and USERD, reads GP_PUT, fetches the GPFIFO entry and decodes the method stream. Translations are read-back proven from driver-published page directories, pins, or content-rescue. [live] for CE; [wip] for GR (discovery now succeeds, see §4).
3. Capture. The decoder accumulates the QMD window and captures the launch candidate; the sema service queues one request on the single ordered workqueue. [live] observed (ring consumed, methods execute).
4. Physical submit. The backend relocates the QMD, copies program/param pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to the ladder channel's GPFIFO and rings the real doorbell. [gated] default-off; latched on a proven real GR submission.
5. Engine. The front-end fetches the entry and executes methods; then the inline-QMD install must materialise the descriptor for the engine to read, currently [wall] — not observed. Context validity, destination mapping and fetch/alignment are exonerated; the recovered-method-stream form is the investigation.
6. Completion. The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X wakes the guest. CE work is live in this shape [gated]; for GR the completion-counter publication is the current second face of the wall.

4. How work is found: discovery, truth, completion

• Ring/USERD discovery has three cooperating sources: injected pins, a kernel-native trusted walk of driver-published page directories, and content-keyed discovery when neither names the ring. Content discovery scans both apertures (emulated vidmem and guest RAM), runs the FB scan first on a 2-second cadence, pairs a ring with the rank's staged ctx-init push, and validates the pair before pinning. Since the 2026-10-08 re-arm work (7abd8f87, 66388560) the discovery no longer retires permanently at a pass cap.
• The semaphore service decodes releases, translates their target pages through the trusted resolver plus a driver-published-VAS pool resolver, defers untranslatable releases with retry telemetry, and applies each release exactly once — in software, or by letting the engine author it under the relay (default off).
• Completion counters use a fixed 20-slot pool (a partial live channel count poisoned slots before the fix); each decoded release lands in the channel's exact slot. Notifiers are serviced by the pump, and family events are delivered as RM events + MSI-X through the guest's registered event objects.

5. Challenges, honestly

• GR dispatch wall. The forward path runs front-end methods, the context is valid and the destination window is accepted — and the descriptor install is still not observed. The mode matrix carries dozens of single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory block, lane controls, shadow windows, pointer relocation).
• Completion publication. A boot where GR rings were discovered and releases applied still saw the guest re-queue the same push and give up. A later sema_dbg=Y run settled it: the release was written to the exact slot, the notifier cleared and the event posted, while the completion counter publication never wrote because the fixed-total autoreg mapping makes it a structural no-op. The leading reading is the counter-slot publication; the GP_GET write-back is the secondary, still-open candidate. Both are debug-gated today.
• Reproducibility (ISSUE-0001). The green/stall split is per-boot. The first fix wave landed and the earlier CE freeze is gone; the stall moved to GR ring discovery, which the latest discovery work addresses. Acceptance is 10 cold boots green, pending.
• Multiple clients (ISSUE-0002). A second CUDA process wedges in guest UVM because host per-client sema/GR state is not reset when the guest re-runs its per-client boot handshake. Per-client state containers and reset are the designed path.

6. Portability

The architecture ports: observe pushes, resolve truth, decode, execute/forward, publish completion. So does the RPC/RM object model and the mode-matrix practice. What does not port is address-level fact: ring conventions, VA windows, QMD offsets, context-image layout, PTE encodings, engine inventories and scratch layouts must be re-derived per driver version and GPU architecture by measurement, never guessed. A new version enters as a profile plus a measured delta; driver 610 additionally sits behind an explicit human approval gate.

7. Future

The planned next steps are reproducibility acceptance, the descriptor-install and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger where possible), the arbitrary-kernel acceptance suite, then the near-native performance program. The unsolved future classes — multi-tenant fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction. 3D graphics reuses this pipeline once compute dispatch is proven; it adds engine breadth (video/image), more channels and synchronisation volume, and a much larger method vocabulary, but not a new plane.

---