GSP and GSP-RM

GSP and GSP-RM

What this page is. The GPU System Processor hosts NVIDIA's Resource Manager in firmware; since Turing, the guest driver's management path is talking to that firmware, not to silicon. This page explains that architecture and the interface the project emulates. Condensed from ../docs/03-architecture/03-gsp-and-gsp-rm.md; the docs tree wins on disagreement.

Status: 2026-10-30, tree master @ b4930168. Tags: [live] proven with the stock guest driver; [wip] partial; [wall] current blocker; [target] designed end state.

1. What the GSP is

The GSP is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that contains, among other things, a full port of NVIDIA's Resource Manager (RM) — the component that owns GPU initialization, memory, engines, interrupts and scheduling.

   ┌──────────────────────────── GUEST ────────────────────────────┐
   │  nvidia.ko                                                   │
   │    RM client path ── RPC ──┐                                  │
   │    GSP boot code           │                                  │
   └────────────────────────────┼──────────────────────────────────┘
                                │  shared memory queues + doorbells
   ┌────────────────────────────┼──────────────────────────────────┐
   │  GSP (firmware, RISC-V)    │                                   │
   │    RM core (RMCore)  ◄─────┘                                   │
   │    RM API  (object model, clients, handles)                    │
   │    Resource server (resserv): allocation & lifetime            │
   │    memory manager, GMMU, FIFO/runlist, interrupts, PMU         │
   └───────────────────────────────────────────────────────────────┘

A driver as complex as NVIDIA's is not written to program silicon directly; it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an RM client: it issues requests, receives callbacks, and observes state through a well-defined protocol.

What the GSP does on bare metal

• It is the device's manager processor. Since Turing, the driver does not program most of the GPU. The GSP runs NVIDIA's signed firmware — a small RTOS plus the RM server — and owns GPU-global state: initialisation, the memory-management unit's deeper page-table levels, engine configuration, context scheduling, interrupts and power/clock management (PMU). The driver's kernel RM boots it, then becomes its client.
• It boots from driver-provided memory. At power-on the driver carves a firmware-protected region of video memory (WPR2), copies the firmware image there, has the device verify signatures (FWSEC), hands control to the boot ROM ("booter"), which starts the RISC-V core; the firmware runs its RTOS and RM initialisation and signals readiness. Everything the driver does afterwards is a request to that server.
• Why this matters for virtualization. Because the management path is a firmware interface — shared-memory queues plus a small RPC protocol, not a pile of register lore — it is small, stable, and observable. That is the interface Open-IOV emulates; it is the reason a virtual NVIDIA GPU is practical at all.

2. The boot handshake we emulate

The guest driver boots the GSP exactly as it would on bare metal:

1. WPR setup — carve a firmware-protected region of VRAM and write descriptors into BAR0 registers (WPR-Management.md).
2. FWSEC — the driver verifies firmware signatures.
3. Booter — the boot ROM consumes the firmware image.
4. RISCV start — the RISC-V core begins executing the firmware.
5. GSP init done — the firmware signals readiness through a status mailbox; the driver proceeds to initialization RPCs.

The emulation implements stage completion rather than the stages themselves — legitimate because the only observable to the driver is the state machine it polls. The same handshake has also been run against real firmware on the physical backend: the firmware comes up, delivers its first status events, serves the static configuration, and stays live until the unload RPC releases it (Physical-GPU-Backend.md).

3. The transport: message queues and RPC

GSP-RM's transport is a shared-memory message queue pair:

  guest system memory (allocated by the guest driver, registered with us)
  ┌───────────────────────────────┐
  │ command queue (guest -> GSP)  │  element ring, 4 KB elements
  ├───────────────────────────────┤
  │ status queue  (GSP -> guest)  │  element ring
  └───────────────────────────────┘
     + a page table describing the queue pages (the buffer is
       non-contiguous: each page is an independent allocation)

Each element begins with a transport header followed by an RPC message whose 32-byte header carries header version, length, function, result and sequence number. Observations worth knowing (all verified against the stock guest):

• length covers the fixed part of the message, not variable payloads — do not use it to bound a payload;
• function numbers are a flat enum shared by several protocols (SET_GUEST_SYSTEM_INFO 1, FREE 10, GET_GSP_STATIC_INFO 65, GSP_RM_CONTROL 76, GSP_RM_ALLOC 103, ...);
• replies must be matched to request sequence numbers — the driver runs a reply-matching state machine, and a slow reply is indistinguishable from a hang.

4. RMCore, the RM API and the resource server

Three RM concepts make every log readable:

• RMCore — the object system. Everything is an object with a class (a 32-bit class ID), a handle in a per-client handle space, and a parent object forming a tree rooted at a client. Traces of GSP_RM_ALLOC are object creation; traces of GSP_RM_CONTROL are method calls on objects.
• RM API — how in-kernel components (and user space via ioctls) allocate objects and call controls, with an implicit-handle and an explicit-handle variant. For a GSP client, part of the API becomes RPCs while some calls are handled locally; memory objects, for example, are allocated locally by the client RM.
• Resource server (resserv) — the allocation authority: it owns handle spaces, enforces privileges and tracks object lifetimes.

5. The client/server split (and why page tables matter)

Modern RM is client/server: the client (guest nvidia.ko) owns per-process state and submits work; the server (GSP-RM) owns GPU-global state — routing, page tables, scheduling domains, engine configuration. For virtualized setups the client tells the server which VA ranges the server owns, and the server manages the deeper page-table levels for those ranges. The client hands over the physical addresses of its page-table levels plus the VA ranges they cover; on real hardware the server then fills in the lower levels for server-owned ranges and the client never writes them.

This is a central fact for the execution work: when the guest's UVM asks the "GPU" to map memory, the mapping is expected to materialize in page tables that only the server side can write — which is us (UVM-in-the-Guest.md, Introduction-to-GMMU.md).

6. Event channels and notifications

Beyond request/reply, the guest needs asynchronous events: event objects registered with a notification configuration (an MSI-X vector or a memory "semaphore" the GPU can write), completion events when channels finish work or errors occur, delivered as a strict bit/handle/state triple on the exact vector. The emulation's MSI-X path is emulation thread → eventfd → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost.

7. How Open-IOV deals with the GSP — two modes

The project handles the GSP in two ways, and they complement each other:

1. Guest-facing GSP emulation (vgsp.ko is the virtual GSP). For each mediated device, the module presents a device whose GSP behaviour is software: it implements the boot state machine the driver polls (stage completion, not silicon), answers the WPR/BAR0 register protocol, serves the message-queue transport and every RPC verb the driver uses during probe, initialisation and runtime, and delivers events via emulated MSI-X. The guest's RM believes it is talking to firmware; it is talking to the emulation.
2. The real GSP as the backend server. When the module owns the physical GPU, it loads the matching real firmware image into the device, runs the actual boot (FWSEC → booter → RISC-V → RM init-done), and becomes the RM client itself — "be the host driver's RM" becomes "be the RM client". The same protocol the guest uses one layer up is used here one layer down, and it is what the work path submits through (Physical-GPU-Backend.md).

Keeping the two in sync is a project discipline: what the guest's RM is promised (objects, controls, page tables, completions) must be answerable by the emulation, and what executes on hardware must round-trip through the real firmware session.

What is emulated today

Layer	State	Notes
GSP boot state machine	[live]	stages complete; the driver's poll loop is satisfied
Message queue transport	[live]	including the non-contiguous page table for the queue buffer
RPC request/reply	[live]	every function the driver uses during init is answered; unknown functions get a well-formed success reply
Static info tables	[wip]	device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list
Controls during init	[wip]	~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving
Object creation	[wip]	handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet
Page tables for server-owned ranges	[wip]	contract received and recorded; forwarded paths run on module-built scratch tables while full server-owned leaf population is the remaining execution work
Event/notification delivery	[live]	MSI-X works; completion counters, notifiers and channel events are serviced and relayed to the guest
Real GSP as the backend server	[live]	the module boots the real firmware, serves its RM client ladder, and executes forwarded work through it
Firmware behaviour (routing, scheduling, real execution)	[live]	forwarded guest CUDA launches and graphics method streams execute on the physical engines — see Execution-Frontier.md

8. Where the firmware knowledge lives

• GSP-570.144W-Firmware-Analysis.md — how the 28.5 MB image was decompiled and mapped.
• Address-Reference.md — firmware-side address tables.
• RPC-Message-Format.md, GSP-RPC-Emulation.md — historical RPC analysis (superseded; kept for the record).
• WPR-Management.md — write-protected region semantics.
• The execution plane and the execution frontier — what turns this protocol into real work on real hardware.