GSP and GSP-RM

GSP and GSP-RM

What this page is. The GPU System Processor hosts NVIDIA's Resource Manager in firmware; since Turing, the guest driver's management path is talking to that firmware, not to silicon. This page explains that architecture and the interface the project emulates. Condensed from ../docs/03-architecture/03-gsp-and-gsp-rm.md; the docs tree wins on disagreement.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] proven with the stock guest driver; [wip] partial; [wall] current blocker; [target] designed end state.

1. What the GSP is

The GSP is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that contains, among other things, a full port of NVIDIA's Resource Manager (RM) — the component that owns GPU initialization, memory, engines, interrupts and scheduling.

   ┌──────────────────────────── GUEST ────────────────────────────┐
   │  nvidia.ko                                                   │
   │    RM client path ── RPC ──┐                                  │
   │    GSP boot code           │                                  │
   └────────────────────────────┼──────────────────────────────────┘
                                │  shared memory queues + doorbells
   ┌────────────────────────────┼──────────────────────────────────┐
   │  GSP (firmware, RISC-V)    │                                   │
   │    RM core (RMCore)  ◄─────┘                                   │
   │    RM API  (object model, clients, handles)                    │
   │    Resource server (resserv): allocation & lifetime            │
   │    memory manager, GMMU, FIFO/runlist, interrupts, PMU         │
   └───────────────────────────────────────────────────────────────┘

A driver as complex as NVIDIA's is not written to program silicon directly; it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an RM client: it issues requests, receives callbacks, and observes state through a well-defined protocol.

2. The boot handshake we emulate

The guest driver boots the GSP exactly as it would on bare metal:

1. WPR setup — carve a firmware-protected region of VRAM and write descriptors into BAR0 registers (WPR-Management.md).
2. FWSEC — the driver verifies firmware signatures.
3. Booter — the boot ROM consumes the firmware image.
4. RISCV start — the RISC-V core begins executing the firmware.
5. GSP init done — the firmware signals readiness through a status mailbox; the driver proceeds to initialization RPCs.

The emulation implements stage completion rather than the stages themselves — legitimate because the only observable to the driver is the state machine it polls. The same handshake has also been run against real firmware on the physical backend: the firmware comes up, delivers its first status events, serves the static configuration, and stays live until the unload RPC releases it (Physical-GPU-Backend.md).

3. The transport: message queues and RPC

GSP-RM's transport is a shared-memory message queue pair:

  guest system memory (allocated by the guest driver, registered with us)
  ┌───────────────────────────────┐
  │ command queue (guest -> GSP)  │  element ring, 4 KB elements
  ├───────────────────────────────┤
  │ status queue  (GSP -> guest)  │  element ring
  └───────────────────────────────┘
     + a page table describing the queue pages (the buffer is
       non-contiguous: each page is an independent allocation)

Each element begins with a transport header followed by an RPC message whose 32-byte header carries header version, length, function, result and sequence number. Observations worth knowing (all verified against the stock guest):

• length covers the fixed part of the message, not variable payloads — do not use it to bound a payload;
• function numbers are a flat enum shared by several protocols (SET_GUEST_SYSTEM_INFO 1, FREE 10, GET_GSP_STATIC_INFO 65, GSP_RM_CONTROL 76, GSP_RM_ALLOC 103, ...);
• replies must be matched to request sequence numbers — the driver runs a reply-matching state machine, and a slow reply is indistinguishable from a hang.

4. RMCore, the RM API and the resource server

Three RM concepts make every log readable:

• RMCore — the object system. Everything is an object with a class (a 32-bit class ID), a handle in a per-client handle space, and a parent object forming a tree rooted at a client. Traces of GSP_RM_ALLOC are object creation; traces of GSP_RM_CONTROL are method calls on objects.
• RM API — how in-kernel components (and user space via ioctls) allocate objects and call controls, with an implicit-handle and an explicit-handle variant. For a GSP client, part of the API becomes RPCs while some calls are handled locally; memory objects, for example, are allocated locally by the client RM.
• Resource server (resserv) — the allocation authority: it owns handle spaces, enforces privileges and tracks object lifetimes.

5. The client/server split (and why page tables matter)

Modern RM is client/server: the client (guest nvidia.ko) owns per-process state and submits work; the server (GSP-RM) owns GPU-global state — routing, page tables, scheduling domains, engine configuration. For virtualized setups the client tells the server which VA ranges the server owns, and the server manages the deeper page-table levels for those ranges. The client hands over the physical addresses of its page-table levels plus the VA ranges they cover; on real hardware the server then fills in the lower levels for server-owned ranges and the client never writes them.

This is a central fact for the execution work: when the guest's UVM asks the "GPU" to map memory, the mapping is expected to materialize in page tables that only the server side can write — which is us (UVM-in-the-Guest.md, Introduction-to-GMMU.md).

6. Event channels and notifications

Beyond request/reply, the guest needs asynchronous events: event objects registered with a notification configuration (an MSI-X vector or a memory "semaphore" the GPU can write), completion events when channels finish work or errors occur, delivered as a strict bit/handle/state triple on the exact vector. The emulation's MSI-X path is emulation thread → eventfd → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost.

7. What we emulate today

Layer	State	Notes
GSP boot state machine	[live]	stages complete; the driver's poll loop is satisfied
Message queue transport	[live]	including the non-contiguous page table for the queue buffer
RPC request/reply	[live]	every function the driver uses during init is answered; unknown functions get a well-formed success reply
Static info tables	[wip]	device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list
Controls during init	[wip]	~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving
Object creation	[wip]	handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet
Page tables for server-owned ranges	[wip]	contract received and recorded; leaf population is the execution work
Event/notification delivery	[wip]	MSI-X works; richer event semantics land with forwarding
Firmware behaviour (routing, scheduling, real execution)	[wall]	the execution frontier — see Execution-Frontier.md

8. Where the firmware knowledge lives

• GSP-570.144W-Firmware-Analysis.md — how the 28.5 MB image was decompiled and mapped.
• Address-Reference.md — firmware-side address tables.
• RPC-Message-Format.md, GSP-RPC-Emulation.md — historical RPC analysis (superseded; kept for the record).
• WPR-Management.md — write-protected region semantics.
• The execution plane and the execution frontier — what turns this protocol into real work on real hardware.