GSP RPC Emulation

GSP vGPU RPC Emulation

Overview

Open-IOV includes emulation mode that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.

Problem

The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:

Handler	Address	Size	Purpose
BAR Trap	0x04CB8188	40KB	BAR1/ForceBarAccess/Aperture
WPR Handler 1	0x04F00B0C	62KB	Write/Read Protect Regions
WPR Handler 2	0x04F00C78	62KB	Write/Read Protect Regions
Migration	0x04179ED8/0x04179FBC	-	Live migration

Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).

Solution: Emulation Mode

Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:

┌─────────────────────────────────────────────────────────────────┐
│                    Emulation Architecture                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Userspace App (libiov.so)                                       │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────┐     ┌─────────────────┐                   │
│  │ Kernel IOCTL    │────▶│ Emulation Layer │                   │
│  │ Handler         │     │ (kernel)        │                   │
│  └─────────────────┘     └────────┬────────┘                   │
│                                    │                            │
│                    ┌───────────────┼───────────────┐           │
│                    ▼               ▼               ▼           │
│             ┌──────────┐    ┌──────────┐    ┌──────────┐      │
│             │ BAR Emu  │    │ WPR Emu  │    │ Mem Emu  │      │
│             │ (kernel) │    │ (kernel) │    │ (kernel) │      │
│             └──────────┘    └──────────┘    └──────────┘      │
│                    │               │               │           │
│                    ▼               ▼               ▼           │
│             ┌──────────────────────────────────────────┐       │
│             │         Userspace Emulator (libiov.so)   │       │
│             │  rpc_emu_bar_*, rpc_emu_wpr_*        │       │
│             └──────────────────────────────────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Auto-Detection (Default Behavior)

Emulation mode is ON by default and auto-detects at driver load:

1. Driver initializes RPC communication with GSP
2. Sends test BAR_TRAP RPC (non-destructive READ)
3. If GSP responds (ret=0) → vGPU handlers present → emulation disabled
4. If GSP times out/errors (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → emulation enabled

dmesg output:
  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support..
  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
  # or
  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode

Module Parameters

Parameter	Type	Default	Description
emulation_mode	bool	1 (ON)	Current state (read-only, shows auto-detected value)
emulation_override	bool	0 (auto)	Override: 1=force emulation ON, 0=force emulation OFF

<syntaxhighlight lang="bash"># Default: auto-detect sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

1. Force emulation ON (testing with GRID GSP)

sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1

1. Force emulation OFF (requires GRID GSP)

sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0</syntaxhighlight>

Emulated RPC Handlers

Kernel Emulation (driver/)

RPC Type	Handler	Emulation
GSP_RPC_ALLOC (0x01)	gsp_vf_mem_alloc	Synthetic handles/VAs from local counter
GSP_RPC_PIN (0x05)	gsp_vf_mem_pin	Synthetic DMA addresses from local counter
GSP_RPC_MAP (0x03)	gsp_vf_mem_map	Identity map (return requested VA)
GSP_RPC_BAR_TRAP (0x0A)	gsp_bar_trap_handler	Local aperture/force_access state
GSP_RPC_MAP (WPR)	gsp_vf_wpr_protect	Local WPR region list
GSP_RPC_UNMAP (WPR)	gsp_vf_wpr_unprotect	Local WPR region removal

Userspace Emulation (userspace/lib/rpc_emulator.c)

Function	Purpose
rpc_emu_bar_config	Local aperture offset/size + backing store
rpc_emu_bar_force_access	Local force_access state
rpc_emu_bar_save/restore	Migration state serialization
rpc_emu_wpr_protect_write/read	Local WPR region tracking
rpc_emu_wpr_unprotect	Local WPR region removal
rpc_emu_wpr_set_timeout	Local timeout tracking
rpc_emu_wpr_check_access	Access violation detection

Emulation Data Structures

BAR State (per VF)

<syntaxhighlight lang="c">struct gsp_bar_state {

   uint64_t aperture_offset;   // BAR1 aperture base
   uint64_t aperture_size;     // BAR1 aperture size (max 256KB)
   bool force_access;          // ForceBarAccess enabled
   uint32_t force_vf;          // VF holding force access
   void *backing_store;        // Emulated BAR memory
   size_t backing_size;        // Backing store size

};</syntaxhighlight>

WPR Region List (per VF)

<syntaxhighlight lang="c">struct gsp_wpr_region {

   struct list_head list;
   uint64_t base;              // Region base address
   uint64_t size;              // Region size
   uint32_t flags;             // GSP_WPR_WRITE/READ/EXECUTE
   uint32_t vf_id;             // Owning VF
   void (*completion_cb)(..); // Write/read complete callback
   void (*timeout_cb)(..);    // Timeout callback
   uint64_t timeout_ns;        // Timeout in nanoseconds

};</syntaxhighlight>

Userspace Emulator State

<syntaxhighlight lang="c">struct emu_bar_state {

   uint64_t aperture_offset;
   uint64_t aperture_size;
   bool force_access;
   uint32_t force_vf;
   void *backing_store;
   size_t backing_size;
   pthread_mutex_t lock;

};

struct emu_wpr_state {

   struct emu_wpr_region *regions;  // Linked list
   pthread_mutex_t lock;

};</syntaxhighlight>

Fallback Logic

Userspace library (libiov.so) implements fallback:

<syntaxhighlight lang="c">vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {

   // 1. Try kernel IOCTL (sends RPC to GSP)
   ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
   
   // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
   if (ret < 0 && dev->emulation_enabled && 
       (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
       return rpc_emu_bar_config(vf->id, offset, size);
   }
   
   return map_errno(errno);

}</syntaxhighlight>

Limitations

Feature	Emulation Status	Notes
BAR Aperture Config	✅ Full	Local state + backing store
ForceBarAccess	✅ Full	Per-VF exclusive access
BAR Save/Restore	✅ Full	Migration support
WPR Write Protect	✅ Full	Region tracking + access check
WPR Read Protect	✅ Full	Region tracking + access check
WPR Timeout	✅ Partial	Timeout stored, callbacks not wired
Memory Alloc	✅ Basic	Synthetic handles/VAs
Memory Pin	✅ Basic	Synthetic DMA addresses
Memory Map	✅ Basic	Identity mapping
Migration	⚠️ Partial	BAR/WPR state only
Scheduling	❌ No	Returns success, no actual scheduling
Notifications	❌ No	No async event delivery

Testing Emulation

<syntaxhighlight lang="bash"># 1. Load driver (auto-enables emulation on consumer GSP) sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

1. 2. Verify emulation enabled in dmesg

dmesg -T | grep -i emulation

1. gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode

1. 3. Run example - should complete without timeouts

sudo LD_LIBRARY_PATH=userspace/lib./userspace/examples/example_vgpu /dev/gsp-vgpu

1. 4. Check debugfs for emulated state

cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions</syntaxhighlight>

Future Work

• ☐ Wire WPR timeout callbacks to kernel timers
• ☐ Implement emulated scheduling with work queues
• ☐ Add async notification support via eventfd
• ☐ Complete migration state for WPR regions
• ☐ Performance optimization (batch RPC emulation)
• ☐ Conformance tests vs GRID GSP behavior

Related Documentation

• BAR Emulation Specification
• WPR Management
• RPC Message Format
• vGPU-IO Integration Guide