GSP RPC Emulation
Jump to navigation
Jump to search
GSP vGPU RPC Emulation
Historical document — superseded. This page describes an earlier emulation-mode design and is retained for reference only. The project does not use NVIDIA's proprietary vGPU stack. Canonical documentation:
../docs/03-architecture/03-gsp-and-gsp-rm.mdand../docs/03-architecture/06-host-module-and-hybrid-model.md.
Overview
Open-IOV includes emulation mode that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.
Problem
The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:
| Handler | Address | Size | Purpose |
|---|---|---|---|
| BAR Trap | 0x04CB8188 | 40KB | BAR1/ForceBarAccess/Aperture |
| WPR Handler 1 | 0x04F00B0C | 62KB | Write/Read Protect Regions |
| WPR Handler 2 | 0x04F00C78 | 62KB | Write/Read Protect Regions |
| Migration | 0x04179ED8/0x04179FBC | - | Live migration |
Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).
Solution: Emulation Mode
Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:
┌─────────────────────────────────────────────────────────────────┐ │ Emulation Architecture │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Userspace App (libiov.so) │ │ │ │ │ ▼ │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Kernel IOCTL │────▶│ Emulation Layer │ │ │ │ Handler │ │ (kernel) │ │ │ └─────────────────┘ └────────┬────────┘ │ │ │ │ │ ┌───────────────┼───────────────┐ │ │ ▼ ▼ ▼ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ BAR Emu │ │ WPR Emu │ │ Mem Emu │ │ │ │ (kernel) │ │ (kernel) │ │ (kernel) │ │ │ └──────────┘ └──────────┘ └──────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌──────────────────────────────────────────┐ │ │ │ Userspace Emulator (libiov.so) │ │ │ │ rpc_emu_bar_*(), rpc_emu_wpr_*() │ │ │ └──────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘
Auto-Detection (Default Behavior)
Emulation mode is ON by default and auto-detects at driver load:
- Driver initializes RPC communication with GSP
- Sends test
BAR_TRAPRPC (non-destructive READ) - If GSP responds (ret=0) → vGPU handlers present → emulation disabled
- If GSP times out/errors (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → emulation enabled
dmesg output: gsp-vgpu: Auto-detecting GSP vGPU RPC handler support... gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode # or gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode
Module Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
emulation_mode
|
bool | 1 (ON) | Current state (read-only, shows auto-detected value) |
emulation_override
|
bool | 0 (auto) | Override: 1=force emulation ON, 0=force emulation OFF |
# Default: auto-detect sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin # Force emulation ON (testing with GRID GSP) sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1 # Force emulation OFF (requires GRID GSP) sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0
Emulated RPC Handlers
Kernel Emulation (driver/)
| RPC Type | Handler | Emulation |
|---|---|---|
GSP_RPC_ALLOC (0x01)
|
gsp_vf_mem_alloc()
|
Synthetic handles/VAs from local counter |
GSP_RPC_PIN (0x05)
|
gsp_vf_mem_pin()
|
Synthetic DMA addresses from local counter |
GSP_RPC_MAP (0x03)
|
gsp_vf_mem_map()
|
Identity map (return requested VA) |
GSP_RPC_BAR_TRAP (0x0A)
|
gsp_bar_trap_handler()
|
Local aperture/force_access state |
GSP_RPC_MAP (WPR)
|
gsp_vf_wpr_protect()
|
Local WPR region list |
GSP_RPC_UNMAP (WPR)
|
gsp_vf_wpr_unprotect()
|
Local WPR region removal |
Userspace Emulation (userspace/lib/rpc_emulator.c)
| Function | Purpose |
|---|---|
rpc_emu_bar_config()
|
Local aperture offset/size + backing store |
rpc_emu_bar_force_access()
|
Local force_access state |
rpc_emu_bar_save/restore()
|
Migration state serialization |
rpc_emu_wpr_protect_write/read()
|
Local WPR region tracking |
rpc_emu_wpr_unprotect()
|
Local WPR region removal |
rpc_emu_wpr_set_timeout()
|
Local timeout tracking |
rpc_emu_wpr_check_access()
|
Access violation detection |
Emulation Data Structures
BAR State (per VF)
struct gsp_bar_state {
uint64_t aperture_offset; // BAR1 aperture base
uint64_t aperture_size; // BAR1 aperture size (max 256KB)
bool force_access; // ForceBarAccess enabled
uint32_t force_vf; // VF holding force access
void *backing_store; // Emulated BAR memory
size_t backing_size; // Backing store size
};
WPR Region List (per VF)
struct gsp_wpr_region {
struct list_head list;
uint64_t base; // Region base address
uint64_t size; // Region size
uint32_t flags; // GSP_WPR_WRITE/READ/EXECUTE
uint32_t vf_id; // Owning VF
void (*completion_cb)(...); // Write/read complete callback
void (*timeout_cb)(...); // Timeout callback
uint64_t timeout_ns; // Timeout in nanoseconds
};
Userspace Emulator State
struct emu_bar_state {
uint64_t aperture_offset;
uint64_t aperture_size;
bool force_access;
uint32_t force_vf;
void *backing_store;
size_t backing_size;
pthread_mutex_t lock;
};
struct emu_wpr_state {
struct emu_wpr_region *regions; // Linked list
pthread_mutex_t lock;
};
Fallback Logic
Userspace library (libiov.so) implements fallback:
vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
// 1. Try kernel IOCTL (sends RPC to GSP)
ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
// 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
if (ret < 0 && dev->emulation_enabled &&
(errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
return rpc_emu_bar_config(vf->id, offset, size);
}
return map_errno(errno);
}
Limitations
| Feature | Emulation Status | Notes |
|---|---|---|
| BAR Aperture Config | ✅ Full | Local state + backing store |
| ForceBarAccess | ✅ Full | Per-VF exclusive access |
| BAR Save/Restore | ✅ Full | Migration support |
| WPR Write Protect | ✅ Full | Region tracking + access check |
| WPR Read Protect | ✅ Full | Region tracking + access check |
| WPR Timeout | ✅ Partial | Timeout stored, callbacks not wired |
| Memory Alloc | ✅ Basic | Synthetic handles/VAs |
| Memory Pin | ✅ Basic | Synthetic DMA addresses |
| Memory Map | ✅ Basic | Identity mapping |
| Migration | ⚠️ Partial | BAR/WPR state only |
| Scheduling | ❌ No | Returns success, no actual scheduling |
| Notifications | ❌ No | No async event delivery |
Testing Emulation
# 1. Load driver (auto-enables emulation on consumer GSP) sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin # 2. Verify emulation enabled in dmesg dmesg -T | grep -i emulation # gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode # 3. Run example - should complete without timeouts sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu # 4. Check debugfs for emulated state cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions
Future Work
- [ ] Wire WPR timeout callbacks to kernel timers
- [ ] Implement emulated scheduling with work queues
- [ ] Add async notification support via eventfd
- [ ] Complete migration state for WPR regions
- [ ] Performance optimization (batch RPC emulation)
- [ ] Conformance tests vs GRID GSP behavior