GSP RPC Emulation

GSP vGPU RPC Emulation

Historical document — superseded. This page describes an earlier emulation-mode design and is retained for reference only. The project does not use NVIDIA's proprietary vGPU stack. Canonical documentation: ../docs/03-architecture/03-gsp-and-gsp-rm.md and ../docs/03-architecture/06-host-module-and-hybrid-model.md.

Overview

Open-IOV includes emulation mode that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.

Problem

The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:

Handler	Address	Size	Purpose
BAR Trap	0x04CB8188	40KB	BAR1/ForceBarAccess/Aperture
WPR Handler 1	0x04F00B0C	62KB	Write/Read Protect Regions
WPR Handler 2	0x04F00C78	62KB	Write/Read Protect Regions
Migration	0x04179ED8/0x04179FBC	-	Live migration

Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).

Solution: Emulation Mode

Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:

┌─────────────────────────────────────────────────────────────────┐
│                    Emulation Architecture                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Userspace App (libiov.so)                                       │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────┐     ┌─────────────────┐                   │
│  │ Kernel IOCTL    │────▶│ Emulation Layer │                   │
│  │ Handler         │     │ (kernel)        │                   │
│  └─────────────────┘     └────────┬────────┘                   │
│                                    │                            │
│                    ┌───────────────┼───────────────┐           │
│                    ▼               ▼               ▼           │
│             ┌──────────┐    ┌──────────┐    ┌──────────┐      │
│             │ BAR Emu  │    │ WPR Emu  │    │ Mem Emu  │      │
│             │ (kernel) │    │ (kernel) │    │ (kernel) │      │
│             └──────────┘    └──────────┘    └──────────┘      │
│                    │               │               │           │
│                    ▼               ▼               ▼           │
│             ┌──────────────────────────────────────────┐       │
│             │         Userspace Emulator (libiov.so)   │       │
│             │  rpc_emu_bar_*(), rpc_emu_wpr_*()        │       │
│             └──────────────────────────────────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Auto-Detection (Default Behavior)

Emulation mode is ON by default and auto-detects at driver load:

1. Driver initializes RPC communication with GSP
2. Sends test BAR_TRAP RPC (non-destructive READ)
3. If GSP responds (ret=0) → vGPU handlers present → emulation disabled
4. If GSP times out/errors (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → emulation enabled

dmesg output:
  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
  # or
  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode

Module Parameters

Parameter	Type	Default	Description
emulation_mode	bool	1 (ON)	Current state (read-only, shows auto-detected value)
emulation_override	bool	0 (auto)	Override: 1=force emulation ON, 0=force emulation OFF

# Default: auto-detect
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# Force emulation ON (testing with GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1

# Force emulation OFF (requires GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0

Emulated RPC Handlers

Kernel Emulation (driver/)

RPC Type	Handler	Emulation
GSP_RPC_ALLOC (0x01)	gsp_vf_mem_alloc()	Synthetic handles/VAs from local counter
GSP_RPC_PIN (0x05)	gsp_vf_mem_pin()	Synthetic DMA addresses from local counter
GSP_RPC_MAP (0x03)	gsp_vf_mem_map()	Identity map (return requested VA)
GSP_RPC_BAR_TRAP (0x0A)	gsp_bar_trap_handler()	Local aperture/force_access state
GSP_RPC_MAP (WPR)	gsp_vf_wpr_protect()	Local WPR region list
GSP_RPC_UNMAP (WPR)	gsp_vf_wpr_unprotect()	Local WPR region removal

Userspace Emulation (userspace/lib/rpc_emulator.c)

Function	Purpose
rpc_emu_bar_config()	Local aperture offset/size + backing store
rpc_emu_bar_force_access()	Local force_access state
rpc_emu_bar_save/restore()	Migration state serialization
rpc_emu_wpr_protect_write/read()	Local WPR region tracking
rpc_emu_wpr_unprotect()	Local WPR region removal
rpc_emu_wpr_set_timeout()	Local timeout tracking
rpc_emu_wpr_check_access()	Access violation detection

Emulation Data Structures

BAR State (per VF)

struct gsp_bar_state {
    uint64_t aperture_offset;   // BAR1 aperture base
    uint64_t aperture_size;     // BAR1 aperture size (max 256KB)
    bool force_access;          // ForceBarAccess enabled
    uint32_t force_vf;          // VF holding force access
    void *backing_store;        // Emulated BAR memory
    size_t backing_size;        // Backing store size
};

WPR Region List (per VF)

struct gsp_wpr_region {
    struct list_head list;
    uint64_t base;              // Region base address
    uint64_t size;              // Region size
    uint32_t flags;             // GSP_WPR_WRITE/READ/EXECUTE
    uint32_t vf_id;             // Owning VF
    void (*completion_cb)(...); // Write/read complete callback
    void (*timeout_cb)(...);    // Timeout callback
    uint64_t timeout_ns;        // Timeout in nanoseconds
};

Userspace Emulator State

struct emu_bar_state {
    uint64_t aperture_offset;
    uint64_t aperture_size;
    bool force_access;
    uint32_t force_vf;
    void *backing_store;
    size_t backing_size;
    pthread_mutex_t lock;
};

struct emu_wpr_state {
    struct emu_wpr_region *regions;  // Linked list
    pthread_mutex_t lock;
};

Fallback Logic

Userspace library (libiov.so) implements fallback:

vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
    // 1. Try kernel IOCTL (sends RPC to GSP)
    ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
    
    // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
    if (ret < 0 && dev->emulation_enabled && 
        (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
        return rpc_emu_bar_config(vf->id, offset, size);
    }
    
    return map_errno(errno);
}

Limitations

Feature	Emulation Status	Notes
BAR Aperture Config	✅ Full	Local state + backing store
ForceBarAccess	✅ Full	Per-VF exclusive access
BAR Save/Restore	✅ Full	Migration support
WPR Write Protect	✅ Full	Region tracking + access check
WPR Read Protect	✅ Full	Region tracking + access check
WPR Timeout	✅ Partial	Timeout stored, callbacks not wired
Memory Alloc	✅ Basic	Synthetic handles/VAs
Memory Pin	✅ Basic	Synthetic DMA addresses
Memory Map	✅ Basic	Identity mapping
Migration	⚠️ Partial	BAR/WPR state only
Scheduling	❌ No	Returns success, no actual scheduling
Notifications	❌ No	No async event delivery

Testing Emulation

# 1. Load driver (auto-enables emulation on consumer GSP)
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# 2. Verify emulation enabled in dmesg
dmesg -T | grep -i emulation
# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode

# 3. Run example - should complete without timeouts
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu

# 4. Check debugfs for emulated state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions

Future Work

• [ ] Wire WPR timeout callbacks to kernel timers
• [ ] Implement emulated scheduling with work queues
• [ ] Add async notification support via eventfd
• [ ] Complete migration state for WPR regions
• [ ] Performance optimization (batch RPC emulation)
• [ ] Conformance tests vs GRID GSP behavior

Related Documentation

• BAR Emulation Specification
• WPR Management
• RPC Message Format
• Integration Guide