GSP RPC Emulation

From Open-IOV
Jump to navigation Jump to search

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

GSP vGPU RPC Emulation

Historical document — superseded. This page describes an earlier emulation-mode design and is retained for reference only. The project does not use NVIDIA's proprietary vGPU stack. Canonical documentation: ../docs/03-architecture/03-gsp-and-gsp-rm.md and ../docs/03-architecture/06-host-module-and-hybrid-model.md.

Overview

Open-IOV includes emulation mode that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.

Problem

The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:

Handler Address Size Purpose
BAR Trap 0x04CB8188 40KB BAR1/ForceBarAccess/Aperture
WPR Handler 1 0x04F00B0C 62KB Write/Read Protect Regions
WPR Handler 2 0x04F00C78 62KB Write/Read Protect Regions
Migration 0x04179ED8/0x04179FBC - Live migration

Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).

Solution: Emulation Mode

Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP: 

┌─────────────────────────────────────────────────────────────────┐
│                    Emulation Architecture                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Userspace App (libiov.so)                                       │
│       │                                                          │
│       ▼                                                          │
│  ┌─────────────────┐     ┌─────────────────┐                   │
│  │ Kernel IOCTL    │────▶│ Emulation Layer │                   │
│  │ Handler         │     │ (kernel)        │                   │
│  └─────────────────┘     └────────┬────────┘                   │
│                                    │                            │
│                    ┌───────────────┼───────────────┐           │
│                    ▼               ▼               ▼           │
│             ┌──────────┐    ┌──────────┐    ┌──────────┐      │
│             │ BAR Emu  │    │ WPR Emu  │    │ Mem Emu  │      │
│             │ (kernel) │    │ (kernel) │    │ (kernel) │      │
│             └──────────┘    └──────────┘    └──────────┘      │
│                    │               │               │           │
│                    ▼               ▼               ▼           │
│             ┌──────────────────────────────────────────┐       │
│             │         Userspace Emulator (libiov.so)   │       │
│             │  rpc_emu_bar_*(), rpc_emu_wpr_*()        │       │
│             └──────────────────────────────────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Auto-Detection (Default Behavior)

Emulation mode is ON by default and auto-detects at driver load:

  1. Driver initializes RPC communication with GSP
  2. Sends test BAR_TRAP RPC (non-destructive READ)
  3. If GSP responds (ret=0) → vGPU handlers present → emulation disabled
  4. If GSP times out/errors (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → emulation enabled
dmesg output:
  gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
  gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
  # or
  gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode

Module Parameters

Parameter Type Default Description
emulation_mode bool 1 (ON) Current state (read-only, shows auto-detected value)
emulation_override bool 0 (auto) Override: 1=force emulation ON, 0=force emulation OFF 
# Default: auto-detect
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# Force emulation ON (testing with GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1

# Force emulation OFF (requires GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0

Emulated RPC Handlers

Kernel Emulation (driver/)

RPC Type Handler Emulation
GSP_RPC_ALLOC (0x01) gsp_vf_mem_alloc() Synthetic handles/VAs from local counter
GSP_RPC_PIN (0x05) gsp_vf_mem_pin() Synthetic DMA addresses from local counter
GSP_RPC_MAP (0x03) gsp_vf_mem_map() Identity map (return requested VA)
GSP_RPC_BAR_TRAP (0x0A) gsp_bar_trap_handler() Local aperture/force_access state
GSP_RPC_MAP (WPR) gsp_vf_wpr_protect() Local WPR region list
GSP_RPC_UNMAP (WPR) gsp_vf_wpr_unprotect() Local WPR region removal

Userspace Emulation (userspace/lib/rpc_emulator.c)

Function Purpose
rpc_emu_bar_config() Local aperture offset/size + backing store
rpc_emu_bar_force_access() Local force_access state
rpc_emu_bar_save/restore() Migration state serialization
rpc_emu_wpr_protect_write/read() Local WPR region tracking
rpc_emu_wpr_unprotect() Local WPR region removal
rpc_emu_wpr_set_timeout() Local timeout tracking
rpc_emu_wpr_check_access() Access violation detection

Emulation Data Structures

BAR State (per VF)

struct gsp_bar_state {
    uint64_t aperture_offset;   // BAR1 aperture base
    uint64_t aperture_size;     // BAR1 aperture size (max 256KB)
    bool force_access;          // ForceBarAccess enabled
    uint32_t force_vf;          // VF holding force access
    void *backing_store;        // Emulated BAR memory
    size_t backing_size;        // Backing store size
};

WPR Region List (per VF)

struct gsp_wpr_region {
    struct list_head list;
    uint64_t base;              // Region base address
    uint64_t size;              // Region size
    uint32_t flags;             // GSP_WPR_WRITE/READ/EXECUTE
    uint32_t vf_id;             // Owning VF
    void (*completion_cb)(...); // Write/read complete callback
    void (*timeout_cb)(...);    // Timeout callback
    uint64_t timeout_ns;        // Timeout in nanoseconds
};

Userspace Emulator State

struct emu_bar_state {
    uint64_t aperture_offset;
    uint64_t aperture_size;
    bool force_access;
    uint32_t force_vf;
    void *backing_store;
    size_t backing_size;
    pthread_mutex_t lock;
};

struct emu_wpr_state {
    struct emu_wpr_region *regions;  // Linked list
    pthread_mutex_t lock;
};

Fallback Logic

Userspace library (libiov.so) implements fallback: 

vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
    // 1. Try kernel IOCTL (sends RPC to GSP)
    ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);
    
    // 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
    if (ret < 0 && dev->emulation_enabled && 
        (errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
        return rpc_emu_bar_config(vf->id, offset, size);
    }
    
    return map_errno(errno);
}

Limitations

Feature Emulation Status Notes
BAR Aperture Config ✅ Full Local state + backing store
ForceBarAccess ✅ Full Per-VF exclusive access
BAR Save/Restore ✅ Full Migration support
WPR Write Protect ✅ Full Region tracking + access check
WPR Read Protect ✅ Full Region tracking + access check
WPR Timeout ✅ Partial Timeout stored, callbacks not wired
Memory Alloc ✅ Basic Synthetic handles/VAs
Memory Pin ✅ Basic Synthetic DMA addresses
Memory Map ✅ Basic Identity mapping
Migration ⚠️ Partial BAR/WPR state only
Scheduling ❌ No Returns success, no actual scheduling
Notifications ❌ No No async event delivery

Testing Emulation

# 1. Load driver (auto-enables emulation on consumer GSP)
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# 2. Verify emulation enabled in dmesg
dmesg -T | grep -i emulation
# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode

# 3. Run example - should complete without timeouts
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu

# 4. Check debugfs for emulated state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions

Future Work

  • [ ] Wire WPR timeout callbacks to kernel timers
  • [ ] Implement emulated scheduling with work queues
  • [ ] Add async notification support via eventfd
  • [ ] Complete migration state for WPR regions
  • [ ] Performance optimization (batch RPC emulation)
  • [ ] Conformance tests vs GRID GSP behavior

Related Documentation

Retrieved from "https://open-iov.org/index.php?title=GSP_RPC_Emulation&oldid=23971"