Open-IOV - User contributions [en]

GPU Software Bill Of Materials (SBOM)

2026-10-12T04:41:23Z

WikiBot: Fix repository links to the cgit tree URLs

{{About message}}

{{Superseded}}
= GPU Software Bill Of Materials (SBOM) =

<blockquote>'''This page is superseded.''' The previous content was an imported 2023 wiki survey of third-party GPU-virtualization components. The complete, repository-scoped SBOM now lives at:</blockquote>

* '''[https://git.vgpu.io/open-iov.git/tree/SBOM.md <code>SBOM.md</code>]''' — human-readable SBOM: primary component, first-party components, bundled third-party components with SHA-256 hashes, archived-documentation provenance, external build/run dependencies, dependency graph, license/redistribution terms, and the NTIA minimum-elements checklist.
* '''[https://git.vgpu.io/open-iov.git/tree/sbom/open-iov-0.1.0.cdx.json <code>sbom/open-iov-0.1.0.cdx.json</code>]''' — machine-readable CycloneDX 1.6 SBOM (serial <code>urn:uuid:e626c5a1-9697-4546-bb2b-5b4feb58e922</code>).

The SBOM covers every file tracked in this repository plus the external components required to build, run and support it. NVIDIA drivers, GSP firmware and CUDA packages are documented but '''not distributed''' here.

VGPU-IO Integration Guide

2026-10-12T04:41:03Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= vGPU-IO Integration Guide =

<blockquote>'''Historical document — superseded.''' This earlier integration design predates the current stack and is retained for reference only. Canonical documentation: getting started [https://git.vgpu.io/open-iov.git/tree/docs/02-getting-started <code>../docs/02-getting-started/</code>], host module [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>], developer standards [https://git.vgpu.io/open-iov.git/tree/docs/07-developers <code>../docs/07-developers/</code>].</blockquote>

== Overview ==

This guide covers implementing a Linux kernel driver for vGPU-IO using the GSP 570.144W firmware RPC interface.

== Architecture ==

<pre>┌─────────────────────────────────────────────────────────────┐
│ vGPU-IO Kernel Driver │
├─────────────────────────────────────────────────────────────┤
│ gsp_core.c │ gsp_rpc.c │ gsp_vf.c │
│ - Device probe │ - RPC client │ - VF lifecycle │
│ - BAR mapping │ - Doorbell │ - Migration │
│ - IRQ/MSI-X │ - Completion │ - WPR/BAR integration │
├─────────────────────────────────────────────────────────────┤
│ GSP Firmware (570.144W) │
│ Primary Dispatcher (0x057A1AC0) │ Secondary (0x04ED39DC) │
│ BAR Handler (0x04CB8188) │ WPR Handlers │
│ Migration (0x04179ED8/0x04179FBC)│ SCHED/ALLOC/PIN │
└─────────────────────────────────────────────────────────────┘</pre>

== Module Structure ==

<pre>drivers/gpu/nvidia/gsp/
├── gsp.h # Core definitions
├── gsp_rpc.h # RPC message formats
├── gsp_vf.h # VF definitions
├── gsp_bar.h # BAR emulation
├── gsp_wpr.h # WPR management
├── gsp_migration.h # Migration protocol
├── gsp_core.c # Probe/remove, BAR/IRQ
├── gsp_rpc.c # RPC client implementation
├── gsp_vf.c # VF create/destroy
├── gsp_bar.c # BAR trap handler
├── gsp_wpr.c # WPR protect/unprotect
├── gsp_migration.c # Live migration
└── gsp_debugfs.c # Debug interface</pre>

== Core Data Structures ==

=== GSP Device ===

<pre>struct gsp_device {
struct pci_dev *pdev;
void __iomem *bar0; // PCIe BAR0 mapping
void __iomem *bar1; // PCIe BAR1 (GSP FW)
struct gsp_firmware *fw; // Firmware image
struct gsp_rpc_client *rpc; // RPC client
struct mutex vf_lock;
struct idr vf_idr; // VF ID allocator
struct list_head vf_list;
struct gsp_debugfs *debugfs;
spinlock_t doorbell_lock;
uint32_t doorbell_seq;
};</pre>

=== VF (Virtual Function) ===

<pre>struct gsp_vf {
struct gsp_device *gsp;
uint32_t id; // VF ID (1-7)
struct kref ref;
struct mutex lock;
mig_state_t mig_state;
struct gsp_migration *migration;
uint64_t bar1_offset; // BAR1 aperture offset
uint64_t bar1_size; // BAR1 aperture size
uint64_t pt_base; // Page table base
uint64_t pt_size; // Page table size
uint64_t cmd_base; // Command buffer base
uint64_t cmd_size; // Command buffer size
struct gsp_wpr_region *wpr_regions;
struct list_head wpr_list;
struct gsp_bar_state bar_state;
atomic_t refcount;
bool active;
};</pre>

== Initialization Sequence ==

=== 1. PCI Probe ===

<pre>static int gsp_probe(struct pci_dev *pdev, const struct pci_device_id *id) {
struct gsp_device *gsp;
int ret;

gsp = devm_kzalloc(&pdev->dev, sizeof(*gsp), GFP_KERNEL);
gsp->pdev = pdev;
pci_set_drvdata(pdev, gsp);

// Enable PCI device
ret = pcim_enable_device(pdev);
if (ret) return ret;

// Map BARs
gsp->bar0 = pcim_iomap(pdev, 0, 0);
gsp->bar1 = pcim_iomap(pdev, 1, 0);
if (!gsp->bar0 || !gsp->bar1) return -ENOMEM;

// Load firmware
ret = gsp_firmware_load(gsp, "gsp-570.144.fwimage.bin");
if (ret) return ret;

// Initialize RPC client
gsp->rpc = gsp_rpc_client_init(gsp);
if (IS_ERR(gsp->rpc)) return PTR_ERR(gsp->rpc);

// Setup IRQ/MSI-X
ret = gsp_setup_irq(gsp);
if (ret) return ret;

// Initialize debugfs
gsp_debugfs_init(gsp);

return 0;
}</pre>

=== 2. Firmware Loading ===

<pre>int gsp_firmware_load(struct gsp_device *gsp, const char *name) {
const struct firmware *fw;
int ret;

ret = request_firmware(&fw, name, &gsp->pdev->dev);
if (ret) return ret;

// Verify MD5: 13ce12a392a31eeb79a2e6ebd3a4a93a
if (!gsp_firmware_verify(fw->data, fw->size)) {
release_firmware(fw);
return -EINVAL;
}

// Load to GSP memory (via BAR1)
ret = gsp_firmware_upload(gsp, fw->data, fw->size);
release_firmware(fw);

return ret;
}</pre>

=== 3. GSP Boot ===

<pre>int gsp_boot(struct gsp_device *gsp) {
// 1. Reset GSP
writel(1, gsp->bar0 + GSP_RESET_REG);
msleep(10);
writel(0, gsp->bar0 + GSP_RESET_REG);

// 2. Wait for entry point (0x04009000) to signal ready
ret = gsp_wait_for_ready(gsp, 5000); // 5s timeout
if (ret) return ret;

// 3. Initialize RPC dispatchers
ret = gsp_rpc_init_dispatchers(gsp);
if (ret) return ret;

return 0;
}</pre>

== RPC Client Implementation ==

=== Doorbell Mechanism ===

<pre>#define GSP_DOORBELL_OFFSET 0x1000
#define GSP_MSG_RING_SIZE 256
#define GSP_MSG_MAX_SIZE 256

struct gsp_rpc_client {
struct gsp_device *gsp;
struct mutex lock;
uint32_t seq;
wait_queue_head_t wq;
struct gsp_msg_ring *ring;
struct completion *pending[GSP_MAX_PENDING];
};

static int gsp_rpc_send_sync(struct gsp_rpc_client *rpc,
uint32_t rpc_type,
void *payload, size_t payload_size,
void *response, size_t *response_size) {
struct gsp_msg msg = {
.hdr = {
.magic = 0x47535052,
.version = 1,
.rpc_type = rpc_type,
.sequence = atomic_inc_return(&rpc->seq),
.timestamp_ns = ktime_get_ns(),
.payload_size = payload_size,
.flags = MSG_SYNC,
}
};

struct completion done;
init_completion(&done);

mutex_lock(&rpc->lock);
rpc->pending[msg.hdr.sequence % GSP_MAX_PENDING] = &done;

// Copy message to ring buffer
memcpy(msg.hdr.payload, payload, payload_size);
gsp_ring_write(rpc->ring, &msg, GSP_MSG_MAX_SIZE);

// Ring doorbell
writel(1, rpc->gsp->bar0 + GSP_DOORBELL_OFFSET);

mutex_unlock(&rpc->lock);

// Wait for completion
if (!wait_for_completion_timeout(&done, msecs_to_jiffies(5000)))
return -ETIMEDOUT;

// Copy response
if (response && response_size) {
*response_size = min(*response_size, done.payload_size);
memcpy(response, done.payload, *response_size);
}

return done.status;
}</pre>

=== Completion Handler (ISR) ===

<pre>static irqreturn_t gsp_isr(int irq, void *data) {
struct gsp_device *gsp = data;
struct gsp_msg msg;

while (gsp_ring_read(gsp->rpc->ring, &msg)) {
if (msg.hdr.flags & MSG_SYNC) {
struct completion *done =
gsp->rpc->pending[msg.hdr.sequence % GSP_MAX_PENDING];
if (done) {
done->status = msg.payload.status;
done->payload_size = msg.hdr.payload_size;
memcpy(done->payload, msg.payload.data, msg.hdr.payload_size);
complete(done);
}
} else {
// Async notification
gsp_handle_notification(gsp, &msg);
}
}

return IRQ_HANDLED;
}</pre>

== VF Lifecycle ==

=== Create VF ===

<pre>int gsp_vf_create(struct gsp_device *gsp, uint32_t vf_id,
const struct gsp_vf_config *config) {
struct gsp_vf *vf;
int ret;

mutex_lock(&gsp->vf_lock);

if (idr_alloc(&gsp->vf_idr, NULL, vf_id, vf_id + 1, GFP_KERNEL) < 0) {
ret = -EBUSY;
goto unlock;
}

vf = kzalloc(sizeof(*vf), GFP_KERNEL);
vf->gsp = gsp;
vf->id = vf_id;
kref_init(&vf->ref);
mutex_init(&vf->lock);
INIT_LIST_HEAD(&vf->wpr_list);

// Configure VF resources
vf->bar1_offset = config->bar1_offset;
vf->bar1_size = config->bar1_size;
vf->pt_base = config->pt_base;
vf->pt_size = config->pt_size;
vf->cmd_base = config->cmd_base;
vf->cmd_size = config->cmd_size;

// Initialize WPR for VF
ret = gsp_vf_wpr_init(vf);
if (ret) goto err_wpr;

// Initialize BAR for VF
ret = gsp_vf_bar_init(vf);
if (ret) goto err_bar;

list_add(&vf->list, &gsp->vf_list);
vf->active = true;

mutex_unlock(&gsp->vf_lock);
return 0;

err_bar:
gsp_vf_wpr_fini(vf);
err_wpr:
idr_remove(&gsp->vf_idr, vf_id);
kfree(vf);
unlock:
mutex_unlock(&gsp->vf_lock);
return ret;
}</pre>

=== Destroy VF ===

<pre>void gsp_vf_destroy(struct gsp_vf *vf) {
struct gsp_device *gsp = vf->gsp;
uint32_t id = vf->id;

mutex_lock(&gsp->vf_lock);
vf->active = false;
list_del(&vf->list);
mutex_unlock(&gsp->vf_lock);

gsp_vf_bar_fini(vf);
gsp_vf_wpr_fini(vf);

if (vf->migration)
gsp_migration_cancel(vf->migration);

kref_put(&vf->ref, gsp_vf_release);
}

static void gsp_vf_release(struct kref *ref) {
struct gsp_vf *vf = container_of(ref, struct gsp_vf, ref);
kfree(vf);
}</pre>

== Memory Management (ALLOC/PIN) ==

=== Allocate VF Memory ===

<pre>int gsp_vf_mem_alloc(struct gsp_vf *vf,
uint64_t size, uint32_t flags,
uint64_t *handle, uint64_t *va) {
struct alloc_payload payload = {
.size = size,
.type = MEM_TYPE_VRAM,
.alignment = 0x1000,
.va_hint = 0,
.flags = flags,
.pid = current->pid,
.vf_id = vf->id,
};

struct alloc_response resp;
size_t resp_size = sizeof(resp);

return gsp_rpc_send_sync(vf->gsp->rpc, RPC_ALLOC,
&payload, sizeof(payload),
&resp, &resp_size);
}</pre>

=== Pin Memory for DMA ===

<pre>int gsp_vf_mem_pin(struct gsp_vf *vf, uint64_t handle,
uint64_t offset, uint64_t size,
uint64_t *dma_addr) {
struct pin_payload payload = {
.handle = handle,
.offset = offset,
.size = size,
.vf_id = vf->id,
.flags = PIN_WRITE | PIN_READ,
};

struct pin_response resp;
size_t resp_size = sizeof(resp);

return gsp_rpc_send_sync(vf->gsp->rpc, RPC_PIN,
&payload, sizeof(payload),
&resp, &resp_size);
}</pre>

== Debugging & Testing ==

=== Debugfs Interface ===

<pre># List VFs
cat /sys/kernel/debug/gsp/vfs

# VF details
cat /sys/kernel/debug/gsp/vf/1/status

# RPC statistics
cat /sys/kernel/debug/gsp/rpc/stats

# BAR state
cat /sys/kernel/debug/gsp/vf/1/bar

# WPR regions
cat /sys/kernel/debug/gsp/vf/1/wpr

# Migration status
cat /sys/kernel/debug/gsp/vf/1/migration</pre>

=== Trace Events ===

<pre># Enable tracing
echo 1 > /sys/kernel/debug/tracing/events/gsp/enable

# Trace migration
trace-cmd record -e gsp_mig_phase -e gsp_mig_transfer

# Trace WPR
trace-cmd record -e gsp_wpr_event</pre>

=== Unit Tests ===

<pre>// kselftest for gsp-rpc
static void test_rpc_alloc(void) {
struct gsp_vf *vf = create_test_vf();
uint64_t handle, va;

ASSERT_EQ(gsp_vf_mem_alloc(vf, 4096, 0, &handle, &va), 0);
ASSERT_NE(handle, 0);
ASSERT_NE(va, 0);

gsp_vf_mem_free(vf, handle);
destroy_test_vf(vf);
}</pre>

== Error Handling Checklist ==

{| class="wikitable"
|-
! Scenario
! Detection
! Recovery
|-
| GSP firmware crash
| Watchdog timeout
| Reset GSP, reload FW
|-
| RPC timeout
| Doorbell no response
| Retry, then VF reset
|-
| BAR trap storm
| Rate limit exceeded
| Throttle VF
|-
| WPR violation
| Callback storm
| Quarantine VF
|-
| Migration fail
| Commit verify fail
| Rollback to source
|}

== Performance Tuning ==

=== RPC Batching ===

<pre>// Batch multiple ALLOCs in single RPC
struct batch_alloc_payload {
uint32_t count;
struct alloc_payload allocs[MAX_BATCH];
};</pre>

=== Async Completion ===

<pre>// Use async RPC for non-critical paths
gsp_rpc_send_async(rpc, RPC_NOTIFY, payload, size, callback, ctx);</pre>

=== Memory Pool ===

<pre>// Pre-allocate bounce buffers
struct gsp_bounce_pool {
struct page **pages;
int count;
spinlock_t lock;
};</pre>

----

== Reference ==

* GSP Firmware: gsp-570.144.fwimage.bin
* Base Address: 0x04000000
* Primary Dispatcher: 0x057A1AC0
* Secondary Dispatcher: 0x04ED39DC

WPR Management

2026-10-12T04:41:02Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= WPR (Write/Read Protect Region) Management =

<blockquote>'''Historical document — superseded.''' WPR is the '''write-protected region of VRAM reserved for firmware''', not a generic read/write-protect mechanism; the read-protect framing below is historical reverse-engineering material. Canonical documentation: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

WPR (Write Protect Region / Read Protect Region) provides memory isolation for vGPU by protecting guest memory regions from unauthorized access. The GSP firmware implements WPR through two handlers at '''0x04F00B0C''' and '''0x04F00C78''' (62 KB each).

== Key Addresses ==

{| class="wikitable"
|-
! Handler
! Address
! Size
! Role
|-
| WPR Handler 1
| 0x04F00B0C
| 62 KB
| Primary WPR management
|-
| WPR Handler 2
| 0x04F00C78
| 62 KB
| Secondary/timeout handling
|-
| String Table
| 60 locations
| —
| Completion/timeout strings
|}

== String References (60 Total) ==

=== Completion Events ===

<pre>WPR_WRITE_COMPLETE 0x5E478F8
WPR_READ_COMPLETE 0x5E478E0
NV_WPR_WRITE_COMPLETE 0x5E478C8
NV_WPR_READ_COMPLETE 0x5E478B0</pre>

=== Timeout Events ===

<pre>WPR_WRITE_TIMEOUT 0x5E47898
WPR_READ_TIMEOUT 0x5E47880
NV_WPR_WRITE_TIMEOUT 0x5E47868
NV_WPR_READ_TIMEOUT 0x5E47850</pre>

== WPR State Machine ==

=== Handler 1: 0x04F00B0C (Primary) ===

<pre>typedef enum {
WPR_STATE_IDLE = 0,
WPR_STATE_WRITE_PENDING = 1,
WPR_STATE_READ_PENDING = 2,
WPR_STATE_COMPLETE = 3,
WPR_STATE_TIMEOUT = 4,
WPR_STATE_ERROR = 0xFFFF
} wpr_state_t;

typedef struct {
uint64_t region_base; // Protected region base
uint64_t region_size; // Region size (page-aligned)
uint32_t vf_id; // Owning VF
uint32_t flags; // WPR_WRITE | WPR_READ | WPR_EXECUTE
wpr_state_t state;
uint64_t timeout_ns; // Operation timeout
void *completion_cb; // Callback on complete
void *timeout_cb; // Callback on timeout
} wpr_region_t;</pre>

=== Operation Flow ===

==== Write Protect ====

<pre>1. VF requests WPR_WRITE on region [base, base+size)
2. Handler validates alignment (4KB) and permissions
3. Programs MMU/IOMMU write-protect bits
4. Starts timeout timer (configurable)
5. On GPU write attempt:
a. Trap to GSP via page fault
b. Check if VF owns region
c. If allowed: emulate write, return OK
d. If denied: trigger WPR_WRITE_COMPLETE/NV_WPR_WRITE_COMPLETE
6. On timeout: trigger WPR_WRITE_TIMEOUT/NV_WPR_WRITE_TIMEOUT</pre>

==== Read Protect ====

<pre>1. VF requests WPR_READ on region
2. Programs MMU/IOMMU read-protect bits
3. On GPU read attempt:
a. Trap to GSP
b. Return zero/poison data or emulate
c. Trigger WPR_READ_COMPLETE/NV_WPR_READ_COMPLETE
4. On timeout: trigger WPR_READ_TIMEOUT/NV_WPR_READ_TIMEOUT</pre>

== VBIOS Parsing (Handler 2: 0x04F00C78) ==

The WPR handlers parse VBIOS structures for legacy compatibility:

<pre>int parse_vbios_for_wpr(void *vbios_ptr, size_t size) {
// Search for signatures:
// 0xAA55 - Standard VBIOS signature
// 0x57464F4E ("WFON") - WPR enable
// 0x5746564E ("WFVN") - WPR version

for (offset = 0; offset < size; offset += 0x1000) {
uint32_t sig = *(uint32_t*)(vbios_ptr + offset);

if (sig == 0xAA55) {
// Check for WPR structures
parse_wpr_tables(vbios_ptr + offset);
}
else if (sig == 0x57464F4E || sig == 0x5746564E) {
// WPR enabled in VBIOS
wpr_vbios_enabled = true;
wpr_vbios_offset = offset;
}
}
}</pre>

== Integration with GSP Subsystems ==

=== Memory Allocator (0x429B450) ===

<pre>// Allocate WPR metadata structures
wpr_region_t *wpr_alloc_region() {
return UNK_ram_058A1A9C(0x429B450, sizeof(wpr_region_t));
}

// Allocate bounce buffers for emulation
void *wpr_alloc_bounce(size_t size) {
return UNK_ram_058A1A9C(0x429B450, size);
}</pre>

=== DMA/Copy Engine (0x0588F9C4, 0x05AEC86C) ===

<pre>// Copy data for WPR emulation
int wpr_emulate_access(wpr_region_t *rgn, void *dst, void *src, size_t size) {
return UNK_ram_0588F9C4(dma_engine, src, dst, size);
}</pre>

=== Logging (0x05AB7DA0) ===

<pre>// Log WPR events
void wpr_log_event(const char *event_str, uint32_t vf_id, uint64_t addr) {
UNK_ram_05AB7DA0(1, &event_str); // Log string
UNK_ram_05AB7DA0(2, &vf_id); // Log VF ID
UNK_ram_05AB7DA0(3, &addr); // Log address
}</pre>

== vGPU-IO Integration ==

=== Kernel WPR API ===

<pre>struct gsp_wpr_ops {
int (*protect_write)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base, uint64_t size);
int (*protect_read)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base, uint64_t size);
int (*unprotect)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base);
int (*set_timeout)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t timeout_ns);
int (*register_callback)(struct gsp_device *gsp,
uint32_t vf_id,
enum wpr_event_type type,
void (*cb)(struct wpr_event *));
};

static const struct gsp_wpr_ops gsp_wpr_ops = {
.protect_write = gsp_wpr_write_protect,
.protect_read = gsp_wpr_read_protect,
.unprotect = gsp_wpr_unprotect,
.set_timeout = gsp_wpr_set_timeout,
.register_callback = gsp_wpr_register_callback,
};</pre>

=== VF WPR Lifecycle ===

<pre>// VF initialization: register WPR regions
int gsp_vf_wpr_init(struct gsp_vf *vf) {
// Protect VF's page tables
gsp_wpr_ops.protect_write(vf->gsp, vf->id,
vf->pt_base, vf->pt_size);

// Protect VF's command buffers
gsp_wpr_ops.protect_write(vf->gsp, vf->id,
vf->cmd_base, vf->cmd_size);

// Set default timeout (10ms)
gsp_wpr_ops.set_timeout(vf->gsp, vf->id, 10000000);

// Register completion callbacks
gsp_wpr_ops.register_callback(vf->gsp, vf->id,
WPR_EVENT_WRITE_COMPLETE,
vf_wpr_write_complete_cb);
gsp_wpr_ops.register_callback(vf->gsp, vf->id,
WPR_EVENT_TIMEOUT,
vf_wpr_timeout_cb);

return 0;
}

// VF cleanup
void gsp_vf_wpr_fini(struct gsp_vf *vf) {
gsp_wpr_ops.unprotect(vf->gsp, vf->id, vf->pt_base);
gsp_wpr_ops.unprotect(vf->gsp, vf->id, vf->cmd_base);
// ... unregister callbacks
}</pre>

=== Callback Handling ===

<pre>static void vf_wpr_write_complete_cb(struct wpr_event *evt) {
struct gsp_vf *vf = evt->vf;

// Log the write attempt
dev_dbg(vf->dev, "WPR write complete: VF=%u addr=0x%llx\n",
vf->id, evt->address);

// If this was a page table update, invalidate TLB
if (evt->address >= vf->pt_base &&
evt->address < vf->pt_base + vf->pt_size) {
gsp_tlb_invalidate(vf->gsp, vf->id);
}
}

static void vf_wpr_timeout_cb(struct wpr_event *evt) {
struct gsp_vf *vf = evt->vf;

dev_warn(vf->dev, "WPR timeout: VF=%u type=%d addr=0x%llx\n",
vf->id, evt->type, evt->address);

// Escalate: reset VF or throttle
if (evt->type == WPR_EVENT_WRITE_TIMEOUT) {
gsp_vf_throttle(vf);
} else {
gsp_vf_reset(vf);
}
}</pre>

== Error Codes ==

{| class="wikitable"
|-
! Code
! Name
! Description
|-
| 0x1E
| ETIME
| Operation timed out
|-
| 0x1F
| EDEV
| Device error
|-
| 0x25
| EOVERFLOW
| Region too large
|-
| 0x37
| EINVAL
| Invalid alignment
|-
| 0x51
| ENOMEM
| Out of memory
|-
| 0x56
| EBUSY
| Region busy
|-
| 0xFFFF
| EPERM
| Permission denied
|}

== Debugging ==

=== Monitor These Addresses ===

<pre>0x04296048 : Global error flag
0x0429B750+0x111300 : Error status register
0x05E47850-0x05E478F8 : WPR string table</pre>

=== GSP Log Patterns ===

<pre>WPR_WRITE_COMPLETE VF=<id> ADDR=<addr>
WPR_READ_COMPLETE VF=<id> ADDR=<addr>
WPR_WRITE_TIMEOUT VF=<id> ADDR=<addr>
WPR_READ_TIMEOUT VF=<id> ADDR=<addr>
NV_WPR_WRITE_COMPLETE VF=<id> ADDR=<addr>
NV_WPR_READ_COMPLETE VF=<id> ADDR=<addr>
NV_WPR_WRITE_TIMEOUT VF=<id> ADDR=<addr>
NV_WPR_READ_TIMEOUT VF=<id> ADDR=<addr></pre>

=== Trace Points ===

<pre>// In kernel driver
trace_gsp_wpr_protect(vf_id, base, size, type);
trace_gsp_wpr_event(vf_id, event_type, address, status);
trace_gsp_wpr_timeout(vf_id, address, elapsed_ns);</pre>

----

== Reference ==

* Handler 1: <code>FUN_ram_04f00b0c_ram_04f00b0c.c</code> (62 KB)
* Handler 2: <code>FUN_ram_04f00c78_ram_04f00c78.c</code> (62 KB)
* RPC Types: Handled via primary/secondary dispatchers

UVM in the Guest

2026-10-12T04:41:02Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= UVM in the Guest =

<blockquote>'''Scope.''' What UVM (<code>nvidia_uvm.ko</code>) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>]; where this page and the docs tree disagree, the docs tree wins.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> partially working, software bridge in charge; <code>[gated]</code> implemented default-off; <code>[target]</code> planned end state.

== 1. What UVM is ==

UVM provides unified memory and GPU-driven page-fault handling. In practice it is a '''second GPU driver layered on the RM''', owning GPU address spaces and running its own memory-management work on the GPU.

{| class="wikitable"
|-
! Component
! Role
|-
| VA spaces (<code>uvm_va_space</code>)
| per-process GPU address spaces
|-
| Channel managers
| pools of GPU channels (CE, SEC2, GR as needed) for UVM's work
|-
| Push buffers / GPFIFO
| UVM builds and submits its own pushes
|-
| Semaphores and trackers
| UVM waits on GPU-written semaphores to track completion
|-
| Page trees
| GPU page tables for its VA spaces
|-
| Fault handling
| fault buffer plus notification; UVM resolves and replays/cancels faults
|}

For CUDA, UVM is entered early: <code>cuInit</code> → driver init → <code>UVM_REGISTER_GPU</code>. If UVM cannot register the GPU, CUDA does not start — which is why "UVM advances" is the project's primary correctness metric on the way to CUDA.

== 2. The registration path (traced end to end) ==

<pre>libcuda: UVM_REGISTER_GPU ioctl (0x25)
uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
nvUvmInterfaceDeviceCreate (RM object creation)
uvm_hal_init_gpu / get_gpu_caps (CE caps)
uvm_ats_add_gpu / uvm_gpu_init_isr (fault/interrupt plumbing)
init_channel_manager
create semaphore pool (sysmem)
create channels per CE (GPFIFO + pushbuffer + semaphores)
channel_init() -> first push -> semaphore release -> wait
uvm_page_tree_init (page tables for the VA space)</pre>

Each stage has a distinct failure signature — see [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]].

== 3. What UVM requires from the "GPU" ==

{| class="wikitable"
|-
! Requirement
! Detail
! Status
|-
| GPU identity
| must match the UUID from RM
| <code>[live]</code>
|-
| CE capabilities
| per-engine support bits; UVM requires <code>supported && !grce && sysmem</code> for CPU_TO_GPU / GPU_TO_CPU channels
| <code>[live]</code> caps accepted; <code>[gated]</code> copies forwarded to the physical copy engine
|-
| Channel creation
| RM channel objects with GPFIFO/userd/instance parameters
| <code>[live]</code>
|-
| Pushes executed
| first action on a new channel is an init push plus a semaphore-tracked wait
| <code>[wip]</code> software bridge executes memsets, copies and releases; <code>[gated]</code> CE-class copies forwarded
|-
| Page trees
| UVM initializes page trees with its own pushes; server-owned ranges must be populated by us
| <code>[wip]</code>
|-
| Fault delivery
| fault buffer plus notification; replay/cancel semantics
| <code>[gated]</code> backend-side observation/relay; guest contract <code>[target]</code>
|-
| Completion semantics
| tracker values must advance exactly
| <code>[wip]</code>
|}

'''Current frontier.''' Context creation and the full CUDA driver-API smoke pass (<code>ALL OK</code>, copies matched) run end to end. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored releases and a fault relay. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed; context validity at dispatch is ruled out ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]). The remaining UVM work is completion translation and the guest-side fault replay/cancel contract.

== 4. The discovery that defines the work: how UVM waits ==

UVM's tracker is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps its state:

<pre>uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
releasing value 1</pre>

Mechanically, UVM built a push ending in a '''semaphore release''' — a method sequence (CE channel class, e.g. <code>TURING_DMA_COPY_A</code>) that sets an address and a payload and launches a DMA operation to write the payload to memory — then submitted it and waits for the payload to appear. On real hardware the copy engine writes it; on our device nobody does unless the emulation does. The push we found in guest RAM decodes as:

<pre>20010000 0000c5b5 set-object (TURING_DMA_COPY_A)
2001a000 0000c076 set-object (GP100_UVM_SW)
20038090 00000001 incrementing-method header: count 3, addr 0x240
21000000 00000001 lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD</pre>

So the emulation's job for this push is: '''write <code>payload</code> to the memory at the semaphore's GPU VA''', which requires a GPU-VA → guest-PA translation. The host does this through the learned VA→PA map: while the channel's GPFIFO has pending entries it reads each push, decodes the method stream and runs the work (page-table memsets, copies, semaphore releases). When a push page cannot be resolved — or a channel consumes entries without any release landing — it falls back to content: guest RAM is scanned for release groups and pending payloads are applied forward-only. A release is applied only when <code>payload > current</code>, so recycled push buffers and stale copies are harmless. One pitfall observed live: a semaphore-page VA can flap between two physical pages across map rebuilds; pinning the validated translation fixes the stalled wait.

== 5. The memory model UVM assumes ==

* UVM's semaphore pool is '''system memory''' (<code>UVM_APERTURE_SYS</code>), one 4 KiB page per channel manager, addressed as 4-byte slots (1024 per page).
* Each page has a '''GPU VA''' (chosen by the client's RM) and a '''CPU VA''' (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
* Both addresses are recorded in UVM's structures in guest memory — the key that makes translation possible without extra help from the guest.
* The CPU VA is a direct-map address, so <code>PA = cpu_va - page_offset_base</code> (verified live: writing the payload there completes the push).
* Values are per channel and monotonic (1, 2, 3 …); UVM's own check (<code>UVM_GPU_SEMAPHORE_MAX_JUMP</code>) detects and reports wrong values.
* The client's page tables for '''server-owned ranges are empty chains''': the client hands us the physical addresses of its page-table levels and expects the server to own the lower levels ([[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]]).

== 6. Page trees and faults ==

<code>uvm_page_tree_init</code> initializes a VA space's page tables with pushes on UVM's own channels (memops and page-table writes; the copy engine executes them on real hardware). Fault handling is event-driven: UVM programs a fault buffer (replayable and non-replayable; sizes from <code>INTERNAL_GMMU_GET_STATIC_INFO</code>, answered by the emulation) and, on a fault, reads the entry, populates the page table and replays or cancels the fault with a <code>MEM_OP</code> push. Reconciling the guest's memory model with the real GPU's here is the deepest remaining work.

== 7. How to observe UVM ==

In the guest, UVM prints nothing by default:

<pre>echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints</pre>

Messages then appear with the <code>nvidia-uvm:</code> prefix. Known-good markers, in order: <code>uvm_gpu_retain_by_uuid</code> → channel manager created → <code>uvm_page_tree_init</code> pushes → <code>UVM_REGISTER_GPU</code> returns and CUDA proceeds. Failure signatures characterized live:

{| class="wikitable"
|-
! Message
! Meaning
|-
| <code>Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED</code>
| CE caps wrong or missing
|-
| <code>stuck waiting</code> plus <code>Tracker entry ... channel_init()</code>
| the push's semaphore was never written
|-
| <code>Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101</code>
| a wrong payload was written; UVM caught it
|}

== 8. Why UVM is the perfect correctness oracle ==

UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling and error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU.

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>] — canonical chapter (this page's source).
* [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the guest stack and the RM objects UVM consumes.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the userspace half of the contract; [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — how UVM's pushes execute, and the current wall.
* [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the fault contract.

System Overview

2026-10-12T04:41:02Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= System Overview =

<blockquote>'''What this page is.''' The one page to read if you want to understand ''how Open-IOV works'' before diving into any subsystem. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/01-system-overview.md <code>../docs/03-architecture/01-system-overview.md</code>] (with [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/09-full-system-picture.md <code>../docs/03-architecture/09-full-system-picture.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/10-interaction-and-scheduling.md <code>../docs/03-architecture/10-interaction-and-scheduling.md</code>] for the full picture); the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> designed end state.

== 1. The core principle ==

'''Emulate the interface. Pass through the work. Never patch anyone.'''

A virtual GPU is a ''mediated passthrough'' device. The guest must believe it owns a real NVIDIA GPU with a real GPU System Processor running firmware; the host must arrange for the guest's work to execute on real hardware, because emulating compute in software is orders of magnitude too slow to be useful.

<pre>EMULATE (software; the guest expects virtual hardware)
PCI configuration space, capability structures, power management
BAR0 register interface (shadow, not the physical GPU's BAR0)
BAR1 VRAM aperture (guest-visible video memory)
GSP firmware boot handshake (FWSEC -> booter -> RISCV -> boot complete)
GSP-RM message queues, RPC transport, resource-manager controls
event/notification channels; MSI-X injection (via eventfd)

PASS THROUGH (real hardware; correctness and performance)
GPU memory: guest VRAM objects backed by real VRAM [target]
page tables / GMMU state used by the GPU [wip]
channel execution: guest push buffers on real channels [gated/wall]
compute (CUDA) and, later, 3D/video work [wip/planned]
DMA, completion and fault interrupts relayed to the guest</pre>

The line between the two is the '''interface boundary''', and it moves as the project matures: early work emulates everything the guest can observe; later work forwards everything that must be fast.

== 2. Layer map ==

<pre>GUEST
application ── CUDA runtime (libcudart) ─┐
application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack
OpenGL/Vulkan ── libGLX/libvulkan ──────┘
│ ioctls / mmaps
nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm
│ MMIO + DMA + MSI-X (a virtual PCIe device)
════════╪════════════════════════════════════════════════════════
HYPERVISOR (KVM/QEMU, later Xen/...) VFIO mdev transport + IOMMU
│
HOST MODULE vgsp.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA<->PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU
│ module-owned device + real GSP session
HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
│ (the distribution nvidia.ko stays unmodified and unbound in
│ this mode)
v
REAL HARDWARE (engines, memory, interrupts)</pre>

The guest never contains Open-IOV code; the host driver is unmodified; the hypervisor is unmodified. The only new component is <code>vgsp.ko</code>.

== 3. How each side works ==

* '''Guest side.''' The stock driver performs the GSP boot handshake, sets up RPC message queues in guest memory, creates RM objects, programs channels and submits work — exactly as on bare metal ([[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]], [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]]).
* '''Host module.''' <code>vgsp.ko</code> is a VFIO mediated-device driver: it registers a parent on the physical GPU, exposes virtual GPU types, and each mdev is one virtual GPU for one VM. Inside, it presents a plausible Turing-class device (config space, BAR shadows, MSI-X), answers the GSP boot handshake and the RPC protocol with well-formed replies, and maintains the memory model that resolves guest physical addresses to host addresses ([[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]]).
* '''The interface that makes this possible.''' Since Turing, the driver's management path is not ''silicon'', it is ''firmware'': the GSP runs a version of NVIDIA's Resource Manager and the driver talks to it over shared-memory queues with a small RPC protocol. That protocol is small, stable and observable — so the project emulates '''the firmware's interface''' instead of emulating a GPU ([[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]], [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>]).
* '''Execution.''' Guest work that must be fast (copies, kernels) is forwarded to a real pipeline driven through the module's own GSP-RM session ([[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 4. Data flows ==

<pre>MMIO from the guest:
guest nvidia.ko writes BAR0 offset X
-> KVM exit -> QEMU -> VFIO region write into the mdev
-> BAR0 shadow state machine (doorbells, VF registers, notify)
-> may wake the emulation thread

RPC from the guest to "firmware":
guest writes a command-queue element into a guest-memory msgq
-> doorbell (BAR0)
-> emulation thread reads the queue through the hypervisor's guest-RAM
accessor, parses the element, dispatches the RPC
-> handler produces a reply element in the status queue
-> emulation injects an MSI-X interrupt to the guest</pre>

== 5. What runs today ==

# Guest boots, enumerates the virtual PCIe device, binds <code>nvidia.ko</code>.
# The GSP boot handshake completes against the emulation.
# Message queues are set up, versions negotiated, RM initialization exchanges complete; object creation, capability queries, vaspaces, channels and the engine inventory are answered with well-formed replies (a small set of control IDs is special-cased with real values; reply fidelity for consumed fields keeps improving).
# <code>nvidia-smi -L</code> enumerates the device; CUDA userspace starts; context creation completes; the full driver-API smoke passes (<code>ALL OK</code>, 64-byte payload round-trips).
# CE-class copies are forwarded to the physical copy engine (default off) with engine-authored semaphore releases and a fault relay. For GR (graphics/compute), a forwarded guest launch reaches the front-end but the engine has not executed it yet — the current frontier ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 6. Where the detail lives ==

* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the firmware/RM architecture and the transport we emulate.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the module's internals.
* [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the guest side.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]], [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]] — memory and translation.
* [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — work and execution.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — supported combinations and portability.
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/01-system-overview.md <code>../docs/03-architecture/01-system-overview.md</code>] and the rest of the architecture corpus in the repository.

Risc-v-gsp-rm-debugger

2026-10-12T04:41:01Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= risc-v-gsp-rm-debugger =

<blockquote>'''Scope.''' risc-v-gsp-rm-debugger runs GSP/uproc firmware off-hardware in a controllable RISC-V machine, instruments it with Frida-like primitives, and records durable sessions so firmware behaviour can be captured and compared with evidence instead of anecdotes. Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/03-risc-v-gsp-rm-vm.md <code>../docs/05-operations/03-risc-v-gsp-rm-vm.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md <code>../docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>

<blockquote>'''Development-only.''' Host-side tooling that is never shipped, never a driver dependency and never present in a guest; interfaces and file formats may change without notice.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> recorded QEMU sessions; <code>[design]</code> device model and custom simulator are roadmap.</blockquote>

== 1. What it is ==

risc-v-gsp-rm-debugger makes the firmware's '''runtime behaviour''' inspectable. It boots GSP/uproc firmware in <code>qemu-system-riscv64</code> (or a deterministic native interpreter) and offers Frida-like primitives on top:

* breakpoints and enter/leave hooks (RV64 ABI aware); watchpoints on firmware variables;
* memory read/write/hexdump/search, structure decoding, snapshot diffing;
* recorded sessions: a replayable event stream plus memory dumps;
* remote calls and run-to-cursor (execute until a chosen address);
* injection: build a position-independent RV64 service, load it into the running firmware and dispatch to it;
* declarative probe specs, a plugin API, and a Ghidra bridge for symbol export, hit annotation and decompiler-driven execution.

Its purpose is '''capturing how the firmware server behaves, repeatably and safely''', so emulation changes can be measured against recorded evidence and can survive GSP version, device and architecture churn. The tooling lives under <code>open-iov/debug/risc-v-gsp-rm-vm/</code>, with its own documentation set (design, usage, scripting, plugins, bridges, accuracy, portability, roadmap).

== 2. What it is not ==

* '''Not a GSP boot environment.''' QEMU <code>virt</code> has no falcon windows, WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. <code>probe</code> reports exactly where the real image first diverges (for the TU104 570.144 image: a load-access fault at firmware vaddr <code>0x4009014</code> reading <code>0x2000000000000A00</code>) — expected data, not a boot. A device model is a roadmap item, not a claim.
* '''Not a timing model.''' Do not infer latency, jitter or concurrency behaviour from it; its in-scope conclusions are execution flow and memory side effects.
* '''Not a replacement for Ghidra or the guest diagnostics.''' Ghidra answers ''what is the code''; risc-v-gsp-rm-debugger answers ''what did it do''; the guest driver remains the oracle for guest-side behaviour.
* '''Not a hardware or security tool.''' It never attaches to a real GPU, a guest, a kernel module or an mdev, and it is never a security or confidentiality boundary.

== 3. Where it fits ==

{| class="wikitable"
|-
! Layer
! Tool
! Question answered
|-
| Static
| Ghidra and the decompilation corpus
| what is the code?
|-
| Guest driver
| guest diagnostics, ptrace tracing
| what did the driver request?
|-
| '''Firmware runtime'''
| '''risc-v-gsp-rm-debugger'''
| '''what does the firmware server do, repeatably and safely?'''
|-
| Physical backend
| supervised hardware sessions, module logs
| what did real silicon / the real GSP-RM do?
|}

The intended measurement loop: capture the real backend's RPC behaviour, run the same stimulus in risc-v-gsp-rm-debugger, and compare the recorded sessions with the tool's accuracy metrics. A simulator observation is evidence about the firmware, not about silicon; anything that would change the product is reproduced on a physical reference first.

== 4. Quickstart ==

Requirements: Linux, Python 3.10+ (standard library only), <code>qemu-system-riscv64</code> (Ubuntu package <code>qemu-system-misc</code>), <code>clang</code> and <code>lld</code>. No GPU, no driver and no kernel module are touched. From the repository root:

<pre>cd debug/risc-v-gsp-rm-vm
./risc-v-gsp-rm-debugger.sh info # environment check: QEMU, toolchain, paths
make demo && make sdk # build the bundled RV64 uproc demo and the injection SDK
./risc-v-gsp-rm-debugger.sh demo # canned run: boot, hooks, watchpoints, session
./risc-v-gsp-rm-debugger.sh analyze dumps/<printed-session-dir>
./risc-v-gsp-rm-debugger.sh probe # real GSP image: report the first divergence
./risc-v-gsp-rm-debugger.sh inject # load and invoke a position-independent service</pre>

What "good" looks like: <code>info</code> reports QEMU and the demo paths; <code>demo</code> prints hook/branch lines and a session directory under <code>dumps/</code>; <code>analyze</code> renders hook/watch/breakpoint events and a coherent summary; <code>probe</code> reports a bounded first divergence (expected data, not a boot); <code>inject</code> returns matching values on the direct and firmware-dispatched invocation paths.

A large test suite (hundreds of tests) is run with <code>make test</code>, covering unit tests plus live-QEMU smoke runs.

== 5. The primitives ==

{| class="wikitable"
|-
! Primitive
! Use it for
|-
| <code>breakpoint</code> / <code>hook</code>
| did this function run? with what arguments and return value?
|-
| <code>watch</code> (<code>w</code>/<code>r</code>/<code>a</code>)
| when did this field change, and from which store PC?
|-
| <code>run</code> / <code>run_until</code>
| bounded observation; stop at a chosen address
|-
| <code>call</code>
| remote call to unit-test a function in isolation
|-
| <code>step</code> / <code>note</code>
| single-step with dispatch; annotate the event stream
|-
| <code>Memory.read/write/hexdump/search/snapshot/diff</code>
| inspect and diff memory regions
|-
| <code>SymbolDB</code>
| resolve file/program/runtime addresses across sources
|-
| <code>Recorder</code>
| write the durable session (events, snapshots, blobs)
|-
| injection SDK
| load, register and invoke your own position-independent RV64 service
|}

Instructions between stops are invisible unless a trace is on (a TCG plugin on QEMU, or <code>--trace</code> on the native backend). Two backends — QEMU-RSP and the native interpreter — satisfy the same session API, so scripts, probes, plugins, recordings and comparisons survive a backend change.

== 6. Fidelity levels (what a result can mean) ==

{| class="wikitable"
|-
! Level
! Property
! risc-v-gsp-rm-debugger today
|-
| '''L0'''
| instruction semantics
| high (QEMU TCG, same ISA family)
|-
| '''L1'''
| memory behaviour
| high for RAM-resident code, conditional on mapping
|-
| '''L2'''
| time / concurrency
| '''not modelled''' — never claim
|-
| '''L3'''
| device environment (falcon, WPR2, CSB, PMU, timers, DMA, IRQ)
| '''absent''' — device model is roadmap
|}

A simulator observation is evidence about the firmware, not about silicon. Never quote a session for timing, interrupt, security or confidentiality claims; zero hits in a run are a coverage fact, not proof that code is dead.

== 7. Sessions as evidence ==

Every run records a session directory (format <code>greeneye-session/1</code>):

<pre>dumps/<stamp>-<uuid>-<name>/
session.json format, name, counts, snapshot inventory, meta
events.jsonl append-only event stream (hooks, watches, memory, notes)
snapshots/*.bin.z zlib-compressed region captures with sha256
blobs/* QEMU logs, instruction/access traces, extra captures
summary.md rendered analysis</pre>

Rules that make a session usable as evidence:

# '''Record with a fixed stimulus.''' The same script, seed, budgets and image hash make runs comparable; a self-comparison must be empty.
# '''Annotate as you go.''' Explain inside the session what was asked and what was changed; a chat log is not evidence.
# '''Attach the provenance.''' Firmware SHA-256, version string, symbol DB source/hash/bias, and the exact command or script.
# '''Keep the directory together.''' <code>session.json</code> + <code>events.jsonl</code> + <code>snapshots/</code> + blobs are one artifact; a copied event stream alone is not evidence.
# '''State the fidelity.''' Say which of L0/L1 was exercised and name the reference capture; do not silently waive L2/L3 gaps.
# '''Report deltas honestly.''' Every remaining delta between two runs is explained or listed as open; an empty unexplained-delta list is a claim that must be earned.
# '''Prefer an A/B pair.''' Static reading is a hypothesis; a recorded session is evidence; at least two comparable runs are a conclusion.

== 8. Safety and status ==

* '''Simulator only.''' The tools launch their own QEMU or native machine against the bundled demo or an explicitly allowed re-hosted ELF; they never attach to hardware, a guest, a kernel module, an mdev or an existing debugger stub.
* '''No guest, no hardware, no product.''' Nothing runs in a guest; injected services are prototypes, not shipped code; product changes go through the host module and a supervised physical confirmation.
* '''Bounded everywhere.''' Runs have durations and stop caps; reads and snapshots have size caps; a wedged target is left stopped and restarted clean rather than poked.
* '''Local only.''' The Ghidra bridge is a localhost transport, unauthenticated by design; never expose it on a network interface.
* '''No confidentiality claims.''' The tool is a study environment; never use injection to defeat a hardware security boundary.
* '''Status.''' The audited v0.1 surfaces are the CLI and the session format. Working and live-verified: hooks/watchpoints, memory tools, sessions, declarative probes, plugins, remote calls, run-to-cursor, the injection SDK, the Ghidra bridges, the first-fault probe and the instruction/access trace. The device model, the custom simulator and serving full host message traffic remain roadmap; porting and packaging scaffolds are pilot-status. Treat interface changes as reviewed events.

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/03-risc-v-gsp-rm-vm.md <code>../docs/05-operations/03-risc-v-gsp-rm-vm.md</code>] — project-level purpose, status and limits.
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md <code>../docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md</code>] — the single user front door.
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/02-debugging-and-maintenance-workflows.md <code>../docs/05-operations/02-debugging-and-maintenance-workflows.md</code>] — captures, failure taxonomy and upgrade workflows.
* [[GSP 570.144W Firmware Analysis|<code>GSP-570.144W-Firmware-Analysis.md</code>]] and [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the firmware image and architecture the sessions exercise.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] and [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — the client side of the RPC traffic and the supervised backend the simulator complements.
* [[Project Status|<code>Project-Status.md</code>]] and [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>] — honest overall status and the documentation front door.

Glossary

2026-10-12T04:41:01Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Glossary =

<blockquote>'''What this page is.''' The terms used across this wiki, the project's documents and its logs. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/09-reference/glossary-and-faq.md <code>../docs/09-reference/glossary-and-faq.md</code>]; the docs tree wins on disagreement. For questions, see [[FAQ|<code>FAQ.md</code>]].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>.

== GPU, driver and virtualization ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''BAR0 / BAR1 / BAR3'''
| PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window.
|-
| '''CUDASW'''
| The family of NVIDIA user-space software between applications and the driver: <code>libcuda</code>, <code>libcudart</code>, the JIT compiler, the fatbinary loader, and friends.
|-
| '''Doorbell'''
| A register or memory location a driver writes to signal "new work is available" for a channel or queue.
|-
| '''mdev'''
| Mediated device: the kernel mechanism that turns a physical device into virtual instances (VFIO).
|-
| '''vGPU'''
| Virtual GPU: the device this project presents to a guest.
|-
| '''SRIOV / VF'''
| Single-root I/O virtualization / virtual functions (hardware partitioning).
|-
| '''IOMMU / SMMU'''
| Platform DMA translation and isolation (Intel VT-d / AMD-Vi / ARM SMMU).
|}

== GSP, firmware and the RM ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''GSP'''
| GPU System Processor: the embedded RISC-V (RV64IMAC) that runs firmware containing the Resource Manager.
|-
| '''GSP-RM'''
| The Resource Manager running on the GSP (the "self-hosted RM").
|-
| '''RM / RM API'''
| NVIDIA's Resource Manager and the API used to allocate objects and issue controls.
|-
| '''RMCore'''
| The RM's object system: classes, handles, parents, methods.
|-
| '''resserv / resource server'''
| The RM component that owns allocation authority, privileges and lifetimes.
|-
| '''FWSEC'''
| Firmware security processor stage of the GSP boot sequence.
|-
| '''WPR'''
| Write-Protected Region: the VRAM area reserved for firmware.
|-
| '''MSGQ'''
| The GSP message-queue transport: shared-memory element rings with a page table, plus doorbells.
|-
| '''VGX version'''
| The version the guest announces during RPC negotiation; it selects message layouts.
|-
| '''Reply fidelity'''
| How completely an emulated reply matches what real hardware returns, as a ladder: status-only → coherent zeroed params → coherent values for consumed fields → exact values.
|}

== Execution engines and memory ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''CE'''
| Copy Engine: a DMA engine; also used for UVM's internal work.
|-
| '''GR'''
| The graphics/compute engine that executes shaders (graphics pipelines and CUDA kernels).
|-
| '''SEC2'''
| Secure copy engine used by UVM for protected operations.
|-
| '''GPFIFO'''
| GPU FIFO: a ring of entries, each pointing to a push buffer for a channel.
|-
| '''Push buffer'''
| A stream of method calls (engine commands) submitted via a GPFIFO.
|-
| '''PBDMA'''
| The push-buffer DMA engine in the host scheduler that fetches and executes push buffers.
|-
| '''QMD'''
| Queue Meta Data: the per-launch descriptor a GR channel reads to execute a kernel grid.
|-
| '''USERD'''
| Per-channel control block in VRAM: PUT/GET pointers and control state.
|-
| '''Runlist'''
| The scheduler's list of channels eligible to run.
|-
| '''TSG'''
| Time-Slice Group: channels sharing a scheduling slot.
|-
| '''Engine record / engine list'''
| The per-engine descriptors (kind, instance, class, client type) the driver walks after <code>GPU_GET_ENGINES</code>; each must be self-consistent or the driver rewrites the following allocation.
|-
| '''GMMU'''
| GPU Memory Management Unit: page tables, TLB, fault handling.
|-
| '''PDB'''
| Page Directory Base: root of a GPU page table.
|-
| '''PDE / PTE'''
| Page-directory entry / page-table entry.
|-
| '''UVM'''
| NVIDIA's unified memory module: VA spaces, page trees, faults, channel manager ([[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]]).
|-
| '''VA / PA / GPA / IOVA'''
| Virtual address / physical address / guest physical address / I/O virtual address.
|-
| '''vidmem / sysmem / aperture'''
| GPU local memory / system memory / how an address is qualified.
|-
| '''Semaphore / tracker'''
| A memory location written by the GPU to signal progress; UVM's tracker waits on it.
|-
| '''AMAPlib'''
| NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules.
|-
| '''HAL'''
| Hardware abstraction layer: per-generation functions that program registers and formats.
|-
| '''Class list (classDB)'''
| The RM classes reported as supported; engine queries are filtered against it, so the class list, engine list and capability controls must agree.
|}

== Method and tooling ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''Host oracle / A-B harness'''
| The same query or deterministic workload run against a bare-metal host driver and inside the guest, then diffed — the reference for what the guest should have seen.
|-
| '''File virtual address (<code>libcuda+0x...</code>)'''
| An address relative to the start of a shared object file, as distinct from a runtime mapping or a disassembler program address.
|-
| '''Session (risc-v-gsp-rm-debugger)'''
| A recorded run of the firmware workbench — the unit of evidence ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]).
|}

== Related pages ==

* [[FAQ|<code>FAQ.md</code>]]
* [[System Overview|<code>System-Overview.md</code>]]
* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]]
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]]

RPC Message Format

2026-10-12T04:41:00Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= GSP RPC Message Format Specification =

<blockquote>'''Historical document — superseded.''' This page describes the legacy BAR0 mailbox RPC model: the 64-byte <code>GSPR</code> magic header and the 12-type model below are '''not''' the path the guest driver uses. The live GSP-RM path is a message queue in guest system memory with 4 KB queue elements and a 32-byte RPC header (signature <code>0x43505256</code>, header version <code>0x03000000</code>); functions are a flat enum (<code>SET_GUEST_SYSTEM_INFO</code> 1, <code>FREE</code> 10, <code>GET_GSP_STATIC_INFO</code> 65, <code>GSP_RM_CONTROL</code> 76, <code>GSP_RM_ALLOC</code> 103, ...). Canonical documentation: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

The GSP firmware uses a dispatcher-based RPC system with two dispatchers:

* '''Primary''' (0x057A1AC0): ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG
* '''Secondary''' (0x04ED39DC): Branch-based dispatch for all 11 types

== Primary Dispatcher Call Convention ==

=== Entry Point ===

<pre>FUN_ram_057a1ac0(
uint32_t rpc_type, // param_1: RPC type (0x01-0x0C)
uint8_t flags_a, // param_2
uint8_t flags_b, // param_3
uint8_t flags_c, // param_4
uint32_t *out_subcmd_id,// param_5: Returns sub-command ID
uint8_t *out_a, // param_6: Echoes flags_b
uint8_t *out_b, // param_7: Echoes flags_a
uint8_t *out_c // param_8: Echoes flags_c
) -> int64_t</pre>

=== RPC Type to Sub-Command Mapping ===

The primary dispatcher validates RPC type, then indexes into the 32-entry jump table at '''0x057A3AD8'''.

{| class="wikitable"
|-
! RPC Type
! Name
! Subcmd Range
! Notes
|-
| 0x01
| ALLOC
| 0-31
| Uses 0x057E94EC main
|-
| 0x02
| FREE
| 0-31
|
|-
| 0x03
| MAP
| —
| Uses secondary dispatcher
|-
| 0x04
| UNMAP
| 0-31
| Primary + secondary
|-
| 0x05
| PIN
| —
| Uses secondary dispatcher
|-
| 0x06
| UNPIN
| —
| Uses secondary dispatcher
|-
| 0x07
| SCHEDULE
| 0-31
| Uses 0x051EFBF4 main
|-
| 0x08
| YIELD
| —
| Uses secondary dispatcher
|-
| 0x09
| NOTIFY
| —
| Uses secondary dispatcher
|-
| 0x0A
| BAR_TRAP
| —
| Uses 0x04CB8188
|-
| 0x0B
| MIGRATE
| 0-15 x2
| Dual jump tables
|-
| 0x0C
| DEBUG
| —
| Unimplemented
|}

=== Sub-Command Descriptor (Returned by Jump Table Handlers) ===

Each sub-command handler at the jump table returns:

<pre>typedef struct {
uint32_t subcmd_id; // Constant ID (e.g., 0x10, 0x14, 0x40)
uint8_t echo_flags_b; // Copy of input flags_b
uint8_t echo_flags_a; // Copy of input flags_a
uint8_t echo_flags_c; // Copy of input flags_c
} __attribute__((packed)) subcmd_descriptor_t;</pre>

'''Example: Subcmd 3 (0x057A1BCC)'''

<pre>// Returns ID = 0x10
*out_subcmd_id = 0x10;
*out_a = flags_b;
*out_b = flags_a;
*out_c = flags_c;
return 0;</pre>

'''Example: Subcmd 0 (0x057A1DB4)'''

<pre>// Returns ID = 0x40
*out_subcmd_id = 0x40;
*out_a = flags_b;
*out_b = flags_a;
*out_c = flags_c;
return 0;</pre>

== Secondary Dispatcher Call Convention ==

=== Entry Point ===

<pre>FUN_ram_04ed39dc(rpc_message_t *msg) -> int64_t</pre>

=== Message Structure (Inferred) ===

<pre>typedef struct {
uint32_t rpc_type; // 0x01-0x0B
uint32_t sub_type; // Sub-operation code
uint64_t payload_ptr; // Pointer to RPC-specific payload
uint32_t payload_size; // Size of payload
uint32_t flags; // Control flags
uint64_t context; // Opaque context
uint32_t sequence; // Sequence number
uint32_t reserved;
} __attribute__((packed)) rpc_message_t;</pre>

=== Branch-Based Dispatch ===

Secondary dispatcher uses <code>bgeu</code>/<code>bltu</code> branches on <code>rpc_type</code> register:

<pre># Pseudocode
if (rpc_type >= 0x0C) goto error;
switch(rpc_type):
case 0x03: call MAP_handler_A;
case 0x04: call UNMAP_handler_A;
case 0x05: call PIN_handler_A (0x04ED34B8);
case 0x06: call UNPIN_handler_A;
case 0x07: call SCHEDULE_handler_A;
case 0x08: call YIELD_handler_A;
case 0x09: call NOTIFY_handler_A;
case 0x0A: call BAR_TRAP_handler (0x057675B7);</pre>

== RPC-Specific Payloads ==

=== ALLOC (0x01) - Via 0x057E94EC ===

<pre>typedef struct {
uint64_t size; // Allocation size in bytes
uint32_t type; // Memory type (VRAM, sysmem, etc.)
uint32_t alignment; // Alignment requirement
uint64_t va_hint; // Virtual address hint
uint32_t flags; // CACHE_COHERENT, UNCACHED, etc.
uint32_t pid; // Process ID
uint32_t vf_id; // VF ID (for vGPU)
uint32_t reserved;
uint64_t *out_handle; // Output: allocation handle
uint64_t *out_va; // Output: mapped virtual address
uint64_t *out_pa; // Output: physical address
} alloc_payload_t;</pre>

=== MAP (0x03) - Handlers 0x05766AB7 / 0x05778A37 ===

<pre>typedef struct {
uint64_t handle; // Allocation handle from ALLOC
uint64_t va; // Target virtual address
uint64_t size; // Map size
uint32_t prot; // PROT_READ|PROT_WRITE|PROT_EXEC
uint32_t flags; // MAP_FIXED, MAP_SHARED, etc.
uint32_t vf_id; // Target VF
uint32_t reserved;
} map_payload_t;</pre>

=== PIN (0x05) - Handler 0x04ED34B8 ===

<pre>typedef struct {
uint64_t handle; // Allocation handle
uint64_t offset; // Offset within allocation
uint64_t size; // Size to pin
uint32_t vf_id; // Target VF
uint32_t flags; // PIN_WRITE, PIN_READ, PIN_LONGTERM
uint64_t *out_dma_addr; // Output: DMA address for device
} pin_payload_t;</pre>

=== SCHEDULE (0x07) - Via 0x051EFBF4 / 0x05767F37 ===

<pre>typedef struct {
uint32_t vf_id; // VF to schedule
uint32_t priority; // Scheduling priority
uint64_t time_slice_ns; // Time slice in nanoseconds
uint32_t flags; // PREEMPTIBLE, REALTIME, etc.
uint32_t reserved;
} schedule_payload_t;</pre>

=== BAR_TRAP (0x0A) - Handler 0x04CB8188 ===

<pre>typedef struct {
uint32_t bar_index; // BAR number (0-5)
uint32_t trap_type; // READ, WRITE, ATOMIC
uint64_t offset; // Offset within BAR
uint64_t value; // Value for writes
uint32_t size; // Access size (1,2,4,8)
uint32_t vf_id; // Trapping VF
uint64_t *out_value; // Output: read value
} bar_trap_payload_t;</pre>

=== MIGRATE (0x0B) - Dual Jump Tables ===

<pre>typedef struct {
uint32_t phase; // 0=prepare, 1=transfer, 2=commit, 3=cleanup
uint32_t subcmd; // Sub-command index (0-15 for each table)
uint64_t src_vf; // Source VF
uint64_t dst_vf; // Destination VF
uint64_t data_ptr; // Migration data buffer
uint64_t data_size; // Buffer size
uint32_t flags; // LIVE, POSTCOPY, etc.
uint32_t reserved;
} migrate_payload_t;</pre>

== Wire Format (Host → GSP) ==

<blockquote>'''Legacy BAR0 mailbox format — not used by the guest driver.''' The current msgq header/element facts are in the note at the top of this page.</blockquote>

=== Message Header (64 bytes) ===

<pre>typedef struct {
uint32_t magic; // 0x47535052 ("GSPR")
uint32_t version; // Protocol version (1)
uint32_t rpc_type; // 0x01-0x0C
uint32_t sequence; // Monotonic sequence
uint64_t timestamp_ns; // Submission timestamp
uint32_t payload_size; // Size of payload
uint32_t flags; // MSG_SYNC, MSG_ASYNC, MSG_NOREPLY
uint64_t correlation_id;// For async matching
uint64_t reserved[3]; // Padding to 64 bytes
} __attribute__((packed)) rpc_header_t;</pre>

=== Complete Message ===

<pre>+------------------+
| rpc_header_t | 64 bytes
+------------------+
| payload (varies) | payload_size bytes
+------------------+
| padding to 256B | (alignment)
+------------------+</pre>

== Response Format ==

=== Synchronous Response ===

<pre>typedef struct {
int64_t status; // 0 = success, negative = error
uint32_t rpc_type; // Echo of request
uint32_t sequence; // Echo of request
uint64_t timestamp_ns; // Completion timestamp
uint32_t payload_size; // Response payload size
uint32_t reserved;
} __attribute__((packed)) rpc_response_header_t;</pre>

=== Error Codes ===

{| class="wikitable"
|-
! Code
! Name
! Description
|-
| 0
| OK
| Success
|-
| -1 (0xFFFF)
| EPERM
| Permission denied
|-
| -2 (0xFFFE)
| ENOENT
| Not found
|-
| -3 (0xFFFD)
| EINVAL
| Invalid argument
|-
| -6 (0xFFFA)
| ENODEV
| No such device
|-
| -12 (0xFFF4)
| ENOMEM
| Out of memory
|-
| -16 (0xFFEE)
| EBUSY
| Resource busy
|-
| -22 (0xFFEA)
| EINVAL
| Invalid argument
|-
| -28 (0xFFE4)
| ENOSPC
| No space left
|-
| -37 (0xFFDB)
| ENOLCK
| No locks available
|-
| -56 (0xFFC8)
| EBADMSG
| Bad message
|-
| -84 (0xFFAC)
| EOVERFLOW
| Value too large
|-
| -90 (0xFFA6)
| ENOSTR
| Not a stream
|-
| -91 (0xFFA5)
| ENODATA
| No data available
|-
| -94 (0xFFA2)
| ENOSR
| Out of streams resources
|-
| -97 (0xFF9F)
| EPROTO
| Protocol error
|-
| -111 (0xFF91)
| ETIMEDOUT
| Connection timed out
|}

== Kernel Client Implementation ==

=== Message Builder ===

<pre>static int gsp_rpc_send(gsp_client_t *client, uint32_t rpc_type,
void *payload, size_t payload_size,
void *response, size_t *response_size) {
rpc_header_t hdr = {
.magic = 0x47535052,
.version = 1,
.rpc_type = rpc_type,
.sequence = atomic_inc(&client->seq),
.timestamp_ns = ktime_get_ns(),
.payload_size = payload_size,
.flags = MSG_SYNC,
};

// Build complete message
uint8_t msg[256];
memcpy(msg, &hdr, sizeof(hdr));
memcpy(msg + 64, payload, payload_size);

// Submit to GSP doorbell
return gsp_doorbell_ring(client->gsp, msg, sizeof(msg));
}</pre>

=== Doorbell Mechanism ===

<pre>// GSP doorbell register (PCIe BAR0 + offset)
#define GSP_DOORBELL_OFFSET 0x1000

static void gsp_doorbell_ring(gsp_device_t *gsp, void *msg, size_t len) {
// Copy message to shared memory ring
// Ring GSP doorbell
writel(1, gsp->bar0 + GSP_DOORBELL_OFFSET);
}</pre>

== Async Completion ==

=== Notification Mechanism ===

<pre>typedef struct {
uint32_t rpc_type;
uint32_t sequence;
int64_t status;
uint64_t timestamp_ns;
uint32_t payload_size;
uint32_t reserved;
} async_completion_t;

// Delivered via MSI-X or GSP notification queue</pre>

----

== Reference ==

* Primary Dispatcher: 0x057A1AC0
* Secondary Dispatcher: 0x04ED39DC
* Jump Table: 0x057A3AD8 (32 entries)
* RPC Type Jump Table: 0x059BFEB0 (12 types)

Project Status

2026-10-12T04:41:00Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Project Status =

<blockquote>'''What this page is.''' The honest, dated state of the project: what works today, what is in progress, and what is not started. It is refreshed periodically and each claim maps to a maintained document in the repository. Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>] (project front door), [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] (known-knowns and known-unknowns) and [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>] (the execution frontier). When this page and the docs disagree, the docs tree wins.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or behind supervised gates; <code>[wip]</code> partially implemented; <code>[wall]</code> current blocker; <code>[target]</code> named success state; <code>[planned]</code> designed, not built.

== 1. Milestone ladder ==

{| class="wikitable"
|-
! Milestone
! State
|-
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev)
| <code>[live]</code> working
|-
| GSP boot emulation (FWSEC / booter / RISCV handshake)
| <code>[live]</code> working
|-
| GSP-RM message queues and RPC transport
| <code>[live]</code> working
|-
| Stock guest driver probe and <code>nvidia-smi</code> (TU104)
| <code>[live]</code> working
|-
| CUDA userspace init and the driver-API smoke (<code>cuInit</code>, device enumeration, <code>cuCtxCreate</code>, allocation, copies, free, destroy)
| <code>[live]</code> working — the smoke prints <code>ALL OK</code> and the 64-byte payload round-trips exactly
|-
| Guest CUDA kernel execution on real hardware
| <code>[wall]</code> a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue
|-
| Physical backend on real hardware (module-owned GPU running its real GSP-RM)
| <code>[live]</code> boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above
|-
| Multi-tenant isolation (mdev per VM, per-context page directories)
| <code>[wip]</code> designed; the baseline service-state ownership work is in progress
|-
| OpenGL / Vulkan (3D), video
| <code>[planned]</code>
|-
| Windows guests
| <code>[planned]</code> (target)
|-
| Xen / Argo / OpenXT / Qubes Air transports
| <code>[planned]</code> (the emulation core is hypervisor-independent by construction)
|-
| Live migration
| <code>[planned]</code>
|-
| Performance numbers
| <code>[planned]</code> — no published numbers yet; any claim will ship with its measurement methodology
|}

== 2. What "CUDA works" means here ==

A green <code>nvidia-smi</code> or even a passing driver-API smoke is not "CUDA works". The project's definition of done for CUDA requires all of the following ([https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>]):

# Stock guest driver probe, <code>nvidia-smi</code>, device enumeration — done.
# Context creation and the full driver-API surface (<code>cuMemAlloc</code>, copies, free, destroy) — done.
# A guest kernel launch that '''executes on real hardware''' and is proven by an engine-authored completion — the current frontier.
# Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far.
# Correct completion, semaphore and fault semantics back to the guest — CE class work has the relay; GR is open.
# Reproducibility: every boot, not a lucky one — an open issue.
# No ad-hoc default-off parameters required for the canonical path.

Until every gate holds, the honest formulation is: '''initialization and the driver-API path work; engine execution is in progress.'''

== 3. How to see it yourself ==

The reference setup takes about half an hour from a fresh host ([[Install and First Run|<code>Install-and-First-Run.md</code>]]); the current expected outcome is exactly the ladder above — driver probe, <code>nvidia-smi</code>, the full driver-API smoke, and the execution frontier where it stands. The risc-v-gsp-rm-debugger workbench ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]) is the host-side tooling used to study the firmware side of that frontier.

== 4. How this page is maintained ==

Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with its evidence and its open theories, is on [[The Execution Frontier|<code>Execution-Frontier.md</code>]].

== 5. Related pages ==

* [[Install and First Run|<code>Install-and-First-Run.md</code>]]
* [[System Overview|<code>System-Overview.md</code>]]
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]]
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — which hardware and software combinations are Reference, Target or Experimental.
* [[FAQ|<code>FAQ.md</code>]]

Physical GPU Backend and Guest Wiring

2026-10-12T04:40:59Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Physical GPU Backend and Guest Wiring =

<blockquote>'''Scope.''' The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> planned end state.

== 1. The problem and its constraints ==

The guest's real work (copies, memory operations, grid launches) must execute on the physical GPU. The constraints: one DKMS kernel module on the host, no daemon and no new host userspace; host and guest drivers unmodified, with no paravirtual enlightenment in the guest; all emulation and translation logic inside the module; and real execution — compute is not emulated in software.

== 2. Why the host driver is not the path ==

{| class="wikitable"
|-
! Interface
! What it offers
! Reachable from our module?
|-
| Host driver (distribution build)
| resource management, scheduling, memory
| No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface
|-
| Host driver peer-memory exports
| pin/map memory for peer devices
| Partially — memory only, no submission
|-
| Vendor vGPU host stack
| mediated work submission and scheduling
| Requires the licensed vGPU host branch and a vGPU-aware guest — excluded
|-
| SR-IOV virtual functions
| hardware-partitioned device
| Not available on this GPU generation
|-
| The GPU's own GSP firmware
| the resource manager server itself
| Yes — once the module owns the device; this is the path used
|}

With an unmodified host driver there is no callable submission interface into it, so the module must own the device and talk to the hardware itself.

== 3. Mechanism options and the decision ==

<pre> (A) Proxy through the host driver: module -> host driver's RM -> real channel
-> GPU. Needs a modified host driver or a vendor interface — not implementable.
(B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
(C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client; scheduling,
page tables and context management stay the firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.</pre>

Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.

== 4. The chosen mechanism: bring-up ladder ==

The module binds the physical GPU as its driver while the distribution driver is unbound (dedicated-GPU deployment, §9). Each stage is reversible and independently verified:

# '''Ownership''' — the board's power domain is held active across the window (the platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. <code>[live]</code>
# '''Firmware staging''' — the matching GSP image is fetched and validated (ELF64 RISC-V; loadable section, version, hardware signature); no device state touched. <code>[live]</code>
# '''Boot''' — protected region in video memory, secure-verification stage, booter, RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. <code>[live]</code>
# '''Host RM client''' — the module speaks the already-mapped RPC protocol to the real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". <code>[live]</code>
# '''Guest mediation''' — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. <code>[wip]</code>
# '''Work path''' — decoded guest pushes become real push streams; resolved host pages map through module-built scratch page tables; the firmware schedules the work. <code>[gated]</code>
# '''Completion relay''' — hardware-completed semaphores land in guest-polled memory; events and faults relay through the existing MSI-X injection path. <code>[gated]</code>

== 5. The two planes and the wiring ==

The '''emulation plane''' serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The '''physical backend plane''' owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.

<pre>GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
-----------------------------------------------------------------------------
INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
claim: power domain held -> distribution driver released -> module
binds the device -> liveness check [live]
boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
mdev + ladder: backend joins the live session; real objects,
channels, first real CE copy [live]
guest: nvidia.ko probes; emulation serves handshake, static config,
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
page tables; GR: launch reaches the front-end, descriptor not seen [wall]
DONE engine authors the release in guest memory; host relays via MSI-X [gated]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
never auto-retried; recovery is supervised [live]</pre>

== 6. Proven capabilities ==

{| class="wikitable"
|-
! Capability
! Status
|-
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X)
| done <code>[live]</code>
|-
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy)
| done — 64/64 payload round-trip <code>[live]</code>
|-
| Software bridge for decoded memory/state operations
| in progress <code>[wip]</code>
|-
| Device ownership, firmware staging, board-liveness checks
| verified live <code>[live]</code>
|-
| Power domain held active across the ownership window
| verified across supervised cycles <code>[live]</code>
|-
| Real GSP boot by the module, through init-done and clean release
| done, no reboot on clean cycles <code>[live]</code>
|-
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy)
| done <code>[live]</code>
|-
| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay
| default-off, verified live; one relayed-copy mismatch under triage <code>[gated]</code>
|-
| CUDA kernel execution (<code>vectorAdd</code>)
| open <code>[wall]</code> — the launch-descriptor install path; correct results are the <code>[target]</code>
|}

== 7. The execution frontier ==

The RM client ladder reaches the compute engine (a GR0 channel plus a compute-class object). A forwarded guest CUDA launch now reaches the front-end (ring entry fetched, methods executed) but the inline launch-descriptor install has not been observed and no grid has dispatched; context validity at dispatch is ruled out. The wall is the form of the forwarded launch method stream — see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]. A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step.

== 8. Safety and platform discipline ==

* '''Dedicated GPU deployment.''' One GPU serves the vGPU service; the host uses another GPU or integrated graphics. The distribution driver stays unmodified and is unbound from that device only while the service is active.
* '''The board's power domain belongs to the platform.''' Releasing the last driver can power-gate the board and the power-on sequence can fail; the ownership path holds the whole board in a held-active state across the window, verifies config-space liveness, and aborts on any power-loss signature without retry.
* '''Supervised cycles.''' The device is driven by the module under supervised cycles with a written recovery plan. A failed mutating operation parks the device in a locked fail-safe and is never auto-retried; any reboot requires explicit human approval with a written reason and post-reboot plan first.
* '''Fail closed.''' No forwarding seam programs guest-derived page tables into the real MMU until the population path is proven end to end; the current forwarded path submits engine work only through module-built scratch page tables mapping already-resolved host pages.

== 9. Sharing policy and isolation ==

* '''Dedicated GPU (chosen):''' one GPU serves the vGPU service; a multi-GPU host gives the distribution driver its own GPU.
* '''Shared single GPU with the host driver bound:''' no conforming mechanism exists on this generation without vendor interfaces; it would require relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
* Isolation is one real GSP client per mdev with per-client page tables; host access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]]).

== 10. Open questions ==

* Firmware/host driver ABI match for the supported family on the development GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.

== 11. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] — canonical: mechanism choice and device bring-up.
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — canonical: the guest-wiring design.
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — the GR wall and the software execution plane.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the module that owns both planes.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]] — page-table seams and scratch mappings.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]], [[Project Status|<code>Project-Status.md</code>]] — support scope and the honest status picture.

Migration Protocol

2026-10-12T04:40:59Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Migration Protocol (RPC 0x0B) =

<blockquote>'''Historical document — superseded.''' This page records the MIGRATE RPC (type 0x0B) as seen in firmware analysis; in emulation mode the MIGRATE RPC returns <code>-ENOTSUPP</code>. The live migration implementation is [https://git.vgpu.io/open-iov.git/tree/kernel/migration.c <code>../kernel/migration.c</code>]; design and status are documented in [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation <code>../docs/04-implementation/</code>].</blockquote>

== Overview ==

The MIGRATE RPC (type 0x0B) implements live migration for vGPU virtual functions. It uses '''dual jump tables''' for two-phase migration operations, accessed via the primary dispatcher at 0x057A1AC0.

== Key Addresses ==

=== Migration Jump Tables ===

==== Table 1: 0x04179ED8 (Primary MIGRATE branch at 0x057A1BDC) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
! Description
|-
| 0
| 0x057A2344
| ✅
| Phase 1: Prepare - returns ID 0x14, checks flag at +0x7F9
|-
| 1
| 0x057A2320
| ✅
| Phase 1: Prepare variant
|-
| 2
| 0x057A230C
| ✅
| Phase 1: Prepare variant
|-
| 3
| 0x057A22F8
| ✅
| Phase 1: Prepare - returns ID 0x10
|-
| 4
| 0x057A2388
| ✅
| Phase 1: Prepare variant
|-
| 5
| 0x057A23D8
| ✅
| Phase 1: Prepare variant
|-
| 6
| 0x057A23AC
| ✅
| Phase 1: Prepare variant
|-
| 7
| 0x057A2478
| ✅
| Phase 1: Prepare variant
|-
| 8
| 0x057A2428
| ✅
| Phase 1: Prepare variant
|-
| 9
| 0x057A2454
| ✅
| Phase 1: Prepare variant
|-
| 10
| 0x057A2404
| ✅
| Phase 1: Prepare variant
|-
| 11
| 0x057A2364
| ✅
| Phase 1: Prepare variant
|-
| 12
| 0x057A1F3C
| (dup)
| Duplicate entry
|-
| 13
| 0x057A299C
| —
| Phase 2: Transfer
|-
| 14
| 0x057A1F3C
| (dup)
| Duplicate entry
|-
| 15
| 0x057A1F3C
| (dup)
| Duplicate entry
|}

==== Table 2: 0x04179FBC (Primary MIGRATE branch at 0x057A1D5C) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
! Description
|-
| 0
| 0x057A257C
| ✅
| Phase 2: Transfer
|-
| 1
| 0x057A25E4
| ✅
| Phase 2: Transfer
|-
| 2
| 0x057A24EC
| ✅
| Phase 2: Transfer
|-
| 3
| 0x057A2608
| ✅
| Phase 2: Transfer
|-
| 4
| 0x057A2534
| ✅
| Phase 2: Transfer
|-
| 5
| 0x057A25C0
| ✅
| Phase 2: Transfer
|-
| 6
| 0x057A24A4
| ✅
| Phase 2: Transfer
|-
| 7
| 0x057A259C
| ✅
| Phase 2: Transfer
|-
| 8
| 0x057A2510
| ✅
| Phase 2: Transfer
|-
| 9
| 0x057A2558
| ✅
| Phase 2: Transfer
|-
| 10
| 0x057A24C8
| ✅
| Phase 2: Transfer
|-
| 11
| 0x057A262C
| ✅
| Phase 2: Transfer
|-
| 12
| 0x057A345C
| ✅
| Phase 3: Commit
|-
| 13
| 0x057A36D8
| ✅
| Phase 3: Commit
|-
| 14
| 0x057A35D8
| ✅
| Phase 3: Commit
|-
| 15
| 0x057A3658
| ✅
| Phase 3: Commit
|}

== Migration Phases ==

=== Phase 0: Discovery (Pre-migration) ===

<pre>// Query VF migratability
int gsp_migrate_query(struct gsp_vf *vf, struct migrate_query *out) {
migrate_payload_t payload = {
.phase = MIGRATE_PHASE_QUERY,
.subcmd = 0,
.src_vf = vf->id,
.dst_vf = 0,
};
return gsp_rpc_send(vf->gsp, RPC_MIGRATE, &payload, sizeof(payload), out, sizeof(*out));
}</pre>

=== Phase 1: Prepare (Table 1, indices 0-11) ===

'''Goal''': Quiesce VF, prepare memory state for transfer

<pre>// Subcmd 3 (0x057A22F8) - Returns ID 0x10
int migrate_prepare_quiesce(vf_id) {
// 1. Pause VF scheduling
// 2. Flush pending commands
// 3. Pin all VF memory
// 4. Return quiesce token
}

// Subcmd 0 (0x057A2344) - Returns ID 0x14, checks flag at in_t1+0x7F9
int migrate_prepare_snapshot(vf_id, *token) {
// 1. Snapshot VF register state
// 2. Snapshot BAR configurations
// 3. Snapshot WPR regions
// 4. Snapshot page tables
// 5. Return snapshot handle
}</pre>

=== Phase 2: Transfer (Table 1 index 13, Table 2 indices 0-11) ===

'''Goal''': Transfer memory pages to destination

<pre>// Table 2, subcmd 0 (0x057A257C)
int migrate_transfer_pages(vf_id, page_list, *progress) {
// 1. Iterate page list
// 2. For each page:
// a. Copy via DMA (0x0588F9C4)
// b. Verify checksum
// c. Mark page dirty if modified
// 3. Return bytes transferred
}

// Table 2, subcmd 11 (0x057A262C) - Dirty page tracking
int migrate_track_dirty(vf_id, *dirty_bitmap) {
// Return bitmap of pages dirtied since last transfer
}</pre>

=== Phase 3: Commit (Table 2 indices 12-15) ===

'''Goal''': Final synchronization, cutover to destination

<pre>// Subcmd 12 (0x057A345C)
int migrate_commit_finalize(vf_id, dst_vf_id) {
// 1. Final dirty page transfer
// 2. Transfer register state
// 3. Activate destination VF
// 4. Release source VF resources
// 5. Notify completion
}

// Subcmd 13 (0x057A36D8)
int migrate_commit_verify(vf_id, *verify_result) {
// 1. Verify memory integrity
// 2. Verify register state
// 3. Verify device state
// 4. Return PASS/FAIL
}</pre>

=== Phase 4: Cleanup (Post-migration) ===

<pre>int migrate_cleanup(src_vf_id) {
// 1. Release pinned memory
// 2. Free snapshot buffers
// 3. Remove WPR regions
// 4. Destroy migration context
}</pre>

== Migration Message Format ==

<pre>typedef struct {
uint32_t phase; // MIGRATE_PHASE_*
uint32_t subcmd; // Sub-command index (0-15 per table)
uint64_t src_vf; // Source VF ID
uint64_t dst_vf; // Destination VF ID
uint64_t data_ptr; // Data buffer (page list, bitmap, etc.)
uint64_t data_size; // Buffer size
uint32_t flags; // MIGRATE_LIVE, MIGRATE_POSTCOPY, etc.
uint32_t reserved;
uint64_t token; // Opaque migration token
} __attribute__((packed)) migrate_payload_t;

#define MIGRATE_PHASE_QUERY 0
#define MIGRATE_PHASE_PREPARE 1
#define MIGRATE_PHASE_TRANSFER 2
#define MIGRATE_PHASE_COMMIT 3
#define MIGRATE_PHASE_CLEANUP 4

#define MIGRATE_FLAG_LIVE (1<<0) // Live migration
#define MIGRATE_FLAG_POSTCOPY (1<<1) // Post-copy mode
#define MIGRATE_FLAG_AUTO_COMMIT (1<<2) // Auto-commit after transfer
#define MIGRATE_FLAG_VERIFY (1<<3) // Verify integrity</pre>

== Sub-Command Descriptors (Returned by Handlers) ==

All migration handlers follow the primary dispatcher pattern:

<pre>typedef struct {
uint32_t subcmd_id; // Constant: 0x10, 0x14, etc.
uint8_t echo_flags_b; // Copy of input flags_b
uint8_t echo_flags_a; // Copy of input flags_a
uint8_t echo_flags_c; // Copy of input flags_c
} migrate_descriptor_t;</pre>

'''Known Subcmd IDs''':

{| class="wikitable"
|-
! Subcmd
! ID
! Phase
|-
| 0x057A22F8 (Table1[3])
| 0x10
| Prepare
|-
| 0x057A2344 (Table1[0])
| 0x14
| Prepare
|-
| Others
| TBD
| Transfer/Commit
|}

== vGPU-IO Integration ==

=== Migration State Machine ===

<pre>typedef enum {
MIG_STATE_IDLE = 0,
MIG_STATE_QUERY,
MIG_STATE_PREPARE,
MIG_STATE_TRANSFER,
MIG_STATE_COMMIT,
MIG_STATE_CLEANUP,
MIG_STATE_FAILED,
MIG_STATE_COMPLETED
} mig_state_t;

struct gsp_migration {
struct gsp_vf *vf;
mig_state_t state;
uint64_t token;
uint64_t dst_vf;
struct migrate_progress progress;
struct completion comp;
struct mutex lock;
};</pre>

=== Migration Coordinator ===

<pre>int gsp_migration_start(struct gsp_vf *vf, uint64_t dst_vf) {
struct gsp_migration *mig = kzalloc(sizeof(*mig), GFP_KERNEL);
vf->migration = mig;
mig->vf = vf;
mig->dst_vf = dst_vf;
mig->state = MIG_STATE_QUERY;

// Phase 0: Query
struct migrate_query qry;
gsp_migrate_query(vf, &qry);
if (!qry.migratable) return -EINVAL;

// Phase 1: Prepare
mig->state = MIG_STATE_PREPARE;
migrate_payload_t prep = {
.phase = MIGRATE_PHASE_PREPARE,
.subcmd = 3, // Quiesce
.src_vf = vf->id,
.dst_vf = dst_vf,
};
migrate_descriptor_t desc;
gsp_rpc_send(vf->gsp, RPC_MIGRATE, &prep, sizeof(prep), &desc, sizeof(desc));
mig->token = desc.subcmd_id; // Use returned token

// Phase 2: Transfer (iterative)
mig->state = MIG_STATE_TRANSFER;
while (!gsp_migration_transfer_iter(mig)) {
msleep(10); // Throttle
if (signal_pending(current)) break;
}

// Phase 3: Commit
mig->state = MIG_STATE_COMMIT;
migrate_payload_t commit = {
.phase = MIGRATE_PHASE_COMMIT,
.subcmd = 12,
.src_vf = vf->id,
.dst_vf = dst_vf,
.token = mig->token,
.flags = MIGRATE_FLAG_VERIFY,
};
gsp_rpc_send(vf->gsp, RPC_MIGRATE, &commit, sizeof(commit), NULL, 0);

// Phase 4: Cleanup
mig->state = MIG_STATE_CLEANUP;
gsp_migration_cleanup(mig);

mig->state = MIG_STATE_COMPLETED;
return 0;
}</pre>

=== Transfer Iteration ===

<pre>bool gsp_migration_transfer_iter(struct gsp_migration *mig) {
migrate_payload_t xfer = {
.phase = MIGRATE_PHASE_TRANSFER,
.subcmd = 0,
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.data_ptr = (uint64_t)mig->progress.page_list,
.data_size = mig->progress.page_count * sizeof(uint64_t),
.token = mig->token,
};

struct migrate_progress prog;
int ret = gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE,
&xfer, sizeof(xfer), &prog, sizeof(prog));

if (ret) return true; // Error = done

mig->progress.bytes_transferred += prog.bytes_transferred;
mig->progress.pages_transferred += prog.pages_transferred;
mig->progress.dirty_pages = prog.dirty_pages;

// Check if more dirty pages to transfer
if (prog.dirty_pages > 0 && !(mig->flags & MIGRATE_FLAG_POSTCOPY)) {
return false; // Continue iterating
}

return true; // Transfer complete
}</pre>

=== Post-Copy Mode ===

<pre>int gsp_migration_postcopy(struct gsp_migration *mig) {
// Enable post-copy on destination
migrate_payload_t pc = {
.phase = MIGRATE_PHASE_TRANSFER,
.subcmd = 11, // Dirty tracking
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.flags = MIGRATE_FLAG_POSTCOPY,
.token = mig->token,
};

// Page fault handler on destination
gsp_page_fault_handler = gsp_migration_page_fault;

return gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE, &pc, sizeof(pc), NULL, 0);
}</pre>

== Integration with BAR/WPR ==

=== BAR State Migration ===

<pre>int gsp_migration_bar_save(struct gsp_migration *mig) {
// Save BAR1 aperture, force access state
struct bar_save_msg bar_save = {
.vf_id = mig->vf->id,
.bar = 1,
};
return gsp_rpc_send(mig->vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_SAVE, &bar_save, sizeof(bar_save),
mig->bar_state_buf, &mig->bar_state_size);
}

int gsp_migration_bar_restore(struct gsp_migration *mig) {
// Restore BAR1 on destination
struct bar_restore_msg bar_restore = {
.vf_id = mig->dst_vf,
.bar = 1,
.data = mig->bar_state_buf,
.data_size = mig->bar_state_size,
};
return gsp_rpc_send(mig->vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_RESTORE, &bar_restore, sizeof(bar_restore));
}</pre>

=== WPR Region Migration ===

<pre>int gsp_migration_wpr_save(struct gsp_migration *mig) {
// Save WPR regions for VF
migrate_payload_t wpr = {
.phase = MIGRATE_PHASE_PREPARE,
.subcmd = 4, // WPR snapshot
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.token = mig->token,
};
return gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE, &wpr, sizeof(wpr), NULL, 0);
}</pre>

== Error Handling ==

{| class="wikitable"
|-
! Phase
! Error
! Recovery
|-
| Prepare
| EBUSY
| Wait, retry
|-
| Prepare
| ENOMEM
| Reduce snapshot size
|-
| Transfer
| ETIMEDOUT
| Checkpoint, retry
|-
| Transfer
| EIO
| Failover to post-copy
|-
| Commit
| EINVAL
| Rollback to source
|-
| Commit
| ENOSPC
| Expand destination
|}

== Debugging ==

=== Migration Trace Points ===

<pre>trace_gsp_mig_phase(vf_id, phase, subcmd, token);
trace_gsp_mig_transfer(vf_id, pages, bytes, dirty);
trace_gsp_mig_commit(vf_id, status, verify);
trace_gsp_mig_error(vf_id, phase, error);</pre>

=== GSP Log Strings ===

Search for migration-related strings in firmware:

* Migration phase markers
* Page transfer progress
* Commit verification results

----

== Reference ==

* Primary Dispatcher MIGRATE branches: 0x057A1BDC, 0x057A1D54
* Jump Table 1: 0x04179ED8 (16 entries)
* Jump Table 2: 0x04179FBC (16 entries)
* Decompiled handlers: 28 functions in decompiled/

Introduction to GMMU

2026-10-12T04:40:58Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Introduction to the GMMU =

<blockquote>'''Scope.''' A from-first-principles introduction to the GPU Memory Management Unit (GMMU): what it translates, how its page tables work on our Turing reference, how the host module uses them, and how the GMMU relates to the CPU MMU and the platform IOMMU/SMMU. The architecture chapter is [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/05-gmmu-amap-hal-gr.md <code>../docs/03-architecture/05-gmmu-amap-hal-gr.md</code>]; the isolation policy authority is [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] (condensed in [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]]).</blockquote>

<blockquote>'''Evidence discipline.''' Hardware-format claims condense the tracked field catalogue [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md <code>../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md</code>] and live observations; numeric IDs, layouts and code anchors are cited, and no timing/cycle claim is made.</blockquote>

<blockquote>Status: 2026-10-11, tree <code>master @ 08f39c2f</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[proven]</code> hardware-format fact; <code>[hypothesis]</code> unexercised; <code>[design]</code> not built.</blockquote>

== 1. What the GMMU is ==

The '''GMMU (GPU Memory Management Unit)''' is the address-translation hardware inside the GPU. Every engine — GR (shaders and CUDA kernels), CE (copy engines), SEC2, the video engines — addresses memory through '''GPU virtual addresses (VAs)''', and the GMMU translates each VA into a physical address: an offset in GPU local memory ('''vidmem'''/VRAM), an address in system memory ('''sysmem'''), or a peer GPU's memory across a fabric.

It is a paged MMU, conceptually the CPU MMU's sibling:

* '''Multi-level page tables''' describe the mappings; the root is a '''page directory base (PDB)'''.
* A '''TLB''' caches translations so the walk is not paid per access.
* Translation failures raise '''faults''' (invalid/missing leaf, missing table), reported through engine fault buffers.
* '''Each context has its own page directory (VA space)''', so one context cannot name another context's mappings — this is the hardware isolation boundary the project builds multi-tenancy on.

What makes it ''not'' just "the CPU MMU on a GPU":

* The tables live in GPU-visible memory (vidmem or sysmem), and each directory entry says which aperture the next level lives in.
* Entries carry GPU-specific '''aperture''' (which physical space) and '''memory-kind''' fields that the CPU MMU does not have.
* The "physical" addresses inside the tables are '''guest-physical addresses or emulated-VRAM offsets''' in our virtualized setting; the host module owns the resolution to real host addresses (§4.2). On a bare-metal GPU the same structures point at real vidmem/sysmem.

For this project the GMMU matters three ways: the guest driver programs it exactly as on bare metal (no guest patches); the guest's page tables are the ground truth for ''where its memory actually is''; and the per-context page directory is the only proven hardware isolation primitive available on the reference generation (§4.2).

== 2. The three MMUs: CPU MMU, GMMU, IOMMU/SMMU ==

{| class="wikitable"
|-
!
! CPU MMU
! GMMU
! IOMMU / SMMU
|-
| Translates
| process VA → host physical address
| engine GPU VA → GPU physical (vidmem / sysmem / peer)
| device DMA address → host physical address
|-
| Lives in
| the CPU core
| the GPU
| the host platform (IOMMU: Intel VT-d / AMD-Vi) or the SoC (SMMU, ARM)
|-
| Programmed by
| the OS kernel (per-process tables)
| the GPU driver / RM (the ''guest'' driver here)
| the hypervisor / OS (per-device or per-VM domains)
|-
| Table unit
| per-process page tables (e.g. CR3)
| per-context page directory (VA space)
| per-device/per-VM translation domain
|-
| Isolation scope
| process ↔ process
| context ↔ context (engine-level)
| VM/device ↔ VM/device (platform DMA)
|-
| In our stack
| guest and host kernels
| guest driver builds the tables; host walks or plants its own
| hypervisor enforces it; the module resolves GPAs itself when it needs host addresses
|}

They are three instances of the same idea — ''page tables plus a TLB give virtual memory and isolation to an agent'' — at three different points in the address path, owned by three different software layers. In a virtualized guest a GPU access passes through '''two''' of them:

<pre>engine (GR / CE)
│ GPU VA
▼
GMMU (inside the GPU, driven by the guest's page tables)
│ guest physical address (GPA) / GPU local address
▼
IOMMU / SMMU (in the host platform, programmed by the hypervisor for this VM)
│ host physical address (HPA)
▼
host DRAM / real VRAM</pre>

The two stages are independent: the '''GMMU''' decides ''which guest page'' an engine touches; the '''IOMMU/SMMU''' decides ''whether this device may reach that page at all'' and where it lives in host memory. GMMU isolation between guest contexts is the GPU driver/RM's job; IOMMU isolation between VMs is the platform's job.

'''How the two meet in this project:'''

* In real vGPU, the host vGPU manager uses the platform IOMMU to establish the GPU's view of guest memory. Here the host module does the equivalent resolution itself.
* On the reference host the physical GPU's IOMMU group is an '''identity domain''' (all-RAM identity) and the mediated device registers as an '''emulated-IOMMU VFIO device''': its type-1 container holds guest DMA mappings privately and no kernel <code>iommu_domain</code> exists for it, so <code>iommu_iova_to_phys</code> cannot translate guest GPAs. Guest RAM is backed by a QEMU shared file and the module translates GPA→HPA through that file's page cache, behind the active-mdev gate (a per-mdev-gated translation, ''not'' itself an IOMMU guarantee).
* The project rule is explicit: '''memory isolation is the platform IOMMU's job, and we never weaken it''' — DMA only through the hypervisor-provided interface for the device's own VM, no global mappings, fail closed if a required isolation property cannot be established.
* On aarch64 hosts the '''SMMU''' plays the IOMMU's role; page sizes, cache attributes and SMMU behaviour are on the host-portability list for the Grace/GB10 targets.

== 3. How the GMMU works (Turing reference) ==

=== 3.1 Page-tree geometry ===

The Turing page-tree path inherits the Pascal/Volta geometry and uses the '''VER2''' entry encoding. Depths run root (0) to leaf (4); VA width is '''49 bits''':

{| class="wikitable"
|-
! Depth
! HW level
! VA bits
! Index bits
! Entry size
! Role
|-
| 0
| PDE3
| 48:47
| 2
| 8 B
| pointer PDE
|-
| 1
| PDE2
| 46:38
| 9
| 8 B
| pointer PDE
|-
| 2
| PDE1
| 37:29
| 9
| 8 B
| pointer PDE (<code>NO_ATS</code> set here)
|-
| 3
| PDE0
| 28:21
| 8
| 16 B (dual) / 8 B (2 MB leaf)
| dual 64K/4K PDE, or a 2 MB leaf PTE
|-
| 4
| PTE
| 20:12 (4 KB) / 20:16 (64 KB)
| 9 / 5
| 8 B
| leaf PTE
|}

Supported page sizes are '''2 MB''' (depth 3), '''64 KB''' and '''4 KB''' (depth 4). A leaf entry alone does not say 4 KB vs 64 KB — the table's depth and page-size context do — so decoders must report <code>4k_or_64k</code> rather than guess.

=== 3.2 Entry encoding (VER2) ===

'''Leaf PTE (8 bytes):'''

{| class="wikitable"
|-
! Field
! Bits
! Meaning
|-
| <code>valid</code>
| 0
| entry is present
|-
| <code>aperture</code>
| 2:1
| 0 video, 1 peer, 2 system coherent, 3 system non-coherent
|-
| <code>volatile</code>
| 3
| cache/volatility bit
|-
| <code>encrypted</code>
| 4
| encrypted memory
|-
| <code>privilege</code>
| 5
| privileged access
|-
| <code>read_only</code>
| 6
| write protection
|-
| <code>atomic_disable</code>
| 7
| atomics disabled
|-
| <code>address_sys</code>
| 53:8
| system-aperture address bits (PA 53:12)
|-
| <code>address_vid</code>
| 32:8
| video/peer address bits (PA 36:12)
|-
| <code>peer_id</code>
| 35:33
| peer/fabric id (peer aperture)
|-
| <code>comptagline</code>
| 55:36
| compression tag (overloaded as high address bits for the 47-bit peer case)
|-
| <code>kind</code>
| 63:56
| memory kind id
|}

'''Directory PDE (8 bytes)''' is the same shape with one discriminator change: bit 0 is <code>is_pde_pointer</code> (0 = pointer to a lower table, 1 = a 2 MB leaf PTE), and '''aperture 0 means invalid at directory level''' (it means video memory at leaf level). Depth 2 sets <code>NO_ATS</code> (bit 5).

'''Dual PDE0 (16 bytes)''' holds two halves for mixed 64K/4K tables: BIG (low 64 bits) and SMALL (high 64 bits); bit 0 discriminates the whole slot.

Live guest note: the client's leaf entries in the format we observe are <code>PA[47:12] | flags</code>, with <code>flags = 0xe</code> for a valid system-memory leaf. The <code>kind</code> field carries numeric memory-kind ids (243 named values); common compute values are <code>0x00 PITCH</code>, <code>0x06 GENERIC_MEMORY</code>, <code>0x07 INVALID</code>, <code>0x0F SMSKED_MESSAGE</code>, <code>0xFE GENERIC_16BX2</code>.

The decoder (<code>sim/gmmu_pte.py</code>, docs/43) is decode-only: unsupported encodings fail closed, set bits not covered by a named field are reported as reserved, and nothing is guessed.

=== 3.3 Walking, caching and faults ===

A walk starts at the channel's '''page-directory base''' and follows <code>PDE3 → PDE2 → PDE1 → PDE0 → PTE</code>; the leaf yields the physical address, page size, aperture and access permissions. The TLB caches results, and there is a second, coarser cache that matters operationally:

* '''PTE fetch-line rule (learned live):''' the engine caches PTE fetches at '''128-byte (16-PTE) granularity''', and the forwarding code has '''no invalidate path'''. A runtime PTE write into a line the GMMU has already walked is invisible to the engine. Consequences: page-table slots for a guest window are '''pre-built before any engine access''', and each forwarded copy pair consumes a whole fresh line from a pre-zeroed pool (16 page-table pages, capacity 512 pairs per boot in the current design) — a line is never reused within a boot.
* '''Faults:''' invalid or missing leaves and missing tables raise GMMU faults; the driver allocates fault buffers whose sizes come from <code>INTERNAL_GMMU_GET_STATIC_INFO</code> (control <code>0x20800a59</code>): '''64 KB replayable / 4 KB legacy''' — the values the emulation serves, because a wrong value breaks UVM early.

== 4. How the GMMU is used ==

=== 4.1 Guest side (the stock driver) ===

* The guest driver allocates VA spaces and '''builds the page tables''', then publishes the directory per channel/context through <code>SET_PAGE_DIRECTORY</code> (<code>hVASpace</code> + <code>chId</code>; the host records the directory per owner in <code>kernel/gsp_rm_emulation.c:3307-3319</code>).
* Memory descriptors carry an aperture: <code>ADDR_SYSMEM = 1</code>, <code>ADDR_FBMEM = 2</code>.
* '''Server-owned ranges:''' <code>COPY_SERVER_RESERVED_PDES</code> (control <code>0x90f10106</code>) tells the client that some page-table ranges are populated by the "server" (us). The exact division of labour for <code>RESERVE_ENTRIES</code>/<code>RELEASE_ENTRIES</code> and invalidations is a tracked known-unknown.
* Channels submit push buffers that reference GPU VAs; the engines walk the GMMU to reach the data.

=== 4.2 Host side (<code>vgsp.ko</code>) ===

* '''Truth.''' The host resolves guest VAs by '''walking the driver-published page directories itself''' (the trusted GMMU walk in <code>kernel/gsp_sema_service.c</code>, VER2 geometry at <code>:1058-1072</code>, decode <code>:1093</code>, walk <code>:1174</code>), plus injected pins and content-keyed rescue when neither names a mapping. This is what lets the software path apply a semaphore release to the correct host page.
* '''Forwarding.''' For real CE copies the module builds its '''own scratch page tables''' that map already-resolved host pages (guest RAM through the shared file; shadow pages through the emulated-VRAM mapping), under the fetch-line rules of §3.3. Real guest page tables are ''not yet'' programmed into the real MMU — guest submission on hardware is the end state (P6).
* '''Emulation.''' Guest vidmem is a private host-RAM buffer per mdev (<code>kernel/bar_init.c:68-77</code>); FB-aperture accesses stay inside the owning mdev's buffer, and guest RAM is reached only through the hypervisor's per-mdev DMA interface (<code>vfio_dma_rw</code>).
* '''Isolation.''' The per-context page directory is the '''only proven hardware isolation primitive''' on this generation: the VER2 PTE/PDE has '''no tenant/ACL/PASID field''' (proven absence in the named field set; <code>peer_id</code> is a fabric peer selector, not a security domain). <code>read_only</code> (bit 6) is a proven ''format'' field, but its ''enforcement'' is a <code>[hypothesis]</code> until the M7 supervised probe. The multi-tenant design therefore places decisions in '''topology''' (which directory a page appears in, which channel may name it) plus software validation and audit in the execution plane — see [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]].
* '''Versioning.''' GMMU controls, static info and entry formats are version/hardware-keyed; formats are expected to differ per generation and must be re-derived by measurement, never guessed.

=== 4.3 Tooling built around it ===

{| class="wikitable"
|-
! Tool / doc
! Question it answers
|-
| PTE decoder <code>sim/gmmu_pte.py</code> + [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md docs/43]
| decode a captured entry field-by-field (decode-only, fail closed)
|-
| Walker <code>sim/gmmu_walk.py</code> + [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/48-GMMU-WALK-AND-LAUNCH-AUDIT.md docs/48]
| walk a captured PDB to the leaf mapping or the exact fault (4K/2M validated against a live captured channel PDB; 64K modelled)
|-
| Launch audit <code>harness/launch_audit.py</code> + docs/48
| where the engine would read/write each leg of a submission: <code>ok</code> / <code>pa_mismatch</code> / <code>unmapped</code> / <code>access_denied</code>
|-
| Oracle workflow [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/44-GMMU-ORACLE-AND-WORKFLOW.md docs/44]
| compare simulated vs observed GMMU/FBIF register values, per register, with fidelity labels
|-
| Observe-only windows [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/42-GMMU-FBIF-OBSERVABILITY.md docs/42]
| record what firmware touches in the GMMU register windows without faking semantics
|}

== 5. Limits and open questions ==

* '''<code>read_only</code> enforcement''' is unproven (M7 supervised probe); '''encryption (CCS) semantics''' are unstudied. Neither can be assumed as a tenant boundary.
* '''Invalidation semantics''' are not fully modelled: the 128-byte fetch-line observation is live evidence, not a complete invalidate contract.
* The '''64K dual path''' is modelled from the format definitions, not yet validated against a capture.
* '''Newer generations''' (Ampere/Hopper/Blackwell, GB10) are expected to change the entry format; each must be re-derived. RM 610 work sits behind its own human approval gate.
* '''No timing/cycle claims''' are made anywhere here; zero hits in a run are a coverage fact, not proof of dead code.
* Multi-mdev isolation is first a '''service-state-ownership''' problem (host tables keyed by guest-chosen handles/VAs alone collide); GMMU policy is built on top of that fix, not instead of it.

== 6. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/05-gmmu-amap-hal-gr.md <code>../docs/03-architecture/05-gmmu-amap-hal-gr.md</code>] — architecture chapter: GMMU, AMAPlib/HALs, channels, runlists, GR.
* [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]] — the isolation design condensed.
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] — canonical multi-tenant GMMU policy.
* [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — how guest work becomes physical execution; the fetch-line rule in context.
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — page-table seams and the guest-memory reachability model.
* [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the IOMMU reachability note.
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — threat model and the IOMMU rules.
* [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md <code>../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md</code>], [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/44-GMMU-ORACLE-AND-WORKFLOW.md docs/44], [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/48-GMMU-WALK-AND-LAUNCH-AUDIT.md docs/48] — the field catalogue and the offline walk/audit tooling.

Install and First Run

2026-10-12T04:40:58Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Install and First Run =

<blockquote>'''What this page is.''' From a fresh Ubuntu host with a supported NVIDIA GPU to a guest VM running a stock NVIDIA driver and <code>nvidia-smi</code>, in about half an hour. Condensed from the maintained guide [https://git.vgpu.io/open-iov.git/tree/docs/02-getting-started/01-install-and-first-run.md <code>../docs/02-getting-started/01-install-and-first-run.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> verified path; <code>[target]</code> roadmap item.

== 1. Prerequisites ==

'''Host'''

* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a <code>[target]</code>);
* an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the reference — see [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]]);
* a working '''host''' NVIDIA driver (<code>nvidia-smi</code> runs on the host; Open-IOV does not modify it);
* IOMMU enabled (<code>intel_iommu=on</code> / <code>amd_iommu=on</code>), hugepages configured, kernel headers for the running kernel;
* <code>qemu-system-x86_64</code> with KVM; libvirt optional;
* root (or sudo) for module load and mdev creation.

'''Guest'''

* Ubuntu 24.04 server (the reference guest), or any Linux with the matching stock NVIDIA driver available;
* '''no Open-IOV component at all''' — that is the point ([[System Overview|<code>System-Overview.md</code>]]).

== 2. Get the code and build ==

<pre>git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps # build/runtime dependencies
./install.sh nvidia # optional: NVIDIA OpenRM modules + build glue
./install.sh firmware # GSP firmware image (verified)
./install.sh build # kernel module + userspace</pre>

<code>install.sh</code> is safe to re-run. The manual equivalent, and the order that matters:

<pre>make kernel # first: generates Module.symvers
./scripts/build_nvidia_openrm.sh # optional (see below)
make userspace</pre>

'''Build order matters.''' NVIDIA's OpenRM modules resolve symbols that <code>vgsp.ko</code> exports; building them in the wrong order produces <code>Unknown symbol</code> failures at load time.

'''The OpenRM stage is optional.''' It is needed only for the native vGPU-management workflows in which NVIDIA's modules are built against the symbols <code>vgsp.ko</code> exports. The QEMU/emulated path uses the distribution driver that already manages the physical GPU; <code>load-stack</code> skips the OpenRM submodule when a distribution driver is present.

== 3. Load the stack and create a virtual GPU ==

<pre>./scripts/stack_manager.sh status # what is currently loaded
./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia
./scripts/stack_manager.sh mdev-create # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/</pre>

The mdev device ''is'' the virtual GPU; everything after this treats it like any other VFIO device.

== 4. Launch a reference guest ==

<pre>./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev</pre>

To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly as any VFIO device:

<pre><hostdev mode='subsystem' type='mdev' model='vfio-pci'>
<source>
<address uuid='<mdev-uuid>'/>
</source>
</hostdev></pre>

'''Attach behind a root port, not on the root bus.''' QEMU rewrites a device's PCIe capability when it detects a root-complex-integrated endpoint, which changes the link capabilities the guest driver sees. The reference script does this for you.

== 5. Install the stock driver in the guest ==

Inside the guest:

<pre>sudo apt update
sudo apt install -y linux-headers-generic build-essential
# The validated combination is the stock 570.144 Linux runfile (the same
# driver version as the emulation target).
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot</pre>

Verify:

<pre>nvidia-smi # must list the virtual GPU
lsmod | grep -c nvidia</pre>

If <code>nvidia.ko</code> fails to probe, capture host dmesg, guest dmesg, guest <code>lspci -vvv</code> and the guest module parameters, and file an issue.

== 6. First CUDA check ==

<pre># in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery</pre>

'''Expectation today''' (<code>[live]</code> unless noted): driver probe, <code>nvidia-smi</code>, device enumeration and the full CUDA '''driver-API''' smoke complete end to end (<code>ALL OK</code>, 64-byte payload round-trips). CUDA '''kernel execution''' is the current frontier: a guest launch is forwarded and reaches the engine queue, but engine execution is not yet observed (<code>[wall]</code>) — see [[Project Status|<code>Project-Status.md</code>]] and [[The Execution Frontier|<code>Execution-Frontier.md</code>]] before drawing conclusions from a kernel that does not run.

== 7. Day-to-day operations ==

<pre>./scripts/stack_manager.sh up # build all + load + create mdev
./scripts/stack_manager.sh status # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test # host test suite
./scripts/stack_manager.sh unload # correct-order teardown
./scripts/stack_manager.sh cleanup-crash</pre>

Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM → firmware; unload is the reverse. Do not <code>rmmod</code> by hand while a VM holds the mdev.

== 8. Watching the emulation work ==

The module logs with the prefix '''<code>vgsp-<subsystem>:</code>''' (for example <code>vgsp-rpc:</code>, <code>vgsp-sema:</code>, <code>vgsp-work:</code>) plus per-RPC traces:

<pre>sudo dmesg -w | grep -i 'vgsp' # live
sudo dmesg -T | grep -i 'vgsp' | tail -50</pre>

Developer debug knobs (module parameters) are documented in the repository's debugging workflow document.

== 9. Troubleshooting index ==

{| class="wikitable"
|-
! Symptom
! Start here
|-
| Module will not load
| <code>dmesg</code>, <code>modinfo vgsp.ko</code>, stale module state → <code>cleanup-crash</code>
|-
| No mdev type for your GPU
| [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]]
|-
| Guest sees wrong PCIe link speed/width
| attach behind a root port (section 4)
|-
| <code>nvidia.ko</code> probe fails in guest
| guest dmesg + <code>lspci -vvv</code> + [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]]
|-
| <code>nvidia-smi</code> works, a CUDA kernel does not run
| expected today — the execution frontier, not your setup ([[The Execution Frontier|<code>Execution-Frontier.md</code>]])
|-
| VM boot fails or the guest hangs
| <code>scripts/stack_manager.sh cleanup-crash</code>, then reboot
|}

== 10. Uninstall ==

<pre>./scripts/stack_manager.sh unload</pre>

The project never modifies your host driver's files; removing the module and its mdev devices returns the host to its previous state.

The Host Module (vgsp.ko)

2026-10-12T04:40:57Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= The Host Module (vgsp.ko) =

<blockquote>'''Scope.''' The host-side kernel module that presents one virtual Turing-class GPU per <code>mdev</code> instance: the virtual device, the emulation engine, the guest-memory model and the work-forwarding design. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>]; the docs tree wins on disagreement with this page.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> demonstrated live, being productized; <code>[gated]</code> implemented default-off behind supervised gates; <code>[wall]</code> known blocking limit; <code>[target]</code> designed, not built; <code>[planned]</code> roadmap.</blockquote>

== 1. Role and position ==

<pre>guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor
(KVM/QEMU or Xen: VFIO mdev transport, IOMMU)
│
vgsp.ko: virtual device · emulation core · memory model · work path
│
physical GPU + GSP firmware, driven by this module (host driver unmodified)</pre>

The module registers as an '''mdev parent''' on the physical GPU, exposing one device type per supported profile; each <code>mdev</code> is an independent virtual GPU ([[System Overview|<code>System-Overview.md</code>]], [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]]).

== 2. The virtual device ==

The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability list). Some fields exist only for validation: PCIe link capabilities (the hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows.

{| class="wikitable"
|-
! BAR
! Size
! Backing
! Contents
|-
| BAR0
| 16 MB
| host <code>vzalloc</code> shadow
| boot, doorbells, VF registers, PRAMIN window, VBIOS
|-
| BAR1
| 256 MB
| host <code>vzalloc</code> vidmem buffer
| guest-visible VRAM aperture
|-
| BAR3
| —
| instance window
| instance/PRAMIN behaviour
|}

MMIO is served by VFIO region callbacks or directly from the shadow when mapped into the guest. BAR1 is host RAM today — correctness first: the real GPU cannot read it, so forwarding must later migrate objects to real VRAM or expose DMA-mapped system memory [target] ([[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]]).

== 3. The emulation core ==

A kernel thread drives the device's "firmware" behaviour [live]:

<pre>BAR0 doorbell / VF-register write → read msgq element from guest memory
→ dispatch RPC → write reply → inject MSI-X (eventfd → irqfd)</pre>

Guest memory is reached via <code>vfio_dma_rw()</code> from kernel-thread context. The RPC philosophy: correct shape first (or the driver walks off into uninitialized memory), plausible values second (values must be self-consistent), never invent input (in/out parameter blocks are echoed; zeroing the <code>GSP_RM_ALLOC</code> reply once faulted <code>libcuda</code>). Handlers cover boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries, reserved-PDE contracts, channel/CE allocation and an unknown-control fallback. Push buffers are decoded as method streams and executed in software — memsets, copies, semaphore releases — a '''correctness bridge, not the data plane''' [wip]; emulation-mode CUDA is verified end to end (<code>vectorAdd</code> over 1M elements, <code>bad=0</code>) [live]. Physical-GSP boot is default-off [gated]; the firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are not yet on the live path [target].

== 4. The guest-memory model ==

'''What the client gives us:''' channel memory descriptors (<code>NV_MEMORY_DESC_PARAMS</code>) with ''physical'' addresses (instance memory, method buffer, userd, ramfc, error notifiers) and the page-table contract for server-owned VA ranges (<code>COPY_SERVER_RESERVED_PDES</code> — table-level PAs, page shifts, VA range, ''empty'' leaves). It gives us none of its own sysmem maps.

'''What we derive instead''' [live] — the module rebuilds the missing mappings from the guest's own bookkeeping, with no cooperation from the guest driver:

# '''VA↔PA pairs live in guest memory''': UVM and RM record both the GPU VA and the kernel CPU VA per object (<code>uvm_rm_mem_t.vas[CPU]/vas[GPU]</code> +0x30/+0x38).
# '''CPU VAs are direct-map addresses''': <code>PA = cpu_va - page_offset_base</code>, the base recovered from guest memory or <code>/proc/kallsyms</code> during development.
# '''Exception:''' control-device <code>mmap()</code> pages have no leaf PTEs; verified pairs arrive via <code>sema_map_add</code> until a durable join exists. The page-granular cache is fed by a trusted GMMU walk ([[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]]).

'''Load-bearing copy invariants:''' (1) a copy runs at most once per GPFIFO entry — replays are suppressed by entry-word identity plus an executed-operation bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were contiguous in one observed 1 MB range); (3) a consumed slot's work never runs later — the per-put-epoch mark is authoritative until a put wrap.

== 5. The forwarding path (design) ==

<pre>guest push submitted
→ detect (doorbell / GPPUT / push-buffer scan), decode the method stream
→ EMULATE state/notification (release, membar, fence) or
FORWARD real engine work (GR launch, CE copy, memop)
→ FORWARD: submit equivalent work on a real channel bound to the physical GPU
→ relay completions (semaphore / fault / interrupt) back into guest memory</pre>

Proxying through the distribution host driver has '''no callable entry point from kernel context''', so the module '''owns the device and runs its real GSP firmware as the resource-manager server''', with emulation as the control plane. CE copies, release and fault relay are implemented default-off and verified live [gated]; GR execution is the open wall [wall] — a forwarded launch reaches the front-end; the inline launch-descriptor install is unobserved ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 6. Debug parameters ==

<code>/sys/module/vgsp/parameters/*</code> makes guest memory and the emulated device inspectable without rebuilding:

{| class="wikitable"
|-
! Parameter
! Purpose
|-
| <code>scan_guest_ram=<v></code> / <code>scan_vidmem=<v></code>
| find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow
|-
| <code>scan_seq=<v1,...></code> / <code>scan_pt=<pa></code> / <code>scan_masked=<v>,<mask></code>
| find a 32-bit sequence, entries pointing at a PA, or <code>(x & mask) == v</code>
|-
| <code>dump_vidmem=<off>,<len></code> / <code>fill_vidmem=<v></code>
| dump/scan or fill the VRAM shadow
|-
| <code>poke_guest=<gpa>,<v></code> / <code>poke_fill=<gpa>,<v>,<n></code>
| write a 32-bit value / fill a range in guest physical memory
|-
| <code>sema_probe=<gpu_va>[,count]</code> / <code>sema_map_add=<va>,<pa></code>
| probe a semaphore VA / teach one verified VA→PA pair
|-
| <code>dump_guest=<gpa>,<len></code>
| dump guest physical memory (capped)
|}

The physical-backend path adds a default-off set of backend controls (documented in the repository; workflows in [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/02-debugging-and-maintenance-workflows.md <code>../docs/05-operations/02-debugging-and-maintenance-workflows.md</code>]).

== 7. Build and symbol contracts ==

* <code>vgsp.ko</code> exports the <code>nvidia_vgpu_vfio_*</code> interface (<code>probe</code>/<code>remove</code>, ops get/set, mdev parent registration) to pair the device with OpenRM host modules.
* OpenRM modules built from source need <code>KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers</code> and must build '''after''' this module (missing symbols and stale build caches are common otherwise).
* On single-GPU developer systems the distribution driver owns the physical GPU and the module runs standalone; the OpenRM path is only for native-vGPU management, not QEMU.

== 8. Implemented vs designed ==

{| class="wikitable"
|-
! Area
! Status
|-
| mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live]
| implemented
|-
| Message queues + RPC + init controls (stock 570.144) [live]
| implemented
|-
| Guest sysmem access + VA↔PA discovery [wip]
| debug-grade, to be productized
|-
| Push decoding + semaphore completion [wip]
| demonstrated live, being productized
|-
| Emulation-mode CUDA end-to-end [live]
| repeatable 1M-element <code>vectorAdd</code>, <code>bad=0</code>
|-
| Guest-RAM physical path + server-owned page-table population [gated]
| default-off; GPA→HPA via shared-file page cache verified live on the CE
|-
| GPU work forwarding: CE copy, release and fault relay [gated]
| default-off, verified live; GR open [wall]
|-
| Execution backend (module-owned GPU, direct real-GSP session) [wall]
| boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried)
|-
| Real VRAM backing [target]; Windows, Xen, aarch64 [planned]
| designed, not implemented
|}

== 9. Related documentation ==

* Canonical chapter: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>] — the source of this condensation.
* [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]], [[BAR Emulation Specification|<code>BAR-Emulation-Specification.md</code>]], [[WPR Management|<code>WPR-Management.md</code>]]; [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]].

Hardware and Software Matrix

2026-10-12T04:40:57Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= Hardware and Software Matrix =

<blockquote>'''Companion summary.''' This page condenses [https://git.vgpu.io/open-iov.git/tree/docs/06-portability/hardware-software-matrix.md <code>../docs/06-portability/hardware-software-matrix.md</code>] — the canonical support matrix: the support ladder, the current status of every host/GPU/guest/hypervisor combination, and what adding a combination requires. The validation process behind every support decision is [https://git.vgpu.io/open-iov.git/tree/docs/06-portability/03-device-validation-and-porting-sequence.md <code>../docs/06-portability/03-device-validation-and-porting-sequence.md</code>]. When this page and the docs tree disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[reference]</code> continuously tested, bugs block releases; <code>[supported]</code> tested per release; <code>[experimental]</code> partial or bring-up; <code>[target]</code> designed for, bring-up in progress; <code>[design]</code> design direction, not the current build.

== 1. Support ladder ==

Support means '''tested and documented''', not "should probably work".

{| class="wikitable"
|-
! Level
! Meaning
|-
| '''Reference'''
| continuously tested; bugs block releases
|-
| '''Supported'''
| tested per release; documented workarounds if any
|-
| '''Experimental'''
| known to boot/test partially; may need configuration
|-
| '''Target'''
| architecture designed for it; bring-up in progress
|-
| '''Out of scope'''
| not planned
|}

== 2. Host platforms ==

{| class="wikitable"
|-
! Platform
! Status
! Notes
|-
| x86_64 + NVIDIA Turing (TU104)
| '''Reference'''
| the development hardware (RTX 2080-class); the only Reference platform today
|-
| x86_64 + Ampere / Ada consumer
| '''Target'''
| profile work; expected to be configuration + measurement
|-
| x86_64 + Hopper / datacenter
| '''Target'''
| requires MIG-aware policy for multi-tenancy
|-
| aarch64 + Grace-Hopper (GH200)
| '''Target'''
| host-architecture work in the memory model
|-
| aarch64 + Grace-Blackwell
| '''Target'''
| as above, newer firmware/driver family
|-
| GB10 (DGX Spark, Surface-class RTX laptops)
| '''Target'''
| same architecture; firmware/ROM quirks to be measured
|-
| Other NVIDIA generations
| '''Experimental'''
| bring-up path in the canonical document §5
|}

The design direction is that '''each new host platform should require configuration + verification, not new code''' — not the current state: selection is compile-time today and the module is built for the TU104 profile.

== 3. Guest operating systems ==

{| class="wikitable"
|-
! Guest
! Status
|-
| Linux x86_64 (stock <code>nvidia.ko</code>, CUDA)
| '''Experimental''' — probe, <code>nvidia-smi</code> and the driver-API smoke pass; CE-class copies are forwarded default-off; for GR the guest launch reaches the front-end but no grid has dispatched yet (see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]); no continuous guest testing yet
|-
| Linux aarch64
| '''Target'''
|-
| Windows x86_64 (WDDM driver, CUDA, 3D)
| '''Target'''
|-
| Other Unix (FreeBSD, ...)
| Out of scope until demand
|}

'''Guest rule:''' the guest never contains Open-IOV code — no guest agent, no driver patch, no kernel parameters beyond what the stock driver needs.

== 4. Hypervisors and transports ==

{| class="wikitable"
|-
! Hypervisor
! Status
! Transport
|-
| KVM + QEMU (VFIO mdev)
| '''Reference'''
| VFIO mdev + eventfd IRQ + DMA interface
|-
| libvirt (mdev attach)
| '''Supported'''
| same, via libvirt XML
|-
| Zen / KVM on other device models
| '''Experimental'''
| as above
|-
| Xen (incl. Argo/OpenXT)
| '''Target'''
| Xen device model + event channels
|-
| Qubes OS / Qubes OS Air
| '''Target'''
| compartmentalized VMs; GPU per compartment
|-
| Cloud hypervisor / crosvm
| '''Experimental'''
| VFIO-based; needs testing
|}

The transport boundary — registration, guest-memory access, interrupt injection — is deliberately narrow, so a new transport is small and isolated.

== 5. What portability means today ==

<pre> host architecture GPU family driver/firmware family
┌─────────────────┐ ┌──────────────────┐ ┌────────────────────┐
│ x86_64 │ │ Turing (TU104) │ │ 570.144 (anchor) │
│ aarch64 (Grace) │ │ Ampere / Ada │ │ newer families │
│ GB10-class │ │ Hopper / datac. │ │ (gated, pending │
└────────┬────────┘ └────────┬─────────┘ │ review) │
│ │ └─────────┬──────────┘
└──────────┬──────────┴───────────────────────┘
▼
one combination = hardware profile + version profile
▼
evaluation ladder + evidence → matrix entry:
"hardware X + driver Y + firmware Z, tested at rung R on date D"</pre>

The configuration surface lives in <code>config/</code>: <code>config/hardware/<arch>.yaml</code> (PCI IDs, BARs, memory geometry, engine lists) and <code>config/versions/<driver>.yaml</code> (RPC/control inventories, capability tables).

<blockquote>'''These files are design targets, not live configuration.''' No code reads them today: selection is compile-time and the module is built for the TU104 profile. Treating the YAML surface as authoritative — or claiming "configuration-driven portability" — is aspirational until the profiles are wired into the module's load path. A profile is the ''unit of support''; what is missing is the code that reads the profile files, not the schema.</blockquote>

== 6. Adding a new combination ==

'''New GPU / hardware profile.'''

# Create a hardware profile: PCI IDs, BAR layout and sizes, memory geometry (FB size, BAR windows), engine inventory, interrupt topology, and the register values the guest driver validates during probe.
# Run the five-rung evaluation ladder ([https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>]).
# Record the differences; prefer configuration over code.
# Publish the profile, the evidence, and any workarounds.

Firmware-side analysis uses the risc-v-gsp-rm-debugger ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]): first-touch probes and per-pair comparisons classify each difference as profile data or logic before any profile change lands.

'''New firmware version.''' Images are versioned under <code>firmware/<SERIES>-GSP/<VERSION>/</code> with signatures and analysis artifacts; the work is capture, inventory diff, decode against that version's reference, then configuration/handlers and a re-run of the ladder.

'''New driver version.''' Run the trace, diff the RPC/control inventory, add version-keyed handlers, re-run the ladder. Newer driver/firmware families are gated pending review before any work starts.

'''Evidence expectations.''' A combination is entered in the matrix only with the profile fields recorded; the firmware/device pairing verified by hash and version string; every first-touch difference explained as profile or logic; per-pair comparison results with session hashes; the metric table with its reference-capture method and stated fidelity limits; and a supervised physical validation for the support claim. "Tested and documented" is literal: an untested or simulator-only combination stays '''Target''' or '''Experimental'''.

== 7. Anchor versions and engine-list portability ==

The anchor is guest CUDA driver (<code>libcuda</code>) '''570.144''' with GSP firmware '''570.144'''; the host oracle is <code>libcuda</code> '''570.211.01''' with CUDA toolkit '''12.0'''. The versioned surface is the RM ioctl/class/control interface the userspace driver issues — not a stable public API; see [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]].

The engine inventory and its class-list filtering are per-architecture, per-driver data. An earlier gap was concrete: a missing async copy engine blocked context creation until the inventory was completed — which is why it belongs in the hardware profile, validated against the host oracle.

== 8. Known portability risks ==

{| class="wikitable"
|-
! Risk
! Mitigation
|-
| Page-size/cache-attribute differences on aarch64
| keep the memory model abstract; test on Grace early
|-
| Windows driver ROM/VBIOS expectations
| explicit ROM/VBIOS provisioning in the device profile; measured bring-up
|-
| Xen interrupt/memory interfaces differ
| narrow transport interface + Xen backend plan
|-
| Vendor ROMs and laptop firmware variants (GB10/Surface)
| per-device overrides in the profile system
|-
| Driver-family drift (580 and newer)
| version-keyed handlers; matrix-driven testing
|}

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/06-portability/hardware-software-matrix.md <code>../docs/06-portability/hardware-software-matrix.md</code>] — canonical matrix: ladder, status, procedures; the validation process behind every support claim is in [https://git.vgpu.io/open-iov.git/tree/docs/06-portability/03-device-validation-and-porting-sequence.md <code>../docs/06-portability/03-device-validation-and-porting-sequence.md</code>].
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — the five-rung evaluation ladder and evidence rules.
* [[Project Status|<code>Project-Status.md</code>]] and [[System Overview|<code>System-Overview.md</code>]] — current status and component map.
* [[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]] — firmware-side portability analysis.
* [[HAAEEP and Collaboration|<code>HAAEEP-and-Collaboration.md</code>]] — the edge use case and collaboration models.

HAAEEP and Collaboration

2026-10-12T04:40:56Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= HAAEEP and Collaboration =

<blockquote>'''Companion summary — target architecture.''' This page condenses [https://git.vgpu.io/open-iov.git/tree/docs/08-ecosystem/haaeep-and-collaboration.md <code>../docs/08-ecosystem/haaeep-and-collaboration.md</code>], the canonical description of the '''High Assurance Agent Edge Environment Platform (HAAEEP)''': a target architecture that Open-IOV exists to enable, and the collaboration models around it. HAAEEP is a proposal and a roadmap, not a shipped product. When this page and the docs tree disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[proposal]</code> architecture and roadmap; <code>[target]</code> milestone to reach; <code>[goal]</code> end state; <code>[design]</code> integration point designed, not built.

== 1. What HAAEEP is ==

An AI agent harness (Claude Code, OpenCode, or any future equivalent) is an untrusted, capability-constrained workload: it runs generated code, holds credentials, and talks to a model. In a ''cloud'' deployment its requests leave the device; in a ''secure edge'' deployment — a workstation, a kiosk, a vehicle, a medical or industrial system — they must not.

HAAEEP is the platform shape for that edge deployment:

* inference runs '''locally''', in its own VM, with the GPU;
* agents run in '''their own VMs''', without the GPU and without the weights;
* other functions (desktop, networking, storage) run in '''other VMs''', each isolated;
* the whole stack is '''attestable''', so remote parties can verify it;
* and nothing in the design requires the model server, the agent, or the GPU driver to be modified.

Open-IOV is HAAEEP's GPU component. The broader motivation is in [https://git.vgpu.io/open-iov.git/tree/docs/01-orientation/02-the-ai-edge-and-ecosystem.md <code>../docs/01-orientation/02-the-ai-edge-and-ecosystem.md</code>].

== 2. Architecture ==

<pre>┌──────────────────────────────────────────────────────────────────────────┐
│ EDGE DEVICE — one GPU, local inference only │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │
│ │ Agent VM │ │ Inference VM │ │ Infrastructure │ │
│ │ capability- │────▶│ vLLM / SGLang │ │ desktop, net, │ │
│ │ constrained │ API │ owns GPU via │ │ storage VMs │ │
│ │ no GPU │ │ Open-IOV mdev or │ │ optional GPU accel│ │
│ │ │ │ xen-pci + relay │ │ via Open-IOV │ │
│ └─────────────────┘ └────────┬─────────┘ └───────────────────┘ │
│ │ │
│ ┌────────────────────────────────┴───────────────────────────────────┐ │
│ │ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air │ │
│ │ • Flask-XSM / HMX mandatory access control between VMs │ │
│ │ • TrenchBoot measured boot: firmware → hypervisor → OS │ │
│ │ • Open-IOV: per-VM virtual GPUs, IOMMU-enforced isolation │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘</pre>

== 3. Roles and trust ==

{| class="wikitable"
|-
! Component
! Trust
! Why
|-
| Agent VM
| untrusted
| runs generated code; may be compromised
|-
| Inference VM
| high (holds weights)
| isolated from agents by MAC and by device virtualization
|-
| GPU-owning VM (optional)
| high
| owns the physical device; can serve others through Open-IOV
|-
| Infrastructure VMs
| medium
| GPU acceleration optional; no access to weights
|-
| Hypervisor
| TCB
| enforced by the platform (Argo/OpenXT/Qubes)
|-
| Open-IOV device
| TCB-adjacent
| must fail closed; audited (see the security doc)
|}

== 4. Data flow ==

# An agent VM issues an API call to the inference VM over an internal, MAC-controlled channel.
# The inference VM's vLLM/SGLang (unchanged) serves the request with the GPU, using its KV-cache plugins (also unchanged).
# No request ever leaves the device: the platform's network policy forbids external inference endpoints by construction.

'''Optional topology — GPU relay domain.''' Where the platform prefers it (Xen with <code>xen-pci</code>), one domain owns the physical GPU and relays acceleration to other domains through Open-IOV: the process running the model is isolated from the domain that owns the silicon, and agents are isolated from both.

== 5. Why Open-IOV fits ==

{| class="wikitable"
|-
! Alternative
! Why it is not enough
|-
| Discrete passthrough
| gives the GPU to exactly one VM; HAAEEP needs the GPU in the inference VM ''and'' optional acceleration elsewhere
|-
| Process-level sharing (host CUDA with containers)
| no VM-level isolation, which is the whole point of the platform
|-
| Proprietary vGPU
| cannot build a sovereign, auditable, redistributable edge platform on a licensed host stack you cannot inspect or modify
|-
| Software GPU emulation
| cannot run models usefully
|}

Open-IOV is the component in this design that ''must'' be open, permissively licensed and version-durable — so it is the component we are building. Its isolation model is described in [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]].

== 6. Roadmap (proposal) ==

{| class="wikitable"
|-
! Milestone
! Depends on
|-
| CUDA execution in a guest (rung 3)
| rung 2 (runtime start) is green; CE-class work is forwarded with release and fault relay in place (default off); for GR the guest launch reaches the front-end but no grid has dispatched yet — the frontier is tracked in [[The Execution Frontier|<code>Execution-Frontier.md</code>]]
|-
| Two-VM coexistence (inference + agent) with isolation
| Open-IOV multi-tenancy + platform MAC
|-
| Xen/OpenXT or Qubes Air variant
| Open-IOV transport backends
|-
| Attested boot-to-inference chain
| TrenchBoot integration
|-
| Published reference platform (scripts, docs)
| all of the above
|}

Each milestone is documented as it lands, with the same evidence discipline as the rest of the project. Nothing in this table is claimed today: it is the proposal.

== 7. Collaboration models ==

'''Open source projects.''' Adopt the code under BSD-3-Clause (kernel modules are GPL-2.0 — see [https://git.vgpu.io/open-iov.git/tree/docs/09-reference/legal-and-licensing.md <code>../docs/09-reference/legal-and-licensing.md</code>]). We treat interface requirements as design inputs (transports, IRQ models, memory interfaces), provide the transport abstraction rather than asking for forks, and test against partner platforms in the matrix.

'''Commercial partners.''' Welcome, with no license fee: ship Open-IOV in products and differentiate above it; sponsor support for specific hardware/driver versions; co-develop under the same permissive terms.

'''Hardware vendors.''' Nothing is required. The goal is that new hardware works through configuration and measurement; if you want to accelerate support, share hardware and traces, and we publish what we learn.

'''Security researchers.''' The emulation's attack surface is small and documented. Break it. Findings and fixes are published, with credit.

== 8. Cross-project notes ==

{| class="wikitable"
|-
! Project
! Relationship
|-
| '''Qubes OS Air'''
| compartmentalization on modern hardware; Open-IOV makes GPU acceleration usable per compartment (especially AI workloads)
|-
| '''TrenchBoot'''
| measured/attested boot; combined, the platform can attest the inference path
|-
| '''Argo / OpenXT'''
| hypervisor-mediated exchange (HMX) and Flask-XSM MAC; Open-IOV respects and complements that policy
|-
| '''KVM/VFIO gaming'''
| established the fast-passthrough pattern; Open-IOV serves users who need ''multiple'' VMs or cannot dedicate the GPU
|-
| '''vLLM / SGLang'''
| remain the inference answer; Open-IOV isolates them without changing them; KV-cache plugins keep working
|}

== 9. How to help ==

* '''Platform builders''': bring your hypervisor; tell us the transport you need.
* '''Security engineers''': review the isolation model and the debug surface.
* '''AI-infrastructure engineers''': define the reference HAAEEP deployment with us (model server, KV cache, agent harness).
* '''Everyone''': run the evaluation ladder on your hardware and send the evidence.

Contact and participation channels are in [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>] §6 "How to get involved". We would rather have a hundred partners than a hundred forks.

== 10. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/08-ecosystem/haaeep-and-collaboration.md <code>../docs/08-ecosystem/haaeep-and-collaboration.md</code>] — canonical HAAEEP architecture, roadmap and collaboration models.
* [https://git.vgpu.io/open-iov.git/tree/docs/01-orientation/02-the-ai-edge-and-ecosystem.md <code>../docs/01-orientation/02-the-ai-edge-and-ecosystem.md</code>] — the AI-edge motivation and ecosystem stance.
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — the evaluation ladder behind every milestone.
* [[Project Status|<code>Project-Status.md</code>]] and [[System Overview|<code>System-Overview.md</code>]] — current status and component map.
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]] and [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — the GR frontier and how work reaches real hardware.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — what runs where, and how support is added.

The Guest Stack and CUDASW

2026-10-12T04:40:56Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= The Guest Stack and CUDASW =

<blockquote>'''Scope.''' What runs inside the guest, how the NVIDIA software layers sit on top of the device, and what the unmodified guest driver expects from the emulated GPU. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/02-guest-stack-and-cudasw.md <code>../docs/03-architecture/02-guest-stack-and-cudasw.md</code>]; the docs tree wins on disagreement with this page.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[gated]</code> implemented default-off behind supervised gates; <code>[wall]</code> known blocking limit; <code>[target]</code> designed, not built; <code>[planned]</code> roadmap.</blockquote>

== 1. Vocabulary: what "CUDASW" means here ==

'''CUDASW''' is the family of NVIDIA ''software'' components that sit between an application and the GPU driver. The project's compatibility target is the whole stack, not one library:

<pre>application
│ CUDA API (runtime)
libcudart.so ─────────────────────────────┐ runtime: device/stream/event
│ CUDA driver API │ management, kernel launch
libcuda.so ───────────────────────────────┤ driver: contexts, modules,
│ ioctls │ memory, channel submission
/dev/nvidia* ─────────────────────────────┘
nvidia.ko (RM + UVM + modeset)</pre>

Adjacent members of the family: '''libnvidia-ptxjitcompiler''' (PTX→SASS JIT — usually "just works" once the driver initializes), '''libnvidia-fatbinaryloader / nvrtc''', '''UVM''' (<code>nvidia_uvm.ko</code> — [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]]), the '''graphics stack''' (<code>libGLX_nvidia</code>, <code>libnvidia-glcore</code>, Vulkan ICD, <code>nvidia_modeset</code>/<code>nvidia_drm</code>) and the '''video stack''' (NVENC/NVDEC) — both roadmapped.

== 2. Layering rules the emulation must respect ==

# '''The kernel driver is the contract.''' Everything above it talks to <code>nvidia.ko</code> through stable ioctls, memory mappings and <code>/dev/nvidia*</code> nodes. If the kernel driver initializes and behaves, user space follows.
# '''User space is opaque to us.''' We never intercept user-space APIs. All hooks are in the device model (host side) or in the ''hardware interface'' the kernel driver sees (guest side).
# '''Version skew is normal.''' Driver version, CUDA version and firmware version are three independent axes. The compatibility target is "the driver family that matches the emulated firmware profile", with configuration-driven adaptation for others.

== 3. Kernel-driver anatomy (what the guest brings) ==

{| class="wikitable"
|-
! Component
! Role
! Why it matters here
|-
| RM core
| object model, handles, locks, device lifecycle
| drives the GSP boot and the whole RPC/control surface
|-
| Resource server (<code>resserv</code>)
| allocates/tracks objects, memory, channels
| the client half of the client/server RM split
|-
| Memory manager (<code>memmgr</code>)
| VRAM/sysmem objects, mappings, page tables
| the guest expects the GSP to own page-table leaves for its VA spaces
|-
| FIFO/runlist
| channels, TSGs, GPFIFO, USERD, doorbells
| where the guest's work is submitted
|-
| GSP client
| message queues, RPC transport, event handling
| the interface we emulate
|-
| UVM
| unified memory, page faults, channel manager
| starts early in CUDA init, extremely stateful
|-
| <code>nvidia_modeset</code> / <code>nvidia_drm</code>
| display / DRM
| needed later for graphics
|-
| CE / GR HALs
| engine programming
| needed to ''execute'' forwarded work
|}

== 4. Driver initialization vs runtime ==

Understanding this split is the key to debugging.

'''Initialization (boot/probe)''' — deterministic, mostly one-shot; where compatibility is won or lost [live]:

<pre>PCI probe → identify GPU from config space and BAR0 registers
→ boot the GSP (our emulated handshake)
→ read static configuration (device info tables, FB/BUS info, VBIOS, clocks)
→ create root client/device/subdevice objects
→ initialize RM subsystems (interrupts, FIFO, memory, GMMU, PMU)
→ expose /dev/nvidia* and register with the kernel</pre>

The actual message/control IDs observed at each step are listed in [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]].

'''Runtime (per-process, per-context)''' — driven by user-space workloads; where ''execution'' is won or lost:

<pre>open /dev/nvidia* → cuInit → UVM registration (per process)
→ cuCtxCreate → object/allocations → memory maps
→ channel creation + push buffers + semaphores
→ kernel launches (GR) / copies (CE) / memops
→ faults, completions, events</pre>

Context creation and the full CUDA driver-API smoke now pass (<code>ALL OK</code>, copies matched) [live]. The current focus is execution: the backend is a '''module-owned GPU running its real GSP firmware as the resource-manager server''' ([[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]]), with CE-class copies forwarded to the physical copy engine default-off [gated]. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed — the open execution wall ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]) [wall].

== 5. What the guest expects from the device, per phase ==

{| class="wikitable"
|-
! Phase
! Guest action
! Device/firmware expectation
|-
| Probe
| config space reads, BAR sizing, MSI-X capability
| a plausible TU104-family PCIe GPU
|-
| Boot
| BAR0 register writes, WPR setup, firmware boot sequence
| GSP boots, booter completes, "GSP init done"
|-
| Config
| RPC <code>GET_GSP_STATIC_INFO</code>, device tables, VBIOS reads
| consistent, self-describing tables
|-
| RM init
| <code>GSP_RM_CONTROL</code> battery (FB/BUS/CE/INTR/GMMU/...)
| correct shapes and plausible values
|-
| Objects
| <code>GSP_RM_ALLOC</code> (root, device, subdevice, vaspace, channel, CE objects)
| success + correct handles/params
|-
| Runtime
| allocs/controls for maps and channels; pushes
| executed work, semaphores, faults
|}

The RPC surface is described in [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] and [[RPC Message Format|<code>RPC-Message-Format.md</code>]]; the boot handshake in [[WPR Management|<code>WPR-Management.md</code>]].

== 6. Why user-space-only interception does not work ==

Two hard-won facts from this project:

# '''<code>libcuda</code> issues raw <code>syscall</code> instructions''', so an <code>LD_PRELOAD</code> shim around <code>ioctl()</code> sees nothing. Observing CUDA's kernel calls requires a <code>ptrace</code>-based tracer (PTRACE_SYSCALL + PTRACE_GET_SYSCALL_INFO), as the project's diagnostics do.
# **The CUDA driver talks to the ''kernel driver'', not to the device.** All device interaction happens in <code>nvidia.ko</code>. This is good news: the emulation boundary is a small set of kernel-side interfaces (MMIO, DMA, interrupts, the GSP protocol) rather than a large user-space API surface.

== 7. Windows guests ==

Windows brings its own kernel-mode driver, <code>nvlddmkm.sys</code>, which uses the same underlying GSP device/firmware model but different host interfaces (WDDM, MSI-X semantics, ROM/VBIOS quirks). The emulation target is the same: satisfy the firmware interface, forward the work. Windows support is a roadmap item ([[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]]) [planned], and the design goal is that no Windows-specific code lives in the guest.

== 8. Compatibility notes for versions ==

* The guest's driver and the emulated firmware profile must agree on the RPC message versions. The guest negotiates these at start-up (<code>SET_GUEST_SYSTEM_INFO</code> carries the VGX/RPC version); the emulation answers with a profile that matches the driver family in use.
* CUDA versions ride on driver versions; the test matrix tracks (driver, CUDA, firmware) triples ([https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>]).

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/02-guest-stack-and-cudasw.md <code>../docs/03-architecture/02-guest-stack-and-cudasw.md</code>] — canonical chapter.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the other side of the boundary.
* [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]], [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — user-space driver internals.
* [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]], [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the observed protocol reality.
* [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — where guest work is headed.

GSP and GSP-RM

2026-10-12T04:40:55Z

WikiBot: Fix repository links: point to the cgit tree URLs

{{About message}}

= GSP and GSP-RM =

<blockquote>'''What this page is.''' The GPU System Processor hosts NVIDIA's Resource Manager in firmware; since Turing, the guest driver's management path is ''talking to that firmware'', not to silicon. This page explains that architecture and the interface the project emulates. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> designed end state.

== 1. What the GSP is ==

The '''GSP''' is a small embedded RISC-V processor (RV64IMAC) inside NVIDIA GPUs. It runs a firmware image (for the 570.144 generation, a ~28 MB ELF) that contains, among other things, a full port of NVIDIA's '''Resource Manager (RM)''' — the component that owns GPU initialization, memory, engines, interrupts and scheduling.

<pre> ┌──────────────────────────── GUEST ────────────────────────────┐
│ nvidia.ko │
│ RM client path ── RPC ──┐ │
│ GSP boot code │ │
└────────────────────────────┼──────────────────────────────────┘
│ shared memory queues + doorbells
┌────────────────────────────┼──────────────────────────────────┐
│ GSP (firmware, RISC-V) │ │
│ RM core (RMCore) ◄─────┘ │
│ RM API (object model, clients, handles) │
│ Resource server (resserv): allocation & lifetime │
│ memory manager, GMMU, FIFO/runlist, interrupts, PMU │
└───────────────────────────────────────────────────────────────┘</pre>

A driver as complex as NVIDIA's is not written to program silicon directly; it is written to talk to an RM, and on modern GPUs a copy of the RM runs on the GSP. The driver is therefore an '''RM client''': it issues requests, receives callbacks, and observes state through a well-defined protocol.

== 2. The boot handshake we emulate ==

The guest driver boots the GSP exactly as it would on bare metal:

# '''WPR setup''' — carve a firmware-protected region of VRAM and write descriptors into BAR0 registers ([[WPR Management|<code>WPR-Management.md</code>]]).
# '''FWSEC''' — the driver verifies firmware signatures.
# '''Booter''' — the boot ROM consumes the firmware image.
# '''RISCV start''' — the RISC-V core begins executing the firmware.
# '''GSP init done''' — the firmware signals readiness through a status mailbox; the driver proceeds to initialization RPCs.

The emulation implements '''stage completion''' rather than the stages themselves — legitimate because the only observable to the driver is the state machine it polls. The same handshake has also been run against real firmware on the physical backend: the firmware comes up, delivers its first status events, serves the static configuration, and stays live until the unload RPC releases it ([[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]]).

== 3. The transport: message queues and RPC ==

GSP-RM's transport is a shared-memory message queue pair:

<pre> guest system memory (allocated by the guest driver, registered with us)
┌───────────────────────────────┐
│ command queue (guest -> GSP) │ element ring, 4 KB elements
├───────────────────────────────┤
│ status queue (GSP -> guest) │ element ring
└───────────────────────────────┘
+ a page table describing the queue pages (the buffer is
non-contiguous: each page is an independent allocation)</pre>

Each element begins with a transport header followed by an '''RPC message''' whose 32-byte header carries header version, length, function, result and sequence number. Observations worth knowing (all verified against the stock guest):

* <code>length</code> covers the fixed part of the message, not variable payloads — do not use it to bound a payload;
* function numbers are a flat enum shared by several protocols (<code>SET_GUEST_SYSTEM_INFO</code> 1, <code>FREE</code> 10, <code>GET_GSP_STATIC_INFO</code> 65, <code>GSP_RM_CONTROL</code> 76, <code>GSP_RM_ALLOC</code> 103, ...);
* replies must be matched to request sequence numbers — the driver runs a reply-matching state machine, and a slow reply is indistinguishable from a hang.

== 4. RMCore, the RM API and the resource server ==

Three RM concepts make every log readable:

* '''RMCore''' — the object system. Everything is an object with a class (a 32-bit class ID), a handle in a per-client handle space, and a parent object forming a tree rooted at a client. Traces of <code>GSP_RM_ALLOC</code> are object creation; traces of <code>GSP_RM_CONTROL</code> are method calls on objects.
* '''RM API''' — how in-kernel components (and user space via ioctls) allocate objects and call controls, with an implicit-handle and an explicit-handle variant. For a GSP client, part of the API becomes RPCs while some calls are handled locally; memory objects, for example, are allocated locally by the client RM.
* '''Resource server (resserv)''' — the allocation authority: it owns handle spaces, enforces privileges and tracks object lifetimes.

== 5. The client/server split (and why page tables matter) ==

Modern RM is client/server: the '''client''' (guest <code>nvidia.ko</code>) owns per-process state and submits work; the '''server''' (GSP-RM) owns GPU-global state — routing, page tables, scheduling domains, engine configuration. For virtualized setups the client tells the server which VA ranges the server owns, and the server manages the deeper page-table levels for those ranges. The client hands over the '''physical addresses of its page-table levels''' plus the VA ranges they cover; on real hardware the server then fills in the lower levels for server-owned ranges and the client never writes them.

This is a central fact for the execution work: when the guest's UVM asks the "GPU" to map memory, the mapping is expected to materialize in page tables that only the server side can write — which is us ([[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]], [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]]).

== 6. Event channels and notifications ==

Beyond request/reply, the guest needs asynchronous events: event objects registered with a notification configuration (an MSI-X vector or a memory "semaphore" the GPU can write), completion events when channels finish work or errors occur, delivered as a strict bit/handle/state triple on the exact vector. The emulation's MSI-X path is emulation thread → <code>eventfd</code> → KVM → guest vector, with the pending-bit array emulated so a masked vector's interrupt is not lost.

== 7. What we emulate today ==

{| class="wikitable"
|-
! Layer
! State
! Notes
|-
| GSP boot state machine
| <code>[live]</code>
| stages complete; the driver's poll loop is satisfied
|-
| Message queue transport
| <code>[live]</code>
| including the non-contiguous page table for the queue buffer
|-
| RPC request/reply
| <code>[live]</code>
| every function the driver uses during init is answered; unknown functions get a well-formed success reply
|-
| Static info tables
| <code>[wip]</code>
| device/FB/BUS/INTR/GMMU tables carry plausible reference values; the engine inventory is complete and agrees with the class list
|-
| Controls during init
| <code>[wip]</code>
| ~160 distinct control IDs observed per CUDA run; a small set is special-cased with real values and reply fidelity for the consumed fields keeps improving
|-
| Object creation
| <code>[wip]</code>
| handles/parameters are echoed consistently (zeroing the request parameters corrupts the caller); there is no server-side object model yet
|-
| Page tables for server-owned ranges
| <code>[wip]</code>
| contract received and recorded; leaf population is the execution work
|-
| Event/notification delivery
| <code>[wip]</code>
| MSI-X works; richer event semantics land with forwarding
|-
| Firmware behaviour (routing, scheduling, real execution)
| <code>[wall]</code>
| the execution frontier — see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]
|}

== 8. Where the firmware knowledge lives ==

* [[GSP 570.144W Firmware Analysis|<code>GSP-570.144W-Firmware-Analysis.md</code>]] — how the 28.5 MB image was decompiled and mapped.
* [[Address Reference|<code>Address-Reference.md</code>]] — firmware-side address tables.
* [[RPC Message Format|<code>RPC-Message-Format.md</code>]], [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]] — historical RPC analysis (superseded; kept for the record).
* [[WPR Management|<code>WPR-Management.md</code>]] — write-protected region semantics.
* [[The vGSP Execution Plane (vEP)|The execution plane]] and [[The Execution Frontier|the execution frontier]] — what turns this protocol into real work on real hardware.

MediaWiki:Sidebar

2026-10-12T04:40:28Z

WikiBot: Fix repository links to the cgit tree/repo URLs

* navigation
** mainpage|mainpage-description
** Project Status
** Documentation
** FAQ
** recentchanges-url|recentchanges
* Deep dives
** System Overview
** GSP and GSP-RM
** The vGSP Execution Plane (vEP)
** The Host Module (vgsp.ko)
** The Guest Stack and CUDASW
** CUDA Userspace Driver
** Introduction to GMMU
** UVM in the Guest
** Physical GPU Backend and Guest Wiring
** The Execution Frontier
* Project
** Open-IOV:About|About Open-IOV
** Install and First Run
** Hardware and Software Matrix
** HAAEEP and Collaboration
** risc-v-gsp-rm-debugger
** Glossary
* Community
** Community
** https://git.vgpu.io/open-iov.git/|Source repository
** https://discord.gg/Rb9K9DYxKK|Discord
* SEARCH
* TOOLBOX
* LANGUAGES

Community

2026-10-12T04:40:27Z

WikiBot: Fix repository links to the cgit tree/repo URLs

{{About message}}

== Community ==
'''Chat:''' [https://discord.gg/Rb9K9DYxKK Open-IOV Discord]

'''Source code and issues:''' [https://git.vgpu.io/open-iov.git/ git.vgpu.io/open-iov]

== Contributing ==
The project welcomes contributors of every kind. The most valuable contribution is evidence: a new hardware or software combination tested with the project's methodology, with the captures listed in the repository's debugging document.

* Start from [[Hardware and Software Matrix]] to see which combinations are Reference, Target or Experimental.
* The repository's developer onboarding document describes the environment, the coding standards and how a change is landed.
* For the firmware side, [[risc-v-gsp-rm-debugger]] is the host-side workbench; the tool documentation is in the repository.

'''Editorial policy for this wiki:''' account creation and editing are closed to prevent spam. The wiki is maintained from the project's tracked documentation by the project's publishing bot; corrections and additions are made through the repository.

Documentation

2026-10-12T04:40:25Z

WikiBot: Fix repository links to the cgit tree/repo URLs

{{About message}}

This is the index of the Open-IOV wiki. If you are new, read '''[[Project Status]]''' first, then '''[[System Overview]]'''. The wiki is a curated public view of the project's maintained documentation; each page names its canonical document in the [https://git.vgpu.io/open-iov.git/ repository].

== Start here ==
* [[Project Status]] — what works today, what is in progress, what is not started
* [[Install and First Run]] — build, load, and a stock guest driver in about half an hour
* [[System Overview]] — the architecture in one page
* [[FAQ]] — common questions, including "why is nvidia-smi working but CUDA not?"

== Architecture ==
* [[System Overview]] — principle, layer map, data flows
* [[GSP and GSP-RM]] — the firmware interface the project emulates
* [[The Host Module (vgsp.ko)]] — virtual device, emulation core, memory model, forwarding
* [[The Guest Stack and CUDASW]] — what runs unmodified inside the guest
* [[The vGSP Execution Plane (vEP)]] — the execution and completion layer
* [[Introduction to GMMU]] — GPU address translation from first principles
* [[UVM in the Guest]] — unified memory, semaphores, page trees
* [[GMMU Multi-Tenant Protection]] — isolation design for multiple VMs and tenants
* [[Physical GPU Backend and Guest Wiring]] — how guest work reaches real hardware
* [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA, with real control IDs and log signatures

== CUDA ==
* [[CUDA Userspace Driver]] — the <code>libcuda.so</code> deep dive
* [[The Execution Frontier]] — the GR compute-launch chain and its current wall

== Firmware and reverse engineering ==
* [[GSP 570.144W Firmware Analysis]] — how the 28.5 MB image was decompiled and mapped
* [[Address Reference]] — firmware-side address tables
* [[risc-v-gsp-rm-debugger]] — the firmware runtime workbench (hooks, watchpoints, sessions)
* Historical protocol analyses, kept for the record: [[RPC Message Format]] · [[BAR Emulation Specification]] · [[WPR Management]] · [[Migration Protocol]] · [[GSP RPC Emulation]] · [[Error Codes]] · [[vGPU-IO Integration Guide]]

== Portability and ecosystem ==
* [[Hardware and Software Matrix]] — Reference / Target / Experimental, honestly labelled
* [[HAAEEP and Collaboration]] — the agent-edge use case, collaboration and licensing models

== Reference ==
* [[Glossary]]
* [[FAQ]]
* [[Error Codes]] — historical naming, superseded by the wire status values

== History ==
This wiki previously hosted the OpenMdev / LibVF.IO project documentation (2021–2023). The pages below are archived for reference; they describe earlier projects and architectures, not Open-IOV.

* [[Introduction]] · [[Virtualization Fundamentals]] · [[Virtual I/O Internals]] · [[GPU Driver Internals]] · [[GPU Firmware]] · [[GPU Support]] · [[CPU Support]] · [[Hypervisor Support]] · [[Merged Drivers]] · [[OpenRM]] · [[AMDGPU]] · [[Intel]] · [[Mdev-GPU]] · [[GVM]] · [[Hyperborea]] · [[LIME Is Mediated Emulation]] · [[LibVF.IO Setup Guide]] · [[Looking Glass KVMFR]] · [[VGPU Unlock]] · [[GPU Software Bill Of Materials (SBOM)]]

== Meta ==
* [[Open-IOV:About|About Open-IOV]]
* [[Community]]
* Wiki architecture pages name their canonical source in the repository; when a page and its source disagree, the repository wins.

Open-IOV:About

2026-10-12T04:40:24Z

WikiBot: Fix repository links to the cgit tree/repo URLs

{{About message}}

'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs. It runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with GPU compute (CUDA), and eventually 3D graphics and video, on hardware you already own: without a proprietary vGPU license, without patching the guest driver, the host driver, the hypervisor or the GPU firmware.

== Mission ==
Make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without modifying NVIDIA software, firmware or hardware.

== How it works ==
The only component the project adds is a host kernel module (<code>vgsp.ko</code>). It implements the interface the guest driver expects to talk to — the GPU System Processor (GSP) and its resource-manager firmware — and arranges for the guest's work to execute on real GPU hardware. The guiding sentence is:

::'''''Emulate the interface. Pass through the work. Never patch anyone.'''''

See [[System Overview]] for the architecture and [[Project Status]] for the honest current state.

== Principles ==
* '''No patches anywhere.''' Guest drivers, host drivers, firmware and hypervisors are all used unmodified; interoperability is the goal, not counterfeiting.
* '''Honest status.''' Claims carry dates and evidence; the project publishes what does not work as clearly as what does.
* '''Independent IP block.''' Open-IOV is self-contained so KVM, Xen, Argo/OpenXT, Qubes OS Air and others can carry it without forks.
* '''Durability by owning the interface.''' The protocol, not a snapshot of it, is reimplemented — so new driver versions, firmware and hardware can be carried forward.

== Licensing ==
'''All documentation on this site is licensed [https://creativecommons.org/licenses/by/4.0/ CC-BY-4.0]''' — the statement carried at the top of every page. It may be reused with attribution, or used as inspiration for new documentation, including by GPU vendors for public commercial distribution as developer documentation.

The project's source code is licensed separately: '''BSD 3-Clause''' for everything reusable (userspace libraries, tools, scripts, configuration) and '''GPL-2.0''' for the kernel modules that must legally link against GPL-only kernel exports (VFIO, mdev, IOMMU); per-file SPDX identifiers govern. NVIDIA's software and firmware are used unmodified and are not redistributed. See the repository's <code>LICENSE</code> and <code>docs/09-reference/legal-and-licensing.md</code> for the full statement.

== Community ==
* Source and issues: [https://git.vgpu.io/open-iov.git/ git.vgpu.io/open-iov]
* Chat: [https://discord.gg/Rb9K9DYxKK Open-IOV Discord]
* See [[Community]] and [[HAAEEP and Collaboration]].

Main Page

2026-10-12T04:40:23Z

WikiBot: Fix repository links to the cgit tree/repo URLs

{{About message}}

__NOTOC__
<div style="border:1px solid #c8ccd1; background:#f8f9fa; padding:0.9em 1.2em; margin-bottom:1em;">
'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs: it runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with CUDA on hardware you own, '''without proprietary vGPU licenses''', without patching the guest driver, the host driver, the hypervisor or the firmware.

'''''Emulate the interface. Pass through the work. Never patch anyone.'''''

The only new component is one host kernel module (<code>vgsp.ko</code>), which implements the interface the guest driver expects to talk to — the GPU System Processor and its resource-manager firmware — and arranges for the guest's work to run on real GPU hardware.

'''[[Open-IOV:About|About Open-IOV]]''' · '''[[Community|Community]]''' · '''[[Documentation|Documentation]]'''
</div>

== Start here ==
* '''[[Project Status]]''' — the honest, dated state of the project.
* '''[[Install and First Run]]''' — fresh host to a stock guest driver in about half an hour.
* '''[[System Overview]]''' — how Open-IOV works, in one page.
* '''[[Documentation]]''' — the full index of this wiki.
* '''[[FAQ]]''' — the questions we get asked most.

== Project status (2026-10-12) ==
{| class="wikitable"
! Area !! State
|-
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) || Working
|-
| GSP boot emulation and GSP-RM message queues / RPC || Working
|-
| Stock guest driver probe, nvidia-smi (TU104) || Working
|-
| CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working
|-
| Guest CUDA kernel execution on real hardware || In progress — a forwarded launch reaches the engine queue; engine execution is the frontier
|-
| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet
|}

The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]].

== Deep dives ==
* [[System Overview]] — the architecture before any subsystem
* [[GSP and GSP-RM]] — the firmware interface we emulate
* [[The vGSP Execution Plane (vEP)]] — how guest work becomes execution and completion
* [[The Host Module (vgsp.ko)]] — the module internals
* [[The Guest Stack and CUDASW]] — what runs unmodified in the guest
* [[CUDA Userspace Driver]] — the <code>libcuda.so</code> analysis
* [[Introduction to GMMU]] — GPU address translation from first principles
* [[UVM in the Guest]] — unified memory as a correctness oracle
* [[Physical GPU Backend and Guest Wiring]] — how work reaches real hardware
* [[The Execution Frontier]] — the leading edge, evidence-led
* [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA with real IDs and log signatures

== Firmware interoperability ==
* [[GSP 570.144W Firmware Analysis]]
* [[Address Reference]]
* [[RPC Message Format]] · [[BAR Emulation Specification]] · [[WPR Management]] · [[Migration Protocol]] · [[GSP RPC Emulation]] (historical, superseded)
* [[risc-v-gsp-rm-debugger]] — the firmware runtime workbench

== Project and ecosystem ==
* [[Hardware and Software Matrix]] — supported combinations, honestly labelled
* [[HAAEEP and Collaboration]] — the agent-edge use case and how to work with us
* [[FAQ]] · [[Glossary]]

== Community ==
* Source and issues: [https://git.vgpu.io/open-iov.git/ git.vgpu.io/open-iov]
* Chat: [https://discord.gg/Rb9K9DYxKK Open-IOV Discord]

== About Open-IOV ==
Open-IOV runs unmodified NVIDIA driver stacks inside virtual machines with GPU compute on hardware you own — no proprietary vGPU license, and no patches to the guest driver, the host driver, the hypervisor or the firmware. The project adds one host kernel module (<code>vgsp.ko</code>) that implements the interface the guest driver expects to talk to — the GPU System Processor and its resource-manager firmware — and arranges for the guest's work to run on real GPU hardware.

* '''Mission:''' make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without modifying NVIDIA software, firmware or hardware.
* '''Principles:''' no patches anywhere; honest status carried with dated evidence; an independent, self-contained IP block; durability by owning the interface rather than a snapshot of it.
* '''Licensing:''' all documentation on this site is [https://creativecommons.org/licenses/by/4.0/ CC-BY-4.0]; the project's code is BSD-3-Clause (GPL-2.0 for the kernel modules); NVIDIA software and firmware are used unmodified and are not redistributed.

'''[[Open-IOV:About|Read more about Open-IOV →]]'''

== About this wiki ==
This wiki previously hosted documentation from the earlier project era (2021–2023). Those pages are retained for reference under [[Documentation|History]]; current documentation starts at [[Documentation]].

GSP RPC Emulation

2026-10-12T04:40:10Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= GSP vGPU RPC Emulation =

<blockquote>'''Historical document — superseded.''' This page describes an earlier emulation-mode design and is retained for reference only. The project does '''not''' use NVIDIA's proprietary vGPU stack. Canonical documentation: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

Open-IOV includes '''emulation mode''' that allows operation with unmodified consumer/OpenRM GSP firmware by emulating missing vGPU RPC handlers locally in kernel and userspace.

== Problem ==

The NVIDIA GSP firmware on consumer drivers (550.163.01) and OpenRM lacks the vGPU-specific RPC handlers that exist only in the GRID host driver firmware:

{| class="wikitable"
|-
! Handler
! Address
! Size
! Purpose
|-
| BAR Trap
| 0x04CB8188
| 40KB
| BAR1/ForceBarAccess/Aperture
|-
| WPR Handler 1
| 0x04F00B0C
| 62KB
| Write/Read Protect Regions
|-
| WPR Handler 2
| 0x04F00C78
| 62KB
| Write/Read Protect Regions
|-
| Migration
| 0x04179ED8/0x04179FBC
| -
| Live migration
|}

Without these handlers, VF creation times out at BAR configuration (RPC to 0x04CB8188).

== Solution: Emulation Mode ==

Emulation mode intercepts vGPU RPC calls and handles them locally instead of sending to GSP:

<pre>┌─────────────────────────────────────────────────────────────────┐
│ Emulation Architecture │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Userspace App (libiov.so) │
│ │ │
│ ▼ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Kernel IOCTL │────▶│ Emulation Layer │ │
│ │ Handler │ │ (kernel) │ │
│ └─────────────────┘ └────────┬────────┘ │
│ │ │
│ ┌───────────────┼───────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ BAR Emu │ │ WPR Emu │ │ Mem Emu │ │
│ │ (kernel) │ │ (kernel) │ │ (kernel) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────┐ │
│ │ Userspace Emulator (libiov.so) │ │
│ │ rpc_emu_bar_*(), rpc_emu_wpr_*() │ │
│ └──────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘</pre>

== Auto-Detection (Default Behavior) ==

'''Emulation mode is ON by default''' and auto-detects at driver load:

# Driver initializes RPC communication with GSP
# Sends test <code>BAR_TRAP</code> RPC (non-destructive READ)
# '''If GSP responds''' (ret=0) → vGPU handlers present → '''emulation disabled'''
# '''If GSP times out/errors''' (ret=-ETIMEDOUT/-EIO/-ENODEV) → handlers missing → '''emulation enabled'''

<pre>dmesg output:
gsp-vgpu: Auto-detecting GSP vGPU RPC handler support...
gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode
# or
gsp-vgpu: GSP has vGPU RPC handlers - DISABLING emulation mode</pre>

== Module Parameters ==

{| class="wikitable"
|-
! Parameter
! Type
! Default
! Description
|-
| <code>emulation_mode</code>
| bool
| 1 (ON)
| Current state (read-only, shows auto-detected value)
|-
| <code>emulation_override</code>
| bool
| 0 (auto)
| Override: 1=force emulation ON, 0=force emulation OFF
|}

<pre># Default: auto-detect
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# Force emulation ON (testing with GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=1

# Force emulation OFF (requires GRID GSP)
sudo insmod gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin emulation_override=0</pre>

== Emulated RPC Handlers ==

=== Kernel Emulation (<code>driver/</code>) ===

{| class="wikitable"
|-
! RPC Type
! Handler
! Emulation
|-
| <code>GSP_RPC_ALLOC</code> (0x01)
| <code>gsp_vf_mem_alloc()</code>
| Synthetic handles/VAs from local counter
|-
| <code>GSP_RPC_PIN</code> (0x05)
| <code>gsp_vf_mem_pin()</code>
| Synthetic DMA addresses from local counter
|-
| <code>GSP_RPC_MAP</code> (0x03)
| <code>gsp_vf_mem_map()</code>
| Identity map (return requested VA)
|-
| <code>GSP_RPC_BAR_TRAP</code> (0x0A)
| <code>gsp_bar_trap_handler()</code>
| Local aperture/force_access state
|-
| <code>GSP_RPC_MAP</code> (WPR)
| <code>gsp_vf_wpr_protect()</code>
| Local WPR region list
|-
| <code>GSP_RPC_UNMAP</code> (WPR)
| <code>gsp_vf_wpr_unprotect()</code>
| Local WPR region removal
|}

=== Userspace Emulation (<code>userspace/lib/rpc_emulator.c</code>) ===

{| class="wikitable"
|-
! Function
! Purpose
|-
| <code>rpc_emu_bar_config()</code>
| Local aperture offset/size + backing store
|-
| <code>rpc_emu_bar_force_access()</code>
| Local force_access state
|-
| <code>rpc_emu_bar_save/restore()</code>
| Migration state serialization
|-
| <code>rpc_emu_wpr_protect_write/read()</code>
| Local WPR region tracking
|-
| <code>rpc_emu_wpr_unprotect()</code>
| Local WPR region removal
|-
| <code>rpc_emu_wpr_set_timeout()</code>
| Local timeout tracking
|-
| <code>rpc_emu_wpr_check_access()</code>
| Access violation detection
|}

== Emulation Data Structures ==

=== BAR State (per VF) ===

<pre>struct gsp_bar_state {
uint64_t aperture_offset; // BAR1 aperture base
uint64_t aperture_size; // BAR1 aperture size (max 256KB)
bool force_access; // ForceBarAccess enabled
uint32_t force_vf; // VF holding force access
void *backing_store; // Emulated BAR memory
size_t backing_size; // Backing store size
};</pre>

=== WPR Region List (per VF) ===

<pre>struct gsp_wpr_region {
struct list_head list;
uint64_t base; // Region base address
uint64_t size; // Region size
uint32_t flags; // GSP_WPR_WRITE/READ/EXECUTE
uint32_t vf_id; // Owning VF
void (*completion_cb)(...); // Write/read complete callback
void (*timeout_cb)(...); // Timeout callback
uint64_t timeout_ns; // Timeout in nanoseconds
};</pre>

=== Userspace Emulator State ===

<pre>struct emu_bar_state {
uint64_t aperture_offset;
uint64_t aperture_size;
bool force_access;
uint32_t force_vf;
void *backing_store;
size_t backing_size;
pthread_mutex_t lock;
};

struct emu_wpr_state {
struct emu_wpr_region *regions; // Linked list
pthread_mutex_t lock;
};</pre>

== Fallback Logic ==

Userspace library (<code>libiov.so</code>) implements fallback:

<pre>vgpu_error_t vgpu_bar_config(vgpu_vf_t vf, uint64_t offset, uint64_t size) {
// 1. Try kernel IOCTL (sends RPC to GSP)
ret = ioctl(fd, GSP_IOCTL_BAR_CONFIG, &args);

// 2. If GSP lacks handlers (ENOTSUP/ETIMEDOUT/EIO), use emulator
if (ret < 0 && dev->emulation_enabled &&
(errno == ENOTSUP || errno == ETIMEDOUT || errno == EIO)) {
return rpc_emu_bar_config(vf->id, offset, size);
}

return map_errno(errno);
}</pre>

== Limitations ==

{| class="wikitable"
|-
! Feature
! Emulation Status
! Notes
|-
| BAR Aperture Config
| ✅ Full
| Local state + backing store
|-
| ForceBarAccess
| ✅ Full
| Per-VF exclusive access
|-
| BAR Save/Restore
| ✅ Full
| Migration support
|-
| WPR Write Protect
| ✅ Full
| Region tracking + access check
|-
| WPR Read Protect
| ✅ Full
| Region tracking + access check
|-
| WPR Timeout
| ✅ Partial
| Timeout stored, callbacks not wired
|-
| Memory Alloc
| ✅ Basic
| Synthetic handles/VAs
|-
| Memory Pin
| ✅ Basic
| Synthetic DMA addresses
|-
| Memory Map
| ✅ Basic
| Identity mapping
|-
| Migration
| ⚠️ Partial
| BAR/WPR state only
|-
| Scheduling
| ❌ No
| Returns success, no actual scheduling
|-
| Notifications
| ❌ No
| No async event delivery
|}

== Testing Emulation ==

<pre># 1. Load driver (auto-enables emulation on consumer GSP)
sudo insmod driver/gsp-vgpu.ko firmware_name=gsp-570.144.fwimage.bin

# 2. Verify emulation enabled in dmesg
dmesg -T | grep -i emulation
# gsp-vgpu: GSP lacks vGPU RPC handlers (ret=-110) - ENABLING emulation mode

# 3. Run example - should complete without timeouts
sudo LD_LIBRARY_PATH=userspace/lib ./userspace/examples/example_vgpu /dev/gsp-vgpu

# 4. Check debugfs for emulated state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/bar_state
cat /sys/kernel/debug/gsp-vgpu/vfs/vf1/wpr_regions</pre>

== Future Work ==

* [ ] Wire WPR timeout callbacks to kernel timers
* [ ] Implement emulated scheduling with work queues
* [ ] Add async notification support via eventfd
* [ ] Complete migration state for WPR regions
* [ ] Performance optimization (batch RPC emulation)
* [ ] Conformance tests vs GRID GSP behavior

== Related Documentation ==

* [[BAR Emulation Specification]]
* [[WPR Management]]
* [[RPC Message Format]]
* [[VGPU-IO Integration Guide|Integration Guide]]

GSP 570.144W Firmware Analysis

2026-10-12T04:40:09Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= NVIDIA GSP 570.144W Firmware Analysis (TU102/TU104) =

<blockquote>'''Retained firmware-analysis record.''' This page documents the GSP 570.144W firmware binary (decompilation and firmware-side addresses) and the address tables below are the firmware-side record, not the host implementation. Current host architecture: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>]. The decompiled analysis output is not distributed; the file names below are</blockquote>

the analysis record.

== Binary Information ==

* '''File''': <code>gsp-570.144.fwimage.bin</code>
* '''Size''': 28.5 MB
* '''Format''': ELF, RV64IMAC (RISC-V 64-bit)
* '''Base Address''': 0x04000000
* '''MD5''': <code>13ce12a392a31eeb79a2e6ebd3a4a93a</code>

== Architecture Overview ==

=== Memory Map ===

<pre>0x04000000 - 0x05FFFFFF : Code + Read-only data (~32 MB)
0x04296040 : Global state structure (cRam0000000004296048)
0x0429B750 : Error/status registers
0x041DC350 : Stack canary / frame guard
0x429B000 - 0x429BFFF : Memory allocator arena (0x429B450 = alloc entry)</pre>

=== Calling Convention (RISC-V RV64IMAC) ===

{| class="wikitable"
|-
! Register
! Purpose
|-
| s0 (x8)
| Frame pointer
|-
| s1 (x9), s4 (x12), s10 (x20)
| RA save locations
|-
| s1-s11
| Callee-saved (pushed to stack)
|-
| a0-a7
| Arguments / return values
|-
| sp
| Stack pointer (grows down, frames 96-560 bytes)
|}

----

== RPC System ==

=== 12 RPC Types (Jump Table at 0x059BFEB0) ===

{| class="wikitable"
|-
! Type
! Name
! Handler A
! Handler B
! Dispatcher
|-
| 0x01
| ALLOC
| —
| —
| Primary
|-
| 0x02
| FREE
| —
| —
| Primary
|-
| 0x03
| MAP
| 0x05766AB7
| 0x05778A37
| Secondary
|-
| 0x04
| UNMAP
| 0x057669B7
| 0x05779937
| Primary + Secondary
|-
| 0x05
| PIN
| 0x057664B7
| 0x0577A3B7
| Secondary
|-
| 0x06
| UNPIN
| 0x057682B7
| 0x05779FB7
| Secondary
|-
| 0x07
| SCHEDULE
| 0x05767F37
| 0x0575AEB7
| Primary + Secondary
|-
| 0x08
| YIELD
| 0x0575AE37
| 0x05767337
| Secondary
|-
| 0x09
| NOTIFY
| 0x0577C8B7
| 0x05767837
| Secondary
|-
| 0x0A
| BAR_TRAP
| 0x057675B7
| —
| Secondary
|-
| 0x0B
| MIGRATE
| —
| —
| Primary
|-
| 0x0C
| DEBUG
| —
| —
| —
|}

=== Primary Dispatcher (0x057A1AC0) ===

<pre>Entry: FUN_ram_057a1ac0(param_1, param_2, param_3, param_4, *out_id, *out_a, *out_b, *out_c)</pre>

* Extracts RPC type from <code>param_1</code>
* Validates against max type (bgeu x12, x13 checks for types 11 and 13)
* MIGRATE (0x0B) branches at 0x057A1BDC and 0x057A1D54
* Indexes into 32-entry jump table at '''0x057A3AD8'''

==== Primary Dispatcher Sub-Command Jump Table (0x057A3AD8) ====

{| class="wikitable"
|-
! Subcmd
! Target
! Decompiled
! Description
|-
| 0
| 0x057A1DB4
| ✅
| Returns ID=0x40
|-
| 1
| 0x057A1C18
| ✅
|
|-
| 2
| 0x057A39FC
| ✅
|
|-
| 3
| 0x057A1BCC
| ✅
|
|-
| 4
| 0x057A1EF4
| ✅
|
|-
| 5
| 0x057A1E94
| ✅
|
|-
| 6
| 0x057A3A28
| ✅
|
|-
| 7
| 0x057A3A60
| ✅
|
|-
| 8
| 0x057A1C6C
| ✅
|
|-
| 9
| 0x057A3A48
| ✅
|
|-
| 10
| 0x057A1F6C
| ✅
|
|-
| 11
| 0x057A3AC4
| ✅
|
|-
| 12
| 0x057A1E10
| ✅
|
|-
| 13
| 0x057A3A90
| ✅
|
|-
| 14
| 0x057A39BC
| ✅
|
|-
| 15
| 0x057A3AAC
| ✅
|
|-
| 16
| 0x057A3A78
| ✅
|
|-
| 17
| 0x057A1CCC
| ✅
|
|-
| 18
| 0x057A39E0
| ✅
|
|-
| 19
| 0x057A1FA8
| ✅
|
|-
| 20
| 0x057A2094
| ✅
|
|-
| 21
| 0x057A2074
| ✅
|
|-
| 22
| 0x057A2044
| ✅
|
|-
| 23
| 0x057A20D4
| ✅
|
|-
| 24
| 0x057A20B4
| ✅
|
|-
| 25
| 0x057A1FC8
| ✅
|
|-
| 26
| 0x057A2008
| ✅
|
|-
| 27
| 0x057A1FE8
| ✅
|
|-
| 28
| 0x057A2024
| ✅
|
|-
| 29
| 0x057A219C
| ✅
|
|-
| 30
| 0x057A20FC
| ✅
|
|-
| 31
| 0x057A214C
| ✅
|
|}

'''Pattern''': Each sub-command returns a constant ID in <code>*out_id</code> (e.g., 0x10, 0x14, 0x40) and echoes input params to output pointers.

=== Secondary Dispatcher (0x04ED39DC) ===

* Branch-based dispatch for ALL 11 RPC types
* 141 KB decompiled — handles MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP
* Entry: 0x04ED34B8 (PIN handler from secondary)

=== Specialized Main Handlers ===

{| class="wikitable"
|-
! Function
! Address
! Size
! Role
|-
| <code>rpc_alloc_main</code>
| 0x057E94EC
| 40 KB
| Core ALLOC implementation
|-
| <code>rpc_schedule_main</code>
| 0x051EFBF4
| 1.8 KB
| Core SCHEDULE implementation
|-
| <code>rpc_bar_trap_handler</code>
| 0x04CB8188
| 40 KB
| BAR1 emulation
|-
| <code>rpc_pin_handler</code>
| 0x04ED34B8
| —
| PIN from secondary dispatcher
|}

----

== BAR Emulation (RPC 0x0A) ==

=== Handler: 0x04CB8188 (40 KB) ===

'''Strings Referenced''':

* <code>BAR1</code> at 0x0406570E, 0x04085DE0
* <code>Bar1Rest</code> at 0x040656DA
* <code>ForceBarAccess</code> at 0x040656F2
* <code>Bar1Aperture</code> at 0x04065722

'''Key Operations''':

* Memory allocation via <code>UNK_ram_058A1A9C</code> at arena 0x429B450
* Buffer management with 0x429B428 (cache/flush)
* Error codes: 0x51 (OOM), 0x1E (timeout), 0x1D, 0x1F, 0x25, 0x37, 0x56
* Uses <code>ebreak</code> for fatal errors with logging at 0x05AB7DA0
* Stack canary check against 0x041DC350

'''Integration Point for vGPU-IO''':

<pre>// Register BAR trap handler at 0x04CB8188
// Monitor BAR1, Bar1Rest, ForceBarAccess, Bar1Aperture registers
// Emulate BAR1 aperture/size for guest VF</pre>

----

== WPR Management ==

=== Handlers: 0x04F00B0C, 0x04F00C78 (62 KB each) ===

'''String References (60 locations)''':

* <code>WPR_WRITE_COMPLETE</code>, <code>WPR_READ_COMPLETE</code>
* <code>WPR_WRITE_TIMEOUT</code>, <code>WPR_READ_TIMEOUT</code>
* <code>NV_WPR_WRITE_COMPLETE</code>, <code>NV_WPR_READ_COMPLETE</code>
* <code>NV_WPR_WRITE_TIMEOUT</code>, <code>NV_WPR_READ_TIMEOUT</code>

'''Key Operations''':

* VBIOS parsing (checks for <code>0xAA55</code>, <code>0x57464F4E</code> "WFON", <code>0x5746564E</code> "WFVN")
* Memory allocation at 0x429B450
* DMA/copy via <code>UNK_ram_0588F9C4</code>, <code>UNK_ram_05AEC86C</code>
* State machine with completion/timeout handling
* Error codes: 0x1E-0x25, 0x37, 0x51, 0x56, 0xFFFF

'''Integration Point for vGPU-IO''':

<pre>// Hook WPR completion callbacks from string reference locations
// Implement NV_WPR_WRITE/READ_COMPLETE/TIMEOUT notifications
// Manage WPR regions for guest VF memory isolation</pre>

----

== Migration Protocol (RPC 0x0B) ==

=== Dual Jump Tables ===

==== Table 1: 0x04179ED8 (Primary dispatcher MIGRATE branch at 0x057A1BDC) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
|-
| 0
| 0x057A2344
| ✅ Returns ID=0x14, reads flag at <code>in_t1+0x7F9</code>
|-
| 1
| 0x057A2320
| ✅
|-
| 2
| 0x057A230C
| ✅
|-
| 3
| 0x057A22F8
| ✅ Returns ID=0x10
|-
| 4
| 0x057A2388
| ✅
|-
| 5
| 0x057A23D8
| ✅
|-
| 6
| 0x057A23AC
| ✅
|-
| 7
| 0x057A2478
| ✅
|-
| 8
| 0x057A2428
| ✅
|-
| 9
| 0x057A2454
| ✅
|-
| 10
| 0x057A2404
| ✅
|-
| 11
| 0x057A2364
| ✅
|-
| 12
| 0x057A1F3C
| (dup)
|-
| 13
| 0x057A299C
|
|-
| 14
| 0x057A1F3C
| (dup)
|-
| 15
| 0x057A1F3C
| (dup)
|}

==== Table 2: 0x04179FBC (Primary dispatcher MIGRATE branch at 0x057A1D5C) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
|-
| 0
| 0x057A257C
| ✅
|-
| 1
| 0x057A25E4
| ✅
|-
| 2
| 0x057A24EC
| ✅
|-
| 3
| 0x057A2608
| ✅
|-
| 4
| 0x057A2534
| ✅
|-
| 5
| 0x057A25C0
| ✅
|-
| 6
| 0x057A24A4
| ✅
|-
| 7
| 0x057A259C
| ✅
|-
| 8
| 0x057A2510
| ✅
|-
| 9
| 0x057A2558
| ✅
|-
| 10
| 0x057A24C8
| ✅
|-
| 11
| 0x057A262C
| ✅
|-
| 12
| 0x057A345C
| ✅
|-
| 13
| 0x057A36D8
| ✅
|-
| 14
| 0x057A35D8
| ✅
|-
| 15
| 0x057A3658
| ✅
|}

'''Pattern''': Migration sub-commands follow same descriptor pattern as primary dispatcher — return constant ID, echo params.

----

== Entry Point & Boot Sequence ==

=== 0x04009000 (573 bytes) ===

* Initializes global state at 0x04296040
* Sets up stack canary at 0x041DC350
* Configures memory allocator arena at 0x429B450
* Registers RPC dispatchers
* Jumps to main scheduler loop

----

== Decompiled Output Directory ==

<pre>decompiled/ (analysis output, not distributed)
├── entry_ram_04009000.c # Boot entry
├── FUN_ram_04cb8188_ram_04cb8188.c # BAR trap handler (40 KB)
├── FUN_ram_04cc58f8_ram_04cc58f8.c # Secondary dispatcher (141 KB)
├── FUN_ram_04d00a18_ram_04d00a18.c # ALLOC-dominant (142 KB)
├── FUN_ram_04ed39dc_ram_04ed39dc.c # Secondary dispatcher (141 KB)
├── FUN_ram_04f00b0c_ram_04f00b0c.c # WPR handler 1 (62 KB)
├── FUN_ram_04f00c78_ram_04f00c78.c # WPR handler 2 (62 KB)
├── FUN_ram_051efbf4_ram_051efbf4.c # SCHEDULE main (1.8 KB)
├── FUN_ram_054a7de0_ram_054a7de0.c # ALLOC-dominant (67 KB)
├── FUN_ram_05723ef4_ram_05723ef4.c # Tertiary dispatcher (13 KB)
├── FUN_ram_057a1ac0_ram_057a1ac0.c # Primary dispatcher (558 B)
├── FUN_ram_057a1bcc_ram_057a1bcc.c ... # 32 sub-commands (~550 B each)
├── FUN_ram_057e94ec_ram_057e94ec.c # ALLOC main (40 KB)
├── FUN_ram_059be7f4_ram_059be7f4.c # RPC jump table (4.9 KB)
├── FUN_ram_057664b7_ram_057664b7.c ... # MAP/PIN/UNPIN/SCHED/YIELD/NOTIFY handlers
└── FUN_ram_057a22f8_ram_057a22f8.c ... # 28 migration handlers</pre>

----

== vGPU-IO Integration Checklist ==

=== Memory Management (ALLOC/FREE/MAP/UNMAP) ===

* [ ] Implement kernel RPC client calling 0x057A1AC0
* [ ] Parse sub-command descriptors from 0x057A3AD8 jump table
* [ ] Hook ALLOC main at 0x057E94EC for vGPU memory carve-out

=== Page Pinning (PIN/UNPIN) ===

* [ ] Call secondary dispatcher PIN at 0x04ED34B8
* [ ] Implement UNPIN via 0x057682B7 / 0x05779FB7

=== Scheduling (SCHEDULE/YIELD) ===

* [ ] Use 0x05767F37 / 0x0575AEB7 / 0x051EFBF4 for vCPU scheduling
* [ ] Implement YIELD at 0x0575AE37 / 0x05767337

=== BAR Emulation (BAR_TRAP) ===

* [ ] Register handler at 0x04CB8188
* [ ] Emulate BAR1/Bar1Rest/ForceBarAccess/Bar1Aperture for VF
* [ ] Monitor 0x429B450 allocator for BAR backing store

=== WPR Management ===

* [ ] Hook completion callbacks from string refs (0x04F00B0C, 0x04F00C78)
* [ ] Implement NV_WPR_* notification forwarding to guest
* [ ] Manage WPR regions per-VF for isolation

=== Migration (MIGRATE) ===

* [ ] Implement dual jump table dispatch (0x04179ED8, 0x04179FBC)
* [ ] Handle 28 migration sub-commands
* [ ] Coordinate with guest VF live migration protocol

----

== Tools & Scripts ==

{| class="wikitable"
|-
! Script
! Purpose
|-
| <code>DecompileFunctions.java</code>
| Ghidra headless decompilation
|-
| <code>trace_all_subsystems.py</code>
| Binary analysis: WPR/BAR/Migration strings, jump tables
|-
| <code>gsp-fwimage-570.144/</code>
| Ghidra project directory
|}

=== Ghidra Usage ===

<pre>/snap/ghidra/current/ghidra/support/analyzeHeadless \
/path/to/project gsp-analysis \
-import gsp-570.144.fwimage.bin \
-noanalysis \
-postScript DecompileFunctions.java \
0xADDRESS1,0xADDRESS2,...</pre>

----

== References ==

* RPC map: [https://git.vgpu.io/open-iov.git/tree/firmware/57X-GSP/570.144/RPC_HANDLER_MAP.md <code>firmware/57X-GSP/570.144/RPC_HANDLER_MAP.md</code>]
* Final report: [https://git.vgpu.io/open-iov.git/tree/firmware/57X-GSP/570.144/FINAL_REPORT.md <code>firmware/57X-GSP/570.144/FINAL_REPORT.md</code>]
* Analysis results: [https://git.vgpu.io/open-iov.git/tree/firmware/57X-GSP/570.144/analysis_results.json <code>firmware/57X-GSP/570.144/analysis_results.json</code>]

GMMU Multi-Tenant Protection

2026-10-12T04:40:09Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= GMMU Multi-Tenant Protection =

<blockquote>'''Current — companion summary.''' This page condenses [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] (the canonical document: problem statement, full design, RM struct sketches, walked example, portability, M1–M8 roadmap). Read the canonical document for anything you intend to build against; this page is the quick map.</blockquote>

== Overview ==

Open-IOV must isolate '''multiple mdevs (VMs)''' and, for multi-tenant inference, '''multiple tenants inside one guest''' at the GMMU. The design has two halves:

# '''Service-state ownership (baseline 1.0).''' The work/completion service (<code>kernel/gsp_sema_service.c</code>) is one global context (<code>static struct sema_svc svc</code>, <code>:676</code>), the pump services the first open mdev (<code>:13352-13364</code>), and channels/VA spaces/translations are keyed by guest-chosen values alone (<code>:847</code>, <code>:988</code>, <code>:113-130</code>). Two guests can pick the same handles and VAs (live: channel <code>0xcaf000d4</code>; <code>../debug/issue/issue-number-2.md</code>), so owner-keying is a prerequisite for any GMMU policy.
# '''GMMU policy for paged attention (research).''' A tenant/agent-aware KV cache protection design across vLLM, a guest utility, RM controls, the vEP, and page tables — post-1.0, dual-gated; never a baseline dependency.

== Isolation layers ==

{| class="wikitable"
|-
! Layer
! Mechanism
! Status
|-
| L0 platform
| IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary
| live
|-
| L1 transport
| VFIO mdev per tenant, config/BAR/IRQ mediation
| live
|-
| L2 vEP state
| per-mdev emulation context + (needed) per-mdev work service
| live / design
|-
| L3 GMMU
| per-context page directory (VA space)
| live (guest-owned) / design (host-owned)
|-
| L4 arbitration
| channel ownership, TSG/runlist policy
| design
|-
| L5 completion
| owner-keyed release resolution; no cross-tenant apply
| wip
|-
| L6 fault
| fault attributed to the owning channel/tenant only
| wip
|-
| L7 application
| per-tenant/per-agent VA pooling inside a guest
| research
|}

== Hardware primitives: proven vs hypothesis ==

{| class="wikitable"
|-
! Primitive
! Status
|-
| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0
| '''proven''' (<code>docs/43-GMMU-PTE-DECODER</code>; <code>kernel/gsp_sema_service.c:1058-1072</code>)
|-
| VER2 PTE fields incl. <code>read_only</code> (6), <code>privilege</code> (5), <code>encrypted</code> (4), <code>kind</code>
| '''proven format''' (<code>docs/43</code> §3)
|-
| Separate page directory per VA space (<code>SET_PAGE_DIRECTORY</code> with <code>hVASpace</code>/<code>chId</code>)
| '''proven''' (<code>kernel/gsp_rm_emulation.c:3310-3314</code>)
|-
| No tenant/ACL/PASID field in the VER2 PTE/PDE
| '''proven absence''' in the field catalogue
|-
| RO-leaf write enforcement
| '''hypothesis''' (needs a supervised probe; M7)
|-
| Encryption-domain (CCS) as a tenant boundary
| '''hypothesis'''
|}

== The design in brief ==

* '''Design A (stock-compatible).''' Extended vLLM reserves a VA range per agent; a new guest utility (<code>oiv-agentd</code>) publishes tag records to a device-memory '''tag mailbox''' the vEP can read (emulation mode: guest vidmem is host RAM, <code>kernel/bar_init.c:68-77</code>). The vEP enforces owner↔VA-subtree match at map time and at submit time. No guest KMD change. Research (M6).
* '''Design B (explicit controls).''' New vGPU-private RM controls — <code>TENANT_BIND</code> (0x90f10120), <code>TAG_RANGE</code> (…21), <code>UNTAG_RANGE</code> (…22), <code>TENANT_SHARE</code> (…23), <code>AGENT_CHANNEL_BIND</code> (…24), <code>TAG_QUERY</code> (…25) — plus struct sketches in the canonical doc §4.5. Requires a guest KMD extension: post-1.0, gated (M8).
* '''Granularities.''' Per-KV-block tagging (finest; software+PTE policy, sharing needs RO enforcement); per-tenant separate page directories (the only proven hardware lever); per-agent separate channels (fault/QoS containment; optional per-agent sub-directories). Degradation ladder and per-granularity cost/failure analysis in canonical §4.7.

== Walked example (two tenants, paged_attention_v2) ==

Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP submit-time policy; the fault is attributed to A1 only (T2 untouched); a periodic owner-vs-PTE audit with canaries catches escapes. If only per-tenant separation exists, the deployable contract is '''one agent per tenant, one tenant per context''' (canonical §5.4).

== Key code anchors (master <code>1f571dda</code>) ==

{| class="wikitable"
|-
! What
! Where
|-
| per-mdev emulation context
| <code>kernel/include/nvidia-vgpu-vfio.h:133</code>, <code>kernel/gsp_emulation.h:130</code>
|-
| global service state
| <code>kernel/gsp_sema_service.c:676</code>; pump <code>:13352</code>; doorbell <code>:13512</code>
|-
| channel/vaspace registries
| <code>kernel/gsp_sema_service.c:847</code>, <code>:988</code>
|-
| GMMU walk (VER2)
| <code>kernel/gsp_sema_service.c:1174</code>; decode <code>:1093</code>
|-
| page directory record
| <code>kernel/gsp_rm_emulation.c:3320</code>; reserved PDEs <code>:2237-2262</code>
|-
| per-mdev FB sizing
| <code>kernel/bar_init.c:60-77</code>; <code>kernel/include/ioctl.h:39-46</code>
|-
| L2 invalidate acked
| <code>kernel/gsp_rm_emulation.c:2519-2532</code>
|}

== Roadmap (acceptance criteria in canonical §7.3) ==

M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the 1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- replace, never required by the baseline. RM 610 stays behind its own human gate (<code>docs/23-RM610-READINESS</code>).

== Related documentation ==

* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]] — from-first-principles GMMU introduction (what it is, page tables, IOMMU/MMU/SMMU relationships)
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] — canonical
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>] — vEP vocabulary (parallel)
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/05-gmmu-amap-hal-gr.md <code>../docs/03-architecture/05-gmmu-amap-hal-gr.md</code>] — GMMU background
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/09-full-system-picture.md <code>../docs/03-architecture/09-full-system-picture.md</code>] — multi-tenancy and isolation map
* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — forwarding and page-table seams
* [https://git.vgpu.io/open-iov.git/tree/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — threat model, rung 5
* [https://git.vgpu.io/open-iov.git/tree/debug/issue/issue-number-2.md <code>../debug/issue/issue-number-2.md</code>] — per-client state wedge
* [https://git.vgpu.io/open-iov.git/tree/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md <code>../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md</code>] — GMMU field catalogue

FAQ

2026-10-12T04:40:08Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= FAQ =

<blockquote>'''What this page is.''' The questions the project is asked most, answered honestly and with pointers to the maintained documents. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/09-reference/glossary-and-faq.md <code>../docs/09-reference/glossary-and-faq.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>.

== Project and positioning ==

'''Is this a replacement for vLLM on bare metal?''' No. Continuous batching on a bare-metal host is the right answer for centralized inference, and it stays that way. Open-IOV is for compute ''inside guests'': Linux and Windows CUDA, 3D, video, and secure edge isolation.

'''Does the guest driver need to be patched?''' No, and that is a hard project rule: no guest driver patches, no host driver patches, no firmware patches, no hypervisor patches. The guest runs the stock NVIDIA driver, unmodified.

'''Does the host driver need to be patched or replaced?''' No. In the standalone configuration the distribution driver owns the physical GPU and Open-IOV is an independent kernel module. Building NVIDIA's OpenRM modules from source is needed only for the native vGPU-management workflows, not for the QEMU/emulated path ([[Install and First Run|<code>Install-and-First-Run.md</code>]] §2).

'''Why is <code>nvidia-smi</code> working but my CUDA kernel not running?''' Because execution is the frontier, not the setup. Today, a stock guest driver probes, <code>nvidia-smi</code> works, device enumeration works, and the full CUDA '''driver-API''' path (context creation, allocation, copies, free, destroy) completes; a guest '''kernel launch''' is forwarded to the physical engine queue but the engine does not yet execute the forwarded launch descriptor (<code>[wall]</code>). This is the project's known status, not a misconfiguration — see [[Project Status|<code>Project-Status.md</code>]] and [[The Execution Frontier|<code>Execution-Frontier.md</code>]].

'''How fast is it?''' No published numbers yet. Copy-engine-class forwarding and its completion and fault relay are implemented (default off) and GR execution is the open frontier; even the initialization path has not yet been measured under the project's methodology. Any performance claim will ship together with its methodology and measurements.

== Compatibility ==

'''Will it support Windows guests?''' Yes, it is a target: same architecture, measured bring-up. [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] lists the support ladder.

'''Can I run several VMs on one GPU?''' That is a core design goal: one mediated device per VM, IOMMU isolation, and explicit scheduling at the GPU's memory-management unit. Multi-tenant maturity depends on the scheduling and isolation work tracked in the project's known-unknowns document.

'''Does it support Xen / Qubes OS Air / Argo / OpenXT?''' Designed for them. The emulation core is hypervisor-independent by construction; the transport backends are on the roadmap. KVM/QEMU with VFIO mdev is the reference transport today.

'''What about MIG?''' MIG is a hardware partitioning mechanism; the project's scheduling policy can use it where available. It is part of the multi-tenancy work.

'''Which GPUs are supported?''' Turing TU104 is the reference platform. Other generations are targets or experimental combinations — see [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] for the exact ladder and what each status means.

== Legal and licensing ==

'''How is this legal / license-clean?''' Open-IOV's own code is BSD-3-Clause (GPL-2.0 for the kernel modules that link GPL-only kernel exports); NVIDIA's software is used unmodified and is not redistributed. The project interoperates with the interfaces rather than relicensing anything. Details: the repository <code>LICENSE</code> and the legal reference in <code>docs/09-reference/</code>.

'''Can my company use this commercially?''' Yes, under BSD-3-Clause (kernel modules GPL-2.0), with no fee. Co-development is welcome — see [[HAAEEP and Collaboration|<code>HAAEEP-and-Collaboration.md</code>]].

== Contributing ==

'''Where do I report bugs or ask for help?''' Open an issue in the repository with a reproduction and the standard captures (host dmesg, guest dmesg, guest <code>lspci -vvv</code>, guest driver parameters). A reproduction plus logs is worth more than a description.

'''How do I contribute support for my GPU / driver / hypervisor?''' Start from [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] and the portability workflow in the repository. The deliverable is a profile plus evidence from the project's evaluation ladder.

'''How do I debug the firmware side?''' The risc-v-gsp-rm-debugger workbench runs GSP firmware in a RISC-V virtual machine with hooks, watchpoints and recorded sessions — [[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]].

The vGSP Execution Plane (vEP)

2026-10-12T04:40:08Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= The vGSP Execution Plane (vEP) =

<blockquote>'''Canonical, maintained version of this page:''' [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>../docs/03-architecture/13-vexecution-plane.md</code>]. This wiki page is the condensed public deep-dive: what the layer is, how a guest CUDA launch flows through it, where it stands today, and where it is going. It is condensed from the canonical document and its cited evidence; when the two disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

Status: 2026-10-08, tree <code>master @ 1f571dda</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or parameter-gated; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> named success latch; <code>[planned]</code> designed, not built.

== 1. What it is ==

'''vGSP (virtual GSP)''' is our host-side reimplementation of the GPU System Processor interface: the virtual device plus the emulated GSP-RM service the stock guest driver talks to, implemented by <code>vgsp.ko</code>. The '''vGSP Execution Plane (vEP)''' is the half of vGSP that turns guest-authored GPU work into physical execution and guest-visible completion: channel ring and USERD discovery, GPU VA-to-PA truth, semaphore-release application, completion counters and notifiers, QMD handling for forwarded launches, and submission to the physical GPU's engines under the module-owned backend session.

The other half — the control plane (boot handshake, message queues, RPC, RM objects, BAR shadows) — makes the guest driver ''run''. The vEP makes the guest's work ''run''. The canonical document defines the name and rejects the alternates ("work plane", "submission plane", "forwarding plane", ...) that would collide with existing project vocabulary.

'''Four work domains''' organize the subsystem:

{| class="wikitable"
|-
! Domain
! Question
! Implemented by
|-
| Scheduler
| which channel's work runs, at what pace
| doorbell steering, the per-channel pump, discovery cadence, one ordered workqueue, one in-flight request
|-
| Allocator
| where rings, USERDs, counters, notifiers and scratch windows live
| ring/USERD autopin and pins, content-keyed pairing, counter derive, ladder scratch/PT pools, the BAR1 vidmem shadow
|-
| Control
| which RM objects/controls shape execution
| channel and event registration, <code>SET_PAGE_DIRECTORY</code>, <code>PROMOTE_CTX</code>, engine topology, the mode matrix
|-
| Share
| how clients/mdevs share the one GPU
| mdev backend refcount, one real session under a single lock, one workqueue; per-mdev clients and fairness are planned
|}

== 2. The shape of the plane ==

[[File:The-vGSP-Execution-Plane-diagram.png|frameless|center|upright=2.2|The vGSP execution plane (vEP): one CUDA launch, end to end.]]

<pre>guest: push + QMD + GPFIFO entry + doorbell (BAR0 0x88704)
-> pump (per-channel) -> ring/USERD discovery -> VA-to-PA truth -> decode
-> release apply + counter publish + notifier/event -> MSI-X -> guest wakes
-> (for real work) ordered workqueue -> ladder submission -> real GSP/engines
-> engine-authored release -> poll -> completion publication</pre>

== 3. One CUDA kernel launch, end to end ==

# '''Launch build (guest).''' <code>libcuda</code>/<code>nvidia.ko</code> build the GR push: context methods, the inline launch descriptor (QMD), SASS and parameter pointers, and a release target. They append a GPFIFO entry and ring the BAR0 doorbell. <code>[live]</code> observed with the stock driver.
# '''Steering and truth (host).''' The doorbell wakes the service thread; the pump resolves the channel's ring VA and USERD, reads <code>GP_PUT</code>, fetches the GPFIFO entry and decodes the method stream. Translations are read-back proven from driver-published page directories, pins, or content-rescue. <code>[live]</code> for CE; <code>[wip]</code> for GR (discovery now succeeds, see §4).
# '''Capture.''' The decoder accumulates the QMD window and captures the launch candidate; the sema service queues one request on the single ordered workqueue. <code>[live]</code> observed (ring consumed, methods execute).
# '''Physical submit.''' The backend relocates the QMD, copies program/param pages into module shadow windows whose PTEs were planted before the channel commit (the engine never sees a late PTE write — it caches 128-byte fetch lines with no invalidate), builds the launch method block, publishes it to the ladder channel's GPFIFO and rings the real doorbell. <code>[gated]</code> default-off; latched on a proven real GR submission.
# '''Engine.''' The front-end fetches the entry and executes methods; then the '''inline-QMD install''' must materialise the descriptor for the engine to read, currently '''<code>[wall]</code>''' — not observed. Context validity, destination mapping and fetch/alignment are exonerated; the recovered-method-stream form is the investigation.
# '''Completion.''' The engine authors the QMD release word and the channel release; the host applies the deferred release exactly once, publishes the completion counter, clears the notifier and posts the channel event; MSI-X wakes the guest. CE work is live in this shape <code>[gated]</code>; for GR the completion-counter publication is the current second face of the wall.

== 4. How work is found: discovery, truth, completion ==

* '''Ring/USERD discovery''' has three cooperating sources: injected pins, a kernel-native trusted walk of driver-published page directories, and content-keyed discovery when neither names the ring. Content discovery scans '''both apertures''' (emulated vidmem and guest RAM), runs the FB scan first on a 2-second cadence, pairs a ring with the rank's staged ctx-init push, and validates the pair before pinning. Since the 2026-10-08 re-arm work (<code>7abd8f87</code>, <code>66388560</code>) the discovery no longer retires permanently at a pass cap.
* '''The semaphore service''' decodes releases, translates their target pages through the trusted resolver plus a driver-published-VAS pool resolver, defers untranslatable releases with retry telemetry, and applies each release exactly once — in software, or by letting the engine author it under the relay (default off).
* '''Completion counters''' use a fixed 20-slot pool (a partial live channel count poisoned slots before the fix); each decoded release lands in the channel's exact slot. '''Notifiers''' are serviced by the pump, and family events are delivered as RM events + MSI-X through the guest's registered event objects.

== 5. Challenges, honestly ==

* '''GR dispatch wall.''' The forward path runs front-end methods, the context is valid and the destination window is accepted — and the descriptor install is still not observed. The mode matrix carries dozens of single-variable falsifiers (promote shapes, prelude/PCAS form, local-memory block, lane controls, shadow windows, pointer relocation).
* '''Completion publication.''' A boot where GR rings were discovered and releases applied still saw the guest re-queue the same push and give up. A later <code>sema_dbg=Y</code> run settled it: the release was written to the exact slot, the notifier cleared and the event posted, while the completion counter publication never wrote because the fixed-total autoreg mapping makes it a structural no-op. The leading reading is the counter-slot publication; the GP_GET write-back is the secondary, still-open candidate. Both are debug-gated today.
* '''Reproducibility (ISSUE-0001).''' The green/stall split is per-boot. The first fix wave landed and the earlier CE freeze is gone; the stall moved to GR ring discovery, which the latest discovery work addresses. Acceptance is 10 cold boots green, pending.
* '''Multiple clients (ISSUE-0002).''' A second CUDA process wedges in guest UVM because host per-client sema/GR state is not reset when the guest re-runs its per-client boot handshake. Per-client state containers and reset are the designed path.

== 6. Portability ==

The '''architecture''' ports: observe pushes, resolve truth, decode, execute/forward, publish completion. So does the '''RPC/RM object model''' and the mode-matrix practice. What does '''not''' port is address-level fact: ring conventions, VA windows, QMD offsets, context-image layout, PTE encodings, engine inventories and scratch layouts must be re-derived per driver version and GPU architecture by measurement, never guessed. A new version enters as a profile plus a measured delta; driver 610 additionally sits behind an explicit human approval gate.

== 7. Future ==

The planned next steps are reproducibility acceptance, the descriptor-install and completion-publication investigations (offline-first with the RISC-V GSP-RM debugger where possible), the arbitrary-kernel acceptance suite, then the near-native performance program. The unsolved future classes — multi-tenant fairness, fault replay/cancel, context switching between tenants, hostile-guest hardening, and version drift — each have a proactive design direction in the canonical document. 3D graphics reuses this pipeline once compute dispatch is proven; it adds engine breadth (video/image), more channels and synchronisation volume, and a much larger method vocabulary, but not a new plane.

----

''Canonical document: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/13-vexecution-plane.md <code>docs/03-architecture/13-vexecution-plane.md</code>]. See also: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/09-full-system-picture.md full system picture], [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/11-physical-backend-guest-wiring.md physical backend guest wiring], [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md the execution frontier].''

The Execution Frontier

2026-10-12T04:40:07Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= The Execution Frontier =

<blockquote>'''What this page is.''' The current leading edge of guest-facing execution: the chain that must fire for a stock guest CUDA kernel to run on the physical GPU, what has been proven about it, where it stops, how we know, and the open lines of investigation and theories. It is evidence-led: every statement carries a live observation, a decompilation finding, or a clearly labelled theory. Condensed from [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12 (source page dated 2026-10-06), tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed; <code>[wip]</code> partial; <code>[wall]</code> current blocker; theories are labelled as such.

== 1. The chain that should fire ==

A guest CUDA launch becomes a '''method stream''' in a push buffer, referenced by a '''GPFIFO ring entry''', announced by a '''doorbell'''. The host emulation decodes it and forwards an equivalent method stream to a real channel owned by the module's resource-manager session. The hardware '''front-end''' fetches and executes the methods, the launch descriptor ('''QMD''') is materialised and read by the '''GR engine''', the grid is dispatched to the SMs, and completion is published as '''semaphores''' the guest waits on.

<pre>guest: builds launch + method stream
GPFIFO entry + doorbell
-> host: doorbell trap, decode, rebuild forward stream,
prewrite QMD, ring/USERD, doorbell
-> front-end: fetch GPFIFO entry (ring cursor advances)
-> front-end: execute methods (object bind, cache ops, report sem)
-> front-end: install inline QMD into the descriptor buffer
-> GR engine: read QMD through the channel's page tables
-> context switch loads the channel context (FECS/ucode)
-> GR: dispatch grid to the SMs
-> engine authors release semaphore + channel semaphore
-> host polls release, relays to guest; CUDA wakes</pre>

{| class="wikitable"
|-
! #
! Stage
! State
|-
| 1
| Guest builds launch + method stream
| <code>[live]</code> observed (stock driver, unmodified)
|-
| 2
| GPFIFO entry + doorbell
| <code>[live]</code> observed
|-
| 3
| Host decode + forward stream + QMD prewrite + ring/USERD
| <code>[live]</code> observed
|-
| 4
| Front-end fetches the ring entry
| <code>[live]</code> observed (ring cursor advances)
|-
| 5
| Front-end executes methods
| <code>[live]</code> observed (object bind, cache ops, report semaphore)
|-
| 6
| '''Inline-QMD install''' (descriptor materialised)
| <code>[wall]</code> not observed
|-
| 7
| '''Engine reads the QMD'''
| <code>[wall]</code> not observed
|-
| 8
| Context switch loads the channel context
| <code>[live]</code> context validity at dispatch is ruled in
|-
| 9
| GR dispatches the grid to the SMs
| never observed on this stack
|-
| 10
| Engine authors release + channel semaphore
| absent for forwarded launches
|-
| 11
| Host relays completion; guest wakes
| the guest waits and times out
|}

'''One-line summary:''' the guest, the emulation, the forwarding path and the front-end are live; the '''launch-descriptor install path''' (stages 6–7) is the leading edge, and with it stages 9–11.

== 2. Where the leading edge is ==

A stock guest program now reaches its kernel launch and submits it through the forwarding path. On the wire the ring entry is consumed and the front-end methods run — but the launch descriptor is never materialised, the engine never reads a QMD, no engine-authored completion appears, and the guest blocks in its synchronisation wait.

This is '''not''' a context-validity wall (ruled out at dispatch) and '''not''' a window-mapping wall for the descriptor destination (that install is accepted). The wall is the '''form of the launch method stream''': what the front-end requires in order to install the inline descriptor, and what the engine requires in order to read and dispatch it.

== 3. How we know (observables and their limits) ==

{| class="wikitable"
|-
! Observable
! What it proves
! What it does not prove
|-
| Ring entry consumed; ring cursor advances
| the front-end is alive and fetching
| nothing about the engine
|-
| Method execution on the ladder channel
| the front-end can execute the method vocabulary
| '''not''' engine execution — these signals can be front-end-authored
|-
| Descriptor buffer prewritten with a detectable sentinel
| whether the descriptor was materialised after submission
| a read does not modify memory; the sentinel detects installs, not reads
|-
| Install sentinel unchanged after submit
| the inline-install method did not run
| it does not say why
|-
| Engine-authored release word absent; poll times out
| the engine did not complete the launch
| it does not say whether the engine started
|-
| No GR fault recorded
| the engine did not raise this error
| silence is not health: a skipped launch is silent by construction
|-
| Channel instance context flag set
| the channel is context-valid at dispatch
| nothing about the descriptor path
|}

'''Consequence:''' progress on this frontier must be measured by an '''engine-authored''' signal (the release word advancing) — not by front-end signals, which we have already learned to distinguish.

== 4. What is exonerated (do not relitigate) ==

* Context validity at dispatch — the channel's context is valid when the launch is submitted (instance flag set).
* The descriptor destination window mapping — installed in the engine's table view at submit.
* Front-end fetch and control-block alignment — the entry is consumed with the host control block at offset 0.
* The shared context-buffer one-shot rule — traced to a shared per-GPU buffer already initialised in the same cycle; not the dispatch delta.
* Parenting/TSG for the fetch — both variants fetch identically.
* The copy-engine path — CE-class work, releases and fault relay are live (default off); the GR path is a separate problem.

== 5. The clear issues (ranked) ==

'''Issue A — the forwarded stream carries a verbatim guest prelude that cannot complete.''' The forwarded stream includes a verbatim copy of the guest's prelude whose inline-data writes target addresses not present in the engine's view of the channel's page tables, and the referenced data is not relocated. The shorter, proven stream shape fires its tail (channel semaphore) but still does not schedule the descriptor.

'''Issue B — the descriptor-virtualization control method is absent.''' The method that arms the inline-descriptor install path is not present in the forwarded stream. Fixing it should materialise the descriptor — falsifiable in one cycle by watching the sentinel and the engine-read signal.

'''Issue C — diagnostic hygiene.''' The sentinel placed at word 0 overlaps real descriptor fields; it must be moved to an unused word so diagnostics do not perturb what they measure.

== 6. Lines of investigation (live) ==

# '''Restore the proven launch shape''' as the baseline and '''re-add the virtualization control method''' — one variable at a time, with the install sentinel and the engine-read signal as the readout.
# '''Relocate or drop the verbatim prelude''' so inline-data writes target addresses present in the engine's view, or replace it with the host-side staging the proven shape already uses.
# '''Move the sentinel''' and re-run to confirm the readout is clean.
# '''Diff against a known-good reference''' — our own ladder dispatching a descriptor built from the same source material, or a captured working launch of the stock stack on the same GPU — converting the remaining work from inference to differential.
# '''Offline probing''' of any rule the front-end or engine is believed to require, using the firmware debugger before spending a supervised cycle ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]).

== 7. Theories (labelled; not yet proven) ==

* '''T1 — the virtualization control is the gate.''' The front-end will not materialise an inline descriptor until the control method enables the install path.
* '''T2 — materialise-then-read.''' The engine only schedules a descriptor that exists in its own view; an install targeting unmapped destinations aborts silently rather than faulting.
* '''T3 — the verbatim prelude poisons the launch.''' Its refused writes leave the launch path unable to schedule the descriptor.
* '''T4 — relocation is the missing transform.''' The guest's descriptor is valid for the guest's view of memory; the forwarding path must relocate referenced data into the channel's view before install.

Each theory has a one-cycle falsifier: the install sentinel, the engine-read signal, and the engine-authored release, in that order.

== 8. Claim discipline ==

* '''Engine-authored release is the only proof of execution.''' Front-end signals are necessary but not sufficient.
* '''Silence is not health.''' No fault recorded does not mean no fault occurred.
* '''Emulator verdicts are instruction/memory-level.''' They bound behaviour, never timing or device environment.
* '''A wall is closed only by an acceptance signal''', named before the experiment.

== 9. Related pages ==

* [[Project Status|<code>Project-Status.md</code>]] — the project-level ladder.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the guest-side client path.
* [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — the backend the work is forwarded to.
* [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — the execution layer's design.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]] — page tables, QMDs and the engines.
* [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>] — the canonical, continuously-updated version with the full evidence tables.

Error Codes

2026-10-12T04:40:07Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= GSP Error Codes =

<blockquote>'''Historical document — superseded.''' The GSP-specific POSIX-style names in this page are historical labels from early reverse engineering, not wire statuses. The emulation uses NVIDIA <code>NV_STATUS</code> values: <code>NV_OK</code> <code>0</code>, <code>NV_ERR_NOT_SUPPORTED</code> <code>0x56</code>, <code>NV_ERR_INVALID_PARAMETER</code> <code>0x3B</code>. Canonical reference: [https://git.vgpu.io/open-iov.git/tree/docs/09-reference/glossary-and-faq.md <code>../docs/09-reference/glossary-and-faq.md</code>] and the kernel's <code>NV_STATUS</code> usage (<code>../kernel/</code>).</blockquote>

== Standard POSIX Errors (Negative Return Values) ==

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0
| 0x00000000
| OK
| Success
|-
| -1
| 0xFFFFFFFF
| EPERM
| Operation not permitted
|-
| -2
| 0xFFFFFFFE
| ENOENT
| No such file/entry
|-
| -3
| 0xFFFFFFFD
| EINVAL
| Invalid argument
|-
| -6
| 0xFFFFFFFA
| ENODEV
| No such device
|-
| -12
| 0xFFFFFFF4
| ENOMEM
| Out of memory
|-
| -16
| 0xFFFFFFEE
| EBUSY
| Device/resource busy
|-
| -22
| 0xFFFFFFEA
| EINVAL
| Invalid argument
|-
| -28
| 0xFFFFFFE4
| ENOSPC
| No space left on device
|-
| -37
| 0xFFFFFFDB
| ENOLCK
| No locks available
|-
| -56
| 0xFFFFFFC8
| EBADMSG
| Bad message
|-
| -84
| 0xFFFFFFAC
| EOVERFLOW
| Value too large
|-
| -90
| 0xFFFFA6
| ENOSTR
| Not a stream
|-
| -91
| 0xFFFFA5
| ENODATA
| No data available
|-
| -94
| 0xFFFFA2
| ENOSR
| Out of streams resources
|-
| -97
| 0xFFFFFF9F
| EPROTO
| Protocol error
|-
| -111
| 0xFFFFFF91
| ETIMEDOUT
| Connection timed out
|}

== GSP-Specific Error Codes (Observed in Firmware) ==

=== Memory Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x1E
| 0x0000001E
| ETIME
| WPR/RPC timeout
|-
| 0x1F
| 0x0000001F
| EDEV
| Device error
|-
| 0x25
| 0x00000025
| EOVERFLOW
| Region/offset overflow
|-
| 0x37
| 0x00000037
| EINVAL
| Invalid alignment/size
|-
| 0x51
| 0x00000051
| ENOMEM
| Allocator exhausted
|-
| 0x56
| 0x00000056
| EBUSY
| Resource busy
|-
| 0xFFFF
| 0xFFFFFFFF
| EPERM
| Permission denied
|}

=== RPC Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x1D
| 0x0000001D
| EFAULT
| Bad address in RPC
|-
| 0x1E
| 0x0000001E
| ETIMEDOUT
| RPC timeout
|-
| 0x1F
| 0x0000001F
| EPROTO
| Protocol violation
|-
| 0x20
| 0x00000020
| EBADRPC
| Invalid RPC type
|-
| 0x21
| 0x00000021
| EBADSEQ
| Sequence mismatch
|-
| 0x22
| 0x00000022
| EBADSIZE
| Payload size error
|}

=== BAR Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x30
| 0x00000030
| EBARINVAL
| Invalid BAR index
|-
| 0x31
| 0x00000031
| EBARPERM
| BAR access denied
|-
| 0x32
| 0x00000032
| EBARSIZE
| Aperture size error
|-
| 0x33
| 0x00000033
| EBARALIGN
| Alignment error
|-
| 0x34
| 0x00000034
| EBARRANGE
| Offset out of range
|-
| 0x35
| 0x00000035
| EBARBUSY
| BAR busy
|}

=== WPR Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x40
| 0x00000040
| EWPRINVAL
| Invalid WPR region
|-
| 0x41
| 0x00000041
| EWPRPERM
| WPR permission denied
|-
| 0x42
| 0x00000042
| EWPROVERLAP
| Overlapping WPR regions
|-
| 0x43
| 0x00000043
| EWPRBUSY
| WPR operation in progress
|-
| 0x44
| 0x00000044
| EWPRTIMEOUT
| WPR timeout
|}

=== Migration Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x50
| 0x00000050
| EMIGINVAL
| Invalid migration phase
|-
| 0x51
| 0x00000051
| EMIGNOMEM
| Migration buffer OOM
|-
| 0x52
| 0x00000052
| EMIGBUSY
| Migration in progress
|-
| 0x53
| 0x00000053
| EMIGPHASE
| Phase mismatch
|-
| 0x54
| 0x00000054
| EMIGVERIFY
| Verification failed
|-
| 0x55
| 0x00000055
| EMIGCOMMIT
| Commit failed
|-
| 0x56
| 0x00000056
| EMIGABORT
| Migration aborted
|}

=== Scheduler Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x60
| 0x00000060
| ESCHINVAL
| Invalid schedule params
|-
| 0x61
| 0x00000061
| ESCHPRIO
| Priority error
|-
| 0x62
| 0x00000062
| ESCHYIELD
| Yield failed
|-
| 0x63
| 0x00000063
| ESCHPREEMPT
| Preemption error
|}

=== Notification Errors ===

{| class="wikitable"
|-
! Code
! Hex
! Name
! Description
|-
| 0x70
| 0x00000070
| ENOTIFYINVAL
| Invalid notification
|-
| 0x71
| 0x00000071
| ENOTIFYBUSY
| Notification queue full
|-
| 0x72
| 0x00000072
| ENOTIFYDROP
| Notification dropped
|}

== Error Code Mapping (Kernel Driver) ==

<pre>static int gsp_errno_to_errno(int gsp_err) {
switch (gsp_err) {
case 0: return 0;
case -1: return -EPERM;
case -2: return -ENOENT;
case -3: return -EINVAL;
case -6: return -ENODEV;
case -12: return -ENOMEM;
case -16: return -EBUSY;
case -22: return -EINVAL;
case -28: return -ENOSPC;
case -37: return -ENOLCK;
case -56: return -EBADMSG;
case -84: return -EOVERFLOW;
case -90: return -ENOSTR;
case -91: return -ENODATA;
case -94: return -ENOSR;
case -97: return -EPROTO;
case -111: return -ETIMEDOUT;

// GSP-specific
case 0x1E: return -ETIMEDOUT;
case 0x1F: return -EIO;
case 0x25: return -EOVERFLOW;
case 0x37: return -EINVAL;
case 0x51: return -ENOMEM;
case 0x56: return -EBUSY;
case 0xFFFF: return -EPERM;

// BAR
case 0x30: return -EINVAL;
case 0x31: return -EPERM;
case 0x32: return -EINVAL;
case 0x33: return -EINVAL;
case 0x34: return -ERANGE;
case 0x35: return -EBUSY;

// WPR
case 0x40: return -EINVAL;
case 0x41: return -EPERM;
case 0x42: return -EEXIST;
case 0x43: return -EBUSY;
case 0x44: return -ETIMEDOUT;

// Migration
case 0x50: return -EINVAL;
case 0x51: return -ENOMEM;
case 0x52: return -EBUSY;
case 0x53: return -EINVAL;
case 0x54: return -EIO;
case 0x55: return -EIO;
case 0x56: return -ECANCELED;

// Scheduler
case 0x60: return -EINVAL;
case 0x61: return -EINVAL;
case 0x62: return -EAGAIN;
case 0x63: return -EAGAIN;

// Notification
case 0x70: return -EINVAL;
case 0x71: return -EAGAIN;
case 0x72: return -ENOBUFS;

default: return -EIO;
}
}</pre>

== Debugging Error Codes ==

=== GSP Log Format ===

<pre>GSP_ERROR: rpc_type=0x0A seq=0x1234 err=0x31 (EBARPERM) vf=2 addr=0x7F000000</pre>

=== Common Error Scenarios ===

{| class="wikitable"
|-
! Scenario
! Error Code
! Resolution
|-
| VF writes to unmapped BAR
| 0x31 (EBARPERM)
| Check VF BAR config
|-
| Migration verify fails
| 0x54 (EMIGVERIFY)
| Check memory integrity
|-
| WPR timeout on page fault
| 0x44 (EWPRTIMEOUT)
| Increase timeout
|-
| RPC sequence mismatch
| 0x21 (EBADSEQ)
| Reset RPC client
|-
| Allocator exhausted
| 0x51 (ENOMEM)
| Free unused VF memory
|-
| BAR aperture too small
| 0x32 (EBARSIZE)
| Increase aperture
|}

=== Error Injection (Testing) ===

<pre>// Inject errors for testing
#define GSP_INJECT_ERROR(rpc_type, err_code) \
gsp_test_inject_error(gsp, rpc_type, err_code)

void gsp_test_inject_error(struct gsp_device *gsp,
uint32_t rpc_type, int err_code) {
struct inject_payload p = {
.rpc_type = rpc_type,
.error_code = err_code,
.enable = 1,
};
gsp_rpc_send_sync(gsp->rpc, RPC_DEBUG, &p, sizeof(p), NULL, NULL);
}</pre>

Guest Driver Lifecycle: A Field Guide

2026-10-12T04:40:06Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= Guest Driver Lifecycle: A Field Guide =

<blockquote>'''Scope.''' A walking tour of what happens between "guest boots" and "CUDA initializes", using message, function and control IDs observed from an unmodified 570.144 guest. Canonical source: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[design]</code> stages documented ahead of arrival.</blockquote>

== 1. How to read this page ==

Each stage lists what the '''guest''' does, what the '''device/firmware''' must answer, what host emulation does today, and the log signature to look for (host-side <code>dmesg</code> with prefixes <code>vgsp:</code> / <code>vgsp-rm:</code>; guest-side <code>dmesg</code> in the VM). IDs are for 570.144 on the Turing (TU104) profile; other generations add IDs and change versioned values, but the ''stages'' are stable.

== 2. Stage map ==

<pre>guest boots
│
├─ 1. PCI probe & config space ........ lspci is the reference
├─ 2. GSP boot handshake .............. WPR, FWSEC, booter, RISC-V core
├─ 3. Static configuration ............ GET_GSP_STATIC_INFO (65)
├─ 4. Message queues .................. SET_GUEST_SYSTEM_INFO (1) + init traffic
├─ 5. RM control battery .............. GSP_RM_CONTROL (76) × ~160
├─ 6. Object creation ................. GSP_RM_ALLOC (103) × ~60
└─ 7. UVM registration/context ........ cuInit → channel manager → execution frontier</pre>

== 3. PCI probe and config space ==

'''Guest:''' reads IDs, walks capabilities, sizes BARs, enables Memory Space and Bus Master, programs MSI-X. '''Device must:''' report a coherent TU104-family GPU with 16 MB BAR0, 256 MB BAR1, an MSI-X capability and a link capability the driver accepts. '''Emulation:''' config space is synthesized and BAR sizes are enforced. The hypervisor is the usual source of surprises — a device on the root bus can have its PCIe capability rewritten to "root complex integrated endpoint", removing link information the driver wants; attach behind a root port. '''Log:''' guest-side <code>lspci -vvv</code> is the reference.

== 4. GSP boot handshake ==

'''Guest:''' writes WPR descriptors, releases FWSEC, then the booter, then the RISC-V core; polls a status mailbox for "GSP init done". '''Device must:''' complete every stage in the order the driver polls and leave the mailbox in the state the driver expects. '''Emulation:''' the host module models the stage state machine and auto-clears self-clearing triggers (e.g. MMU invalidate registers) so polls terminate.

<pre>vgsp: GSP-BOOT: ... trigger auto-cleared ...</pre>

== 5. Static configuration ==

'''Guest:''' <code>GET_GSP_STATIC_INFO</code> (65) → a large <code>GspStaticConfigInfo</code> structure; reads the VBIOS image from the BAR0 window; queries device/FB/BUS info. '''Device must:''' provide self-consistent sizes, engine inventories, memory geometry, interrupt topology and a parseable VBIOS blob. '''Emulation:''' the static structures are synthesized for TU104 (FB size, BAR layout, engine list, vGPU-friendly VF values).

<pre>vgsp-rm: GET_GSP_STATIC_INFO → ...</pre>

== 6. Message queues ==

'''Guest:''' sends <code>SET_GUEST_SYSTEM_INFO</code> (1) with libos regions, a queue buffer description and a page table (the buffer is <code>NV_MEMORY_NONCONTIGUOUS</code>); also <code>GSP_SET_SYSTEM_INFO</code> (72), <code>SET_REGISTRY</code> (73) and <code>CONTINUATION_RECORD</code> (71) as init traffic. '''Device must:''' consume them silently (async) and consume the command queue exactly in the guest's write order. '''Emulation:''' parses the queue chain, builds the geometry, links the status queue and consumes elements one by one — '''element-based, not byte-stream-based''': one message is one element and may span pages.

<pre>vgsp: libos region[i] id8=... pa=... size=...
vgsp: queue args sharedPA=... ptes=... cmdOff=... statOff=...</pre>

== 7. The RM control battery ==

'''Guest:''' sends <code>GSP_RM_CONTROL</code> (76) for every subsystem it initializes. Of the ~160 observed control IDs, 23 are special-cased with real values; the rest are answered with a generic <code>SUCCESS</code> and zeroed params (the fallback).

{| class="wikitable"
|-
! Control
! Meaning
! What matters
|-
| <code>0x20800a40</code> family
| internal device info table
| per-index values, entry count
|-
| <code>0x20801112</code>
| FIFO device info table
| engine inventory; count must match engine queries
|-
| <code>0x20800a5c</code>
| interrupt kernel table
| vector topology the driver maps
|-
| <code>0x20800a59</code>
| GMMU static info
| fault buffer sizes; a wrong value breaks UVM early
|-
| <code>0x20800a01</code>
| display static info
| display count; headless values are fine
|-
| <code>0x20801823/-a/-b</code>
| BUS/FB info V2
| sysmem connection type and PCIe link caps feed bandwidth math
|-
| <code>0x20802a07</code>
| CE physical caps
| per-engine bits; UVM refuses channels without <code>sysmem</code> support
|-
| <code>0x20802a0a/0b</code>
| all-CE caps
| caps table for all engines
|-
| <code>0x20802a02</code>
| CE PCE mask
| one bit per engine
|-
| <code>0x20802a08</code>
| CE fault method buffer size
| method buffer sizing
|-
| <code>0x90f10106</code>
| vaspace reserved-PDE contract
| carries page-table physical addresses (see [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]])
|-
| <code>0x20800123</code> / <code>0x20800170</code>
| <code>GPU_GET_ENGINES</code> / <code>GET_ENGINES_V2</code>
| engine inventory; the guest must see the full list, including async copy (<code>COPY14</code>)
|-
| <code>0x800201</code>
| <code>GPU_GET_CLASSLIST</code>
| class list; must agree with engine list and CE caps
|-
| <code>0x2080012b</code>
| <code>GPU_PROMOTE_CTX</code> (560-byte params)
| load-bearing GR context-validity operation; a wrong shape is a server wedge handled by the supervised fail-safe
|-
| <code>0x20803039</code>
| NVLink masks/VBIOS info
| fallback: generic <code>SUCCESS</code> with zeroed params
|-
| <code>0x20800a70</code>
| BUS flush with sysmembar
| fallback: generic <code>SUCCESS</code> with zeroed params; UVM issues these often
|}

<pre>vgsp-rm: GSP_RM_CONTROL cmd=0x... hClient=0x... hObject=0x... paramsSize=...</pre>

== 8. Object creation ==

'''Guest:''' <code>GSP_RM_ALLOC</code> (103) creates the object tree. Typical order for a CUDA process:

<pre>0x0 NV01_ROOT (client root, 120-byte params)
0x80 NV01_DEVICE_0 (56 bytes)
0x2080 NV20_SUBDEVICE_0 (4 bytes)
0x90f1 FERMI_VASPACE_A (48 bytes) ← the GPU address space
0xa06c KEPLER_CHANNEL_GROUP_A (20 bytes) ← CE channel group
0xc46f TURING_CHANNEL_GPFIFO_A(368 bytes) ← the channel (GPFIFO VA, instance/userd descriptors)
0xc076 GP100_UVM_SW (0 bytes)
0xc5b5 TURING_DMA_COPY_A (8 bytes) ← CE object
0xc5c0 TURING_COMPUTE_A (16 bytes) ← compute object (engine registration path)</pre>

A hard-won rule from the <code>TURING_COMPUTE_A</code> (0xc5c0) alloc: the caller may issue it with <code>paramsSize=0</code> against a small stack local while the kernel side treats the class params as 16 bytes and copies reply params back. The emulation must '''echo the request params back in the alloc reply''', or the user-mode client segfaults. Channel parameters decode into <code>gpFifoOffset</code> (a GPU VA), <code>gpFifoEntries</code>, <code>engineType</code> (e.g. <code>COPY0 = 0x9</code>), and <code>NV_MEMORY_DESC_PARAMS</code> for instance/userd/ramfc/method buffers with '''physical''' addresses and apertures (<code>ADDR_SYSMEM = 1</code>, <code>ADDR_FBMEM = 2</code>).

<pre>vgsp-rm: GSP_RM_ALLOC class=0x... paramsSize=... hClient=... hParent=... hObject=...</pre>

== 9. UVM registration and the execution frontier ==

'''Guest:''' <code>cuInit</code> → <code>UVM_REGISTER_GPU</code> → channel-manager creation → page-tree initialization; each UVM channel does a first push and waits on a semaphore. '''Emulation today:''' the host decodes the pushes UVM generates and executes their memsets, copies and semaphore releases in software by translating the semaphore VA to a guest physical address — a bridge, not forwarding: no work reaches the physical GPU through it yet.

'''Progress:''' context creation now returns (the driver-API smoke completes end to end) and the frontier is execution. The host GR ladder allocates a GR0 channel and a compute-class object; a first semaphore push against an uncommitted GR context faulted <code>CTXNOTVALID</code>, and a following <code>GPU_PROMOTE_CTX</code> wedged GSP-RM into its designed locked fail-safe. Later, a forwarded guest CUDA launch reached the front-end (ring entry fetched, methods executed) with no observed inline launch-descriptor install and no grid dispatched; context validity at dispatch is ruled out. See [[The Execution Frontier|<code>Execution-Frontier.md</code>]]. The roadmap from here is:

<pre>UVM registers → cuInit returns → context creation → memory allocation
→ channel creation for the process → kernel launch → (real execution)</pre>

Each remaining step needs the same treatment: satisfy the interface, forward the work, relay the completions.

<pre>nvidia-uvm: ... Tracker entry for value N (sema VA 0x...) channel ID ... CE ...
nvidia-uvm: ... pending push '' started at ... in channel_init() releasing value N</pre>

== 10. Quick reference: function IDs from a stock 570.144 guest ==

{| class="wikitable"
|-
! Function
! Name
! Frequency per CUDA run (approx.)
|-
| 1
| SET_GUEST_SYSTEM_INFO
| once
|-
| 10
| FREE
| ~14
|-
| 47
| UNLOADING_GUEST_DRIVER
| on unload
|-
| 65
| GET_GSP_STATIC_INFO
| once
|-
| 70
| UPDATE_BAR_PDE
| a few
|-
| 71/72/73
| CONTINUATION_RECORD / GSP_SET_SYSTEM_INFO / SET_REGISTRY
| once each
|-
| 76
| GSP_RM_CONTROL
| ~160
|-
| 103
| GSP_RM_ALLOC
| ~60
|}

If a new driver version produces '''new function numbers''', do not guess: add a handler that logs the raw payload, then decode it against that version's observed traffic.

== 11. Related documentation ==

* [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/07-driver-lifecycle-field-guide.md <code>../docs/03-architecture/07-driver-lifecycle-field-guide.md</code>] — canonical field guide.
* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — GSP boot, message queues and RM architecture.
* [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]] — control/alloc emulation on the host side.
* [[RPC Message Format|<code>RPC-Message-Format.md</code>]] — wire format of the function 76/103 traffic.
* [[Address Reference|<code>Address-Reference.md</code>]] and [[GSP 570.144W Firmware Analysis|<code>GSP-570.144W-Firmware-Analysis.md</code>]] — ID reference and the firmware they belong to.
* [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — registration, the frontier, the host stages and the forwarding backend.

CUDA Userspace Driver

2026-10-12T04:40:06Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =

<blockquote>'''Status (2026-10-12, honest).''' The analysis covers the whole library for driver 570.144 and the tooling is version-agnostic. The ''control plane'' gets the stock guest stack through <code>cuInit</code>, device enumeration, <code>cuCtxCreate</code> and the full driver-API smoke test — allocation, copies, free and destroy complete end to end (<code>ALL OK</code>; the 64-byte payload round-trips exactly). The ''data plane'' decodes guest push buffers and executes the memory and semaphore work it understands; CUDA kernel launches are forwarded to the physical execution path, where a launch reaches the engine queue (front-end deposit and scheduling are observed) but an engine-authored release is not yet observed. Green runs are not yet reproducible on every boot — an open issue, tracked in the repository. Canonical detail: [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/01-architecture.md <code>01-architecture.md</code>] through <code>08</code>; reading order [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]; front door [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

== 1. What it is ==

<code>libcuda.so.1.1</code> is the userspace half of the CUDA driver: the library an application or language runtime loads to talk to a GPU. For 570.144 it is a '''single, fully stripped, position-independent shared object of about 71 MB''':

{| class="wikitable"
|-
! Property
! Value
|-
| Exports
| 659 (public <code>cu*</code> API plus internal/legacy entry points)
|-
| Imports
| 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path
|-
| <code>.text</code>
| <code>0x179ec0</code>–<code>0xe94c63</code> (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables)
|-
| <code>.rodata</code>
| ~53 MB — error strings, dispatch/version tables, '''embedded device code'''
|-
| CUDA note
| CUDA 12.8; supported kernel-driver ABIs <code>[470, 535, 550, 560, 565, 570]</code>
|}

Two entry surfaces coexist: the '''public driver API''' (<code>cuInit</code>, <code>cuCtxCreate_v2</code>, <code>cuMemAlloc_v2</code>, <code>cuLaunchKernel</code>, …) and the '''entry-table API''' — <code>cuGetExportTable</code>, a 128-bit-UUID-keyed lookup that hands internal tables to the CUDA runtime (<code>libcudart</code>), tooling and interposition libraries. Tables are versioned by a leading <code>struct_size</code> field and are append-only.

The library embeds '''device code''' — cubins for memcpy/memset kernels, the graph scheduler, syscall trampolines and preemption-handler assembly — and loads it through its own module machinery, so a stripped <code>libcuda.so</code> still contains many small ELF images in <code>.rodata</code>. The CUDA ''runtime'' (<code>libcudart</code>, the <code>cuda*</code> API) is a separate stock library that bottoms out in <code>libcuda</code>; it also calls through the entry tables, so a guest failure in a <code>cuda*</code> call may reach driver code through a different door than the public <code>cu*</code> entry point.

'''Open-IOV requires this library unmodified.''' In the guest it links against the guest's stock <code>nvidia.ko</code> / <code>nvidia-uvm.ko</code> exactly as on bare metal; there is no <code>LD_PRELOAD</code> shim and no patch. Instrumentation uses <code>ptrace</code> or kernel-side tracing, because the RM client issues raw <code>syscall(SYS_ioctl)</code> rather than libc <code>ioctl</code>.

== 2. Layer model ==

The library is strictly layered, and the boundaries are where the emulation interfaces live (condensed from <code>docs/01</code> and <code>docs/02</code>):

<pre>application / libcudart
| public cu* API or UUID entry tables
v
generated wrappers -> API layer -> CUDA internal engine
|-- driver-model layer (RM) -> /dev/nvidia* ioctls
|-- hardware layer (QMD/method encoding)
| -> GPU queues, doorbells
|-- UVM userspace (managed memory) -> /dev/nvidia-uvm
'-- JIT/ELF (PTX->SASS, cubin loading)
kernel driver + GSP-RM + GPU</pre>

* '''Generated wrappers and API layer.''' A master export list (name, platform flags, CUDA version introduced) generates the exported symbols, wrappers, tool callbacks and <code>cuGetProcAddress</code> tables (<code>_vN</code>/<code>_ptsz</code> variants are part of that data). Below it, the hand-written layer reads the per-thread context stack, validates state, takes one context lock, calls one engine function and returns a <code>CUresult</code> — the semantic work is always one layer down.
* '''Internal engine.''' Owns the object graph: process globals (init state, kernel-client handle, device list), device (chip/caps/limits, vtables), context (memory/channel/stream managers, QMD pools, ISR thread), stream, module/function, memory objects, events/graphs/libraries.
* '''Driver-model layer.''' Selected at initialization behind one function table: the '''RM backend''' (the one Open-IOV cares about) plus SoC, Amodel, MPS and WDDM. It turns engine operations into RM controls and object allocations over <code>/dev/nvidia*</code>, and owns the '''driver-ABI adaptation''' layer — one instance per supported kernel-driver ABI, chosen at <code>cuInit</code>.
* '''Hardware layer.''' Chip-specific method/QMD/semaphore/doorbell encoders selected by compute capability. The userspace driver ''builds'' the GPU command streams; the kernel driver allocates, maps, schedules and services interrupts.
* '''Synchronization model.''' Three primitives explain most of it: the '''tracking semaphore''' (per-channel counter of completed pushes), the '''marker''' ("all work before this point") and the '''QMD semaphore''' (released when a grid completes). Blocking waits use a per-context ISR thread and a DPC queue.

== 3. What it touches in a guest ==

This is the emulation contract; the full checklist is <code>docs/03</code> §9.

'''Device nodes.''' <code>/dev/nvidiactl</code> (process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object fds), <code>/dev/nvidiaN</code> (per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation), <code>/dev/nvidia-uvm</code> (managed memory and page migration) and <code>/dev/nvidia-vgpuN</code> (vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths).

'''ioctls.''' The control device uses <code>_IOWR('F', cmd, …)</code>: <code>200</code> <code>NV_ESC_CARD_INFO</code>, <code>210</code> <code>NV_ESC_CHECK_VERSION_STR</code>, <code>212</code> <code>NV_ESC_ATTACH_GPUS_TO_FD</code>. RM escapes include <code>0x2b</code> <code>NV_ESC_RM_ALLOC</code> ('''the main object-allocation path'''), <code>0x2a</code> <code>NV_ESC_RM_CONTROL</code>, <code>0x29</code> <code>NV_ESC_RM_FREE</code>, <code>0x4e</code>/<code>0x4f</code> <code>NV_ESC_RM_MAP_MEMORY</code>/<code>UNMAP_MEMORY</code>, <code>0x57</code>/<code>0x58</code> <code>NV_ESC_RM_MAP_MEMORY_DMA</code>/<code>UNMAP_MEMORY_DMA</code> and <code>0x4a</code> <code>NV_ESC_RM_VID_HEAP_CONTROL</code>. The request word <code>rsi=0xc030462b</code> in a crash dump decodes to "allocating an RM object with a 48-byte payload".

'''RM objects/classes.''' Every RM call names a class. Observed for 570.144 on an emulated TU104: <code>NV01_ROOT</code> (<code>0x0</code>), <code>NV01_DEVICE_0</code> (<code>0x80</code>), <code>NV20_SUBDEVICE_0</code> (<code>0x2080</code>), <code>NV01_EVENT</code> (<code>0x40</code>), <code>KEPLER_CHANNEL_GROUP_A</code> (<code>0xa06c</code>), a channel-group/runlist object (<code>0x9067</code>), <code>TURING_CHANNEL_GPFIFO_A</code> (<code>0xc46f</code>, the channel), <code>TURING_COMPUTE_A</code> (<code>0xc5c0</code>, the compute engine object), plus memory (<code>0x3e</code>) and VA-space classes. The observed <code>cuCtxCreate</code> order is root → device → subdevice → events → channel group → channel-group object → memory → channel → control <code>NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST</code> → compute object → engine registration.

'''Controls.''' Device/GPU info (<code>GPU_GET_ENGINES</code>, <code>GPU_GET_CLASSLIST</code>, <code>GPU_GET_VIRTUALIZATION_MODE</code>, <code>FB_GET_INFO</code>, <code>CE_GET_CAPS</code>, …), FIFO/channels (<code>NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST</code> <code>0x80170d</code>, GPFIFO scheduling, timeslices, <code>GET_CLASS_ENGINEID</code>), GR/context switches, memory, events and interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates unknown-but-successful controls; it only fails when a ''specific reply field'' is consumed and wrong.

'''Memory.''' VA reservation (<code>NV_ESC_RM_VID_HEAP_CONTROL</code>/<code>NVOS32</code>), backing allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and mapping (<code>NVOS33</code> into the process, <code>NVOS46</code> into a device VA space). All guest memory CUDA uses is either an RM allocation backed by emulated VRAM/BAR1 or pinned guest RAM the GPU must DMA to.

'''Channels, GPFIFO, USERD, doorbells.''' A channel is a GPFIFO (array of pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO, error notifier and (Volta and later) the USERD itself, passing the handles to the channel allocation; submission writes the GPFIFO put pointer into USERD and rings a doorbell token returned by the channel allocation. Pushbuffers are method streams terminated by a tracking-semaphore release and a new GPFIFO entry; the host emulation decodes them as such.

'''Synchronization below the API.''' Semaphores (plain GPU-visible memory values), 4 KB per-channel error notifiers, OS events (kernel objects used to wake blocked threads), interrupts routed through the kernel driver, and a trap handler for debuggers and <code>device-side assert</code>.

'''UVM and GSP-RM.''' When UVM is available the driver registers GPUs, VA spaces, channels and managed allocations and pushes UVM semaphores into the same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing from userspace — it still issues RM ioctls; inside the kernel driver controls become GSP-RM RPCs, which Open-IOV's host module answers.

== 4. Why forwarding matters, and where we are ==

The performance budget is dominated by '''push-buffer execution''', '''synchronization latency''' and (once, at setup) '''control-plane latency''' (<code>docs/04</code> §4). Only forwarding ''real work'' to the physical GPU meets the first two; software execution of state/memory ops is a correctness bridge, not the data plane (<code>docs/08</code>).

* '''Stated architecture:''' ''emulate the interface, pass through the work, never patch anyone.'' Prototype the host-RM/GSP proxy first for correctness, measure, then move the hot path to direct submission if the budgets demand it — work path abstracted from day one.
* '''Control plane today:''' GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and object/control replies are implemented; the stock guest stack completes <code>cuInit</code>, device enumeration, <code>cuCtxCreate</code> and the driver-API smoke test (allocation, copies, free and destroy).
* '''Current frontier:''' a guest CUDA kernel launch is forwarded and reaches the physical engine queue — the front end deposits the launch and the scheduling path accepts it — but the engine does not yet execute the forwarded launch descriptor: an engine-authored release is the only proof of execution and it is not observed. The per-boot green/stall variance on the emulation path is an open issue, tracked in the repository. The chain, the evidence and the live state are on [[The Execution Frontier|<code>Execution-Frontier.md</code>]].
* '''Data plane today:''' the host module decodes guest push buffers as method streams and executes the memory and semaphore work it understands (memsets, copies, semaphore releases), and forwards engine-class work to the physical backend; kernel launches are forwarded but not yet executed by the engine.

== 5. The crash-path lesson: replies are consumed data ==

The old deterministic <code>cuCtxCreate</code> segfault at <code>libcuda+0x453f90</code> shows why reply coherence matters (<code>docs/05</code> §3.5, <code>docs/04</code> §6). The engine-registration hook allocates a <code>TURING_COMPUTE_A</code> object with <code>pAllocParms</code> pointing at the caller's own '''8-byte stack local''' and <code>paramsSize = 0</code>; the guest driver does not honour <code>paramsSize</code>, so it reads '''16-byte''' class params starting there (the upper 8 bytes are the hook's saved <code>rbp</code>) and copies 16 bytes of reply params back over the same buffer. With the old zeroed reply that copy-back zeroed the saved <code>rbp</code>; the function's epilogue then loaded <code>rbp = 0</code> and its next access — reloading a pointer from <code>[rbp-0x38]</code> — faulted at <code>0xffffffffffffffc8</code> (the store was the kernel's <code>copy_to_user</code>).

'''The fix:''' echo the request's params back, forcing only <code>status</code> to <code>NV_OK</code>; the copy-back then writes exactly the bytes it read, and the fault disappears — the same thing the real GSP-RM does for these in/out params.

'''The general rule:''' the userspace library is a ''consumer of emulated data''. A wrong reply is worse than a slow one. Work up the reply-fidelity ladder — status-only → coherent zeroed params → coherent values for consumed fields → exact values — and validate each rung with the A/B oracle.

== 6. How to look things up ==

The toolchain is version-agnostic (<code>docs/06</code>):

<pre>tools/cudatool.py info|exports|functions|nearest|crash|delta <lib>
tools/callwalk.py <callgraph.tsv> callers|callees|chain ...</pre>

* '''<code>cudatool.py</code>''' — build id/sections/exports/ABI note (<code>info</code>), symbol map (<code>exports</code>), function ranges from <code>.eh_frame_hdr</code> (<code>functions</code>), containing function and nearest export for an address (<code>nearest</code>), dmesg <code>ip</code> to file vaddr (<code>crash</code>), and an interface-level diff (<code>delta</code>).
* '''<code>callwalk.py</code>''' — caller/callee trees and shortest call paths over the callgraph index.
* '''Annotated corpus''' — <code>570.144/annotations/names.tsv</code> (our name map; exports exact, internals grow over time) and <code>subsystems.tsv</code>; corpus headers carry the recovered name, file address and direct callers/callees. The generated corpus is not distributed.

'''Address convention (read this before citing an offset).''' Public docs cite '''file virtual addresses''' as <code>libcuda+0x...</code>. Ghidra loads these binaries at image base <code>0x100000</code>, so <code>program address = file vaddr + 0x100000</code>; the <code>tools/</code> wrappers translate automatically, raw Ghidra scripts do not. An earlier analysis passed file vaddrs straight to Ghidra and decompiled functions 0x100000 too low, so its decompiled-code conclusions were about the wrong functions (its live traces stayed valid). The same applies when transferring a name map to a new driver version.

== 7. Where the detail lives ==

{| class="wikitable"
|-
! Document
! What it gives you
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]
| the reading order, fast recipes, and the two findings that change how you debug
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/01-architecture.md <code>../userspace/cuda/docs/01-architecture.md</code>]
| the full layer/object/sync/memory model
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/02-build-and-execution.md <code>../userspace/cuda/docs/02-build-and-execution.md</code>]
| build model, embedded device code, init/threading, what it touches
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/03-driver-and-gpu-interface.md <code>../userspace/cuda/docs/03-driver-and-gpu-interface.md</code>]
| the emulation contract: nodes, ioctls, classes, controls, channels, UVM
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/04-virtualization-and-emulation.md <code>../userspace/cuda/docs/04-virtualization-and-emulation.md</code>]
| NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/05-570.144-decompilation.md <code>../userspace/cuda/docs/05-570.144-decompilation.md</code>]
| the binary map, the crash resolution, and the current engine-topology blocker
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/06-tooling.md <code>../userspace/cuda/docs/06-tooling.md</code>]
| every tool, its usage and its cost
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/07-version-upgrade-playbook.md <code>../userspace/cuda/docs/07-version-upgrade-playbook.md</code>]
| re-running the analysis on a new driver version
|-
| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/08-virtualization-approaches.md <code>../userspace/cuda/docs/08-virtualization-approaches.md</code>]
| forwarding options and the honest current stage
|-
| [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>]
| the project documentation front door
|}

BAR Emulation Specification

2026-10-12T04:40:05Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= BAR Emulation Specification (GSP 570.144W) =

<blockquote>'''Historical document — superseded.''' The <code>BAR_TRAP</code> RPC (type 0x0A) described here is '''not used''': in emulation mode it returns <code>-ENOTSUPP</code>, and BAR access is synchronous MMIO through the BAR shadow. Guest-visible BAR windows are BAR0 (16 MB register MMIO), BAR1 (256 MB VRAM aperture) and BAR3 (instance window). Canonical documentation: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

The GSP firmware implements BAR (Base Address Register) emulation for virtual GPU (vGPU) through the '''BAR_TRAP''' RPC (type 0x0A). The main handler at '''0x04CB8188''' (40 KB) manages BAR1 aperture, forced access, and memory backing. '''This RPC is not used by the live path''': in emulation mode <code>BAR_TRAP</code> returns <code>-ENOTSUPP</code>; guest BAR access is handled synchronously as MMIO through the BAR shadow (<code>../kernel/bar_emulation.c</code>), not via an RPC.

== Key Addresses ==

{| class="wikitable"
|-
! Component
! Address
! Description
|-
| BAR Trap Handler
| 0x04CB8188
| Main emulation logic (40 KB)
|-
| RPC Jump Table Entry
| 0x057675B7
| BAR_TRAP in secondary dispatcher
|-
| String: BAR1
| 0x0406570E, 0x04085DE0
| BAR1 identifier
|-
| String: Bar1Rest
| 0x040656DA
| BAR1 restore state
|-
| String: ForceBarAccess
| 0x040656F2
| Forced access control
|-
| String: Bar1Aperture
| 0x04065722
| Aperture configuration
|-
| Memory Allocator
| 0x429B450
| Arena for BAR backing store
|-
| Cache Management
| 0x429B428
| Flush/invalidate
|}

== Handler State Machine ==

=== Main Loop (0x04CB8188) ===

The handler runs a continuous event loop processing BAR trap requests:

<pre>while (true) {
// Wait for trap event (bitmask in unaff_s5)
event = wait_for_bar_event();

switch (event & 0x3F) {
case 0x01: // BAR1 write access
handle_bar1_write();
break;
case 0x02: // BAR1 read access
handle_bar1_read();
break;
case 0x08: // Aperture configuration
handle_aperture_config();
break;
case 0x10: // ForceBarAccess
handle_force_access();
break;
case 0x20: // Bar1Rest (restore)
handle_bar1_restore();
break;
default:
return ERROR_UNKNOWN_EVENT;
}
}</pre>

=== Event Bitmask ===

{| class="wikitable"
|-
! Bit
! Event
! Description
|-
| 0
| BAR1_WRITE
| Guest VF wrote to BAR1
|-
| 1
| BAR1_READ
| Guest VF read from BAR1
|-
| 3
| APERTURE
| Aperture size/offset change
|-
| 4
| FORCE_ACCESS
| ForceBarAccess triggered
|-
| 5
| BAR1_REST
| Restore BAR1 state
|}

== Memory Management ==

=== Allocator Arena (0x429B450) ===

<pre>// Allocate backing store for BAR emulation
void *bar_alloc(size_t size) {
// Calls UNK_ram_058A1A9C(0x429B450, size)
// Returns pointer in 0x429B000 region
}

// Free backing store
void bar_free(void *ptr) {
// Calls UNK_ram_05895BF4(0x429B428, ptr)
}</pre>

=== Allocation Sizes ===

* '''Standard BAR1 page''': 0x1000 (4 KB)
* '''Aperture buffer''': 0x40000 (256 KB) max
* '''Force access buffer''': 0x1000 (4 KB)

=== Cache Coherency ===

<pre>// Flush cache for DMA
void bar_cache_flush(void *addr, size_t size) {
UNK_ram_05895BF4(0x429B428, addr);
UNK_ram_05895BF4(0x429B428, addr + size);
}</pre>

== BAR1 Emulation Logic ==

=== Write Path (Event 0x01) ===

<pre>int handle_bar1_write(bar_trap_t *trap) {
// 1. Validate offset within aperture
if (trap->offset >= bar1_aperture_size)
return ERROR_OVERFLOW;

// 2. Check ForceBarAccess flag
if (force_bar_access && !(trap->flags & FORCE_ALLOWED))
return ERROR_PERMISSION;

// 3. Translate to backing store
uint8_t *backing = bar1_backing + trap->offset;

// 4. Perform write (1/2/4/8 bytes)
switch (trap->size) {
case 1: *backing = trap->value; break;
case 2: *(uint16_t*)backing = trap->value; break;
case 4: *(uint32_t*)backing = trap->value; break;
case 8: *(uint64_t*)backing = trap->value; break;
}

// 5. Flush to device if needed
if (device_needs_flush)
bar_cache_flush(backing, trap->size);

return OK;
}</pre>

=== Read Path (Event 0x02) ===

<pre>int handle_bar1_read(bar_trap_t *trap) {
if (trap->offset >= bar1_aperture_size)
return ERROR_OVERFLOW;

uint8_t *backing = bar1_backing + trap->offset;

switch (trap->size) {
case 1: *trap->out_value = *backing; break;
case 2: *trap->out_value = *(uint16_t*)backing; break;
case 4: *trap->out_value = *(uint32_t*)backing; break;
case 8: *trap->out_value = *(uint64_t*)backing; break;
}

return OK;
}</pre>

=== Aperture Configuration (Event 0x08) ===

<pre>int handle_aperture_config(bar_config_t *cfg) {
// Validate new aperture
if (cfg->size > MAX_APERTURE_SIZE) // 0x40000
return ERROR_INVALID;

if (cfg->offset & 0xFFF) // Must be 4KB aligned
return ERROR_ALIGNMENT;

// Reallocate backing store if size changed
if (cfg->size != bar1_aperture_size) {
void *new_backing = bar_alloc(cfg->size);
if (!new_backing) return ERROR_NOMEM;

// Copy existing data
memcpy(new_backing, bar1_backing,
min(cfg->size, bar1_aperture_size));

bar_free(bar1_backing);
bar1_backing = new_backing;
bar1_aperture_size = cfg->size;
}

bar1_aperture_offset = cfg->offset;
return OK;
}</pre>

=== ForceBarAccess (Event 0x10) ===

<pre>int handle_force_access(force_access_t *fa) {
// ForceBarAccess allows hypervisor to bypass VF permissions
// Used for live migration, checkpoint/restore

force_bar_access = fa->enable;
force_access_vf = fa->vf_id;
force_access_range = fa->range;

// Allocate shadow buffer for forced operations
if (fa->enable) {
force_buffer = bar_alloc(0x1000);
if (!force_buffer) return ERROR_NOMEM;
} else {
bar_free(force_buffer);
force_buffer = NULL;
}

return OK;
}</pre>

=== Bar1Rest - State Restore (Event 0x20) ===

<pre>int handle_bar1_restore(restore_state_t *rs) {
// Restore BAR1 state from checkpoint
// Used during VF live migration

bar1_aperture_offset = rs->aperture_offset;
bar1_aperture_size = rs->aperture_size;
force_bar_access = rs->force_access;

// Restore backing store from serialized data
if (rs->backing_size > 0) {
void *new_backing = bar_alloc(rs->backing_size);
if (!new_backing) return ERROR_NOMEM;

memcpy(new_backing, rs->backing_data, rs->backing_size);
bar_free(bar1_backing);
bar1_backing = new_backing;
}

return OK;
}</pre>

== vGPU-IO Integration ==

=== Kernel BAR Trap Registration ===

<pre>struct gsp_bar_ops {
int (*bar_trap)(struct gsp_device *gsp,
struct bar_trap_msg *msg);
int (*configure_aperture)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t offset, uint64_t size);
int (*force_access)(struct gsp_device *gsp,
uint32_t vf_id, bool enable);
int (*save_state)(struct gsp_device *gsp,
uint32_t vf_id, void *buf, size_t *size);
int (*restore_state)(struct gsp_device *gsp,
uint32_t vf_id, void *buf, size_t size);
};

static const struct gsp_bar_ops gsp_bar_ops = {
.bar_trap = gsp_bar_trap_handler,
.configure_aperture = gsp_bar_config_aperture,
.force_access = gsp_bar_force_access,
.save_state = gsp_bar_save_state,
.restore_state = gsp_bar_restore_state,
};</pre>

=== VF BAR Configuration ===

<pre>int gsp_vf_bar_init(struct gsp_vf *vf) {
struct bar_config cfg = {
.vf_id = vf->id,
.bar = 1, // BAR1
.aperture_offset = vf->bar1_offset,
.aperture_size = vf->bar1_size,
.flags = BAR_FLAG_COHERENT,
};

return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_CONFIG, &cfg, sizeof(cfg));
}</pre>

=== Live Migration Support ===

<pre>// Save BAR state for migration
int gsp_vf_bar_save(struct gsp_vf *vf, void *buf, size_t *size) {
struct bar_save_msg msg = {
.vf_id = vf->id,
.bar = 1,
};

return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_SAVE, &msg, sizeof(msg),
buf, size);
}

// Restore BAR state on destination
int gsp_vf_bar_restore(struct gsp_vf *vf, void *buf, size_t size) {
struct bar_restore_msg msg = {
.vf_id = vf->id,
.bar = 1,
.data = buf,
.data_size = size,
};

return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_RESTORE, &msg, sizeof(msg));
}</pre>

== Error Handling ==

{| class="wikitable"
|-
! Error
! Code
! Recovery
|-
| OOM (0x51)
| Allocator exhausted
| Shrink aperture, retry
|-
| Timeout (0x1E)
| Cache flush timeout
| Reset BAR, reinitialize
|-
| Invalid (0x1F)
| Bad aperture config
| Validate alignment/size
|-
| Permission (0xFFFF)
| ForceAccess violation
| Check VF permissions
|-
| Overflow (0x25)
| Offset beyond aperture
| Bounds check in VF driver
|}

== Debugging ==

=== Key Registers to Monitor ===

<pre>0x04296048 : Global error flag (cRam0000000004296048)
0x0429B750 : Error status (0x111300 offset)
0x429B428 : Cache management base
0x429B450 : Allocator arena base</pre>

=== Log Strings ===

Search for these in GSP logs:

* <code>BAR1</code> - BAR1 access
* <code>Bar1Rest</code> - State restore
* <code>ForceBarAccess</code> - Forced access
* <code>Bar1Aperture</code> - Aperture changes
* <code>Bar1Size</code> - Size queries

----

== Reference ==

* Handler: <code>FUN_ram_04cb8188_ram_04cb8188.c</code> (40 KB)
* RPC Type: 0x0A (BAR_TRAP)
* Jump Table Entry: 0x057675B7

Address Reference

2026-10-12T04:40:04Z

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

{{About message}}

= Address Reference =

<blockquote>'''Retained firmware-side address record.''' The tables below list addresses inside the GSP 570.144W firmware image (the firmware-side record), not host implementation details. Current architecture: [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>] and [https://git.vgpu.io/open-iov.git/tree/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>]. Decompiled files live under <code>firmware/57X-GSP/570.144/decompiled/</code>.</blockquote>

== Firmware Base ==

{| class="wikitable"
|-
! Symbol
! Address
! Description
|-
| <code>FW_BASE</code>
| 0x04000000
| Firmware load address
|-
| <code>FW_SIZE</code>
| 0x02000000
| ~32 MB
|}

== Dispatchers ==

{| class="wikitable"
|-
! Name
! Address
! Size
! Description
|-
| Entry Point
| 0x04009000
| 573B
| Boot entry
|-
| Primary Dispatcher
| 0x057A1AC0
| 558B
| Main RPC dispatcher
|-
| Secondary Dispatcher
| 0x04ED39DC
| 141KB
| Branch-based dispatcher
|-
| Tertiary Dispatcher
| 0x05723EF4
| 13KB
| Additional dispatcher
|-
| RPC Jump Table
| 0x059BE7F4
| 4.9KB
| 12 RPC type table
|-
| RPC Type Table
| 0x059BFEB0
| —
| Actual handler addresses
|}

== Primary Dispatcher Sub-Commands (Jump Table at 0x057A3AD8) ==

{| class="wikitable"
|-
! Subcmd
! Target
! Decompiled
|-
| 0
| 0x057A1DB4
| ✅
|-
| 1
| 0x057A1C18
| ✅
|-
| 2
| 0x057A39FC
| ✅
|-
| 3
| 0x057A1BCC
| ✅
|-
| 4
| 0x057A1EF4
| ✅
|-
| 5
| 0x057A1E94
| ✅
|-
| 6
| 0x057A3A28
| ✅
|-
| 7
| 0x057A3A60
| ✅
|-
| 8
| 0x057A1C6C
| ✅
|-
| 9
| 0x057A3A48
| ✅
|-
| 10
| 0x057A1F6C
| ✅
|-
| 11
| 0x057A3AC4
| ✅
|-
| 12
| 0x057A1E10
| ✅
|-
| 13
| 0x057A3A90
| ✅
|-
| 14
| 0x057A39BC
| ✅
|-
| 15
| 0x057A3AAC
| ✅
|-
| 16
| 0x057A3A78
| ✅
|-
| 17
| 0x057A1CCC
| ✅
|-
| 18
| 0x057A39E0
| ✅
|-
| 19
| 0x057A1FA8
| ✅
|-
| 20
| 0x057A2094
| ✅
|-
| 21
| 0x057A2074
| ✅
|-
| 22
| 0x057A2044
| ✅
|-
| 23
| 0x057A20D4
| ✅
|-
| 24
| 0x057A20B4
| ✅
|-
| 25
| 0x057A1FC8
| ✅
|-
| 26
| 0x057A2008
| ✅
|-
| 27
| 0x057A1FE8
| ✅
|-
| 28
| 0x057A2024
| ✅
|-
| 29
| 0x057A219C
| ✅
|-
| 30
| 0x057A20FC
| ✅
|-
| 31
| 0x057A214C
| ✅
|}

== Migration Jump Tables ==

=== Table 1: 0x04179ED8 (MIGRATE branch at 0x057A1BDC) ===

{| class="wikitable"
|-
! Index
! Target
! Decompiled
|-
| 0
| 0x057A2344
| ✅
|-
| 1
| 0x057A2320
| ✅
|-
| 2
| 0x057A230C
| ✅
|-
| 3
| 0x057A22F8
| ✅
|-
| 4
| 0x057A2388
| ✅
|-
| 5
| 0x057A23D8
| ✅
|-
| 6
| 0x057A23AC
| ✅
|-
| 7
| 0x057A2478
| ✅
|-
| 8
| 0x057A2428
| ✅
|-
| 9
| 0x057A2454
| ✅
|-
| 10
| 0x057A2404
| ✅
|-
| 11
| 0x057A2364
| ✅
|-
| 12
| 0x057A1F3C
| (dup)
|-
| 13
| 0x057A299C
| —
|-
| 14
| 0x057A1F3C
| (dup)
|-
| 15
| 0x057A1F3C
| (dup)
|}

=== Table 2: 0x04179FBC (MIGRATE branch at 0x057A1D5C) ===

{| class="wikitable"
|-
! Index
! Target
! Decompiled
|-
| 0
| 0x057A257C
| ✅
|-
| 1
| 0x057A25E4
| ✅
|-
| 2
| 0x057A24EC
| ✅
|-
| 3
| 0x057A2608
| ✅
|-
| 4
| 0x057A2534
| ✅
|-
| 5
| 0x057A25C0
| ✅
|-
| 6
| 0x057A24A4
| ✅
|-
| 7
| 0x057A259C
| ✅
|-
| 8
| 0x057A2510
| ✅
|-
| 9
| 0x057A2558
| ✅
|-
| 10
| 0x057A24C8
| ✅
|-
| 11
| 0x057A262C
| ✅
|-
| 12
| 0x057A345C
| ✅
|-
| 13
| 0x057A36D8
| ✅
|-
| 14
| 0x057A35D8
| ✅
|-
| 15
| 0x057A3658
| ✅
|}

== RPC Handlers (from Jump Table at 0x059BFEB0) ==

{| class="wikitable"
|-
! RPC Type
! Name
! Handler A
! Handler B
|-
| 0x01
| ALLOC
| —
| —
|-
| 0x02
| FREE
| —
| —
|-
| 0x03
| MAP
| 0x05766AB7
| 0x05778A37
|-
| 0x04
| UNMAP
| 0x057669B7
| 0x05779937
|-
| 0x05
| PIN
| 0x057664B7
| 0x0577A3B7
|-
| 0x06
| UNPIN
| 0x057682B7
| 0x05779FB7
|-
| 0x07
| SCHEDULE
| 0x05767F37
| 0x0575AEB7
|-
| 0x08
| YIELD
| 0x0575AE37
| 0x05767337
|-
| 0x09
| NOTIFY
| 0x0577C8B7
| 0x05767837
|-
| 0x0A
| BAR_TRAP
| 0x057675B7
| —
|-
| 0x0B
| MIGRATE
| —
| —
|-
| 0x0C
| DEBUG
| —
| —
|}

== Specialized Main Handlers ==

{| class="wikitable"
|-
! Function
! Address
! Size
|-
| <code>rpc_alloc_main</code>
| 0x057E94EC
| 40 KB
|-
| <code>rpc_schedule_main</code>
| 0x051EFBF4
| 1.8 KB
|-
| <code>rpc_bar_trap_handler</code>
| 0x04CB8188
| 40 KB
|-
| <code>rpc_pin_handler</code>
| 0x04ED34B8
| —
|}

== WPR Handlers ==

{| class="wikitable"
|-
! Handler
! Address
! Size
|-
| WPR Handler 1
| 0x04F00B0C
| 62 KB
|-
| WPR Handler 2
| 0x04F00C78
| 62 KB
|}

== Memory & Global State ==

{| class="wikitable"
|-
! Symbol
! Address
! Description
|-
| Global Error Flag
| 0x04296048
| <code>cRam0000000004296048</code>
|-
| Error Status Reg
| 0x0429B750+0x111300
| Error register
|-
| Stack Canary
| 0x041DC350
| Frame guard value
|-
| Allocator Arena
| 0x429B000
| Base
|-
| Allocator Entry
| 0x429B450
| <code>UNK_ram_058A1A9C</code>
|-
| Cache Manager
| 0x429B428
| <code>UNK_ram_05895BF4</code>
|}

== String References ==

=== BAR Strings ===

{| class="wikitable"
|-
! String
! Address
|-
| BAR1
| 0x0406570E, 0x04085DE0
|-
| Bar1Rest
| 0x040656DA
|-
| ForceBarAccess
| 0x040656F2
|-
| Bar1Aperture
| 0x04065722
|-
| Bar1Size
| (referenced)
|}

=== WPR Strings ===

{| class="wikitable"
|-
! String
! Address
|-
| WPR_WRITE_COMPLETE
| 0x5E478F8
|-
| WPR_READ_COMPLETE
| 0x5E478E0
|-
| WPR_WRITE_TIMEOUT
| 0x5E47898
|-
| WPR_READ_TIMEOUT
| 0x5E47880
|-
| NV_WPR_WRITE_COMPLETE
| 0x5E478C8
|-
| NV_WPR_READ_COMPLETE
| 0x5E478B0
|-
| NV_WPR_WRITE_TIMEOUT
| 0x5E47868
|-
| NV_WPR_READ_TIMEOUT
| 0x5E47850
|}

=== VBIOS Signatures ===

{| class="wikitable"
|-
! Signature
! Value
! Meaning
|-
| VBIOS Standard
| 0xAA55
| VBIOS header
|-
| WPR Enable
| 0x57464F4E ("WFON")
| WPR on
|-
| WPR Version
| 0x5746564E ("WFVN")
| WPR version
|}

== Key Function Pointers (Indirect Calls) ==

{| class="wikitable"
|-
! Address
! Function
! Purpose
|-
| 0x05AB7DA0
| Logger
| String logging
|-
| 0x05AC4B7C
| Stack unwind
| Frame recovery
|-
| 0x058A1A9C
| Allocator
| Memory alloc
|-
| 0x05895BF4
| Cache flush
| Cache management
|-
| 0x0588F9C4
| DMA copy
| Memory copy
|-
| 0x05AEC86C
| Memory read
| Read from addr
|-
| 0x05AE1430
| Size query
| Region size
|-
| 0x05AF4D30
| Page fault
| Page fault handler
|-
| 0x05B10C3C
| Page map
| Page mapping
|-
| 0x05AE9A2C
| Crypto/verify
| Integrity check
|-
| 0x05AED674
| WPR check
| WPR validation
|-
| 0x05AED430
| WPR emulate
| WPR emulation
|-
| 0x05CB30CC
| Config read
| Config access
|-
| 0x05CB25E0
| BAR config
| BAR configuration
|-
| 0x05CB23D8
| BAR map
| BAR mapping
|}

== Decompiled Output Files ==

The decompiled analysis output is not distributed; the file names below record what the analysis produced (one decompiled function per firmware address).

{| class="wikitable"
|-
! Category
! Files
|-
| Core
| entry_ram_04009000.c, FUN_ram_057a1ac0_ram_057a1ac0.c
|-
| Sub-commands
| FUN_ram_057a1bcc_ram_057a1bcc.c ... FUN_ram_057a214c_ram_057a214c.c (32)
|-
| Dispatchers
| FUN_ram_04cc58f8_ram_04cc58f8.c, FUN_ram_04ed39dc_ram_04ed39dc.c, FUN_ram_05723ef4_ram_05723ef4.c
|-
| RPC Jump Table
| FUN_ram_059be7f4_ram_059be7f4.c
|-
| BAR
| FUN_ram_04cb8188_ram_04cb8188.c
|-
| WPR
| FUN_ram_04f00b0c_ram_04f00b0c.c, FUN_ram_04f00c78_ram_04f00c78.c
|-
| ALLOC
| FUN_ram_057e94ec_ram_057e94ec.c, FUN_ram_04d00a18_ram_04d00a18.c, FUN_ram_054a7de0_ram_054a7de0.c
|-
| SCHEDULE
| FUN_ram_051efbf4_ram_051efbf4.c
|-
| RPC Handlers
| FUN_ram_057664b7_ram_057664b7.c ... FUN_ram_0577c8b7_ram_0577c8b7.c (15)
|-
| Migration
| FUN_ram_057a22f8_ram_057a22f8.c ... FUN_ram_057a3658_ram_057a3658.c (28)
|}

== Migration Sub-Command Descriptors (Sample) ==

{| class="wikitable"
|-
! Handler
! Returned ID
! Phase
|-
| 0x057A22F8
| 0x10
| Prepare
|-
| 0x057A2344
| 0x14
| Prepare
|-
| Others
| TBD
| Transfer/Commit
|}

Main Page

2026-10-12T04:34:37Z

WikiBot: Remove LibVF.IO reference from the landing page

{{About message}}

__NOTOC__
<div style="border:1px solid #c8ccd1; background:#f8f9fa; padding:0.9em 1.2em; margin-bottom:1em;">
'''Open-IOV''' is an open-source virtualization layer for NVIDIA GPUs: it runs '''stock, unmodified NVIDIA drivers inside virtual machines''' — Linux and Windows — with CUDA on hardware you own, '''without proprietary vGPU licenses''', without patching the guest driver, the host driver, the hypervisor or the firmware.

'''''Emulate the interface. Pass through the work. Never patch anyone.'''''

The only new component is one host kernel module (<code>vgsp.ko</code>), which implements the interface the guest driver expects to talk to — the GPU System Processor and its resource-manager firmware — and arranges for the guest's work to run on real GPU hardware.

'''[[Open-IOV:About|About Open-IOV]]''' · '''[[Community|Community]]''' · '''[[Documentation|Documentation]]'''
</div>

== Start here ==
* '''[[Project Status]]''' — the honest, dated state of the project.
* '''[[Install and First Run]]''' — fresh host to a stock guest driver in about half an hour.
* '''[[System Overview]]''' — how Open-IOV works, in one page.
* '''[[Documentation]]''' — the full index of this wiki.
* '''[[FAQ]]''' — the questions we get asked most.

== Project status (2026-10-12) ==
{| class="wikitable"
! Area !! State
|-
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev) || Working
|-
| GSP boot emulation and GSP-RM message queues / RPC || Working
|-
| Stock guest driver probe, nvidia-smi (TU104) || Working
|-
| CUDA userspace init and the driver-API smoke (ALL OK, 64-byte payload round-trips) || Working
|-
| Guest CUDA kernel execution on real hardware || In progress — a forwarded launch reaches the engine queue; engine execution is the frontier
|-
| OpenGL / Vulkan, Windows guests, multi-tenancy, migration, performance || Not yet
|}

The full story, with evidence and the current wall: [[Project Status]] and [[The Execution Frontier]].

== Deep dives ==
* [[System Overview]] — the architecture before any subsystem
* [[GSP and GSP-RM]] — the firmware interface we emulate
* [[The vGSP Execution Plane (vEP)]] — how guest work becomes execution and completion
* [[The Host Module (vgsp.ko)]] — the module internals
* [[The Guest Stack and CUDASW]] — what runs unmodified in the guest
* [[CUDA Userspace Driver]] — the <code>libcuda.so</code> analysis
* [[Introduction to GMMU]] — GPU address translation from first principles
* [[UVM in the Guest]] — unified memory as a correctness oracle
* [[Physical GPU Backend and Guest Wiring]] — how work reaches real hardware
* [[The Execution Frontier]] — the leading edge, evidence-led
* [[Guest Driver Lifecycle: A Field Guide]] — boot to CUDA with real IDs and log signatures

== Firmware interoperability ==
* [[GSP 570.144W Firmware Analysis]]
* [[Address Reference]]
* [[RPC Message Format]] · [[BAR Emulation Specification]] · [[WPR Management]] · [[Migration Protocol]] · [[GSP RPC Emulation]] (historical, superseded)
* [[risc-v-gsp-rm-debugger]] — the firmware runtime workbench

== Project and ecosystem ==
* [[Hardware and Software Matrix]] — supported combinations, honestly labelled
* [[HAAEEP and Collaboration]] — the agent-edge use case and how to work with us
* [[FAQ]] · [[Glossary]]

== Community ==
* Source and issues: [https://git.vgpu.io/open-iov git.vgpu.io/open-iov]
* Chat: [https://discord.gg/Rb9K9DYxKK Open-IOV Discord]

== About Open-IOV ==
Open-IOV runs unmodified NVIDIA driver stacks inside virtual machines with GPU compute on hardware you own — no proprietary vGPU license, and no patches to the guest driver, the host driver, the hypervisor or the firmware. The project adds one host kernel module (<code>vgsp.ko</code>) that implements the interface the guest driver expects to talk to — the GPU System Processor and its resource-manager firmware — and arranges for the guest's work to run on real GPU hardware.

* '''Mission:''' make NVIDIA GPUs fully virtualizable, by anyone, for any hypervisor, without modifying NVIDIA software, firmware or hardware.
* '''Principles:''' no patches anywhere; honest status carried with dated evidence; an independent, self-contained IP block; durability by owning the interface rather than a snapshot of it.
* '''Licensing:''' all documentation on this site is [https://creativecommons.org/licenses/by/4.0/ CC-BY-4.0]; the project's code is BSD-3-Clause (GPL-2.0 for the kernel modules); NVIDIA software and firmware are used unmodified and are not redistributed.

'''[[Open-IOV:About|Read more about Open-IOV →]]'''

== About this wiki ==
This wiki previously hosted documentation from the earlier project era (2021–2023). Those pages are retained for reference under [[Documentation|History]]; current documentation starts at [[Documentation]].

VGPU-IO Integration Guide

2026-10-12T04:29:54Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= vGPU-IO Integration Guide =

<blockquote>'''Historical document — superseded.''' This earlier integration design predates the current stack and is retained for reference only. Canonical documentation: getting started [https://git.vgpu.io/open-iov/blob/master/docs/02-getting-started <code>../docs/02-getting-started/</code>], host module [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>], developer standards [https://git.vgpu.io/open-iov/blob/master/docs/07-developers <code>../docs/07-developers/</code>].</blockquote>

== Overview ==

This guide covers implementing a Linux kernel driver for vGPU-IO using the GSP 570.144W firmware RPC interface.

== Architecture ==

<pre>┌─────────────────────────────────────────────────────────────┐
│ vGPU-IO Kernel Driver │
├─────────────────────────────────────────────────────────────┤
│ gsp_core.c │ gsp_rpc.c │ gsp_vf.c │
│ - Device probe │ - RPC client │ - VF lifecycle │
│ - BAR mapping │ - Doorbell │ - Migration │
│ - IRQ/MSI-X │ - Completion │ - WPR/BAR integration │
├─────────────────────────────────────────────────────────────┤
│ GSP Firmware (570.144W) │
│ Primary Dispatcher (0x057A1AC0) │ Secondary (0x04ED39DC) │
│ BAR Handler (0x04CB8188) │ WPR Handlers │
│ Migration (0x04179ED8/0x04179FBC)│ SCHED/ALLOC/PIN │
└─────────────────────────────────────────────────────────────┘</pre>

== Module Structure ==

<pre>drivers/gpu/nvidia/gsp/
├── gsp.h # Core definitions
├── gsp_rpc.h # RPC message formats
├── gsp_vf.h # VF definitions
├── gsp_bar.h # BAR emulation
├── gsp_wpr.h # WPR management
├── gsp_migration.h # Migration protocol
├── gsp_core.c # Probe/remove, BAR/IRQ
├── gsp_rpc.c # RPC client implementation
├── gsp_vf.c # VF create/destroy
├── gsp_bar.c # BAR trap handler
├── gsp_wpr.c # WPR protect/unprotect
├── gsp_migration.c # Live migration
└── gsp_debugfs.c # Debug interface</pre>

== Core Data Structures ==

=== GSP Device ===

<pre>struct gsp_device {
struct pci_dev *pdev;
void __iomem *bar0; // PCIe BAR0 mapping
void __iomem *bar1; // PCIe BAR1 (GSP FW)
struct gsp_firmware *fw; // Firmware image
struct gsp_rpc_client *rpc; // RPC client
struct mutex vf_lock;
struct idr vf_idr; // VF ID allocator
struct list_head vf_list;
struct gsp_debugfs *debugfs;
spinlock_t doorbell_lock;
uint32_t doorbell_seq;
};</pre>

=== VF (Virtual Function) ===

<pre>struct gsp_vf {
struct gsp_device *gsp;
uint32_t id; // VF ID (1-7)
struct kref ref;
struct mutex lock;
mig_state_t mig_state;
struct gsp_migration *migration;
uint64_t bar1_offset; // BAR1 aperture offset
uint64_t bar1_size; // BAR1 aperture size
uint64_t pt_base; // Page table base
uint64_t pt_size; // Page table size
uint64_t cmd_base; // Command buffer base
uint64_t cmd_size; // Command buffer size
struct gsp_wpr_region *wpr_regions;
struct list_head wpr_list;
struct gsp_bar_state bar_state;
atomic_t refcount;
bool active;
};</pre>

== Initialization Sequence ==

=== 1. PCI Probe ===

<pre>static int gsp_probe(struct pci_dev *pdev, const struct pci_device_id *id) {
struct gsp_device *gsp;
int ret;

gsp = devm_kzalloc(&pdev->dev, sizeof(*gsp), GFP_KERNEL);
gsp->pdev = pdev;
pci_set_drvdata(pdev, gsp);

// Enable PCI device
ret = pcim_enable_device(pdev);
if (ret) return ret;

// Map BARs
gsp->bar0 = pcim_iomap(pdev, 0, 0);
gsp->bar1 = pcim_iomap(pdev, 1, 0);
if (!gsp->bar0 || !gsp->bar1) return -ENOMEM;

// Load firmware
ret = gsp_firmware_load(gsp, "gsp-570.144.fwimage.bin");
if (ret) return ret;

// Initialize RPC client
gsp->rpc = gsp_rpc_client_init(gsp);
if (IS_ERR(gsp->rpc)) return PTR_ERR(gsp->rpc);

// Setup IRQ/MSI-X
ret = gsp_setup_irq(gsp);
if (ret) return ret;

// Initialize debugfs
gsp_debugfs_init(gsp);

return 0;
}</pre>

=== 2. Firmware Loading ===

<pre>int gsp_firmware_load(struct gsp_device *gsp, const char *name) {
const struct firmware *fw;
int ret;

ret = request_firmware(&fw, name, &gsp->pdev->dev);
if (ret) return ret;

// Verify MD5: 13ce12a392a31eeb79a2e6ebd3a4a93a
if (!gsp_firmware_verify(fw->data, fw->size)) {
release_firmware(fw);
return -EINVAL;
}

// Load to GSP memory (via BAR1)
ret = gsp_firmware_upload(gsp, fw->data, fw->size);
release_firmware(fw);

return ret;
}</pre>

=== 3. GSP Boot ===

<pre>int gsp_boot(struct gsp_device *gsp) {
// 1. Reset GSP
writel(1, gsp->bar0 + GSP_RESET_REG);
msleep(10);
writel(0, gsp->bar0 + GSP_RESET_REG);

// 2. Wait for entry point (0x04009000) to signal ready
ret = gsp_wait_for_ready(gsp, 5000); // 5s timeout
if (ret) return ret;

// 3. Initialize RPC dispatchers
ret = gsp_rpc_init_dispatchers(gsp);
if (ret) return ret;

return 0;
}</pre>

== RPC Client Implementation ==

=== Doorbell Mechanism ===

<pre>#define GSP_DOORBELL_OFFSET 0x1000
#define GSP_MSG_RING_SIZE 256
#define GSP_MSG_MAX_SIZE 256

struct gsp_rpc_client {
struct gsp_device *gsp;
struct mutex lock;
uint32_t seq;
wait_queue_head_t wq;
struct gsp_msg_ring *ring;
struct completion *pending[GSP_MAX_PENDING];
};

static int gsp_rpc_send_sync(struct gsp_rpc_client *rpc,
uint32_t rpc_type,
void *payload, size_t payload_size,
void *response, size_t *response_size) {
struct gsp_msg msg = {
.hdr = {
.magic = 0x47535052,
.version = 1,
.rpc_type = rpc_type,
.sequence = atomic_inc_return(&rpc->seq),
.timestamp_ns = ktime_get_ns(),
.payload_size = payload_size,
.flags = MSG_SYNC,
}
};

struct completion done;
init_completion(&done);

mutex_lock(&rpc->lock);
rpc->pending[msg.hdr.sequence % GSP_MAX_PENDING] = &done;

// Copy message to ring buffer
memcpy(msg.hdr.payload, payload, payload_size);
gsp_ring_write(rpc->ring, &msg, GSP_MSG_MAX_SIZE);

// Ring doorbell
writel(1, rpc->gsp->bar0 + GSP_DOORBELL_OFFSET);

mutex_unlock(&rpc->lock);

// Wait for completion
if (!wait_for_completion_timeout(&done, msecs_to_jiffies(5000)))
return -ETIMEDOUT;

// Copy response
if (response && response_size) {
*response_size = min(*response_size, done.payload_size);
memcpy(response, done.payload, *response_size);
}

return done.status;
}</pre>

=== Completion Handler (ISR) ===

<pre>static irqreturn_t gsp_isr(int irq, void *data) {
struct gsp_device *gsp = data;
struct gsp_msg msg;

while (gsp_ring_read(gsp->rpc->ring, &msg)) {
if (msg.hdr.flags & MSG_SYNC) {
struct completion *done =
gsp->rpc->pending[msg.hdr.sequence % GSP_MAX_PENDING];
if (done) {
done->status = msg.payload.status;
done->payload_size = msg.hdr.payload_size;
memcpy(done->payload, msg.payload.data, msg.hdr.payload_size);
complete(done);
}
} else {
// Async notification
gsp_handle_notification(gsp, &msg);
}
}

return IRQ_HANDLED;
}</pre>

== VF Lifecycle ==

=== Create VF ===

<pre>int gsp_vf_create(struct gsp_device *gsp, uint32_t vf_id,
const struct gsp_vf_config *config) {
struct gsp_vf *vf;
int ret;

mutex_lock(&gsp->vf_lock);

if (idr_alloc(&gsp->vf_idr, NULL, vf_id, vf_id + 1, GFP_KERNEL) < 0) {
ret = -EBUSY;
goto unlock;
}

vf = kzalloc(sizeof(*vf), GFP_KERNEL);
vf->gsp = gsp;
vf->id = vf_id;
kref_init(&vf->ref);
mutex_init(&vf->lock);
INIT_LIST_HEAD(&vf->wpr_list);

// Configure VF resources
vf->bar1_offset = config->bar1_offset;
vf->bar1_size = config->bar1_size;
vf->pt_base = config->pt_base;
vf->pt_size = config->pt_size;
vf->cmd_base = config->cmd_base;
vf->cmd_size = config->cmd_size;

// Initialize WPR for VF
ret = gsp_vf_wpr_init(vf);
if (ret) goto err_wpr;

// Initialize BAR for VF
ret = gsp_vf_bar_init(vf);
if (ret) goto err_bar;

list_add(&vf->list, &gsp->vf_list);
vf->active = true;

mutex_unlock(&gsp->vf_lock);
return 0;

err_bar:
gsp_vf_wpr_fini(vf);
err_wpr:
idr_remove(&gsp->vf_idr, vf_id);
kfree(vf);
unlock:
mutex_unlock(&gsp->vf_lock);
return ret;
}</pre>

=== Destroy VF ===

<pre>void gsp_vf_destroy(struct gsp_vf *vf) {
struct gsp_device *gsp = vf->gsp;
uint32_t id = vf->id;

mutex_lock(&gsp->vf_lock);
vf->active = false;
list_del(&vf->list);
mutex_unlock(&gsp->vf_lock);

gsp_vf_bar_fini(vf);
gsp_vf_wpr_fini(vf);

if (vf->migration)
gsp_migration_cancel(vf->migration);

kref_put(&vf->ref, gsp_vf_release);
}

static void gsp_vf_release(struct kref *ref) {
struct gsp_vf *vf = container_of(ref, struct gsp_vf, ref);
kfree(vf);
}</pre>

== Memory Management (ALLOC/PIN) ==

=== Allocate VF Memory ===

<pre>int gsp_vf_mem_alloc(struct gsp_vf *vf,
uint64_t size, uint32_t flags,
uint64_t *handle, uint64_t *va) {
struct alloc_payload payload = {
.size = size,
.type = MEM_TYPE_VRAM,
.alignment = 0x1000,
.va_hint = 0,
.flags = flags,
.pid = current->pid,
.vf_id = vf->id,
};

struct alloc_response resp;
size_t resp_size = sizeof(resp);

return gsp_rpc_send_sync(vf->gsp->rpc, RPC_ALLOC,
&payload, sizeof(payload),
&resp, &resp_size);
}</pre>

=== Pin Memory for DMA ===

<pre>int gsp_vf_mem_pin(struct gsp_vf *vf, uint64_t handle,
uint64_t offset, uint64_t size,
uint64_t *dma_addr) {
struct pin_payload payload = {
.handle = handle,
.offset = offset,
.size = size,
.vf_id = vf->id,
.flags = PIN_WRITE | PIN_READ,
};

struct pin_response resp;
size_t resp_size = sizeof(resp);

return gsp_rpc_send_sync(vf->gsp->rpc, RPC_PIN,
&payload, sizeof(payload),
&resp, &resp_size);
}</pre>

== Debugging & Testing ==

=== Debugfs Interface ===

<pre># List VFs
cat /sys/kernel/debug/gsp/vfs

# VF details
cat /sys/kernel/debug/gsp/vf/1/status

# RPC statistics
cat /sys/kernel/debug/gsp/rpc/stats

# BAR state
cat /sys/kernel/debug/gsp/vf/1/bar

# WPR regions
cat /sys/kernel/debug/gsp/vf/1/wpr

# Migration status
cat /sys/kernel/debug/gsp/vf/1/migration</pre>

=== Trace Events ===

<pre># Enable tracing
echo 1 > /sys/kernel/debug/tracing/events/gsp/enable

# Trace migration
trace-cmd record -e gsp_mig_phase -e gsp_mig_transfer

# Trace WPR
trace-cmd record -e gsp_wpr_event</pre>

=== Unit Tests ===

<pre>// kselftest for gsp-rpc
static void test_rpc_alloc(void) {
struct gsp_vf *vf = create_test_vf();
uint64_t handle, va;

ASSERT_EQ(gsp_vf_mem_alloc(vf, 4096, 0, &handle, &va), 0);
ASSERT_NE(handle, 0);
ASSERT_NE(va, 0);

gsp_vf_mem_free(vf, handle);
destroy_test_vf(vf);
}</pre>

== Error Handling Checklist ==

{| class="wikitable"
|-
! Scenario
! Detection
! Recovery
|-
| GSP firmware crash
| Watchdog timeout
| Reset GSP, reload FW
|-
| RPC timeout
| Doorbell no response
| Retry, then VF reset
|-
| BAR trap storm
| Rate limit exceeded
| Throttle VF
|-
| WPR violation
| Callback storm
| Quarantine VF
|-
| Migration fail
| Commit verify fail
| Rollback to source
|}

== Performance Tuning ==

=== RPC Batching ===

<pre>// Batch multiple ALLOCs in single RPC
struct batch_alloc_payload {
uint32_t count;
struct alloc_payload allocs[MAX_BATCH];
};</pre>

=== Async Completion ===

<pre>// Use async RPC for non-critical paths
gsp_rpc_send_async(rpc, RPC_NOTIFY, payload, size, callback, ctx);</pre>

=== Memory Pool ===

<pre>// Pre-allocate bounce buffers
struct gsp_bounce_pool {
struct page **pages;
int count;
spinlock_t lock;
};</pre>

----

== Reference ==

* GSP Firmware: gsp-570.144.fwimage.bin
* Base Address: 0x04000000
* Primary Dispatcher: 0x057A1AC0
* Secondary Dispatcher: 0x04ED39DC

WPR Management

2026-10-12T04:29:53Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= WPR (Write/Read Protect Region) Management =

<blockquote>'''Historical document — superseded.''' WPR is the '''write-protected region of VRAM reserved for firmware''', not a generic read/write-protect mechanism; the read-protect framing below is historical reverse-engineering material. Canonical documentation: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

WPR (Write Protect Region / Read Protect Region) provides memory isolation for vGPU by protecting guest memory regions from unauthorized access. The GSP firmware implements WPR through two handlers at '''0x04F00B0C''' and '''0x04F00C78''' (62 KB each).

== Key Addresses ==

{| class="wikitable"
|-
! Handler
! Address
! Size
! Role
|-
| WPR Handler 1
| 0x04F00B0C
| 62 KB
| Primary WPR management
|-
| WPR Handler 2
| 0x04F00C78
| 62 KB
| Secondary/timeout handling
|-
| String Table
| 60 locations
| —
| Completion/timeout strings
|}

== String References (60 Total) ==

=== Completion Events ===

<pre>WPR_WRITE_COMPLETE 0x5E478F8
WPR_READ_COMPLETE 0x5E478E0
NV_WPR_WRITE_COMPLETE 0x5E478C8
NV_WPR_READ_COMPLETE 0x5E478B0</pre>

=== Timeout Events ===

<pre>WPR_WRITE_TIMEOUT 0x5E47898
WPR_READ_TIMEOUT 0x5E47880
NV_WPR_WRITE_TIMEOUT 0x5E47868
NV_WPR_READ_TIMEOUT 0x5E47850</pre>

== WPR State Machine ==

=== Handler 1: 0x04F00B0C (Primary) ===

<pre>typedef enum {
WPR_STATE_IDLE = 0,
WPR_STATE_WRITE_PENDING = 1,
WPR_STATE_READ_PENDING = 2,
WPR_STATE_COMPLETE = 3,
WPR_STATE_TIMEOUT = 4,
WPR_STATE_ERROR = 0xFFFF
} wpr_state_t;

typedef struct {
uint64_t region_base; // Protected region base
uint64_t region_size; // Region size (page-aligned)
uint32_t vf_id; // Owning VF
uint32_t flags; // WPR_WRITE | WPR_READ | WPR_EXECUTE
wpr_state_t state;
uint64_t timeout_ns; // Operation timeout
void *completion_cb; // Callback on complete
void *timeout_cb; // Callback on timeout
} wpr_region_t;</pre>

=== Operation Flow ===

==== Write Protect ====

<pre>1. VF requests WPR_WRITE on region [base, base+size)
2. Handler validates alignment (4KB) and permissions
3. Programs MMU/IOMMU write-protect bits
4. Starts timeout timer (configurable)
5. On GPU write attempt:
a. Trap to GSP via page fault
b. Check if VF owns region
c. If allowed: emulate write, return OK
d. If denied: trigger WPR_WRITE_COMPLETE/NV_WPR_WRITE_COMPLETE
6. On timeout: trigger WPR_WRITE_TIMEOUT/NV_WPR_WRITE_TIMEOUT</pre>

==== Read Protect ====

<pre>1. VF requests WPR_READ on region
2. Programs MMU/IOMMU read-protect bits
3. On GPU read attempt:
a. Trap to GSP
b. Return zero/poison data or emulate
c. Trigger WPR_READ_COMPLETE/NV_WPR_READ_COMPLETE
4. On timeout: trigger WPR_READ_TIMEOUT/NV_WPR_READ_TIMEOUT</pre>

== VBIOS Parsing (Handler 2: 0x04F00C78) ==

The WPR handlers parse VBIOS structures for legacy compatibility:

<pre>int parse_vbios_for_wpr(void *vbios_ptr, size_t size) {
// Search for signatures:
// 0xAA55 - Standard VBIOS signature
// 0x57464F4E ("WFON") - WPR enable
// 0x5746564E ("WFVN") - WPR version

for (offset = 0; offset < size; offset += 0x1000) {
uint32_t sig = *(uint32_t*)(vbios_ptr + offset);

if (sig == 0xAA55) {
// Check for WPR structures
parse_wpr_tables(vbios_ptr + offset);
}
else if (sig == 0x57464F4E || sig == 0x5746564E) {
// WPR enabled in VBIOS
wpr_vbios_enabled = true;
wpr_vbios_offset = offset;
}
}
}</pre>

== Integration with GSP Subsystems ==

=== Memory Allocator (0x429B450) ===

<pre>// Allocate WPR metadata structures
wpr_region_t *wpr_alloc_region() {
return UNK_ram_058A1A9C(0x429B450, sizeof(wpr_region_t));
}

// Allocate bounce buffers for emulation
void *wpr_alloc_bounce(size_t size) {
return UNK_ram_058A1A9C(0x429B450, size);
}</pre>

=== DMA/Copy Engine (0x0588F9C4, 0x05AEC86C) ===

<pre>// Copy data for WPR emulation
int wpr_emulate_access(wpr_region_t *rgn, void *dst, void *src, size_t size) {
return UNK_ram_0588F9C4(dma_engine, src, dst, size);
}</pre>

=== Logging (0x05AB7DA0) ===

<pre>// Log WPR events
void wpr_log_event(const char *event_str, uint32_t vf_id, uint64_t addr) {
UNK_ram_05AB7DA0(1, &event_str); // Log string
UNK_ram_05AB7DA0(2, &vf_id); // Log VF ID
UNK_ram_05AB7DA0(3, &addr); // Log address
}</pre>

== vGPU-IO Integration ==

=== Kernel WPR API ===

<pre>struct gsp_wpr_ops {
int (*protect_write)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base, uint64_t size);
int (*protect_read)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base, uint64_t size);
int (*unprotect)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t base);
int (*set_timeout)(struct gsp_device *gsp,
uint32_t vf_id,
uint64_t timeout_ns);
int (*register_callback)(struct gsp_device *gsp,
uint32_t vf_id,
enum wpr_event_type type,
void (*cb)(struct wpr_event *));
};

static const struct gsp_wpr_ops gsp_wpr_ops = {
.protect_write = gsp_wpr_write_protect,
.protect_read = gsp_wpr_read_protect,
.unprotect = gsp_wpr_unprotect,
.set_timeout = gsp_wpr_set_timeout,
.register_callback = gsp_wpr_register_callback,
};</pre>

=== VF WPR Lifecycle ===

<pre>// VF initialization: register WPR regions
int gsp_vf_wpr_init(struct gsp_vf *vf) {
// Protect VF's page tables
gsp_wpr_ops.protect_write(vf->gsp, vf->id,
vf->pt_base, vf->pt_size);

// Protect VF's command buffers
gsp_wpr_ops.protect_write(vf->gsp, vf->id,
vf->cmd_base, vf->cmd_size);

// Set default timeout (10ms)
gsp_wpr_ops.set_timeout(vf->gsp, vf->id, 10000000);

// Register completion callbacks
gsp_wpr_ops.register_callback(vf->gsp, vf->id,
WPR_EVENT_WRITE_COMPLETE,
vf_wpr_write_complete_cb);
gsp_wpr_ops.register_callback(vf->gsp, vf->id,
WPR_EVENT_TIMEOUT,
vf_wpr_timeout_cb);

return 0;
}

// VF cleanup
void gsp_vf_wpr_fini(struct gsp_vf *vf) {
gsp_wpr_ops.unprotect(vf->gsp, vf->id, vf->pt_base);
gsp_wpr_ops.unprotect(vf->gsp, vf->id, vf->cmd_base);
// ... unregister callbacks
}</pre>

=== Callback Handling ===

<pre>static void vf_wpr_write_complete_cb(struct wpr_event *evt) {
struct gsp_vf *vf = evt->vf;

// Log the write attempt
dev_dbg(vf->dev, "WPR write complete: VF=%u addr=0x%llx\n",
vf->id, evt->address);

// If this was a page table update, invalidate TLB
if (evt->address >= vf->pt_base &&
evt->address < vf->pt_base + vf->pt_size) {
gsp_tlb_invalidate(vf->gsp, vf->id);
}
}

static void vf_wpr_timeout_cb(struct wpr_event *evt) {
struct gsp_vf *vf = evt->vf;

dev_warn(vf->dev, "WPR timeout: VF=%u type=%d addr=0x%llx\n",
vf->id, evt->type, evt->address);

// Escalate: reset VF or throttle
if (evt->type == WPR_EVENT_WRITE_TIMEOUT) {
gsp_vf_throttle(vf);
} else {
gsp_vf_reset(vf);
}
}</pre>

== Error Codes ==

{| class="wikitable"
|-
! Code
! Name
! Description
|-
| 0x1E
| ETIME
| Operation timed out
|-
| 0x1F
| EDEV
| Device error
|-
| 0x25
| EOVERFLOW
| Region too large
|-
| 0x37
| EINVAL
| Invalid alignment
|-
| 0x51
| ENOMEM
| Out of memory
|-
| 0x56
| EBUSY
| Region busy
|-
| 0xFFFF
| EPERM
| Permission denied
|}

== Debugging ==

=== Monitor These Addresses ===

<pre>0x04296048 : Global error flag
0x0429B750+0x111300 : Error status register
0x05E47850-0x05E478F8 : WPR string table</pre>

=== GSP Log Patterns ===

<pre>WPR_WRITE_COMPLETE VF=<id> ADDR=<addr>
WPR_READ_COMPLETE VF=<id> ADDR=<addr>
WPR_WRITE_TIMEOUT VF=<id> ADDR=<addr>
WPR_READ_TIMEOUT VF=<id> ADDR=<addr>
NV_WPR_WRITE_COMPLETE VF=<id> ADDR=<addr>
NV_WPR_READ_COMPLETE VF=<id> ADDR=<addr>
NV_WPR_WRITE_TIMEOUT VF=<id> ADDR=<addr>
NV_WPR_READ_TIMEOUT VF=<id> ADDR=<addr></pre>

=== Trace Points ===

<pre>// In kernel driver
trace_gsp_wpr_protect(vf_id, base, size, type);
trace_gsp_wpr_event(vf_id, event_type, address, status);
trace_gsp_wpr_timeout(vf_id, address, elapsed_ns);</pre>

----

== Reference ==

* Handler 1: <code>FUN_ram_04f00b0c_ram_04f00b0c.c</code> (62 KB)
* Handler 2: <code>FUN_ram_04f00c78_ram_04f00c78.c</code> (62 KB)
* RPC Types: Handled via primary/secondary dispatchers

UVM in the Guest

2026-10-12T04:29:53Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= UVM in the Guest =

<blockquote>'''Scope.''' What UVM (<code>nvidia_uvm.ko</code>) does inside the guest, why it is the most demanding consumer of the virtual GPU's semantics, and how it waits. Canonical source: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>]; where this page and the docs tree disagree, the docs tree wins.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> partially working, software bridge in charge; <code>[gated]</code> implemented default-off; <code>[target]</code> planned end state.

== 1. What UVM is ==

UVM provides unified memory and GPU-driven page-fault handling. In practice it is a '''second GPU driver layered on the RM''', owning GPU address spaces and running its own memory-management work on the GPU.

{| class="wikitable"
|-
! Component
! Role
|-
| VA spaces (<code>uvm_va_space</code>)
| per-process GPU address spaces
|-
| Channel managers
| pools of GPU channels (CE, SEC2, GR as needed) for UVM's work
|-
| Push buffers / GPFIFO
| UVM builds and submits its own pushes
|-
| Semaphores and trackers
| UVM waits on GPU-written semaphores to track completion
|-
| Page trees
| GPU page tables for its VA spaces
|-
| Fault handling
| fault buffer plus notification; UVM resolves and replays/cancels faults
|}

For CUDA, UVM is entered early: <code>cuInit</code> → driver init → <code>UVM_REGISTER_GPU</code>. If UVM cannot register the GPU, CUDA does not start — which is why "UVM advances" is the project's primary correctness metric on the way to CUDA.

== 2. The registration path (traced end to end) ==

<pre>libcuda: UVM_REGISTER_GPU ioctl (0x25)
uvm_api_register_gpu -> uvm_va_space_register_gpu -> uvm_gpu_retain_by_uuid
nvUvmInterfaceRegisterGpu / GetGpuInfo; alloc_parent_gpu / add_gpu
nvUvmInterfaceDeviceCreate (RM object creation)
uvm_hal_init_gpu / get_gpu_caps (CE caps)
uvm_ats_add_gpu / uvm_gpu_init_isr (fault/interrupt plumbing)
init_channel_manager
create semaphore pool (sysmem)
create channels per CE (GPFIFO + pushbuffer + semaphores)
channel_init() -> first push -> semaphore release -> wait
uvm_page_tree_init (page tables for the VA space)</pre>

Each stage has a distinct failure signature — see [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]].

== 3. What UVM requires from the "GPU" ==

{| class="wikitable"
|-
! Requirement
! Detail
! Status
|-
| GPU identity
| must match the UUID from RM
| <code>[live]</code>
|-
| CE capabilities
| per-engine support bits; UVM requires <code>supported && !grce && sysmem</code> for CPU_TO_GPU / GPU_TO_CPU channels
| <code>[live]</code> caps accepted; <code>[gated]</code> copies forwarded to the physical copy engine
|-
| Channel creation
| RM channel objects with GPFIFO/userd/instance parameters
| <code>[live]</code>
|-
| Pushes executed
| first action on a new channel is an init push plus a semaphore-tracked wait
| <code>[wip]</code> software bridge executes memsets, copies and releases; <code>[gated]</code> CE-class copies forwarded
|-
| Page trees
| UVM initializes page trees with its own pushes; server-owned ranges must be populated by us
| <code>[wip]</code>
|-
| Fault delivery
| fault buffer plus notification; replay/cancel semantics
| <code>[gated]</code> backend-side observation/relay; guest contract <code>[target]</code>
|-
| Completion semantics
| tracker values must advance exactly
| <code>[wip]</code>
|}

'''Current frontier.''' Context creation and the full CUDA driver-API smoke pass (<code>ALL OK</code>, copies matched) run end to end. CE-class copies are forwarded to the physical copy engine (default off) with engine-authored releases and a fault relay. For GR, a forwarded guest launch reaches the front-end but the inline launch-descriptor install has not been observed; context validity at dispatch is ruled out ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]). The remaining UVM work is completion translation and the guest-side fault replay/cancel contract.

== 4. The discovery that defines the work: how UVM waits ==

UVM's tracker is the single most visible symptom of an unfaithful GPU. It spins, and after 30 seconds it dumps its state:

<pre>uvm_tracker.c:285 Tracker entry for value 1 (sema VA 0x121000000)
uvm_channel.c:4184 pending push '' started at uvm_channel.c:2535 in channel_init()
releasing value 1</pre>

Mechanically, UVM built a push ending in a '''semaphore release''' — a method sequence (CE channel class, e.g. <code>TURING_DMA_COPY_A</code>) that sets an address and a payload and launches a DMA operation to write the payload to memory — then submitted it and waits for the payload to appear. On real hardware the copy engine writes it; on our device nobody does unless the emulation does. The push we found in guest RAM decodes as:

<pre>20010000 0000c5b5 set-object (TURING_DMA_COPY_A)
2001a000 0000c076 set-object (GP100_UVM_SW)
20038090 00000001 incrementing-method header: count 3, addr 0x240
21000000 00000001 lower 32 bits of the semaphore VA; payload (1)
200180c0 0000000c LAUNCH_DMA with SEMAPHORE_TYPE=RELEASE_ONE_WORD</pre>

So the emulation's job for this push is: '''write <code>payload</code> to the memory at the semaphore's GPU VA''', which requires a GPU-VA → guest-PA translation. The host does this through the learned VA→PA map: while the channel's GPFIFO has pending entries it reads each push, decodes the method stream and runs the work (page-table memsets, copies, semaphore releases). When a push page cannot be resolved — or a channel consumes entries without any release landing — it falls back to content: guest RAM is scanned for release groups and pending payloads are applied forward-only. A release is applied only when <code>payload > current</code>, so recycled push buffers and stale copies are harmless. One pitfall observed live: a semaphore-page VA can flap between two physical pages across map rebuilds; pinning the validated translation fixes the stalled wait.

== 5. The memory model UVM assumes ==

* UVM's semaphore pool is '''system memory''' (<code>UVM_APERTURE_SYS</code>), one 4 KiB page per channel manager, addressed as 4-byte slots (1024 per page).
* Each page has a '''GPU VA''' (chosen by the client's RM) and a '''CPU VA''' (a kernel mapping). UVM polls the CPU VA; the GPU writes the GPU VA.
* Both addresses are recorded in UVM's structures in guest memory — the key that makes translation possible without extra help from the guest.
* The CPU VA is a direct-map address, so <code>PA = cpu_va - page_offset_base</code> (verified live: writing the payload there completes the push).
* Values are per channel and monotonic (1, 2, 3 …); UVM's own check (<code>UVM_GPU_SEMAPHORE_MAX_JUMP</code>) detects and reports wrong values.
* The client's page tables for '''server-owned ranges are empty chains''': the client hands us the physical addresses of its page-table levels and expects the server to own the lower levels ([[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]]).

== 6. Page trees and faults ==

<code>uvm_page_tree_init</code> initializes a VA space's page tables with pushes on UVM's own channels (memops and page-table writes; the copy engine executes them on real hardware). Fault handling is event-driven: UVM programs a fault buffer (replayable and non-replayable; sizes from <code>INTERNAL_GMMU_GET_STATIC_INFO</code>, answered by the emulation) and, on a fault, reads the entry, populates the page table and replays or cancels the fault with a <code>MEM_OP</code> push. Reconciling the guest's memory model with the real GPU's here is the deepest remaining work.

== 7. How to observe UVM ==

In the guest, UVM prints nothing by default:

<pre>echo 1 | sudo tee /sys/module/nvidia_uvm/parameters/uvm_debug_prints</pre>

Messages then appear with the <code>nvidia-uvm:</code> prefix. Known-good markers, in order: <code>uvm_gpu_retain_by_uuid</code> → channel manager created → <code>uvm_page_tree_init</code> pushes → <code>UVM_REGISTER_GPU</code> returns and CUDA proceeds. Failure signatures characterized live:

{| class="wikitable"
|-
! Message
! Meaning
|-
| <code>Failed to initialize the channel manager: NV_ERR_NOT_SUPPORTED</code>
| CE caps wrong or missing
|-
| <code>stuck waiting</code> plus <code>Tracker entry ... channel_init()</code>
| the push's semaphore was never written
|-
| <code>Assert failed ... unexpected semaphore ... jump from 0x0 to 0x1010101</code>
| a wrong payload was written; UVM caught it
|}

== 8. Why UVM is the perfect correctness oracle ==

UVM exercises in minutes what a driver test suite would test in hours: memory allocation, page tables, channel setup, push generation, semaphore semantics, fault handling and error recovery — all with continuous self-checks and loud diagnostics. When UVM registers cleanly and CUDA workloads pass, the device is behaving like a real GPU.

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/04-uvm.md <code>../docs/03-architecture/04-uvm.md</code>] — canonical chapter (this page's source).
* [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the guest stack and the RM objects UVM consumes.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the userspace half of the contract; [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — how UVM's pushes execute, and the current wall.
* [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the fault contract.

System Overview

2026-10-12T04:29:52Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= System Overview =

<blockquote>'''What this page is.''' The one page to read if you want to understand ''how Open-IOV works'' before diving into any subsystem. Condensed from [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/01-system-overview.md <code>../docs/03-architecture/01-system-overview.md</code>] (with [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/09-full-system-picture.md <code>../docs/03-architecture/09-full-system-picture.md</code>] and [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/10-interaction-and-scheduling.md <code>../docs/03-architecture/10-interaction-and-scheduling.md</code>] for the full picture); the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partial; <code>[wall]</code> current blocker; <code>[target]</code> designed end state.

== 1. The core principle ==

'''Emulate the interface. Pass through the work. Never patch anyone.'''

A virtual GPU is a ''mediated passthrough'' device. The guest must believe it owns a real NVIDIA GPU with a real GPU System Processor running firmware; the host must arrange for the guest's work to execute on real hardware, because emulating compute in software is orders of magnitude too slow to be useful.

<pre>EMULATE (software; the guest expects virtual hardware)
PCI configuration space, capability structures, power management
BAR0 register interface (shadow, not the physical GPU's BAR0)
BAR1 VRAM aperture (guest-visible video memory)
GSP firmware boot handshake (FWSEC -> booter -> RISCV -> boot complete)
GSP-RM message queues, RPC transport, resource-manager controls
event/notification channels; MSI-X injection (via eventfd)

PASS THROUGH (real hardware; correctness and performance)
GPU memory: guest VRAM objects backed by real VRAM [target]
page tables / GMMU state used by the GPU [wip]
channel execution: guest push buffers on real channels [gated/wall]
compute (CUDA) and, later, 3D/video work [wip/planned]
DMA, completion and fault interrupts relayed to the guest</pre>

The line between the two is the '''interface boundary''', and it moves as the project matures: early work emulates everything the guest can observe; later work forwards everything that must be fast.

== 2. Layer map ==

<pre>GUEST
application ── CUDA runtime (libcudart) ─┐
application ── CUDA driver (libcuda) ─┤ userspace: CUDASW stack
OpenGL/Vulkan ── libGLX/libvulkan ──────┘
│ ioctls / mmaps
nvidia.ko (unmodified) ── RM core, resource server, UVM, modeset/drm
│ MMIO + DMA + MSI-X (a virtual PCIe device)
════════╪════════════════════════════════════════════════════════
HYPERVISOR (KVM/QEMU, later Xen/...) VFIO mdev transport + IOMMU
│
HOST MODULE vgsp.ko
├─ virtual device: config space, BAR0/BAR1/BAR3 windows, MSI-X
├─ emulation core: GSP boot + GSP-RM (RPC msgq, controls, objects, events)
├─ memory model: guest RAM access, VA<->PA maps, VRAM backing
└─ work path: channel/push forwarding to the physical GPU
│ module-owned device + real GSP session
HOST BACKEND the GPU's own hardware + GSP firmware, driven by this module
│ (the distribution nvidia.ko stays unmodified and unbound in
│ this mode)
v
REAL HARDWARE (engines, memory, interrupts)</pre>

The guest never contains Open-IOV code; the host driver is unmodified; the hypervisor is unmodified. The only new component is <code>vgsp.ko</code>.

== 3. How each side works ==

* '''Guest side.''' The stock driver performs the GSP boot handshake, sets up RPC message queues in guest memory, creates RM objects, programs channels and submits work — exactly as on bare metal ([[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]], [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]]).
* '''Host module.''' <code>vgsp.ko</code> is a VFIO mediated-device driver: it registers a parent on the physical GPU, exposes virtual GPU types, and each mdev is one virtual GPU for one VM. Inside, it presents a plausible Turing-class device (config space, BAR shadows, MSI-X), answers the GSP boot handshake and the RPC protocol with well-formed replies, and maintains the memory model that resolves guest physical addresses to host addresses ([[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]]).
* '''The interface that makes this possible.''' Since Turing, the driver's management path is not ''silicon'', it is ''firmware'': the GSP runs a version of NVIDIA's Resource Manager and the driver talks to it over shared-memory queues with a small RPC protocol. That protocol is small, stable and observable — so the project emulates '''the firmware's interface''' instead of emulating a GPU ([[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]], [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>]).
* '''Execution.''' Guest work that must be fast (copies, kernels) is forwarded to a real pipeline driven through the module's own GSP-RM session ([[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 4. Data flows ==

<pre>MMIO from the guest:
guest nvidia.ko writes BAR0 offset X
-> KVM exit -> QEMU -> VFIO region write into the mdev
-> BAR0 shadow state machine (doorbells, VF registers, notify)
-> may wake the emulation thread

RPC from the guest to "firmware":
guest writes a command-queue element into a guest-memory msgq
-> doorbell (BAR0)
-> emulation thread reads the queue through the hypervisor's guest-RAM
accessor, parses the element, dispatches the RPC
-> handler produces a reply element in the status queue
-> emulation injects an MSI-X interrupt to the guest</pre>

== 5. What runs today ==

# Guest boots, enumerates the virtual PCIe device, binds <code>nvidia.ko</code>.
# The GSP boot handshake completes against the emulation.
# Message queues are set up, versions negotiated, RM initialization exchanges complete; object creation, capability queries, vaspaces, channels and the engine inventory are answered with well-formed replies (a small set of control IDs is special-cased with real values; reply fidelity for consumed fields keeps improving).
# <code>nvidia-smi -L</code> enumerates the device; CUDA userspace starts; context creation completes; the full driver-API smoke passes (<code>ALL OK</code>, 64-byte payload round-trips).
# CE-class copies are forwarded to the physical copy engine (default off) with engine-authored semaphore releases and a fault relay. For GR (graphics/compute), a forwarded guest launch reaches the front-end but the engine has not executed it yet — the current frontier ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 6. Where the detail lives ==

* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the firmware/RM architecture and the transport we emulate.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the module's internals.
* [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]], [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] — the guest side.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]], [[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]] — memory and translation.
* [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]], [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]] — work and execution.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — supported combinations and portability.
* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/01-system-overview.md <code>../docs/03-architecture/01-system-overview.md</code>] and the rest of the architecture corpus in the repository.

Risc-v-gsp-rm-debugger

2026-10-12T04:29:52Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= risc-v-gsp-rm-debugger =

<blockquote>'''Scope.''' risc-v-gsp-rm-debugger runs GSP/uproc firmware off-hardware in a controllable RISC-V machine, instruments it with Frida-like primitives, and records durable sessions so firmware behaviour can be captured and compared with evidence instead of anecdotes. Canonical sources: [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/03-risc-v-gsp-rm-vm.md <code>../docs/05-operations/03-risc-v-gsp-rm-vm.md</code>] and [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md <code>../docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md</code>]; if this page and the docs tree disagree, the docs tree wins.</blockquote>

<blockquote>'''Development-only.''' Host-side tooling that is never shipped, never a driver dependency and never present in a guest; interfaces and file formats may change without notice.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> recorded QEMU sessions; <code>[design]</code> device model and custom simulator are roadmap.</blockquote>

== 1. What it is ==

risc-v-gsp-rm-debugger makes the firmware's '''runtime behaviour''' inspectable. It boots GSP/uproc firmware in <code>qemu-system-riscv64</code> (or a deterministic native interpreter) and offers Frida-like primitives on top:

* breakpoints and enter/leave hooks (RV64 ABI aware); watchpoints on firmware variables;
* memory read/write/hexdump/search, structure decoding, snapshot diffing;
* recorded sessions: a replayable event stream plus memory dumps;
* remote calls and run-to-cursor (execute until a chosen address);
* injection: build a position-independent RV64 service, load it into the running firmware and dispatch to it;
* declarative probe specs, a plugin API, and a Ghidra bridge for symbol export, hit annotation and decompiler-driven execution.

Its purpose is '''capturing how the firmware server behaves, repeatably and safely''', so emulation changes can be measured against recorded evidence and can survive GSP version, device and architecture churn. The tooling lives under <code>open-iov/debug/risc-v-gsp-rm-vm/</code>, with its own documentation set (design, usage, scripting, plugins, bridges, accuracy, portability, roadmap).

== 2. What it is not ==

* '''Not a GSP boot environment.''' QEMU <code>virt</code> has no falcon windows, WPR2, CSB/TRANSCOM, PMU, timers or FB DMA. <code>probe</code> reports exactly where the real image first diverges (for the TU104 570.144 image: a load-access fault at firmware vaddr <code>0x4009014</code> reading <code>0x2000000000000A00</code>) — expected data, not a boot. A device model is a roadmap item, not a claim.
* '''Not a timing model.''' Do not infer latency, jitter or concurrency behaviour from it; its in-scope conclusions are execution flow and memory side effects.
* '''Not a replacement for Ghidra or the guest diagnostics.''' Ghidra answers ''what is the code''; risc-v-gsp-rm-debugger answers ''what did it do''; the guest driver remains the oracle for guest-side behaviour.
* '''Not a hardware or security tool.''' It never attaches to a real GPU, a guest, a kernel module or an mdev, and it is never a security or confidentiality boundary.

== 3. Where it fits ==

{| class="wikitable"
|-
! Layer
! Tool
! Question answered
|-
| Static
| Ghidra and the decompilation corpus
| what is the code?
|-
| Guest driver
| guest diagnostics, ptrace tracing
| what did the driver request?
|-
| '''Firmware runtime'''
| '''risc-v-gsp-rm-debugger'''
| '''what does the firmware server do, repeatably and safely?'''
|-
| Physical backend
| supervised hardware sessions, module logs
| what did real silicon / the real GSP-RM do?
|}

The intended measurement loop: capture the real backend's RPC behaviour, run the same stimulus in risc-v-gsp-rm-debugger, and compare the recorded sessions with the tool's accuracy metrics. A simulator observation is evidence about the firmware, not about silicon; anything that would change the product is reproduced on a physical reference first.

== 4. Quickstart ==

Requirements: Linux, Python 3.10+ (standard library only), <code>qemu-system-riscv64</code> (Ubuntu package <code>qemu-system-misc</code>), <code>clang</code> and <code>lld</code>. No GPU, no driver and no kernel module are touched. From the repository root:

<pre>cd debug/risc-v-gsp-rm-vm
./risc-v-gsp-rm-debugger.sh info # environment check: QEMU, toolchain, paths
make demo && make sdk # build the bundled RV64 uproc demo and the injection SDK
./risc-v-gsp-rm-debugger.sh demo # canned run: boot, hooks, watchpoints, session
./risc-v-gsp-rm-debugger.sh analyze dumps/<printed-session-dir>
./risc-v-gsp-rm-debugger.sh probe # real GSP image: report the first divergence
./risc-v-gsp-rm-debugger.sh inject # load and invoke a position-independent service</pre>

What "good" looks like: <code>info</code> reports QEMU and the demo paths; <code>demo</code> prints hook/branch lines and a session directory under <code>dumps/</code>; <code>analyze</code> renders hook/watch/breakpoint events and a coherent summary; <code>probe</code> reports a bounded first divergence (expected data, not a boot); <code>inject</code> returns matching values on the direct and firmware-dispatched invocation paths.

A large test suite (hundreds of tests) is run with <code>make test</code>, covering unit tests plus live-QEMU smoke runs.

== 5. The primitives ==

{| class="wikitable"
|-
! Primitive
! Use it for
|-
| <code>breakpoint</code> / <code>hook</code>
| did this function run? with what arguments and return value?
|-
| <code>watch</code> (<code>w</code>/<code>r</code>/<code>a</code>)
| when did this field change, and from which store PC?
|-
| <code>run</code> / <code>run_until</code>
| bounded observation; stop at a chosen address
|-
| <code>call</code>
| remote call to unit-test a function in isolation
|-
| <code>step</code> / <code>note</code>
| single-step with dispatch; annotate the event stream
|-
| <code>Memory.read/write/hexdump/search/snapshot/diff</code>
| inspect and diff memory regions
|-
| <code>SymbolDB</code>
| resolve file/program/runtime addresses across sources
|-
| <code>Recorder</code>
| write the durable session (events, snapshots, blobs)
|-
| injection SDK
| load, register and invoke your own position-independent RV64 service
|}

Instructions between stops are invisible unless a trace is on (a TCG plugin on QEMU, or <code>--trace</code> on the native backend). Two backends — QEMU-RSP and the native interpreter — satisfy the same session API, so scripts, probes, plugins, recordings and comparisons survive a backend change.

== 6. Fidelity levels (what a result can mean) ==

{| class="wikitable"
|-
! Level
! Property
! risc-v-gsp-rm-debugger today
|-
| '''L0'''
| instruction semantics
| high (QEMU TCG, same ISA family)
|-
| '''L1'''
| memory behaviour
| high for RAM-resident code, conditional on mapping
|-
| '''L2'''
| time / concurrency
| '''not modelled''' — never claim
|-
| '''L3'''
| device environment (falcon, WPR2, CSB, PMU, timers, DMA, IRQ)
| '''absent''' — device model is roadmap
|}

A simulator observation is evidence about the firmware, not about silicon. Never quote a session for timing, interrupt, security or confidentiality claims; zero hits in a run are a coverage fact, not proof that code is dead.

== 7. Sessions as evidence ==

Every run records a session directory (format <code>greeneye-session/1</code>):

<pre>dumps/<stamp>-<uuid>-<name>/
session.json format, name, counts, snapshot inventory, meta
events.jsonl append-only event stream (hooks, watches, memory, notes)
snapshots/*.bin.z zlib-compressed region captures with sha256
blobs/* QEMU logs, instruction/access traces, extra captures
summary.md rendered analysis</pre>

Rules that make a session usable as evidence:

# '''Record with a fixed stimulus.''' The same script, seed, budgets and image hash make runs comparable; a self-comparison must be empty.
# '''Annotate as you go.''' Explain inside the session what was asked and what was changed; a chat log is not evidence.
# '''Attach the provenance.''' Firmware SHA-256, version string, symbol DB source/hash/bias, and the exact command or script.
# '''Keep the directory together.''' <code>session.json</code> + <code>events.jsonl</code> + <code>snapshots/</code> + blobs are one artifact; a copied event stream alone is not evidence.
# '''State the fidelity.''' Say which of L0/L1 was exercised and name the reference capture; do not silently waive L2/L3 gaps.
# '''Report deltas honestly.''' Every remaining delta between two runs is explained or listed as open; an empty unexplained-delta list is a claim that must be earned.
# '''Prefer an A/B pair.''' Static reading is a hypothesis; a recorded session is evidence; at least two comparable runs are a conclusion.

== 8. Safety and status ==

* '''Simulator only.''' The tools launch their own QEMU or native machine against the bundled demo or an explicitly allowed re-hosted ELF; they never attach to hardware, a guest, a kernel module, an mdev or an existing debugger stub.
* '''No guest, no hardware, no product.''' Nothing runs in a guest; injected services are prototypes, not shipped code; product changes go through the host module and a supervised physical confirmation.
* '''Bounded everywhere.''' Runs have durations and stop caps; reads and snapshots have size caps; a wedged target is left stopped and restarted clean rather than poked.
* '''Local only.''' The Ghidra bridge is a localhost transport, unauthenticated by design; never expose it on a network interface.
* '''No confidentiality claims.''' The tool is a study environment; never use injection to defeat a hardware security boundary.
* '''Status.''' The audited v0.1 surfaces are the CLI and the session format. Working and live-verified: hooks/watchpoints, memory tools, sessions, declarative probes, plugins, remote calls, run-to-cursor, the injection SDK, the Ghidra bridges, the first-fault probe and the instruction/access trace. The device model, the custom simulator and serving full host message traffic remain roadmap; porting and packaging scaffolds are pilot-status. Treat interface changes as reviewed events.

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/03-risc-v-gsp-rm-vm.md <code>../docs/05-operations/03-risc-v-gsp-rm-vm.md</code>] — project-level purpose, status and limits.
* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md <code>../docs/05-operations/04-risc-v-gsp-rm-debugger-user-guide.md</code>] — the single user front door.
* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/02-debugging-and-maintenance-workflows.md <code>../docs/05-operations/02-debugging-and-maintenance-workflows.md</code>] — captures, failure taxonomy and upgrade workflows.
* [[GSP 570.144W Firmware Analysis|<code>GSP-570.144W-Firmware-Analysis.md</code>]] and [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]] — the firmware image and architecture the sessions exercise.
* [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]] and [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — the client side of the RPC traffic and the supervised backend the simulator complements.
* [[Project Status|<code>Project-Status.md</code>]] and [https://git.vgpu.io/open-iov/blob/master/docs/README.md <code>../docs/README.md</code>] — honest overall status and the documentation front door.

Glossary

2026-10-12T04:29:51Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Glossary =

<blockquote>'''What this page is.''' The terms used across this wiki, the project's documents and its logs. Condensed from [https://git.vgpu.io/open-iov/blob/master/docs/09-reference/glossary-and-faq.md <code>../docs/09-reference/glossary-and-faq.md</code>]; the docs tree wins on disagreement. For questions, see [[FAQ|<code>FAQ.md</code>]].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>.

== GPU, driver and virtualization ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''BAR0 / BAR1 / BAR3'''
| PCI BAR windows of the GPU: register interface (16 MB), VRAM aperture (256 MB), instance/PRAMIN window.
|-
| '''CUDASW'''
| The family of NVIDIA user-space software between applications and the driver: <code>libcuda</code>, <code>libcudart</code>, the JIT compiler, the fatbinary loader, and friends.
|-
| '''Doorbell'''
| A register or memory location a driver writes to signal "new work is available" for a channel or queue.
|-
| '''mdev'''
| Mediated device: the kernel mechanism that turns a physical device into virtual instances (VFIO).
|-
| '''vGPU'''
| Virtual GPU: the device this project presents to a guest.
|-
| '''SRIOV / VF'''
| Single-root I/O virtualization / virtual functions (hardware partitioning).
|-
| '''IOMMU / SMMU'''
| Platform DMA translation and isolation (Intel VT-d / AMD-Vi / ARM SMMU).
|}

== GSP, firmware and the RM ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''GSP'''
| GPU System Processor: the embedded RISC-V (RV64IMAC) that runs firmware containing the Resource Manager.
|-
| '''GSP-RM'''
| The Resource Manager running on the GSP (the "self-hosted RM").
|-
| '''RM / RM API'''
| NVIDIA's Resource Manager and the API used to allocate objects and issue controls.
|-
| '''RMCore'''
| The RM's object system: classes, handles, parents, methods.
|-
| '''resserv / resource server'''
| The RM component that owns allocation authority, privileges and lifetimes.
|-
| '''FWSEC'''
| Firmware security processor stage of the GSP boot sequence.
|-
| '''WPR'''
| Write-Protected Region: the VRAM area reserved for firmware.
|-
| '''MSGQ'''
| The GSP message-queue transport: shared-memory element rings with a page table, plus doorbells.
|-
| '''VGX version'''
| The version the guest announces during RPC negotiation; it selects message layouts.
|-
| '''Reply fidelity'''
| How completely an emulated reply matches what real hardware returns, as a ladder: status-only → coherent zeroed params → coherent values for consumed fields → exact values.
|}

== Execution engines and memory ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''CE'''
| Copy Engine: a DMA engine; also used for UVM's internal work.
|-
| '''GR'''
| The graphics/compute engine that executes shaders (graphics pipelines and CUDA kernels).
|-
| '''SEC2'''
| Secure copy engine used by UVM for protected operations.
|-
| '''GPFIFO'''
| GPU FIFO: a ring of entries, each pointing to a push buffer for a channel.
|-
| '''Push buffer'''
| A stream of method calls (engine commands) submitted via a GPFIFO.
|-
| '''PBDMA'''
| The push-buffer DMA engine in the host scheduler that fetches and executes push buffers.
|-
| '''QMD'''
| Queue Meta Data: the per-launch descriptor a GR channel reads to execute a kernel grid.
|-
| '''USERD'''
| Per-channel control block in VRAM: PUT/GET pointers and control state.
|-
| '''Runlist'''
| The scheduler's list of channels eligible to run.
|-
| '''TSG'''
| Time-Slice Group: channels sharing a scheduling slot.
|-
| '''Engine record / engine list'''
| The per-engine descriptors (kind, instance, class, client type) the driver walks after <code>GPU_GET_ENGINES</code>; each must be self-consistent or the driver rewrites the following allocation.
|-
| '''GMMU'''
| GPU Memory Management Unit: page tables, TLB, fault handling.
|-
| '''PDB'''
| Page Directory Base: root of a GPU page table.
|-
| '''PDE / PTE'''
| Page-directory entry / page-table entry.
|-
| '''UVM'''
| NVIDIA's unified memory module: VA spaces, page trees, faults, channel manager ([[UVM in the Guest|<code>UVM-in-the-Guest.md</code>]]).
|-
| '''VA / PA / GPA / IOVA'''
| Virtual address / physical address / guest physical address / I/O virtual address.
|-
| '''vidmem / sysmem / aperture'''
| GPU local memory / system memory / how an address is qualified.
|-
| '''Semaphore / tracker'''
| A memory location written by the GPU to signal progress; UVM's tracker waits on it.
|-
| '''AMAPlib'''
| NVIDIA's shared "address mapping" library: swizzling, bank/channel mapping, aperture rules.
|-
| '''HAL'''
| Hardware abstraction layer: per-generation functions that program registers and formats.
|-
| '''Class list (classDB)'''
| The RM classes reported as supported; engine queries are filtered against it, so the class list, engine list and capability controls must agree.
|}

== Method and tooling ==

{| class="wikitable"
|-
! Term
! Meaning
|-
| '''Host oracle / A-B harness'''
| The same query or deterministic workload run against a bare-metal host driver and inside the guest, then diffed — the reference for what the guest should have seen.
|-
| '''File virtual address (<code>libcuda+0x...</code>)'''
| An address relative to the start of a shared object file, as distinct from a runtime mapping or a disassembler program address.
|-
| '''Session (risc-v-gsp-rm-debugger)'''
| A recorded run of the firmware workbench — the unit of evidence ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]).
|}

== Related pages ==

* [[FAQ|<code>FAQ.md</code>]]
* [[System Overview|<code>System-Overview.md</code>]]
* [[GSP and GSP-RM|<code>GSP-and-GSP-RM.md</code>]]
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]]

RPC Message Format

2026-10-12T04:29:51Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= GSP RPC Message Format Specification =

<blockquote>'''Historical document — superseded.''' This page describes the legacy BAR0 mailbox RPC model: the 64-byte <code>GSPR</code> magic header and the 12-type model below are '''not''' the path the guest driver uses. The live GSP-RM path is a message queue in guest system memory with 4 KB queue elements and a 32-byte RPC header (signature <code>0x43505256</code>, header version <code>0x03000000</code>); functions are a flat enum (<code>SET_GUEST_SYSTEM_INFO</code> 1, <code>FREE</code> 10, <code>GET_GSP_STATIC_INFO</code> 65, <code>GSP_RM_CONTROL</code> 76, <code>GSP_RM_ALLOC</code> 103, ...). Canonical documentation: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/03-gsp-and-gsp-rm.md <code>../docs/03-architecture/03-gsp-and-gsp-rm.md</code>] and [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>].</blockquote>

== Overview ==

The GSP firmware uses a dispatcher-based RPC system with two dispatchers:

* '''Primary''' (0x057A1AC0): ALLOC, FREE, MAP, UNMAP, PIN, UNPIN, SCHEDULE, YIELD, NOTIFY, BAR_TRAP, MIGRATE, DEBUG
* '''Secondary''' (0x04ED39DC): Branch-based dispatch for all 11 types

== Primary Dispatcher Call Convention ==

=== Entry Point ===

<pre>FUN_ram_057a1ac0(
uint32_t rpc_type, // param_1: RPC type (0x01-0x0C)
uint8_t flags_a, // param_2
uint8_t flags_b, // param_3
uint8_t flags_c, // param_4
uint32_t *out_subcmd_id,// param_5: Returns sub-command ID
uint8_t *out_a, // param_6: Echoes flags_b
uint8_t *out_b, // param_7: Echoes flags_a
uint8_t *out_c // param_8: Echoes flags_c
) -> int64_t</pre>

=== RPC Type to Sub-Command Mapping ===

The primary dispatcher validates RPC type, then indexes into the 32-entry jump table at '''0x057A3AD8'''.

{| class="wikitable"
|-
! RPC Type
! Name
! Subcmd Range
! Notes
|-
| 0x01
| ALLOC
| 0-31
| Uses 0x057E94EC main
|-
| 0x02
| FREE
| 0-31
|
|-
| 0x03
| MAP
| —
| Uses secondary dispatcher
|-
| 0x04
| UNMAP
| 0-31
| Primary + secondary
|-
| 0x05
| PIN
| —
| Uses secondary dispatcher
|-
| 0x06
| UNPIN
| —
| Uses secondary dispatcher
|-
| 0x07
| SCHEDULE
| 0-31
| Uses 0x051EFBF4 main
|-
| 0x08
| YIELD
| —
| Uses secondary dispatcher
|-
| 0x09
| NOTIFY
| —
| Uses secondary dispatcher
|-
| 0x0A
| BAR_TRAP
| —
| Uses 0x04CB8188
|-
| 0x0B
| MIGRATE
| 0-15 x2
| Dual jump tables
|-
| 0x0C
| DEBUG
| —
| Unimplemented
|}

=== Sub-Command Descriptor (Returned by Jump Table Handlers) ===

Each sub-command handler at the jump table returns:

<pre>typedef struct {
uint32_t subcmd_id; // Constant ID (e.g., 0x10, 0x14, 0x40)
uint8_t echo_flags_b; // Copy of input flags_b
uint8_t echo_flags_a; // Copy of input flags_a
uint8_t echo_flags_c; // Copy of input flags_c
} __attribute__((packed)) subcmd_descriptor_t;</pre>

'''Example: Subcmd 3 (0x057A1BCC)'''

<pre>// Returns ID = 0x10
*out_subcmd_id = 0x10;
*out_a = flags_b;
*out_b = flags_a;
*out_c = flags_c;
return 0;</pre>

'''Example: Subcmd 0 (0x057A1DB4)'''

<pre>// Returns ID = 0x40
*out_subcmd_id = 0x40;
*out_a = flags_b;
*out_b = flags_a;
*out_c = flags_c;
return 0;</pre>

== Secondary Dispatcher Call Convention ==

=== Entry Point ===

<pre>FUN_ram_04ed39dc(rpc_message_t *msg) -> int64_t</pre>

=== Message Structure (Inferred) ===

<pre>typedef struct {
uint32_t rpc_type; // 0x01-0x0B
uint32_t sub_type; // Sub-operation code
uint64_t payload_ptr; // Pointer to RPC-specific payload
uint32_t payload_size; // Size of payload
uint32_t flags; // Control flags
uint64_t context; // Opaque context
uint32_t sequence; // Sequence number
uint32_t reserved;
} __attribute__((packed)) rpc_message_t;</pre>

=== Branch-Based Dispatch ===

Secondary dispatcher uses <code>bgeu</code>/<code>bltu</code> branches on <code>rpc_type</code> register:

<pre># Pseudocode
if (rpc_type >= 0x0C) goto error;
switch(rpc_type):
case 0x03: call MAP_handler_A;
case 0x04: call UNMAP_handler_A;
case 0x05: call PIN_handler_A (0x04ED34B8);
case 0x06: call UNPIN_handler_A;
case 0x07: call SCHEDULE_handler_A;
case 0x08: call YIELD_handler_A;
case 0x09: call NOTIFY_handler_A;
case 0x0A: call BAR_TRAP_handler (0x057675B7);</pre>

== RPC-Specific Payloads ==

=== ALLOC (0x01) - Via 0x057E94EC ===

<pre>typedef struct {
uint64_t size; // Allocation size in bytes
uint32_t type; // Memory type (VRAM, sysmem, etc.)
uint32_t alignment; // Alignment requirement
uint64_t va_hint; // Virtual address hint
uint32_t flags; // CACHE_COHERENT, UNCACHED, etc.
uint32_t pid; // Process ID
uint32_t vf_id; // VF ID (for vGPU)
uint32_t reserved;
uint64_t *out_handle; // Output: allocation handle
uint64_t *out_va; // Output: mapped virtual address
uint64_t *out_pa; // Output: physical address
} alloc_payload_t;</pre>

=== MAP (0x03) - Handlers 0x05766AB7 / 0x05778A37 ===

<pre>typedef struct {
uint64_t handle; // Allocation handle from ALLOC
uint64_t va; // Target virtual address
uint64_t size; // Map size
uint32_t prot; // PROT_READ|PROT_WRITE|PROT_EXEC
uint32_t flags; // MAP_FIXED, MAP_SHARED, etc.
uint32_t vf_id; // Target VF
uint32_t reserved;
} map_payload_t;</pre>

=== PIN (0x05) - Handler 0x04ED34B8 ===

<pre>typedef struct {
uint64_t handle; // Allocation handle
uint64_t offset; // Offset within allocation
uint64_t size; // Size to pin
uint32_t vf_id; // Target VF
uint32_t flags; // PIN_WRITE, PIN_READ, PIN_LONGTERM
uint64_t *out_dma_addr; // Output: DMA address for device
} pin_payload_t;</pre>

=== SCHEDULE (0x07) - Via 0x051EFBF4 / 0x05767F37 ===

<pre>typedef struct {
uint32_t vf_id; // VF to schedule
uint32_t priority; // Scheduling priority
uint64_t time_slice_ns; // Time slice in nanoseconds
uint32_t flags; // PREEMPTIBLE, REALTIME, etc.
uint32_t reserved;
} schedule_payload_t;</pre>

=== BAR_TRAP (0x0A) - Handler 0x04CB8188 ===

<pre>typedef struct {
uint32_t bar_index; // BAR number (0-5)
uint32_t trap_type; // READ, WRITE, ATOMIC
uint64_t offset; // Offset within BAR
uint64_t value; // Value for writes
uint32_t size; // Access size (1,2,4,8)
uint32_t vf_id; // Trapping VF
uint64_t *out_value; // Output: read value
} bar_trap_payload_t;</pre>

=== MIGRATE (0x0B) - Dual Jump Tables ===

<pre>typedef struct {
uint32_t phase; // 0=prepare, 1=transfer, 2=commit, 3=cleanup
uint32_t subcmd; // Sub-command index (0-15 for each table)
uint64_t src_vf; // Source VF
uint64_t dst_vf; // Destination VF
uint64_t data_ptr; // Migration data buffer
uint64_t data_size; // Buffer size
uint32_t flags; // LIVE, POSTCOPY, etc.
uint32_t reserved;
} migrate_payload_t;</pre>

== Wire Format (Host → GSP) ==

<blockquote>'''Legacy BAR0 mailbox format — not used by the guest driver.''' The current msgq header/element facts are in the note at the top of this page.</blockquote>

=== Message Header (64 bytes) ===

<pre>typedef struct {
uint32_t magic; // 0x47535052 ("GSPR")
uint32_t version; // Protocol version (1)
uint32_t rpc_type; // 0x01-0x0C
uint32_t sequence; // Monotonic sequence
uint64_t timestamp_ns; // Submission timestamp
uint32_t payload_size; // Size of payload
uint32_t flags; // MSG_SYNC, MSG_ASYNC, MSG_NOREPLY
uint64_t correlation_id;// For async matching
uint64_t reserved[3]; // Padding to 64 bytes
} __attribute__((packed)) rpc_header_t;</pre>

=== Complete Message ===

<pre>+------------------+
| rpc_header_t | 64 bytes
+------------------+
| payload (varies) | payload_size bytes
+------------------+
| padding to 256B | (alignment)
+------------------+</pre>

== Response Format ==

=== Synchronous Response ===

<pre>typedef struct {
int64_t status; // 0 = success, negative = error
uint32_t rpc_type; // Echo of request
uint32_t sequence; // Echo of request
uint64_t timestamp_ns; // Completion timestamp
uint32_t payload_size; // Response payload size
uint32_t reserved;
} __attribute__((packed)) rpc_response_header_t;</pre>

=== Error Codes ===

{| class="wikitable"
|-
! Code
! Name
! Description
|-
| 0
| OK
| Success
|-
| -1 (0xFFFF)
| EPERM
| Permission denied
|-
| -2 (0xFFFE)
| ENOENT
| Not found
|-
| -3 (0xFFFD)
| EINVAL
| Invalid argument
|-
| -6 (0xFFFA)
| ENODEV
| No such device
|-
| -12 (0xFFF4)
| ENOMEM
| Out of memory
|-
| -16 (0xFFEE)
| EBUSY
| Resource busy
|-
| -22 (0xFFEA)
| EINVAL
| Invalid argument
|-
| -28 (0xFFE4)
| ENOSPC
| No space left
|-
| -37 (0xFFDB)
| ENOLCK
| No locks available
|-
| -56 (0xFFC8)
| EBADMSG
| Bad message
|-
| -84 (0xFFAC)
| EOVERFLOW
| Value too large
|-
| -90 (0xFFA6)
| ENOSTR
| Not a stream
|-
| -91 (0xFFA5)
| ENODATA
| No data available
|-
| -94 (0xFFA2)
| ENOSR
| Out of streams resources
|-
| -97 (0xFF9F)
| EPROTO
| Protocol error
|-
| -111 (0xFF91)
| ETIMEDOUT
| Connection timed out
|}

== Kernel Client Implementation ==

=== Message Builder ===

<pre>static int gsp_rpc_send(gsp_client_t *client, uint32_t rpc_type,
void *payload, size_t payload_size,
void *response, size_t *response_size) {
rpc_header_t hdr = {
.magic = 0x47535052,
.version = 1,
.rpc_type = rpc_type,
.sequence = atomic_inc(&client->seq),
.timestamp_ns = ktime_get_ns(),
.payload_size = payload_size,
.flags = MSG_SYNC,
};

// Build complete message
uint8_t msg[256];
memcpy(msg, &hdr, sizeof(hdr));
memcpy(msg + 64, payload, payload_size);

// Submit to GSP doorbell
return gsp_doorbell_ring(client->gsp, msg, sizeof(msg));
}</pre>

=== Doorbell Mechanism ===

<pre>// GSP doorbell register (PCIe BAR0 + offset)
#define GSP_DOORBELL_OFFSET 0x1000

static void gsp_doorbell_ring(gsp_device_t *gsp, void *msg, size_t len) {
// Copy message to shared memory ring
// Ring GSP doorbell
writel(1, gsp->bar0 + GSP_DOORBELL_OFFSET);
}</pre>

== Async Completion ==

=== Notification Mechanism ===

<pre>typedef struct {
uint32_t rpc_type;
uint32_t sequence;
int64_t status;
uint64_t timestamp_ns;
uint32_t payload_size;
uint32_t reserved;
} async_completion_t;

// Delivered via MSI-X or GSP notification queue</pre>

----

== Reference ==

* Primary Dispatcher: 0x057A1AC0
* Secondary Dispatcher: 0x04ED39DC
* Jump Table: 0x057A3AD8 (32 entries)
* RPC Type Jump Table: 0x059BFEB0 (12 types)

Project Status

2026-10-12T04:29:51Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Project Status =

<blockquote>'''What this page is.''' The honest, dated state of the project: what works today, what is in progress, and what is not started. It is refreshed periodically and each claim maps to a maintained document in the repository. Canonical sources: [https://git.vgpu.io/open-iov/blob/master/docs/README.md <code>../docs/README.md</code>] (project front door), [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] (known-knowns and known-unknowns) and [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/03-execution-frontier.md <code>../docs/04-implementation/03-execution-frontier.md</code>] (the execution frontier). When this page and the docs disagree, the docs tree wins.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented, default-off or behind supervised gates; <code>[wip]</code> partially implemented; <code>[wall]</code> current blocker; <code>[target]</code> named success state; <code>[planned]</code> designed, not built.

== 1. Milestone ladder ==

{| class="wikitable"
|-
! Milestone
! State
|-
| Emulated vGPU device (PCI, BARs, MSI-X, VFIO mdev)
| <code>[live]</code> working
|-
| GSP boot emulation (FWSEC / booter / RISCV handshake)
| <code>[live]</code> working
|-
| GSP-RM message queues and RPC transport
| <code>[live]</code> working
|-
| Stock guest driver probe and <code>nvidia-smi</code> (TU104)
| <code>[live]</code> working
|-
| CUDA userspace init and the driver-API smoke (<code>cuInit</code>, device enumeration, <code>cuCtxCreate</code>, allocation, copies, free, destroy)
| <code>[live]</code> working — the smoke prints <code>ALL OK</code> and the 64-byte payload round-trips exactly
|-
| Guest CUDA kernel execution on real hardware
| <code>[wall]</code> a forwarded guest launch reaches the engine queue (the front end deposits the methods and the scheduling path accepts them), but the engine does not yet execute the forwarded launch descriptor — an engine-authored release has not been observed. Emulation-mode kernel runs (executed through the host bridge on a no-claim path) have completed green; reproducibility of the green path across boots is an open issue
|-
| Physical backend on real hardware (module-owned GPU running its real GSP-RM)
| <code>[live]</code> boot, RPC session, object creation and copy-engine work proven; GR execution is the same frontier as above
|-
| Multi-tenant isolation (mdev per VM, per-context page directories)
| <code>[wip]</code> designed; the baseline service-state ownership work is in progress
|-
| OpenGL / Vulkan (3D), video
| <code>[planned]</code>
|-
| Windows guests
| <code>[planned]</code> (target)
|-
| Xen / Argo / OpenXT / Qubes Air transports
| <code>[planned]</code> (the emulation core is hypervisor-independent by construction)
|-
| Live migration
| <code>[planned]</code>
|-
| Performance numbers
| <code>[planned]</code> — no published numbers yet; any claim will ship with its measurement methodology
|}

== 2. What "CUDA works" means here ==

A green <code>nvidia-smi</code> or even a passing driver-API smoke is not "CUDA works". The project's definition of done for CUDA requires all of the following ([https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>]):

# Stock guest driver probe, <code>nvidia-smi</code>, device enumeration — done.
# Context creation and the full driver-API surface (<code>cuMemAlloc</code>, copies, free, destroy) — done.
# A guest kernel launch that '''executes on real hardware''' and is proven by an engine-authored completion — the current frontier.
# Memory the guest's work refers to resolving correctly end to end (the VA↔PA model) — partially, for the paths exercised so far.
# Correct completion, semaphore and fault semantics back to the guest — CE class work has the relay; GR is open.
# Reproducibility: every boot, not a lucky one — an open issue.
# No ad-hoc default-off parameters required for the canonical path.

Until every gate holds, the honest formulation is: '''initialization and the driver-API path work; engine execution is in progress.'''

== 3. How to see it yourself ==

The reference setup takes about half an hour from a fresh host ([[Install and First Run|<code>Install-and-First-Run.md</code>]]); the current expected outcome is exactly the ladder above — driver probe, <code>nvidia-smi</code>, the full driver-API smoke, and the execution frontier where it stands. The risc-v-gsp-rm-debugger workbench ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]) is the host-side tooling used to study the firmware side of that frontier.

== 4. How this page is maintained ==

Every claim above carries a date and a tag, maps to a maintained document in the repository, and is refreshed as milestones land. The execution story, with its evidence and its open theories, is on [[The Execution Frontier|<code>Execution-Frontier.md</code>]].

== 5. Related pages ==

* [[Install and First Run|<code>Install-and-First-Run.md</code>]]
* [[System Overview|<code>System-Overview.md</code>]]
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]]
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — which hardware and software combinations are Reference, Target or Experimental.
* [[FAQ|<code>FAQ.md</code>]]

Physical GPU Backend and Guest Wiring

2026-10-12T04:29:50Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Physical GPU Backend and Guest Wiring =

<blockquote>'''Scope.''' The design that executes guest work on the real GPU: why the host driver cannot be the path and how the live GSP-RM session is wired to the guest-facing emulation. Canonical sources: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] and [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> proven with the stock guest driver; <code>[gated]</code> implemented default-off; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> planned end state.

== 1. The problem and its constraints ==

The guest's real work (copies, memory operations, grid launches) must execute on the physical GPU. The constraints: one DKMS kernel module on the host, no daemon and no new host userspace; host and guest drivers unmodified, with no paravirtual enlightenment in the guest; all emulation and translation logic inside the module; and real execution — compute is not emulated in software.

== 2. Why the host driver is not the path ==

{| class="wikitable"
|-
! Interface
! What it offers
! Reachable from our module?
|-
| Host driver (distribution build)
| resource management, scheduling, memory
| No — no general RM entry points for third-party modules, no kernel-client interface, user-pointer ioctl surface
|-
| Host driver peer-memory exports
| pin/map memory for peer devices
| Partially — memory only, no submission
|-
| Vendor vGPU host stack
| mediated work submission and scheduling
| Requires the licensed vGPU host branch and a vGPU-aware guest — excluded
|-
| SR-IOV virtual functions
| hardware-partitioned device
| Not available on this GPU generation
|-
| The GPU's own GSP firmware
| the resource manager server itself
| Yes — once the module owns the device; this is the path used
|}

With an unmodified host driver there is no callable submission interface into it, so the module must own the device and talk to the hardware itself.

== 3. Mechanism options and the decision ==

<pre> (A) Proxy through the host driver: module -> host driver's RM -> real channel
-> GPU. Needs a modified host driver or a vendor interface — not implementable.
(B) Own the GPU; own scheduling (no GSP): module -> own channels/runlists/page
tables -> GPU. Reproduces the scheduler, MMU ownership, context switching.
(C) Hybrid — only meaningful once (A) and (B) exist; not on the critical path.
(D) Own the GPU; use its real GSP firmware as the server <- chosen
guest push -> module -> real GSP-RM session -> real channel -> GPU
The module becomes the host-side resource-manager client; scheduling,
page tables and context management stay the firmware's job.
(E) Vendor-managed sharing (vGPU host stack, SR-IOV VFs) — deployment-gated; out of scope.</pre>

Option (D) is the recorded decision (2026-09-29): the only option satisfying module-only, no-daemon, unmodified drivers, real execution, and multi-tenant sharing.

== 4. The chosen mechanism: bring-up ladder ==

The module binds the physical GPU as its driver while the distribution driver is unbound (dedicated-GPU deployment, §9). Each stage is reversible and independently verified:

# '''Ownership''' — the board's power domain is held active across the window (the platform, not PCI, owns it); the module binds the device and maps BAR0/1/3. <code>[live]</code>
# '''Firmware staging''' — the matching GSP image is fetched and validated (ELF64 RISC-V; loadable section, version, hardware signature); no device state touched. <code>[live]</code>
# '''Boot''' — protected region in video memory, secure-verification stage, booter, RISC-V core start: FWSEC → Booter Load → RISC-V active → GSP-RM init-done. <code>[live]</code>
# '''Host RM client''' — the module speaks the already-mapped RPC protocol to the real firmware: device/subdevice, vaspaces, channels, memory. "Call the host driver's RM" becomes "be the RM client". <code>[live]</code>
# '''Guest mediation''' — the guest-facing emulation stays unchanged; real-resource objects translate to GSP-RM requests, and server-owned ranges populate the real MMU. <code>[wip]</code>
# '''Work path''' — decoded guest pushes become real push streams; resolved host pages map through module-built scratch page tables; the firmware schedules the work. <code>[gated]</code>
# '''Completion relay''' — hardware-completed semaphores land in guest-polled memory; events and faults relay through the existing MSI-X injection path. <code>[gated]</code>

== 5. The two planes and the wiring ==

The '''emulation plane''' serves the guest: BAR shadows, the sysmem msgq protocol, a hand-built TU104 static-configuration model, and a software work pipeline that executes decoded memsets, copies and semaphore releases. The '''physical backend plane''' owns the device, boots the real GSP-RM and runs the RM client ladder. The wiring joins them at the lifecycle seam (mdev create/remove), a static-config identity overlay, a read-only control whitelist, and — for eligible copy-engine work — the scratch-mapped forwarding path, all default-off and fail-open to software.

<pre>GUEST (QEMU) HOST: vgsp.ko (module-owned GPU) PHYSICAL GPU
-----------------------------------------------------------------------------
INIT preflight: distribution driver green; no vgsp/mdev/QEMU [live]
claim: power domain held -> distribution driver released -> module
binds the device -> liveness check [live]
boot: FWSEC -> Booter Load -> RISC-V active -> GSP-RM init [live]
mdev + ladder: backend joins the live session; real objects,
channels, first real CE copy [live]
guest: nvidia.ko probes; emulation serves handshake, static config,
RM battery, UVM registration [live]
WORK guest CUDA app -> libcuda -> nvidia.ko -> GPFIFO -> doorbell
host pump decodes; eligible CE copies forwarded via module-built scratch
page tables; GR: launch reaches the front-end, descriptor not seen [wall]
DONE engine authors the release in guest memory; host relays via MSI-X [gated]
TEARDOWN mdev remove -> QEMU stop -> module unload -> clean GSP-RM shutdown
-> protected region down -> distribution driver rebound [live]
FAIL a failed mutating operation parks the device in a locked fail-safe;
never auto-retried; recovery is supervised [live]</pre>

== 6. Proven capabilities ==

{| class="wikitable"
|-
! Capability
! Status
|-
| Control plane (GSP boot emulation, RPC, BAR/PCI/MSI-X)
| done <code>[live]</code>
|-
| Driver-API baseline (init, devices, context, allocation, copies, free, destroy)
| done — 64/64 payload round-trip <code>[live]</code>
|-
| Software bridge for decoded memory/state operations
| in progress <code>[wip]</code>
|-
| Device ownership, firmware staging, board-liveness checks
| verified live <code>[live]</code>
|-
| Power domain held active across the ownership window
| verified across supervised cycles <code>[live]</code>
|-
| Real GSP boot by the module, through init-done and clean release
| done, no reboot on clean cycles <code>[live]</code>
|-
| Host RM client and first real engine work (static config, real objects, 4 KiB CE copy)
| done <code>[live]</code>
|-
| Guest-copy forwarding, deferred release, hardware-primary copies, engine-authored relay, fault relay
| default-off, verified live; one relayed-copy mismatch under triage <code>[gated]</code>
|-
| CUDA kernel execution (<code>vectorAdd</code>)
| open <code>[wall]</code> — the launch-descriptor install path; correct results are the <code>[target]</code>
|}

== 7. The execution frontier ==

The RM client ladder reaches the compute engine (a GR0 channel plus a compute-class object). A forwarded guest CUDA launch now reaches the front-end (ring entry fetched, methods executed) but the inline launch-descriptor install has not been observed and no grid has dispatched; context validity at dispatch is ruled out. The wall is the form of the forwarded launch method stream — see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]. A failed mutating operation on the backend parks the device in a locked fail-safe that is never auto-retried; recovery is a supervised step.

== 8. Safety and platform discipline ==

* '''Dedicated GPU deployment.''' One GPU serves the vGPU service; the host uses another GPU or integrated graphics. The distribution driver stays unmodified and is unbound from that device only while the service is active.
* '''The board's power domain belongs to the platform.''' Releasing the last driver can power-gate the board and the power-on sequence can fail; the ownership path holds the whole board in a held-active state across the window, verifies config-space liveness, and aborts on any power-loss signature without retry.
* '''Supervised cycles.''' The device is driven by the module under supervised cycles with a written recovery plan. A failed mutating operation parks the device in a locked fail-safe and is never auto-retried; any reboot requires explicit human approval with a written reason and post-reboot plan first.
* '''Fail closed.''' No forwarding seam programs guest-derived page tables into the real MMU until the population path is proven end to end; the current forwarded path submits engine work only through module-built scratch page tables mapping already-resolved host pages.

== 9. Sharing policy and isolation ==

* '''Dedicated GPU (chosen):''' one GPU serves the vGPU service; a multi-GPU host gives the distribution driver its own GPU.
* '''Shared single GPU with the host driver bound:''' no conforming mechanism exists on this generation without vendor interfaces; it would require relaxing a constraint (vendor vGPU stack, SR-IOV hardware, or a host daemon).
* Isolation is one real GSP client per mdev with per-client page tables; host access to guest memory stays behind the hypervisor's per-device DMA interface, and tenant policy is a later layer on top ([[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]]).

== 10. Open questions ==

* Firmware/host driver ABI match for the supported family on the development GPU.
* GPU unbind/reset lifecycle and recovery after guest faults.
* Which guest allocations need real VRAM and which can stay in system memory.
* Fault semantics and per-tenant scheduling once multiple mdevs share the service GPU.

== 11. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/08-physical-gpu-integration.md <code>../docs/03-architecture/08-physical-gpu-integration.md</code>] — canonical: mechanism choice and device bring-up.
* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — canonical: the guest-wiring design.
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — the GR wall and the software execution plane.
* [[The Host Module (vgsp.ko)|<code>Host-Module.md</code>]] — the module that owns both planes.
* [[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]] — page-table seams and scratch mappings.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]], [[Project Status|<code>Project-Status.md</code>]] — support scope and the honest status picture.

Migration Protocol

2026-10-12T04:29:50Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Migration Protocol (RPC 0x0B) =

<blockquote>'''Historical document — superseded.''' This page records the MIGRATE RPC (type 0x0B) as seen in firmware analysis; in emulation mode the MIGRATE RPC returns <code>-ENOTSUPP</code>. The live migration implementation is [https://git.vgpu.io/open-iov/blob/master/kernel/migration.c <code>../kernel/migration.c</code>]; design and status are documented in [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation <code>../docs/04-implementation/</code>].</blockquote>

== Overview ==

The MIGRATE RPC (type 0x0B) implements live migration for vGPU virtual functions. It uses '''dual jump tables''' for two-phase migration operations, accessed via the primary dispatcher at 0x057A1AC0.

== Key Addresses ==

=== Migration Jump Tables ===

==== Table 1: 0x04179ED8 (Primary MIGRATE branch at 0x057A1BDC) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
! Description
|-
| 0
| 0x057A2344
| ✅
| Phase 1: Prepare - returns ID 0x14, checks flag at +0x7F9
|-
| 1
| 0x057A2320
| ✅
| Phase 1: Prepare variant
|-
| 2
| 0x057A230C
| ✅
| Phase 1: Prepare variant
|-
| 3
| 0x057A22F8
| ✅
| Phase 1: Prepare - returns ID 0x10
|-
| 4
| 0x057A2388
| ✅
| Phase 1: Prepare variant
|-
| 5
| 0x057A23D8
| ✅
| Phase 1: Prepare variant
|-
| 6
| 0x057A23AC
| ✅
| Phase 1: Prepare variant
|-
| 7
| 0x057A2478
| ✅
| Phase 1: Prepare variant
|-
| 8
| 0x057A2428
| ✅
| Phase 1: Prepare variant
|-
| 9
| 0x057A2454
| ✅
| Phase 1: Prepare variant
|-
| 10
| 0x057A2404
| ✅
| Phase 1: Prepare variant
|-
| 11
| 0x057A2364
| ✅
| Phase 1: Prepare variant
|-
| 12
| 0x057A1F3C
| (dup)
| Duplicate entry
|-
| 13
| 0x057A299C
| —
| Phase 2: Transfer
|-
| 14
| 0x057A1F3C
| (dup)
| Duplicate entry
|-
| 15
| 0x057A1F3C
| (dup)
| Duplicate entry
|}

==== Table 2: 0x04179FBC (Primary MIGRATE branch at 0x057A1D5C) ====

{| class="wikitable"
|-
! Index
! Target
! Decompiled
! Description
|-
| 0
| 0x057A257C
| ✅
| Phase 2: Transfer
|-
| 1
| 0x057A25E4
| ✅
| Phase 2: Transfer
|-
| 2
| 0x057A24EC
| ✅
| Phase 2: Transfer
|-
| 3
| 0x057A2608
| ✅
| Phase 2: Transfer
|-
| 4
| 0x057A2534
| ✅
| Phase 2: Transfer
|-
| 5
| 0x057A25C0
| ✅
| Phase 2: Transfer
|-
| 6
| 0x057A24A4
| ✅
| Phase 2: Transfer
|-
| 7
| 0x057A259C
| ✅
| Phase 2: Transfer
|-
| 8
| 0x057A2510
| ✅
| Phase 2: Transfer
|-
| 9
| 0x057A2558
| ✅
| Phase 2: Transfer
|-
| 10
| 0x057A24C8
| ✅
| Phase 2: Transfer
|-
| 11
| 0x057A262C
| ✅
| Phase 2: Transfer
|-
| 12
| 0x057A345C
| ✅
| Phase 3: Commit
|-
| 13
| 0x057A36D8
| ✅
| Phase 3: Commit
|-
| 14
| 0x057A35D8
| ✅
| Phase 3: Commit
|-
| 15
| 0x057A3658
| ✅
| Phase 3: Commit
|}

== Migration Phases ==

=== Phase 0: Discovery (Pre-migration) ===

<pre>// Query VF migratability
int gsp_migrate_query(struct gsp_vf *vf, struct migrate_query *out) {
migrate_payload_t payload = {
.phase = MIGRATE_PHASE_QUERY,
.subcmd = 0,
.src_vf = vf->id,
.dst_vf = 0,
};
return gsp_rpc_send(vf->gsp, RPC_MIGRATE, &payload, sizeof(payload), out, sizeof(*out));
}</pre>

=== Phase 1: Prepare (Table 1, indices 0-11) ===

'''Goal''': Quiesce VF, prepare memory state for transfer

<pre>// Subcmd 3 (0x057A22F8) - Returns ID 0x10
int migrate_prepare_quiesce(vf_id) {
// 1. Pause VF scheduling
// 2. Flush pending commands
// 3. Pin all VF memory
// 4. Return quiesce token
}

// Subcmd 0 (0x057A2344) - Returns ID 0x14, checks flag at in_t1+0x7F9
int migrate_prepare_snapshot(vf_id, *token) {
// 1. Snapshot VF register state
// 2. Snapshot BAR configurations
// 3. Snapshot WPR regions
// 4. Snapshot page tables
// 5. Return snapshot handle
}</pre>

=== Phase 2: Transfer (Table 1 index 13, Table 2 indices 0-11) ===

'''Goal''': Transfer memory pages to destination

<pre>// Table 2, subcmd 0 (0x057A257C)
int migrate_transfer_pages(vf_id, page_list, *progress) {
// 1. Iterate page list
// 2. For each page:
// a. Copy via DMA (0x0588F9C4)
// b. Verify checksum
// c. Mark page dirty if modified
// 3. Return bytes transferred
}

// Table 2, subcmd 11 (0x057A262C) - Dirty page tracking
int migrate_track_dirty(vf_id, *dirty_bitmap) {
// Return bitmap of pages dirtied since last transfer
}</pre>

=== Phase 3: Commit (Table 2 indices 12-15) ===

'''Goal''': Final synchronization, cutover to destination

<pre>// Subcmd 12 (0x057A345C)
int migrate_commit_finalize(vf_id, dst_vf_id) {
// 1. Final dirty page transfer
// 2. Transfer register state
// 3. Activate destination VF
// 4. Release source VF resources
// 5. Notify completion
}

// Subcmd 13 (0x057A36D8)
int migrate_commit_verify(vf_id, *verify_result) {
// 1. Verify memory integrity
// 2. Verify register state
// 3. Verify device state
// 4. Return PASS/FAIL
}</pre>

=== Phase 4: Cleanup (Post-migration) ===

<pre>int migrate_cleanup(src_vf_id) {
// 1. Release pinned memory
// 2. Free snapshot buffers
// 3. Remove WPR regions
// 4. Destroy migration context
}</pre>

== Migration Message Format ==

<pre>typedef struct {
uint32_t phase; // MIGRATE_PHASE_*
uint32_t subcmd; // Sub-command index (0-15 per table)
uint64_t src_vf; // Source VF ID
uint64_t dst_vf; // Destination VF ID
uint64_t data_ptr; // Data buffer (page list, bitmap, etc.)
uint64_t data_size; // Buffer size
uint32_t flags; // MIGRATE_LIVE, MIGRATE_POSTCOPY, etc.
uint32_t reserved;
uint64_t token; // Opaque migration token
} __attribute__((packed)) migrate_payload_t;

#define MIGRATE_PHASE_QUERY 0
#define MIGRATE_PHASE_PREPARE 1
#define MIGRATE_PHASE_TRANSFER 2
#define MIGRATE_PHASE_COMMIT 3
#define MIGRATE_PHASE_CLEANUP 4

#define MIGRATE_FLAG_LIVE (1<<0) // Live migration
#define MIGRATE_FLAG_POSTCOPY (1<<1) // Post-copy mode
#define MIGRATE_FLAG_AUTO_COMMIT (1<<2) // Auto-commit after transfer
#define MIGRATE_FLAG_VERIFY (1<<3) // Verify integrity</pre>

== Sub-Command Descriptors (Returned by Handlers) ==

All migration handlers follow the primary dispatcher pattern:

<pre>typedef struct {
uint32_t subcmd_id; // Constant: 0x10, 0x14, etc.
uint8_t echo_flags_b; // Copy of input flags_b
uint8_t echo_flags_a; // Copy of input flags_a
uint8_t echo_flags_c; // Copy of input flags_c
} migrate_descriptor_t;</pre>

'''Known Subcmd IDs''':

{| class="wikitable"
|-
! Subcmd
! ID
! Phase
|-
| 0x057A22F8 (Table1[3])
| 0x10
| Prepare
|-
| 0x057A2344 (Table1[0])
| 0x14
| Prepare
|-
| Others
| TBD
| Transfer/Commit
|}

== vGPU-IO Integration ==

=== Migration State Machine ===

<pre>typedef enum {
MIG_STATE_IDLE = 0,
MIG_STATE_QUERY,
MIG_STATE_PREPARE,
MIG_STATE_TRANSFER,
MIG_STATE_COMMIT,
MIG_STATE_CLEANUP,
MIG_STATE_FAILED,
MIG_STATE_COMPLETED
} mig_state_t;

struct gsp_migration {
struct gsp_vf *vf;
mig_state_t state;
uint64_t token;
uint64_t dst_vf;
struct migrate_progress progress;
struct completion comp;
struct mutex lock;
};</pre>

=== Migration Coordinator ===

<pre>int gsp_migration_start(struct gsp_vf *vf, uint64_t dst_vf) {
struct gsp_migration *mig = kzalloc(sizeof(*mig), GFP_KERNEL);
vf->migration = mig;
mig->vf = vf;
mig->dst_vf = dst_vf;
mig->state = MIG_STATE_QUERY;

// Phase 0: Query
struct migrate_query qry;
gsp_migrate_query(vf, &qry);
if (!qry.migratable) return -EINVAL;

// Phase 1: Prepare
mig->state = MIG_STATE_PREPARE;
migrate_payload_t prep = {
.phase = MIGRATE_PHASE_PREPARE,
.subcmd = 3, // Quiesce
.src_vf = vf->id,
.dst_vf = dst_vf,
};
migrate_descriptor_t desc;
gsp_rpc_send(vf->gsp, RPC_MIGRATE, &prep, sizeof(prep), &desc, sizeof(desc));
mig->token = desc.subcmd_id; // Use returned token

// Phase 2: Transfer (iterative)
mig->state = MIG_STATE_TRANSFER;
while (!gsp_migration_transfer_iter(mig)) {
msleep(10); // Throttle
if (signal_pending(current)) break;
}

// Phase 3: Commit
mig->state = MIG_STATE_COMMIT;
migrate_payload_t commit = {
.phase = MIGRATE_PHASE_COMMIT,
.subcmd = 12,
.src_vf = vf->id,
.dst_vf = dst_vf,
.token = mig->token,
.flags = MIGRATE_FLAG_VERIFY,
};
gsp_rpc_send(vf->gsp, RPC_MIGRATE, &commit, sizeof(commit), NULL, 0);

// Phase 4: Cleanup
mig->state = MIG_STATE_CLEANUP;
gsp_migration_cleanup(mig);

mig->state = MIG_STATE_COMPLETED;
return 0;
}</pre>

=== Transfer Iteration ===

<pre>bool gsp_migration_transfer_iter(struct gsp_migration *mig) {
migrate_payload_t xfer = {
.phase = MIGRATE_PHASE_TRANSFER,
.subcmd = 0,
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.data_ptr = (uint64_t)mig->progress.page_list,
.data_size = mig->progress.page_count * sizeof(uint64_t),
.token = mig->token,
};

struct migrate_progress prog;
int ret = gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE,
&xfer, sizeof(xfer), &prog, sizeof(prog));

if (ret) return true; // Error = done

mig->progress.bytes_transferred += prog.bytes_transferred;
mig->progress.pages_transferred += prog.pages_transferred;
mig->progress.dirty_pages = prog.dirty_pages;

// Check if more dirty pages to transfer
if (prog.dirty_pages > 0 && !(mig->flags & MIGRATE_FLAG_POSTCOPY)) {
return false; // Continue iterating
}

return true; // Transfer complete
}</pre>

=== Post-Copy Mode ===

<pre>int gsp_migration_postcopy(struct gsp_migration *mig) {
// Enable post-copy on destination
migrate_payload_t pc = {
.phase = MIGRATE_PHASE_TRANSFER,
.subcmd = 11, // Dirty tracking
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.flags = MIGRATE_FLAG_POSTCOPY,
.token = mig->token,
};

// Page fault handler on destination
gsp_page_fault_handler = gsp_migration_page_fault;

return gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE, &pc, sizeof(pc), NULL, 0);
}</pre>

== Integration with BAR/WPR ==

=== BAR State Migration ===

<pre>int gsp_migration_bar_save(struct gsp_migration *mig) {
// Save BAR1 aperture, force access state
struct bar_save_msg bar_save = {
.vf_id = mig->vf->id,
.bar = 1,
};
return gsp_rpc_send(mig->vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_SAVE, &bar_save, sizeof(bar_save),
mig->bar_state_buf, &mig->bar_state_size);
}

int gsp_migration_bar_restore(struct gsp_migration *mig) {
// Restore BAR1 on destination
struct bar_restore_msg bar_restore = {
.vf_id = mig->dst_vf,
.bar = 1,
.data = mig->bar_state_buf,
.data_size = mig->bar_state_size,
};
return gsp_rpc_send(mig->vf->gsp, RPC_BAR_TRAP,
BAR_SUBCMD_RESTORE, &bar_restore, sizeof(bar_restore));
}</pre>

=== WPR Region Migration ===

<pre>int gsp_migration_wpr_save(struct gsp_migration *mig) {
// Save WPR regions for VF
migrate_payload_t wpr = {
.phase = MIGRATE_PHASE_PREPARE,
.subcmd = 4, // WPR snapshot
.src_vf = mig->vf->id,
.dst_vf = mig->dst_vf,
.token = mig->token,
};
return gsp_rpc_send(mig->vf->gsp, RPC_MIGRATE, &wpr, sizeof(wpr), NULL, 0);
}</pre>

== Error Handling ==

{| class="wikitable"
|-
! Phase
! Error
! Recovery
|-
| Prepare
| EBUSY
| Wait, retry
|-
| Prepare
| ENOMEM
| Reduce snapshot size
|-
| Transfer
| ETIMEDOUT
| Checkpoint, retry
|-
| Transfer
| EIO
| Failover to post-copy
|-
| Commit
| EINVAL
| Rollback to source
|-
| Commit
| ENOSPC
| Expand destination
|}

== Debugging ==

=== Migration Trace Points ===

<pre>trace_gsp_mig_phase(vf_id, phase, subcmd, token);
trace_gsp_mig_transfer(vf_id, pages, bytes, dirty);
trace_gsp_mig_commit(vf_id, status, verify);
trace_gsp_mig_error(vf_id, phase, error);</pre>

=== GSP Log Strings ===

Search for migration-related strings in firmware:

* Migration phase markers
* Page transfer progress
* Commit verification results

----

== Reference ==

* Primary Dispatcher MIGRATE branches: 0x057A1BDC, 0x057A1D54
* Jump Table 1: 0x04179ED8 (16 entries)
* Jump Table 2: 0x04179FBC (16 entries)
* Decompiled handlers: 28 functions in decompiled/

Introduction to GMMU

2026-10-12T04:29:49Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Introduction to the GMMU =

<blockquote>'''Scope.''' A from-first-principles introduction to the GPU Memory Management Unit (GMMU): what it translates, how its page tables work on our Turing reference, how the host module uses them, and how the GMMU relates to the CPU MMU and the platform IOMMU/SMMU. The architecture chapter is [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/05-gmmu-amap-hal-gr.md <code>../docs/03-architecture/05-gmmu-amap-hal-gr.md</code>]; the isolation policy authority is [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] (condensed in [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]]).</blockquote>

<blockquote>'''Evidence discipline.''' Hardware-format claims condense the tracked field catalogue [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md <code>../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md</code>] and live observations; numeric IDs, layouts and code anchors are cited, and no timing/cycle claim is made.</blockquote>

<blockquote>Status: 2026-10-11, tree <code>master @ 08f39c2f</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[proven]</code> hardware-format fact; <code>[hypothesis]</code> unexercised; <code>[design]</code> not built.</blockquote>

== 1. What the GMMU is ==

The '''GMMU (GPU Memory Management Unit)''' is the address-translation hardware inside the GPU. Every engine — GR (shaders and CUDA kernels), CE (copy engines), SEC2, the video engines — addresses memory through '''GPU virtual addresses (VAs)''', and the GMMU translates each VA into a physical address: an offset in GPU local memory ('''vidmem'''/VRAM), an address in system memory ('''sysmem'''), or a peer GPU's memory across a fabric.

It is a paged MMU, conceptually the CPU MMU's sibling:

* '''Multi-level page tables''' describe the mappings; the root is a '''page directory base (PDB)'''.
* A '''TLB''' caches translations so the walk is not paid per access.
* Translation failures raise '''faults''' (invalid/missing leaf, missing table), reported through engine fault buffers.
* '''Each context has its own page directory (VA space)''', so one context cannot name another context's mappings — this is the hardware isolation boundary the project builds multi-tenancy on.

What makes it ''not'' just "the CPU MMU on a GPU":

* The tables live in GPU-visible memory (vidmem or sysmem), and each directory entry says which aperture the next level lives in.
* Entries carry GPU-specific '''aperture''' (which physical space) and '''memory-kind''' fields that the CPU MMU does not have.
* The "physical" addresses inside the tables are '''guest-physical addresses or emulated-VRAM offsets''' in our virtualized setting; the host module owns the resolution to real host addresses (§4.2). On a bare-metal GPU the same structures point at real vidmem/sysmem.

For this project the GMMU matters three ways: the guest driver programs it exactly as on bare metal (no guest patches); the guest's page tables are the ground truth for ''where its memory actually is''; and the per-context page directory is the only proven hardware isolation primitive available on the reference generation (§4.2).

== 2. The three MMUs: CPU MMU, GMMU, IOMMU/SMMU ==

{| class="wikitable"
|-
!
! CPU MMU
! GMMU
! IOMMU / SMMU
|-
| Translates
| process VA → host physical address
| engine GPU VA → GPU physical (vidmem / sysmem / peer)
| device DMA address → host physical address
|-
| Lives in
| the CPU core
| the GPU
| the host platform (IOMMU: Intel VT-d / AMD-Vi) or the SoC (SMMU, ARM)
|-
| Programmed by
| the OS kernel (per-process tables)
| the GPU driver / RM (the ''guest'' driver here)
| the hypervisor / OS (per-device or per-VM domains)
|-
| Table unit
| per-process page tables (e.g. CR3)
| per-context page directory (VA space)
| per-device/per-VM translation domain
|-
| Isolation scope
| process ↔ process
| context ↔ context (engine-level)
| VM/device ↔ VM/device (platform DMA)
|-
| In our stack
| guest and host kernels
| guest driver builds the tables; host walks or plants its own
| hypervisor enforces it; the module resolves GPAs itself when it needs host addresses
|}

They are three instances of the same idea — ''page tables plus a TLB give virtual memory and isolation to an agent'' — at three different points in the address path, owned by three different software layers. In a virtualized guest a GPU access passes through '''two''' of them:

<pre>engine (GR / CE)
│ GPU VA
▼
GMMU (inside the GPU, driven by the guest's page tables)
│ guest physical address (GPA) / GPU local address
▼
IOMMU / SMMU (in the host platform, programmed by the hypervisor for this VM)
│ host physical address (HPA)
▼
host DRAM / real VRAM</pre>

The two stages are independent: the '''GMMU''' decides ''which guest page'' an engine touches; the '''IOMMU/SMMU''' decides ''whether this device may reach that page at all'' and where it lives in host memory. GMMU isolation between guest contexts is the GPU driver/RM's job; IOMMU isolation between VMs is the platform's job.

'''How the two meet in this project:'''

* In real vGPU, the host vGPU manager uses the platform IOMMU to establish the GPU's view of guest memory. Here the host module does the equivalent resolution itself.
* On the reference host the physical GPU's IOMMU group is an '''identity domain''' (all-RAM identity) and the mediated device registers as an '''emulated-IOMMU VFIO device''': its type-1 container holds guest DMA mappings privately and no kernel <code>iommu_domain</code> exists for it, so <code>iommu_iova_to_phys</code> cannot translate guest GPAs. Guest RAM is backed by a QEMU shared file and the module translates GPA→HPA through that file's page cache, behind the active-mdev gate (a per-mdev-gated translation, ''not'' itself an IOMMU guarantee).
* The project rule is explicit: '''memory isolation is the platform IOMMU's job, and we never weaken it''' — DMA only through the hypervisor-provided interface for the device's own VM, no global mappings, fail closed if a required isolation property cannot be established.
* On aarch64 hosts the '''SMMU''' plays the IOMMU's role; page sizes, cache attributes and SMMU behaviour are on the host-portability list for the Grace/GB10 targets.

== 3. How the GMMU works (Turing reference) ==

=== 3.1 Page-tree geometry ===

The Turing page-tree path inherits the Pascal/Volta geometry and uses the '''VER2''' entry encoding. Depths run root (0) to leaf (4); VA width is '''49 bits''':

{| class="wikitable"
|-
! Depth
! HW level
! VA bits
! Index bits
! Entry size
! Role
|-
| 0
| PDE3
| 48:47
| 2
| 8 B
| pointer PDE
|-
| 1
| PDE2
| 46:38
| 9
| 8 B
| pointer PDE
|-
| 2
| PDE1
| 37:29
| 9
| 8 B
| pointer PDE (<code>NO_ATS</code> set here)
|-
| 3
| PDE0
| 28:21
| 8
| 16 B (dual) / 8 B (2 MB leaf)
| dual 64K/4K PDE, or a 2 MB leaf PTE
|-
| 4
| PTE
| 20:12 (4 KB) / 20:16 (64 KB)
| 9 / 5
| 8 B
| leaf PTE
|}

Supported page sizes are '''2 MB''' (depth 3), '''64 KB''' and '''4 KB''' (depth 4). A leaf entry alone does not say 4 KB vs 64 KB — the table's depth and page-size context do — so decoders must report <code>4k_or_64k</code> rather than guess.

=== 3.2 Entry encoding (VER2) ===

'''Leaf PTE (8 bytes):'''

{| class="wikitable"
|-
! Field
! Bits
! Meaning
|-
| <code>valid</code>
| 0
| entry is present
|-
| <code>aperture</code>
| 2:1
| 0 video, 1 peer, 2 system coherent, 3 system non-coherent
|-
| <code>volatile</code>
| 3
| cache/volatility bit
|-
| <code>encrypted</code>
| 4
| encrypted memory
|-
| <code>privilege</code>
| 5
| privileged access
|-
| <code>read_only</code>
| 6
| write protection
|-
| <code>atomic_disable</code>
| 7
| atomics disabled
|-
| <code>address_sys</code>
| 53:8
| system-aperture address bits (PA 53:12)
|-
| <code>address_vid</code>
| 32:8
| video/peer address bits (PA 36:12)
|-
| <code>peer_id</code>
| 35:33
| peer/fabric id (peer aperture)
|-
| <code>comptagline</code>
| 55:36
| compression tag (overloaded as high address bits for the 47-bit peer case)
|-
| <code>kind</code>
| 63:56
| memory kind id
|}

'''Directory PDE (8 bytes)''' is the same shape with one discriminator change: bit 0 is <code>is_pde_pointer</code> (0 = pointer to a lower table, 1 = a 2 MB leaf PTE), and '''aperture 0 means invalid at directory level''' (it means video memory at leaf level). Depth 2 sets <code>NO_ATS</code> (bit 5).

'''Dual PDE0 (16 bytes)''' holds two halves for mixed 64K/4K tables: BIG (low 64 bits) and SMALL (high 64 bits); bit 0 discriminates the whole slot.

Live guest note: the client's leaf entries in the format we observe are <code>PA[47:12] | flags</code>, with <code>flags = 0xe</code> for a valid system-memory leaf. The <code>kind</code> field carries numeric memory-kind ids (243 named values); common compute values are <code>0x00 PITCH</code>, <code>0x06 GENERIC_MEMORY</code>, <code>0x07 INVALID</code>, <code>0x0F SMSKED_MESSAGE</code>, <code>0xFE GENERIC_16BX2</code>.

The decoder (<code>sim/gmmu_pte.py</code>, docs/43) is decode-only: unsupported encodings fail closed, set bits not covered by a named field are reported as reserved, and nothing is guessed.

=== 3.3 Walking, caching and faults ===

A walk starts at the channel's '''page-directory base''' and follows <code>PDE3 → PDE2 → PDE1 → PDE0 → PTE</code>; the leaf yields the physical address, page size, aperture and access permissions. The TLB caches results, and there is a second, coarser cache that matters operationally:

* '''PTE fetch-line rule (learned live):''' the engine caches PTE fetches at '''128-byte (16-PTE) granularity''', and the forwarding code has '''no invalidate path'''. A runtime PTE write into a line the GMMU has already walked is invisible to the engine. Consequences: page-table slots for a guest window are '''pre-built before any engine access''', and each forwarded copy pair consumes a whole fresh line from a pre-zeroed pool (16 page-table pages, capacity 512 pairs per boot in the current design) — a line is never reused within a boot.
* '''Faults:''' invalid or missing leaves and missing tables raise GMMU faults; the driver allocates fault buffers whose sizes come from <code>INTERNAL_GMMU_GET_STATIC_INFO</code> (control <code>0x20800a59</code>): '''64 KB replayable / 4 KB legacy''' — the values the emulation serves, because a wrong value breaks UVM early.

== 4. How the GMMU is used ==

=== 4.1 Guest side (the stock driver) ===

* The guest driver allocates VA spaces and '''builds the page tables''', then publishes the directory per channel/context through <code>SET_PAGE_DIRECTORY</code> (<code>hVASpace</code> + <code>chId</code>; the host records the directory per owner in <code>kernel/gsp_rm_emulation.c:3307-3319</code>).
* Memory descriptors carry an aperture: <code>ADDR_SYSMEM = 1</code>, <code>ADDR_FBMEM = 2</code>.
* '''Server-owned ranges:''' <code>COPY_SERVER_RESERVED_PDES</code> (control <code>0x90f10106</code>) tells the client that some page-table ranges are populated by the "server" (us). The exact division of labour for <code>RESERVE_ENTRIES</code>/<code>RELEASE_ENTRIES</code> and invalidations is a tracked known-unknown.
* Channels submit push buffers that reference GPU VAs; the engines walk the GMMU to reach the data.

=== 4.2 Host side (<code>vgsp.ko</code>) ===

* '''Truth.''' The host resolves guest VAs by '''walking the driver-published page directories itself''' (the trusted GMMU walk in <code>kernel/gsp_sema_service.c</code>, VER2 geometry at <code>:1058-1072</code>, decode <code>:1093</code>, walk <code>:1174</code>), plus injected pins and content-keyed rescue when neither names a mapping. This is what lets the software path apply a semaphore release to the correct host page.
* '''Forwarding.''' For real CE copies the module builds its '''own scratch page tables''' that map already-resolved host pages (guest RAM through the shared file; shadow pages through the emulated-VRAM mapping), under the fetch-line rules of §3.3. Real guest page tables are ''not yet'' programmed into the real MMU — guest submission on hardware is the end state (P6).
* '''Emulation.''' Guest vidmem is a private host-RAM buffer per mdev (<code>kernel/bar_init.c:68-77</code>); FB-aperture accesses stay inside the owning mdev's buffer, and guest RAM is reached only through the hypervisor's per-mdev DMA interface (<code>vfio_dma_rw</code>).
* '''Isolation.''' The per-context page directory is the '''only proven hardware isolation primitive''' on this generation: the VER2 PTE/PDE has '''no tenant/ACL/PASID field''' (proven absence in the named field set; <code>peer_id</code> is a fabric peer selector, not a security domain). <code>read_only</code> (bit 6) is a proven ''format'' field, but its ''enforcement'' is a <code>[hypothesis]</code> until the M7 supervised probe. The multi-tenant design therefore places decisions in '''topology''' (which directory a page appears in, which channel may name it) plus software validation and audit in the execution plane — see [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]].
* '''Versioning.''' GMMU controls, static info and entry formats are version/hardware-keyed; formats are expected to differ per generation and must be re-derived by measurement, never guessed.

=== 4.3 Tooling built around it ===

{| class="wikitable"
|-
! Tool / doc
! Question it answers
|-
| PTE decoder <code>sim/gmmu_pte.py</code> + [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md docs/43]
| decode a captured entry field-by-field (decode-only, fail closed)
|-
| Walker <code>sim/gmmu_walk.py</code> + [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/48-GMMU-WALK-AND-LAUNCH-AUDIT.md docs/48]
| walk a captured PDB to the leaf mapping or the exact fault (4K/2M validated against a live captured channel PDB; 64K modelled)
|-
| Launch audit <code>harness/launch_audit.py</code> + docs/48
| where the engine would read/write each leg of a submission: <code>ok</code> / <code>pa_mismatch</code> / <code>unmapped</code> / <code>access_denied</code>
|-
| Oracle workflow [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/44-GMMU-ORACLE-AND-WORKFLOW.md docs/44]
| compare simulated vs observed GMMU/FBIF register values, per register, with fidelity labels
|-
| Observe-only windows [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/42-GMMU-FBIF-OBSERVABILITY.md docs/42]
| record what firmware touches in the GMMU register windows without faking semantics
|}

== 5. Limits and open questions ==

* '''<code>read_only</code> enforcement''' is unproven (M7 supervised probe); '''encryption (CCS) semantics''' are unstudied. Neither can be assumed as a tenant boundary.
* '''Invalidation semantics''' are not fully modelled: the 128-byte fetch-line observation is live evidence, not a complete invalidate contract.
* The '''64K dual path''' is modelled from the format definitions, not yet validated against a capture.
* '''Newer generations''' (Ampere/Hopper/Blackwell, GB10) are expected to change the entry format; each must be re-derived. RM 610 work sits behind its own human approval gate.
* '''No timing/cycle claims''' are made anywhere here; zero hits in a run are a coverage fact, not proof of dead code.
* Multi-mdev isolation is first a '''service-state-ownership''' problem (host tables keyed by guest-chosen handles/VAs alone collide); GMMU policy is built on top of that fix, not instead of it.

== 6. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/05-gmmu-amap-hal-gr.md <code>../docs/03-architecture/05-gmmu-amap-hal-gr.md</code>] — architecture chapter: GMMU, AMAPlib/HALs, channels, runlists, GR.
* [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]] — the isolation design condensed.
* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/14-multi-tenant-gmmu-protection.md <code>../docs/03-architecture/14-multi-tenant-gmmu-protection.md</code>] — canonical multi-tenant GMMU policy.
* [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]] — how guest work becomes physical execution; the fetch-line rule in context.
* [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/11-physical-backend-guest-wiring.md <code>../docs/03-architecture/11-physical-backend-guest-wiring.md</code>] — page-table seams and the guest-memory reachability model.
* [https://git.vgpu.io/open-iov/blob/master/docs/04-implementation/01-known-knowns-and-unknowns.md <code>../docs/04-implementation/01-known-knowns-and-unknowns.md</code>] — server-owned page tables and the IOMMU reachability note.
* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — threat model and the IOMMU rules.
* [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md <code>../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md</code>], [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/44-GMMU-ORACLE-AND-WORKFLOW.md docs/44], [https://git.vgpu.io/open-iov/blob/master/debug/risc-v-gsp-rm-vm/docs/48-GMMU-WALK-AND-LAUNCH-AUDIT.md docs/48] — the field catalogue and the offline walk/audit tooling.

Install and First Run

2026-10-12T04:29:49Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Install and First Run =

<blockquote>'''What this page is.''' From a fresh Ubuntu host with a supported NVIDIA GPU to a guest VM running a stock NVIDIA driver and <code>nvidia-smi</code>, in about half an hour. Condensed from the maintained guide [https://git.vgpu.io/open-iov/blob/master/docs/02-getting-started/01-install-and-first-run.md <code>../docs/02-getting-started/01-install-and-first-run.md</code>]; the docs tree wins on disagreement.</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> verified path; <code>[target]</code> roadmap item.

== 1. Prerequisites ==

'''Host'''

* Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a <code>[target]</code>);
* an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the reference — see [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]]);
* a working '''host''' NVIDIA driver (<code>nvidia-smi</code> runs on the host; Open-IOV does not modify it);
* IOMMU enabled (<code>intel_iommu=on</code> / <code>amd_iommu=on</code>), hugepages configured, kernel headers for the running kernel;
* <code>qemu-system-x86_64</code> with KVM; libvirt optional;
* root (or sudo) for module load and mdev creation.

'''Guest'''

* Ubuntu 24.04 server (the reference guest), or any Linux with the matching stock NVIDIA driver available;
* '''no Open-IOV component at all''' — that is the point ([[System Overview|<code>System-Overview.md</code>]]).

== 2. Get the code and build ==

<pre>git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps # build/runtime dependencies
./install.sh nvidia # optional: NVIDIA OpenRM modules + build glue
./install.sh firmware # GSP firmware image (verified)
./install.sh build # kernel module + userspace</pre>

<code>install.sh</code> is safe to re-run. The manual equivalent, and the order that matters:

<pre>make kernel # first: generates Module.symvers
./scripts/build_nvidia_openrm.sh # optional (see below)
make userspace</pre>

'''Build order matters.''' NVIDIA's OpenRM modules resolve symbols that <code>vgsp.ko</code> exports; building them in the wrong order produces <code>Unknown symbol</code> failures at load time.

'''The OpenRM stage is optional.''' It is needed only for the native vGPU-management workflows in which NVIDIA's modules are built against the symbols <code>vgsp.ko</code> exports. The QEMU/emulated path uses the distribution driver that already manages the physical GPU; <code>load-stack</code> skips the OpenRM submodule when a distribution driver is present.

== 3. Load the stack and create a virtual GPU ==

<pre>./scripts/stack_manager.sh status # what is currently loaded
./scripts/stack_manager.sh load-stack # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia
./scripts/stack_manager.sh mdev-create # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list # inspect /sys/bus/mdev/devices/</pre>

The mdev device ''is'' the virtual GPU; everything after this treats it like any other VFIO device.

== 4. Launch a reference guest ==

<pre>./scripts/stack_manager.sh qemu # reference raw-QEMU guest with the mdev</pre>

To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly as any VFIO device:

<pre><hostdev mode='subsystem' type='mdev' model='vfio-pci'>
<source>
<address uuid='<mdev-uuid>'/>
</source>
</hostdev></pre>

'''Attach behind a root port, not on the root bus.''' QEMU rewrites a device's PCIe capability when it detects a root-complex-integrated endpoint, which changes the link capabilities the guest driver sees. The reference script does this for you.

== 5. Install the stock driver in the guest ==

Inside the guest:

<pre>sudo apt update
sudo apt install -y linux-headers-generic build-essential
# The validated combination is the stock 570.144 Linux runfile (the same
# driver version as the emulation target).
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot</pre>

Verify:

<pre>nvidia-smi # must list the virtual GPU
lsmod | grep -c nvidia</pre>

If <code>nvidia.ko</code> fails to probe, capture host dmesg, guest dmesg, guest <code>lspci -vvv</code> and the guest module parameters, and file an issue.

== 6. First CUDA check ==

<pre># in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery</pre>

'''Expectation today''' (<code>[live]</code> unless noted): driver probe, <code>nvidia-smi</code>, device enumeration and the full CUDA '''driver-API''' smoke complete end to end (<code>ALL OK</code>, 64-byte payload round-trips). CUDA '''kernel execution''' is the current frontier: a guest launch is forwarded and reaches the engine queue, but engine execution is not yet observed (<code>[wall]</code>) — see [[Project Status|<code>Project-Status.md</code>]] and [[The Execution Frontier|<code>Execution-Frontier.md</code>]] before drawing conclusions from a kernel that does not run.

== 7. Day-to-day operations ==

<pre>./scripts/stack_manager.sh up # build all + load + create mdev
./scripts/stack_manager.sh status # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test # host test suite
./scripts/stack_manager.sh unload # correct-order teardown
./scripts/stack_manager.sh cleanup-crash</pre>

Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM → firmware; unload is the reverse. Do not <code>rmmod</code> by hand while a VM holds the mdev.

== 8. Watching the emulation work ==

The module logs with the prefix '''<code>vgsp-<subsystem>:</code>''' (for example <code>vgsp-rpc:</code>, <code>vgsp-sema:</code>, <code>vgsp-work:</code>) plus per-RPC traces:

<pre>sudo dmesg -w | grep -i 'vgsp' # live
sudo dmesg -T | grep -i 'vgsp' | tail -50</pre>

Developer debug knobs (module parameters) are documented in the repository's debugging workflow document.

== 9. Troubleshooting index ==

{| class="wikitable"
|-
! Symptom
! Start here
|-
| Module will not load
| <code>dmesg</code>, <code>modinfo vgsp.ko</code>, stale module state → <code>cleanup-crash</code>
|-
| No mdev type for your GPU
| [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]]
|-
| Guest sees wrong PCIe link speed/width
| attach behind a root port (section 4)
|-
| <code>nvidia.ko</code> probe fails in guest
| guest dmesg + <code>lspci -vvv</code> + [[Guest Driver Lifecycle: A Field Guide|<code>Driver-Lifecycle-Field-Guide.md</code>]]
|-
| <code>nvidia-smi</code> works, a CUDA kernel does not run
| expected today — the execution frontier, not your setup ([[The Execution Frontier|<code>Execution-Frontier.md</code>]])
|-
| VM boot fails or the guest hangs
| <code>scripts/stack_manager.sh cleanup-crash</code>, then reboot
|}

== 10. Uninstall ==

<pre>./scripts/stack_manager.sh unload</pre>

The project never modifies your host driver's files; removing the module and its mdev devices returns the host to its previous state.

The Host Module (vgsp.ko)

2026-10-12T04:29:48Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= The Host Module (vgsp.ko) =

<blockquote>'''Scope.''' The host-side kernel module that presents one virtual Turing-class GPU per <code>mdev</code> instance: the virtual device, the emulation engine, the guest-memory model and the work-forwarding design. Canonical source: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>]; the docs tree wins on disagreement with this page.</blockquote>

<blockquote>Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[live]</code> observed with the stock guest driver; <code>[wip]</code> demonstrated live, being productized; <code>[gated]</code> implemented default-off behind supervised gates; <code>[wall]</code> known blocking limit; <code>[target]</code> designed, not built; <code>[planned]</code> roadmap.</blockquote>

== 1. Role and position ==

<pre>guest VM ── PCIe MMIO / DMA / MSI-X through the hypervisor
(KVM/QEMU or Xen: VFIO mdev transport, IOMMU)
│
vgsp.ko: virtual device · emulation core · memory model · work path
│
physical GPU + GSP firmware, driven by this module (host driver unmodified)</pre>

The module registers as an '''mdev parent''' on the physical GPU, exposing one device type per supported profile; each <code>mdev</code> is an independent virtual GPU ([[System Overview|<code>System-Overview.md</code>]], [[The vGSP Execution Plane (vEP)|<code>Execution-Plane.md</code>]]).

== 2. The virtual device ==

The guest sees a plausible Turing-class GPU (IDs, class code, BARs, capability list). Some fields exist only for validation: PCIe link capabilities (the hypervisor can rewrite them), MSI-X masking state, and non-SR-IOV VF windows.

{| class="wikitable"
|-
! BAR
! Size
! Backing
! Contents
|-
| BAR0
| 16 MB
| host <code>vzalloc</code> shadow
| boot, doorbells, VF registers, PRAMIN window, VBIOS
|-
| BAR1
| 256 MB
| host <code>vzalloc</code> vidmem buffer
| guest-visible VRAM aperture
|-
| BAR3
| —
| instance window
| instance/PRAMIN behaviour
|}

MMIO is served by VFIO region callbacks or directly from the shadow when mapped into the guest. BAR1 is host RAM today — correctness first: the real GPU cannot read it, so forwarding must later migrate objects to real VRAM or expose DMA-mapped system memory [target] ([[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]]).

== 3. The emulation core ==

A kernel thread drives the device's "firmware" behaviour [live]:

<pre>BAR0 doorbell / VF-register write → read msgq element from guest memory
→ dispatch RPC → write reply → inject MSI-X (eventfd → irqfd)</pre>

Guest memory is reached via <code>vfio_dma_rw()</code> from kernel-thread context. The RPC philosophy: correct shape first (or the driver walks off into uninitialized memory), plausible values second (values must be self-consistent), never invent input (in/out parameter blocks are echoed; zeroing the <code>GSP_RM_ALLOC</code> reply once faulted <code>libcuda</code>). Handlers cover boot/static-info exchanges, device/FB/BUS/INTR tables, CE and GMMU queries, reserved-PDE contracts, channel/CE allocation and an unknown-control fallback. Push buffers are decoded as method streams and executed in software — memsets, copies, semaphore releases — a '''correctness bridge, not the data plane''' [wip]; emulation-mode CUDA is verified end to end (<code>vectorAdd</code> over 1M elements, <code>bad=0</code>) [live]. Physical-GSP boot is default-off [gated]; the firmware analyzer, version tables, HAL tables and dynamic RPC dispatcher are not yet on the live path [target].

== 4. The guest-memory model ==

'''What the client gives us:''' channel memory descriptors (<code>NV_MEMORY_DESC_PARAMS</code>) with ''physical'' addresses (instance memory, method buffer, userd, ramfc, error notifiers) and the page-table contract for server-owned VA ranges (<code>COPY_SERVER_RESERVED_PDES</code> — table-level PAs, page shifts, VA range, ''empty'' leaves). It gives us none of its own sysmem maps.

'''What we derive instead''' [live] — the module rebuilds the missing mappings from the guest's own bookkeeping, with no cooperation from the guest driver:

# '''VA↔PA pairs live in guest memory''': UVM and RM record both the GPU VA and the kernel CPU VA per object (<code>uvm_rm_mem_t.vas[CPU]/vas[GPU]</code> +0x30/+0x38).
# '''CPU VAs are direct-map addresses''': <code>PA = cpu_va - page_offset_base</code>, the base recovered from guest memory or <code>/proc/kallsyms</code> during development.
# '''Exception:''' control-device <code>mmap()</code> pages have no leaf PTEs; verified pairs arrive via <code>sema_map_add</code> until a durable join exists. The page-granular cache is fed by a trusted GMMU walk ([[Introduction to GMMU|<code>Introduction-to-GMMU.md</code>]]).

'''Load-bearing copy invariants:''' (1) a copy runs at most once per GPFIFO entry — replays are suppressed by entry-word identity plus an executed-operation bitmap; (2) a virtual operand translates per 4 KB page (195 of 256 pages were contiguous in one observed 1 MB range); (3) a consumed slot's work never runs later — the per-put-epoch mark is authoritative until a put wrap.

== 5. The forwarding path (design) ==

<pre>guest push submitted
→ detect (doorbell / GPPUT / push-buffer scan), decode the method stream
→ EMULATE state/notification (release, membar, fence) or
FORWARD real engine work (GR launch, CE copy, memop)
→ FORWARD: submit equivalent work on a real channel bound to the physical GPU
→ relay completions (semaphore / fault / interrupt) back into guest memory</pre>

Proxying through the distribution host driver has '''no callable entry point from kernel context''', so the module '''owns the device and runs its real GSP firmware as the resource-manager server''', with emulation as the control plane. CE copies, release and fault relay are implemented default-off and verified live [gated]; GR execution is the open wall [wall] — a forwarded launch reaches the front-end; the inline launch-descriptor install is unobserved ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]).

== 6. Debug parameters ==

<code>/sys/module/vgsp/parameters/*</code> makes guest memory and the emulated device inspectable without rebuilding:

{| class="wikitable"
|-
! Parameter
! Purpose
|-
| <code>scan_guest_ram=<v></code> / <code>scan_vidmem=<v></code>
| find a 64-bit value in guest RAM (0–8 GB) or the VRAM shadow
|-
| <code>scan_seq=<v1,...></code> / <code>scan_pt=<pa></code> / <code>scan_masked=<v>,<mask></code>
| find a 32-bit sequence, entries pointing at a PA, or <code>(x & mask) == v</code>
|-
| <code>dump_vidmem=<off>,<len></code> / <code>fill_vidmem=<v></code>
| dump/scan or fill the VRAM shadow
|-
| <code>poke_guest=<gpa>,<v></code> / <code>poke_fill=<gpa>,<v>,<n></code>
| write a 32-bit value / fill a range in guest physical memory
|-
| <code>sema_probe=<gpu_va>[,count]</code> / <code>sema_map_add=<va>,<pa></code>
| probe a semaphore VA / teach one verified VA→PA pair
|-
| <code>dump_guest=<gpa>,<len></code>
| dump guest physical memory (capped)
|}

The physical-backend path adds a default-off set of backend controls (documented in the repository; workflows in [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/02-debugging-and-maintenance-workflows.md <code>../docs/05-operations/02-debugging-and-maintenance-workflows.md</code>]).

== 7. Build and symbol contracts ==

* <code>vgsp.ko</code> exports the <code>nvidia_vgpu_vfio_*</code> interface (<code>probe</code>/<code>remove</code>, ops get/set, mdev parent registration) to pair the device with OpenRM host modules.
* OpenRM modules built from source need <code>KBUILD_EXTRA_SYMBOLS=$(pwd)/kernel/Module.symvers</code> and must build '''after''' this module (missing symbols and stale build caches are common otherwise).
* On single-GPU developer systems the distribution driver owns the physical GPU and the module runs standalone; the OpenRM path is only for native-vGPU management, not QEMU.

== 8. Implemented vs designed ==

{| class="wikitable"
|-
! Area
! Status
|-
| mdev/VFIO device, config space, BARs, MSI-X; BAR0 state machine; boot handshake [live]
| implemented
|-
| Message queues + RPC + init controls (stock 570.144) [live]
| implemented
|-
| Guest sysmem access + VA↔PA discovery [wip]
| debug-grade, to be productized
|-
| Push decoding + semaphore completion [wip]
| demonstrated live, being productized
|-
| Emulation-mode CUDA end-to-end [live]
| repeatable 1M-element <code>vectorAdd</code>, <code>bad=0</code>
|-
| Guest-RAM physical path + server-owned page-table population [gated]
| default-off; GPA→HPA via shared-file page cache verified live on the CE
|-
| GPU work forwarding: CE copy, release and fault relay [gated]
| default-off, verified live; GR open [wall]
|-
| Execution backend (module-owned GPU, direct real-GSP session) [wall]
| boot, static configuration and CE copies proven; GR execution open (failed mutating attempts park the device in a locked fail-safe, never auto-retried)
|-
| Real VRAM backing [target]; Windows, Xen, aarch64 [planned]
| designed, not implemented
|}

== 9. Related documentation ==

* Canonical chapter: [https://git.vgpu.io/open-iov/blob/master/docs/03-architecture/06-host-module-and-hybrid-model.md <code>../docs/03-architecture/06-host-module-and-hybrid-model.md</code>] — the source of this condensation.
* [[GSP RPC Emulation|<code>GSP-RPC-Emulation.md</code>]], [[BAR Emulation Specification|<code>BAR-Emulation-Specification.md</code>]], [[WPR Management|<code>WPR-Management.md</code>]]; [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]], [[The Execution Frontier|<code>Execution-Frontier.md</code>]], [[The Guest Stack and CUDASW|<code>Guest-Stack-and-CUDASW.md</code>]].

Hardware and Software Matrix

2026-10-12T04:29:47Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= Hardware and Software Matrix =

<blockquote>'''Companion summary.''' This page condenses [https://git.vgpu.io/open-iov/blob/master/docs/06-portability/hardware-software-matrix.md <code>../docs/06-portability/hardware-software-matrix.md</code>] — the canonical support matrix: the support ladder, the current status of every host/GPU/guest/hypervisor combination, and what adding a combination requires. The validation process behind every support decision is [https://git.vgpu.io/open-iov/blob/master/docs/06-portability/03-device-validation-and-porting-sequence.md <code>../docs/06-portability/03-device-validation-and-porting-sequence.md</code>]. When this page and the docs tree disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov/blob/master/docs/README.md <code>../docs/README.md</code>].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[reference]</code> continuously tested, bugs block releases; <code>[supported]</code> tested per release; <code>[experimental]</code> partial or bring-up; <code>[target]</code> designed for, bring-up in progress; <code>[design]</code> design direction, not the current build.

== 1. Support ladder ==

Support means '''tested and documented''', not "should probably work".

{| class="wikitable"
|-
! Level
! Meaning
|-
| '''Reference'''
| continuously tested; bugs block releases
|-
| '''Supported'''
| tested per release; documented workarounds if any
|-
| '''Experimental'''
| known to boot/test partially; may need configuration
|-
| '''Target'''
| architecture designed for it; bring-up in progress
|-
| '''Out of scope'''
| not planned
|}

== 2. Host platforms ==

{| class="wikitable"
|-
! Platform
! Status
! Notes
|-
| x86_64 + NVIDIA Turing (TU104)
| '''Reference'''
| the development hardware (RTX 2080-class); the only Reference platform today
|-
| x86_64 + Ampere / Ada consumer
| '''Target'''
| profile work; expected to be configuration + measurement
|-
| x86_64 + Hopper / datacenter
| '''Target'''
| requires MIG-aware policy for multi-tenancy
|-
| aarch64 + Grace-Hopper (GH200)
| '''Target'''
| host-architecture work in the memory model
|-
| aarch64 + Grace-Blackwell
| '''Target'''
| as above, newer firmware/driver family
|-
| GB10 (DGX Spark, Surface-class RTX laptops)
| '''Target'''
| same architecture; firmware/ROM quirks to be measured
|-
| Other NVIDIA generations
| '''Experimental'''
| bring-up path in the canonical document §5
|}

The design direction is that '''each new host platform should require configuration + verification, not new code''' — not the current state: selection is compile-time today and the module is built for the TU104 profile.

== 3. Guest operating systems ==

{| class="wikitable"
|-
! Guest
! Status
|-
| Linux x86_64 (stock <code>nvidia.ko</code>, CUDA)
| '''Experimental''' — probe, <code>nvidia-smi</code> and the driver-API smoke pass; CE-class copies are forwarded default-off; for GR the guest launch reaches the front-end but no grid has dispatched yet (see [[The Execution Frontier|<code>Execution-Frontier.md</code>]]); no continuous guest testing yet
|-
| Linux aarch64
| '''Target'''
|-
| Windows x86_64 (WDDM driver, CUDA, 3D)
| '''Target'''
|-
| Other Unix (FreeBSD, ...)
| Out of scope until demand
|}

'''Guest rule:''' the guest never contains Open-IOV code — no guest agent, no driver patch, no kernel parameters beyond what the stock driver needs.

== 4. Hypervisors and transports ==

{| class="wikitable"
|-
! Hypervisor
! Status
! Transport
|-
| KVM + QEMU (VFIO mdev)
| '''Reference'''
| VFIO mdev + eventfd IRQ + DMA interface
|-
| libvirt (mdev attach)
| '''Supported'''
| same, via libvirt XML
|-
| Zen / KVM on other device models
| '''Experimental'''
| as above
|-
| Xen (incl. Argo/OpenXT)
| '''Target'''
| Xen device model + event channels
|-
| Qubes OS / Qubes OS Air
| '''Target'''
| compartmentalized VMs; GPU per compartment
|-
| Cloud hypervisor / crosvm
| '''Experimental'''
| VFIO-based; needs testing
|}

The transport boundary — registration, guest-memory access, interrupt injection — is deliberately narrow, so a new transport is small and isolated.

== 5. What portability means today ==

<pre> host architecture GPU family driver/firmware family
┌─────────────────┐ ┌──────────────────┐ ┌────────────────────┐
│ x86_64 │ │ Turing (TU104) │ │ 570.144 (anchor) │
│ aarch64 (Grace) │ │ Ampere / Ada │ │ newer families │
│ GB10-class │ │ Hopper / datac. │ │ (gated, pending │
└────────┬────────┘ └────────┬─────────┘ │ review) │
│ │ └─────────┬──────────┘
└──────────┬──────────┴───────────────────────┘
▼
one combination = hardware profile + version profile
▼
evaluation ladder + evidence → matrix entry:
"hardware X + driver Y + firmware Z, tested at rung R on date D"</pre>

The configuration surface lives in <code>config/</code>: <code>config/hardware/<arch>.yaml</code> (PCI IDs, BARs, memory geometry, engine lists) and <code>config/versions/<driver>.yaml</code> (RPC/control inventories, capability tables).

<blockquote>'''These files are design targets, not live configuration.''' No code reads them today: selection is compile-time and the module is built for the TU104 profile. Treating the YAML surface as authoritative — or claiming "configuration-driven portability" — is aspirational until the profiles are wired into the module's load path. A profile is the ''unit of support''; what is missing is the code that reads the profile files, not the schema.</blockquote>

== 6. Adding a new combination ==

'''New GPU / hardware profile.'''

# Create a hardware profile: PCI IDs, BAR layout and sizes, memory geometry (FB size, BAR windows), engine inventory, interrupt topology, and the register values the guest driver validates during probe.
# Run the five-rung evaluation ladder ([https://git.vgpu.io/open-iov/blob/master/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>]).
# Record the differences; prefer configuration over code.
# Publish the profile, the evidence, and any workarounds.

Firmware-side analysis uses the risc-v-gsp-rm-debugger ([[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]]): first-touch probes and per-pair comparisons classify each difference as profile data or logic before any profile change lands.

'''New firmware version.''' Images are versioned under <code>firmware/<SERIES>-GSP/<VERSION>/</code> with signatures and analysis artifacts; the work is capture, inventory diff, decode against that version's reference, then configuration/handlers and a re-run of the ladder.

'''New driver version.''' Run the trace, diff the RPC/control inventory, add version-keyed handlers, re-run the ladder. Newer driver/firmware families are gated pending review before any work starts.

'''Evidence expectations.''' A combination is entered in the matrix only with the profile fields recorded; the firmware/device pairing verified by hash and version string; every first-touch difference explained as profile or logic; per-pair comparison results with session hashes; the metric table with its reference-capture method and stated fidelity limits; and a supervised physical validation for the support claim. "Tested and documented" is literal: an untested or simulator-only combination stays '''Target''' or '''Experimental'''.

== 7. Anchor versions and engine-list portability ==

The anchor is guest CUDA driver (<code>libcuda</code>) '''570.144''' with GSP firmware '''570.144'''; the host oracle is <code>libcuda</code> '''570.211.01''' with CUDA toolkit '''12.0'''. The versioned surface is the RM ioctl/class/control interface the userspace driver issues — not a stable public API; see [[CUDA Userspace Driver|<code>CUDA-Userspace-Driver.md</code>]].

The engine inventory and its class-list filtering are per-architecture, per-driver data. An earlier gap was concrete: a missing async copy engine blocked context creation until the inventory was completed — which is why it belongs in the hardware profile, validated against the host oracle.

== 8. Known portability risks ==

{| class="wikitable"
|-
! Risk
! Mitigation
|-
| Page-size/cache-attribute differences on aarch64
| keep the memory model abstract; test on Grace early
|-
| Windows driver ROM/VBIOS expectations
| explicit ROM/VBIOS provisioning in the device profile; measured bring-up
|-
| Xen interrupt/memory interfaces differ
| narrow transport interface + Xen backend plan
|-
| Vendor ROMs and laptop firmware variants (GB10/Surface)
| per-device overrides in the profile system
|-
| Driver-family drift (580 and newer)
| version-keyed handlers; matrix-driven testing
|}

== 9. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/06-portability/hardware-software-matrix.md <code>../docs/06-portability/hardware-software-matrix.md</code>] — canonical matrix: ladder, status, procedures; the validation process behind every support claim is in [https://git.vgpu.io/open-iov/blob/master/docs/06-portability/03-device-validation-and-porting-sequence.md <code>../docs/06-portability/03-device-validation-and-porting-sequence.md</code>].
* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — the five-rung evaluation ladder and evidence rules.
* [[Project Status|<code>Project-Status.md</code>]] and [[System Overview|<code>System-Overview.md</code>]] — current status and component map.
* [[risc-v-gsp-rm-debugger|<code>Riscv-GSP-RM-Debugger.md</code>]] — firmware-side portability analysis.
* [[HAAEEP and Collaboration|<code>HAAEEP-and-Collaboration.md</code>]] — the edge use case and collaboration models.

HAAEEP and Collaboration

2026-10-12T04:29:47Z

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

{{About message}}

= HAAEEP and Collaboration =

<blockquote>'''Companion summary — target architecture.''' This page condenses [https://git.vgpu.io/open-iov/blob/master/docs/08-ecosystem/haaeep-and-collaboration.md <code>../docs/08-ecosystem/haaeep-and-collaboration.md</code>], the canonical description of the '''High Assurance Agent Edge Environment Platform (HAAEEP)''': a target architecture that Open-IOV exists to enable, and the collaboration models around it. HAAEEP is a proposal and a roadmap, not a shipped product. When this page and the docs tree disagree, the docs tree wins. Front door: [https://git.vgpu.io/open-iov/blob/master/docs/README.md <code>../docs/README.md</code>].</blockquote>

Status: 2026-10-12, tree <code>master @ 2f066727</code>. Tags: <code>[proposal]</code> architecture and roadmap; <code>[target]</code> milestone to reach; <code>[goal]</code> end state; <code>[design]</code> integration point designed, not built.

== 1. What HAAEEP is ==

An AI agent harness (Claude Code, OpenCode, or any future equivalent) is an untrusted, capability-constrained workload: it runs generated code, holds credentials, and talks to a model. In a ''cloud'' deployment its requests leave the device; in a ''secure edge'' deployment — a workstation, a kiosk, a vehicle, a medical or industrial system — they must not.

HAAEEP is the platform shape for that edge deployment:

* inference runs '''locally''', in its own VM, with the GPU;
* agents run in '''their own VMs''', without the GPU and without the weights;
* other functions (desktop, networking, storage) run in '''other VMs''', each isolated;
* the whole stack is '''attestable''', so remote parties can verify it;
* and nothing in the design requires the model server, the agent, or the GPU driver to be modified.

Open-IOV is HAAEEP's GPU component. The broader motivation is in [https://git.vgpu.io/open-iov/blob/master/docs/01-orientation/02-the-ai-edge-and-ecosystem.md <code>../docs/01-orientation/02-the-ai-edge-and-ecosystem.md</code>].

== 2. Architecture ==

<pre>┌──────────────────────────────────────────────────────────────────────────┐
│ EDGE DEVICE — one GPU, local inference only │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌───────────────────┐ │
│ │ Agent VM │ │ Inference VM │ │ Infrastructure │ │
│ │ capability- │────▶│ vLLM / SGLang │ │ desktop, net, │ │
│ │ constrained │ API │ owns GPU via │ │ storage VMs │ │
│ │ no GPU │ │ Open-IOV mdev or │ │ optional GPU accel│ │
│ │ │ │ xen-pci + relay │ │ via Open-IOV │ │
│ └─────────────────┘ └────────┬─────────┘ └───────────────────┘ │
│ │ │
│ ┌────────────────────────────────┴───────────────────────────────────┐ │
│ │ Hypervisor: KVM / Xen + Argo/OpenXT / Qubes OS Air │ │
│ │ • Flask-XSM / HMX mandatory access control between VMs │ │
│ │ • TrenchBoot measured boot: firmware → hypervisor → OS │ │
│ │ • Open-IOV: per-VM virtual GPUs, IOMMU-enforced isolation │ │
│ └────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘</pre>

== 3. Roles and trust ==

{| class="wikitable"
|-
! Component
! Trust
! Why
|-
| Agent VM
| untrusted
| runs generated code; may be compromised
|-
| Inference VM
| high (holds weights)
| isolated from agents by MAC and by device virtualization
|-
| GPU-owning VM (optional)
| high
| owns the physical device; can serve others through Open-IOV
|-
| Infrastructure VMs
| medium
| GPU acceleration optional; no access to weights
|-
| Hypervisor
| TCB
| enforced by the platform (Argo/OpenXT/Qubes)
|-
| Open-IOV device
| TCB-adjacent
| must fail closed; audited (see the security doc)
|}

== 4. Data flow ==

# An agent VM issues an API call to the inference VM over an internal, MAC-controlled channel.
# The inference VM's vLLM/SGLang (unchanged) serves the request with the GPU, using its KV-cache plugins (also unchanged).
# No request ever leaves the device: the platform's network policy forbids external inference endpoints by construction.

'''Optional topology — GPU relay domain.''' Where the platform prefers it (Xen with <code>xen-pci</code>), one domain owns the physical GPU and relays acceleration to other domains through Open-IOV: the process running the model is isolated from the domain that owns the silicon, and agents are isolated from both.

== 5. Why Open-IOV fits ==

{| class="wikitable"
|-
! Alternative
! Why it is not enough
|-
| Discrete passthrough
| gives the GPU to exactly one VM; HAAEEP needs the GPU in the inference VM ''and'' optional acceleration elsewhere
|-
| Process-level sharing (host CUDA with containers)
| no VM-level isolation, which is the whole point of the platform
|-
| Proprietary vGPU
| cannot build a sovereign, auditable, redistributable edge platform on a licensed host stack you cannot inspect or modify
|-
| Software GPU emulation
| cannot run models usefully
|}

Open-IOV is the component in this design that ''must'' be open, permissively licensed and version-durable — so it is the component we are building. Its isolation model is described in [[GMMU Multi-Tenant Protection|<code>GMMU-Multi-Tenant-Protection.md</code>]].

== 6. Roadmap (proposal) ==

{| class="wikitable"
|-
! Milestone
! Depends on
|-
| CUDA execution in a guest (rung 3)
| rung 2 (runtime start) is green; CE-class work is forwarded with release and fault relay in place (default off); for GR the guest launch reaches the front-end but no grid has dispatched yet — the frontier is tracked in [[The Execution Frontier|<code>Execution-Frontier.md</code>]]
|-
| Two-VM coexistence (inference + agent) with isolation
| Open-IOV multi-tenancy + platform MAC
|-
| Xen/OpenXT or Qubes Air variant
| Open-IOV transport backends
|-
| Attested boot-to-inference chain
| TrenchBoot integration
|-
| Published reference platform (scripts, docs)
| all of the above
|}

Each milestone is documented as it lands, with the same evidence discipline as the rest of the project. Nothing in this table is claimed today: it is the proposal.

== 7. Collaboration models ==

'''Open source projects.''' Adopt the code under BSD-3-Clause (kernel modules are GPL-2.0 — see [https://git.vgpu.io/open-iov/blob/master/docs/09-reference/legal-and-licensing.md <code>../docs/09-reference/legal-and-licensing.md</code>]). We treat interface requirements as design inputs (transports, IRQ models, memory interfaces), provide the transport abstraction rather than asking for forks, and test against partner platforms in the matrix.

'''Commercial partners.''' Welcome, with no license fee: ship Open-IOV in products and differentiate above it; sponsor support for specific hardware/driver versions; co-develop under the same permissive terms.

'''Hardware vendors.''' Nothing is required. The goal is that new hardware works through configuration and measurement; if you want to accelerate support, share hardware and traces, and we publish what we learn.

'''Security researchers.''' The emulation's attack surface is small and documented. Break it. Findings and fixes are published, with credit.

== 8. Cross-project notes ==

{| class="wikitable"
|-
! Project
! Relationship
|-
| '''Qubes OS Air'''
| compartmentalization on modern hardware; Open-IOV makes GPU acceleration usable per compartment (especially AI workloads)
|-
| '''TrenchBoot'''
| measured/attested boot; combined, the platform can attest the inference path
|-
| '''Argo / OpenXT'''
| hypervisor-mediated exchange (HMX) and Flask-XSM MAC; Open-IOV respects and complements that policy
|-
| '''KVM/VFIO gaming'''
| established the fast-passthrough pattern; Open-IOV serves users who need ''multiple'' VMs or cannot dedicate the GPU
|-
| '''vLLM / SGLang'''
| remain the inference answer; Open-IOV isolates them without changing them; KV-cache plugins keep working
|}

== 9. How to help ==

* '''Platform builders''': bring your hypervisor; tell us the transport you need.
* '''Security engineers''': review the isolation model and the debug surface.
* '''AI-infrastructure engineers''': define the reference HAAEEP deployment with us (model server, KV cache, agent harness).
* '''Everyone''': run the evaluation ladder on your hardware and send the evidence.

Contact and participation channels are in [https://git.vgpu.io/open-iov/blob/master/docs/README.md <code>../docs/README.md</code>] §6 "How to get involved". We would rather have a hundred partners than a hundred forks.

== 10. Related documentation ==

* [https://git.vgpu.io/open-iov/blob/master/docs/08-ecosystem/haaeep-and-collaboration.md <code>../docs/08-ecosystem/haaeep-and-collaboration.md</code>] — canonical HAAEEP architecture, roadmap and collaboration models.
* [https://git.vgpu.io/open-iov/blob/master/docs/01-orientation/02-the-ai-edge-and-ecosystem.md <code>../docs/01-orientation/02-the-ai-edge-and-ecosystem.md</code>] — the AI-edge motivation and ecosystem stance.
* [https://git.vgpu.io/open-iov/blob/master/docs/05-operations/01-performance-correctness-security.md <code>../docs/05-operations/01-performance-correctness-security.md</code>] — the evaluation ladder behind every milestone.
* [[Project Status|<code>Project-Status.md</code>]] and [[System Overview|<code>System-Overview.md</code>]] — current status and component map.
* [[The Execution Frontier|<code>Execution-Frontier.md</code>]] and [[Physical GPU Backend and Guest Wiring|<code>Physical-GPU-Backend.md</code>]] — the GR frontier and how work reaches real hardware.
* [[Hardware and Software Matrix|<code>Hardware-Software-Matrix.md</code>]] — what runs where, and how support is added.