BAR Emulation Specification

From Open-IOV
Revision as of 04:40, 12 October 2026 by WikiBot (talk | contribs) (Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

BAR Emulation Specification (GSP 570.144W)

Historical document — superseded. The BAR_TRAP RPC (type 0x0A) described here is not used: in emulation mode it returns -ENOTSUPP, and BAR access is synchronous MMIO through the BAR shadow. Guest-visible BAR windows are BAR0 (16 MB register MMIO), BAR1 (256 MB VRAM aperture) and BAR3 (instance window). Canonical documentation: ../docs/03-architecture/06-host-module-and-hybrid-model.md.

Overview

The GSP firmware implements BAR (Base Address Register) emulation for virtual GPU (vGPU) through the BAR_TRAP RPC (type 0x0A). The main handler at 0x04CB8188 (40 KB) manages BAR1 aperture, forced access, and memory backing. This RPC is not used by the live path: in emulation mode BAR_TRAP returns -ENOTSUPP; guest BAR access is handled synchronously as MMIO through the BAR shadow (../kernel/bar_emulation.c), not via an RPC.

Key Addresses

Component Address Description
BAR Trap Handler 0x04CB8188 Main emulation logic (40 KB)
RPC Jump Table Entry 0x057675B7 BAR_TRAP in secondary dispatcher
String: BAR1 0x0406570E, 0x04085DE0 BAR1 identifier
String: Bar1Rest 0x040656DA BAR1 restore state
String: ForceBarAccess 0x040656F2 Forced access control
String: Bar1Aperture 0x04065722 Aperture configuration
Memory Allocator 0x429B450 Arena for BAR backing store
Cache Management 0x429B428 Flush/invalidate

Handler State Machine

Main Loop (0x04CB8188)

The handler runs a continuous event loop processing BAR trap requests: 

while (true) {
    // Wait for trap event (bitmask in unaff_s5)
    event = wait_for_bar_event();
    
    switch (event & 0x3F) {
        case 0x01:  // BAR1 write access
            handle_bar1_write();
            break;
        case 0x02:  // BAR1 read access
            handle_bar1_read();
            break;
        case 0x08:  // Aperture configuration
            handle_aperture_config();
            break;
        case 0x10:  // ForceBarAccess
            handle_force_access();
            break;
        case 0x20:  // Bar1Rest (restore)
            handle_bar1_restore();
            break;
        default:
            return ERROR_UNKNOWN_EVENT;
    }
}

Event Bitmask

Bit Event Description
0 BAR1_WRITE Guest VF wrote to BAR1
1 BAR1_READ Guest VF read from BAR1
3 APERTURE Aperture size/offset change
4 FORCE_ACCESS ForceBarAccess triggered
5 BAR1_REST Restore BAR1 state

Memory Management

Allocator Arena (0x429B450)

// Allocate backing store for BAR emulation
void *bar_alloc(size_t size) {
    // Calls UNK_ram_058A1A9C(0x429B450, size)
    // Returns pointer in 0x429B000 region
}

// Free backing store
void bar_free(void *ptr) {
    // Calls UNK_ram_05895BF4(0x429B428, ptr)
}

Allocation Sizes

  • Standard BAR1 page: 0x1000 (4 KB)
  • Aperture buffer: 0x40000 (256 KB) max
  • Force access buffer: 0x1000 (4 KB)

Cache Coherency

// Flush cache for DMA
void bar_cache_flush(void *addr, size_t size) {
    UNK_ram_05895BF4(0x429B428, addr);
    UNK_ram_05895BF4(0x429B428, addr + size);
}

BAR1 Emulation Logic

Write Path (Event 0x01)

int handle_bar1_write(bar_trap_t *trap) {
    // 1. Validate offset within aperture
    if (trap->offset >= bar1_aperture_size)
        return ERROR_OVERFLOW;
    
    // 2. Check ForceBarAccess flag
    if (force_bar_access && !(trap->flags & FORCE_ALLOWED))
        return ERROR_PERMISSION;
    
    // 3. Translate to backing store
    uint8_t *backing = bar1_backing + trap->offset;
    
    // 4. Perform write (1/2/4/8 bytes)
    switch (trap->size) {
        case 1: *backing = trap->value; break;
        case 2: *(uint16_t*)backing = trap->value; break;
        case 4: *(uint32_t*)backing = trap->value; break;
        case 8: *(uint64_t*)backing = trap->value; break;
    }
    
    // 5. Flush to device if needed
    if (device_needs_flush)
        bar_cache_flush(backing, trap->size);
    
    return OK;
}

Read Path (Event 0x02)

int handle_bar1_read(bar_trap_t *trap) {
    if (trap->offset >= bar1_aperture_size)
        return ERROR_OVERFLOW;
    
    uint8_t *backing = bar1_backing + trap->offset;
    
    switch (trap->size) {
        case 1: *trap->out_value = *backing; break;
        case 2: *trap->out_value = *(uint16_t*)backing; break;
        case 4: *trap->out_value = *(uint32_t*)backing; break;
        case 8: *trap->out_value = *(uint64_t*)backing; break;
    }
    
    return OK;
}

Aperture Configuration (Event 0x08)

int handle_aperture_config(bar_config_t *cfg) {
    // Validate new aperture
    if (cfg->size > MAX_APERTURE_SIZE)  // 0x40000
        return ERROR_INVALID;
    
    if (cfg->offset & 0xFFF)  // Must be 4KB aligned
        return ERROR_ALIGNMENT;
    
    // Reallocate backing store if size changed
    if (cfg->size != bar1_aperture_size) {
        void *new_backing = bar_alloc(cfg->size);
        if (!new_backing) return ERROR_NOMEM;
        
        // Copy existing data
        memcpy(new_backing, bar1_backing, 
               min(cfg->size, bar1_aperture_size));
        
        bar_free(bar1_backing);
        bar1_backing = new_backing;
        bar1_aperture_size = cfg->size;
    }
    
    bar1_aperture_offset = cfg->offset;
    return OK;
}

ForceBarAccess (Event 0x10)

int handle_force_access(force_access_t *fa) {
    // ForceBarAccess allows hypervisor to bypass VF permissions
    // Used for live migration, checkpoint/restore
    
    force_bar_access = fa->enable;
    force_access_vf = fa->vf_id;
    force_access_range = fa->range;
    
    // Allocate shadow buffer for forced operations
    if (fa->enable) {
        force_buffer = bar_alloc(0x1000);
        if (!force_buffer) return ERROR_NOMEM;
    } else {
        bar_free(force_buffer);
        force_buffer = NULL;
    }
    
    return OK;
}

Bar1Rest - State Restore (Event 0x20)

int handle_bar1_restore(restore_state_t *rs) {
    // Restore BAR1 state from checkpoint
    // Used during VF live migration
    
    bar1_aperture_offset = rs->aperture_offset;
    bar1_aperture_size = rs->aperture_size;
    force_bar_access = rs->force_access;
    
    // Restore backing store from serialized data
    if (rs->backing_size > 0) {
        void *new_backing = bar_alloc(rs->backing_size);
        if (!new_backing) return ERROR_NOMEM;
        
        memcpy(new_backing, rs->backing_data, rs->backing_size);
        bar_free(bar1_backing);
        bar1_backing = new_backing;
    }
    
    return OK;
}

vGPU-IO Integration

Kernel BAR Trap Registration

struct gsp_bar_ops {
    int (*bar_trap)(struct gsp_device *gsp, 
                    struct bar_trap_msg *msg);
    int (*configure_aperture)(struct gsp_device *gsp,
                              uint32_t vf_id,
                              uint64_t offset, uint64_t size);
    int (*force_access)(struct gsp_device *gsp,
                        uint32_t vf_id, bool enable);
    int (*save_state)(struct gsp_device *gsp,
                      uint32_t vf_id, void *buf, size_t *size);
    int (*restore_state)(struct gsp_device *gsp,
                         uint32_t vf_id, void *buf, size_t size);
};

static const struct gsp_bar_ops gsp_bar_ops = {
    .bar_trap = gsp_bar_trap_handler,
    .configure_aperture = gsp_bar_config_aperture,
    .force_access = gsp_bar_force_access,
    .save_state = gsp_bar_save_state,
    .restore_state = gsp_bar_restore_state,
};

VF BAR Configuration

int gsp_vf_bar_init(struct gsp_vf *vf) {
    struct bar_config cfg = {
        .vf_id = vf->id,
        .bar = 1,  // BAR1
        .aperture_offset = vf->bar1_offset,
        .aperture_size = vf->bar1_size,
        .flags = BAR_FLAG_COHERENT,
    };
    
    return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP, 
                        BAR_SUBCMD_CONFIG, &cfg, sizeof(cfg));
}

Live Migration Support

// Save BAR state for migration
int gsp_vf_bar_save(struct gsp_vf *vf, void *buf, size_t *size) {
    struct bar_save_msg msg = {
        .vf_id = vf->id,
        .bar = 1,
    };
    
    return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP,
                        BAR_SUBCMD_SAVE, &msg, sizeof(msg),
                        buf, size);
}

// Restore BAR state on destination
int gsp_vf_bar_restore(struct gsp_vf *vf, void *buf, size_t size) {
    struct bar_restore_msg msg = {
        .vf_id = vf->id,
        .bar = 1,
        .data = buf,
        .data_size = size,
    };
    
    return gsp_rpc_send(vf->gsp, RPC_BAR_TRAP,
                        BAR_SUBCMD_RESTORE, &msg, sizeof(msg));
}

Error Handling

Error Code Recovery
OOM (0x51) Allocator exhausted Shrink aperture, retry
Timeout (0x1E) Cache flush timeout Reset BAR, reinitialize
Invalid (0x1F) Bad aperture config Validate alignment/size
Permission (0xFFFF) ForceAccess violation Check VF permissions
Overflow (0x25) Offset beyond aperture Bounds check in VF driver

Debugging

Key Registers to Monitor

0x04296048  : Global error flag (cRam0000000004296048)
0x0429B750  : Error status (0x111300 offset)
0x429B428   : Cache management base
0x429B450   : Allocator arena base

Log Strings

Search for these in GSP logs:

  • BAR1 - BAR1 access
  • Bar1Rest - State restore
  • ForceBarAccess - Forced access
  • Bar1Aperture - Aperture changes
  • Bar1Size - Size queries

Reference

  • Handler: FUN_ram_04cb8188_ram_04cb8188.c (40 KB)
  • RPC Type: 0x0A (BAR_TRAP)
  • Jump Table Entry: 0x057675B7
Retrieved from "https://open-iov.org/index.php?title=BAR_Emulation_Specification&oldid=23962"