Virtual I/O Internals

The following document will attempt to detail the internals of a Virtual Function IO (VFIO) driven Mediated Device (Mdev).

Comparison of Approaches

RPC Mode	SR-IOV Mode
Host requires insight about guest of workload.	Host ignorance of guest workload.
Error reporting.	No guest driver error reporting.
In depth dynamic monitoring.	Basic dynamic monitoring.
Software defined MMU guest separation.	Firmware defined MMU guest separation.
Requires deferred instructions to be supported by host software (support libraries).	Guest is ignorant of host supported software such as support libraries.

RPC Mode

Instruction Execution

RPC Mode moves instruction information across a virtual function interface (VF) using Remote Procedure Calls generally by way of soft interrupt (IOCTLs). Guest GPU instructions passed from the guest as Remote Procedure Calls are Just-in-time recompiled on the host for execution by a device driver.

Memory Management

Guests may be presented with emulated memory regions which use indirect emulated communication requiring a VM-exit (slow) or instead the guest may be presented with passthrough memory regions which use direct communication requiring no VM-exit (fast).

Scheduling

Scheduling

RPC Mode Requirements:

Sensitive Instruction List.

Instruction Shim/Binary Translator.

HPA<->GPA Boundary Enforcement.

SR-IOV Mode

Instruction Execution

SR-IOV Mode involves the communication of instructions from a virtual function (VF) through direct communication to the PCI BAR.

Memory Management

Guests are presenting with passthrough memory regions by the device firmware.

Scheduling

RPC Mode Requirements:

Device SR-IOV support.

HPA<->GPA Boundary Enforcement.

Both Modes

VFIO file descriptor

VFIO devices are mapped based on a file offset representing the virtual PCI device.

BAR regions in a VGA PCI device.

00:00.0 VGA compatible controller
Region 0 Bar0 (starts at offset 0)
Region 1 Bar1
Region 2 Bar2
Region 3 Bar3
Region 4 Bar4
Region 5 Bar5 (IO port space)
Expansion ROM

VFIO representation of PCI BAR regions offsets.

<- File Offset ->
0 -> A	A -> A+B	A+B -> A+B+C	A+B+C -> A+B+C+D	...
Region 0 (A)	Region 1 (B)	Region 2 (C)	Region 3 (D)	...