Introduction
This page will provide introductory context on where to begin reading. It will aim to give you a sense of primitives or foundational concepts used for GPU Virtualization.
Terminology
To get started with basic terminology around IO Virtualization read the Glossary page. This will get you familiar with a few terms and concepts used in IO Virtualization documentation.
Primitives
The primitives section will cover a few notes on the basics of virtualizing a GPU or IO device.
PCI BAR
PCI-e devices such as GPUs are used for input and output to and from a computer system. This input and output occurs using TLPs or Transaction Layer Packets over the PCI-e packet network. The IO device is presented to the host operating system as BARs or Base Address Registers.
GPU Drivers
GPUs need software to communicate from a host computer system to the device - this software is called a GPU Driver and it typically interacts with the device using MMIO or Memory Mapped IO. MMIO is a set of operations used to allow normal CPU instructions to address registers in RAM in order to communicate with the PCI-e device. In the context of virtualization these registers must be presented into the device in a way that the guest operating system expects. Typically this is accomplished using VFIO or Virtual Function IO. VFIO is used for many methods of virtualizing the device including discrete assignment of a physical device, or virtual (partitioned) device assignment created with or without hardware assistance.
IOMMU
MMU or Memory Management Units handle hardware enforced access to memory paging in computer systems. In the context of virtualized MMIO (Memory Mapped Input Output) devices an MMU is required to hardware enforce access to virtual devices (a Memory Management Unit for Input Output is known as an IOMMU). In the case of VFIO, groups of IOMMU devices (IOMMU Groups) are used to control access to the device. These IOMMU Groups often represent individual devices or sub-devices (in the context of Unassisted Virtualization (VFIO-Mdev), Hardware-Assisted Virtualization (SR-IOV), or Hardware-Assisted Paravirtualization) which may be bound to a VFIO bus driver (such as vfio-pci) which results in a VFIO Group (unit of ownership).
Virtualization vs. Paravirtualization
Virtualization provides support for unmodified guest software or drivers in the context of GPUs.
Paravirtualization provides support for "enlightened" guests or modified software which is aware of virtualization.
Unassisted vs. Assisted
Similar to virtualization on CPUs with tools like Xen, IO devices can be virtualized on different kinds of computer hardware. Some devices provide hardware virtualization assistance (such as HVM mode in Xen) and some do not (such as PV mode in Xen), as a result an appropriate mode should be used depending on the availability of assistance.
Further Reading
This section will cover what you should read to obtain more information about virtualization.
Virtualization Fundamentals
The Virtualization Fundamentals article provides links to some of the foundational work which underly modern virtualization industry and development today. If you'd like to understand both the theories on which virtualization builds as well as the history of industry you should read several articles here. Notable Popek and Goldberg Virtualization Requirements as well as Xen and the Art of Virtualization provide a great foundations-first approach to learning about virtualization.
Virtual I/O Internals
The Virtual I/O Internals article provides a detailed look into the interface layers between IO driver internals, the hypervisor, and the guest OS. This is a great article to read if you'd like to better understand how virtual machines can interact with both discretely assigned devices as well as those which have been partitioned using software or firmware.
GPU Driver Internals
The GPU Driver Internals article provides information on how GPU drivers are structured internally as well as how they achieve virtualization. This includes high level architecture, device initialization, as well as memory mapping, and scheduling.