View source for Mdev-GPU

This page makes use of terms that are defined in the [https://openmdev.io/index.php/Glossary ''OpenMdev Glossary'']. If you're curious what a given term means you can check there for a definition.



This page will be used to document the '''Mdev-GPU API''' as well as provide an overview of the internals of the Mdev-GPU package.

Mdev-GPU's source code can be found [https://github.com/Arc-Compute/Mdev-GPU here].

Module docs can be found [https://docs.linux-gvm.org/mdev-gpu/ here].<blockquote>An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This [https://creativecommons.org/licenses/by/4.0/ CC-BY-4.0] licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.</blockquote>

=== Compilation ===
Compiling Mdev-GPU from source is very simple however it can take a long time to complete (10 minutes for existing sources).

If you'd like to compile Mdev-GPU yourself you can do so by cloning [https://github.com/Arc-Compute/Mdev-GPU the repo] and running the following two commands from within the repository directory:

<code>curl -sSL <nowiki>https://get.haskellstack.org/</nowiki> | sh</code>

<code>stack install</code>

These commands will generate all binaries provided by Mdev-GPU under the path <code>~/.local/bin/</code>.

If you would prefer not to wait for the program to compile you can find pre-compiled x86_64 binaries under the [https://github.com/Arc-Compute/Mdev-GPU/releases/tag/0.1.0.0 Releases section of the repository on GitHub].

=== Installation ===
The binaries compiled on your local machine by Stack in the <code>~/.local/bin/</code> path or downloaded from [https://github.com/Arc-Compute/Mdev-GPU/releases Releases on GitHub] should be copied to <code>/usr/bin/</code>. This can be accomplished both via install script (currently testing in LibVF.IO) or via management layer-agnostic package.

=== Mdev-GPU Configuration ===
The Mdev-GPU repository provides Systemd service and example configurations for Mdev-CLI in the [https://github.com/Arc-Compute/Mdev-GPU/tree/master/etc /etc/ folder]. On installation the directories & files contained in the repository's /etc/ folder should be copied to the absolute path <code>/etc/</code> on the user's computer.

==== Systemd ====
GVM/Mdev-GPU exposes several of it's functions as Systemd services.

As of the 0.1.0.0 release the following Systemd services are available:

* '''mdev-post.service'''

===== mdev-post.service =====
As the name would suggest '''[https://github.com/Arc-Compute/Mdev-GPU/blob/master/etc/systemd/system/mdev-post.service mdev-post.service]''' posts the GPU(s) in a state where they are ready to be virtualized. When the service is invoked the mdev-cli util registers Mdev callbacks and supported mediated device (Mdev) types with the Mediated Core according to the user's specified configuration at '''<code>/etc/gvm/mdev-gpu/generate-vgpu-types.yaml</code>'''. If you'd like to place your configuration file elsewhere and still have your configuration invoked on boot you can edit the systemd service [https://github.com/Arc-Compute/Mdev-GPU/blob/9993fd0781ddddfcca2cb9bee6455ea2c024cc23/etc/systemd/system/mdev-post.service#L10 ExecStart= line] to specify an alternative path.

==== Mdev-CLI ====
Configuration for Mdev-GPU can be found in the path '''<code>/etc/gvm/mdev-gpu/</code>'''. This path will by default contain several vendor-specific folders which may include configuration examples for various devices/architectures as well as the file <code>'''generate-vgpu-types.yaml'''</code> which you can configure to specify your desired Virtual GPU (vGPU) / VFIO-Mdev topologies as they are presented in the sysfsdev mdev_bus path and as seen by tools such as [https://github.com/mdevctl/mdevctl mdevctl]. Once you have configured the available types your liking you can refresh the supported types registered with the [https://openmdev.io/index.php/Virtual_IO_Internals#Mediated_Core Mediated Core] by rebooting your machine or reloading the vendor driver kernel module then restarting '''mdev-post.service''' or by invoking '''mdev-cli'''.


The available yaml properties which may be specified in '''<code>/etc/gvm/mdev-gpu/generate-vgpu-types.yaml</code>''' are shown below. Note that fields with the <sup>[required]</sup> tag must be specified in the yaml file whereas fields with the <sup>[optional]</sup> tag may be omitted from configuration if you would prefer to rely upon default parameters instead.

===== num =====
<sup>[required]</sup>

num refers to the Mediated Device (Mdev) type number. This value specifies the profile's numerical identifier in the sysfsdev mdev_bus path.

===== vDevId =====
<sup>[optional]</sup>

The vGPU (Virtual GPU) PCI Device ID which should be presented to the guest.

===== pDevId =====
<sup>[optional]</sup>

The pGPU (Physical GPU) PCI Device ID which should be presented to the host.

===== name =====
<sup>[optional]</sup>

The GPU name presented to the guest.

===== gpuClass =====
<sup>[optional]</sup>

The class of GPU to use.

available strings:

"Compute"

"Graphics"

===== maxInstances =====
<sup>[required]</sup>

maxInstances refers to the maximum number of instances available of any given Mediated Device (Mdev) profile. This number can be configured arbitrarily however it should ideally not be set to a value which allows for fbLen/fbRes values that allow a number of Mdev devices to be created which would exceed your device's available VRAM.

===== virtDisplay =====
<sup>[optional]</sup>

virtDisplay controls the topologies of virtual displays. virtDisplay exposes the following user configurable properties:

====== numHeads ======
<sup>[optional]</sup>

Number of heads [virtual monitors] available for each Mediated Device (Mdev).

====== maxResX ======
<sup>[optional]</sup>

Maximum width of the mediated display.

====== maxResY ======
<sup>[optional]</sup>

Maximum height of the mediated display.

====== frlConfig ======
<sup>[optional]</sup>

Frame rate limiter for the mediated display.

====== frlEnable ======
<sup>[optional]</sup>

Do we enable frame rate limiting or not?

===== ecc =====
<sup>[optional]</sup>

ecc refers to if error correcting memory should be used.

===== fbLen =====
<sup>[required]</sup>

fbLen is the framebuffer length (in megabytes).

===== fbRes =====
<sup>[required]</sup>

fbRes is the framebuffer reserved by the hypervisor (in megabytes). In most circumstances this value should scale to around 1/7th the size of your chosen fbLen value.

===== mapVideoSize =====
<sup>[optional]</sup>

Mappable video size in MBs, 24 by default.

===== encCap =====
<sup>[optional]</sup>

Percentage we use the NvEnc hardware.

===== bar1Len =====
<sup>[optional]</sup>

bar1Len refers to the size of the Base Address Registers for BAR1 (MSI) as presented by the [https://openmdev.io/index.php/Virtual_I/O_Internals#VFIO_device_file_descriptor VFIO File Descriptor].