Open-IOV - User contributions [en]

OpenRM

2022-06-14T20:13:47Z

Temple-os:

OpenRM refers to the open sourcing of the NVIDIA kernel module for Linux, specifically the RM Core or the internal resource manager.

Through the open sourced code ([https://github.com/NVIDIA/open-gpu-kernel-modules located here]) we have been able to deduce 7 top level APIs that are being exposed to the Linux developers.

== Interaction With OpenRM ==
All of OpenRM is a high scale object oriented abstraction layer for the GPU, all resources are internally classes which have control definitions (methods) on how you can use them. These classes can also represent physical resources or engines.

=== Alloc (Client) ===
This is analogous to calling an object constructor in a higher level programming language, such as C#.

The data for this call requires the following components:

# hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
# hObjectParent - Parent of the resource we are allocating.
# hObjectNew - New object to allocate.
# hClass - Class of the object we are dealing with.
# pAllocParams - Pointer to values passed into the constructor call.
# status - Status of the constructor call (did it succeed or fail)

=== Alloc Memory 64 ===
This creates a CPU's mapping to a resource.

The data for this call requires the following components:

# hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
# hObjectParent - Parent of the resource we create the CPU mapping to.
# hObjectNew - Mapping object resource id.
# hClass - Class of the mapped object.
# flags - Flags to use on the mapped object.
# pMemory - pointer for the memory to map.
# pLimit - Edge of memory map.
# status - Status of the constructor call.

NOTE: Seems to be deprecated.

=== Control ===
This is a control command which is analogous to calling a method on a class.

The data for this call requires the following components:

# hClient - RM Client Id.
# hObject - Object id.
# cmd - Method to call.
# params - Parameters into this method.
# paramsSize - Size of the params.
# status - Status of the control call.

NOTE: The control will overwrite the object we pass in over the params pointer.

=== Dup Object ===
Duplicates the object, or clones the object, which is a deep copy. This procedure can be used across processes in order to produce shared memory.

The data for this call requires the following components:

# hClient - RM Client Id.
# hParent - Parent of the object.
# hObject - Object destination for duplicating.
# hClientSrc - Client source id.
# hObjectSrc - Object source.
# flags - Flags to use.
# status - Status of the duplication.

=== Free ===
Frees an object from the object tree.

The data for this call requires the following components:

# hRoot - Client Id.
# hObjectParent - Parent of the object.
# hObjectOld - Object to call the destructor on.
# status - Status of the free.

=== Vid Heap Control ===
TODO: Figure out what this does.

NOTE: Seems to be deprecated.

=== Map Memory ===
This seems to be the same as alloc memory but seems to be used more often.

The data for this call requires the following components:

# hClient - Client Id.
# hDevice - Device id.
# hMemory - Memory resource.
# offset - Offset of the memory.
# length - Length of the memory.
# pLinearAddress - Pointer for returned address.
# status - Status of the map memory.
# flags - Flags to use.

=== Unmap Memory ===
This unmaps the CPU memory mapping from the resource.

The data for this call requires the following components:

# hClient - Client Id.
# hDevice - Device Id.
# hMemory - Memory Id.
# pLinearAddress - pointer to virtual address.
# status - Status of the unmap command.
# flags - Flags to use.

=== Alloc Context DMA 2 ===
Allocates a context for DMA.

The data for this call requires the following components:

# hObjectParent - Object parent.
# hSubDevice - Sub device to allocate on.
# hObjectNew - Context id.
# hClass - Type of class.
# flags - Flags to use.
# selector - UNKNOWN.
# hMemory - Handle for Memory resource to DMA map.
# offset - Offset for the DMA mapping.
# limit - Limit of the DMA map.
# status - Status of the allocation.

NOTE: Seems to be deprecated.

=== Map Memory DMA ===
Map a region of memory for DMA access.

The data for this call requires the following components:

# hClient - Client Id.
# hDevice - Device Id.
# hDma - DMA Id.
# hMemory - Memory Id.
# offset - Offset of the region.
# limit - Limit of the region.
# flags - flags to use.
# dmaOffset - Offset of the mapping.
# status - Status of the map Memory DMA.

=== Un-map Memory DMA ===
Unmaps a region of memory for DMA access.

The data for this call requires the following components:

# hClient - Client Id.
# hDevice - Device Id.
# hDma - DMA handle.
# hMemory - Memory handle.
# flags - Flags to use.
# dmaOffset - DMA Offset to unmap.
# status - Status of the unmap Memory DMA.

=== Bind Context DMA ===
Binds a context for DMA interaction to a channel.

The data for this call requires the following components:

# hClient - Client Id.
# hChannel - Channel Id.
# hCtxDma - Context for DMA Id.
# status - Status of the bind context DMA.

NOTE: Seems to be deprecated.

=== Share ===
Sharing of a resource between multiple clients.

TODO: Understand this sharing more.

The data for this call requires the following components:

# hClient - Client Id.
# hObject - Resource Id.
# sharePolicy - Share policy.
# status - Status of sharing.

=== Add VBlank Callback ===
TODO: Figure out what this does.

NOTE: Seems to be deprecated.

== Engines ==
There are engines which internally act as classes inside the OpenRM driver.

These classes have a few key components that make them unique.

=== Table of Engines ===
{| class="wikitable sortable mw-collapsible"
|+Engines
!Engine
!Description
|-
|Invalid
|Does not exist.
|-
|Software
|Abstraction layer for software processes inside the OS Software.
|-
|GPU
|Interaction layer with the GPU.
|-
|Falcon
|Interaction with the Falcon on the GPU.
|-
|Master Control
|Unknown.
|-
|Kernel Master Control
|Interaction with the PMC.[https://github.com/NVIDIA/open-gpu-kernel-modules/blob/d8f3bcff924776518f1e63286537c3cf365289ac/src/nvidia/generated/g_kernel_mc_nvoc.h]
|-
|Private Ring
|Interaction with a private messaging ring.
|-
|Software Interrupt
|Interaction with software interrupts.
|-
|Memory System
|Interaction with the memory system.
|-
|Kernel Memory System
|Interaction with the kernel memory system.
|-
|Memory Manager
|Interaction with the memory management system.
|-
|FBFlcn
|Interaction with the FBFlcn subsystem.
|-
|TMR
|Interaction with the TMR on specific chips.
|-
|DMA
|Interaction with a GPU's DMA access.
|-
|Kernel FIFO
|First in First out queue in the GPU driver.
|-
|FIFO
|Interaction with a generic FIFO.
|-
|OS
|Interaction with the OS.
|-
|BUS
|PCI Bus interaction
|-
|Kernel BUS
|Internal module BUS.
|-
|INFOROM
|Interaction with the INFOROM Region on the GPU.
|-
|PERF
|Unknown.
|-
|Kernel PERF
|Unknown.
|-
|BIF
|Unknown.
|-
|Kernel BIF
|Unknown.
|-
|SF
|Unknown.
|-
|GPIO
|Interaction with GPIO.
|-
|CLK
|Clock manager system.
|-
|Kernel Display
|Kernel Display system.
|-
|DISP
|Display system.
|-
|DPU
|Data Processing Unit.
|-
|
|
|-
|
|
|}

OpenRM

2022-06-14T19:11:01Z

Temple-os:

OpenRM

2022-06-14T19:08:08Z

Temple-os:

OpenRM

2022-06-13T21:03:46Z

Temple-os: Documented OpenRM Kernel API.

Articles

2022-06-13T14:21:42Z

Temple-os: Added libRM link under projects

===Abstract===
[[Introductory Concepts & Definitions]]

[[Virtualization Fundamentals]]

[[Mediated Device Internals]]

[[Merged Drivers]]

=== Design Documents ===
[https://projectacrn.github.io/latest/developer-guides/hld/hv-memmgt.html Memory Management High Level Design (ARCN)]

===Projects===
[https://openmdev.io/index.php/LibVF.IO LibVF.IO]

[[arcd-gpud]]

[[Hyperborea]]

[https://openmdev.io/index.php/Looking_Glass_KVMFR Looking Glass]

[https://openxt.atlassian.net/wiki/spaces/OD/pages/10747915/What+is+OpenXT OpenXT]

[https://gitlab.com/vglass OpenXT: vGlass]

[https://github.com/OpenXT/surfman OpenXT: Surfman (legacy DRM)]

[https://www.bromium.com/opensource/ Bromium/uXen]

[https://xenproject.org/help/documentation/ Xen Project]

[https://www.qubes-os.org/doc/ Qubes OS]

[https://projectacrn.github.io/2.1/tutorials/using_celadon_as_uos.html Intel Celadon]

[https://openmdev.io/index.php/VGPU_Unlock vGPU_Unlock]

[[LibRM]]
===Hardware Support===
[https://openmdev.io/index.php/GPU_Support GPU Support]

[https://openmdev.io/index.php/CPU_Support CPU Support]

=== Software Support ===
[https://openmdev.io/index.php/Hypervisor_Support Hypervisor Support]

=== API Documentation ===

==== Kernel APIs ====
[https://elixir.bootlin.com/linux/latest/source/Documentation/driver-api/vfio.rst VFIO] - [https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/vfio.h vfio.h]

[https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api/vfio-mediated-device.rst?h=driver-core-next&id=7de3697e9cbd4bd3d62bafa249d57990e1b8f294 VFIO Mediated Device] - [https://elixir.bootlin.com/linux/latest/source/include/linux/mdev.h mdev.h] - [https://elixir.bootlin.com/linux/latest/source/samples/vfio-mdev/mtty.c mtty.c]

[https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git/tree/Documentation/driver-api Kernel.org Driver Core Documentation]

[https://docs.microsoft.com/en-us/windows-hardware/drivers/display/iommu-based-gpu-isolation NT Kernel (Windows) IOMMU-based GPU Isolation]

==== Driver APIs ====
[https://01.org/linuxgraphics/gfx-docs/drm/gpu/i915.html i915 Driver API]

[https://projectacrn.github.io/2.1/api/GVT-g_api.html Intel GVT-g APIs]

[https://nouveau.freedesktop.org/Development.html Nouveau Tools & APIs]

[https://openmdev.io/index.php/OpenRM OpenRM Driver API]

[https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html AMDGPU Driver API]

==== Virtualization APIs ====
[https://qemu-project.gitlab.io/qemu/interop/qemu-qmp-ref.html QEMU Machine Protocol (QMP) Reference Manual]

[https://projectacrn.github.io/2.1/developer-guides/hld/ivshmem-hld.html Inter-VM Shared Memory (IVSHMEM)]
===User Guides===
[https://arccompute.com/blog/libvfio-commodity-gpu-multiplexing/ LibVF.IO Setup Guide]

[https://looking-glass.io/docs/stable/ Looking Glass Quickstart Guide]

[https://github.com/intel/gvt-linux/wiki/GVTg_Setup_Guide Intel GVT-g Setup Guide]

[https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization/tree/master/docs AMD GPU-IOV Module Docs]

[https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF PCI passthrough via OVMF]

[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/index RedHat Virtualization Guide]

=== Developer Guides ===
[https://rayanfam.com/tags/hypervisor/ Hypervisor From Scratch]

[https://lwn.net/Kernel/LDD3/ Linux Device Drivers (3rd Edition)]

[https://dri.freedesktop.org/docs/drm/gpu/ GPU Driver Developer's Guide]

[https://dri.freedesktop.org/docs/drm/PCI/pci.html# How To Write PCI Drivers]

[https://doc.dpdk.org/guides-16.04/prog_guide/ivshmem_lib.html Data Plane Development Kit: IVSHMEM Programming Guide]
=== Communities & Mailing Lists ===
[https://discord.gg/Rb9K9DYxKK LibVF.IO Discord]

[https://lists.freedesktop.org/mailman/listinfo/intel-gfx Intel-gfx Mailing List]

[https://lists.freedesktop.org/mailman/listinfo/nouveau Nouveau Mailing List]

[https://lists.freedesktop.org/mailman/listinfo/amd-gfx AMD-gfx Mailing List]

[https://listman.redhat.com/mailman/listinfo/vfio-users VFIO-users Mailing List]

[https://forum.level1techs.com/c/software/vfio/132 <nowiki>Level1Techs Forum [VFIO Topic]</nowiki>]

[https://old.reddit.com/r/VFIO/ VFIO Subreddit]

File:Import Graph For LibVF.IO.jpg

2022-01-04T21:04:03Z

Temple-os:

An import graph for different files in the LibVF.IO project and what is imported from where.