OpenRM
OpenRM refers to the open sourcing of the NVIDIA kernel module for Linux, specifically the RM Core or the internal resource manager.
Through the open sourced code (located here) we have been able to deduce 7 top level APIs that are being exposed to the Linux developers.
An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.
This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.
Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.
Interaction With OpenRM
All of OpenRM is a high scale object oriented abstraction layer for the GPU, all resources are internally classes which have control definitions (methods) on how you can use them. These classes can also represent physical resources or engines.
Alloc (Client)
This is analogous to calling an object constructor in a higher level programming language, such as C#.
The data for this call requires the following components:
- hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
- hObjectParent - Parent of the resource we are allocating.
- hObjectNew - New object to allocate.
- hClass - Class of the object we are dealing with.
- pAllocParams - Pointer to values passed into the constructor call.
- status - Status of the constructor call (did it succeed or fail)
Alloc Memory 64
This creates a CPU's mapping to a resource.
The data for this call requires the following components:
- hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
- hObjectParent - Parent of the resource we create the CPU mapping to.
- hObjectNew - Mapping object resource id.
- hClass - Class of the mapped object.
- flags - Flags to use on the mapped object.
- pMemory - pointer for the memory to map.
- pLimit - Edge of memory map.
- status - Status of the constructor call.
NOTE: Seems to be deprecated.
Control
This is a control command which is analogous to calling a method on a class.
The data for this call requires the following components:
- hClient - RM Client Id.
- hObject - Object id.
- cmd - Method to call.
- params - Parameters into this method.
- paramsSize - Size of the params.
- status - Status of the control call.
NOTE: The control will overwrite the object we pass in over the params pointer.
Dup Object
Duplicates the object, or clones the object, which is a deep copy. This procedure can be used across processes in order to produce shared memory.
The data for this call requires the following components:
- hClient - RM Client Id.
- hParent - Parent of the object.
- hObject - Object destination for duplicating.
- hClientSrc - Client source id.
- hObjectSrc - Object source.
- flags - Flags to use.
- status - Status of the duplication.
Free
Frees an object from the object tree.
The data for this call requires the following components:
- hRoot - Client Id.
- hObjectParent - Parent of the object.
- hObjectOld - Object to call the destructor on.
- status - Status of the free.
Vid Heap Control
TODO: Figure out what this does.
NOTE: Seems to be deprecated.
Map Memory
This seems to be the same as alloc memory but seems to be used more often.
The data for this call requires the following components:
- hClient - Client Id.
- hDevice - Device id.
- hMemory - Memory resource.
- offset - Offset of the memory.
- length - Length of the memory.
- pLinearAddress - Pointer for returned address.
- status - Status of the map memory.
- flags - Flags to use.
Unmap Memory
This unmaps the CPU memory mapping from the resource.
The data for this call requires the following components:
- hClient - Client Id.
- hDevice - Device Id.
- hMemory - Memory Id.
- pLinearAddress - pointer to virtual address.
- status - Status of the unmap command.
- flags - Flags to use.
Alloc Context DMA 2
Allocates a context for DMA.
The data for this call requires the following components:
- hObjectParent - Object parent.
- hSubDevice - Sub device to allocate on.
- hObjectNew - Context id.
- hClass - Type of class.
- flags - Flags to use.
- selector - UNKNOWN.
- hMemory - Handle for Memory resource to DMA map.
- offset - Offset for the DMA mapping.
- limit - Limit of the DMA map.
- status - Status of the allocation.
NOTE: Seems to be deprecated.
Map Memory DMA
Map a region of memory for DMA access.
The data for this call requires the following components:
- hClient - Client Id.
- hDevice - Device Id.
- hDma - DMA Id.
- hMemory - Memory Id.
- offset - Offset of the region.
- limit - Limit of the region.
- flags - flags to use.
- dmaOffset - Offset of the mapping.
- status - Status of the map Memory DMA.
Un-map Memory DMA
Unmaps a region of memory for DMA access.
The data for this call requires the following components:
- hClient - Client Id.
- hDevice - Device Id.
- hDma - DMA handle.
- hMemory - Memory handle.
- flags - Flags to use.
- dmaOffset - DMA Offset to unmap.
- status - Status of the unmap Memory DMA.
Bind Context DMA
Binds a context for DMA interaction to a channel.
The data for this call requires the following components:
- hClient - Client Id.
- hChannel - Channel Id.
- hCtxDma - Context for DMA Id.
- status - Status of the bind context DMA.
NOTE: Seems to be deprecated.
Sharing of a resource between multiple clients.
TODO: Understand this sharing more.
The data for this call requires the following components:
- hClient - Client Id.
- hObject - Resource Id.
- sharePolicy - Share policy.
- status - Status of sharing.
Add VBlank Callback
TODO: Figure out what this does.
NOTE: Seems to be deprecated.
Engines
There are engines which internally act as classes inside the OpenRM driver.
These classes have a few key components that make them unique.
Table of Engines
Engine | Description |
---|---|
Invalid | Does not exist. |
Software | Abstraction layer for software processes inside the OS Software. |
GPU | Interaction layer with the GPU. |
Falcon | Interaction with the Falcon on the GPU. |
Master Control | Unknown. |
Kernel Master Control | Interaction with the PMC.[1] |
Private Ring | Interaction with a private messaging ring. |
Software Interrupt | Interaction with software interrupts. |
Memory System | Interaction with the memory system. |
Kernel Memory System | Interaction with the kernel memory system. |
Memory Manager | Interaction with the memory management system. |
FBFlcn | Interaction with the FBFlcn subsystem. |
TMR | Interaction with the TMR on specific chips. |
DMA | Interaction with a GPU's DMA access. |
Kernel FIFO | First in First out queue in the GPU driver. |
FIFO | Interaction with a generic FIFO. |
OS | Interaction with the OS. |
BUS | PCI Bus interaction |
Kernel BUS | Internal module BUS. |
INFOROM | Interaction with the INFOROM Region on the GPU. |
PERF | Unknown. |
Kernel PERF | Unknown. |
BIF | Unknown. |
Kernel BIF | Unknown. |
SF | Unknown. |
GPIO | Interaction with GPIO. |
CLK | Clock manager system. |
Kernel Display | Kernel Display system. |
DISP | Display system. |
DPU | Data Processing Unit. |