OpenRM

OpenRM refers to the open sourcing of the NVIDIA kernel module for Linux, specifically the RM Core or the internal resource manager.

Through the open sourced code (located here) we have been able to deduce 7 top level APIs that are being exposed to the Linux developers.

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

Interaction With OpenRM

All of OpenRM is a high scale object oriented abstraction layer for the GPU, all resources are internally classes which have control definitions (methods) on how you can use them. These classes can also represent physical resources or engines.

Alloc (Client)

This is analogous to calling an object constructor in a higher level programming language, such as C#.

The data for this call requires the following components:

1. hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
2. hObjectParent - Parent of the resource we are allocating.
3. hObjectNew - New object to allocate.
4. hClass - Class of the object we are dealing with.
5. pAllocParams - Pointer to values passed into the constructor call.
6. status - Status of the constructor call (did it succeed or fail)

Alloc Memory 64

This creates a CPU's mapping to a resource.

The data for this call requires the following components:

1. hRoot - RM Client Id (given by the kernel module for an individual process that is interacting with the kernel module).
2. hObjectParent - Parent of the resource we create the CPU mapping to.
3. hObjectNew - Mapping object resource id.
4. hClass - Class of the mapped object.
5. flags - Flags to use on the mapped object.
6. pMemory - pointer for the memory to map.
7. pLimit - Edge of memory map.
8. status - Status of the constructor call.

NOTE: Seems to be deprecated.

Control

This is a control command which is analogous to calling a method on a class.

The data for this call requires the following components:

1. hClient - RM Client Id.
2. hObject - Object id.
3. cmd - Method to call.
4. params - Parameters into this method.
5. paramsSize - Size of the params.
6. status - Status of the control call.

NOTE: The control will overwrite the object we pass in over the params pointer.

Dup Object

Duplicates the object, or clones the object, which is a deep copy. This procedure can be used across processes in order to produce shared memory.

The data for this call requires the following components:

1. hClient - RM Client Id.
2. hParent - Parent of the object.
3. hObject - Object destination for duplicating.
4. hClientSrc - Client source id.
5. hObjectSrc - Object source.
6. flags - Flags to use.
7. status - Status of the duplication.

Free

Frees an object from the object tree.

The data for this call requires the following components:

1. hRoot - Client Id.
2. hObjectParent - Parent of the object.
3. hObjectOld - Object to call the destructor on.
4. status - Status of the free.

Vid Heap Control

TODO: Figure out what this does.

NOTE: Seems to be deprecated.

Map Memory

This seems to be the same as alloc memory but seems to be used more often.

The data for this call requires the following components:

1. hClient - Client Id.
2. hDevice - Device id.
3. hMemory - Memory resource.
4. offset - Offset of the memory.
5. length - Length of the memory.
6. pLinearAddress - Pointer for returned address.
7. status - Status of the map memory.
8. flags - Flags to use.

Unmap Memory

This unmaps the CPU memory mapping from the resource.

The data for this call requires the following components:

1. hClient - Client Id.
2. hDevice - Device Id.
3. hMemory - Memory Id.
4. pLinearAddress - pointer to virtual address.
5. status - Status of the unmap command.
6. flags - Flags to use.

Alloc Context DMA 2

Allocates a context for DMA.

The data for this call requires the following components:

1. hObjectParent - Object parent.
2. hSubDevice - Sub device to allocate on.
3. hObjectNew - Context id.
4. hClass - Type of class.
5. flags - Flags to use.
6. selector - UNKNOWN.
7. hMemory - Handle for Memory resource to DMA map.
8. offset - Offset for the DMA mapping.
9. limit - Limit of the DMA map.
10. status - Status of the allocation.

NOTE: Seems to be deprecated.

Map Memory DMA

Map a region of memory for DMA access.

The data for this call requires the following components:

1. hClient - Client Id.
2. hDevice - Device Id.
3. hDma - DMA Id.
4. hMemory - Memory Id.
5. offset - Offset of the region.
6. limit - Limit of the region.
7. flags - flags to use.
8. dmaOffset - Offset of the mapping.
9. status - Status of the map Memory DMA.

Un-map Memory DMA

Unmaps a region of memory for DMA access.

The data for this call requires the following components:

1. hClient - Client Id.
2. hDevice - Device Id.
3. hDma - DMA handle.
4. hMemory - Memory handle.
5. flags - Flags to use.
6. dmaOffset - DMA Offset to unmap.
7. status - Status of the unmap Memory DMA.

Bind Context DMA

Binds a context for DMA interaction to a channel.

The data for this call requires the following components:

1. hClient - Client Id.
2. hChannel - Channel Id.
3. hCtxDma - Context for DMA Id.
4. status - Status of the bind context DMA.

NOTE: Seems to be deprecated.

Share

Sharing of a resource between multiple clients.

TODO: Understand this sharing more.

The data for this call requires the following components:

1. hClient - Client Id.
2. hObject - Resource Id.
3. sharePolicy - Share policy.
4. status - Status of sharing.

Add VBlank Callback

TODO: Figure out what this does.

NOTE: Seems to be deprecated.

Engines

There are engines which internally act as classes inside the OpenRM driver.

These classes have a few key components that make them unique.

Table of Engines

Engines

Engine	Description
Invalid	Does not exist.
Software	Abstraction layer for software processes inside the OS Software.
GPU	Interaction layer with the GPU.
Falcon	Interaction with the Falcon on the GPU.
Master Control	Unknown.
Kernel Master Control	Interaction with the PMC.[1]
Private Ring	Interaction with a private messaging ring.
Software Interrupt	Interaction with software interrupts.
Memory System	Interaction with the memory system.
Kernel Memory System	Interaction with the kernel memory system.
Memory Manager	Interaction with the memory management system.
FBFlcn	Interaction with the FBFlcn subsystem.
TMR	Interaction with the TMR on specific chips.
DMA	Interaction with a GPU's DMA access.
Kernel FIFO	First in First out queue in the GPU driver.
FIFO	Interaction with a generic FIFO.
OS	Interaction with the OS.
BUS	PCI Bus interaction
Kernel BUS	Internal module BUS.
INFOROM	Interaction with the INFOROM Region on the GPU.
PERF	Unknown.
Kernel PERF	Unknown.
BIF	Unknown.
Kernel BIF	Unknown.
SF	Unknown.
GPIO	Interaction with GPIO.
CLK	Clock manager system.
Kernel Display	Kernel Display system.
DISP	Display system.
DPU	Data Processing Unit.