Difference between revisions of "Graphics Execution"

From Open-IOV
Jump to navigation Jump to search
(New page (2026-10-30 wiki refresh): Graphics Execution)
 
m (2026-11-09 wiki refresh: display/KMS bring-up (NvKms, /dev/dri, GL 4.6), direct-GL channel submissions, current X/GLX completion wall)
Line 5: Line 5:
<blockquote>'''Scope.''' How the emulated vGPU runs '''3D graphics''' work — the OpenGL and Vulkan command streams a stock guest submits — on the physical GPU, and what is proven about it. This is the shared machinery behind [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]] and the Windows/DirectX target ([[DirectX|<code>DirectX.md</code>]]); CUDA compute uses the same submission and completion path for a different engine workload ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]). Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/05-graphics-execution-plan.md <code>../docs/04-implementation/05-graphics-execution-plan.md</code>] (the plan and milestone ladder) and [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/04-real-render-walls.md <code>../docs/04-implementation/04-real-render-walls.md</code>] (the measured walls); the docs tree wins on disagreement.</blockquote>
<blockquote>'''Scope.''' How the emulated vGPU runs '''3D graphics''' work — the OpenGL and Vulkan command streams a stock guest submits — on the physical GPU, and what is proven about it. This is the shared machinery behind [[OpenGL|<code>OpenGL.md</code>]], [[Vulkan|<code>Vulkan.md</code>]] and the Windows/DirectX target ([[DirectX|<code>DirectX.md</code>]]); CUDA compute uses the same submission and completion path for a different engine workload ([[The Execution Frontier|<code>Execution-Frontier.md</code>]]). Canonical sources: [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/05-graphics-execution-plan.md <code>../docs/04-implementation/05-graphics-execution-plan.md</code>] (the plan and milestone ladder) and [https://git.vgpu.io/open-iov.git/tree/docs/04-implementation/04-real-render-walls.md <code>../docs/04-implementation/04-real-render-walls.md</code>] (the measured walls); the docs tree wins on disagreement.</blockquote>


Status: 2026-10-30, tree <code>master @ b4930168</code>. Tags: <code>[live]</code> observed end to end; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> defined success state; <code>[planned]</code> designed, not built.
Status: 2026-11-09, tree <code>master @ 4acc81e8</code>. Tags: <code>[live]</code> observed end to end; <code>[wip]</code> partially built; <code>[wall]</code> current blocker; <code>[target]</code> defined success state; <code>[planned]</code> designed, not built.


== 1. What "real pixels" means ==
== 1. What "real pixels" means ==
Line 33: Line 33:
# '''Verbatim 3D method-stream replay''' <code>[live]</code>. The emulation captures the X server's graphics-class (<code>0xC597</code>, <code>TURING_A</code>) section from its channel, relocates the operand addresses into host-built shadow pages the engine can reach, and submits the stream unchanged. The engine retires it: the uploads (<code>LINE_LENGTH_IN</code>/<code>LAUNCH_DMA</code>), the state, program and texture binds, and the release all execute with no MMU fault. This proves the ''mechanics'' of 3D execution — object bind, window placement, operand relocation, completion — on real hardware.
# '''Verbatim 3D method-stream replay''' <code>[live]</code>. The emulation captures the X server's graphics-class (<code>0xC597</code>, <code>TURING_A</code>) section from its channel, relocates the operand addresses into host-built shadow pages the engine can reach, and submits the stream unchanged. The engine retires it: the uploads (<code>LINE_LENGTH_IN</code>/<code>LAUNCH_DMA</code>), the state, program and texture binds, and the release all execute with no MMU fault. This proves the ''mechanics'' of 3D execution — object bind, window placement, operand relocation, completion — on real hardware.
# '''Synthesized launch from a captured section''' <code>[live]</code>. A compute-shaped launch built from the captured section (operand list, program, a 64-word launch descriptor) reaches the engine and retires real <code>sm_75</code> SASS; the Turing shader-program-header form (<code>V3</code>) is accepted with zero header faults across repeated launches. This proved engine-side dispatch before the verbatim lane existed. The program itself is still a development stand-in at this rung — execution proof, not pixels.
# '''Synthesized launch from a captured section''' <code>[live]</code>. A compute-shaped launch built from the captured section (operand list, program, a 64-word launch descriptor) reaches the engine and retires real <code>sm_75</code> SASS; the Turing shader-program-header form (<code>V3</code>) is accepted with zero header faults across repeated launches. This proved engine-side dispatch before the verbatim lane existed. The program itself is still a development stand-in at this rung — execution proof, not pixels.
# '''Forwarding the client's own pushes''' <code>[wall]</code>. The goal state: the GL client's own GPFIFO ring and method streams are read, decoded and executed with the client's real per-draw state. The GL client's channel is created today, but it never submits — its pushes have not been made readable — and this lane is gated on the ring/push-readability work (§6).
# '''Forwarding the client's own pushes''' <code>[wip]</code>. The GL client's own channels now register, submit and retire: the direct-GL GR channel (<code>0xbeef0100</code>) and its CE siblings (<code>0xbeef0101</code>/<code>0xbeef0102</code>) are discovered, their rings and releases are decoded, and the X server's DDX kick channel (<code>0xbfef0002</code>) is decoded from its ring head with its class-2D notification authored. What remains is completing the GLX context-creation handshake and the client's kick-slot wait (§6), not making the pushes readable.


== 4. The channel and object picture ==
== 4. The channel and object picture ==


* '''Every submission lives on a channel''': a GPFIFO ring plus engine objects, with a USERD the driver advances its put pointer in and a doorbell the host module traps. Channels are typed by engine: a graphics channel carries 3D state and draws; copy-class channels carry memory movements.
* '''Every submission lives on a channel''': a GPFIFO ring plus engine objects, with a USERD the driver advances its put pointer in and a doorbell the host module traps. Channels are typed by engine: a graphics channel carries 3D state and draws; copy-class channels carry memory movements.
* '''The graphics object class''' the guest's 3D stream binds is the Turing graphics class, <code>0xC597</code>. The stream the emulation can capture today comes from the '''X server's channel''' (<code>OpenGL</code>); the 2D present/readback work rides lighter 2D-class and copy-engine shapes.
* '''The graphics object class''' the guest's 3D stream binds is the Turing graphics class, <code>0xC597</code>. Streams now come from both the '''X server's channel''' and the '''direct GL client's own channels''' (<code>0xbeef0100</code> GR plus the <code>0xbeef0101</code>/<code>0xbeef0102</code> CE siblings); the 2D present/readback work rides lighter 2D-class and copy-engine shapes.
* '''Display and modeset''' <code>[live]</code>. The guest's modeset path is served by an emulated display engine: the display push-buffer channels (core <code>0xc57d</code>, window <code>0xc57e</code>, immediate/window <code>0xc57b</code>) are registered, their push buffers parsed, and the notifier/<code>UPDATE</code> completion authored. This is what lets nvidia-modeset/NvKms register (<code>/dev/dri</code>), X settle and GLX clients obtain a real GL context. The display-capability seam is parameterised in the campaign configuration.
* '''Cells of state''': 3D streams are long—hundreds of method words per draw — and mix state binds (viewport, render target, ZS, blend/ROP), program and constant binds, vertex/index streams and draw-issuing methods. The capture machinery recognises the init-shaped operands today; the per-draw state vocabulary is being extended ([G-B in the plan]( ../docs/04-implementation/05-graphics-execution-plan.md)).
* '''Cells of state''': 3D streams are long—hundreds of method words per draw — and mix state binds (viewport, render target, ZS, blend/ROP), program and constant binds, vertex/index streams and draw-issuing methods. The capture machinery recognises the init-shaped operands today; the per-draw state vocabulary is being extended ([G-B in the plan]( ../docs/04-implementation/05-graphics-execution-plan.md)).
* '''Synchronisation''': tracking semaphores, notifications and fence values written by the engine; the guest's swap/present waits sit on top of those.
* '''Synchronisation''': tracking semaphores, notifications and fence values written by the engine; the guest's swap/present waits sit on top of those.
Line 45: Line 46:


The acceptance signal is the '''engine-authored release''': a semaphore the engine writes when the work completes. For GL, completion flows through the channel's semaphores/notifications to the X server and the app's swap wait; for Vulkan, the client's fence value is released; for CUDA, the launch's release group wakes the client. The emulation relays engine completions into guest-observable memory (MSI-X plus a poller), and the guest resumes from there.
The acceptance signal is the '''engine-authored release''': a semaphore the engine writes when the work completes. For GL, completion flows through the channel's semaphores/notifications to the X server and the app's swap wait; for Vulkan, the client's fence value is released; for CUDA, the launch's release group wakes the client. The emulation relays engine completions into guest-observable memory (MSI-X plus a poller), and the guest resumes from there.
For the display path, completion is the notifier write and <code>UPDATE -&gt; FINISHED</code>; a completion event deferred by the reply path is force-retried until it posts, so the X server's mode set finishes.


During bring-up the module also carries a '''diagnostics-only mirror''' for apps that spin on a shared progress word; it exists so a run can proceed while the natural completion path is being finished. The 60-fps milestone explicitly requires it switched off, along with any external poke.
During bring-up the module also carries a '''diagnostics-only mirror''' for apps that spin on a shared progress word; it exists so a run can proceed while the natural completion path is being finished. The 60-fps milestone explicitly requires it switched off, along with any external poke.
Line 50: Line 53:
== 6. Where the frontier is (ranked walls) ==
== 6. Where the frontier is (ranked walls) ==


# '''The GL client's submissions''' <code>[wall]</code>. The client's channel exists but never submits: direct measurement shows the client process writes no put pointer in any mapping the service can see, and its pushes are not yet reachable. Everything downstream waits on this.
# '''The X GLX context-creation completion''' <code>[wall]</code>. With the UVM channel-pool and direct-GL submission walls closed, the X server spins in its RM ioctl path (<code>_nv018351rm</code>) while a GL client's GLX context request waits on the X reply. The X DDX's own channels and the GLX reply are the current named objects. (Fullscreen GZDoom's client-triggered mode set can still wedge X; the campaign path is a settled X plus windowed GZDoom.)
# '''Program (SASS) delivery''' <code>[wall]</code>. Graphics programs reach the device through the driver's copy/utility-channel uploads; those inline-data writes are not yet materialised for the GL/Vulkan clients, and the program page the engine fetches is empty. With the data-set gate bypassed, the engine fetches the launch and MMU-faults (<code>FAULT_PDE</code>) at the program VA — proof the channel and context are alive, and that mapping + content are the missing pair.
# '''The client kick-slot word''' <code>[wall]</code>. The GL client polls a kick-slot GPU address for its completion; the RM notifier page is a different object, so the completion must be authored at the client's kick-slot address.
# '''Non-black frames''' <code>[wall]</code>. Once draws execute with real state and real programs, the capture must show motion; every frame captured so far is black.
# '''Non-black frames''' <code>[wall]</code>. Once the GLX context completes and the client's kick wait resolves, the capture must show motion; every frame captured so far is black.
# '''Sustained frame rate, and two VMs''' <code>[target]</code>. The defined end state: 60 fps in one VM with no forcing, then two VMs on two mediated devices with non-black frames in both.
# '''Sustained frame rate, and two VMs''' <code>[target]</code>. The defined end state: 60 fps in one VM with no forcing, then two VMs on two mediated devices with non-black frames in both.


Line 65: Line 68:
| <code>[live]</code>
| <code>[live]</code>
|-
|-
| <code>glxinfo</code> enumerates the emulated GPU with the NVIDIA renderer
| Display engine serves the modeset push buffers; notifier plus <code>UPDATE -&gt; FINISHED</code>
| <code>[live]</code> (campaign configuration)
|-
| nvidia-modeset/NvKms registers (<code>/dev/dri/card0</code>, <code>renderD128</code>, <code>fb0</code>); <code>nvidia-smi -L</code> enumerates
| <code>[live]</code>
| <code>[live]</code>
|-
|-
| GZDoom launches and runs on the emulated vGPU
| <code>glxinfo</code> reports the emulated GPU with the NVIDIA renderer (OpenGL 4.6.0, direct rendering)
| <code>[live]</code>
|-
| GZDoom obtains a real GL context; windowed reaches renderer startup
| <code>[live]</code> (pixels frontier)
| <code>[live]</code> (pixels frontier)
|-
|-
Line 84: Line 93:
|-
|-
| Direct GL client: device-init gate handled, channels allocate, readback path returns clean
| Direct GL client: device-init gate handled, channels allocate, readback path returns clean
| <code>[wip]</code> (client submissions not yet)
| <code>[live]</code>
|-
| Direct GL client channels submit and retire (GR <code>0xbeef0100</code>, CE <code>0xbeef0101</code>/<code>0xbeef0102</code>); DDX kick channel decoded
| <code>[live]</code>
|-
| X GLX context-creation completion and the client kick-slot word
| <code>[wall]</code>
|-
|-
| Non-black pixels from the emulated vGPU's 3D pipeline
| Non-black pixels from the emulated vGPU's 3D pipeline
Line 96: Line 111:


* No non-black pixel has been produced by the emulated vGPU's 3D pipeline yet.
* No non-black pixel has been produced by the emulated vGPU's 3D pipeline yet.
* Display/KMS bring-up (NvKms, <code>/dev/dri</code>, X, a real GL context) is not a rendered-frame claim; the GLX context-creation and kick-slot completions are the current walls.
* No frame-rate or latency numbers: the scheduled measurement is defined (60 fps over 60 s, no forcing) but not yet met.
* No frame-rate or latency numbers: the scheduled measurement is defined (60 fps over 60 s, no forcing) but not yet met.
* The <code>virgl</code>-based demo videos are a different, labelled path — not this one.
* The <code>virgl</code>-based demo videos are a different, labelled path — not this one.

Revision as of 17:57, 9 November 2026

An absence of critical technical documentation has historically slowed growth and adoption of developer ecosystems for GPU virtualization.

This CC-BY-4.0 licensed content can either be used with attribution, or used as inspiration for new documentation, created by GPU vendors for public commercial distribution as developer documentation.

Where possible, this documentation will clearly label dates and versions of observed-but-not-guaranteed behaviour vs. vendor-documented stable interfaces/behaviour with guarantees of forward or backward compatibility.

Graphics Execution

Scope. How the emulated vGPU runs 3D graphics work — the OpenGL and Vulkan command streams a stock guest submits — on the physical GPU, and what is proven about it. This is the shared machinery behind OpenGL.md, Vulkan.md and the Windows/DirectX target (DirectX.md); CUDA compute uses the same submission and completion path for a different engine workload (Execution-Frontier.md). Canonical sources: ../docs/04-implementation/05-graphics-execution-plan.md (the plan and milestone ladder) and ../docs/04-implementation/04-real-render-walls.md (the measured walls); the docs tree wins on disagreement.

Status: 2026-11-09, tree master @ 4acc81e8. Tags: [live] observed end to end; [wip] partially built; [wall] current blocker; [target] defined success state; [planned] designed, not built.

1. What "real pixels" means

The acceptance target is strict: a stock guest GL or Vulkan application's draw commands are executed by the physical GPU's fixed-function 3D pipeline (vertex fetch, primitive assembly, rasterisation, fragment work, ROP), the render target is readable by the guest, and that image reaches the capture path through the emulated mdev. The pixels must not come from another GPU, a software rasteriser, or a host display round-trip.

(The project has also published two deliberately-labelled two-VM demo videos that use the standard virtio-gpu/virgl virtual GPU on a host integrated GPU, and a CUDA-animation video that does run on the emulated vGPU through the compute path. The virgl videos are honest demonstrations of the guest experience; they are not this execution path, and their captions say so.)

2. The path a graphics command takes

The graphics execution paths: guest OpenGL/Vulkan/CUDA commands to the physical engines, and completions back.
guest apps (GL / Vulkan / CUDA)
  -> guest user space (GLX client + GL core, Vulkan ICD, libcuda)
  -> guest X server (GLX module + NVIDIA DDX)           [OpenGL]
  -> nvidia.ko: channels, GPFIFO rings, doorbells       [all APIs]
  -> vgsp.ko: capture sections, decode pushes, forward
  -> physical GPU front-end -> GR engine (3D + compute) / copy engines
  <- engine-authored releases written into guest memory
  <- completion relay (MSI-X + wakes) back to the guest
ASCII fallback above; the diagram shows the three execution lanes.

3. The three execution lanes

Graphics work reaches the engines today in one of three shapes, in increasing fidelity:

  1. Verbatim 3D method-stream replay [live]. The emulation captures the X server's graphics-class (0xC597, TURING_A) section from its channel, relocates the operand addresses into host-built shadow pages the engine can reach, and submits the stream unchanged. The engine retires it: the uploads (LINE_LENGTH_IN/LAUNCH_DMA), the state, program and texture binds, and the release all execute with no MMU fault. This proves the mechanics of 3D execution — object bind, window placement, operand relocation, completion — on real hardware.
  2. Synthesized launch from a captured section [live]. A compute-shaped launch built from the captured section (operand list, program, a 64-word launch descriptor) reaches the engine and retires real sm_75 SASS; the Turing shader-program-header form (V3) is accepted with zero header faults across repeated launches. This proved engine-side dispatch before the verbatim lane existed. The program itself is still a development stand-in at this rung — execution proof, not pixels.
  3. Forwarding the client's own pushes [wip]. The GL client's own channels now register, submit and retire: the direct-GL GR channel (0xbeef0100) and its CE siblings (0xbeef0101/0xbeef0102) are discovered, their rings and releases are decoded, and the X server's DDX kick channel (0xbfef0002) is decoded from its ring head with its class-2D notification authored. What remains is completing the GLX context-creation handshake and the client's kick-slot wait (§6), not making the pushes readable.

4. The channel and object picture

  • Every submission lives on a channel: a GPFIFO ring plus engine objects, with a USERD the driver advances its put pointer in and a doorbell the host module traps. Channels are typed by engine: a graphics channel carries 3D state and draws; copy-class channels carry memory movements.
  • The graphics object class the guest's 3D stream binds is the Turing graphics class, 0xC597. Streams now come from both the X server's channel and the direct GL client's own channels (0xbeef0100 GR plus the 0xbeef0101/0xbeef0102 CE siblings); the 2D present/readback work rides lighter 2D-class and copy-engine shapes.
  • Display and modeset [live]. The guest's modeset path is served by an emulated display engine: the display push-buffer channels (core 0xc57d, window 0xc57e, immediate/window 0xc57b) are registered, their push buffers parsed, and the notifier/UPDATE completion authored. This is what lets nvidia-modeset/NvKms register (/dev/dri), X settle and GLX clients obtain a real GL context. The display-capability seam is parameterised in the campaign configuration.
  • Cells of state: 3D streams are long—hundreds of method words per draw — and mix state binds (viewport, render target, ZS, blend/ROP), program and constant binds, vertex/index streams and draw-issuing methods. The capture machinery recognises the init-shaped operands today; the per-draw state vocabulary is being extended ([G-B in the plan]( ../docs/04-implementation/05-graphics-execution-plan.md)).
  • Synchronisation: tracking semaphores, notifications and fence values written by the engine; the guest's swap/present waits sit on top of those.

5. Completion, fences and presents

The acceptance signal is the engine-authored release: a semaphore the engine writes when the work completes. For GL, completion flows through the channel's semaphores/notifications to the X server and the app's swap wait; for Vulkan, the client's fence value is released; for CUDA, the launch's release group wakes the client. The emulation relays engine completions into guest-observable memory (MSI-X plus a poller), and the guest resumes from there.

For the display path, completion is the notifier write and UPDATE -> FINISHED; a completion event deferred by the reply path is force-retried until it posts, so the X server's mode set finishes.

During bring-up the module also carries a diagnostics-only mirror for apps that spin on a shared progress word; it exists so a run can proceed while the natural completion path is being finished. The 60-fps milestone explicitly requires it switched off, along with any external poke.

6. Where the frontier is (ranked walls)

  1. The X GLX context-creation completion [wall]. With the UVM channel-pool and direct-GL submission walls closed, the X server spins in its RM ioctl path (_nv018351rm) while a GL client's GLX context request waits on the X reply. The X DDX's own channels and the GLX reply are the current named objects. (Fullscreen GZDoom's client-triggered mode set can still wedge X; the campaign path is a settled X plus windowed GZDoom.)
  2. The client kick-slot word [wall]. The GL client polls a kick-slot GPU address for its completion; the RM notifier page is a different object, so the completion must be authored at the client's kick-slot address.
  3. Non-black frames [wall]. Once the GLX context completes and the client's kick wait resolves, the capture must show motion; every frame captured so far is black.
  4. Sustained frame rate, and two VMs [target]. The defined end state: 60 fps in one VM with no forcing, then two VMs on two mediated devices with non-black frames in both.

7. What is proven today

Capability State
X server initialises on the emulated vGPU (driver DMA sanity push, ScreenInit) [live]
Display engine serves the modeset push buffers; notifier plus UPDATE -> FINISHED [live] (campaign configuration)
nvidia-modeset/NvKms registers (/dev/dri/card0, renderD128, fb0); nvidia-smi -L enumerates [live]
glxinfo reports the emulated GPU with the NVIDIA renderer (OpenGL 4.6.0, direct rendering) [live]
GZDoom obtains a real GL context; windowed reaches renderer startup [live] (pixels frontier)
GL section capture: the X server's 0xC597 stream decoded [live]
Verbatim 3D method-stream replay retired by the engine (no MMU fault) [live]
Synthesized launch executes real sm_75 SASS; Turing V3 program header accepted [live]
2D screen fills and copy-engine present ops execute [live]
Direct GL client: device-init gate handled, channels allocate, readback path returns clean [live]
Direct GL client channels submit and retire (GR 0xbeef0100, CE 0xbeef0101/0xbeef0102); DDX kick channel decoded [live]
X GLX context-creation completion and the client kick-slot word [wall]
Non-black pixels from the emulated vGPU's 3D pipeline [wall]
Windows/DirectX guests [planned]

8. What is not claimed

  • No non-black pixel has been produced by the emulated vGPU's 3D pipeline yet.
  • Display/KMS bring-up (NvKms, /dev/dri, X, a real GL context) is not a rendered-frame claim; the GLX context-creation and kick-slot completions are the current walls.
  • No frame-rate or latency numbers: the scheduled measurement is defined (60 fps over 60 s, no forcing) but not yet met.
  • The virgl-based demo videos are a different, labelled path — not this one.

9. Related pages