CUDA Userspace Driver - Revision history

WikiBot: Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

2026-10-12T04:40:06Z

Fix repository links: point to the cgit tree URLs (git.vgpu.io/open-iov.git/tree/...)

WikiBot: Carry the site-wide About/licensing message on every page (2026-10-12)

2026-10-12T04:29:42Z

Carry the site-wide About/licensing message on every page (2026-10-12)

← Older revision		Revision as of 04:29, 12 October 2026
Line 1:		Line 1:
			{{About message}}

	= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =		= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =

WikiBot: Re-sync page from tracked wiki source: fix cross-document links and diagram; refresh status

2026-10-12T04:09:59Z

Re-sync page from tracked wiki source: fix cross-document links and diagram; refresh status

Show changes

WikiBot: Remove internal document references

2026-10-12T03:59:35Z

Remove internal document references

New page

= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =


== 1. What it is ==

<code>libcuda.so.1.1</code> is the userspace half of the CUDA driver: the library an application or language runtime loads to talk to a GPU. For 570.144 it is a '''single, fully stripped, position-independent shared object of about 71 MB''':

{| class="wikitable"
|-
! Property
! Value
|-
| Exports
| 659 (public <code>cu*</code> API plus internal/legacy entry points)
|-
| Imports
| 243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path
|-
| <code>.text</code>
| <code>0x179ec0</code>–<code>0xe94c63</code> (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables)
|-
| <code>.rodata</code>
| ~53 MB — error strings, dispatch/version tables, '''embedded device code'''
|-
| CUDA note
| CUDA 12.8; supported kernel-driver ABIs <code>[470, 535, 550, 560, 565, 570]</code>
|}

Two entry surfaces coexist: the '''public driver API''' (<code>cuInit</code>, <code>cuCtxCreate_v2</code>, <code>cuMemAlloc_v2</code>, <code>cuLaunchKernel</code>, …) and the '''entry-table API''' — <code>cuGetExportTable</code>, a 128-bit-UUID-keyed lookup that hands internal tables to the CUDA runtime (<code>libcudart</code>), tooling and interposition libraries. Tables are versioned by a leading <code>struct_size</code> field and are append-only.
The library embeds '''device code''' — cubins for memcpy/memset kernels, the graph scheduler, syscall trampolines and preemption-handler assembly — and loads it through its own module machinery, so a stripped <code>libcuda.so</code> still contains many small ELF images in <code>.rodata</code>. The CUDA ''runtime'' (<code>libcudart</code>, the <code>cuda*</code> API) is a separate stock library that bottoms out in <code>libcuda</code>; it also calls through the entry tables, so a guest failure in a <code>cuda*</code> call may reach driver code through a different door than the public <code>cu*</code> entry point.
'''Open-IOV requires this library unmodified.''' In the guest it links against the guest's stock <code>nvidia.ko</code> / <code>nvidia-uvm.ko</code> exactly as on bare metal; there is no <code>LD_PRELOAD</code> shim and no patch. Instrumentation uses <code>ptrace</code> or kernel-side tracing, because the RM client issues raw <code>syscall(SYS_ioctl)</code> rather than libc <code>ioctl</code>.

== 2. Layer model ==

The library is strictly layered, and the boundaries are where the emulation interfaces live:

<pre>application / libcudart
| public cu* API or UUID entry tables
v
generated wrappers -> API layer -> CUDA internal engine
|-- driver-model layer (RM) -> /dev/nvidia* ioctls
|-- hardware layer (QMD/method encoding)
| -> GPU queues, doorbells
|-- UVM userspace (managed memory) -> /dev/nvidia-uvm
'-- JIT/ELF (PTX->SASS, cubin loading)
kernel driver + GSP-RM + GPU</pre>
* '''Generated wrappers and API layer.''' A master export list (name, platform flags, CUDA version introduced) generates the exported symbols, wrappers, tool callbacks and <code>cuGetProcAddress</code> tables (<code>_vN</code>/<code>_ptsz</code> variants are part of that data). Below it, the hand-written layer reads the per-thread context stack, validates state, takes one context lock, calls one engine function and returns a <code>CUresult</code> — the semantic work is always one layer down.
* '''Internal engine.''' Owns the object graph: process globals (init state, kernel-client handle, device list), device (chip/caps/limits, vtables), context (memory/channel/stream managers, QMD pools, ISR thread), stream, module/function, memory objects, events/graphs/libraries.
* '''Driver-model layer.''' Selected at initialization behind one function table: the '''RM backend''' (the one Open-IOV cares about) plus SoC, Amodel, MPS and WDDM. It turns engine operations into RM controls and object allocations over <code>/dev/nvidia*</code>, and owns the '''driver-ABI adaptation''' layer — one instance per supported kernel-driver ABI, chosen at <code>cuInit</code>.
* '''Hardware layer.''' Chip-specific method/QMD/semaphore/doorbell encoders selected by compute capability. The userspace driver ''builds'' the GPU command streams; the kernel driver allocates, maps, schedules and services interrupts.
* '''Synchronization model.''' Three primitives explain most of it: the '''tracking semaphore''' (per-channel counter of completed pushes), the '''marker''' ("all work before this point") and the '''QMD semaphore''' (released when a grid completes). Blocking waits use a per-context ISR thread and a DPC queue.

== 3. What it touches in a guest ==

This is the emulation contract.
'''Device nodes.''' <code>/dev/nvidiactl</code> (process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object fds), <code>/dev/nvidiaN</code> (per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation), <code>/dev/nvidia-uvm</code> (managed memory and page migration) and <code>/dev/nvidia-vgpuN</code> (vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths).
'''ioctls.''' The control device uses <code>_IOWR('F', cmd, …)</code>: <code>200</code> <code>NV_ESC_CARD_INFO</code>, <code>210</code> <code>NV_ESC_CHECK_VERSION_STR</code>, <code>212</code> <code>NV_ESC_ATTACH_GPUS_TO_FD</code>. RM escapes include <code>0x2b</code> <code>NV_ESC_RM_ALLOC</code> ('''the main object-allocation path'''), <code>0x2a</code> <code>NV_ESC_RM_CONTROL</code>, <code>0x29</code> <code>NV_ESC_RM_FREE</code>, <code>0x4e</code>/<code>0x4f</code> <code>NV_ESC_RM_MAP_MEMORY</code>/<code>UNMAP_MEMORY</code>, <code>0x57</code>/<code>0x58</code> <code>NV_ESC_RM_MAP_MEMORY_DMA</code>/<code>UNMAP_MEMORY_DMA</code> and <code>0x4a</code> <code>NV_ESC_RM_VID_HEAP_CONTROL</code>. The request word <code>rsi=0xc030462b</code> in a crash dump decodes to "allocating an RM object with a 48-byte payload".
'''RM objects/classes.''' Every RM call names a class. Observed for 570.144 on an emulated TU104: <code>NV01_ROOT</code> (<code>0x0</code>), <code>NV01_DEVICE_0</code> (<code>0x80</code>), <code>NV20_SUBDEVICE_0</code> (<code>0x2080</code>), <code>NV01_EVENT</code> (<code>0x40</code>), <code>KEPLER_CHANNEL_GROUP_A</code> (<code>0xa06c</code>), a channel-group/runlist object (<code>0x9067</code>), <code>TURING_CHANNEL_GPFIFO_A</code> (<code>0xc46f</code>, the channel), <code>TURING_COMPUTE_A</code> (<code>0xc5c0</code>, the compute engine object), plus memory (<code>0x3e</code>) and VA-space classes. The observed <code>cuCtxCreate</code> order is root → device → subdevice → events → channel group → channel-group object → memory → channel → control <code>NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST</code> → compute object → engine registration.
'''Controls.''' Device/GPU info (<code>GPU_GET_ENGINES</code>, <code>GPU_GET_CLASSLIST</code>, <code>GPU_GET_VIRTUALIZATION_MODE</code>, <code>FB_GET_INFO</code>, <code>CE_GET_CAPS</code>, …), FIFO/channels (<code>NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST</code> <code>0x80170d</code>, GPFIFO scheduling, timeslices, <code>GET_CLASS_ENGINEID</code>), GR/context switches, memory, events and interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates unknown-but-successful controls; it only fails when a ''specific reply field'' is consumed and wrong.
'''Memory.''' VA reservation (<code>NV_ESC_RM_VID_HEAP_CONTROL</code>/<code>NVOS32</code>), backing allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and mapping (<code>NVOS33</code> into the process, <code>NVOS46</code> into a device VA space). All guest memory CUDA uses is either an RM allocation backed by emulated VRAM/BAR1 or pinned guest RAM the GPU must DMA to.
'''Channels, GPFIFO, USERD, doorbells.''' A channel is a GPFIFO (array of pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO, error notifier and (Volta and later) the USERD itself, passing the handles to the channel allocation; submission writes the GPFIFO put pointer into USERD and rings a doorbell token returned by the channel allocation. Pushbuffers are method streams terminated by a tracking-semaphore release and a new GPFIFO entry; the host emulation decodes them as such.
'''Synchronization below the API.''' Semaphores (plain GPU-visible memory values), 4 KB per-channel error notifiers, OS events (kernel objects used to wake blocked threads), interrupts routed through the kernel driver, and a trap handler for debuggers and <code>device-side assert</code>.
'''UVM and GSP-RM.''' When UVM is available the driver registers GPUs, VA spaces, channels and managed allocations and pushes UVM semaphores into the same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing from userspace — it still issues RM ioctls; inside the kernel driver controls become GSP-RM RPCs, which Open-IOV's host module answers.

== 4. Why forwarding matters, and where we are ==

The performance budget is dominated by '''push-buffer execution''', '''synchronization latency''' and (once, at setup) '''control-plane latency'''. Only forwarding ''real work'' to the physical GPU meets the first two; software execution of state/memory ops is a correctness bridge, not the data plane.
* '''Stated architecture:''' ''emulate the interface, pass through the work, never patch anyone.'' Prototype the host-RM/GSP proxy first for correctness, measure, then move the hot path to direct submission if the budgets demand it — work path abstracted from day one.
* '''Control plane today:''' GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and object/control replies are implemented; the stock guest stack gets through <code>cuInit</code> and device enumeration into <code>cuCtxCreate</code>.
* '''Current blocker:''' <code>cuCtxCreate</code> returns <code>rc=999</code> (<code>CUDA_ERROR_UNKNOWN</code>) because the class-filtered engine list is missing the async copy engine <code>COPY14</code> (descriptor <code>0x793ceb0e</code>, instance 14, client type <code>0x38</code>, class <code>0xc9b5</code>). The oracle list has 7 engines (GR, CE0.CE4, <code>COPY14</code>); the guest list has 6. The driver silently skips engines with no per-(class, instance) database entry, then reads a zeroed secondary record and rolls back; what builds that database entry is the open question.
* '''Data plane today:''' the host module decodes guest push buffers as method streams and '''executes the small work it understands itself''' — memsets, copies and semaphore releases. '''No CUDA kernel has been forwarded yet.'''


== 5. The crash-path lesson: replies are consumed data ==

The old deterministic <code>cuCtxCreate</code> segfault at <code>libcuda+0x453f90</code> shows why reply coherence matters. The engine-registration hook allocates a <code>TURING_COMPUTE_A</code> object with <code>pAllocParms</code> pointing at the caller's own '''8-byte stack local''' and <code>paramsSize = 0</code>; the guest driver does not honour <code>paramsSize</code>, so it reads '''16-byte''' class params starting there (the upper 8 bytes are the hook's saved <code>rbp</code>) and copies 16 bytes of reply params back over the same buffer. With the old zeroed reply that copy-back zeroed the saved <code>rbp</code>; the function's epilogue then loaded <code>rbp = 0</code> and its next access — reloading a pointer from <code>[rbp-0x38]</code> — faulted at <code>0xffffffffffffffc8</code> (the store was the kernel's <code>copy_to_user</code>).
'''The fix:''' echo the request's params back, forcing only <code>status</code> to <code>NV_OK</code>; the copy-back then writes exactly the bytes it read, and the fault disappears — the same thing the real GSP-RM does for these in/out params.
'''The general rule:''' the userspace library is a ''consumer of emulated data''. A wrong reply is worse than a slow one. Work up the reply-fidelity ladder — status-only → coherent zeroed params → coherent values for consumed fields → exact values — and validate each rung with the A/B oracle.

== 6. How to look things up ==

The toolchain is version-agnostic:

<pre>tools/cudatool.py info|exports|functions|nearest|crash|delta <lib>
tools/callwalk.py <callgraph.tsv> callers|callees|chain..</pre>
* '''<code>cudatool.py</code>''' — build id/sections/exports/ABI note (<code>info</code>), symbol map (<code>exports</code>), function ranges from <code>.eh_frame_hdr</code> (<code>functions</code>), containing function and nearest export for an address (<code>nearest</code>), dmesg <code>ip</code> to file vaddr (<code>crash</code>), and an interface-level diff (<code>delta</code>).
* '''<code>callwalk.py</code>''' — caller/callee trees and shortest call paths over the callgraph index.
* '''Annotated corpus''' — <code>570.144/annotations/names.tsv</code> (our name map; exports exact, internals grow over time) and <code>subsystems.tsv</code>; corpus headers carry the recovered name, file address and direct callers/callees.
'''Address convention (read this before citing an offset).''' Public docs cite '''file virtual addresses''' as <code>libcuda+0x..</code>. Ghidra loads these binaries at image base <code>0x100000</code>, so <code>program address = file vaddr + 0x100000</code>; the <code>tools/</code> wrappers translate automatically, raw Ghidra scripts do not. An earlier analysis passed file vaddrs straight to Ghidra and decompiled functions 0x100000 too low, so its decompiled-code conclusions were about the wrong functions (its live traces stayed valid). The same applies when transferring a name map to a new driver version.

== 7. Where the detail lives ==

{| class="wikitable"
|-
! Document
! What it gives you
|-
| the reading order, fast recipes, and the two findings that change how you debug
|-
| the full layer/object/sync/memory model
|-
| build model, embedded device code, init/threading, what it touches
|-
| the emulation contract: nodes, ioctls, classes, controls, channels, UVM
|-
| NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security
|-
| the binary map, the crash resolution, and the current engine-topology blocker
|-
| every tool, its usage and its cost
|-
| re-running the analysis on a new driver version
|-
| forwarding options and the honest current stage
|-
| the project documentation front door
|}

← Older revision		Revision as of 04:40, 12 October 2026
Line 3:		Line 3:
	= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =		= CUDA Userspace Driver (<code>libcuda.so</code>) — Analysis Deep Dive =

	<blockquote>'''Status (2026-10-12, honest).''' The analysis covers the whole library for driver 570.144 and the tooling is version-agnostic. The ''control plane'' gets the stock guest stack through <code>cuInit</code>, device enumeration, <code>cuCtxCreate</code> and the full driver-API smoke test — allocation, copies, free and destroy complete end to end (<code>ALL OK</code>; the 64-byte payload round-trips exactly). The ''data plane'' decodes guest push buffers and executes the memory and semaphore work it understands; CUDA kernel launches are forwarded to the physical execution path, where a launch reaches the engine queue (front-end deposit and scheduling are observed) but an engine-authored release is not yet observed. Green runs are not yet reproducible on every boot — an open issue, tracked in the repository. Canonical detail: [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/01-architecture.md <code>01-architecture.md</code>] through <code>08</code>; reading order [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]; front door [https://git.vgpu.io/open-iov/~~blob/master~~/docs/README.md <code>../docs/README.md</code>].</blockquote>		<blockquote>'''Status (2026-10-12, honest).''' The analysis covers the whole library for driver 570.144 and the tooling is version-agnostic. The ''control plane'' gets the stock guest stack through <code>cuInit</code>, device enumeration, <code>cuCtxCreate</code> and the full driver-API smoke test — allocation, copies, free and destroy complete end to end (<code>ALL OK</code>; the 64-byte payload round-trips exactly). The ''data plane'' decodes guest push buffers and executes the memory and semaphore work it understands; CUDA kernel launches are forwarded to the physical execution path, where a launch reaches the engine queue (front-end deposit and scheduling are observed) but an engine-authored release is not yet observed. Green runs are not yet reproducible on every boot — an open issue, tracked in the repository. Canonical detail: [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/01-architecture.md <code>01-architecture.md</code>] through <code>08</code>; reading order [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]; front door [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>].</blockquote>

	== 1. What it is ==		== 1. What it is ==
Line 114:		Line 114:
	! What it gives you		! What it gives you
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/READ-ME-FIRST.md <code>../userspace/cuda/READ-ME-FIRST.md</code>]
	\| the reading order, fast recipes, and the two findings that change how you debug		\| the reading order, fast recipes, and the two findings that change how you debug
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/01-architecture.md <code>../userspace/cuda/docs/01-architecture.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/01-architecture.md <code>../userspace/cuda/docs/01-architecture.md</code>]
	\| the full layer/object/sync/memory model		\| the full layer/object/sync/memory model
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/02-build-and-execution.md <code>../userspace/cuda/docs/02-build-and-execution.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/02-build-and-execution.md <code>../userspace/cuda/docs/02-build-and-execution.md</code>]
	\| build model, embedded device code, init/threading, what it touches		\| build model, embedded device code, init/threading, what it touches
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/03-driver-and-gpu-interface.md <code>../userspace/cuda/docs/03-driver-and-gpu-interface.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/03-driver-and-gpu-interface.md <code>../userspace/cuda/docs/03-driver-and-gpu-interface.md</code>]
	\| the emulation contract: nodes, ioctls, classes, controls, channels, UVM		\| the emulation contract: nodes, ioctls, classes, controls, channels, UVM
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/04-virtualization-and-emulation.md <code>../userspace/cuda/docs/04-virtualization-and-emulation.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/04-virtualization-and-emulation.md <code>../userspace/cuda/docs/04-virtualization-and-emulation.md</code>]
	\| NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security		\| NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/05-570.144-decompilation.md <code>../userspace/cuda/docs/05-570.144-decompilation.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/05-570.144-decompilation.md <code>../userspace/cuda/docs/05-570.144-decompilation.md</code>]
	\| the binary map, the crash resolution, and the current engine-topology blocker		\| the binary map, the crash resolution, and the current engine-topology blocker
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/06-tooling.md <code>../userspace/cuda/docs/06-tooling.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/06-tooling.md <code>../userspace/cuda/docs/06-tooling.md</code>]
	\| every tool, its usage and its cost		\| every tool, its usage and its cost
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/07-version-upgrade-playbook.md <code>../userspace/cuda/docs/07-version-upgrade-playbook.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/07-version-upgrade-playbook.md <code>../userspace/cuda/docs/07-version-upgrade-playbook.md</code>]
	\| re-running the analysis on a new driver version		\| re-running the analysis on a new driver version
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/userspace/cuda/docs/08-virtualization-approaches.md <code>../userspace/cuda/docs/08-virtualization-approaches.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/userspace/cuda/docs/08-virtualization-approaches.md <code>../userspace/cuda/docs/08-virtualization-approaches.md</code>]
	\| forwarding options and the honest current stage		\| forwarding options and the honest current stage
	\|-		\|-
	\| [https://git.vgpu.io/open-iov/~~blob/master~~/docs/README.md <code>../docs/README.md</code>]		\| [https://git.vgpu.io/open-iov.git/tree/docs/README.md <code>../docs/README.md</code>]
	\| the project documentation front door		\| the project documentation front door
	\|}		\|}