CUDA Userspace Driver

CUDA Userspace Driver (libcuda.so) — Analysis Deep Dive

1. What it is

libcuda.so.1.1 is the userspace half of the CUDA driver: the library an application or language runtime loads to talk to a GPU. For 570.144 it is a single, fully stripped, position-independent shared object of about 71 MB:

Property	Value
Exports	659 (public cu* API plus internal/legacy entry points)
Imports	243 — libc/libpthread/libdl/librt only; no NVIDIA userspace dependency on the CUDA path
.text	0x179ec0–0xe94c63 (13.7 MB); 29,494 functions found by full analysis (28,925 with unwind tables)
.rodata	~53 MB — error strings, dispatch/version tables, embedded device code
CUDA note	CUDA 12.8; supported kernel-driver ABIs [470, 535, 550, 560, 565, 570]

Two entry surfaces coexist: the public driver API (cuInit, cuCtxCreate_v2, cuMemAlloc_v2, cuLaunchKernel, …) and the entry-table API — cuGetExportTable, a 128-bit-UUID-keyed lookup that hands internal tables to the CUDA runtime (libcudart), tooling and interposition libraries. Tables are versioned by a leading struct_size field and are append-only. The library embeds device code — cubins for memcpy/memset kernels, the graph scheduler, syscall trampolines and preemption-handler assembly — and loads it through its own module machinery, so a stripped libcuda.so still contains many small ELF images in .rodata. The CUDA runtime (libcudart, the cuda* API) is a separate stock library that bottoms out in libcuda; it also calls through the entry tables, so a guest failure in a cuda* call may reach driver code through a different door than the public cu* entry point. Open-IOV requires this library unmodified. In the guest it links against the guest's stock nvidia.ko / nvidia-uvm.ko exactly as on bare metal; there is no LD_PRELOAD shim and no patch. Instrumentation uses ptrace or kernel-side tracing, because the RM client issues raw syscall(SYS_ioctl) rather than libc ioctl.

2. Layer model

The library is strictly layered, and the boundaries are where the emulation interfaces live:

application / libcudart
      | public cu* API or UUID entry tables
      v
generated wrappers -> API layer -> CUDA internal engine
                                      |-- driver-model layer (RM) -> /dev/nvidia* ioctls
                                      |-- hardware layer (QMD/method encoding)
                                      |     -> GPU queues, doorbells
                                      |-- UVM userspace (managed memory) -> /dev/nvidia-uvm
                                      '-- JIT/ELF (PTX->SASS, cubin loading)
kernel driver + GSP-RM + GPU

• Generated wrappers and API layer. A master export list (name, platform flags, CUDA version introduced) generates the exported symbols, wrappers, tool callbacks and cuGetProcAddress tables (_vN/_ptsz variants are part of that data). Below it, the hand-written layer reads the per-thread context stack, validates state, takes one context lock, calls one engine function and returns a CUresult — the semantic work is always one layer down.
• Internal engine. Owns the object graph: process globals (init state, kernel-client handle, device list), device (chip/caps/limits, vtables), context (memory/channel/stream managers, QMD pools, ISR thread), stream, module/function, memory objects, events/graphs/libraries.
• Driver-model layer. Selected at initialization behind one function table: the RM backend (the one Open-IOV cares about) plus SoC, Amodel, MPS and WDDM. It turns engine operations into RM controls and object allocations over /dev/nvidia*, and owns the driver-ABI adaptation layer — one instance per supported kernel-driver ABI, chosen at cuInit.
• Hardware layer. Chip-specific method/QMD/semaphore/doorbell encoders selected by compute capability. The userspace driver builds the GPU command streams; the kernel driver allocates, maps, schedules and services interrupts.
• Synchronization model. Three primitives explain most of it: the tracking semaphore (per-channel counter of completed pushes), the marker ("all work before this point") and the QMD semaphore (released when a grid completes). Blocking waits use a per-context ISR thread and a DPC queue.

3. What it touches in a guest

This is the emulation contract. Device nodes. /dev/nvidiactl (process-wide control: version negotiation, client creation, GPU attach, OS events, exported-object fds), /dev/nvidiaN (per-GPU: object alloc/free, RM controls, memory alloc/map, event allocation), /dev/nvidia-uvm (managed memory and page migration) and /dev/nvidia-vgpuN (vGPU plugin node; NVIDIA's own vGPU stack only, not used by stock CUDA paths). ioctls. The control device uses _IOWR('F', cmd, …): 200 NV_ESC_CARD_INFO, 210 NV_ESC_CHECK_VERSION_STR, 212 NV_ESC_ATTACH_GPUS_TO_FD. RM escapes include 0x2b NV_ESC_RM_ALLOC (the main object-allocation path), 0x2a NV_ESC_RM_CONTROL, 0x29 NV_ESC_RM_FREE, 0x4e/0x4f NV_ESC_RM_MAP_MEMORY/UNMAP_MEMORY, 0x57/0x58 NV_ESC_RM_MAP_MEMORY_DMA/UNMAP_MEMORY_DMA and 0x4a NV_ESC_RM_VID_HEAP_CONTROL. The request word rsi=0xc030462b in a crash dump decodes to "allocating an RM object with a 48-byte payload". RM objects/classes. Every RM call names a class. Observed for 570.144 on an emulated TU104: NV01_ROOT (0x0), NV01_DEVICE_0 (0x80), NV20_SUBDEVICE_0 (0x2080), NV01_EVENT (0x40), KEPLER_CHANNEL_GROUP_A (0xa06c), a channel-group/runlist object (0x9067), TURING_CHANNEL_GPFIFO_A (0xc46f, the channel), TURING_COMPUTE_A (0xc5c0, the compute engine object), plus memory (0x3e) and VA-space classes. The observed cuCtxCreate order is root → device → subdevice → events → channel group → channel-group object → memory → channel → control NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST → compute object → engine registration. Controls. Device/GPU info (GPU_GET_ENGINES, GPU_GET_CLASSLIST, GPU_GET_VIRTUALIZATION_MODE, FB_GET_INFO, CE_GET_CAPS, …), FIFO/channels (NV0080_CTRL_CMD_FIFO_GET_CHANNELLIST 0x80170d, GPFIFO scheduling, timeslices, GET_CLASS_ENGINEID), GR/context switches, memory, events and interrupts, vGPU/licensing queries, and perf/clocks. The driver tolerates unknown-but-successful controls; it only fails when a specific reply field is consumed and wrong. Memory. VA reservation (NV_ESC_RM_VID_HEAP_CONTROL/NVOS32), backing allocation (vidmem/sysmem, pinned host memory, imported handles, UVM) and mapping (NVOS33 into the process, NVOS46 into a device VA space). All guest memory CUDA uses is either an RM allocation backed by emulated VRAM/BAR1 or pinned guest RAM the GPU must DMA to. Channels, GPFIFO, USERD, doorbells. A channel is a GPFIFO (array of pushbuffer descriptors) plus engine objects. The driver allocates the GPFIFO, error notifier and (Volta and later) the USERD itself, passing the handles to the channel allocation; submission writes the GPFIFO put pointer into USERD and rings a doorbell token returned by the channel allocation. Pushbuffers are method streams terminated by a tracking-semaphore release and a new GPFIFO entry; the host emulation decodes them as such. Synchronization below the API. Semaphores (plain GPU-visible memory values), 4 KB per-channel error notifiers, OS events (kernel objects used to wake blocked threads), interrupts routed through the kernel driver, and a trap handler for debuggers and device-side assert. UVM and GSP-RM. When UVM is available the driver registers GPUs, VA spaces, channels and managed allocations and pushes UVM semaphores into the same streams; without it CUDA degrades to zero-copy. GSP-RM changes nothing from userspace — it still issues RM ioctls; inside the kernel driver controls become GSP-RM RPCs, which Open-IOV's host module answers.

4. Why forwarding matters, and where we are

The performance budget is dominated by push-buffer execution, synchronization latency and (once, at setup) control-plane latency. Only forwarding real work to the physical GPU meets the first two; software execution of state/memory ops is a correctness bridge, not the data plane.

• Stated architecture: emulate the interface, pass through the work, never patch anyone. Prototype the host-RM/GSP proxy first for correctness, measure, then move the hot path to direct submission if the budgets demand it — work path abstracted from day one.
• Control plane today: GSP-RM emulation, RPC handling, BAR/PCI/MSI-X emulation and object/control replies are implemented; the stock guest stack gets through cuInit and device enumeration into cuCtxCreate.
• Current blocker: cuCtxCreate returns rc=999 (CUDA_ERROR_UNKNOWN) because the class-filtered engine list is missing the async copy engine COPY14 (descriptor 0x793ceb0e, instance 14, client type 0x38, class 0xc9b5). The oracle list has 7 engines (GR, CE0.CE4, COPY14); the guest list has 6. The driver silently skips engines with no per-(class, instance) database entry, then reads a zeroed secondary record and rolls back; what builds that database entry is the open question.
• Data plane today: the host module decodes guest push buffers as method streams and executes the small work it understands itself — memsets, copies and semaphore releases. No CUDA kernel has been forwarded yet.

5. The crash-path lesson: replies are consumed data

The old deterministic cuCtxCreate segfault at libcuda+0x453f90 shows why reply coherence matters. The engine-registration hook allocates a TURING_COMPUTE_A object with pAllocParms pointing at the caller's own 8-byte stack local and paramsSize = 0; the guest driver does not honour paramsSize, so it reads 16-byte class params starting there (the upper 8 bytes are the hook's saved rbp) and copies 16 bytes of reply params back over the same buffer. With the old zeroed reply that copy-back zeroed the saved rbp; the function's epilogue then loaded rbp = 0 and its next access — reloading a pointer from [rbp-0x38] — faulted at 0xffffffffffffffc8 (the store was the kernel's copy_to_user). The fix: echo the request's params back, forcing only status to NV_OK; the copy-back then writes exactly the bytes it read, and the fault disappears — the same thing the real GSP-RM does for these in/out params. The general rule: the userspace library is a consumer of emulated data. A wrong reply is worse than a slow one. Work up the reply-fidelity ladder — status-only → coherent zeroed params → coherent values for consumed fields → exact values — and validate each rung with the A/B oracle.

6. How to look things up

The toolchain is version-agnostic:

tools/cudatool.py info|exports|functions|nearest|crash|delta <lib>
tools/callwalk.py <callgraph.tsv> callers|callees|chain..

• cudatool.py — build id/sections/exports/ABI note (info), symbol map (exports), function ranges from .eh_frame_hdr (functions), containing function and nearest export for an address (nearest), dmesg ip to file vaddr (crash), and an interface-level diff (delta).
• callwalk.py — caller/callee trees and shortest call paths over the callgraph index.
• Annotated corpus — 570.144/annotations/names.tsv (our name map; exports exact, internals grow over time) and subsystems.tsv; corpus headers carry the recovered name, file address and direct callers/callees.

Address convention (read this before citing an offset). Public docs cite file virtual addresses as libcuda+0x... Ghidra loads these binaries at image base 0x100000, so program address = file vaddr + 0x100000; the tools/ wrappers translate automatically, raw Ghidra scripts do not. An earlier analysis passed file vaddrs straight to Ghidra and decompiled functions 0x100000 too low, so its decompiled-code conclusions were about the wrong functions (its live traces stayed valid). The same applies when transferring a name map to a new driver version.

7. Where the detail lives

Document	What it gives you
the reading order, fast recipes, and the two findings that change how you debug
the full layer/object/sync/memory model
build model, embedded device code, init/threading, what it touches
the emulation contract: nodes, ioctls, classes, controls, channels, UVM
NVIDIA vGPU vs Open-IOV, anti-fragility, performance, security
the binary map, the crash resolution, and the current engine-topology blocker
every tool, its usage and its cost
re-running the analysis on a new driver version
forwarding options and the honest current stage
the project documentation front door