Install and First Run

Install and First Run

What this page is. From a fresh Ubuntu host with a supported NVIDIA GPU to a guest VM running a stock NVIDIA driver and nvidia-smi, in about half an hour. Condensed from the maintained guide ../docs/02-getting-started/01-install-and-first-run.md; the docs tree wins on disagreement.

Status: 2026-10-12, tree master @ 2f066727. Tags: [live] verified path; [target] roadmap item.

1. Prerequisites

Host

• Ubuntu 22.04 or 24.04, x86_64 (aarch64 is a [target]);
• an NVIDIA GPU covered by the current emulation profile (Turing TU104 is the reference — see Hardware-Software-Matrix.md);
• a working host NVIDIA driver (nvidia-smi runs on the host; Open-IOV does not modify it);
• IOMMU enabled (intel_iommu=on / amd_iommu=on), hugepages configured, kernel headers for the running kernel;
• qemu-system-x86_64 with KVM; libvirt optional;
• root (or sudo) for module load and mdev creation.

Guest

• Ubuntu 24.04 server (the reference guest), or any Linux with the matching stock NVIDIA driver available;
• no Open-IOV component at all — that is the point (System-Overview.md).

2. Get the code and build

git clone https://git.vgpu.io/open-iov.git
cd open-iov
./install.sh deps        # build/runtime dependencies
./install.sh nvidia      # optional: NVIDIA OpenRM modules + build glue
./install.sh firmware    # GSP firmware image (verified)
./install.sh build       # kernel module + userspace

install.sh is safe to re-run. The manual equivalent, and the order that matters:

make kernel                      # first: generates Module.symvers
./scripts/build_nvidia_openrm.sh # optional (see below)
make userspace

Build order matters. NVIDIA's OpenRM modules resolve symbols that vgsp.ko exports; building them in the wrong order produces Unknown symbol failures at load time.

The OpenRM stage is optional. It is needed only for the native vGPU-management workflows in which NVIDIA's modules are built against the symbols vgsp.ko exports. The QEMU/emulated path uses the distribution driver that already manages the physical GPU; load-stack skips the OpenRM submodule when a distribution driver is present.

3. Load the stack and create a virtual GPU

./scripts/stack_manager.sh status        # what is currently loaded
./scripts/stack_manager.sh load-stack    # vfio/mdev prereqs -> vgsp.ko -> (optional) nvidia
./scripts/stack_manager.sh mdev-create   # create a vGPU mediated device
./scripts/stack_manager.sh mdev-list     # inspect /sys/bus/mdev/devices/

The mdev device is the virtual GPU; everything after this treats it like any other VFIO device.

4. Launch a reference guest

./scripts/stack_manager.sh qemu          # reference raw-QEMU guest with the mdev

To integrate with your own stack (libvirt, Xen, ...), attach the mdev exactly as any VFIO device:

<hostdev mode='subsystem' type='mdev' model='vfio-pci'>
  <source>
    <address uuid='<mdev-uuid>'/>
  </source>
</hostdev>

Attach behind a root port, not on the root bus. QEMU rewrites a device's PCIe capability when it detects a root-complex-integrated endpoint, which changes the link capabilities the guest driver sees. The reference script does this for you.

5. Install the stock driver in the guest

Inside the guest:

sudo apt update
sudo apt install -y linux-headers-generic build-essential
# The validated combination is the stock 570.144 Linux runfile (the same
# driver version as the emulation target).
sudo sh ./NVIDIA-Linux-x86_64-570.144.run
sudo reboot

Verify:

nvidia-smi                # must list the virtual GPU
lsmod | grep -c nvidia

If nvidia.ko fails to probe, capture host dmesg, guest dmesg, guest lspci -vvv and the guest module parameters, and file an issue.

6. First CUDA check

# in the guest
git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/1_Utilities/deviceQuery && make && ./deviceQuery

Expectation today ([live] unless noted): driver probe, nvidia-smi, device enumeration and the full CUDA driver-API smoke complete end to end (ALL OK, 64-byte payload round-trips). CUDA kernel execution is the current frontier: a guest launch is forwarded and reaches the engine queue, but engine execution is not yet observed ([wall]) — see Project-Status.md and Execution-Frontier.md before drawing conclusions from a kernel that does not run.

7. Day-to-day operations

./scripts/stack_manager.sh up          # build all + load + create mdev
./scripts/stack_manager.sh status      # modules, mdev, vfio, firmware
./scripts/stack_manager.sh test        # host test suite
./scripts/stack_manager.sh unload      # correct-order teardown
./scripts/stack_manager.sh cleanup-crash

Load order is prerequisites → kernel module → (optional) NVIDIA OpenRM → firmware; unload is the reverse. Do not rmmod by hand while a VM holds the mdev.

8. Watching the emulation work

The module logs with the prefix vgsp-<subsystem>: (for example vgsp-rpc:, vgsp-sema:, vgsp-work:) plus per-RPC traces:

sudo dmesg -w | grep -i 'vgsp'          # live
sudo dmesg -T | grep -i 'vgsp' | tail -50

Developer debug knobs (module parameters) are documented in the repository's debugging workflow document.

9. Troubleshooting index

Symptom	Start here
Module will not load	dmesg, modinfo vgsp.ko, stale module state → cleanup-crash
No mdev type for your GPU	Hardware-Software-Matrix.md
Guest sees wrong PCIe link speed/width	attach behind a root port (section 4)
nvidia.ko probe fails in guest	guest dmesg + lspci -vvv + Driver-Lifecycle-Field-Guide.md
nvidia-smi works, a CUDA kernel does not run	expected today — the execution frontier, not your setup (Execution-Frontier.md)
VM boot fails or the guest hangs	scripts/stack_manager.sh cleanup-crash, then reboot

10. Uninstall

./scripts/stack_manager.sh unload

The project never modifies your host driver's files; removing the module and its mdev devices returns the host to its previous state.