GMMU Multi-Tenant Protection

GMMU Multi-Tenant Protection

Current — companion summary. This page condenses ../docs/03-architecture/14-multi-tenant-gmmu-protection.md (the canonical document: problem statement, full design, RM struct sketches, walked example, portability, M1–M8 roadmap). Read the canonical document for anything you intend to build against; this page is the quick map.

Overview

Open-IOV must isolate multiple mdevs (VMs) and, for multi-tenant inference, multiple tenants inside one guest at the GMMU. The design has two halves:

1. Service-state ownership (baseline 1.0). The work/completion service (kernel/gsp_sema_service.c) is one global context (static struct sema_svc svc, :676), the pump services the first open mdev (:13352-13364), and channels/VA spaces/translations are keyed by guest-chosen values alone (:847, :988, :113-130). Two guests can pick the same handles and VAs (live: channel 0xcaf000d4; ../debug/issue/issue-number-2.md), so owner-keying is a prerequisite for any GMMU policy.
2. GMMU policy for paged attention (research). A tenant/agent-aware KV cache protection design across vLLM, a guest utility, RM controls, the vEP, and page tables — post-1.0, dual-gated; never a baseline dependency.

Isolation layers

Layer	Mechanism	Status
L0 platform	IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary	live
L1 transport	VFIO mdev per tenant, config/BAR/IRQ mediation	live
L2 vEP state	per-mdev emulation context + (needed) per-mdev work service	live / design
L3 GMMU	per-context page directory (VA space)	live (guest-owned) / design (host-owned)
L4 arbitration	channel ownership, TSG/runlist policy	design
L5 completion	owner-keyed release resolution; no cross-tenant apply	wip
L6 fault	fault attributed to the owning channel/tenant only	wip
L7 application	per-tenant/per-agent VA pooling inside a guest	research

Hardware primitives: proven vs hypothesis

Primitive	Status
4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0	proven (docs/43-GMMU-PTE-DECODER; kernel/gsp_sema_service.c:1058-1072)
VER2 PTE fields incl. read_only (6), privilege (5), encrypted (4), kind	proven format (docs/43 §3)
Separate page directory per VA space (SET_PAGE_DIRECTORY with hVASpace/chId)	proven (kernel/gsp_rm_emulation.c:3310-3314)
No tenant/ACL/PASID field in the VER2 PTE/PDE	proven absence in the field catalogue
RO-leaf write enforcement	hypothesis (needs a supervised probe; M7)
Encryption-domain (CCS) as a tenant boundary	hypothesis

The design in brief

• Design A (stock-compatible). Extended vLLM reserves a VA range per agent; a new guest utility (oiv-agentd) publishes tag records to a device-memory tag mailbox the vEP can read (emulation mode: guest vidmem is host RAM, kernel/bar_init.c:68-77). The vEP enforces owner↔VA-subtree match at map time and at submit time. No guest KMD change. Research (M6).
• Design B (explicit controls). New vGPU-private RM controls — TENANT_BIND (0x90f10120), TAG_RANGE (…21), UNTAG_RANGE (…22), TENANT_SHARE (…23), AGENT_CHANNEL_BIND (…24), TAG_QUERY (…25) — plus struct sketches in the canonical doc §4.5. Requires a guest KMD extension: post-1.0, gated (M8).
• Granularities. Per-KV-block tagging (finest; software+PTE policy, sharing needs RO enforcement); per-tenant separate page directories (the only proven hardware lever); per-agent separate channels (fault/QoS containment; optional per-agent sub-directories). Degradation ladder and per-granularity cost/failure analysis in canonical §4.7.

Walked example (two tenants, paged_attention_v2)

Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP submit-time policy; the fault is attributed to A1 only (T2 untouched); a periodic owner-vs-PTE audit with canaries catches escapes. If only per-tenant separation exists, the deployable contract is one agent per tenant, one tenant per context (canonical §5.4).

Key code anchors (master 1f571dda)

What	Where
per-mdev emulation context	kernel/include/nvidia-vgpu-vfio.h:133, kernel/gsp_emulation.h:130
global service state	kernel/gsp_sema_service.c:676; pump :13352; doorbell :13512
channel/vaspace registries	kernel/gsp_sema_service.c:847, :988
GMMU walk (VER2)	kernel/gsp_sema_service.c:1174; decode :1093
page directory record	kernel/gsp_rm_emulation.c:3320; reserved PDEs :2237-2262
per-mdev FB sizing	kernel/bar_init.c:60-77; kernel/include/ioctl.h:39-46
L2 invalidate acked	kernel/gsp_rm_emulation.c:2519-2532

Roadmap (acceptance criteria in canonical §7.3)

M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the 1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- replace, never required by the baseline. RM 610 stays behind its own human gate (docs/23-RM610-READINESS).

Related documentation

• Introduction-to-GMMU.md — from-first-principles GMMU introduction (what it is, page tables, IOMMU/MMU/SMMU relationships)
• ../docs/03-architecture/14-multi-tenant-gmmu-protection.md — canonical
• ../docs/03-architecture/13-vexecution-plane.md — vEP vocabulary (parallel)
• ../docs/03-architecture/05-gmmu-amap-hal-gr.md — GMMU background
• ../docs/03-architecture/09-full-system-picture.md — multi-tenancy and isolation map
• ../docs/03-architecture/11-physical-backend-guest-wiring.md — forwarding and page-table seams
• ../docs/05-operations/01-performance-correctness-security.md — threat model, rung 5
• ../debug/issue/issue-number-2.md — per-client state wedge
• ../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md — GMMU field catalogue