GMMU Multi-Tenant Protection
GMMU Multi-Tenant Protection
Current — companion summary. This page condenses
../docs/03-architecture/14-multi-tenant-gmmu-protection.md(the canonical document: problem statement, full design, RM struct sketches, walked example, portability, M1–M8 roadmap). Read the canonical document for anything you intend to build against; this page is the quick map.
Overview
Open-IOV must isolate multiple mdevs (VMs) and, for multi-tenant inference, multiple tenants inside one guest at the GMMU. The design has two halves:
- Service-state ownership (baseline 1.0). The work/completion service (
kernel/gsp_sema_service.c) is one global context (static struct sema_svc svc,:676), the pump services the first open mdev (:13352-13364), and channels/VA spaces/translations are keyed by guest-chosen values alone (:847,:988,:113-130). Two guests can pick the same handles and VAs (live: channel0xcaf000d4;../debug/issue/issue-number-2.md), so owner-keying is a prerequisite for any GMMU policy. - GMMU policy for paged attention (research). A tenant/agent-aware KV cache protection design across vLLM, a guest utility, RM controls, the vEP, and page tables — post-1.0, dual-gated; never a baseline dependency.
Isolation layers
| Layer | Mechanism | Status |
|---|---|---|
| L0 platform | IOMMU/SMMU per-VM DMA, KVM memory/vCPU boundary | live |
| L1 transport | VFIO mdev per tenant, config/BAR/IRQ mediation | live |
| L2 vEP state | per-mdev emulation context + (needed) per-mdev work service | live / design |
| L3 GMMU | per-context page directory (VA space) | live (guest-owned) / design (host-owned) |
| L4 arbitration | channel ownership, TSG/runlist policy | design |
| L5 completion | owner-keyed release resolution; no cross-tenant apply | wip |
| L6 fault | fault attributed to the owning channel/tenant only | wip |
| L7 application | per-tenant/per-agent VA pooling inside a guest | research |
Hardware primitives: proven vs hypothesis
| Primitive | Status |
|---|---|
| 4-level tree, 49-bit VA, 4K/64K/2M pages, dual PDE0 | proven (docs/43-GMMU-PTE-DECODER; kernel/gsp_sema_service.c:1058-1072)
|
VER2 PTE fields incl. read_only (6), privilege (5), encrypted (4), kind
|
proven format (docs/43 §3)
|
Separate page directory per VA space (SET_PAGE_DIRECTORY with hVASpace/chId)
|
proven (kernel/gsp_rm_emulation.c:3310-3314)
|
| No tenant/ACL/PASID field in the VER2 PTE/PDE | proven absence in the field catalogue |
| RO-leaf write enforcement | hypothesis (needs a supervised probe; M7) |
| Encryption-domain (CCS) as a tenant boundary | hypothesis |
The design in brief
- Design A (stock-compatible). Extended vLLM reserves a VA range per agent; a new guest utility (
oiv-agentd) publishes tag records to a device-memory tag mailbox the vEP can read (emulation mode: guest vidmem is host RAM,kernel/bar_init.c:68-77). The vEP enforces owner↔VA-subtree match at map time and at submit time. No guest KMD change. Research (M6). - Design B (explicit controls). New vGPU-private RM controls —
TENANT_BIND(0x90f10120),TAG_RANGE(…21),UNTAG_RANGE(…22),TENANT_SHARE(…23),AGENT_CHANNEL_BIND(…24),TAG_QUERY(…25) — plus struct sketches in the canonical doc §4.5. Requires a guest KMD extension: post-1.0, gated (M8). - Granularities. Per-KV-block tagging (finest; software+PTE policy, sharing needs RO enforcement); per-tenant separate page directories (the only proven hardware lever); per-agent separate channels (fault/QoS containment; optional per-agent sub-directories). Degradation ladder and per-granularity cost/failure analysis in canonical §4.7.
Walked example (two tenants, paged_attention_v2)
Two agents A1/A2 each own a VA pool and channels. A malicious T1 kernel naming an A2 VA hits a missing PTE (GMMU fault) and/or is refused by vEP submit-time policy; the fault is attributed to A1 only (T2 untouched); a periodic owner-vs-PTE audit with canaries catches escapes. If only per-tenant separation exists, the deployable contract is one agent per tenant, one tenant per context (canonical §5.4).
Key code anchors (master 1f571dda)
| What | Where |
|---|---|
| per-mdev emulation context | kernel/include/nvidia-vgpu-vfio.h:133, kernel/gsp_emulation.h:130
|
| global service state | kernel/gsp_sema_service.c:676; pump :13352; doorbell :13512
|
| channel/vaspace registries | kernel/gsp_sema_service.c:847, :988
|
| GMMU walk (VER2) | kernel/gsp_sema_service.c:1174; decode :1093
|
| page directory record | kernel/gsp_rm_emulation.c:3320; reserved PDEs :2237-2262
|
| per-mdev FB sizing | kernel/bar_init.c:60-77; kernel/include/ioctl.h:39-46
|
| L2 invalidate acked | kernel/gsp_rm_emulation.c:2519-2532
|
Roadmap (acceptance criteria in canonical §7.3)
M1 per-mdev service state — M2 two-guest isolation gate — M3 per-mdev real GSP clients + fairness — M4 independent VRAM sizes/sparse backing — M5 invalidate + eviction/reclaim (I1–I6 invariants) — M6 Design A prototype — M7 RO/permission probe — M8 Design B / CCS / 610 (gated). M1–M5 are the 1.0 programme; M6–M8 are post-1.0 research, dual-gated, duplicate-not- replace, never required by the baseline. RM 610 stays behind its own human gate (docs/23-RM610-READINESS).
Related documentation
Introduction-to-GMMU.md— from-first-principles GMMU introduction (what it is, page tables, IOMMU/MMU/SMMU relationships)../docs/03-architecture/14-multi-tenant-gmmu-protection.md— canonical../docs/03-architecture/13-vexecution-plane.md— vEP vocabulary (parallel)../docs/03-architecture/05-gmmu-amap-hal-gr.md— GMMU background../docs/03-architecture/09-full-system-picture.md— multi-tenancy and isolation map../docs/03-architecture/11-physical-backend-guest-wiring.md— forwarding and page-table seams../docs/05-operations/01-performance-correctness-security.md— threat model, rung 5../debug/issue/issue-number-2.md— per-client state wedge../debug/risc-v-gsp-rm-vm/docs/43-GMMU-PTE-DECODER.md— GMMU field catalogue