Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 24 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 24 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Per-VF degradation — one unreadable VF is skipped with a warning instead of failing discovery or profile listing wholesale, so a flaky sysfs read cannot blank the host's advertised GPU capacity. Only when no VF is readable does discovery fail, so a wholesale outage cannot demote a vGPU host to passthrough while assignments exist. The open-handle probe stays strict on purpose: it authorizes clearing a reused VF path, so an incomplete scan fails the release rather than risk a false "not in use".
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

High Risk
Touches GPU assignment, sysfs writes that can clear live VF types, and resource/admission reporting. A discovery or release bug could steal capacity, demote a vGPU host to passthrough, or wipe another instance’s assignment.

Overview
Adds NVIDIA’s vendor VFIO vGPU path (Linux 6.8 + R580: write type IDs to nvidia/current_vgpu_type) next to mdev, and routes discovery, profile listing, destroy, and reconcile through DiscoverVGPU.

Placement uses creatable_vgpu_types, least-loaded GPU selection, and create/verify/rollback. Availability is a best-effort count of free VFs advertising a type. Release is guarded by an in-process owner map plus a strict open-VFIO-handle scan so a reused VF path is not cleared while a VMM holds it. Reconcile drops orphans unless they are protected or still open; vendor VFIO reconcile no-ops if the protected set is missing.

Unreadable VFs are skipped so one bad sysfs node cannot blank capacity; discovery fails only when no VF is readable, so a vGPU host is not demoted to passthrough. CreateVGPU still rejects vendor VFIO until lifecycle persistence lands. Resource status now prefers vGPU discovery over passthrough, and docs/tests cover stop/start assignment plus the new sysfs backend.

Reviewed by Cursor Bugbot for commit e44df0b. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5b47670 to ffbf8a0 Compare August 7, 2026 20:52
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ffbf8a0 to 78801e7 Compare August 8, 2026 01:05
Comment thread integration/vgpu_test.go Outdated
@yummybomb

Copy link
Copy Markdown
Contributor Author

added 7f5233f: report GPUProfile.Available as the count of free VFs advertising the profile type (creatable-instance units), matching mdev's summed available_instances, the OpenAPI description, and the integration test's decrement assertion. note the gpuProfileSlots metric steps up on vendor VFIO hosts (per-GPU → per-free-VF units). go test ./lib/devices ./lib/resources green; the hardware integration test was not run locally.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7f5233f to 0baacb3 Compare August 10, 2026 07:26
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at abe29bb (pushed)

go get github.com/kernel/hypeman-go-staging@abe29bbba87d87198b10ccfbf4678d350739b146
python code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

230 files generated at 1e0016e (pushed)

typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at db26009 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76BJxZSD-steep-span
Timestamp2026-08-20T22:09:00.892Z
stlc8413509
Spec hash101b9ae4ce99
Config hash55e15f6f4434

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-20 22:09:31 UTC

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch 2 times, most recently from 1f1f24f to c2bcb69 Compare August 20, 2026 20:17
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from c2bcb69 to 1f1f24f Compare August 20, 2026 21:57
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
The instance lifecycle already routes create/start/stop/delete through
CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would
activate the backend before assignment durability and release guards
exist. Reject vendor VFIO creates for now; destroy stays wired so
existing assignments remain releasable. The integration test skips on
vendor VFIO hosts at this layer and no longer asserts the transitional
stop-retention behavior.
Counting every free VF advertising a type overreports concurrent
capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q
assignment revokes the type from every other VF on that GPU. Bound each
GPU's contribution by both its free VFs and how many times the profile
framebuffer fits into the GPU's remaining framebuffer, using the largest
still-creatable profile as a lower bound on what remains.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and
GetGPUStatus turns a discovery error into a host with no GPU, so one
flaky sysfs read blanked out the host's entire GPU capacity for
admission and monitoring.

Skip unreadable VFs with a warning and keep the readable inventory: a
skipped VF is never selected for placement and never reconciled, both
safe directions. When no VF is readable, discovery still fails so a
wholesale sysfs outage cannot demote a vGPU host to passthrough while
assignments exist.

Also document that vendor VFIO vGPUs are known broken on Cloud
Hypervisor upstream and QEMU is the required hypervisor for GPU
instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read
failed, blanking every advertised profile while discoverVFs directly
above it already skips unreadable VFs for exactly that reason. Skip and
warn instead; underreporting is the safe direction for status and
admission.

Also document why openVFIOPaths stays strict where mdev's scan is lax
(it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat
in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement
wholesale when profileMetadata or selectLeastLoadedVF hit the same VF,
so /resources could advertise capacity a create then failed to use.
Skip the VF in both loops; it simply stops being a placement candidate.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 1f1f24f to d28e3d0 Compare August 20, 2026 22:04

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d28e3d0. Configure here.

log.WarnContext(ctx, "preserving vendor VFIO vGPU held open without a live instance claim", "vf", vf.PCIAddress)
continue
}
if err := s.destroyWithOpenPaths(ctx, vf.PCIAddress, "", openPaths); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale open-handle scan race

Medium Severity

reconcile takes one openVFIOPaths snapshot and passes it into destroyWithOpenPaths, which skips a fresh scan under vendorVFIOMu when that map is non-nil. A VMM can open the VF after the snapshot and before the current_vgpu_type clear, so reconcile can wipe an assignment a running guest already holds. The normal destroy path avoids this by passing nil and rescanning under the lock.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit d28e3d0. Configure here.

A process that exits between the /proc listing and its fd walk surfaces
ENOENT or ESRCH; it holds nothing open, so skipping it cannot produce a
false "not in use" answer. Everything else still fails the scan closed.
An mdev assignment carrying neither a UUID nor a device path would
resolve to DestroyMdev("."); release nothing instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants