kvm vga passthrough

tse | 2020-10-31 | | topics: Technology > notes | keywords: kvm vga linux

id: d310bc11-3ebf-4400-aa27-5af603d27064

My setup for VGA passthrough with KVM

I wanted to experiment with VGA passthrough once again on Linux. In addition to the doing the VGA passthrough I wanted to:

have a convenient way of passing input back and forth
forward certain controllers directly to the VM
force cpu pinning

This is a cheatsheet for the resources used, as well as the configuration options I ended up with.

Dependencies

In addition to the userland applications (virt-manager, qemu, etc) and services certain kernel options must be enabled:

https://wiki.gentoo.org/wiki/GPU_passthrough_with_libvirt_qemu_kvm
https://wiki.gentoo.org/wiki/Evdev
https://wiki.gentoo.org/wiki/QEMU

Once the setup is complete one can check it via the kvm-ok script provided by the cpu-checker package (https://manpages.ubuntu.com/manpages/xenial/man1/kvm-ok.1.html)

grub configuration

First we need to determine the ids of the pci devices we wish to passthrough (via lspci -nn), and append them to the GRUB_CMDLINE_LINUX in /etc/default/grub: iommu=on iommu=pt amd_iommu=on pcie_acs_override=downstream,multifunction kvm.ignore_msrs=1 vfio-pci.ids=10de:1b81,10de:10f0

dracut

add the following to /etc/dracut.conf.d/vfio.conf

hostonly="yes"
hostonly_cmdline="amd_iommu=on iommu=pt"
force_drivers+="vfio_pci vfio vfio_iommu_type1 vfio_virqfd"

polkit

add

daemon without authentication */
polkit.addRule(function(action, subject) {
    if (action.id == "org.libvirt.unix.manage" &&
        subject.isInGroup("kvm")) {
            return polkit.Result.YES;
    }
});

to /etc/polkit-1/rules.d/50-libvirt.rules

qemu.conf

Make sure that the user and group are set correctly

add the devices to be managed by evdev to the list cgroup_device_acl, in my case:

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm",
    "/dev/input/by-id/usb-04d9_USB_Keyboard-event-kbd",
    "/dev/input/by-id/usb-Corsair_Corsair_Gaming_M65_Pro_RGB_Mouse_0C04201CAF0C904259322463F5001942-event-mouse"
]

uncomment nographics_allow_host_audio = 1

/etc/libvirt/libvirtd.conf

unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"

/etc/libvirt/libvirt.conf

uri_default = "qemu:///system"

Add user to kvm and libvirt groups

gpasswd -a tse kvm
gpasswd -a tse libvirt

virtual machine .xml

We now need to configure the virtual machine so that we can make use of evdev to pass the devices back and forth to the virtual machine EDITOR=vim virsh edit win10 and add the following, before closing the domain tag:

<qemu:commandline>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/usb-04d9_USB_Keyboard-event-kbd,grab_all=on,repeat=on'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd21,evdev=/dev/input/by-id/ckb-Corsair_Gaming_M65_Pro_RGB_Mouse_vKB_-event'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/ckb-Corsair_Gaming_M65_Pro_RGB_Mouse_vM_-event'/>
    <qemu:arg value='-audiodev'/>
    <qemu:arg value='pa,id=snd0,server=/run/user/1000/pulse/native'/>
    <qemu:arg value='-mem-path'/>
    <qemu:arg value='/mnt/hugepages/'/>
</qemu:commandline>

This also enables hugepages (https://wiki.debian.org/Hugepages).

nvidia error 43

If you get the nvidia error 43 it can be fixed by adding the vendor_id tag under hyperv: <vendor_id state='on' value='123456789ab'/>

and the following needs needs to be added to the features (after hyperv) tag:

<kvm>
    <hidden state='on'/>
</kvm>
<ioapic driver='kvm'/>

CPU Pinning

In order to enable CPU pinning add the following:

  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='8' cpuset='15'/>
    <vcpupin vcpu='9' cpuset='4'/>
    <vcpupin vcpu='10' cpuset='16'/>
    <vcpupin vcpu='11' cpuset='5'/>
    <vcpupin vcpu='12' cpuset='17'/>
    <vcpupin vcpu='13' cpuset='6'/>
    <vcpupin vcpu='14' cpuset='18'/>
    <vcpupin vcpu='15' cpuset='7'/>
    <emulatorpin cpuset='0,19'/>
  </cputune>

(https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#CPU_pinning)

using VirtIO for mouse/keyboard:

https://passthroughpo.st/using-evdev-passthrough-seamless-vm-input/

<input type='mouse' bus='virtio'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
</input>
<input type='keyboard' bus='virtio'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>

Audio

In virt-manager I added the audio interface (kingston hyperx) to the vm.

USB controller

In order to have usb devices being automatically detected by the vm I decided to pass a usb controller. 1) list usb controllers: lspci |grep -i usb

2) list all the iommu groups (iommu_groups.sh) and make sure that we have a usb controller in an IOMMU group by itself:

#!/bin/bash
#https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

3) using the device_to_group.sh script check if the peripherals are connected to the correct controller:

#!/bin/bash

# from https://mathiashueber.com/usb-device-passthrough-setup-virtual-machines/

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

add devices in virt-manager

Once the usb controller pci device has been identified (together with the graphical card) they should be added in virt-manager (in my case 30:00.3 - usb controller -, 2e:00.0 and 2e:00.1 - nvidia -).

Startup script:

#!/bin/bash

if [ "${1}" = "start" ]; then
    mount /mnt/hugepages
    su -c 'sysctl vm.nr_hugepages=8192'
#    virsh -c qemu:///system start win10
    virsh -c qemu:///system create win10.xml
    exit 0
fi

if [ "${1}" = "stop" ]; then
    umount /mnt/hugepages
    su -c 'sysctl vm.nr_hugepages=0'
    exit 0
fi

echo "specify start or stop"
exit 1

win10.xml

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win10
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10</name>
  <uuid>5cbb13f2-969f-4e74-8d63-6b938f35004b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='8' cpuset='15'/>
    <vcpupin vcpu='9' cpuset='4'/>
    <vcpupin vcpu='10' cpuset='16'/>
    <vcpupin vcpu='11' cpuset='5'/>
    <vcpupin vcpu='12' cpuset='17'/>
    <vcpupin vcpu='13' cpuset='6'/>
    <vcpupin vcpu='14' cpuset='18'/>
    <vcpupin vcpu='15' cpuset='7'/>
    <emulatorpin cpuset='0,19'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/edk2-x86_64-code.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-model' check='partial'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/tse/Downloads/Win10_20H2_English_x64.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/nvme-eui.6479a73cd229cc55'/>
      <target dev='sdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='1' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:14:80:dc'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </input>
    <input type='keyboard' bus='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x2e' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x2e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0951'/>
        <product id='0x16a4'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x30' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='1'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <hub type='usb'>
      <address type='usb' bus='0' port='5'/>
    </hub>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/usb-04d9_USB_Keyboard-event-kbd,grab_all=on,repeat=on'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd21,evdev=/dev/input/by-id/ckb-Corsair_Gaming_M65_Pro_RGB_Mouse_vKB_-event'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/ckb-Corsair_Gaming_M65_Pro_RGB_Mouse_vM_-event'/>
    <qemu:arg value='-audiodev'/>
    <qemu:arg value='pa,id=snd0,server=/run/user/1000/pulse/native'/>
    <qemu:arg value='-mem-path'/>
    <qemu:arg value='/mnt/hugepages/'/>
  </qemu:commandline>
</domain>