[PVE-User] Radeon passthrough, migrating to vfio (PVE 3.4 to 4.3)

[Arjen]

Hi,

I'm having an problem at home with moving from Proxmox VE 3.4 to 4.3
with a GNU/Linux VM with VGA passthrough. I'm wondering if someone here
has experienced the same problem, or knows how to investigate it.

On Proxmox 3.4 with kernel 2.6.32, the a VM with Fedora 21 work
beautifully with various PCI passthrough devices: on-board audio, USB3
controller and a AMD Radeon 7750. This has enabled me to happily run my
desktop as a VM for several years (since Proxmox 3.2).

However, the same VM shows freezes the (same) host system running the
latest Proxmox 4.3. As soon as Xorg starts, the radeon driver reports
errors about ring 0 stalls and GPU lockup. It shows "random" colors on
the connected display, tries to reset the GPU and several seconds later
the host freezes to the point where it can only be reset by cutting
power. This makes it difficult to get proper logs from the VM.

vm.conf (abbreviated):
agent: 1
cores: 4
cpu: SandyBridge
hotplug: 0
machine: q35
memory: 6144
ostype: l26
scsihw: virtio-scsi-pci
sockets: 1
tablet: 0
vga: std
hostpci0: 01:00,pcie=1
hostpci1: 00:1b.0
# for PVE 4.3 -> hostpci2: 00:14.0
# for PVE 3.4 -> args: -device pci-assign,host=00:14.0,id=hostusb3

That VM works fine with PCI passthrough of the on-board audio device and
USB3 controller and using qxl or std as the video device. I also created
a new VM with a newer linux kernel (Ubuntu, kernel 4.8), which exhibits
the same problem. As other PCI device passthrough works fine with other
VM's on both versions of Proxmox VE, I'm guessing it is an interaction
between the Linux radeon driver and vfio (which differs from Proxmox 3.4
style passthrough).

I read the Proxmox wiki about PCI passthrough, but because of hardware
limitations, I can only use SeaBIOS + q35 because the radeon kernel
module assumes a PCIE root device and the ROM does not support UEFI.
I was not able to find any information about such problems on the
internet. I found reports about radeon ring 0 stalls and GPU lockup, but
they never involved virtualization and their work-arounds did not help me.

Has anyone run into this or something similar? And have you found a fix?
Thank you for reading, and for any pointers that you may have for me.

kind regards,
Arjen

[Arjen]

Apologies for replying to myself, but it looks like this is not an issue
with Proxmox VE 4. The Linux radeon driver appears not just to work on
Linux guests with my graphics card when using VFIO. (It also does not
work with PVE kernel 3.10.)

Has anyone here been able to successfully passthrough a AMD Radeon (GCN
1.x) graphics card to a Linux guest?

kind regards, Arjen

Post by Arjen
Hi,
I'm having an problem at home with moving from Proxmox VE 3.4 to 4.3
with a GNU/Linux VM with VGA passthrough. I'm wondering if someone here
has experienced the same problem, or knows how to investigate it.
On Proxmox 3.4 with kernel 2.6.32, the a VM with Fedora 21 work
beautifully with various PCI passthrough devices: on-board audio, USB3
controller and a AMD Radeon 7750. This has enabled me to happily run my
desktop as a VM for several years (since Proxmox 3.2).
However, the same VM shows freezes the (same) host system running the
latest Proxmox 4.3. As soon as Xorg starts, the radeon driver reports
errors about ring 0 stalls and GPU lockup. It shows "random" colors on
the connected display, tries to reset the GPU and several seconds later
the host freezes to the point where it can only be reset by cutting
power. This makes it difficult to get proper logs from the VM.
agent: 1
cores: 4
cpu: SandyBridge
hotplug: 0
machine: q35
memory: 6144
ostype: l26
scsihw: virtio-scsi-pci
sockets: 1
tablet: 0
vga: std
hostpci0: 01:00,pcie=1
hostpci1: 00:1b.0
# for PVE 4.3 -> hostpci2: 00:14.0
# for PVE 3.4 -> args: -device pci-assign,host=00:14.0,id=hostusb3
That VM works fine with PCI passthrough of the on-board audio device and
USB3 controller and using qxl or std as the video device. I also created
a new VM with a newer linux kernel (Ubuntu, kernel 4.8), which exhibits
the same problem. As other PCI device passthrough works fine with other
VM's on both versions of Proxmox VE, I'm guessing it is an interaction
between the Linux radeon driver and vfio (which differs from Proxmox 3.4
style passthrough).
I read the Proxmox wiki about PCI passthrough, but because of hardware
limitations, I can only use SeaBIOS + q35 because the radeon kernel
module assumes a PCIE root device and the ROM does not support UEFI.
I was not able to find any information about such problems on the
internet. I found reports about radeon ring 0 stalls and GPU lockup, but
they never involved virtualization and their work-arounds did not help me.
Has anyone run into this or something similar? And have you found a fix?
Thank you for reading, and for any pointers that you may have for me.
kind regards,
Arjen