[PVE-User] Slow live migration in Proxmox 5

[Sten Aus]

Hi

Has anyone seen migration speed drop when upgrading from Proxmox 4 to
Proxmox 5?

I have seen at least one poor network performance issue in Proxmox
Forums
(https://forum.proxmox.com/threads/pve-5-2-netlink-ovs-vswitchd-attribute-type-5-has-an-invalid-length.44039/#post-224515).


I have one small cluster (2 nodes) with 1x10Gbit network each and I get
migration speeds 30-40MB/s. In Proxmox 4 I got 100+MB/s, which is more
sensible for 10Gbps. I am using openvswitch. Cluster has it's own
network for multicast and migration.

I have replaced "migartion_unsecure: 1" with "migration: type=insecure"
in datacenter.cfg, but this seems to have no effect. There's always
lines in migration log: "ssh tunnel ver 1".

Only if I add both of them to datacenter.cfg I will get log saying:
"deprecated setting 'migration_unsecure' and new 'migration: type' set
at same time! Ignore 'migration_unsecure'"


But if I remove both lines (old deprecated one and new one) I will get
bigger speeds:
2018-10-09 11:07:24 migration speed: 136.53 MB/s - downtime 49 ms
Which is weird.


Not sure if I can update bigger cluster to Proxmox 5 now.

Has anyone else seen this?


All the best
Sten


---

# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-5-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-8
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-2.6.32-47-pve: 2.6.32-179
pve-kernel-2.6.32-46-pve: 2.6.32-177
pve-kernel-2.6.32-42-pve: 2.6.32-165
ceph: 11.2.1-1~bpo90+1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-29
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

[Woods, Ken A (DNR)]

Just to be clear, if the migration took ~5 minutes under v4, now they’re taking ~15 minutes?

Hi
Has anyone seen migration speed drop when upgrading from Proxmox 4 to Proxmox 5?
I have seen at least one poor network performance issue in Proxmox Forums (https://urldefense.proofpoint.com/v2/url?u=https-3A__forum.proxmox.com_threads_pve-2D5-2D2-2Dnetlink-2Dovs-2Dvswitchd-2Dattribute-2Dtype-2D5-2Dhas-2Dan-2Dinvalid-2Dlength.44039_-23post-2D224515&d=DwIGaQ&c=teXCf5DW4bHgLDM-H5_GmQ&r=THf3d3FQjCY5FQHo3goSprNAh9vsOWPUM7J0jwvvVwM&m=_OmRthCFKEgaCZ7o1JJB1aPyRAGN6sWZLhdiSksIhAo&s=Q-RwTCu10LxUglOZwvVP72npd2y6a3o8yT-6S19uCzI&e=).
I have one small cluster (2 nodes) with 1x10Gbit network each and I get migration speeds 30-40MB/s. In Proxmox 4 I got 100+MB/s, which is more sensible for 10Gbps. I am using openvswitch. Cluster has it's own network for multicast and migration.
I have replaced "migartion_unsecure: 1" with "migration: type=insecure" in datacenter.cfg, but this seems to have no effect. There's always lines in migration log: "ssh tunnel ver 1".
"deprecated setting 'migration_unsecure' and new 'migration: type' set at same time! Ignore 'migration_unsecure'"
2018-10-09 11:07:24 migration speed: 136.53 MB/s - downtime 49 ms
Which is weird.
Not sure if I can update bigger cluster to Proxmox 5 now.
Has anyone else seen this?
All the best
Sten
---
# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-5-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-8
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-2.6.32-47-pve: 2.6.32-179
pve-kernel-2.6.32-46-pve: 2.6.32-177
pve-kernel-2.6.32-42-pve: 2.6.32-165
ceph: 11.2.1-1~bpo90+1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-29
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
_______________________________________________
pve-user mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__pve.proxmox.com_cgi-2Dbin_mailman_listinfo_pve-2Duser&d=DwIGaQ&c=teXCf5DW4bHgLDM-H5_GmQ&r=THf3d3FQjCY5FQHo3goSprNAh9vsOWPUM7J0jwvvVwM&m=_OmRthCFKEgaCZ7o1JJB1aPyRAGN6sWZLhdiSksIhAo&s=cZODwDqx16RHiin42bV1xSJw3xRBOyCzxx_fJgULEdI&e=

[Kai Zimmer]

Hi Sten,

not exactly your problem, but maybe related: after upgrading from PVE
4.2 to PVE 5.1.2 i see unresponsive/unreachable VMs during high disk i/o.
What i tried so far is :
- disabling  the Spectre/Meltdown patches which steal some performance
(up to 20%) via kernel parameters.
- adding 'nobarrier'-parameter on PVE host ext4 partitions (because i
use qcow2 disk images)
Both didn't fix it.
Now i'm about to update LSI Controller Firmware and drivers, because we
see a similar problem after a Kernel upgrade from 4.x to 5.x on a
non-Proxmox machine.

So maybe it's not the network but a disk speed problem in your case also?

Best,
Kai

Post by Sten Aus
Hi
Has anyone seen migration speed drop when upgrading from Proxmox 4 to
Proxmox 5?
I have seen at least one poor network performance issue in Proxmox
Forums
(https://forum.proxmox.com/threads/pve-5-2-netlink-ovs-vswitchd-attribute-type-5-has-an-invalid-length.44039/#post-224515).
I have one small cluster (2 nodes) with 1x10Gbit network each and I
get migration speeds 30-40MB/s. In Proxmox 4 I got 100+MB/s, which is
more sensible for 10Gbps. I am using openvswitch. Cluster has it's own
network for multicast and migration.
type=insecure" in datacenter.cfg, but this seems to have no effect.
There's always lines in migration log: "ssh tunnel ver 1".
"deprecated setting 'migration_unsecure' and new 'migration: type' set
at same time! Ignore 'migration_unsecure'"
But if I remove both lines (old deprecated one and new one) I will get
2018-10-09 11:07:24 migration speed: 136.53 MB/s - downtime 49 ms
Which is weird.
Not sure if I can update bigger cluster to Proxmox 5 now.
Has anyone else seen this?
All the best
Sten
---
# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-5-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-8
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.4.134-1-pve: 4.4.134-112
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.62-1-pve: 4.4.62-88
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-2.6.32-47-pve: 2.6.32-179
pve-kernel-2.6.32-46-pve: 2.6.32-177
pve-kernel-2.6.32-42-pve: 2.6.32-165
ceph: 11.2.1-1~bpo90+1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-29
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user