[PVE-User] high cpu load on 100mbits/sec download with virtio nic

[Maxime AUGER]

Thank you for your statement

I wget on the debian8 GUEST

-> GUEST-CPU = 25% of 1x vcpu
-> HOST-CPU = 30% of 2x cores (each one)

Ex with "-O /dev/null" : host TOP (the cpu decreases a little bit)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6833 root 20 0 1972312 892460 5900 S 31.1 43.7 184:54.60 /usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid -daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a -s+
6772 root 20 0 0 0 0 R 11.9 0.0 1:37.54 [vhost-6712]
6834 root 20 0 1972312 892460 5900 S 7.9 43.7 173:24.10 /usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid -daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a -s+
6712 root 20 0 1972312 892460 5900 S 1.3 43.7 7:50.63 /usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid -daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a -s+




-----Message d'origine-----
De : pve-user [mailto:pve-user-***@pve.proxmox.com] De la part de Mark Schouten
Envoyé : jeudi 7 juin 2018 17:03
À : PVE User List <pve-***@pve.proxmox.com>
Objet : Re: [PVE-User] high cpu load on 100mbits/sec download with virtio nic

I'm doubtful about the performance I get.
On proxmox HOST a single wget download consume 12% single CPU load at
11MBytes/sec (1518 packets size)

Where are you wgetting to? Benchmarking this is better with iperf or so.

If you want to wget, do wget -O /dev/null so you're not suffering from diskperformance..

[Maxime AUGER]

Hello,

Let me clarify my statement.
GUEST CPU load is acceptable (25% of a single CPU)
It is the cumulative load of the kvm process and the vhost thread that is high, on the HOST side
kvm-thread-1=30%
kvm-thred-2=30%
vhost(net)=10%
70% CPU without i/o disk (-O /dev/null)

it is x10 the load observed with the same conditions under old Vmware ESXi (same GUEST system and wget process)
I think it is a kvm issue, but I'm curious about proxmox's positioning in relation to this performance level


-----Message d'origine-----
De : pve-user [mailto:pve-user-***@pve.proxmox.com] De la part de Josh Knight
Envoyé : jeudi 7 juin 2018 20:47
À : PVE User List <pve-***@pve.proxmox.com>
Objet : Re: [PVE-User] high cpu load on 100mbits/sec download with virtio nic

I'm not convinced this is a proxmox issue, or even an issue to begin with.

I'm running proxmox 5.1-49, in my Linux 4.1 guest when I run wget -O /dev/null <https to ~1.2GB iso> I'm seeing ~30% according to top as well.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12788 root 20 0 65512 7616 5988 S 31.3 0.1 0:03.00 wget


Even on a physical box running Ubuntu I'm getting around 20-30% or more.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36070 root 20 0 32888 4028 3576 S 37.6 0.0 0:03.15 wget


This could be an issue/quirk with the way top is calculating cpu usage, or it's just a process using available cpu as normal. I couldn't reproduce high load using ping -f from a remote host to my VM, and iotop confirmed that -O /dev/null wasn't somehow writing to disk. I was able to lower the CPU usage by running wget with --limit-rate=.

Related, I would not recommend macvtap if you're running routers. At least running on an Ubuntu 16.04 host, under load of ~1Gb we were seeing anywhere from 0.01% to 0.16% packet loss before it reached the guest's virtio interface. We switched to linux bridge and then finally to openvswitch with the default switching config (no custom openflow rules.)


Josh Knight

Post by Maxime AUGER
Thank you for your statement
I wget on the debian8 GUEST
-> GUEST-CPU = 25% of 1x vcpu
-> HOST-CPU = 30% of 2x cores (each one)
Ex with "-O /dev/null" : host TOP (the cpu decreases a little bit)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6833 root 20 0 1972312 892460 5900 S 31.1 43.7 184:54.60
/usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev
socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon
chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid
-daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a -s+
6772 root 20 0 0 0 0 R 11.9 0.0 1:37.54 [vhost-6712]
6834 root 20 0 1972312 892460 5900 S 7.9 43.7 173:24.10
/usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev
socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon
chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid
-daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a -s+
6712 root 20 0 1972312 892460 5900 S 1.3 43.7 7:50.63
/usr/bin/kvm -id 108 -name fg-v4-bench-8 -chardev
socket,id=qmp,path=/var/run/qemu-server/108.qmp,server,nowait -mon
chardev=qmp,mode=control -pidfile /var/run/qemu-server/108.pid
-daemonize -smbios type=1,uuid=b1572470-0605-44da-9bae-0b2a6d43954a
-s+
-----Message d'origine-----
Mark Schouten Envoyé : jeudi 7 juin 2018 17:03 À : PVE User List
100mbits/sec download with virtio nic

I'm doubtful about the performance I get.
On proxmox HOST a single wget download consume 12% single CPU load
at 11MBytes/sec (1518 packets size)

Where are you wgetting to? Benchmarking this is better with iperf or so.
If you want to wget, do wget -O /dev/null so you're not suffering from diskperformance..
--
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark
Schouten
| Tuxis Internet Engineering
KvK: 61527076 | http://www.tuxis.nl/
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-***@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Alwin Antreich]

Post by Maxime AUGER
Hello,
Let me clarify my statement.
GUEST CPU load is acceptable (25% of a single CPU)
It is the cumulative load of the kvm process and the vhost thread that is high, on the HOST side
kvm-thread-1=30%
kvm-thred-2=30%
vhost(net)=10%
70% CPU without i/o disk (-O /dev/null)
it is x10 the load observed with the same conditions under old Vmware ESXi (same GUEST system and wget process)
I think it is a kvm issue, but I'm curious about proxmox's positioning in relation to this performance level
-----Message d'origine-----
Envoyé : jeudi 7 juin 2018 20:47
Objet : Re: [PVE-User] high cpu load on 100mbits/sec download with virtio nic
I'm not convinced this is a proxmox issue, or even an issue to begin with.
I'm running proxmox 5.1-49, in my Linux 4.1 guest when I run wget -O /dev/null <https to ~1.2GB iso> I'm seeing ~30% according to top as well.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12788 root 20 0 65512 7616 5988 S 31.3 0.1 0:03.00 wget
Even on a physical box running Ubuntu I'm getting around 20-30% or more.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36070 root 20 0 32888 4028 3576 S 37.6 0.0 0:03.15 wget
This could be an issue/quirk with the way top is calculating cpu usage, or it's just a process using available cpu as normal. I couldn't reproduce high load using ping -f from a remote host to my VM, and iotop confirmed that -O /dev/null wasn't somehow writing to disk. I was able to lower the CPU usage by running wget with --limit-rate=.
Related, I would not recommend macvtap if you're running routers. At least running on an Ubuntu 16.04 host, under load of ~1Gb we were seeing anywhere from 0.01% to 0.16% packet loss before it reached the guest's virtio interface. We switched to linux bridge and then finally to openvswitch with the default switching config (no custom openflow rules.)
Josh Knight

I guess, this could be well related to the Meltdown/Spectre mitigation.
At least it would fit to the "old" ESXi showing a different performance
hit.

Try to set the CPU type to host or activate PCID/Spectre-ctrl if
applicable.
https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_cpu

--
Cheers,
Alwin

[Maxime AUGER]

Hello,

I can t enable PCID/Spectre-ctrl, host does not support it
Same results using CPU type host
I notice that GUEST %CPU decrease by allocating a single vcpu
In that situation I have same 7% CPU on PROXMOX and VMWARE GUESTS
But on host side it s drastically different
On VMWARE HOST : 8% CPU
On PROXMOX HOST : 50% CPU (40% qemu process and 10% vhost kernel thread)

I also notice that with single vcpu ksoftirq CPU time consuming on GUEST is close to zero
With dual vcpu (or more) ksoftirq increase even when GUEST is idling

It appears to be an improper use of the CPU whether with the HOST CPU type or kvm64
I started with last stable proxmox ISO (5.2), I dont know how to improve CPU utilisation

Do you notice the same behavior, GUEST/HOST CPU usage difference ?

Thank you

-----Message d'origine-----
De : pve-user [mailto:pve-user-***@pve.proxmox.com] De la part de Alwin Antreich
Envoyé : samedi 9 juin 2018 19:48
À : PVE User List <pve-***@pve.proxmox.com>
Objet : ###SPAM### Re: [PVE-User] high cpu load on 100mbits/sec download with virtio nic

Post by Maxime AUGER
Hello,
Let me clarify my statement.
GUEST CPU load is acceptable (25% of a single CPU) It is the
cumulative load of the kvm process and the vhost thread that is high,
on the HOST side kvm-thread-1=30% kvm-thred-2=30% vhost(net)=10% 70%
CPU without i/o disk (-O /dev/null)
it is x10 the load observed with the same conditions under old Vmware
ESXi (same GUEST system and wget process) I think it is a kvm issue,
but I'm curious about proxmox's positioning in relation to this
performance level
-----Message d'origine-----
Josh Knight Envoyé : jeudi 7 juin 2018 20:47 À : PVE User List
100mbits/sec download with virtio nic
I'm not convinced this is a proxmox issue, or even an issue to begin with.
I'm running proxmox 5.1-49, in my Linux 4.1 guest when I run wget -O /dev/null <https to ~1.2GB iso> I'm seeing ~30% according to top as well.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12788 root 20 0 65512 7616 5988 S 31.3 0.1 0:03.00 wget
Even on a physical box running Ubuntu I'm getting around 20-30% or more.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36070 root 20 0 32888 4028 3576 S 37.6 0.0 0:03.15 wget
This could be an issue/quirk with the way top is calculating cpu usage, or it's just a process using available cpu as normal. I couldn't reproduce high load using ping -f from a remote host to my VM, and iotop confirmed that -O /dev/null wasn't somehow writing to disk. I was able to lower the CPU usage by running wget with --limit-rate=.
Related, I would not recommend macvtap if you're running routers. At
least running on an Ubuntu 16.04 host, under load of ~1Gb we were
seeing anywhere from 0.01% to 0.16% packet loss before it reached the
guest's virtio interface. We switched to linux bridge and then
finally to openvswitch with the default switching config (no custom
openflow rules.)
Josh Knight

I guess, this could be well related to the Meltdown/Spectre mitigation.
At least it would fit to the "old" ESXi showing a different performance hit.

Try to set the CPU type to host or activate PCID/Spectre-ctrl if applicable.
https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_cpu

[Maxime AUGER]

Hello,
it's not my intention, my goal is to migrate on KVM a hundred VM currently driven by VMWARE
I am aware of the architecture differences between vmware and kvm
the comparison may make no sense
but I can't migrate without doing a minimum of validation testing
Currently I notice an overhead which is likely to be problematic to scale
if this is the normal behavior of KVM virtualization I would load less VM but that doesn't seem to be the way it is.
So to discern this I would like to have your feedback
the key question I ask
On your PROXMOX setup is the visible CPU usage on the linux GUEST and HOST sides are drastically different on idle ? and with a wget -O /dev/null ... at 100mbits rate ?

-----Message d'origine-----
De : Woods, Ken A (DNR) [mailto:***@alaska.gov]
Envoyé : mardi 12 juin 2018 18:20
À : Maxime AUGER <***@auranext.com>
Objet : Re: [PVE-User] high cpu load on 100mbits/sec download with virtio nic

Are you sure your goal is to not to just trash proxmox?

Post by Maxime AUGER
Hello,
I can t enable PCID/Spectre-ctrl, host does not support it Same
results using CPU type host I notice that GUEST %CPU decrease by
allocating a single vcpu In that situation I have same 7% CPU on
PROXMOX and VMWARE GUESTS But on host side it s drastically different
On VMWARE HOST : 8% CPU On PROXMOX HOST : 50% CPU (40% qemu process
and 10% vhost kernel thread)
I also notice that with single vcpu ksoftirq CPU time consuming on
GUEST is close to zero With dual vcpu (or more) ksoftirq increase even
when GUEST is idling
It appears to be an improper use of the CPU whether with the HOST CPU
type or kvm64 I started with last stable proxmox ISO (5.2), I dont
know how to improve CPU utilisation
Do you notice the same behavior, GUEST/HOST CPU usage difference ?
Thank you
-----Message d'origine-----
Alwin Antreich Envoyé : samedi 9 juin 2018 19:48 À : PVE User List
load on 100mbits/sec download with virtio nic

Post by Maxime AUGER
Hello,
Let me clarify my statement.
GUEST CPU load is acceptable (25% of a single CPU) It is the
cumulative load of the kvm process and the vhost thread that is high,
on the HOST side kvm-thread-1=30% kvm-thred-2=30% vhost(net)=10% 70%
CPU without i/o disk (-O /dev/null)
it is x10 the load observed with the same conditions under old Vmware
ESXi (same GUEST system and wget process) I think it is a kvm issue,
but I'm curious about proxmox's positioning in relation to this
performance level
-----Message d'origine-----
Josh Knight Envoyé : jeudi 7 juin 2018 20:47 À : PVE User List
100mbits/sec download with virtio nic
I'm not convinced this is a proxmox issue, or even an issue to begin with.
I'm running proxmox 5.1-49, in my Linux 4.1 guest when I run wget -O /dev/null <https to ~1.2GB iso> I'm seeing ~30% according to top as well.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12788 root 20 0 65512 7616 5988 S 31.3 0.1 0:03.00 wget
Even on a physical box running Ubuntu I'm getting around 20-30% or more.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
36070 root 20 0 32888 4028 3576 S 37.6 0.0 0:03.15 wget
This could be an issue/quirk with the way top is calculating cpu usage, or it's just a process using available cpu as normal. I couldn't reproduce high load using ping -f from a remote host to my VM, and iotop confirmed that -O /dev/null wasn't somehow writing to disk. I was able to lower the CPU usage by running wget with --limit-rate=.
Related, I would not recommend macvtap if you're running routers. At
least running on an Ubuntu 16.04 host, under load of ~1Gb we were
seeing anywhere from 0.01% to 0.16% packet loss before it reached the
guest's virtio interface. We switched to linux bridge and then
finally to openvswitch with the default switching config (no custom
openflow rules.)
Josh Knight

I guess, this could be well related to the Meltdown/Spectre mitigation.
At least it would fit to the "old" ESXi showing a different performance hit.
Try to set the CPU type to host or activate PCID/Spectre-ctrl if applicable.
https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_cpu
--
Cheers,
Alwin
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Daniel Berteaud]

Post by Maxime AUGER
Hello,
it's not my intention, my goal is to migrate on KVM a hundred VM currently driven by VMWARE
I am aware of the architecture differences between vmware and kvm
the comparison may make no sense
but I can't migrate without doing a minimum of validation testing

IMHO, you should push your host to its limit, eg, run as many VM as you
can until you're CPU bound (on host side). Then you'll be able to check
if the density is less than, equal to, or better than vmware. I don't
think comparing a % cpu usage is meaningful on different platforms as
there can be too many differences in how this is mesured (and how the
CPU governor is handled when nearly idle)

++
--
Logo FWS

*Daniel Berteaud*

FIREWALL-SERVICES SAS.
Société de Services en Logiciels Libres
Tel : [phone] <tel:[phone]>
Matrix: @dani:fws.fr
/www.firewall-services.com/