[PVE-User] Intel X710 10Gbit network card

[Sten Aus]

Hi

Has anyone used this 10Gbit network card?
I'm having problems. Physical host can communicate (vmbr is working),
but no VM is getting traffic on tap devices.

So, I found a driver update from Intel page (from 1.2.37 to 1.238), but
still no luck using this 10Gbit card.

Weird is that PVE host gets traffic on this vmbr, but when I attach it
to VM, it does not.

*brctl show* output shows clearly that tap device is connected to
correct bridge.
vmbr10 8000.6805ca32f119 no eth5.10
tap145i0

Any ideas?

Sten

[Michael Rasmussen]

On Wed, 15 Apr 2015 16:52:57 +0300

Post by Sten Aus
Hi
Has anyone used this 10Gbit network card?

What kernel, 2.6.32 or 3.10?
--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
He hated being thought of as one of those people that wore stupid
ornamental armour. It was gilt by association.
-- Terry Pratchett, "Night Watch"

[Michael Rasmussen]

On Wed, 15 Apr 2015 16:44:03 +0200

Post by Michael Rasmussen
On Wed, 15 Apr 2015 16:52:57 +0300

Post by Sten Aus
Hi
Has anyone used this 10Gbit network card?

What kernel, 2.6.32 or 3.10?

Have you loaded the i40e and i40evf driver?
https://downloadcenter.intel.com/download/24693
--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
"They are both businesses - if you have given them enough money, I'm
sure they'll do whatever the hell you ask:->"
-- David Welton

[Sten Aus]

Hi

Using kernel 2.6.32-37-pve, and i40e was loaded. Now installed i40evf,
too. Should I configure i40evf somehow? Using Intel Corporation 82598EB
and 82598EB 10Gbit PCI-E cards on other servers and have never
experienced this before.

After a (node and VM) reboot no changes on connectivity at the VM side.

# ethtool -i eth5
driver: i40e
version: 1.2.38
firmware-version: f4.22.27454 a1.2 n4.25 e143f
bus-info: 0000:06:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Post by Michael Rasmussen
On Wed, 15 Apr 2015 16:44:03 +0200

Post by Michael Rasmussen
On Wed, 15 Apr 2015 16:52:57 +0300

Post by Sten Aus
Hi
Has anyone used this 10Gbit network card?

What kernel, 2.6.32 or 3.10?

Have you loaded the i40e and i40evf driver?
https://downloadcenter.intel.com/download/24693
_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Michael Rasmussen]

On Wed, 15 Apr 2015 20:23:13 +0300

Hi
Using kernel 2.6.32-37-pve, and i40e was loaded. Now installed i40evf, too. Should I configure i40evf somehow? Using Intel Corporation 82598EB and 82598EB 10Gbit PCI-E cards on other servers and have never experienced this before.

What does tcpdump disclose running on host and in VM?

On host tcpdump -i vmbr10

Btw. Are there bonding involved?
--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Keep grandma off the streets -- legalize bingo.

[Sten Aus]

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the VM tap
device is not getting ARPs back. I mean this tap device what qemu brings
up/makes.

Post by Michael Rasmussen
Btw. Are there bonding involved?

No, no bonding, only one 10Gbit port and a lot of vmbrs (vlans) on it.

PS! Not working with 3.10.0-8-pve as well.

[Michael Rasmussen]

On Wed, 15 Apr 2015 21:01:35 +0300

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the VM tap device is not getting ARPs back. I mean this tap device what qemu brings up/makes.

A long shot. Try to disable hardware checksum offload since it is know
to cause problems in virtualized environments.
You can try with ethtool before making permanent changes: ethtool -K
eth5 tx off

If that works you could add it permanently to udev:

Create this file: /etc/udev/rules.d/81-bridge.rules
With the following content:
ACTION=="add", SUBSYSTEM=="net", NAME=="vmbr*", RUN+="/sbin/ethtool -K
$env{INTERFACE} tx off"
--
Hilsen/Regards
Michael Rasmussen

Get my public GnuPG keys:
michael <at> rasmussen <dot> cc
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E
mir <at> datanom <dot> net
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C
mir <at> miras <dot> org
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917
--------------------------------------------------------------
/usr/games/fortune -es says:
Detroit is Cleveland without the glitter.

[Sten Aus]

Thanks for the tip, but this was not the case.

Post by Michael Rasmussen
On Wed, 15 Apr 2015 21:01:35 +0300

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the VM tap device is not getting ARPs back. I mean this tap device what qemu brings up/makes.

A long shot. Try to disable hardware checksum offload since it is know
to cause problems in virtualized environments.
You can try with ethtool before making permanent changes: ethtool -K
eth5 tx off
Create this file: /etc/udev/rules.d/81-bridge.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="vmbr*", RUN+="/sbin/ethtool -K
$env{INTERFACE} tx off"

[Sten Aus]

Got a little bit further

Turned on stp for this vmbr and now VM gets traffic for about 10-15
seconds and then no packet flow again.

Note that packets are going from VM to (for example) DHCP server, but
they will get lost on their way back. They will arrive at the eth5 and
vmbr, but not to tap, which is connected to the VM by qemu.

Tried to upgrade Intel X710 firmware as well, no luck there. Not to
mention the drivers small upgrade which I already did yesterday.

Post by Sten Aus
Thanks for the tip, but this was not the case.

Post by Michael Rasmussen
On Wed, 15 Apr 2015 21:01:35 +0300

Post by Sten Aus

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the VM
tap device is not getting ARPs back. I mean this tap device what
qemu brings up/makes.

A long shot. Try to disable hardware checksum offload since it is know
to cause problems in virtualized environments.
You can try with ethtool before making permanent changes: ethtool -K
eth5 tx off
Create this file: /etc/udev/rules.d/81-bridge.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="vmbr*", RUN+="/sbin/ethtool -K
$env{INTERFACE} tx off"

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Sten Aus]

I dont want to shout before the night, but it seems that the link is
working!

I have downgraded drivers to

Post by Sten Aus
driver: i40e
version: 1.1.23

As my coworker set up a new environment with Ubuntu and run different
kvm's there and it worked, I saw that there was a difference between
driver versions. Ubuntu was using

Post by Sten Aus
driver: i40e
version: 0.3.36-k

So, somehow I thought that why not try to downgrade.

Post by Sten Aus
Got a little bit further
Turned on stp for this vmbr and now VM gets traffic for about 10-15
seconds and then no packet flow again.
Note that packets are going from VM to (for example) DHCP server, but
they will get lost on their way back. They will arrive at the eth5 and
vmbr, but not to tap, which is connected to the VM by qemu.
Tried to upgrade Intel X710 firmware as well, no luck there. Not to
mention the drivers small upgrade which I already did yesterday.

Post by Sten Aus
Thanks for the tip, but this was not the case.

Post by Michael Rasmussen
On Wed, 15 Apr 2015 21:01:35 +0300

Post by Sten Aus

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the VM
tap device is not getting ARPs back. I mean this tap device what
qemu brings up/makes.

A long shot. Try to disable hardware checksum offload since it is know
to cause problems in virtualized environments.
You can try with ethtool before making permanent changes: ethtool -K
eth5 tx off
Create this file: /etc/udev/rules.d/81-bridge.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="vmbr*", RUN+="/sbin/ethtool -K
$env{INTERFACE} tx off"

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Sten Aus]

Forgot to mention that with newer drivers (1.2.37 (stock) and 1.2.38
(new from Intel support page)) I had a situation where I turned on STP
on my vmbr10 Linux bridge, VM had connectivity for about 10-20 seconds
and then it all failed again.

Post by Sten Aus
I dont want to shout before the night, but it seems that the link is
working!
I have downgraded drivers to

Post by Sten Aus
driver: i40e
version: 1.1.23

As my coworker set up a new environment with Ubuntu and run different
kvm's there and it worked, I saw that there was a difference between
driver versions. Ubuntu was using

Post by Sten Aus
driver: i40e
version: 0.3.36-k

So, somehow I thought that why not try to downgrade.

Post by Sten Aus
Got a little bit further
Turned on stp for this vmbr and now VM gets traffic for about 10-15
seconds and then no packet flow again.
Note that packets are going from VM to (for example) DHCP server, but
they will get lost on their way back. They will arrive at the eth5
and vmbr, but not to tap, which is connected to the VM by qemu.
Tried to upgrade Intel X710 firmware as well, no luck there. Not to
mention the drivers small upgrade which I already did yesterday.

Post by Sten Aus
Thanks for the tip, but this was not the case.

Post by Michael Rasmussen
On Wed, 15 Apr 2015 21:01:35 +0300

Post by Sten Aus

Post by Michael Rasmussen
What does tcpdump disclose running on host and in VM?
On host tcpdump -i vmbr10

Coworker did a lot of tcpdumps during the day and it appears the
VM tap device is not getting ARPs back. I mean this tap device
what qemu brings up/makes.

A long shot. Try to disable hardware checksum offload since it is know
to cause problems in virtualized environments.
You can try with ethtool before making permanent changes: ethtool -K
eth5 tx off
Create this file: /etc/udev/rules.d/81-bridge.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="vmbr*", RUN+="/sbin/ethtool -K
$env{INTERFACE} tx off"

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user