[PVE-User] networking adjustment | hope to get some feedback

Discussion:

2018-06-18 09:56:20 UTC

Hi all,

After having bought some new networking equipment, and gaining more
insight over the last two years, I am planning to make some adjustments
to our proxmox/ceph setup, and I would *greatly* appreciate some
feedback :-)

We are running a three-identical-server proxmox/ceph setup, with on each
server:

NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)

Since we bought new hardware, I can connect each server to our HP
chassis, over a dual 10G bonded LACP connection.

I obviously need to keep the (NIC1) public IP, but since the ceph
monitors ip is difficult to change, I'd like to keep the (NIC2)
10.10.89.x as well.

I also need to keep the (tagged and untagged) VLAN's for proxmox and the
VMs running on it.

I realise that it used to be recommened to split cluster and client
traffic, but consensus nowadays on the ceph mailinglist seems to be:
keep it simple and don't split, unless specifically required. With this
in mind, I would also like to consolidate networking and run all traffic
over this dual lacp-bonded 10G connection to our HP chassis, including
the VLANs.

But how to achieve this..? :-) (and here come the questions...)

My idea is to first enable (active) LACP on our ProCurve 5400 chassis
ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
to select a different Truck Group (Trk1, Trk2 & Trk3) for each
dual-cable-connection to a server..?

And will the port-configured VLANs on the lacp-member-ports (both tagged
and untagged) continue to flow normally through this lacp bond..?

Then, about configuration on proxmox, would something like below do the
trick..?

auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer2+3

auto vmbr0
iface vmbr0 inet static
address a.b.c.10/11/12 (public IPs)
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
down ip addr del 10.100.222.1/24 dev vmbr0 || true

Any feedback on the above? As this is production, I'd like to be
reasonably sure that this would work, before trying.

Your comments will be very much appreciated!

MJ

2018-06-21 12:31:28 UTC

Permalink

Hi,

So, I setup a test rig, with (only) two proxmox test-servers, with two
NICs per server to test.

iface eth1 inet manual
iface eth2 inet manual
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
up ip addr add 10.10.89.10/24 dev vmbr0 || true
down ip addr del 10.10.89.10/24 dev vmbr0 || true
trunk D1-D2 Trk1 LACP

and

The above config allows me to assign VLANs to lacp trunks ("Trk1",
"Trk2") in the chassis webinterface like you would do with ports.

Then I did some reading on load balancing between the trunked ports, and
figured that load balancing based on L4 would perhaps work better for
us, so I changed it with

trunk-load-balance L4

Since we are running the public and cluster network over the same wires,
I don't think we can enable jumbo frames. Or would there be a way to
make ceph traffic use a specific vlan, so we can enable jumbo frames on
that vlan?

I realise that this is perhaps all very specific to our environment, but
again: if there is anyone here with insights, tips, trics, please,
feedback is welcome.

For example: with the dual 10G LACP connection to each server, we can
only use mtu size 1500. Are we loosing much there..? Or would there be a
way around this, somehow?

I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
could never ping the other machine, so that somehow doesn't work. :-(

(thinking that with the ceph ip on the bond0, my VMs would not be able
to see that traffic..?)

Again: all feedback welcome.

MJ

Hi all,
After having bought some new networking equipment, and gaining more
insight over the last two years, I am planning to make some adjustments
to our proxmox/ceph setup, and I would *greatly* appreciate some
feedback :-)
We are running a three-identical-server proxmox/ceph setup, with on each
NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
Since we bought new hardware, I can connect each server to our HP
chassis, over a dual 10G bonded LACP connection.
I obviously need to keep the (NIC1) public IP, but since the ceph
monitors ip is difficult to change, I'd like to keep the (NIC2)
10.10.89.x as well.
I also need to keep the (tagged and untagged) VLAN's for proxmox and the
VMs running on it.
I realise that it used to be recommened to split cluster and client
keep it simple and don't split, unless specifically required. With this
in mind, I would also like to consolidate networking and run all traffic
over this dual lacp-bonded 10G connection to our HP chassis, including
the VLANs.
But how to achieve this..? :-) (and here come the questions...)
My idea is to first enable (active) LACP on our ProCurve 5400 chassis
ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
to select a different Truck Group (Trk1, Trk2 & Trk3) for each
dual-cable-connection to a server..?
And will the port-configured VLANs on the lacp-member-ports (both tagged
and untagged) continue to flow normally through this lacp bond..?
Then, about configuration on proxmox, would something like below do the
trick..?
auto bond0
iface bond0 inet manual
      slaves eth0 eth1
      bond_miimon 100
      bond_mode 802.3ad
      bond_xmit_hash_policy layer2+3
auto vmbr0
iface vmbr0 inet static
      address a.b.c.10/11/12 (public IPs)
      netmask 255.255.255.0
      gateway a.b.c.1
      bridge_ports bond0
      bridge_stp off
      bridge_fd 0
      up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
      down ip addr del 10.100.222.1/24 dev vmbr0 || true
Any feedback on the above? As this is production, I'd like to be
reasonably sure that this would work, before trying.
Your comments will be very much appreciated!
MJ
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Alexandre DERUMIER

2018-06-22 00:54:48 UTC

Permalink

Post by mj
For example: with the dual 10G LACP connection to each server, we can
only use mtu size 1500. Are we loosing much there..? Or would there be a
way around this, somehow?

you can setup mtu 9000 on your bridge and bond.
if your vms have mtu 1500 (inside the vm), the packet will use 1500 mtu

Post by mj
I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
could never ping the other machine, so that somehow doesn't work. :-(

you can setup ip on interface which is plugged in a bridge.

but you can use vlan interface on the bond for example

- management proxmox without vlan && ceph on dedicated vlan
------------------------------------------------------------
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4

auto vmbr0
iface vmbr0 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0

#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....

or with dedicated vlan for proxmox management && ceph
------------------------------------------------------

auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4

auto vmbr0
iface vmbr0 inet static
bridge_ports bond0
bridge_stp off
bridge_fd 0

#dedicated proxmox vlan
auto bond0.99
iface bond0.99 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1

#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....

----- Mail original -----
De: "mj" <***@merit.unu.edu>
À: "proxmoxve" <pve-***@pve.proxmox.com>
Envoyé: Jeudi 21 Juin 2018 14:31:28
Objet: Re: [PVE-User] networking adjustment | hope to get some feedback

Hi,

So, I setup a test rig, with (only) two proxmox test-servers, with two
NICs per server to test.

and

trunk-load-balance L4

Hi all,
After having bought some new networking equipment, and gaining more
insight over the last two years, I am planning to make some adjustments
to our proxmox/ceph setup, and I would *greatly* appreciate some
feedback :-)
We are running a three-identical-server proxmox/ceph setup, with on each
NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
Since we bought new hardware, I can connect each server to our HP
chassis, over a dual 10G bonded LACP connection.
I obviously need to keep the (NIC1) public IP, but since the ceph
monitors ip is difficult to change, I'd like to keep the (NIC2)
10.10.89.x as well.
I also need to keep the (tagged and untagged) VLAN's for proxmox and the
VMs running on it.
I realise that it used to be recommened to split cluster and client
keep it simple and don't split, unless specifically required. With this
in mind, I would also like to consolidate networking and run all traffic
over this dual lacp-bonded 10G connection to our HP chassis, including
the VLANs.
But how to achieve this..? :-) (and here come the questions...)
My idea is to first enable (active) LACP on our ProCurve 5400 chassis
ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
to select a different Truck Group (Trk1, Trk2 & Trk3) for each
dual-cable-connection to a server..?
And will the port-configured VLANs on the lacp-member-ports (both tagged
and untagged) continue to flow normally through this lacp bond..?
Then, about configuration on proxmox, would something like below do the
trick..?
auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer2+3
auto vmbr0
iface vmbr0 inet static
address a.b.c.10/11/12 (public IPs)
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
down ip addr del 10.100.222.1/24 dev vmbr0 || true
Any feedback on the above? As this is production, I'd like to be
reasonably sure that this would work, before trying.
Your comments will be very much appreciated!
MJ
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-***@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

2018-06-22 09:54:44 UTC

Permalink

Thank you Alexandre!

Post by Alexandre DERUMIER

Post by mj
For example: with the dual 10G LACP connection to each server, we can
only use mtu size 1500. Are we loosing much there..? Or would there be a
way around this, somehow?

you can setup mtu 9000 on your bridge and bond.
if your vms have mtu 1500 (inside the vm), the packet will use 1500 mtu

you can setup ip on interface which is plugged in a bridge.
but you can use vlan interface on the bond for example
- management proxmox without vlan && ceph on dedicated vlan
------------------------------------------------------------
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....
or with dedicated vlan for proxmox management && ceph
------------------------------------------------------
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
bridge_ports bond0
bridge_stp off
bridge_fd 0
#dedicated proxmox vlan
auto bond0.99
iface bond0.99 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....
----- Mail original -----
Envoyé: Jeudi 21 Juin 2018 14:31:28
Objet: Re: [PVE-User] networking adjustment | hope to get some feedback
Hi,
So, I setup a test rig, with (only) two proxmox test-servers, with two
NICs per server to test.

and

The above config allows me to assign VLANs to lacp trunks ("Trk1",
"Trk2") in the chassis webinterface like you would do with ports.
Then I did some reading on load balancing between the trunked ports, and
figured that load balancing based on L4 would perhaps work better for
us, so I changed it with

trunk-load-balance L4

Since we are running the public and cluster network over the same wires,
I don't think we can enable jumbo frames. Or would there be a way to
make ceph traffic use a specific vlan, so we can enable jumbo frames on
that vlan?
I realise that this is perhaps all very specific to our environment, but
again: if there is anyone here with insights, tips, trics, please,
feedback is welcome.
For example: with the dual 10G LACP connection to each server, we can
only use mtu size 1500. Are we loosing much there..? Or would there be a
way around this, somehow?
I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
could never ping the other machine, so that somehow doesn't work. :-(
(thinking that with the ceph ip on the bond0, my VMs would not be able
to see that traffic..?)
Again: all feedback welcome.
MJ

Hi all,
After having bought some new networking equipment, and gaining more
insight over the last two years, I am planning to make some adjustments
to our proxmox/ceph setup, and I would *greatly* appreciate some
feedback :-)
We are running a three-identical-server proxmox/ceph setup, with on each
NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
Since we bought new hardware, I can connect each server to our HP
chassis, over a dual 10G bonded LACP connection.
I obviously need to keep the (NIC1) public IP, but since the ceph
monitors ip is difficult to change, I'd like to keep the (NIC2)
10.10.89.x as well.
I also need to keep the (tagged and untagged) VLAN's for proxmox and the
VMs running on it.
I realise that it used to be recommened to split cluster and client
keep it simple and don't split, unless specifically required. With this
in mind, I would also like to consolidate networking and run all traffic
over this dual lacp-bonded 10G connection to our HP chassis, including
the VLANs.
But how to achieve this..? :-) (and here come the questions...)
My idea is to first enable (active) LACP on our ProCurve 5400 chassis
ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
to select a different Truck Group (Trk1, Trk2 & Trk3) for each
dual-cable-connection to a server..?
And will the port-configured VLANs on the lacp-member-ports (both tagged
and untagged) continue to flow normally through this lacp bond..?
Then, about configuration on proxmox, would something like below do the
trick..?
auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer2+3
auto vmbr0
iface vmbr0 inet static
address a.b.c.10/11/12 (public IPs)
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
down ip addr del 10.100.222.1/24 dev vmbr0 || true
Any feedback on the above? As this is production, I'd like to be
reasonably sure that this would work, before trying.
Your comments will be very much appreciated!
MJ
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user