[PVE-User] network conversion mesh network into lacp bonding setup

[lists]

Hi all,

I am going to make some change to our proxmox networking, and I'd like
some fresh eyes to take a look at my plans... :-)

We have three pve hosts, ceph network is a meshed 10G network setup,
directly connecting the pve hosts to each other. Client access is on a
'regular' ip 1G NIC.

Sample /etc/network/interfaces (server pve10) for current config:

# client access
auto vmbr0
iface vmbr0 inet static
address 192.168.89.10
netmask 255.255.255.0
gateway 192.168.89.1
bridge_ports eth0
bridge_stp off
bridge_fd 0

# to pve2/ceph
auto eth2
iface eth2 inet static
address 10.10.89.1
netmask 255.255.255.0
mtu 9000
up route add -net 10.10.89.2 netmask 255.255.255.255 dev eth2
down route del -net 10.10.89.2 netmask 255.255.255.255 dev eth2

# to pve3/ceph
auto eth3
iface eth3 inet static
address 10.10.89.1
netmask 255.255.255.0
mtu 9000
up route add -net 10.10.89.3 netmask 255.255.255.255 dev eth3
down route del -net 10.10.89.3 netmask 255.255.255.255 dev eth3

See for more info on the meshed network:
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server. It works
very nicely.

Now, we want to change networking from the above to: dual 10G lacp bonds
per server to our hp procurve chassis.

So, in order to change as little as possible, I would like to keep ceph
config the same, meaning: retain all IPs/config, and use something like
this:

auto bond0
iface bond0 inet manual
slaves eth2 eth3
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4

allow-hotplug vmbr0
auto vmbr0

# client access
iface vmbr0 inet static
address 192.168.89.10
netmask 255.255.255.0
gateway 192.168.89.1
bridge_ports bond0
bridge_stp off
bridge_fd 0

# ip for access to other cephs
iface vmbr0 inet static
address 10.10.89.1
netmask 255.255.255.0

Then cable eth2 / eth3 to the lacp ports on the HP procurve.

I am assuming that this would make all traffic (ceph and VMs) float over
the same two 10G lacp wires, and both ceph and VMs would not notice any
difference. I'm also assuming that no other config changes would be
required at all.

So, any errors in the above reasoning? I realise we cannot have jumbo
frames in this setup, but I don't think I mind. I also realise that
currently we have seperated ceph and VMs traffic, and in the new
situation we don't anymore, but this seems accepted (perhaps even
recommended) for small networks like ours at the ceph mailinglist nowadays.

So... feedback to all of the above please... :-)

Thanks!
MJ