[PVE-User] 2node cluster

[Tonči Stipičević]

Hello to all

I'm testing pve-storage-zsync and it works pretty well on my 3node
cluster .   One VM from origin node is replicating to other two nodes
and when I shutdown origin node , I can start VM from it's replicated
image on one of the other nodes w/o any issues/problems. If I make some
changes to this replicated image , after starting "dead" node back
again, replication overrides those changes and so on ...

So this is possible because we still have quorum .  However real
scenario will have only two nodes in cluster and the problem arises when
one node shuts down and there is no quorum any more.

Then we have to lower "votes" on the surviving node.

Since I want to make zfs-sync (disaster recovery) test with 2node
cluster ,  the question would be: Is it possible to start "dead" node
back again and restore "votes" value on the surviving node ?      Will
the cluster accept this "dead" node back ?

I have never done this before  and just want to be prepared for
reinstalling dead node if it cannot get back into the cluster again ....

Thank you very much in advance

and

BR


Tonci

[Dean Mumby]

just create a proxmox vm within each node and join them to the cluster , then you should be able to have each node run independently.


Regards
Dean Mumby

----- Original Message -----
From: "Tonči Stipičević" <***@suma-informatika.hr>
To: pve-***@pve.proxmox.com
Sent: Wednesday, 20 June, 2018 12:44:59
Subject: [PVE-User] 2node cluster

Hello to all

I'm testing pve-storage-zsync and it works pretty well on my 3node
cluster .   One VM from origin node is replicating to other two nodes
and when I shutdown origin node , I can start VM from it's replicated
image on one of the other nodes w/o any issues/problems. If I make some
changes to this replicated image , after starting "dead" node back
again, replication overrides those changes and so on ...

So this is possible because we still have quorum .  However real
scenario will have only two nodes in cluster and the problem arises when
one node shuts down and there is no quorum any more.

Then we have to lower "votes" on the surviving node.

Since I want to make zfs-sync (disaster recovery) test with 2node
cluster ,  the question would be: Is it possible to start "dead" node
back again and restore "votes" value on the surviving node ?      Will
the cluster accept this "dead" node back ?

I have never done this before  and just want to be prepared for
reinstalling dead node if it cannot get back into the cluster again ....

Thank you very much in advance

and

BR


Tonci

[Thomas Lamprecht]

Post by Dean Mumby
just create a proxmox vm within each node and join them to the cluster , then you should be able to have each node run independently.

please don't do that, uneccessarily complicated and not good practice
to depend an additional vote to itself through indirection...
And if you add one in both nodes you win absolute nothing, as the cluster
then expects 4 votes, with 3 for quorum. If one node (and it's VM) go
down you have 2 votes left, i.e. no quorum...

If you have a case where one node will always be up and the second may
be shutdown from time to time just give the first "primary" node two
votes via editing /etc/pve/corosync.conf [1]. Just increase the
'quorum_votes' for the respective node entry.

cheers,
Thomas

[1]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#edit-corosync-conf

Post by Dean Mumby
Regards
Dean Mumby
----- Original Message -----
Sent: Wednesday, 20 June, 2018 12:44:59
Subject: [PVE-User] 2node cluster
Hello to all
I'm testing pve-storage-zsync and it works pretty well on my 3node
cluster .   One VM from origin node is replicating to other two nodes
and when I shutdown origin node , I can start VM from it's replicated
image on one of the other nodes w/o any issues/problems. If I make some
changes to this replicated image , after starting "dead" node back
again, replication overrides those changes and so on ...
So this is possible because we still have quorum .  However real
scenario will have only two nodes in cluster and the problem arises when
one node shuts down and there is no quorum any more.
Then we have to lower "votes" on the surviving node.
Since I want to make zfs-sync (disaster recovery) test with 2node
cluster ,  the question would be: Is it possible to start "dead" node
back again and restore "votes" value on the surviving node ?      Will
the cluster accept this "dead" node back ?
I have never done this before  and just want to be prepared for
reinstalling dead node if it cannot get back into the cluster again ....
Thank you very much in advance
and
BR
Tonci
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[Andreas Heinlein]

Post by Thomas Lamprecht
please don't do that, uneccessarily complicated and not good practice
to depend an additional vote to itself through indirection...
And if you add one in both nodes you win absolute nothing, as the cluster
then expects 4 votes, with 3 for quorum. If one node (and it's VM) go
down you have 2 votes left, i.e. no quorum...
If you have a case where one node will always be up and the second may
be shutdown from time to time just give the first "primary" node two
votes via editing /etc/pve/corosync.conf [1]. Just increase the
'quorum_votes' for the respective node entry.
cheers,
Thomas

Just out of curiosity, does Proxmox support STONITH/node level fencing
using e.g. PDUs or UPS'?

Thanks,
Andreas

[Thomas Lamprecht]

Post by Andreas Heinlein

Post by Thomas Lamprecht
please don't do that, uneccessarily complicated and not good practice
to depend an additional vote to itself through indirection...
And if you add one in both nodes you win absolute nothing, as the cluster
then expects 4 votes, with 3 for quorum. If one node (and it's VM) go
down you have 2 votes left, i.e. no quorum...
If you have a case where one node will always be up and the second may
be shutdown from time to time just give the first "primary" node two
votes via editing /etc/pve/corosync.conf [1]. Just increase the
'quorum_votes' for the respective node entry.
cheers,
Thomas

Just out of curiosity, does Proxmox support STONITH/node level fencing
using e.g. PDUs or UPS'?

There's a series from me adding this support, not yet applied.
It planned since start of PVE 4 but we did not get too much request and
poke-ings about it to accelerate the efforts (and as always a lot of other
things where to do), thus it only got code reviewed recently.
Actually we only talked about this today here and it's on my todo list to
finally apply it, *knocks wood*.

cheers,
Thomas

[Andreas Heinlein]

Post by Thomas Lamprecht

Post by Andreas Heinlein

Post by Thomas Lamprecht
please don't do that, uneccessarily complicated and not good practice
to depend an additional vote to itself through indirection...
And if you add one in both nodes you win absolute nothing, as the cluster
then expects 4 votes, with 3 for quorum. If one node (and it's VM) go
down you have 2 votes left, i.e. no quorum...
If you have a case where one node will always be up and the second may
be shutdown from time to time just give the first "primary" node two
votes via editing /etc/pve/corosync.conf [1]. Just increase the
'quorum_votes' for the respective node entry.
cheers,
Thomas

Just out of curiosity, does Proxmox support STONITH/node level fencing
using e.g. PDUs or UPS'?

There's a series from me adding this support, not yet applied.
It planned since start of PVE 4 but we did not get too much request and
poke-ings about it to accelerate the efforts (and as always a lot of other
things where to do), thus it only got code reviewed recently.
Actually we only talked about this today here and it's on my todo list to
finally apply it, *knocks wood*.
cheers,
Thomas

I might be interested, at least I could help testing. I have a APC
Master Switch AP9210 which is currently unused. I could probably also
test fencing through iLO. I will probably not find much time for that
before my holiday, though, which is in July/August. But let me know if I
can do something.

Bye,
Andreas

[Thomas Lamprecht]

Post by Tonči Stipičević
Hello to all
I'm testing pve-storage-zsync and it works pretty well on my 3node cluster .   One VM from origin node is replicating to other two nodes and when I shutdown origin node , I can start VM from it's replicated image on one of the other nodes w/o any issues/problems. If I make some changes to this replicated image , after starting "dead" node back again, replication overrides those changes and so on ...
So this is possible because we still have quorum .  However real scenario will have only two nodes in cluster and the problem arises when one node shuts down and there is no quorum any more.
Then we have to lower "votes" on the surviving node.
Since I want to make zfs-sync (disaster recovery) test with 2node cluster ,  the question would be: Is it possible to start "dead" node back again and restore "votes" value on the surviving node ?      Will the cluster accept this "dead" node back ?
I have never done this before  and just want to be prepared for reinstalling dead node if it cannot get back into the cluster again ....
Thank you very much in advance

personally i would just run a raspberry or similar proxmox node as a third tiebreaker node.

Would work. I often recommend qdevices[1] in this case,
instead of doing a full PVE cluster stack on a raspberry...

It's a simple daemon which gets polled when quroum changes
to act as a tie breaker.

You would need to install corosync-qnetd on the tie breaker,
e.g. Raspberry (it's packaged in Debian Stretch).

Then ensure you can access the external device (raspi) via
ssh: ssh-copy-id ***@qdev-address

Then you can setup the connection with:
# corosync-qdevice-net-certutil -Q -n <clustername> <qdev-address> <cluster-node1-addr> <cluster-node2-addr>

(with <xyz> replaced with the respective values)

Ensure edevice is started on external:

# systemctl restart corosync-qnetd.service

The edit corosync.conf [2] and add a device section to the quorum
section, e.g., my quorum section looks like:

[...]

quorum {
device {
model: net
net {
algorithm: ffsplit
host: 192.168.30.15
tls: on
}
votes: 1
}
provider: corosync_votequorum
}

Then on all cluster nodes ensure that the qdevice service is running:
# systemctl restart corosync-qdeviece.service

And you should be done.

It may not look to easy to setup, but it is not hard either, especially if
you read through the man page a bit. Note that only cluster with even node
count should do this, you will reduce reliability with uneven count, as there
ffsplit algorithm cannot be used and last-man-standing has a few very
non-ideal properties...
For even node count you can only win, if setup correctly.

Just typing this up because I find it a nice workaround for 2 + 1 clusters,
e.g. 2 nodes PVE plus a Linux storage/network box or non-powerful tie
breaker, and it seems to be not that known :)

cheers,
Thomas


[1]: https://www.mankier.com/8/corosync-qdevice
[2]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#edit-corosync-conf