Discussion:
[PVE-User] PVE HBA best practice
Gilberto Nunes
2018-04-20 15:12:43 UTC
Permalink
Hi there PVE Users!

I have here a PowerEdge Dell R740 and a Storage MD1400.
This is a DAS Dell solution.
Further, we will have another server, that we will buy later, to compose a
cluster/HA solution.
So, what is the proper way to use the HBA within Proxmox, in order to make
it reliable and prepared to work later as Cluster/HA solution?
May I have ZFS pools?
May I have see the disk as local discs, and use it directly or what?

I need some guide in this matter, some doc will be appreciated.

Thanks a lot.
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36
Alex Chekholko
2018-04-20 19:08:49 UTC
Permalink
"if you can connect the MD1400 to 2 servers at the same time you may have
some kind of SAS failover cluster. "
https://github.com/ewwhite/zfs-ha/wiki
but it will need to be tested to see how it interacts with PVE. Should be
OK if you just use it as an NFSv3 storage pool in PVE.
Post by Gilberto Nunes
Hi there PVE Users!
I have here a PowerEdge Dell R740 and a Storage MD1400.
This is a DAS Dell solution.
Further, we will have another server, that we will buy later, to compose a
cluster/HA solution.
So, what is the proper way to use the HBA within Proxmox, in order to make
it reliable and prepared to work later as Cluster/HA solution?
May I have ZFS pools?
May I have see the disk as local discs, and use it directly or what?
I need some guide in this matter, some doc will be appreciated.
Thanks a lot.
---
Gilberto Nunes Ferreira
if you only have directly attached storage it is hard to make a HA
solution. since once the server with the storage dies the other can not see
it any more.
normally in proxmox you have 3 servers (you can use a third lightweight
server as witness if you only need 2 servers. eg a raspberry pi or similar)
All the 3 servers can access the same storage. and run the same vm's from
the shared storage. the shared storage can be a ceph cluster. a SAN, or NAS
solution. but it should also be HA or you have just moved the Single point
of failure from the vm hypervisor to the storage solution. Now if a
server dies, the leftover quorum of 2 servers decides the one server is
gone and starts it's VM's on one of the 2 remaining servers.
normally the single server loosing connection to the quorum should stop
all vm's. but you should implement a STONITH solution to prevent vm disk
image corruption.
with directly attached storage in servers you have a problem. you can
either have storage in all servers and use zfs replication between them. it
will not be as up to date as a shared system when it fail tho. Or you can
build a HA storage system as a backend for your proxmox HA virtualization
system.
a server in front of the MD1400 can be a NFS/ISCSI server allowing
multiple proxmox servers to use it as shared storage for proxmox HA. the
storage solution is single point of failure.
if you can connect the MD1400 to 2 servers at the same time you may have
some kind of SAS failover cluster. something like HA-LVM
https://www.linuxjournal.com/content/high-availability-storage-ha-lvm.
MD1400 will be the new single point of failure. but it is less complex then
a server so less likely to die. but HA-LVM is added complexity. and
sometimes a simple NFS server is good enough (tm)
if you have 4 similarly sized servers you can build a ceph cluster.
https://pve.proxmox.com/wiki/Ceph_Server you can even install proxmox
and ceph together on 4 nodes in a hyperconverged solution, but this do
require intimate knowledge of ceph and proxmox and enough juice in your
server to deal with both jobs at the same time (especially in a failure
situation when load will spike) https://pve.proxmox.com/wiki/H
yper-converged_Infrastructure
good luck with proxmox !
Ronny Aasen
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Gilberto Nunes
2018-04-20 19:17:56 UTC
Permalink
Hi there
Even with FC Switch will be difficult to implement HA solution, since, I
guest, each PVE server will see the disks as local disks...

Perhaps I can achieve Storage Replication wich use ZFS replication to do
the job.




---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36
Post by Alex Chekholko
"if you can connect the MD1400 to 2 servers at the same time you may have
some kind of SAS failover cluster. "
https://github.com/ewwhite/zfs-ha/wiki
but it will need to be tested to see how it interacts with PVE. Should be
OK if you just use it as an NFSv3 storage pool in PVE.
Post by Gilberto Nunes
Hi there PVE Users!
I have here a PowerEdge Dell R740 and a Storage MD1400.
This is a DAS Dell solution.
Further, we will have another server, that we will buy later, to
compose a
Post by Gilberto Nunes
cluster/HA solution.
So, what is the proper way to use the HBA within Proxmox, in order to
make
Post by Gilberto Nunes
it reliable and prepared to work later as Cluster/HA solution?
May I have ZFS pools?
May I have see the disk as local discs, and use it directly or what?
I need some guide in this matter, some doc will be appreciated.
Thanks a lot.
---
Gilberto Nunes Ferreira
if you only have directly attached storage it is hard to make a HA
solution. since once the server with the storage dies the other can not
see
it any more.
normally in proxmox you have 3 servers (you can use a third lightweight
server as witness if you only need 2 servers. eg a raspberry pi or
similar)
All the 3 servers can access the same storage. and run the same vm's from
the shared storage. the shared storage can be a ceph cluster. a SAN, or
NAS
solution. but it should also be HA or you have just moved the Single
point
of failure from the vm hypervisor to the storage solution. Now if a
server dies, the leftover quorum of 2 servers decides the one server is
gone and starts it's VM's on one of the 2 remaining servers.
normally the single server loosing connection to the quorum should stop
all vm's. but you should implement a STONITH solution to prevent vm disk
image corruption.
with directly attached storage in servers you have a problem. you can
either have storage in all servers and use zfs replication between them.
it
will not be as up to date as a shared system when it fail tho. Or you can
build a HA storage system as a backend for your proxmox HA virtualization
system.
a server in front of the MD1400 can be a NFS/ISCSI server allowing
multiple proxmox servers to use it as shared storage for proxmox HA. the
storage solution is single point of failure.
if you can connect the MD1400 to 2 servers at the same time you may have
some kind of SAS failover cluster. something like HA-LVM
https://www.linuxjournal.com/content/high-availability-storage-ha-lvm.
MD1400 will be the new single point of failure. but it is less complex
then
a server so less likely to die. but HA-LVM is added complexity. and
sometimes a simple NFS server is good enough (tm)
if you have 4 similarly sized servers you can build a ceph cluster.
https://pve.proxmox.com/wiki/Ceph_Server you can even install proxmox
and ceph together on 4 nodes in a hyperconverged solution, but this do
require intimate knowledge of ceph and proxmox and enough juice in your
server to deal with both jobs at the same time (especially in a failure
situation when load will spike) https://pve.proxmox.com/wiki/H
yper-converged_Infrastructure
good luck with proxmox !
Ronny Aasen
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Simone Piccardi
2018-04-26 17:13:11 UTC
Permalink
Post by Gilberto Nunes
Hi there
Even with FC Switch will be difficult to implement HA solution, since, I
guest, each PVE server will see the disks as local disks...
You can use LVM (not thin) as shared storage for disk images.

Simone
--
Simone Piccardi Truelite Srl
***@truelite.it (email/jabber) Via Monferrato, 6
Tel. +39-347-1032433 50142 Firenze
http://www.truelite.it Tel. +39-055-7879597 Fax. +39-055-7333336
Stefan Sänger
2018-04-21 06:47:41 UTC
Permalink
Hi Gilberto,

the Powevault MD1xxx series are only shelved with SAS support, but
without the actual controller. They are intended as expansion units -
either to a single server (which is basically like making the case
bigger and adding more disks) or to extend an actual storage.

If you go for a MD3xxx device you actually get what you want - available
with SAS, FC or iSCSI.

Whila a storage switch for this is recommended, with only 2 attached
servers it is not necessarily requried.

HA solution then leans towards LVM storage.

We actually have a HA solution including SAS switches and 10 servers
(mixed, 4 rack servers and 6 blades in a small bladecenter) running for
a couple of years now.

I would always recommend using Switches and setting up multipath here...


best regards,

Stefan
Post by Gilberto Nunes
Hi there PVE Users!
I have here a PowerEdge Dell R740 and a Storage MD1400.
This is a DAS Dell solution.
Further, we will have another server, that we will buy later, to compose a
cluster/HA solution.
So, what is the proper way to use the HBA within Proxmox, in order to make
it reliable and prepared to work later as Cluster/HA solution?
May I have ZFS pools?
May I have see the disk as local discs, and use it directly or what?
I need some guide in this matter, some doc will be appreciated.
Thanks a lot.
---
Gilberto Nunes Ferreira
(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram
Skype: gilberto.nunes36
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Gilberto Nunes
2018-04-21 11:40:31 UTC
Permalink
Hi

Thanks you guys, for advice.


---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36
Post by Stefan Sänger
Hi Gilberto,
the Powevault MD1xxx series are only shelved with SAS support, but without
the actual controller. They are intended as expansion units - either to a
single server (which is basically like making the case bigger and adding
more disks) or to extend an actual storage.
If you go for a MD3xxx device you actually get what you want - available
with SAS, FC or iSCSI.
Whila a storage switch for this is recommended, with only 2 attached
servers it is not necessarily requried.
HA solution then leans towards LVM storage.
We actually have a HA solution including SAS switches and 10 servers
(mixed, 4 rack servers and 6 blades in a small bladecenter) running for a
couple of years now.
I would always recommend using Switches and setting up multipath here...
best regards,
Stefan
Post by Gilberto Nunes
Hi there PVE Users!
I have here a PowerEdge Dell R740 and a Storage MD1400.
This is a DAS Dell solution.
Further, we will have another server, that we will buy later, to compose a
cluster/HA solution.
So, what is the proper way to use the HBA within Proxmox, in order to make
it reliable and prepared to work later as Cluster/HA solution?
May I have ZFS pools?
May I have see the disk as local discs, and use it directly or what?
I need some guide in this matter, some doc will be appreciated.
Thanks a lot.
---
Gilberto Nunes Ferreira
(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram
Skype: gilberto.nunes36
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Loading...