Discussion:
[PVE-User] DRBD sync speed
Adam Weremczuk
2018-10-10 13:23:50 UTC
Permalink
Hi all,

I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2

I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode
balance-rr.

The bond is working fine; I get a transfer rate of 150 MB/s with scp.

Following this guide:
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
was going  smoothly up until:

drbdadm -- --overwrite-data-of-peer primary r0/0

cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:3898301536
    [>....................] sync'ed:  0.1% (3806932/3806944)M
    finish: 483:25:13 speed: 2,188 (2,188) K/sec

The transfer rate is horribly slow and at this pace it's going to take
20 days for two 4 TB volumes to sync!

That's almost 15 times slower comparing with the guide video (8:30):


The volumes have been zeroed and contain no live data yet.

My sdb disks are logical drives (hardware RAID) set up as RAID50 with
the defaults:

Strip size: 128 KB
Access policy: RW
Read policy: Normal
Write policy: Write Back with BBU
IO policy: Direct
Drive Cache: Disable
Disable BGI: No

Performance looks good when tested with hdparm:

hdparm -tT /dev/sdb1

/dev/sdb1:
 Timing cached reads:   15056 MB in  1.99 seconds = 7550.46 MB/sec
 Timing buffered disk reads: 2100 MB in  3.00 seconds = 699.81 MB/sec

The volumes have been zeroed and contain no live data yet.

Any idea why the sync rate is so painfully slow and how to improve it?

Regards,
Adam
dorsy
2018-10-10 13:25:50 UTC
Permalink
DRBD related, google for DRBD resync speed :)
Post by Adam Weremczuk
Hi all,
I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
I have 2 identical servers connected with 2 x 1 Gbps links in
bond_mode balance-rr.
The bond is working fine; I get a transfer rate of 150 MB/s with scp.
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
drbdadm -- --overwrite-data-of-peer primary r0/0
cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1
wo:f oos:3898301536
    [>....................] sync'ed:  0.1% (3806932/3806944)M
    finish: 483:25:13 speed: 2,188 (2,188) K/sec
The transfer rate is horribly slow and at this pace it's going to take
20 days for two 4 TB volumes to sync!
http://youtu.be/WQGi8Nf0kVc
The volumes have been zeroed and contain no live data yet.
My sdb disks are logical drives (hardware RAID) set up as RAID50 with
Strip size: 128 KB
Access policy: RW
Read policy: Normal
Write policy: Write Back with BBU
IO policy: Direct
Drive Cache: Disable
Disable BGI: No
hdparm -tT /dev/sdb1
 Timing cached reads:   15056 MB in  1.99 seconds = 7550.46 MB/sec
 Timing buffered disk reads: 2100 MB in  3.00 seconds = 699.81 MB/sec
The volumes have been zeroed and contain no live data yet.
Any idea why the sync rate is so painfully slow and how to improve it?
Regards,
Adam
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Gilberto Nunes
2018-10-10 13:27:07 UTC
Permalink
Hi

I think this problem is related DRDB and not PVE!

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36
Post by Adam Weremczuk
Hi all,
I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode
balance-rr.
The bond is working fine; I get a transfer rate of 150 MB/s with scp.
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
drbdadm -- --overwrite-data-of-peer primary r0/0
cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:3898301536
[>....................] sync'ed: 0.1% (3806932/3806944)M
finish: 483:25:13 speed: 2,188 (2,188) K/sec
The transfer rate is horribly slow and at this pace it's going to take
20 days for two 4 TB volumes to sync!
http://youtu.be/WQGi8Nf0kVc
The volumes have been zeroed and contain no live data yet.
My sdb disks are logical drives (hardware RAID) set up as RAID50 with
Strip size: 128 KB
Access policy: RW
Read policy: Normal
Write policy: Write Back with BBU
IO policy: Direct
Drive Cache: Disable
Disable BGI: No
hdparm -tT /dev/sdb1
Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec
Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec
The volumes have been zeroed and contain no live data yet.
Any idea why the sync rate is so painfully slow and how to improve it?
Regards,
Adam
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Adam Weremczuk
2018-10-10 13:34:21 UTC
Permalink
Yes, there must be something wrong with the defaults.

This seems to be a good starting point:
https://serverfault.com/questions/740311/drbd-terrible-sync-performance-on-10gige/740370

It looks more complex than I expected.
Post by Gilberto Nunes
Hi
I think this problem is related DRDB and not PVE!
---
Gilberto Nunes Ferreira
(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram
Skype: gilberto.nunes36
Post by Adam Weremczuk
Hi all,
I'm trying out DRBD Pacemaker HA Cluster on Proxmox 5.2
I have 2 identical servers connected with 2 x 1 Gbps links in bond_mode
balance-rr.
The bond is working fine; I get a transfer rate of 150 MB/s with scp.
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
drbdadm -- --overwrite-data-of-peer primary r0/0
cat /proc/drbd
version: 8.4.10 (api:1/proto:86-101)
srcversion: 17A0C3A0AF9492ED4B9A418
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:10944 nr:0 dw:0 dr:10992 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f
oos:3898301536
[>....................] sync'ed: 0.1% (3806932/3806944)M
finish: 483:25:13 speed: 2,188 (2,188) K/sec
The transfer rate is horribly slow and at this pace it's going to take
20 days for two 4 TB volumes to sync!
http://youtu.be/WQGi8Nf0kVc
The volumes have been zeroed and contain no live data yet.
My sdb disks are logical drives (hardware RAID) set up as RAID50 with
Strip size: 128 KB
Access policy: RW
Read policy: Normal
Write policy: Write Back with BBU
IO policy: Direct
Drive Cache: Disable
Disable BGI: No
hdparm -tT /dev/sdb1
Timing cached reads: 15056 MB in 1.99 seconds = 7550.46 MB/sec
Timing buffered disk reads: 2100 MB in 3.00 seconds = 699.81 MB/sec
The volumes have been zeroed and contain no live data yet.
Any idea why the sync rate is so painfully slow and how to improve it?
Regards,
Adam
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Loading...