[PVE-User] sparse and compression

Hi Migual,

first at all: man zfs!

Post by Miguel GonzÃ¡lez
Is it advisable to use sparse on ZFS pools performance wise? And
compression? Which kind of compression?

Sparse or not doesn't matter on SSDs. I would use compression because of less
r/w to the disc and modern CPUs can handle lz4 quite well.

Also keep in mind: A sparse volume only stays sparse if trim/discard is used!

volblocksize is important: ZFS is using 8k as default. For ZFS filesystem a
recordsize of 128K is used.

Some older test:
zpool/vm-zvols/bsize_1k written 10.3G
zpool/vm-zvols/bsize_1k logicalused 1.82G
zpool/vm-zvols/bsize_4k written 2.60G
zpool/vm-zvols/bsize_4k logicalused 1.76G
zpool/vm-zvols/bsize_8k written 2.60G
zpool/vm-zvols/bsize_8k logicalused 1.78G
zpool/vm-zvols/bsize_16k written 1.87G
zpool/vm-zvols/bsize_16k logicalused 1.70G
zpool/vm-zvols/bsize_32k written 1.87G
zpool/vm-zvols/bsize_32k logicalused 1.71G
zpool/vm-zvols/bsize_64k written 1.72G
zpool/vm-zvols/bsize_64k logicalused 1.72G
zpool/vm-zvols/bsize_128k written 1.75G
zpool/vm-zvols/bsize_128k logicalused 1.75G

Post by Miguel GonzÃ¡lez
Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

No, sparse or not is set at creation.

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

Because of blocksizes. Check zfs get all and read theory about ZFS.

Here's an example for a non-sparse 50GB Volume for a VM:

zpool/vm-zvols/foobar 51.6G 2.19T 34.6G -
zpool/vm-zvols/foobar used 51.6G
zpool/vm-zvols/foobar referenced 34.6G
zpool/vm-zvols/foobar compressratio 1.02x
zpool/vm-zvols/foobar volsize 50G
zpool/vm-zvols/foobar volblocksize 8K
zpool/vm-zvols/foobar compression lz4
zpool/vm-zvols/foobar refreservation 51.6G
zpool/vm-zvols/foobar usedbydataset 34.6G
zpool/vm-zvols/foobar usedbyrefreservation 17.0G
zpool/vm-zvols/foobar refcompressratio 1.02x
zpool/vm-zvols/foobar written 34.6G
zpool/vm-zvols/foobar logicalused 24.2G
zpool/vm-zvols/foobar logicalreferenced 24.2G

Andreas

Miguel González

2017-12-11 14:27:47 UTC

Post by Andreas Herrmann
Also keep in mind: A sparse volume only stays sparse if trim/discard is used!

How do I use trim/discard? Has this to be set in guest level, right?

Post by Andreas Herrmann
volblocksize is important: ZFS is using 8k as default. For ZFS filesystem a
recordsize of 128K is used.

Can I change recordsize to 128K after creation or do I need to create a
new zpool for that?

Thanks for your promptly answer!

Miguel

---
This email has been checked for viruses by AVG.
http://www.avg.com

Andreas Herrmann

2017-12-11 14:35:35 UTC

Hi,

Post by Andreas Herrmann
Also keep in mind: A sparse volume only stays sparse if trim/discard is used!

How do I use trim/discard? Has this to be set in guest level, right?

https://pve.proxmox.com/wiki/Qemu_trim/discard_and_virtio_scsi

Post by Andreas Herrmann
volblocksize is important: ZFS is using 8k as default. For ZFS filesystem a
recordsize of 128K is used.

Can I change recordsize to 128K after creation or do I need to create a
new zpool for that?

Play and learn:

zfs set volblocksize=16K zpool/vm-zvols/test
cannot set property for 'zpool/vm-zvols/test': 'volblocksize' is readonly

You really should read 'man zfs'

Andreas

Fabian Grünbichler

2017-12-11 13:17:31 UTC

Post by Miguel GonzÃ¡lez
Dear all,
Is it advisable to use sparse on ZFS pools performance wise? And
compression? Which kind of compression?

sparse just tells ZFS to not reserve space, it does not make a
difference performance wise. if you do over provision and attempt to use
more space than you actuall have, you can corrupt volumes / run into I/O
errors though, like with most storages.

compression is advisable, it costs (almost) nothing and usually
increases performance and saves space. the default (on which is lz4) is
fine.

Post by Miguel GonzÃ¡lez
Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

sparse will only affect newly created volumes. you can "convert" sparse
volumes to fully reserved ones and vice versa manually though.

compression only affects data written after it has been enabled, and
already written data stays compressed if you turn it off again. if you
want to fully switch from compressed to uncompressed or vice versa, you
need to re-write all the data.

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

Miguel González

2017-12-11 14:23:34 UTC

Post by Miguel GonzÃ¡lez
Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

sparse will only affect newly created volumes. you can "convert" sparse
volumes to fully reserved ones and vice versa manually though.

how can I convert manually from non-sparse to sparse? Creating a new
zpool and copy disk with dd? Or any other easier way?

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

Yes, raid1

Thanks for your promptly reply!

Miguel

---
This email has been checked for viruses by AVG.
http://www.avg.com

Fabian Grünbichler

2017-12-11 14:29:04 UTC

Post by Miguel GonzÃ¡lez
Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

sparse will only affect newly created volumes. you can "convert" sparse
volumes to fully reserved ones and vice versa manually though.

how can I convert manually from non-sparse to sparse? Creating a new
zpool and copy disk with dd? Or any other easier way?

(un)set the reservations appropriately. like I said, "sparse" is
entirely virtual for ZFS, the only difference is whether the full size
is reserved upon creation or not.

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

Yes, raid1
Thanks for your promptly reply!

raid1 (aka mirror)? or raidZ-1 ? those are two very different things ;)

Miguel González

2017-12-11 14:34:37 UTC

Post by Miguel GonzÃ¡lez
Can I change a zpool to sparse on the fly or do I need to turn off all
VMs before doing so?

sparse will only affect newly created volumes. you can "convert" sparse
volumes to fully reserved ones and vice versa manually though.

how can I convert manually from non-sparse to sparse? Creating a new
zpool and copy disk with dd? Or any other easier way?

(un)set the reservations appropriately. like I said, "sparse" is
entirely virtual for ZFS, the only difference is whether the full size
is reserved upon creation or not.

from Andreas comment maybe i should look more into change blocksize.

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

Yes, raid1
Thanks for your promptly reply!

raid1 (aka mirror)? or raidZ-1 ? those are two very different things ;)

from zfs perspective is called mirror-0 (not softraid underneath):

zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not
support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0h39m with 0 errors on Sun Dec 10 02:03:43 2017
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0

---
This email has been checked for viruses by AVG.
http://www.avg.com

Andreas Herrmann

2017-12-11 14:47:56 UTC

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

No! 'zpool list' will show what is used on disk. zfs list is totally
transparent to zpool layout. Have a look at 'zpool get all' for the ashift
setting.

Example for raidz1 (4x 960GB SSDs):
***@foobar:~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpool 3.41T 102G 3.31T - 8% 2% 1.00x ONLINE -

***@foobar:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zpool 237G 2.17T 140K /zpool

zpool ALLOC is smaller than zfs USED in this example. Why? Try to unserstand
the difference between 'referenced' and 'used'. My volumes aren't sparse but
discard is used.

Andreas

Miguel González

2017-12-11 15:10:29 UTC

Post by Andreas Herrmann
Hi

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

No! 'zpool list' will show what is used on disk. zfs list is totally
transparent to zpool layout. Have a look at 'zpool get all' for the ashift
setting.
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpool 3.41T 102G 3.31T - 8% 2% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
zpool 237G 2.17T 140K /zpool
zpool ALLOC is smaller than zfs USED in this example. Why? Try to unserstand
the difference between 'referenced' and 'used'. My volumes aren't sparse but
discard is used.

I have search around about how to understand those columns. I didn´t
find anything on the wiki that explains this.

This is my zfs list:

NAME USED AVAIL REFER MOUNTPOINT
rpool 207G 61.9G 104K /rpool
rpool/ROOT 6.10G 61.9G 96K /rpool/ROOT
rpool/ROOT/pve-1 6.10G 61.9G 6.10G /
rpool/data 197G 61.9G 96K /rpool/data
rpool/data/vm-100-disk-1 108G 61.9G 108G -
rpool/data/vm-102-disk-1 37.1G 77.9G 21.1G -
rpool/data/vm-102-disk-2 51.6G 81.8G 31.7G -
rpool/swap 4.25G 64.9G 1.25G -

If I run zfs get all I get:

rpool/data/vm-100-disk-1 written 108G
rpool/data/vm-100-disk-1 logicalused 129G
rpool/data/vm-100-disk-1 logicalreferenced 129G

rpool/data/vm-102-disk-1 written 21.1G
rpool/data/vm-102-disk-1 logicalused 27.1G
rpool/data/vm-102-disk-1 logicalreferenced 27.1G

rpool/data/vm-102-disk-2 written 31.7G
rpool/data/vm-102-disk-2 logicalused 36.2G
rpool/data/vm-102-disk-2 logicalreferenced 36.2G

So even If I´m having 8k blocksize and non-sparse the written data is
quite close to the real usage in the guests VMs.

All this comes from that I was running out of space when running
pve-zsync to perform a copy of the VM in other node.

I have found out that snapshots were taken some part of the data (30 Gb).

Any way to run a pve-zsync only a day that doesn´t consume snapshots on
this machine (Maybe running from the target machine?)

Thanks

Miguel

---
This email has been checked for viruses by AVG.
http://www.avg.com

Andreas Herrmann

2017-12-11 15:22:59 UTC

Hi Miguel,

Post by Andreas Herrmann
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zpool 3.41T 102G 3.31T - 8% 2% 1.00x ONLINE -
NAME USED AVAIL REFER MOUNTPOINT
zpool 237G 2.17T 140K /zpool
zpool ALLOC is smaller than zfs USED in this example. Why? Try to unserstand
the difference between 'referenced' and 'used'. My volumes aren't sparse but
discard is used.

I have search around about how to understand those columns. I didn´t
find anything on the wiki that explains this.

Why should Proxmox explain the theory of ZFS? Please have a look at 'man zfs'.
There you'll find all you need.

Andreas

Fabian Grünbichler

2017-12-11 15:37:57 UTC

Post by Andreas Herrmann
Hi

Post by Miguel GonzÃ¡lez
Why a virtual disk shows as 60G when originally It was 36 Gb in raw format?
NAME USED AVAIL REFER MOUNTPOINT
rpool/data/vm-102-disk-1 60.0G 51.3G 20.9G -

wild guess - you are using raidz of some kind? ashift is set to 12 /
auto-detected?

No! 'zpool list' will show what is used on disk. zfs list is totally
transparent to zpool layout. Have a look at 'zpool get all' for the ashift
setting.

I know. in most cases when people are surprised by their zvols taking up
more space than expected, it is because they are using raidz and don't
know about the interaction between ashift=12, raidz and small
volblocksize.

your output is pretty worthless, as "REFER" only refers to the pool
dataset, and not its children. I do know the difference between used and
referenced, which is not (directly) related to discard at all. discard
can obviously get your referenced value down ;)

see the following for an example where a 10G volume takes more than 10G
of space in 'zfs list' output:

$ zfs list testpool -r -o name,used,referenced,volsize
NAME USED REFER VOLSIZE
testpool 14.3G 140K -
testpool/test 14.3G 14.3G 10G

$ zpool list testpool
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 39.8G 19.7G 20.1G - 0% 49% 1.00x ONLINE -

the only difference between a sparse and non-sparse zvol is whether
refreservation is set, which affects the usedbyrefreservation value
which in turn (might / probably will) affect the used value. no relation
to discard at all.

Miguel González

2017-12-11 16:05:17 UTC