Phil Kauffman
2018-11-13 17:54:57 UTC
I use my proxmox node at home to host VM's but also for my main ZFS array.
I use a tool called 'znapzend' to mirror various datasets to other servers.
Recently I noticed the following issue while using the 'znapzendzetup' command. 'znapzendzetup' will show you the backup plans you have set for any dataset.
I was hoping someone could help me troubleshoot this issue. It feels very much like this one (https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1763067), but I'm not sure.
# znapzendzetup list
# ... it just hangs here
OK. so strace it...
# strace znapzendzetup list
Here are the last few lines of the above strace
<snip>
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "org.znapzend:dst_a_plan\t60minute"..., 8192) = 577
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28633, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]}) = 0
read(4, "", 8192) = 0
close(4) = 0
wait4(28633, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 28633
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3585, ...}) = 0
pipe([4, 5]) = 0
pipe([6, 7]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb932193590) = 28634
close(7) = 0
close(5) = 0
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "tank/audio\n", 8192) = 11
read(4, "", 8192) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28634, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]}) = 0
close(4) = 0
wait4(28634, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 28634
pipe([4, 5]) = 0
pipe([6, 7]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb932193590) = 28635
close(7) = 0
close(5) = 0
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4,
</snip>
# uname -a
Linux luna 4.15.18-8-pve #1 SMP PVE 4.15.18-28 (Tue, 30 Oct 2018 14:27:50 +0100) x86_64 GNU/Linux
# dmesg| grep ZFS
[ 7.059616] ZFS: Loaded module v0.7.11-1, ZFS pool version 5000, ZFS filesystem version 5
# zpool status
pool: proxmox
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0h54m with 0 errors on Sun Nov 11 01:18:13 2018
config:
NAME STATE READ WRITE CKSUM
proxmox ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
<redacted_ssd1> ONLINE 0 0 0
<redacted_ssd2> ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: scrub repaired 0B in 17h32m with 0 errors on Sun Nov 11 17:57:00 2018
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
<redacted_hdd1> ONLINE 0 0 0
<redacted_hdd2> ONLINE 0 0 0
<redacted_hdd3> ONLINE 0 0 0
<redacted_hdd4> ONLINE 0 0 0
Cheers,
Phil
I use a tool called 'znapzend' to mirror various datasets to other servers.
Recently I noticed the following issue while using the 'znapzendzetup' command. 'znapzendzetup' will show you the backup plans you have set for any dataset.
I was hoping someone could help me troubleshoot this issue. It feels very much like this one (https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1763067), but I'm not sure.
# znapzendzetup list
# ... it just hangs here
OK. so strace it...
# strace znapzendzetup list
Here are the last few lines of the above strace
<snip>
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "org.znapzend:dst_a_plan\t60minute"..., 8192) = 577
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28633, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]}) = 0
read(4, "", 8192) = 0
close(4) = 0
wait4(28633, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 28633
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3585, ...}) = 0
pipe([4, 5]) = 0
pipe([6, 7]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb932193590) = 28634
close(7) = 0
close(5) = 0
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4, "tank/audio\n", 8192) = 11
read(4, "", 8192) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=28634, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]}) = 0
close(4) = 0
wait4(28634, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 28634
pipe([4, 5]) = 0
pipe([6, 7]) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb932193590) = 28635
close(7) = 0
close(5) = 0
read(6, "", 4) = 0
close(6) = 0
ioctl(4, TCGETS, 0x7ffef62d8970) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
read(4,
</snip>
# uname -a
Linux luna 4.15.18-8-pve #1 SMP PVE 4.15.18-28 (Tue, 30 Oct 2018 14:27:50 +0100) x86_64 GNU/Linux
# dmesg| grep ZFS
[ 7.059616] ZFS: Loaded module v0.7.11-1, ZFS pool version 5000, ZFS filesystem version 5
# zpool status
pool: proxmox
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0h54m with 0 errors on Sun Nov 11 01:18:13 2018
config:
NAME STATE READ WRITE CKSUM
proxmox ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
<redacted_ssd1> ONLINE 0 0 0
<redacted_ssd2> ONLINE 0 0 0
errors: No known data errors
pool: tank
state: ONLINE
scan: scrub repaired 0B in 17h32m with 0 errors on Sun Nov 11 17:57:00 2018
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
<redacted_hdd1> ONLINE 0 0 0
<redacted_hdd2> ONLINE 0 0 0
<redacted_hdd3> ONLINE 0 0 0
<redacted_hdd4> ONLINE 0 0 0
Cheers,
Phil