Discussion:
[PVE-User] Backup to samba share stalling/locking up
Lindsay Mathieson
2014-02-24 06:38:17 UTC
Permalink
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.

It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.

The samba share is mounted via the following in fstab:

//LOB/backup-offsite /mnt/smb-backup cifs username=******,password=****** 0 0


This is happening prior to the last set of updates and after they are applied.

Any ideas?

Side Note: It is proving incredibly difficult to setup a backup for
vm's on all nodes to a usb drive.
--
Lindsay
Lindsay Mathieson
2014-02-24 06:41:48 UTC
Permalink
On 24 February 2014 16:38, Lindsay Mathieson
Post by Lindsay Mathieson
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.
It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.
Seeing this in dmesg

CIFS VFS: No writable handles for inode
Post by Lindsay Mathieson
--
Lindsay
--
Lindsay
Lindsay Mathieson
2014-02-24 07:28:33 UTC
Permalink
On 24 February 2014 16:41, Lindsay Mathieson
Post by Lindsay Mathieson
On 24 February 2014 16:38, Lindsay Mathieson
Post by Lindsay Mathieson
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.
It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.
Seeing this in dmesg
CIFS VFS: No writable handles for inode
adding forcedirectio to the cifs mount options seems to have resolved the issue.

Going to test with oplocks turned off at the server side as well.
Bart Lageweg | Bizway
2014-02-24 08:00:30 UTC
Permalink
Hi Lindasy,

We are using the rack QNAP on 2 locations with NFS, backup first to local and copy to QNAP (40+ nodes).

/etc/vzdump.conf

Script: /usr/local/bin/backup-hook.pl

#!/usr/bin/perl -w
# hook script for vzdump (--script option
use strict;

print "HOOK: " . join (' ', @ARGV) . "\n";

# config
my $phase = shift;
my $mode = shift; # stop/suspend/snapshot
my $vmid = shift;
my $vmtype = $ENV{VMTYPE}; # openvz/qemu
my $dumpdir = $ENV{DUMPDIR};
my $hostname = $ENV{HOSTNAME};
my $tarfile = $ENV{TARFILE};
my $logfile = $ENV{LOGFILE};
my $backuplocatie = "/mnt/pve/sharenameNFS/dump";

my %dispatch = (
"job-start" => \&nop,
"job-end" => \&nop,
"job-abort" => \&nop,
"backup-start" => \&nop,
"backup-end" => \&backup_end,
"backup-abort" => \&nop,
"log-end" => \&log_end,
"pre-stop" => \&nop,
"pre-restart" => \&nop,
);

sub upload {
system("mv -f $backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo $backuplocatie/vzdump-qemu-oud-$hostname-$vmid.vma.lzo");
system("cp $tarfile $backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo") == 0 ||
die "upload cp $tarfile $backuplocatie/dump/vzdump-qemu-$hostname-$vmid.vma.tar.lzo backup-host failed";

print "HOOK: upload " . $tarfile . " to $backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo done\n"; }

sub nop {
# nothing
}

sub backup_end {
upload($tarfile);
}

sub log_end {
# upload($logfile);
}





-----Oorspronkelijk bericht-----
Van: pve-user-***@pve.proxmox.com [mailto:pve-user-***@pve.proxmox.com] Namens Lindsay Mathieson
Verzonden: maandag 24 februari 2014 7:42
Aan: ProxMox Users
Onderwerp: Re: [PVE-User] Backup to samba share stalling/locking up
Post by Lindsay Mathieson
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.
It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.
Seeing this in dmesg

CIFS VFS: No writable handles for inode
Post by Lindsay Mathieson
--
Lindsay
--
Lindsay
Pongrácz István
2014-02-24 08:31:25 UTC
Permalink
Hi Bart,

May I ask, how much time does it take to complete the backup and how much disk space does it use?

Thank you!
Best regards,
István



----------------eredeti üzenet-----------------
Feladó: "Bart Lageweg | Bizway" ***@bizway.nl
Címzett: "'Lindsay Mathieson'"
, "ProxMox Users"
Dátum: Mon, 24 Feb 2014 08:00:30 +0000
-------------------------------------------------
Post by Bart Lageweg | Bizway
Hi Lindasy,
We are using the rack QNAP on 2 locations with NFS, backup first to local and copy to
QNAP (40+ nodes).
/etc/vzdump.conf
Script: /usr/local/bin/backup-hook.pl
#!/usr/bin/perl -w
# hook script for vzdump (--script option
use strict;
# config
my $phase = shift;
my $mode = shift; # stop/suspend/snapshot
my $vmid = shift;
my $vmtype = $ENV{VMTYPE}; # openvz/qemu
my $dumpdir = $ENV{DUMPDIR};
my $hostname = $ENV{HOSTNAME};
my $tarfile = $ENV{TARFILE};
my $logfile = $ENV{LOGFILE};
my $backuplocatie = "/mnt/pve/sharenameNFS/dump";
my %dispatch = (
"job-start" => \&nop,
"job-end" => \&nop,
"job-abort" => \&nop,
"backup-start" => \&nop,
"backup-end" => \&backup_end,
"backup-abort" => \&nop,
"log-end" => \&log_end,
"pre-stop" => \&nop,
"pre-restart" => \&nop,
);
sub upload {
system("mv -f $backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo
$backuplocatie/vzdump-qemu-oud-$hostname-$vmid.vma.lzo");
system("cp $tarfile
$backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo") == 0 ||
die "upload cp $tarfile
$backuplocatie/dump/vzdump-qemu-$hostname-$vmid.vma.tar.lzo
backup-host failed";
print "HOOK: upload " . $tarfile . " to
$backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo done\n"; }
sub nop {
# nothing
}
sub backup_end {
upload($tarfile);
}
sub log_end {
# upload($logfile);
}
-----Oorspronkelijk bericht-----
Verzonden: maandag 24 februari 2014 7:42
Aan: ProxMox Users
Onderwerp: Re: [PVE-User] Backup to samba share stalling/locking up
On 24 February 2014 16:38, Lindsay Mathieson
Post by Lindsay Mathieson
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.
It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.
Seeing this in dmesg
CIFS VFS: No writable handles for inode
Post by Lindsay Mathieson
--
Lindsay
--
Lindsay
_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Bart Lageweg | Bizway
2014-02-24 08:43:13 UTC
Permalink
Hi István,

Our limit is 6 hour, because good raid etc the impact is minimal.

Sample configuration

25 VM's on 2x Dell R515 - 4170HE H700+BBU SATA - DRBD (4x Gbit)

DRBD A - 864 GB used
Local disk - 339 GB used (backup) - QNAP NFS (offside location! 2 versions VM) - 678 GB - 5 hours

DRBD B - 758 GB used
Local disk - 334 GB used (backup) - QNAP NFS (offside location! 2 versions VM) - 668 GB - 6 hours

We also backup in the same time the OS with there own backup solution.

Because we are using DRBD, we can backup to the local disk without interrupt the VM or blocking I/O on DRDB storage.
In the past we got also a lot of locking errors QNAP/NFS/SAMBA etc. so first backup to local and then copy to the QNAP is working perfect for 2 years+

Thanks



-----Oorspronkelijk bericht-----
Van: Pongrácz István [mailto:***@gmail.com]
Verzonden: maandag 24 februari 2014 9:31
Aan: Bart Lageweg | Bizway; 'Lindsay Mathieson'; ProxMox Users
Onderwerp: Re: [PVE-User] Backup to samba share stalling/locking up

Hi Bart,

May I ask, how much time does it take to complete the backup and how much disk space does it use?

Thank you!
Best regards,
István



----------------eredeti üzenet-----------------
Feladó: "Bart Lageweg | Bizway" ***@bizway.nl
Címzett: "'Lindsay Mathieson'"
, "ProxMox Users"
Dátum: Mon, 24 Feb 2014 08:00:30 +0000
-------------------------------------------------
Post by Bart Lageweg | Bizway
Hi Lindasy,
We are using the rack QNAP on 2 locations with NFS, backup first to
local and copy to QNAP (40+ nodes).
/etc/vzdump.conf
Script: /usr/local/bin/backup-hook.pl
#!/usr/bin/perl -w
# hook script for vzdump (--script option use strict;
# config
my $phase = shift;
my $mode = shift; # stop/suspend/snapshot my $vmid = shift; my $vmtype
= $ENV{VMTYPE}; # openvz/qemu my $dumpdir = $ENV{DUMPDIR}; my
$hostname = $ENV{HOSTNAME}; my $tarfile = $ENV{TARFILE}; my $logfile =
$ENV{LOGFILE}; my $backuplocatie = "/mnt/pve/sharenameNFS/dump";
my %dispatch = (
"job-start" => \&nop,
"job-end" => \&nop,
"job-abort" => \&nop,
"backup-start" => \&nop,
"backup-end" => \&backup_end,
"backup-abort" => \&nop,
"log-end" => \&log_end,
"pre-stop" => \&nop,
"pre-restart" => \&nop,
);
sub upload {
system("mv -f $backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo
$backuplocatie/vzdump-qemu-oud-$hostname-$vmid.vma.lzo");
system("cp $tarfile
$backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo") == 0 || die
"upload cp $tarfile
$backuplocatie/dump/vzdump-qemu-$hostname-$vmid.vma.tar.lzo
backup-host failed";
print "HOOK: upload " . $tarfile . " to
$backuplocatie/vzdump-qemu-$hostname-$vmid.vma.lzo done\n"; }
sub nop {
# nothing
}
sub backup_end {
upload($tarfile);
}
sub log_end {
# upload($logfile);
}
-----Oorspronkelijk bericht-----
Verzonden: maandag 24 februari 2014 7:42
Aan: ProxMox Users
Onderwerp: Re: [PVE-User] Backup to samba share stalling/locking up
On 24 February 2014 16:38, Lindsay Mathieson
Post by Lindsay Mathieson
I'm testing backing up a vm to a USB Drive attached to a Qnap NAS,
shared out via samba.
It consistently stalls at 17% and can't be stopped via the web gui.
The only thing that works is a reboot of the server and the vm has to
be unlocked.
Seeing this in dmesg
CIFS VFS: No writable handles for inode
Post by Lindsay Mathieson
--
Lindsay
--
Lindsay
_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
http://pve.proxmox.com/cgi-bin/
Loading...