Duplicity backups on Ubuntu / Debian

home | blog | Terrible people and places | Covid-19 links | Teh Internet | guest blog |rants | placeholder | political | projects | Gwen and Liam | Citadel patched | Tools | Scouts

Update - no longer a thing - 0.7.3 is out now!
Update: If you are running 0.7.2, you should fix it with this:
If you are running 0.7.3, the patch should be in place for you - https://bugs.launchpad.net/duplicity/+bug/1434702

First off, you need the dependencies as the version currently in the apt repository is out of date (might need to check that before doing all this in future folks)....

apt-get install librsync1
apt-get install librsync-dev
apt-get install python-dev
apt-get install python-gnupginterface
apt-get install gnupg2
apt-get install ncftp
apt-get install build-essentials
apt-get install build-essential

The build-essentials vs build-essential is a Ubuntu vs Debian thing, one will not exist on your install depending on which flavor you choose.

Now unpack the source, and install:

tar -zxvf duplicity-0.6.06.tar.gz
cd duplicity-0.6.06
python ./setup.py install

Create some scripts to back yourself up (be creative and use man duplicity)...

Check what you just backed up with:

duplicity list-current-files --no-encryption ssh://user:password@host//backup/path/etc


duplicity list-current-files ssh://user:password@host//backup/path/etc

duplicity list-current-files file:///backup/path/etc
(for local filesystem)

If you use gpg and an un-trusted store.

Get a file back:

duplicity --no-encryption --file-to-restore network/interfaces ssh://user:password@host//backup/path/etc ./etc/localdir-or-filename (must not exist).

Get whole backup run back:
duplicity --no-encryption restore scp://user:password@host//backup/path/etc ./etc/

Get the status of a volume backup
duplicity --no-encryption collection-status scp://user:password@host//backup/path/etc

(shows date of last backup, counts of sets, and if there are incomplete backup sets).

Here are some hints, I will try to put more later:

If you are getting password errors, make sure of the ownership of the files on the storage end of things.
I think if a backup is run with a different user, it will change owners on all the older backup files.
And if you have moved hosts and changed host names, but are keeping the history, try the --allow-source-mismatch flag.
- Yet another thing... Strange errors? Try to ssh as the user you are using for the backup run to the host to accept the fingerprint!

Duplicity hanging on all operations?
If you are running 0.6.18, your build against pycrypto and paramiko in Debian / Ubuntu might be too old.
Upgrade (if possible), or rip em out and build from source.

Duplicity server timing out errors like this:?
BackendException: sftp put of /tmp/duplicity-fD1fg0-tempdir/mktemp-bTxB9t-266 (as duplicity-full.blahblahblah.difftar.gz) failed: Server connection dropped

Upgrade your paramiko package!
If that is not the issue, you could up the socket timeout (option --timeout=100) default 30 seconds.
Make sure you have the latest paramikio with: pip search paramiko
If you can't upgrade, make sure you don't have conflicts with installed versions (i.e. python-paramiko and python-crypto) from your systems package management.
if you can't fix your paramiko / pycrypto issues, try: --ssh-backend pexpect

Does the fix above for the connection dropped error not fix your issue?
Make sure your connection is not cutting over due to some sort of HA reason (UCarp, or similar).
Long running full backups can be interrupted by fail-overs and won't notice the missing ssh connection.
It seems like the mainline Duplicity v 0.6.19 has the ability to pick up where it left off though! (Cool stuff!)

Other suggestions for failing connections during long, long, long sessions...
Split the backup in to parts (smaller chunks). So you don't have an all or nothing proposition.
For fulls, backup to local storage and then rsync that (as rsync does decent restarts). Suppose Duplicity may now too, but not sure yet (too soon to tell in my usage estimations.)

Having cleanup issues for old backups with the scp / ssh back end? At least for version 0.6.21 -
Give this a shot - Hint: just add --num-retries 1
  - old line -
duplicity --allow-source-mismatch --ssh-askpass remove-older-than 30D --extra-clean --no-encryption --force scp://user@host//path/host//filesystem
This would return to the shell with 50 for an error for the file not found, or 139 if the program segfaulted (happened sometimes, but not always).

  - new line -
duplicity --allow-source-mismatch --ssh-askpass remove-older-than 30D --extra-clean --num-retries 1 --no-encryption --force scp://user@host//path/host//filesystem

If I added --num-retries 1 after the --extra-clean, it works fine.

restores: (old versions of duplicity 0.6 series - restore 35 day ago backup, don't use encryption key, ask for password on command line - seems key):
- older versions are fussy about optargs on invocation of the duplicity command.  Think some are fixed in later versions...
duplicity restore -t 35D --no-encryption --ssh-askpass scp://backupuser@i.p.addr.ess//backupserverpath/backupdir ./localresotrefolder