The long lost art of automatic file synchronization with unison via ssh

June 16, 2011 —

I guess one of the reasons why many people use dropbox is, that it makes file synchronization trivially easy for the end-user. However, it only provides limited space and you have to trust all your data to a third party (or use some kind of encrypted files, which makes it non-trivial). But thanks to unison and ssh, you can easily implement automated and secure file-synchronization across several computers.

Here is a little how-to for a simple desktop-laptop-setup. First make sure that you have installed the same version of unison on both computers. Then create the directory ~/.unison and a file named default.prf in it; this is the profile unison will use when invoked without a specified profile. This is how it could look like:

# .unison/default.prf Unison preferences file

# First define the roots for synchronization; all further paths will be
# relative to these.

# my home directory
root = /home/deaddy
# the home directory on my laptop dh0
root = ssh://dh0/home/deaddy

logfile = /home/deaddy.unison/unison.log

# now include another preference file, containing a list of directories you
# want to synchronize
include common

And here is what my common-file looks like:

# .unison/common containing paths to synchronize in default profile
# every file or folder you want to have synchronized is just appended by 
# path = folder or path = file. Remember that these are relative to your root
# specified in the .prf-File

path =.code
path = work
path = documents
path = .vimrc
path = .surf/bookmarks.txt

# I keep some scripts in .local/bin
path = .local 
# However, webkit does save some data in .local which I don't want to have
# synchronized
ignore = Path .local/share/webkit

# since I have the same unison setup on my laptop (just a different hostname in
# the remote root) I want to be able to change this list on both computers and
# have it changes propagated to the other one

path = .unison/common

Actually my list is a bit longer, but this should be enough to give you the idea. Now you have everything set up to run unison for the first time. It will notice you, that there were no archive files found, which is normal for the first run, so just confirm with return. Now unison will ask you which file you want to have synchronized in which direction; however, this is kind of cumbersome if most files are identically, so you can reduce the noise by aborting and invoking unison -auto in order to see only conflicting files.

When you have synchronized the folders, you might want to automate this task. So first we generate a new ssh-key for this task:

ssh-keygen -t rsa -f ~/.ssh/unison_rsa

When asked for a passphrase, just hit return, since you won’t be able to enter a passphrase in the cronjob. Now copy the contents of ~/.ssh/unison_rsa.pub to your ~/.ssh/authorized_keys on the remote host. You should be able to login on the remote machine via ssh -i '~/.ssh/unison_rsa' . If this fails, you should debug ssh (Is public key authentication enabled? Is the public key correctly copied?), else we can continue. Since this key is not password-protected, edit the ~/.ssh/authorized_keys again and prepend command="unison -server" to the newly added key. This will restrict logins to this command. The -server argument is necessary so unison on the remote host can act as an server for your client. As my friend Stefan pointed out, one can improve the restrictions further, i.e. one could prepend

command="/usr/bin/unison -server",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty.

Now we’re almost done. Enter crontab -e into your shell and add a cronjob like this (here it synchronizes every 10 minutes):

*/10    *    *    *    *    unison -batch -dumbtty -sshargs '-i /home/deaddy/.ssh/unison_rsa' 1>/dev/null 2>&amp;1

Replace /home/deaddy/.ssh/unison_rsa with the path to the newly generated private key, and you’re good to go. Somehow my crondaemon does not like output, so I redirect standard out to /dev/null, and standard error to standard out. Thanks to dasjoe for this hint. Also I ran into trouble without -dumbtty which tells unison, that the terminal (i.e. the crondaemon) does not support special characters. Finally -batch tells unison to synchronize all trivial changes, so if you’re editing a file only on one computer, the changes will be propagated to the other one without your interaction. However conflicting changes still need to be resolved manually. For this just use unison -auto .

I now have unison running for years, however I never came around to automate it, because the cronjob won’t load environment variables like your ssh-agent. However, by using the -sshargs argument I now can use a specific key. But now I have finally a fully automated synchronization of all my important files between my laptop and desktop, and thanks to dyndns and ssh I can also synchronize them when I’m not at home, without worrying that the connection is unsafe. Also, unison uses the rsync-algorithm, so after the first run it’s quite fast (searching about 50G of files for changes takes only a few seconds) and uses little bandwidth.