How to Synchronize Files Across Locations

rsync is a utility that can be used to copy files and for keeping files the same on different systems as a rudimentary version control system. The benefit to using rsync over scp is that if an scp is stopped for any reason (poor wireless connection, large files, etc) the restart will begin as if no files were copied over.

You should not use rsync to synchronize files between active and archive storage.

The rsync utility will only copy the files that were not successfully moved over in the previous command. Once you have SSH access between two machines, you can synchronize dir1 folder ( home directory in this example) from local to a remote computer by using syntax:

rsync -a ~/dir1 username@remote_host:destination_directory

…where remote host is the Roar host name as in scp command. If dir1 were on the remote system instead of the local system, the syntax would be:

rsync -a username@remote_host:/home/username/dir1 place_on_local_machine

If you are transferring files that have not been compressed yet, like text files, you can reduce the network transfer by adding compression with the -z option:

rsync -az source_dir username@remote_host:target_dir

The -P flag is very helpful. It combines the flags –progress and –partial. The former gives you a progress bar for the transfers and the latter allows you to resume interrupted transfers:

rsync -azP source_dir username@remote_host:target_dir

In order to keep two directories synchronized it is necessary to delete files from the destination directory if they are removed from the source. rsync does not delete anything from the destination directory by default. To change this behavior use the –delete option:

rsync -a --delete source_dir username@remote_host:taget_dir

If you wish to exclude certain files or directories located inside a directory you are syncing, you can do so by specifying them in a comma-separated list following –exclude= option:

rsync -a --exclude=pattern_to_exclude source_dir username@remote_host:target_dir

One common pitfall that can affect users transferring files between systems with different usernames and groups can be the permissions assigned to the files being rsync-ed. The –chmod option can be used both to set the permissions for the user, group and other independently, as well as to set any directory permissions for inheritance of files created within the directory after the transfer is complete.  Multiple commands can be strung together using commas. For example, the following will provide full permissions for the user, read and execute permissions for others in the group and will cause all of the future files created within any directories being transferred to inherit the group that the directory has.

rsync -a--chmod u=rwx,g=rx,Dg+s source_dir username@remote_host:target_dir