Linux backups powered by Rsync

Backup like a PRO using one most popular linux backup tool alternatives called rsync.

RSync backups data and does it very clean and well. Rsync only transfers those data that have been modified and changed so that the destination host has an exact replica from the source host. Rysnc is a command line backup tool that handles data transfers in an effective and secure manner like any other known commercial backup softwares around. Rync blends in and integrates flawlessly with linux shell commands combined linux I/O redirections.

Here's an altenative approach based from recent entry on creating data backups from simple one to an enterprise backup data sets using rsync.

Man Rsync:

Rsync uses a reliable algorithm to bring remote and host files into sync very quickly. Rsync is fast because it just sends the differences in the files over the network instead of sending the complete files. Rsync is often used as a very powerful mirroring process or just as a more capable replacement for the rcp command. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm.


INSTALLATION:
~~~~~~~~~~~~~~~~

Rsync installation is installed by default System Tool group installation. If rsync is not available from command line and from rpm database, you can install from yum repo using yum like so

# yum -y install rsync

To verify that rsync has been installed successfully

# rpm -qa rsync

There are two different approach on establishing rsync communication between two hosts.

First is by using a remote-shell program such as ssh or rsh. If you want to use this approach, it is required that openssh server or an active ssh connection is present from both sending and receiving host. Openssh installation and configuration would not be covered from this entry. This entry would assume that ssh daemon service is currently configured and listening properly to assigned host port.

Secondly, is by using rsync listening INETD service directly.

This entry hope to cover both of them two worlds.

Rsync usage from command line terminal
======================================

Transfer and/or update files from local host into a remote host using rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Basic rsync argument is to specify file glob, a source and a destination folder. Destination can be a local host or a remote host. A basic example to transfer from local to remote would be

# rsync -t *.mp3 remoteusername@remotehost:remote_destination_folder

The above would transfer, using rsync, all *.mp3 files from current local directory into remote_destination_folder of remotehost using remoteusername for authentication. A destination host can be locally or remote host and can be specified as hostname or IP address.

Legend:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-t preserve time of selected files
*.mp3 selected files to be transferred
remoteusername bash enabled user account from destination host
remotehost destination host where remoteusername is allowed to have access
remote_destination_folder destination folder from destination host owned and accessible
by remoteusername
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For recursive traversing of directory folders for transferring data from current host to another host using rsync, this would be like so

# rsync -avrzt /var/www myuser@server1:/var/backup/
Legends:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-a archived mode enabled
-v verbosed mode enabled
-t preserve time stamp
-z compression transfer enabled
-r resursive mode enabled
/var/www selected folder or files glob source location
myuser user account from destination host
server1 destination host
/var/backup destination folder
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The above command would issues rsync with verbose mode enabled, compression enabled, arhived mode of rsync data transfer/update. Rsync transfers files and folders from /var/www (including the www folder) into /var/backup folder of server1 host using myuser as login credentials. The /var/backup from destination host is owned or writeable by myuser . All rsync files are created with file ownership and permission owned by myuser having 600 file mode. The transfer would be done preserving symbolic links, devices, attributes, permissions and ownerships of files.

# rsync -avz /var/www/ myuser@server1:/var/backup/www

Appending / from selected file glob like the above command issues the same rsync command with same argument. The only difference is that source folder is not created .

Transfer files from remote host to local host using rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

# rsync -avrzt remoteusername@remotehost:remote_source_folder destination_folder
# rsync -avrz myuser@server1:/var/backup/www /var/www/

To transfer multiple files with different file extension from local host to remote host using rsync with file glob would be

# rsync -avrz file1.mp3 *.doc myuser@server1:/home/folder/location

To transfer multiple files from multiple directory sources using rsync would be

# rsync -avz `find /etc -name *.conf` user@remotehost:/user/folder

Linux command line tools when combined with each other creates another set of power tools. The above command searches all *.conf files from /etc folder and transfer them to /usr/folder of remotehost via rsync and using user the user login name. This can be handy if you like to backup specific pattern of files from your system or user accounts individual address books, files like that.

To transfer multiple files from multiple directory with some file excemptions using rsync and grep from local to remote would be like so:

# rsync -avz `find /etc -name *.conf | grep -v yum.conf` user@remotehost:/user/folder

Alternatively,

# rsync -avz $(find /etc -name *.conf | grep -v yum.conf) user@remotehost:/user/folder

and for multiple file rsync with multiple file from multiple source location with multiple file transfer exceptions would be like so

# rsync -avz `find /etc -name *.conf | grep -v 'yum.conf\|xorg.conf'` user@remotehost:/user/folder

To transfer of all your back up files ending in .tar file extension from any location would be like so

# rsync -avz `find / -name *.tar` user@remotehost:/user/folder

You will noticed that rsync can transfer data at high speed rate using rsync algorithm specially if those files and folders to be transferred are existing already from remote host.

To transfer multiple file(s) with multiple exclusions using rsync would be like so:

# rsync -avz * user@remotehost:/location --exclude=*.php --exclude=*.mp3

To transfer files in batch mode based from file lists using rsync would be

Assuming listing.txt is created with contents like below

# cat listing.txt
~~~~~~~~~~~~~~~~~~~~~
files0123.txt
files0124.txt
files6124.txt
files6126.txt
...
snipped
...
files32126.txt
~~~~~~~~~~~~~~~~~~~~~

and feeding the above file to rsync as batch mode input like so

# rsync -avzt --files-from=listing.txt user@remotehost

:/destination/

To rsync multiple folder source location using rsync would be like so

# rsync -avz --files-from=/home1 /home2 user@remotehost:/destination/

If both location contains abc.txt, the latest abc.txt would be transferred to remote host. If /home1/www exist and /home2/www exist, the files from both source kicatuib would be merged into /destination/www .

Fire up two new terminal windows, and from the first window, establish ssh connection from local to remote rsync destination. Then from the second windows try to issue these rsync command. You will notice that rsync never ask any password any more since there is an existing ssh connection with local host to remote host.

More rsync arguments
====================

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-W transfer the whole files without considering any update changes from existing destination file or folder
--progress show progress bar and/or percentage
-4 prefers IPv4
-6 prefers IPv6
--bwlimit=KBPS execute rsync with bandwidth limit rate in KBPS
-h display a more human readable screen output
--log-file=FILE dumps file activity into a file
--ignore-existing ignores already existing copy from destination host
--max-size tells rsync to avoid transferring files larger than specified size like
--max-size=10m avoid file transfer with 10MB in filesize
--port tells rsync to connect to rsync server with a different port, default is 873
--stats tells more file transfer info and rsync algorithm stats on the fly
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

At this point, be reminded to kick the black ninja box to refresh possible monitor related eye strains.





Using Rsync in Daemon Xinetd Mode
=================================

Edit /etc/xinetd.d/rsync and modify

disable = yes
to
disable = no

To run rsync in deamon mode via xinetd, make sure you have similar lines from your /etc/xinetd.d/rsync file like shown below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
service rsync
{
disable = no
socket_type = stream
wait = no
user = root
server = /usr/bin/rsync
server_args = --daemon --config=/etc/rsync/rsyncd.conf -v
log_on_failure += USERID
}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create /etc/rsync/rsyncd.conf as a default conf file for rsync in daemon xinetd mode.

Sample rsyncd.conf file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uid = rsync-user
gid = rsync-user
use chroot = yes
read only = yes
pid file = /var/run/rsyncd.pid

# access list, edit the IP network block to suit your needs
hosts allow=192.168.0.0/255.255.0.0 192.168.1.0/255.255.255.0
# deny anything else
hosts deny=*

# limit connections
max connections = 5
#greeting file
motd file = /etc/rsync/rsyncd.motd
#log file
log file = /var/log/rsync.log

#rsync shared folder
[myrsync]
#make your UID/GUID above owns the below folder and files
#all files and folder would be seen by rsync client
path=/home/rsync-user/rsync-folder
comment = Linux Rsync Server
exclude = *.mp3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create rsync greeting MOTD file like so

# cat /etc/rsync/rsyncd.motd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Welcome to my Rsync Server!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Restart xinetd service for the changes to take effect like so

# service xinetd restart

Verify that rsync is running as xinetd daemon service mode using one tool like ss

# ss -a | grep rsync
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
LISTEN 0 0 *:rsync *:*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Rsync uses port 873 as its default port for xinetd service. Make sure it is also open from your current firewall settings. A sample firewall rule for opeing rsync from your firewall would be like so

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j ACCEPT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# service iptables restart

If you wish to have a passwordless rsync, you need to refer to passwordless/passphraseless ssh from one of last month's entry.

Rsync in daemon mode via command line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An alternative to run rsync in daemon mode is by specifying it from command line. Like so

# rsync --daemon --address host-IP-address --config=/etc/rsync/rsyncd.conf -v --port=873

Legend:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--daemon enables rsync to be run as daemon mode from terminal
--port specifies port number to use
--config specified rsync conf file
host-IP-address IP address where to bind rsync from
-v enables verbose mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Listing out files and folder from rsync server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To list out the files from rsync server, simply point your rsync client to the host running rsync in daemon mode or server mode. To test your rsync server from a client linux host would be like so

# rsync host-IP-address::

Alternatively,

# rsync rsync://host-IP-address/myrsync
# rsync rsync://host-IP-address/myrsync/folder1

From the above rsync command, you should be seeing files from /home/rsync-user/rsync-folder folder from rsync server you defined from /etc/rsync/rsyncd.conf.

Syncing File and Folders from Rsync Server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To sync and download files from rsync server.

# rsync rsync://host-IP-address/myrsync/folder1 .

The above command downloads files including the whole folder1 folder from rsync server and saves it to current directory

# rsync rsync://host-IP-address/myrsync/folder1/ .

The above command downloads only files from folder1 to current folder location

Rsync Log Monitoring
~~~~~~~~~~~~~~~~~~~~

Monitoring rsync server messages for any errors or system messages would be like so

# tail -f /var/log/rsync.log

Final Note:

Rsync can also be used for source code control and management for delivering a centralized and distributed sync source codes from rsync server to group of programmers or source developers among departments, more like a CVS approach.

Using rsync linux command provides many abilities and benefits. To name a few, mirror an entire harddisk or partitions, folders and files, entire domain websites, mail spools for replica servers, user's home folder, a mirror FTP site and much more.

Backup like the enterprise way, use RSync.

Have a nice day ahead!

Related Posts:

Linux Backup using RSnapShot
Linux Backup using Tar
Linux Backup using RDiff
Bandwidth-Effificent and Encrypted Linux Backup