backup using rsync – part 3

Here we are going to use rsync as a daemon.

Configuration

Server

1. similar to smb setup, what to share for rsync

Edit or create /etc/rsyncd.conf

lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
[ServerRsyncShareName]
#this will be root of our share
 path = /path/to/rsync/share
 comment =
#effective access to share as specific user
 uid = username
 gid = groupname
#read only?
 read only = yes
#allow to list this share when client request
 list = yes
#which users allowed to connect - this are rsync users - not system ones
 auth users = user1,user2
#user and passwords definition location
 secrets file = /etc/rsyncd.secrets
 #restrict access to specific IP addresses
 # hosts allow = 192.168.1.0/255.255.255.

2. create users – they are totally independent of system users

Edit or create /etc/rsyncd.secrets
user1:password1
user2:password2
Then run sudo chmod 600 /etc/rsyncd.secrets

3. create service

Edit or create /lib/systemd/system/rsync.service
[Unit]
Description=fast remote file copy program daemon
ConditionPathExists=/etc/rsyncd.conf
[Service]
ExecStart=/usr/bin/rsync --daemon --no-detach
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
Make the system aware of the new service:
sudo systemctl daemon-reload
To control the service run any of these commands:
sudo systemctl start|stop|restart|status|enable|disable rsync.service
Enable if you want rsync daemon to start after a reboot.

4. If server has firewall allow port 873/tcp

e.g.
sudo ufw allow proto tcp to any port 873
Now other rsync program can connect to your rsync server – and read/write files from the defined shares

Client

it is not possible to use a password explicitly in a rsync command. if used in non-interactive mode then either export the password as a variable (export RSYNC_PASSWORD=password1) or use the password file parameter –password-file
Command to read from the server:
rsync -avzh  --verbose  --timeout=60  --contimeout=60  --progress  --stats user1@serverIP::OurRsyncShareName /destination/folder
To write to the server (if readonly=no)
rsync -avzh  --verbose  --timeout=60  --contimeout=60  --progress  --stats  /source/folder user1@serverIP::OurRsyncShareName
For instance, your crontab could look like that:
*/5 7-22 * * * (export RSYNC_PASSWORD=SuperLongPassword12345 ; /usr/bin/flock --verbose -n /tmp/db-rsync -c "/usr/bin/rsync -avP --contimeout=60 --timeout=240 --append-verify --stop-at=22:55 --log-file=/home/username/log/my_rsync.log --bwlimit=1000 -b --backup-dir=/media/username/5TB/rsync-backupdir/ rsyncclient@192.168.1.2::RPi01 /media/username/5TB/backup-RPi01" >>/home/username/log/my_rsync.log 2>&1)
*/5 23,0-6 * * * (export RSYNC_PASSWORD=SuperLongPassword12345 ; /usr/bin/flock --verbose -n /tmp/db-rsync -c "/usr/bin/rsync -avP --contimeout=60 --timeout=240 --append-verify --stop-at=22:55 --log-file=/home/username/log/my_rsync.log  -b --backup-dir=/media/username/5TB/rsync-backupdir/ rsyncclient@192.168.1.2::RPi01 /media/username/5TB/backup-RPi01" >>/home/username/log/my_rsync.log 2>&1)
Note1. OurRsyncShareName is server root folder for rsync so it can be used as  user1@serverIP::OurRsyncShareName/folder1/folder2 if we need to access only a subfolder of the share.
Note2. serverIP:: these two double colons define rsync to daemon connection. single double colon represents ssh mode
Note3. unlike in ssh mode, in daemon mode, the traffic in sent over clear channel – do not use on public networks unless secured otherwise (stunnel or vpn are just two obvious examples of how to do it).

Results

  • Around 9% of the files (in GB) were in the –backup-dir folder and were re-downloaded by rsync. Given the fact that the transfer took over a week with multiple disconnections and reboots in between, I suspect that is not bad.
  • We run a checksum on both sides and 100% of the files were ok, unlike what we attempted to do in part 1. So this is a viable solution.

I will spend some time, later this week, writing a part 4 to share the results, pros and cons of each solution and tips that we learned along the way.

Leave a Reply

Your e-mail address will not be published. Required fields are marked *