Here we are going to use rsync as a daemon.
Configuration
Server
1. similar to smb setup, what to share for rsync
Edit or create /etc/rsyncd.conf
lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid [ServerRsyncShareName] #this will be root of our share path = /path/to/rsync/share comment = #effective access to share as specific user uid = username gid = groupname #read only? read only = yes #allow to list this share when client request list = yes #which users allowed to connect - this are rsync users - not system ones auth users = user1,user2 #user and passwords definition location secrets file = /etc/rsyncd.secrets #restrict access to specific IP addresses # hosts allow = 192.168.1.0/255.255.255.
2. create users – they are totally independent of system users
Edit or create /etc/rsyncd.secrets
user1:password1 user2:password2
Then run
sudo chmod 600 /etc/rsyncd.secrets3. create service
Edit or create /lib/systemd/system/rsync.service
[Unit] Description=fast remote file copy program daemon ConditionPathExists=/etc/rsyncd.conf [Service] ExecStart=/usr/bin/rsync --daemon --no-detach Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target
Make the system aware of the new service:
sudo systemctl daemon-reload
To control the service run any of these commands:
sudo systemctl start|stop|restart|status|enable|disable rsync.service
Enable if you want rsync daemon to start after a reboot.
4. If server has firewall allow port 873/tcp
e.g.
sudo ufw allow proto tcp to any port 873
Now other rsync program can connect to your rsync server – and read/write files from the defined shares
Client
it is not possible to use a password explicitly in a rsync command. if used in non-interactive mode then either export the password as a variable (export RSYNC_PASSWORD=password1) or use the password file parameter –password-file
Command to read from the server:
rsync -avzh --verbose --timeout=60 --contimeout=60 --progress --stats user1@serverIP::OurRsyncShareName /destination/folder
To write to the server (if readonly=no)
rsync -avzh --verbose --timeout=60 --contimeout=60 --progress --stats /source/folder user1@serverIP::OurRsyncShareName
For instance, your crontab could look like that:
*/5 7-22 * * * (export RSYNC_PASSWORD=SuperLongPassword12345 ; /usr/bin/flock --verbose -n /tmp/db-rsync -c "/usr/bin/rsync -avP --contimeout=60 --timeout=240 --append-verify --stop-at=22:55 --log-file=/home/username/log/my_rsync.log --bwlimit=1000 -b --backup-dir=/media/username/5TB/rsync-backupdir/ rsyncclient@192.168.1.2::RPi01 /media/username/5TB/backup-RPi01" >>/home/username/log/my_rsync.log 2>&1) */5 23,0-6 * * * (export RSYNC_PASSWORD=SuperLongPassword12345 ; /usr/bin/flock --verbose -n /tmp/db-rsync -c "/usr/bin/rsync -avP --contimeout=60 --timeout=240 --append-verify --stop-at=22:55 --log-file=/home/username/log/my_rsync.log -b --backup-dir=/media/username/5TB/rsync-backupdir/ rsyncclient@192.168.1.2::RPi01 /media/username/5TB/backup-RPi01" >>/home/username/log/my_rsync.log 2>&1)
Note1. OurRsyncShareName is server root folder for rsync so it can be used as user1@serverIP::OurRsyncShareName/folder1/folder2 if we need to access only a subfolder of the share.
Note2. serverIP:: these two double colons define rsync to daemon connection. single double colon represents ssh mode
Note3. unlike in ssh mode, in daemon mode, the traffic in sent over clear channel – do not use on public networks unless secured otherwise (stunnel or vpn are just two obvious examples of how to do it).
Results
- Around 9% of the files (in GB) were in the –backup-dir folder and were re-downloaded by rsync. Given the fact that the transfer took over a week with multiple disconnections and reboots in between, I suspect that is not bad.
- We run a checksum on both sides and 100% of the files were ok, unlike what we attempted to do in part 1. So this is a viable solution.
I will spend some time, later this week, writing a part 4 to share the results, pros and cons of each solution and tips that we learned along the way.