I have an rsync job that has been added to a crontab and when it's running, I can only check that there's a rsync PID and confirm with htop that it's eating up n amount of CPU and RAM.
What I'd like to do is to monitor what files are actually being rsync'ed in realtime...when I want to. FYI I haven't passed any verbose option to the command nor have I added some logging. I really just want to check what's being rsync'ed on demand.
Any idea how I could achieve that?
6 Answers
You can do (in POSIX shells, including bash):
strace -e open,openat $(ps -o lwp= -LC rsync | sed 's/^/-p/')
Or in zsh:
strace -e open,openat -p${^$(ps -o lwp= -LC rsync)}
Or in fish
strace -e open,openat -p(ps -o lwp= -LC rsync)
Or in rc/es:
strace -e open,openat -p`{ps -o lwp= -LC rsync}
To see what it's doing, or
lsof -ad3-999 -c rsync
to see what files it currently has opened.
The most simple solution would be redirecting the output of rsync to a logfile.
rsync -avz /something /somwhere >> ~/rsynclog
Another way you could is if you know the rough directory you'll be syncing (ie, we'll use the directory 'movies' for example) you can use a combination of lsof and grep:
lsof | grep rsync | grep movies
lsof will list your open files, pipes the output to grep to find any opened by rsync, pipes that output to grep to find the directory/file which is open.
Here are two ways
With screen: Attach screen session to your cron job:
screen rsync --progress src dst
this will allow you to re attach to the rsync jobb anytime you want to check what files its currently processing (just be sure to be the same user as the one that launched the rsync job) with
screen -x
With logging add logging to your rsync job:
rsync --log-file=/tmp/rsync-status.txt src dst
then follow the log in real time with:
tail -f /tmp/rsync-status.txt
As Király István suggested, I am running
rsync -ravz /Users/jkirby/Music/iTunes/* .
which gives output like
Jeffs-MBP-2:2016-08-15 jkirby$ rsync -ravz /Users/jkirby/Music/iTunes/* .
building file list ... done
Temp File 1.tmp
Temp File.tmp
From that output I can see what directory is being copied.
In the case where rsync is copying a lot of large files slowly, I monitor that directory using watch like so. This way I can see the temp file that rsync creates and I can see the size growing on the file currently being copied.
watch -n1 "~/Music/iTunes"
Monitor running process under GNU/Linux
Monitor rsync transfert on both ends
By running Linux kernel, you could query /proc pseudo filesystem. There is no need to resort to strace or lsof, (could be useful on small configuration where this tools could even not be installed).
ps -C rsync fw
PID TTY STAT TIME COMMAND
8645 ? Ss 0:00 /usr/bin/rsync --daemon --no-detach
13763 ? S 0:01 \_ /usr/bin/rsync --daemon --no-detach
13764 ? S 1:30 \_ /usr/bin/rsync --daemon --no-detach
ps -C rsync fwo pid,lstart,stat,cmd
PID STARTED STAT CMD
8645 Sat Jul 10 14:34:12 2021 Ss /usr/bin/rsync --daemon --no-detach
13763 Sat Jul 10 15:51:23 2021 S \_ /usr/bin/rsync --daemon --no-detach
13764 Sat Jul 10 15:51:23 2021 S \_ /usr/bin/rsync --daemon --no-detach
The 1st process is the daemon, looking for start time show two process started by remote rsync.
So, looking for the last one:
mapfile -t allrsyncpid < <(ps -C rsync fho pid)
rsyncpid=$((${allrsyncpid[@]: -1}))
Then with this $rsyncpid variable:
ls -l /proc/$rsyncpid/fd
total 0
lr-x------ 1 root root 64 jui 10 15:51 0 -> /dev/null
lrwx------ 1 root root 64 jui 10 15:51 1 -> socket:[106279330]
lrwx------ 1 root root 64 jui 10 15:51 2 -> socket:[106279330]
lrwx------ 1 root root 64 jui 10 15:51 3 -> socket:[106279331]
lrwx------ 1 root root 64 jui 10 15:51 4 -> /path/to/some/file.IxOWxh
lrwx------ 1 root root 64 jui 10 15:51 5 -> socket:[106323027]
lrwx------ 1 root root 64 jui 10 15:51 6 -> socket:[106320690]
Then
ls -l $(readlink /proc/$rsyncpid/fd/4)
-rw------- 1 root root 41024 Jun 10 15:59 /path/to/other/file.ZIjEDk
If you repeat this, you could see wich file are growing and how.
while [ -d /proc/$rsyncpid ] ;do
ls -l $(readlink /proc/$rsyncpid/fd/4)
read -t .5 foo && break
done
-rw------- 1 root root 0 Jul 10 16:16 /path/to/some/file.nMM3Lz
-rw------- 1 root root 0 Jul 10 16:16 /path/to/other/file.H2b61i
-rw------- 1 root root 0 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7
-rw------- 1 root root 2097152 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7
-rw------- 1 root root 42359296 Jul 10 16:16 /path/to/some/bigfile.B0Xgg7
Until you press return or the end of rsync process.
Of course, as you are watching for /proc entries, created and deleted very quickly by another process, there are some chance ls is executed too late after readlink. So this kind of message is not harmful:
ls: cannot access '/path/to/some/file.fKSZss': No such file or directory