SEOClerks

rsync slow with large number of files, how can I improve it?



Enter a reason for deleting this comment

rsync slow with large number of files, how can I improve it?

I have a huge amount of files I am transferring using rsync. I use it as a backup to transfer from my primary system to an NFS file system that is mounted. This is a great solution, however, it is very slow. It takes hours to complete it seems (I never timed it).

Here is my rsync call:

rsync -zvr --delete /var/www/site/public_html/pics/ /mount/backup/pics/

These are user profile pictures and since I have well over 100,000 users, I have a lot of profile pictures to transfer. They all reside in a single directly. These files are around 1-15kb.

What can I do to speed this up?

Comments

Please login or sign up to leave a comment

Join
Void

rsync is slow and takes a lot of CPU/Memory. If you have a ton of files in a directory, it is probably hanging at the stat() stage. rsync will also make multiple passes per file. Instead of using rsync, you could try tar. 

tar -cf foo.tar /var/www/site/public_html/pics/

It should operate with much less resource usage although it may still take a bit on a large amount of files. 
You can also explode it to where you want to save it:
tar -cf foo.tar /var/www/clerks/public_html/pics/ | tar -x foo.tar /mount/backup/pics




Are you sure you want to delete this post?