SEOClerks

Clear
Sign In Join

rsync slow with large number of files, how can I improve it?

  • Drake83 Level 1
    • 1

    Write the reason you're deleting this FAQ

    rsync slow with large number of files, how can I improve it?

    I have a huge amount of files I am transferring using rsync. I use it as a backup to transfer from my primary system to an NFS file system that is mounted. This is a great solution, however, it is very slow. It takes hours to complete it seems (I never timed it).

    Here is my rsync call:

    rsync -zvr --delete /var/www/site/public_html/pics/ /mount/backup/pics/

    These are user profile pictures and since I have well over 100,000 users, I have a lot of profile pictures to transfer. They all reside in a single directly. These files are around 1-15kb.

    What can I do to speed this up?

    #Rsync #Slow #Files Webmaster Related
    Share post:

    Comments

    Please login or sign up to leave a comment

    Join
    Void

    Void

    rsync is slow and takes a lot of CPU/Memory. If you have a ton of files in a directory, it is probably hanging at the stat() stage. rsync will also make multiple passes per file. Instead of using rsync, you could try tar. 
    tar -cf foo.tar /var/www/site/public_html/pics/

    It should operate with much less resource usage although it may still take a bit on a large amount of files. 
    You can also explode it to where you want to save it:
    tar -cf foo.tar /var/www/clerks/public_html/pics/ | tar -x foo.tar /mount/backup/pics
    11 years ago

    Are you sure you want to delete this post?