Saturday, March 27, 2010

Robocopy - My New Best Friend

I’m not sure how I managed to miss this free and useful utility, but Robocopy has recently become one of my favorite Windows command line tools.  With a single command, you can keep two directories in sync with each other in a very fast, efficient manner.

In the past, I’ve managed to do some fairly cleaver things with Xcopy and other batch files, but I never felt that it was working very efficiently, and it was difficult to write and maintain.  Now, with the Robocopy tool I can quickly copy new files from my working directory, off to a second location for backup or disaster recovery purposes.

Although it has many switches and options, most of my applications utilize either no special switches, or the /mir option.

Without switches, Robocopy will send all new files over to the destination.  This is very similar in concept to xcopy /d, which copies over files that have a newer timestamp.  Although it’s useful in certain applications, such as wanting to keep a copy of all files in a project, even ones that were deleted mid-stream, this isn’t always what you want.  The problem is that if a file is deleted from the source, usually you want your backup copy to reflect that deletion.

And for that is the /mir switch.  With this switch, it copies over all newer files just like before, but now it will analyze the destination, and if a file exists in the destination but not in the source, it will delete it.  This is an excellent way of keeping a copy of your data files on an external hard drive or on a second PC.  Like I said earlier, the mechanism is quite efficient, so even if you’re replicating files to a server off site, only the changes (at a file-level, not block level unfortunately) are copied over.

A case in point, I replicate about 100GB of user data (approximately 80 users) data every day across a 3.0Mbps WAN link, and it takes right at an hour most days.  Your mileage will vary…


I don’t have installations of each to verify, but I believe Robocopy is installed by default on Windows Vista and 7, as well as Server 2008 and 2008R2.  For XP and Server 2003/2003R2, you will need to download it as part of the Windows Resource Kit from Microsoft.  Installation is a no-brainer as shown in the screenshots below.


Once it’s installed, you can execute the command such as the following:

robocopy c:\source e:\destination

Where C:\source is the directory you want to back up, and E:\destination is your backup location.  This is a relatively safe operation, as no files would ever be deleted on either end. 

Once you are comfortable that things are being replicated correctly, if you want to replicate the deletions in C:\source to E:\destination, add the /mir switch.

robocopy c:\source e:\destination /mir

And that’s it, you’ll have an exact copy of C:\source in C:\destination.


Finally, a few other useful switches. 

The combination of /r:3 /w:3 tells Robocopy to Retry files that are in use 3 times, and to Wait 3 seconds between tries.  By default it waits 30 seconds, and retries a million times.  That seems a bit excessive.  Most of the time for me, if a file is in use for more than 9 seconds, then it’s probably safe to move on and try backing it up again tomorrow.

A friend recently found out (almost the hard way) that the /xjd switch makes it safer when backing up user data on Windows Vista and 7 machines.  According to the documentation, this switch excludes junction points for directories.  In practice, it can prevent circular references to some of Window’s special directories such as C:\Users\username\Application Data\.  Note, this option doesn’t exist on Server 2003 or XP.

And finally, if you are scheduling the job, the /log:"c:\logs\backup.log" parameter is handy so that you have a record of what went on.  Note that at least one Server 2008, you cannot write the log file directly to the root of the C: drive, so I now always create a C:\logs\ directory to dump my robocopy logs into.

For me, a typical Robocopy command winds up looking like this:

robocopy c:\source e:\destination /r:3 /w:3 /xjd /mir /log:"c:\logs\backup.log"


Keep in mind that this process is intended to copy a working directory to a backup drive/site.  If you have two working directories (such is the case if you have copies of your photos on both your laptop and desktop) that you want merged, then there are probably better tools out there. 

For that situation, I use FreeFileSync, which is an open-source project on SourceForge.  The reason for using a GUI is simple – you don’t always want to keep the file with the newest timestamp.  By having a GUI with a list of the proposed changes on screen prior to any action taking place, you get the opportunity to override the defaults.

Good luck and don’t be DUMB!  Remember, Disaster Usually Motivates Backups!