Sunday was a Unix day, which might seem impressive except that I really don’t know what I’m doing. Had I known what I’m doing, duplicating one remote directory (this web site, all several hundred megs of it) into another remote directory (a new server) would take about five minutes. Instead it took me about 20 hours. But I learned a lot.
wget
The server I was pulling from did not have rsync installed and would not let me connect via SSH (secure shell). I could telnet, but that didn’t help me. Since rsync was out of the question, I found wget, which is often used to mirror sites via ftp. One thing I couldn’t get to work was to copy directly from the old server to the new one, so I decided to download the whole site to my hard drive and then sync it up to the new server.
I don’t have the developer tools installed yet, but thankfully Apple has a pre-compiled package available from the OS X web site: Wget 1.8.1
Wget is very easy to use. The only stumbling block I had was the need to point to my www directory explicitly, wget wouldn’t follow the symlink (before last night, I didn’t know what a symlink was. They’re basically aliases). I found the explicit path by getting info on a file from the server using Fetch. Once I had the path correct, wget worked perfectly with the following command:
wget -m --passive-ftp ftp://[user]:[password]@[host][explicit path to root directory]/
The commands at the beginning tell wget to mirror (m) and to use passive FTP (–passive-ftp).
rsync
I first learned about rsync while looking for an open source (free) disk mirroring solution for a file server. At the time, rsync didn’t support OS X’s HFS+ filesystem so icons and creator codes weren’t duplicated. Since then macosxlabs.org has developed RsyncX which I hope to try out soon.
I used rsync for two different things. First, I wanted to back up the site I just downloaded by burning it to a CD. OS X doesn’t seem to be able to create a disk image from a folder lke OS 9 could do, and the one shareware application which claimed that ability kept returning errors. I ended up creating a blank CD-master disk image with Disk Copy, then using rsync to duplicate the downloaded folder to the disk image. This is the command I used:
rsync -vr [source path, w/o trailing slash] [disk image path]
The -vr command tells rsync to be “verbose” (v) while duplicating and to recursively copy all directories (r).
The other use for rsync was to mirror my local site onto the new remote server.
rsync -vtrp -e ssh [local path]/ [user]@host:[remote directory]/
The additional commands tell rsync to preserve the original file times (t) and permissions (p).
A trailing slash on the source path tells rsync copy the files from that directory into the target directory. If there is no trailing slash, the directory itself will be copied and created if necessary.
The following examples use a fictional file system which contains:
sourceDir/
a.file
b.file
destinationDir/
1.file
2.file
These two simplified examples demostrate the effect of the trailing slash:
rsync /sourceDir /destinationDir/
would copy the directory sourceDir into destinationDir resulting in:
destinationDir/
1.file
2.file
sourceDir/
a.file
b.file
rsync /sourceDir/ /destinationDir/
would sync the contents of sourceDir into destinationDir resulting in:
destinationDir/
1.file
2.file
a.file
b.file
My last stumbling block was specifying the target directory correctly. All the examples I could find had the target starting with a slash, but when I tried that, it bounced up to the root of the server (not my local root folder) because I’m uploading to a directory before switching my domain over. Once I realized that the “mkdir…permission denied (1)” error was a result of trying to create a directory outside of my personal space rsync worked perfectly. After several hours of searching for answers of course.
I didn’t find any one resource which answered all my questions, but the following sites are good places to start. Otherwise, Google is your friend.
