Just a blog update. Posting has been slowed by the hectic summer intro, but look forward to weekly posts again soon! http://t.co/DmIMl97hEs— Geoffrey Hannigan (@iprophage) June 9, 2015
These days it is becoming more and more common to have to work on multiple computer systems. Whether this is between a work computer and personal computer, or between your local computer and a high performance server or super computer, it is important to move easily between systems. Over the years I have used a few different systems for file transfer, and I wanted to share some of the insight I have gained through these experiences.
CyberduckOnce I started working more heavily in the realm of computational biology, I had to move a lot of files, especially from my local computer to our lab's remote server which was way more powerful. One of the first programs I used to accomplish this task was Cyberduck. This program offer you a nice Graphic User Interface (GUI) for moving files around a remote computer, as well as transferring files, using a point-and-click interface. This is a pretty nice program, but I found it offered less functionality compared to the other programs I am going to talk about next, so I made the switch.
FileZillaThe next GUI I used was FileZilla. This program offers a similar feel to Cyberduck, with more functionality. It gives great control to the user when it comes to moving files between computers, and also has a great system for editing remote text files that is a lot nicer than Cyberduck. I used this for a while after Cyberduck, and found it to be pretty useful. However I finally got to the point where the command line was just easier to use than a GUI, so I made another switch to using some pretty powerful and standard command line tools.
Bash Tools: mv, scp, rsyncAfter you get used to using the command line, it becomes a lot easier than using a GUI (point-and-click) in a lot of ways. One of the ways this became easier was moving files around computers, and transferring files between computer systems. The tools I am talking about are standard Bash tools, and will be available on any Unix-like operating systems like Linux or Mac OS.
Instead of pointing and clicking to move files around like we are initially taught, it can be easier to use the mv command in the command like. Simply type 'mv <File to move> <File destination>'. This is actually a lot faster than using the mouse because you don't have to worry about clicking through windows and dragging files between them. Simply specify the paths and you are good to go. If you want to generate a copy of the file so that the original also stays put, just use the cp command in the same way.
For transferring files between computer systems, I like to use scp (secure copy). Like the cp command, this allows you to securely copy your files between systems. All you need is the address of the destination computer, as well as the file paths like so: 'scp /path/to/local/file.txt firstname.lastname@example.org:/path/to/file/destination/file.txt'. To recursively copy a directory, just add a -r flag after the scp (see the manual for other uses).
Finally, for particularly large file transfers, I like to use the rsync tool. This is similar to scp, but easily picks up transfers where it left off when the connection is interrupted (if you are using scp, you are going to have to start the transfer over). I like to use the following syntax: 'rsync -ravz --progress /path/to/local/file.txt email@example.com:/path/to/file/destination/file.txt'. Go ahead and try this out for transferring those large directories with a lot of files.
So in the end, I would suggest FileZilla to get you started. It offers a familiar GUI that especially appeals to beginners who are not yet used to the command line interface. Once you get used to the command line, I definitely suggest utilizing the standard bash tools. They are easy to use and very robust. As always, if you have any questions about using these programs, let me know in the comments below. Furthermore, if you have other tools that you think are awesome, please share! I would love to hear about some cool new programs, and I'm sure other readers will too!