Skip to end of metadata
Go to start of metadata

UNL has come to an arrangement with Box.com to provide unlimited cloud storage to every student, staff, and faculty member.  This can be useful when used with jobs to automatically upload results when the job has completed.  Combined with Box Sync, the uploaded files can be sync'd to your laptop or desktop upon job completion. The upload and download speed of Box is about 20 to 30 MB/s in good network traffic conditions. Users can use a tool called lftp to transfer files between HCC clusters and their Box accounts.

Step-by-step guide for Lftp

  1. You need to create your UNL Box.com account here.
  2. Since we are going to be using webdav protocol to access your Box.com storage, you need to create an External Password.  In the Box.com interface, you can create it at Account Settings > Create External Password.



  3.  After logging into the cluster of your choice, load the lftp module by entering the command below at the prompt:

    Load the lftp module
    module load lftp  
  4. Connect to Box using your full email as the username and external password you created:

    Connect to Box
    lftp -u <username>,<password> ftps://ftp.box.com
  5. Test the connection by running the ls command.  You should see a listing of your Box files.  Assuming it works, add a bookmark named "box" to use when connecting later:

    Add lftp bookmark
    lftp demo2@unl.edu@ftp.box.com:/> bookmark add box
  6. Exit lftp by typing quit.  To reconnect later, use bookmark name:

    Connect using bookmark name
    lftp box
  7. To upload or download files, use the get and put commands.  For example:

    Transferring files
    [demo@login.crane ~]$ lftp box
    lftp demo2@unl.edu@ftp.box.com:/> put myfile.txt
    lftp demo2@unl.edu@ftp.box.com:/> get my_other_file.txt
  8. To download directories, use the mirror command.  To upload directories, use the mirror command with the -R option.  For example, to download a directory named my_box_dir to your current directory:

    Download a directory from Box
    [demo@login.crane ~]$ lftp box
    lftp demo2@unl.edu@ftp.box.com:/> mirror my_box_dir
    

    To upload a directory named my_hcc_dir to Box, use mirror with the -R option:

    Upload a directory to Box
    [demo@login.crane ~]$ lftp box
    lftp demo2@unl.edu@ftp.box.com:/> mirror -R my_hcc_dir
  9. Lftp also supports using scripts to transfer files.  This can be used to automatic downloading or uploading files during jobs.  For example, create a file called "transfer.sh" with the following lines:

    transfer.sh
    open box
    get some_input_file.tar.gz
    put my_output_file.tar.gz

    To run this script, do:

    Run transfer.sh
    module load lftp
    lftp -f transfer.sh