Skip to end of metadata
Go to start of metadata

UNL has come to an arrangement with Box.com to provide unlimited cloud storage to every student, staff, and faculty member.  This can be useful when used with jobs to automatically upload results when the job has completed.  Combined with Box Sync, the uploaded files can be sync'd to your laptop or desktop upon job completion. The upload and download speed of Box is about 20 to 30 MB/s in good network traffic conditions.  There are two programs that can be used to transfer files to/from Box - cadaver or lftp.  Instructions are provided for both options.

Step-by-step guide for Cadaver

  1. You need to create your UNL Box.com account here.
     
  2. Since we are going to be using webdav protocol to access your Box.com storage, you need to create an External Password.  In the Box.com interface, you can create it at Account Settings > Create External Password.

     
  3. Create a .netrc file in order to automatically login to your box account without typing the password.  The file needs to be in your home directory, ie ~/.netrc.  You can easily create this file using the nano text editor by using the command:

    nano ~/.netrc

    The file should contain the following lines:

    machine dav.box.com
    	login <box_username>@unl.edu
    	password <external_password>

    Once you have typed or pasted these lines into the file, press CTRL-X to exit. Follow the prompts to save the file as .netrc.

  4. Be sure to have the correct permissions on the file.  You can change the permissions with the command:

    $ chmod 600 ~/.netrc
  5. Try out the webdav client by issuing the command:

    $ cadaver https://dav.box.com/dav

    It should give you a prompt like:

    dav:/dav/> 

    Within this prompt, you can view files and navigate through the file system using the usual Bash commands cd and ls. To download files from Box, use the command:

    get <filename>

    Or, alternately, to upload files to your Box, use:

    put <filename>

    To exit the prompt, press ctrl-d

  6. Within a submit script, you can upload and download files by using commands such as:

    #!/bin/sh
    #SBATCH ...
    ....
    cat << EOF | cadaver https://dav.box.com/dav 
    get inputfile.txt
    EOF
     
    cat << EOF | cadaver https://dav.box.com/dav 
    put outputfile.txt
    EOF
  7. The files should automatically appear in your Box account, and be sync'd to your computer if you have the sync client installed.

Step-by-step guide for Lftp

  1.  Create an external password for Box as described in steps 1 and 2 above.
  2.  Load the lftp module:

    Load the lftp module
    module load lftp  
  3. Connect to Box using your full email as the username and external password you created:

    Connect to Box
    lftp -u <username>,<password> ftps://ftp.box.com
  4. Test the connection by running the ls command.  You should see a listing of your Box files.  Assuming it works, add a bookmark named "box" to use when connecting later:

    Add lftp bookmark
    lftp demo2@unl.edu@ftp.box.com:/> bookmark add box
  5. Exit lftp by typing quit.  To reconnect later, use bookmark name:

    Connect using bookmark name
    lftp box
  6. To upload or download files, use the get and put commands.  For example:

    Transferring files
    [demo@login.crane ~]$ lftp box
    lftp demo2@unl.edu@ftp.box.com:/> put myfile.txt
    lftp demo2@unl.edu@ftp.box.com:/> get my_other_file.txt
  7. Lftp also supports using scripts to transfer files.  This can be used to automatic downloading or uploading files during jobs.  For example, create a file called "transfer.sh" with the following lines:

    transfer.sh
    open box
    get some_input_file.tar.gz
    put my_output_file.tar.gz

    To run this script, do:

    Run transfer.sh
    module load lftp
    lftp -f transfer.sh