Unprotected Large Data Transfer

Upload to Google Drive

When uploading large datasets to notebook instances, the process may timeout. In this case, it is recommended to upload your file into Google Drive and download it from the terminal in Jupyter Notebooks.

1. Use the New button in the top left corner of Google Drive to upload your file.
   
   
   
   
2. In the file’s Share option, access will be restricted by default. To change this, select Anyone with a link from the dropdown menu. This will allow file access via Jupyter Notebook’s terminal later.
   
   
   
   
3. Click the Download button to bring up the “Google Drive - Virus scan warning” screen.
   

   Screen Opens in Separate Tab
   

   If this screen automatically opens in another tab:
   
   
   
   Then, append &confirm=yes to the end of the URL in the search bar, save it, and skip to step 4.
   
   

   Screen Opens as Pop-up
   

   If the “Google Drive - Virus scan warning” window opens within the same tab:
   
   
   
   Then:

   • Copy/paste the file’s share link in the search bar
     
     
     
     
   • Locate/copy the file ID in the URL
     
     
     
     
   • Combine that ID with these URL parameters
     
     
     
     https://drive.google.com/u/0/uc?id={REPLACE WITH ID}&export=download&confirm=yes
     
     
4. Once the proper URL is obtained, the file can be downloaded into your Jupyter Notebook through the integrated terminal. Use the wget command, the URL to your Google Drive file, and wget’s -O option to download the file under a specific name in your current directory.
   

You can start a new terminal session by clicking the blue “+” button in the sidebar of your Jupyter Notebook, then selecting Terminal to start a new terminal session.

Command template:

  $ wget "https://example.url" -O {exampleFileName.tar.gz}

Example:

 $ wget "https://drive.google.com/u/0/uc?id=160dDIhVQLNkUgBgaSIMxuJz0WKoDowyv&export=download&confirm=yes" -O nih.tar.gz

Check download:

  $ ls -la

If the dataset is sensitive or protected, email itd-research.ci@sdsu.edu to evaluate more secure options for transferring research data.