Link Search Menu Expand Document (external link)
SDSU Research & Cyberinfrastructure

Unprotected Large Data Transfer

Upload to Google Drive

When uploading large datasets to notebook instances, the process may timeout. In this case, it is recommended to upload your file into Google Drive and download it from the terminal in Jupyter Notebooks.

  1. Use the New button in the top left corner of Google Drive to upload your file.

    Google Drive Upload

  2. In the file’s Share option, access will be restricted by default. To change this, select Anyone with a link from the dropdown menu. This will allow file access via Jupyter Notebook’s terminal later.

    Share Access

  3. Click the Download button to bring up the “Google Drive - Virus scan warning” screen.

    Screen Opens in Separate Tab

    If this screen automatically opens in another tab:

    Virus scan warning (Separate Tab)

    Then, append &confirm=yes to the end of the URL in the search bar, save it, and skip to step 4.

    Screen Opens as Pop-up

    If the “Google Drive - Virus scan warning” window opens within the same tab:

    Virus scan warning (Same Tab)

    Then:

    • Copy/paste the file’s share link in the search bar

      URL View Mode

    • Locate/copy the file ID in the URL

      File ID in URL

    • Combine that ID with these URL parameters

      URL Download Mode

      https://drive.google.com/u/0/uc?id={REPLACE WITH ID}&export=download&confirm=yes

  4. Once the proper URL is obtained, the file can be downloaded into your Jupyter Notebook through the integrated terminal. Use the wget command, the URL to your Google Drive file, and wget’s -O option to download the file under a specific name in your current directory.

You can start a new terminal session by clicking the blue “+” button in the sidebar of your Jupyter Notebook, then selecting Terminal to start a new terminal session.

Command template:

  $ wget "https://example.url" -O {exampleFileName.tar.gz}

Example:

 $ wget "https://drive.google.com/u/0/uc?id=160dDIhVQLNkUgBgaSIMxuJz0WKoDowyv&export=download&confirm=yes" -O nih.tar.gz

Check download:

  $ ls -la

Download in Terminal

If the dataset is sensitive or protected, email itd-research.ci@sdsu.edu to evaluate more secure options for transferring research data.