Skip to main content

Utilities

tree_py​

A utility added in the notebooks repo created by Deven using nbdev. This lists the directory structure as a tree

Following workflow is how to setup this in colab environment

  1. This requires a Colab pro subscription as we need to access terminal.
  2. Open a new colab notebook and mount the drive.
  3. Open terminal window. This is basically a tmux shell.
  4. Have github credentials ready. Accessing colab require your github username and a github token.
  5. Clone the notebooks repo to Mydrive/repos/ directory.
  6. Go in the notebooks directory and install the package
    pip install -e .
  7. Now you can use the command tree_py -h to get the usage information and use it. tree py output

rclone​

Used this to clone files from google drive to Linux environment. Followed this tutorial and rclone documentation.

  1. Install rclone

    sudo apt-get install curl
    curl https://rclone.org/install.sh | sudo bash

  2. Setup the remote

    rclone config

    1. Press n and give a name for the new connection. new connection

    2. Then it asks for the storage type. Scroll the list of options and find the number corresponding to Google Drive. In my case it was 18. drive option

    3. Then leave client_id and client_secret blank in the next prompts.

      It mentions that leaving these empty will result in low preformance. If the performance of the clone command is not adequate, then we should revisit this.

    4. Then it will ask for scope. Give full access. scope

    5. Leave service_account_file empty.

    6. Give n to Edit advanced config?

    7. For Use web browser to automatically authenticate rclone with remote? give Y for WSL or local computers, give N for headless machines (like compute-canada)

      Headless setup prompt is shown below. It requires another machine with rclone installed and still have access to a web-browser like WSL. headless

    8. For Configure this as a Shared Drive (Team Drive)? give N. It will print the configuration and token.

    9. It will ask if you want to keep this remote that's just created. Give Y.

    10. Quit the config and you are done.

  3. Copy files using rclone copy command. E.g.

    rclone copy hydrotrek-g-drive:/BSM1/outputs/ .