Skip to main content

Setting up a compute cluster

This document goes over some basic tips and tricks you can use to setup your /project space inside a compute cluster.

When you are given access to a compute cluster, you are added as a user in that entire Linux user group. So, when you SSH into the cluster with your credentials, you are just logging into a Linux VM.

caution

This Setting up python environment manager section has been tested on WSL2 and Ubuntu 20.04 and CentOS. It may not work on other distros. It was tried on RedHat, but it requires some modifications to be made.

The flow of the document is as follows:

SSH into the cluster with VSCode​

The entire process is documented over → here

Cofiguration​

Dotfiles contain cofiguration settings for a tool/command that you are using in Linux. I personally prefer using fastsetup from fastai

Setting up python environment manager​

I personally prefer using conda as my virtual environment manager. Follow the instructions in the fastsetup repo to setup conda as your virutal environment manager.

Run the setup-conda.sh script to install Mambaforge in your cluster.

git clone https://github.com/fastai/fastsetup.git
cd fastsetup
source setup-conda.sh
. ~/.bashrc
conda install -yq mamba

vim and tmux settings​

You can copy the .vimrc and .tmux from the dotfiles repo and use them as your config for vim and tmux. After going back and forth, I found two very modern and customizable vim and tmux configs

  1. amix/vimrc: The ultimate Vim configuration (vimrc) (github.com)
  2. gpakosz/.tmux: 🇫🇷 Oh my tmux! My self-contained, pretty & versatile tmux configuration made with ❤️ (github.com)

These links have detailed explanations on how to setup and use their configs.

Going beyond SLURM 101​

These are some good reference links.

  1. https://researchcomputing.princeton.edu/education/external-online-resources/slurm
  2. https://kb.iu.edu/d/awrz
  3. https://docs.alliancecan.ca/wiki/Running_jobs

Solving cuda errors​

Here are some links to the forums and other pages where you can ask questions.

  1. Understanding GPU memory
  2. fast.ai forums
  3. PyTorch forums
  4. Huggingface forums

Optional: Distributed compute​

Since, in a compute-cluster you have multiple nodes having multiple GPUs, understanding how gradient synchronization and how multiple GPUs work in parallel is critical.

Distributed and Parallel Training Tutorials — PyTorch Tutorials 2.0.0+cu117 documentation

Issues encountered with RedHat (EPA Cluster)​

  1. There were issues encountered with the set command and the getting the download URL for the operating system.
  2. Also, the default shell in the EPA Cluster is tcsh, the default in 20.04 is bash.
  3. The setup-conda also needs to be modified to add the conda initialization in the .tcshrc file.