Experiment tracking

GQC uses Weights and Biases (wandb) for its experiment tracking for pytorch, fastai, LightGBM and XGBoost.

wandb has great out of the box support for most of the most ML frameworks.

danger

We currently do NOT have a subscription for Weights and Biases. When you use a personal account in Weights and Biases, you get Unlimited hours of tracking, but if you create a Team with multiple members in it, you are limited to only 250 hours of tracking.

GQC has already reached the limit of 250 hours of tracking.

Using wandb in online mode

By default wandb works in online mode, meaning that your model is training, you will see the parameters (loss, metrics) being logged real-time on the wandb website.

Refer to documentation available on the official site to use the appropriate callback → W&B Tutorials

Using wandb in offline mode

1. One might need to use this option, when internet access is unavailable.
2. The narval cluster in Compute-Canada does not have internet access, hence we need to use wandb in offline mode.
3. In this situation, follow these steps,
   1. Set the environment variable WANDB_MODE=offline to save the metrics locally, no internet required. (OR) run the command wandb offline in your terminal.
   2. When you're ready, run wandb init in your directory to set the project name.
   3. Run wandb sync YOUR_RUN_DIRECTORY to push the metrics to our cloud service and see your results in our hosted web app.
   4. You can check via API whether your run is offline by using run.settings._offline or run.settings.mode after your wandb.init().
   5. These steps are taken from → here