Hyperparameter Tuning

This document contains the major list of hyperparamters that we can tune while training neural networks and even tree based models (LightGBM and XGBoost).

Learning rate

fastai

1. We use fastai for most of image recognition and CCTV training pipeline.
2. fastai out of the box has support for finding the ideal learning rate using learn.lr_find()

Vanilla pytorch and PyTorch Lightning

1. https://github.com/davidtvs/pytorch-lr-finder this link uses the same implementation used by the fastai folks
2. https://pytorch-lightning.readthedocs.io/en/1.4.9/advanced/lr_finder.html official support from the Pytorch Lightning community

Keras Tuner

If you are using keras to build your deep learning model, take a look at KerasTuner → here

Number of layers in the neural network

This is purely done on the basis of empherical evidence and experimenting. There is no one right answer, the answer is subjective and it will be different for each problem.

Loss functions

Based on different problems, you must use different loss functions.

1. Default is BinaryCrossEntropy
2. For imbalanced datasets, Weighted BinaryCrossEntropy
3. For segmentation (with class imbalance), FocalLoss
4. and so on

Learning rate schedulers (lr_schedulers)

There was various learning rate schedulers available, please refer to link → here

Tree based parameters

For finding optimal parameters in LightGBM, XGBoost and Random Forrest types of models, use the following links

1. Random search and Grid Search
2. Optuna
3. RayTune