MLOps

Our current system does not include any tools or methods for Design or Deploy. To make the runs reproducible Weights and Biases is used for experiment tracking and logging.

Workflow

Current

1. Start a notebook using nbdev
2. When the notebook is created configure it to work with wandb
3. Track runs, various parameters, graphs and other plots
4. Push the notebook to the appropriate repo, repeat from Step 1.

New

1. Start with designing the experiment in a design tool like mlflow, metaflow, etc.
2. Create notebooks with nbdev as usual
3. Track experiments with wandb
4. Write unittests in nbdev
5. Optional: Deploy (or operate) to see the results

Design

• There are a lot of tools for designing the entire ML experiment.
• mlflow is a popular option
• metaflow is another one

Experiment tracking

• We are sticking with Weights and Biases for this.
• It has integration with all the frameworks we have used so far.
• It is easy to get started with and the export functionality is also available.

Train

Hyperparameter tuning

Options include:

• Optuna
• Ray distributed
• Weights and Biases Sweep

Deploy

We can deploy a few the models we've trained using fastAPI.

• Other options include streamlit for quick experimentation and giving it a basic UI.
• Comprehensive option includes something with a React frontend, FastAPI backend and a lot more.

Future thoughts

• For deploying, REST endpoints can be made for say image_classification, image_segmentation or wildfire_prediction, etc.