Model Training Times and Batch Sizes
One Stage Models:
cctv-multilabel-sd1_d_all_VB.pkl Batch Size: 12, epochs: 5 trained on the video_group 01202023 from the SD1_700 (prev: 2023_05_26) dataset. Memory allocated: 8 GB. Time taken for training : 24 minutes.
cctv-multilabel-sd1_single_vidgrp_FB Batch Size: 18, epochs: 5 trained on the video_group 01202023 from the SD1_700 (prev: 2023_05_26) dataset. Memory allocated: 12 GB. Time taken for training: 24 minutes, 39 seconds.
cctv-multilabel-test_model_FB Batch Size: 18, epochs: 5 trained on the single video_group 01202023 from the SD1_700 (prev: 2023_05_26) dataset. Memory allocated: 12 GB. Time taken for training: 24 minutes, 35 seconds.
cctv-multilabel-sd1_d_all_VB_larger Batch Size: 12, epochs: 5 trained on two video_groups 01202023 and 01182019 from the SD1_700 (prev 2023_05_26) dataset. Memory allocated: 8 GB. Time taken for training: 167 minutes, 30 seconds.
Two Stage models:
cctv-first-stage-sd1_d_all_VB Batch Size: 12, epochs: 5 trained on one video_group 01202023 from the SD1_700 (prev 2023_05_26) dataset. Memory allocated: 8 GB. Time taken for training: 12 minutes 40 seconds. wandb link: https://wandb.ai/lence_ubc/cctv-sd1-singlelabel/runs/kiv0gjz4?nw=nwuserdevengqc
cctv-second-stage-sd1_d_all_VB Batch Size: 12, epochs: 5 trained on one video_group 01202023 from the SD1_700 (prev 2023_05_26) dataset. Memory allocated: 8GB. Time taken for training: 2 minutes 5 seconds. wandb link: https://wandb.ai/lence_ubc/cctv-sd1-defect-multilabel/runs/nyux3vb0?nw=nwuserdevengqc
cctv-first-stage-sd1_d_all_VB_larger Batch Size: 12, epochs: 5 trained on two video_groups 01202023 and 01182019 from the SD1_700 (prev 2023_05_26) dataset. Memory allocated: 8 GB. Time taken for training: 87 minutes 30 seconds. wandb link: https://wandb.ai/lence_ubc/cctv-sd1-singlelabel/runs/yldo3i4q?nw=nwuserdevengqc
cctv-second-stage-sd1_d_all_VB_larger Batch Size: 12, epochs: 5 trained on two video_groups 01202023 and 01182019 from the SD1_700 (prev 2023_05_26) dataset. Memory allocated: 8 GB. Time taken for training: 25 minutes 6 seconds. wandb link: https://wandb.ai/lence_ubc/cctv-sd1-defect-multilabel/runs/ayil7640?nw=nwuserdevengqc