Different Resnet, loss function, and labelling methods

The following runs are used to evaluate which model configuration perform better. The average F1 and F2 scores of the 10 runs are used as the evaluation metric.

The ten runs have different training datasets and testing datasets. The idea of splitting the run into 10 runs is because I want to see how the model configuration performs on different testing dataset. The idea is similar to k-fold cross validation, where each fold has trained on the different training dataset and validated on the different validation datasets. The reason why I train the model this way instead of the actual k-fold cross-validation is that I want to make sure that the testing datasets consist of images that the model has never seen before (images from different videos) since multiple frames from the same video are similar to each other, and those frames can be ended up in testing and validation datasets resulting in a high validation score.

The ten runs are used to evaluate the different model configuration. The actual model for each model configuration is a model trained and validated using the all dataset.

For the model I used in the fine-tuning, I created a new model that used all datasets (including the testing dataset) in training and validation. For the DVN resnet 50 model I used in the fine-tuning with SD1, I created a model using all images from the dataset.

Labels from video vs Labels from access database

• For image label csv produced base on the video annotation, ~30,000 out of 41,000 of the training dataset were labeled as ND
• For image label csv produced base on the access database, ~22,000 out of 40,000 of the training dataset were labeled as ND

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)		Labels in test dataset
	video annotation	access database	video annotation	access database	video annotation	access database	video annotation	access database
39/50	0.9626	0.9534	0.5514	0.5988	0.5323	0.5984	['ND', 'TB', 'TBA', 'AMH']	['ND', 'TB', 'TBA', 'AMH']
40/51	0.8776	0.9081	0.2901	0.2046	0.2554	0.1562	['ND', 'TB', 'TBA', 'AMH']	['ND', 'DAGS', 'TB', 'TBA', 'AMH']
41/52	0.9708	0.9505	0.1269	0.4626	0.1178	0.4378	['ND', 'TB', 'TBB', 'TBA', 'AMH']	['ND', 'TB', 'TBA', 'AMH']*
42/53	1	0.9905	N/A	0.9231	N/A	0.8824	['ND']	['ND', 'AMH']
43/54	0.9881	0.9638	0.9136	0.9486	0.8836	0.9256	['ND', 'TF', 'TB', 'AMH']	['ND', 'TF', 'TB', 'AMH']
44/55	0.9979	1	0.5668	1	0.5464	1	['ND', 'TF', 'AMH']	['ND', 'TF', 'AMH']
45/56	0.8938	0.8693	0.3442	0.3558	0.3485	0.3411	['ND', 'TBA', 'TB', 'AMH', 'RFJ']	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
46/57	0.9361	0.8737	0.5451	0.6619	0.4783	0.6309	['ND', 'DAGS', 'TB', 'AMH']	['ND', 'DAGS', 'TB', 'AMH']
47/58	0.9576	0.8233	0.6841	0.5658	0.651	0.5627	['ND', 'TB', 'DAGS', 'AMH']	['ND', 'TB', 'TF', 'TBA', 'DAGS', 'AMH']
48/59	0.9712	0.9821	0.2861	0.3107	0.2708	0.3168	['ND', 'TB', 'AMH', 'DAGS', 'TBB', 'SSS', 'TBA']	['ND', 'TB', 'AMH', "TBB', 'SSS', 'TBA']**
Average	0.95557	0.93147	0.4787	0.60319	0.453788889	0.58519

*ISJ class was removed from test dataset used in RunID 52 because the model did not have ISJ as a class

**SZ class was removed from test dataset used in RunID 59 because the model did not have SZ as a class

Weighted loss function

List of weight assigned for each class

Defect	Weigth
'AMH'	0.65
'AOC'	8.04
'BSV'	5.15
'CC'	226.74
'CS'	26.36
'DAE'	2.71
'DAGS'	0.14
'FC'	24.12
'FM'	20.99
'IDJ'	19.89
'IRJ'	10.7
'ISGT'	6.26
'ISJ'	377.9
'JOM'	6.37
'JSM'	6.3
'LD'	5.94
'LR'	6.2
'MMC'	1.96
'ND'	0.05
'OBB'	12.32
'OBJ'	8.52
'OBN'	8.65
'OBR'	8.79
'RFB'	226.74
'RFJ'	5.37
'RFL'	6.63
'RMB'	16.67
'SAV'	14
'SCP'	19.89
'SRI'	5.53
'SSS'	3.8
'SZ'	4.93
'TB'	0.32
'TBA'	2.69
'TBB'	7.61
'TBI'	21.8
'TF'	0.67
'TFA'	4.52
'TFC'	7.18

Weigths assigned to pos_weight parameter of loss function

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)		Label in test dataset
	no weighted loss func	weighted loss func	no weighted loss func	weighted loss func	no weighted loss func	weighted loss func
50	0.953	0.703	0.599	0.448	0.598	0.389	['ND', 'TB', 'TBA', 'AMH']
51	0.908	0.950	0.205	0.183	0.156	0.139	['ND', 'DAGS', 'TB', 'TBA', 'AMH']
52	0.951	0.840	0.463	0.404	0.438		['ND', 'TB', 'TBA', 'AMH']
53	0.991	0.929	0.923	0.444	0.882	0.333	['ND', 'AMH']
54	0.964	0.911	0.949	0.728	0.926	0.676	['ND', 'TF', 'TB', 'AMH']
55	1.000	0.978	1.000	0.696	1.000	0.643	['ND', 'TF', 'AMH']
56	0.869	0.845	0.356	0.265	0.341	0.259	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
57	0.874	0.762	0.662	0.599	0.631	0.523	['ND', 'DAGS', 'TB', 'AMH']
58	0.823	0.599	0.566	0.366	0.563	0.344	['ND', 'TB', 'TF', 'TBA', 'DAGS', 'AMH']
59	0.982		0.311		0.317		['ND', 'TB', 'AMH', "TBB', 'SSS', 'TBA']
Average	0.931	0.835	0.603	0.459	0.585	0.413

Weigths assigned to weight parameter of loss function

Run ID	F1 normal		F1 average (no ND)		F2 average (no ND)		Label in test dataset
	no weighted loss func	weighted loss func	no weighted loss func	weighted loss func	no weighted loss func	weighted loss func
50	0.953	0.808	0.599	0.325	0.598	0.268	['ND', 'TB', 'TBA', 'AMH']
51	0.908	0.581	0.205	0.000	0.156	0.000	['ND', 'DAGS', 'TB', 'TBA', 'AMH']
52	0.951	0.931	0.463	0.364	0.438	0.328	['ND', 'TB', 'TBA', 'AMH']
53	0.991	0.773	0.923	0.833	0.882	0.758	['ND', 'AMH']
54	0.964	0.820	0.949	0.734	0.926	0.725	['ND', 'TF', 'TB', 'AMH']
55	1.000	0.997	1.000	0.330	1.000	0.274	['ND', 'TF', 'AMH']
56	0.869	0.864	0.356	0.311	0.341	0.306	['ND', 'TBA', 'TB', 'AMH', 'RFJ']
57	0.874	0.854	0.662	0.500	0.631	0.478	['ND', 'DAGS', 'TB', 'AMH']
58	0.823	0.835	0.566	0.336	0.563	0.313	['ND', 'TB', 'TF', 'TBA', 'DAGS', 'AMH']
59	0.982	0.681	0.311	0.262	0.317	0.269	['ND', 'TB', 'AMH', "TBB', 'SSS', 'TBA']
Average	0.931	0.814	0.603	0.400	0.585	0.372

Resnet50 vs Resnet101

Train Data	Image labelling method	Backbone	Fine Tune	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)	Test Dataset
DNV dataset (10 runs)	Video’s annotation	Resnet50	-	0.940*	0.454*	2 DNV videos (10 runs)
DNV dataset (10 runs)	Video’s annotation	Resnet101	-	0.945*	0.503*	2 DNV videos (10 runs)
DNV dataset (10 runs)	Access Database	Resnet50	-	0.940*	0.585*	2 DNV videos (10 runs)
DNV dataset (10 runs)	Access Database	Resnet101	-	0.910*	0.458*	2 DNV videos (10 runs)
DNV dataset (all videos)	Access Database	Resnet50	SD1 dataset	0.719	0.853	30 SD1 images
DNV dataset (all videos)	Access Database	Resnet101	SD1 dataset	0.704	0.826	30 SD1 images
SewerML dataset	-	Resnet50	SD1 dataset			30 SD1 images

*These scores are the average F2 score of the 10 runs.