Week 01

Oct 31

get the Q&A for the NASSCO training
Azure for OCR
Use Azure output for custom OCR
Use the counter instead of the distance to improve the distance based labelling
Streamlit
- copy all the unblurred and blurred images to public folders
- choose the csv containing the path of file paths to the image in the public folders

Use Streamlit to relabel the images from 33 videos. When this is done, will submit the fastai multi-labels classification run using those images to compute canada.
I intend to use this to increase images with the observation (defect) labels by looking at images that have distance close to the observation distance as well as discard images do not show defects during zoom in (example: photo of the pipe's ceiling and black frame before the inspector annotated MSA (survey is abandoned))
Suggestion I found from Haurun, 2021 paper:
- Only grab ND images if the distance is 1m away from the observation (defect) distance to ensure no defect is visible in those images. This won't be a problem because we have a lot of images of ND comparing to other classes

Finished the building the settler block. However, we got negative value of Nitrate and nitrite nitrogen concentration (SNO). I will need to check for the error. However, my plan is to continue building the next blocks in order to push this model forward.
Submit the following runs to compute canada:
- Defect vs Non defect run with SewerML sample and full and DNV dataset
- Multi labels with DNV dataset (33 + 39 videoes)
- Water level estimation run with SewerML only images with VA labels for Class10 and Class15. Half and full size

Will meet with Dr. Lence on Thursday around 4pm. Will try to settle any pending tasks from Sean and go over thesis outline.

For half images, F1 micro and F1 macro scores decrease slightly when we train water level estimation with only images that have VA (water level label) for both class10 and class15 standard
For class10, the water estimation model has F1 micro score of 0.799 on validation dataset. The paper has F1 micro score of 0.397 on the testing dataset.

I will work on relabelling the first four videos (1,12, 14, 20) and will submit job to compute canada with the new label

Type of model	runID	Input Data	Image Size	Epoch with best Valid_loss	Valid_loss	Note
Water level estimation_Class10	05	SewerML_Train_Val_WL	Full size	9	>0.5
Water level estimation_Class15	02	SewerML_Train_Val_WL	Full size	valid loss is increasing	>0.5
Water level estimation_Class10	03	SewerML_Train_Val_WL	half size	6	>0.5
Water level estimation_Class15	04	SewerML_Train_Val_WL	half size	7	>0.5

Water level estimation_Class10	07	SewerML_Train_Val_va_WL	Full size	7	>0.5	only images with VA label
Water level estimation_Class15	08	SewerML_Train_Val_va_WL	Full size	6	>0.5
Water level estimation_Class10	09	SewerML_Train_Val_va_WL	half size	6	>0.5	peak at epoch 7 before going down
Water level estimation_Class15	10	SewerML_Train_Val_va_WL	half size	9	>0.5

Multi-label defect	07	SewerML_Train_Val	Full size	10	0.08602
Multi-label defect	10	SewerML_Train_Val	half size	8	0.08922
Multi-label defect	13	SewerML_Train_Val_no_va_label	Full size	10	0.07377	replace VA with ND
Multi-label defect	12	SewerML_Train_Val_no_va_label	half size	9	0.07662
Multi-label defect	03	SewerML_Train_v1	Full size	10	0.08275
Multi-label defect		SewerML_Train_v1	half size

I finished the Activated Sludge state variables to Anaerobic Digester state variables block.
I will work on remodifying my image_labelling script this weekends. I have not heard back from Pavan about Azure. So I will test the pipe line once I heard back from him.