Skip to main content

Steps to train DNV model

Questions by Deven​

1. Where and how all the notebooks organized. Since I'll be organizing them in nbdev. I will need that information​

Ans: I have moved notebooks to create labels csv in to this folder. The notebooks that used to run fastai on colab are stored in this folder.

2. Where and how are the datasets organized?​

Ans: The datasets are stored inside this folder

Folder NameContent
videovideo MP4 files received from DNV. The files are organized in three folders. The name of the folder is the name given by DNV.
Azure_JSONJSON file produced when running CCTV OCR Video Processor v2 with Azure notebook. The files are ogranized by the video filename.
Gvision_JSONJSON file produced when running CCTV OCR Video Processor (release) notebook. The files are organized by the video filename.
CCTV_Original_Framesunblurred images that were created by extracting one frame per each second. The files are organized by the video filename.
CCTV_Training_Datablurred images and text extracted from each image stored in csv file. The files are organized by the video filename.

The image label csv files that will be used to train the model are stored in this folder. The files are created when running create_image_labels_v2 notebook.

Folder NameContent
Azureimage label csv files created from text extracted via Azure csv file located in CCTV_Training_Data
Gvisionimage label csv files created from text extracted via Gvision csv file located in CCTV_Training_Data/Gvision
Access Databasemaster_access csv file and videoname_filename_vs_access_database csv file

The blurred image from the CCTV_Training_Data folder and image label CSV files from Azure are the two main file types to train DNV defect classification models.

Note: The image label csv file from each video are combined to create merged image label csv that used to train the DNV defect classification model.

  1. azure_33_39_videos.csv contains image label csv files of videos found in Lynn Valley Oct 6 2021 and Lynn Valley March 3 2022 folders.
  2. labels_Lynn_Valley_Oct_6_2021.csv contains image label csv files of videos found in Lynn Valley Oct 6 2021 folder.
  3. labels_Lynn_Valley_March_3_2022.csv contains image label csv files of videos found in Lynn Valley March 3 2022 folder.

train_test_csv folder contains 10 set of training and testing datasets generated from images from videos found in Lynn Valley Oct 6 2021 and Lynn Valley March 3 2022 folders.

3. Is there any specific notebook sequence that I need to follow to label, organize or store the images that come from DNV?​

Ans:

  1. Run CCTV OCR Video Processor v2 with Azure notebook to capture frame from video and create csv file that contains text found in each frame. The outputs are save in CCTV_Training_Data
  2. Run create_image_labels_v2 notebook to create image label csv file.
    1. You will need to add path to text extracted csv file located in CCTV_Training_Data
    2. There are two ways to create image label label:
      1. According to annotation captured from image. The output csv will be stored in image_label/Azure
      2. According to access database (including continuous defect)(there is still bug that needed to be fixed). The output csv will be stored in image_label/Azure/continuous_defect
  3. Run split_data_to_train-val_test notebook to split to data to train and test dataset. (I will need to clean up this notebook)

To train the model​

  1. Clone compute-canada repository from github
  2. Create a folder called train inside sewer_ml directory
  3. Upload the images that you want to use for training to train folder
  4. Make sure you current path is in sewer_ml directory in the terminal
  5. In the terminal, enter python ./script_runner.py -s {python script} -r {run_id} -b {batchsize} -e {epochs_number} -i {name of csv file} -m {model_name}
    1. Example: python ./script_runner.py -s fastai_multi_label_v2_470_700 -r 47 -b 32 -e 10 -i train_8 -m fastai_defect_DNV
  6. The outputs will be stored inside outputs folder

py scripts to train the model​

  1. fastai_multi_label_v1_470_700.py: using fastai to randomly split dataset into training and validation dataset. It can post a problem if a rare label only exists in validation dataset. The image size is set to DNV image size (470, 700).
  2. fastai_multi_label_v2_470_700.py: using train_test_split to split dataset into training and validation dataset. I have not encountered the problem with a rare label only exists in validation dataset.The image size is set to DNV image size (470, 700).

To run prediction model​

  1. Clone compute-canada repository from github
  2. Create a folder called train inside sewer_ml directory
  3. Upload the images that you want to use for prediction to train folder
  4. Make sure you current path is in sewer_ml directory in the terminal
  5. In the terminal, enter python ./script_runner.py -s {python script} -r {run_id} -i {name of csv file} -m {model_name}
    1. Example: python ./script_runner.py -s fastai_multi_label_predict_DNV -r 48 -i test_9 -m 48_fastai_defect_DNV_32_10_train_9
  6. The outputs will be stored inside outputs folder.

py scripts to run prediction​

  1. fastai_multi_label_predict_DNV.py.

NOTE: When I run the prediction script on Compute Canada, I will need to modify fastai_multi_label_v1_470_700.py or fastai_multi_label_v2_470_700.py by uncommenting line 20 to call wand.login()

Location of my model pkl files​

I have stored my 10 model pkl files that created when randomly set a side 2 videos for testing in CCTV/DNV/models

Dec 6 notes​

  1. Location of access database csv
    1. master_access.csv contains list of defects found in each videos (all the videos from three folders)
    2. videoname_filename_vs_access_database.csv is used to match the actual .MP4 file name to the name input in access database
      1. MP4 file name = column header ['videofile_name']. Note all the csv files produced from Azure are saved under name found in this column.
        1. csv files produced from Azure are stored here
      2. name of the video input in access database = column header ['videoname_in_access]
  2. Location of image label csv
    1. create_image_labels_v2.ipynb is what I used to label each image with defect
      1. There are two ways to label image
        1. According to annotation captured from image
        2. According to access database (including continuous defect)(there is still bug that needed to be fixed)
    2. location of image label csv using annotation captured from image and OCR via Azure
      1. train_test_csv folder contains 10 set of training dataset and testing dataset.
        1. Two videos from Azure_33_39.csv file (image label containing all videos except SANMN00304_1.MP4) were set aside for testing.
        2. I have generated 10 set of training dataset and testing dataset from Azure_33_39.csv
        3. notebook used to split data to training and testing dataset
          1. Run the cells under Split the test dataset according to pipe material header
            1. All unique pipe materials have at least 6 videos. Therefore, we do not have to exclude any pipe material from doing random generating test dataset.
            2. 35 videos of AC pipe
            3. 16 videos of PVC pipe
            4. 9 videos of RCP pipe
            5. 8 videos of VCP pipe
            6. 6 videos of CP pipe
          2. We need to exclude videos that have rare defect (number of defect <=2 in access database) from getting picked as test dataset
            1. list of videos with rare labels:
              1. 'SANMN00304_1.MP4',
              2. 'SANMN04161_63.MP4',
              3. 'SANMN05599_57.MP4',
              4. 'SANMN04377_61.MP4',
              5. 'SANMN05502_58.MP4',
              6. 'SANMN04267_74.MP4',
              7. 'SANMN86600_4.MP4',
              8. 'SANMN00893_1.MP4',
              9. 'SANMN01000_5.MP4',
              10. 'SANMN00937_2.MP4',
              11. 'SANMN04254_42.MP4',
              12. 'SANMN00834_10.MP4',
              13. 'SANMN03805_41.MP4',
              14. 'SANMN05724_53.MP4',
              15. 'SANMN07555_38.MP4',
              16. 'SANMN08088_46.MP4',
              17. 'SANMN05099_48.MP4',
              18. 'SANMN08142_44.MP4',
              19. 'SANMN03959_68.MP4',
              20. 'SANMN00868_27.MP4'
        4. List of test dataset:
          1. ['SANMN04339_62.MP4', 'SANMN05955_57.MP4'],
          2. ['SANMN04167_64.MP4', 'SANMN01138_6.MP4'],
          3. ['SANMN01140_8.MP4', 'SANMN06083_74.MP4'],
          4. ['SANMN04050_82.MP4', 'SANMN08983_19.MP4'],
          5. ['SANMN01275_21.MP4', 'SANMN06099_58.MP4'],
          6. ['SANMN01275_21.MP4', 'SANMN00947_25.MP4'],
          7. ['SANMN04339_62.MP4', 'SANMN03959_67.MP4'],
          8. ['SANMN05390_52.MP4', 'SANMN06083_74.MP4'],
          9. ['SANMN05893_72.MP4', 'SANMN05283_51.MP4'],
          10. ['SANMN03745_18.MP4', 'SANMN00850_28.MP4']
        5. The rest of the videos that were not included in the test dataset were used in training
      2. continuous_defect folder contains image label csv created using access database
Last updated on by Vannary, Deven