Skip to main content

Week 03

March 14​

  • moving all utility data to hard drive (make sure i have duplicated files on hard drive)

  • creating an issue on compute canada how to move copy to hard drive

    • get back to Compute canada about the transfer issue

  • run the new extracted labels on cov and sd1

  • having all the defects in test dataset

  • single model vs (non defect vs defect model and defect classification model)

  • A count of train and validation and test images of each label (show the images)

  • put 486 videos together (priority)

  • send all emails to compute canada

  • run notebook to distance

  • switch to FME

  • SD1 middle frame distance extraction:

    • Here is the list of expressions that the current code failed to recognize as a distance:

      1. SD1-video_type_1:
        • One-digit number with one decimal place (both with and without space after the period) (e.g., 4. 9, 0.0, 5.4)
        • Number that has a comma instead of a period and is not followed by FT or T (e.g., 25,1)
        • Number without a decimal place (e.g., 31.)
        • Number with a negative side (e.g., -0, 6)
      2. SD1-video_type_2:
        • Number with a negative side (e.g., -010, 0 F)
        • Number with a space before a period (e.g., 0000 .0 F)
      3. SD1-video_type_3:
        • One-digit number with one decimal place (e.g., 8.7 FT)
      4. SD1-video_type_4:
        • Azure OCR saw zero as the letter O (e.g., 000. OLF)

    • The distance for all images from SD1-video_type_5 were extracted correctly by the current code

    • Here is the list of expressions that the current code was able to recognize as a distance:

      1. SD1-video_type_1:
        • Two more digit numbers with one decimal place (with and without space after the period) (e.g., 135.6, 48. 7)
        • Number followed by FT , T (e.g., 20. 9T, 33,4FT, 52, 1FT, 0. OT, 65.6FT)
      2. SD1-video_type_2:
        • Four-digit numbers with one decimal place (without space between a period) (e.g., 0000.0 F , 0016.0 F)
      3. SD1-videos_type_3:
        • Two more digit numbers with one decimal place (with and without space after the period) (e.g.,17.2 FT , 78. 0 FT)
      4. SD1-videos_type_4:
        • three-digit numbers with one decimal place and followed by LF (e.g., 006. 1LF)
      5. SD1-video_type_5:
        • Two more digit numbers with one decimal place without space between a period (e.g., 58.0 FT)
        • We need to get more frames from SD1-video_type_5.

March 15​

Fail to extract distance from the following expression:

  • SD1-video_type_1:
    • 208 18 (no period)
    • 31.

FME-to-PYTHON​

Module 9 to 11 are not related to flow path. These three modules require find the intersection between key feature with water main polyline. New functions will be created to use for these three modules.

  1. function to get segments of water main lines that intersect with the key feature (such as road and soil)

    • use geopandas overlay function to find the intersection line.
    • This function will store type of road and type of soil of the intersection line in the attribute table.

  2. function to calculate the percentage of lines segments over total water main pipe length

    • use geopandas.GeoSeries.Length to get the length of line segments

  3. function to calculate the average break key

    • refer to break key table
    • the previous function to get the weighting factor (percentage of lines segments over total water main pipe length)

  • Module 9:

    • Determines type of road the water main is buried underneath.
    • Input files: road polygon, water main polyline.
    • For each water main, use function 1 to get intersection lines and their corresponding type of road.
    • For each water main, use function 2 to get the percentage cover factor of each line over the total water main pipe length.
    • For each water main, use function 3 to get the weighted break key

  • Module 10:

    • Determine the percentage of pipe that is underneath the road
    • Input files: road polygon, water main polyline.
    • For each water main, use function 1 to get intersection lines.
    • For each water main, use function 2 to get the percentage cover factor of each line over the total water main pipe length.
    • Assign the break key according to the percentage cover factor.

  • Module 11

    • Determine the type of soil a pipe is located in.
    • Input files: soil polygon, water main polyline.
    • For each water main, use function 1 to get intersection lines and their corresponding soil type
    • For each water main, use function 2 to get the percentage cover factor of each line over the total water main pipe length.
    • For each water main, use function 3 to get the weighted break key.

For each module, break key value will be stored in dataframe with the following columns:

  • min_value: minimum value for a break key value
  • max_value: maximum value for a break key value
  • break key value

Expressions that did not get extracted as distance​

  1. SD1-video_type_1
    • 10182022-104134 AM-CHANDLER MANCHESTER
      • number without a period (e.g., 49 7, 9, 1)
      • number without decimal place (e.g., 31.)
    • 10202022-80702 AM-TEST (SD1-video_type_1)
      • Azure extracted distance as 21/ 2FT. The current code assigned 21/2 as a distance. This value cannot be converted to a float.
      • number without a period (e.g., 250)
  2. SD1-video_type_2
    • 10192022-101015 AM-JOJO NICHOLAS and 10192022-104040 AM-JOJO NICHOLAS
      • extracted distance correctly
  3. SD1-video_type_3
    • 11222022-103822 AM-COREY BREWER
      • Azure OCR extracted 0 as @ (e.g., 162.@ FT)
    • 11222022-114345 AM-COREY BREWER
      • distance was not extracted from a frame that contains only distance value (e.g., frame only contains 2.4 FT in the text field)
  4. SD1-video_type_4
    • 1122022-73739 AM-COREY BREWER
      • number with O or @ instead of 0 (e.g., 002. OLF, 014. OLF, 035. @LF, 039 OLF, 000, OLF)
    • 1132022-112314 AM-COREY BREWER
      • number with O instead of 0 (e.g., 006. OLF, 005, OLF)
  5. SD1-video_type_5
    • 11302022-104009 AM-COREY BREWER
      • number without a period (e.g., 01 1 FT, 6 FT, 96 3 FT)
      • number with @ instead of 0 (e.g., 115.@ FT)
    • 11302022-120214 PM-COREY BREWER
      • number followed by F (e.g., 212.7 F)
      • number without decimal place and FT is found in the next line (e.g., 284. and FT is on the next line)

Other observations​

  • the rest of the frames with nan distance is because Azure OCR did not recognize the distance field.
  • For some frames, Azure OCR extracted numbers incorrectly (e.g. 203.3 FT as 803.3 FT)
  • For some frames, the current code saw the manhole id as a distance. This only happened with frames that display the CCTV survey information (e.g., 0540133 was extracted as 540). The peak of distance vs frame_index graph happened due to this.
  • In the 1132022-112314 AM-COREY BREWER video, the camera was moved to a different location at 10.45 min. The distance followed by LF was reset to zero. However, the distance at the left corner followed by FT was the same as the frame before 10.45 min. I will have to check the access database to see which distance value was entered in the access database.

March 17​

  • submit run with lr_find with sample 00 with bs 40, epoch = 3,

    • add ideal_lr = learn.lr_find() inside learn.distrib_ctx

  • SaveModelCallback(every_epoch=2, with_opt = True) (submit with 32 GB) for 10 epochs

    • pretrained = False, no image transform, no weighted loss

  • separate out the test logic

  • prepare the script for:

    • pretrained = False, no image transform, no weighted loss
    • pretrained = True, no image transform, no weighted loss
    • pretrained = False, with image transform, no weighted loss
    • pretrained = True, with image transform, no weighted loss
    • pretrained = False, no image transform, with weighted loss
    • pretrained = True, no image transform, with weighted loss
    • pretrained = False, with image transform, with weighted loss
    • pretrained = True, with image transform, with weighted loss

  • nd vs defect (80 videos and 486 videos) (10 epochs) (pretrained)

  • label SD1 images (excluding video type 4) and submit jobs

    • save count of images that distance did not get extracted correctly
  • how is our process different from object detection technique
  • water level model (aggregate bigger water level percentage )
  • confusion matrix
  • do 80% Tr and Valid and 20% test
    • 75% Tr and 25% V (single fold)
    • cross validation use k_fold =4 to get 75% and 25% V
  • model with all images from SD1, DNV, COV
  • transfer learning between utility
Last updated on by Viraj