DNV CCTV

• Link to concept of Fastai
• Oct 6 Pavan: All videos will be standardized to 700 (width) x 470 (height) size
• Oct 20 Vannary: DNV images were labeled based on the distance value. For example, if the inspector annotated observation at 7 m. All frames with 7m will get labeled as the annotation made by the inspector. (Will need to figure a better way to label image)
• Oct 24 Vannary: From Oct 24 onward, all DNV model did not include labels MWL, MCU, MSA, and MGO
• Oct 25 Vannary: All run submitted on October 25 and after should not have a problem with wandb when loading the model because we commented out wandb login() and only called wandb init() right before training.

DNV CCTV footage

• Vannary had downloaded the first set of uploaded data from GQC's Gdrive.
• Those files can be deleted by Deven and Vannary can request Sean to upload missing videos (15GB max).
• Vannary has access to 2TB on her Gdrive (413 GB is used)

JSON files generated from Google Vision API

• Received on October 03 2022 from Pavan

• Pavan has generated JSON files containing the OCR text data, for the frames containing text (in the middle region) of the video: '/content/drive/MyDrive/VS_Research/CCTV/DNV/data/test_1/SANMN00893_1.mp4'

  1. Video was first sampled with 1 frame per each second. The original frame rate of the video is ~30 fps. So we have frame array like: [0, 30, 60, 90, ...]
  2. Before sending to OCR we do a binary classification to screen the frames containing text in the middle region. (For now, we used the Google vision API to do this classification. But we'll move on to using our own method to do this) We get a filtered array for frames containing text in the middle like: [30,60,330,360,390,...]
  3. Then we run OCR on the identified frames and get OCR data for the full frame including distance and date/time data. They are saved as JSON files.

• Find the JSON files at https://drive.google.com/drive/folders/1--fXy7KIcwNcMDomcwFRbZ9aWPQjxlZJ?usp=sharing Naming convention: <videofile_name><frame_id>.json

  • Eg: SANMN00893_1_000060.json

• These JSON files will have to be processed in order to obtain the required data.

  • Ex:
    • 

• OCR results are present in the "textAnnotations" section in the JSON:

• It starts with a summary of all text that has been identified as shown above and then there are subsections for each text segment.

Access Database

• There are 221 PACP condition codes. We will need to select which labels to include.

• From Access dataset, the SANMN00893_1 (InspectionID = 2) has 6 PACP code:
  • AMH Access Point Manhole
  • MWL Miscellaneous Water Level (VA)
  • RFL Roots Fine Lateral (RO)
  • TB Tap Break-in/Hammer (GR, PH, PB, OS, OP, OK are related to connection pipes)
  • TBB Tap Break-in Abandoned (GR, PH, PB, OS, OP, OK are related to connection pipes)
  • SSS Surface Damage Surface Spalling (OB)
• There were mistakes in an assess dataset. ConditionID 9 - RFL should be at distance 14m and not 0.0m. The annotation in the video was correct.
• The water level at 0.0 m is assigned to be 5% in an Access database. The value should be to the nearest 5%. There is no annotation of water level at 0.0m in the video, but exist in the access database. In the Access database, the water level at 3.1 m is assigned to be 40%.
• MWL will be entered at 0.0 m. After this initial entry, only significant changes of at least 10% in water level are recorded.

Notes Google VISION API and Access Database

Error by Google VISION API

• The current code, the text is recognized as a distance if the text is followed by letter m. Some image frames do not contain distance attribute because Google vision API failed to recognize letter m or could not recognize any text.
• Sometimes, Google vision API detected distance as 71 .6m instead of 71.6m. Therefore, we got 0.6 in the distance column instead of 71.6.
• Google vision API output wrong distance image. Distance between each frame should not be greater than 2 m.
• The current code, the text is recognized as label if the text located after Observation:. Sometime Google vision API sees Observation: as Observation (without ":"). (Fixed by Pavan)

1. Need to clean up the wrong observation labels that got from Google vision API
   1. Create the dictionary of all the known defect of the video
   2. Need to include FH (End of Survey) in the dictionary (added on Oct 27)
   3. REGEX needs to be enhanced to handle observation text nuances
   4. Google vision API detect TB (Tap as TBCTap. If we follow the dictionary rule set in OCR notebook, the output will be TBC and not TB. We will need to write a function to handle this. Suggestion: use mode to get label that likely to happen at the given rows with label within certain number of row.
2. Need to clean up the abnormal distance labels obtained from Google vision API
   1. replace distance that is greater than the end of survey distance with np.nan
   2. replace the distance of the current row with np.nan if the difference between the two values is greater than 6 (video SANMN04254_44 has a big jump in distance from 0 to 3.1m. Therefore the difference will need to increase from 2 to greater than 3.1m. 6 was chosen because the distance between the two frames should within 5 m) (change on Oct 27)
   3. replace values where next row is less than the current row with np.nan
   4. call function look_backward and look_forward to replace distance value with np.nan with distance value of the closet row or the average between backward and forward distance if the number of row is equal.
   5. There is a problem with all_frame_SANMN04254_44 csv file that need to be checked. All the distance column is labeled as 0m.video SANMN04254_44 has a big jump in distance from 0 to 3.1m. Therefore the difference will need to increase from 2 to greater than 3.1m. 6 was chosen because the distance between the two frames should within 5 m
   6. Dec 17 2022. clean_row_with_dist_incr_grt_num(df,num=4). In SANMN07555_40.csv there is a jump from 4.7 to 9.0 because the Azure mistook 4.9 as 9.0.

Videos and Access Database

3 folders that have CCTV footages:

• 1 DHL Lynn Valley Oct 6 2021 (34 videos)
• DHL Clements Ave Oct 2020 (1 video)
• DHL Lynn Valley March 3 2022 (46 videos)

1 DHL Lynn Valley Oct 6 2021 folder

1. We will need to separate video with MSA (survey was abandoned) from other videos.
   1. What would we do with images after the survey was abandoned?
      1. For example, if the camera is under water, do we label this image as ND?
   2. For now, we will remove any frame that come after MSA label and frames with MSA and MCU (camera underwater) labels
   3. Video SANMN03277_29 (Inspection ID: 1), the survey was abandoned at 19.3 m because current water level and debris is prohibiting camera movement.
   4. Video SANMN01000_5 (Inspection ID: 6), the survey was abandoned at 1.9 m because of concrete obstruction.
   5. Video SANMN01139_9 (Inspection ID: 10), the survey was abandoned at 48.4 m because debris in pipe. Additional cleaning was needed.
   6. Video SANMN00834_10 (Inspection ID: 11), the survey was abandoned at 53 m because of grease. Additional cleaning was needed.
   7. Video SANMN00478_14 (Inspection ID: 15), the survey was abandoned at 16 m because of water level due to sanitation debris. Additional flushing was required.
   8. Video SANMN03707_15 (Inspection ID: 16) (no video), the survey was abandoned at 0.0 m because benching was too tight.
   9. Video SANMN03005_17 (Inspection ID: 18), the survey was abandoned at 5.3 m because it could not pass deviation.
   10. Video SANMN08983_19 (Inspection ID: 20), the survey was abandoned at 1.9 m because there was too much flow.
       1. all images are underwater. Exclude images from this video for now.
2. The corresponding inspection condition of some videos in #1 DHL Lynn Valley Oct 6 2021 folder are located in other access database folder
   1. Video SANMN01751_41: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN01751_39.MP4, inspection id: 11)
   2. Video SANMN02735_79: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN02735_77.MP4, inspection id: 49)
   3. Video SANMN02808_38: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN02808_36.MP4, inspection id: 8)
   4. Video SANMN03805_43: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN03805_41.MP4, inspection id: 13)
   5. Video SANMN05099_50: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN05099_48.MP4, inspection id: 20)
   6. Video SANMN05724_55: access database located in DHL Lynn Valley March 3 2022 folder. (Video name in the access database: SANMN05724_53.MP4, inspection id: 25)
      1. At 26.8 m, there is a tab break that looks like infiltration.
3. Only video SANMN05099_50 has broken label (B) at 75.9 m.

DHL Clements Ave (1 video)

1. There is discrepancy between label shown in the video and the one in Access Database
   1. At distance = 17.4, the video has four labels, the access database only has three labels
   2. The inspector accidentally entered IDJ (Infiltration Dripper at Joint) and then he corrected himself by entering IDC (Infiltration Dripper at Connection)
2. Sometime, the text still is visible on the screen even though the camera has moved out from the defect spot.
3. ISJ is labeled as continuous defect in access database. We cannot get this information from the video.
4. When the water level is high, the camera will have to pan upward (did not look at the center of the pipe)
5. The camera is moving at the faster rate than the videos from the previous folder (1 DHL Lynn Valley Oct 6 2021 folder)

Label image dataset from csv files generating from running Google vision API

Proposed steps to label the DNV image dataset

1. Separate video with MSA label from the rest of other video. We will need to figure out a way to handle these videos separately.
   1. For now, we will remove any frame that come after MSA label and frames with MSA and MCU (camera underwater) labels.
2. Remove the first frame of each video because it is a blank frame.
3. FH (End of survey) is not a part of the PACP condition code. (Please do not be confused with FH2, FH3, and FH4, which are used to describe Fracture longitudinal Hinge 2,3,4). For now Remove all rows that contain FH.
4. Fix the distance column of the CSV file of rows where there is no observation.
   1. Check for an image with a value greater than the distance of the last image, NaN, or missing a value.
   2. Get the distance value from the closest image (difference between two row numbers) with the correct distance value that comes before the current image.
   3. Get the distance value from the closest image (difference between two row numbers) with the correct distance value that comes after the current image.
   4. Count the number of rows from the current image to the two images.
      1. Set the distance value of the current image the same as the image with a smaller difference between two row numbers.
      2. If the number of rows is equal, find average between the two distance value and report distance in one decimal place.
5. Fix the distance column of the CSV file of rows where there is an observation
   1. Search for the closest image (smallest difference between row number of current image and the image) in both directions (images before and after the current image). Get the distance value from the closet image that has the same observation label.
6. Find the distance that has an observation label. For each distance and label pair, images with the distance value will be labelled as their corresponding label.
   1. In case there is more than one label per distance. All labels will be included.
   2. There will be an additional step between steps 5 and 6 to label images within the distance of the observation label.
      1. for example, if the label is annotated at distance 3.8. Will need to do spatial correlation between image at 3.8 and image within +/- 0.5 distance of 3.8m
7. Images with distances that have been labelled as MWL or other PACP codes related to water level will also have a label of non-defect (ND) if no other defect or observation is presented.
8. The remaining images with no label will be labelled as non-defect (ND).

Water level label processing

1. Access dataset will be used to get the water level of images using the Distance, PACP_Code, and Value_Percent columns of the PACP_Conditions table.
   1. Images with the same distance as the MWL label of the PACP_Conditions will be labelled as MWL and assign the water level value to the water level column. NULL will be given to the rest of the image for now.
   2. From the video footage, we did not see the observation label of MWL (water level) and AMH (Access Manhole) at the beginning of the video. However, in the Access Dataset, each video begins with AMH and MWL at 0.0 m before logging other defects or observations. We can ignore AMH at 0.0 m as the camera pointed toward the pipe, not the manhole.
   3. Additional step will be needed to determine whether the water is visible in the image. For example, the frame was captured when the camera panned upward. To avoid this uncertainty, we can only assign the water level to images with MWL for now.

Proposed steps to create random 10 dataset of train-validation images and test images

1. Create distribution of labels of each video
2. Training data must contain at least one image of every type of label (Vannary)
3. Filter out label that not related to defect code (Vannary)
4. Divide the videos into train-validation video and testing videos
5. For each split, set different seed numbers. Look at the distribution of labels of each seed and decide whether this is a good distribution to run the model

Problems with the current model

There are three major problems with our model:

1. The image is labelled according to the distance where inspectors input their observations.
   • The problem is the defect might be visible before and after those distances.
   • Look at the video manually to see where within the distance from the annotated distance that defect can be seen.
   • Ask Pavan to create a utility function where the inputs are the current frame, and the number of previous frames you want to do spatial correlation with. Then, plot the spatial correlation vs the number of frames from the selected frame to find the spatial correlation threshold.
2. Some labels are too specific. For example, from the straight view, the inspector could not tell whether the tap break is active or abandoned. He/She will need to zoom in and move the camera around to determine the state of the tap break.
   • Need to find out how to handle look around situation
3. There is class imbalance in the dataset
   • Find out a way to handle class imbalance. Can SMOTE handle this? (Deven:I don't think SMOTE can handle this. SMOTE is usually used for applications in Tabular data. What we need in this case would be a weighted loss function.)
     • Focal loss is an improved version of cross-entropy loss(CE) that tries to handle class imbalance problem by assigning more weights to hard or easily misclassified example.
   • We can try duplicate frames 30 times. Right now, we grab one frame per second. The original video has 30fps. The name of our image is in the interval of 30. The duplicated frame can be labelled as [31, 32,....59](Deven: I am not sure if duplicating the frames would solve our problem.)

Class imbalance

1. Haurum (2021) method:
   1. Calculate Class imbalance (CI) in the datasetby calculating the ratio between the largest and smallest class.
   2. Calculate to see how often several classes are present at the same time. Starting counting class with the highest and add the next highest class until we reach the class with the least number in the observation. The author then calculate label cardinality (LC) for each split.

Matching videos to SANMAIN shape file

• The sanity main id can only be found at the beginning of some videos. However, all the videos will include the information regarding the starting and ending manholes/node.
• To match videos to SAN MAIN shape file, we will use the starting and ending manholes/nodes.
  • Extract the starting and ending manhole for each video.
  • Creating video name csv file containing video name, starting manhole/node, and ending manhole/node columns.
  • Creating sanitary main shp file containing the ASSET_ID and manhole ID that the main goes through.
  • Using starting manhole and ending manhole to join the point 2 and point 3 dataframes.
• For videos that contain 'SANMNxxxxx_xx'in the filename, the ASSET_ID can extracted directly from the file name.

from 73 videos used to train the model, there are 71 unique sewer ID.

• SANMN03959 have two videos (SANMN03959_67.MP4 and SANM03959_70.MP4)

• SANMN04050 have two videos (SANMN04050A_83.mp4 and SANMN04050_82.mp4)

• SANMN86600 was not found inside the shape file.

• total surveyed length = 3733.3 meters

• Out of 81 videos uploaded in 2022 (including - SANMN08983_19.mp4 (not use for training)), two sanitary mains are not found in SanMain shp file:

  • SANMN08983
  • SANMN86600

• total surveyed length = 3890.8 meters