Week 5

Mar 03/27/23

Training with EasyOCR

The training sample given on their website how they want the labels. The model expects cropped words with their annotated labels inside a labels.csv.

That would mean that all the images that we got the text extracted from that would have to be cropped and then annotated to create the required training dataset.

The resulting csv would look something like this,

filename, words

,Freudian %

They have provided the required training scripts, this can be explored. The required links are available over here ↓

1. https://github.com/JaidedAI/EasyOCR/blob/master/custom_model.md
2. https://github.com/JaidedAI/EasyOCR/tree/master/trainer

TAO

Sudhir wants to explore this.

1. https://developer.nvidia.com/tao-toolkit-get-started

Meeting with Vannary

1. Issue with weighted loss function across multiple gpus #119 (low priority)
2. Runs with 692 videos
   1. method 1: concatenate all seperate test sets DNV, SD1, COV
   2. method 2: combine all: then split 80 (75-25 t&v) 20 test
3. Multi-label defect classification
   1. DNV
   2. SD1
   3. COV
   4. All combined
      1. This is done
4. Run ND vs D
   1. DNV
      1. Both runs are made for both 80 and 486 (486 also includes the previous 80) videos
      2. Runs synced with wandb
      3. Results for test
         1. with fine_tune → done
         2. with fit_flat_cos → not yet
      4. Results compiled?
         1. with fine_tune → done
         2. with fit_flat_cos → not yet
   2. COV
      1. not yet
   3. SD1
      1. not yet
   4. All combined
      1. blah
5. Only defect (no ND) model
   1. DNV
   2. SD1
   3. COV
   4. All combined

Issues with data validation

1. Frames being mislabelled (ND images getting classified as an image containing defect)
   1. point1
   2. point2
   3. point3

fme-to-python

1. break_value assignment remaining for:
   1. mod1: distance based assignment
   2. mod2: distance based assignment
   3. mod3: this is bldg_type based: layout is shared with Vannary

Mar 03/28/23

1. Found some good hits for converting rst to markdown
   1. https://docs.readthedocs.io/en/stable/guides/migrate-rest-myst.html#how-to-convert-existing-restructuredtext-documentation-to-myst
   2. https://rst-to-myst.readthedocs.io/en/stable/index.html
   3. https://stackoverflow.com/questions/13396856/markdown-output-for-sphinx-based-documentation This was explored, results are not good

Priorities

1. Concept-app
   1. Need to write the design up, I have the skeleton ready
2. Doc on AI-tooling and others
   1. Document has been started need to show it to Sudhir and Jake and then I need to ask for some input on the same

Mar 03/29/23

1. Training with EasyOCR
   1. This is done.
   2. Need to find the prediction rate and check the inference pipeline.

Mar 03/31/23

Possible workflow for Vannary

1. Extract frames into zip files (done) (about 107 zips for 1066 wRC videos)
2. Azure OCR produces text and corresponding bounding boxes (done)
3. Frames are snipped from the standard extracted directory to extract the distance region in each middle-frame
   1. Deven is creating the app to find the distance region to snip the image using the middle frames
   2. Distance regions will be snipped from all extracted frames from the zip files based on the previous point. Organize the snips so that they are consistent with the extracted frames zip files (extracted snips directory - names of the zips should be identical).
4. Stitch a maximum of 200 snips at a time from the same video using ffmpeg to create a single file which would be sent to Azure OCR
   1. Find the text corresponding to each snip
   2. Process the text to find the distance and assign it to the corresponding frame
   3. Write a csv for each video for all the extracted frames (frame_id, distance)
   4. Group the csv files into zip files according to the original group
5. Assign defects to frames based on the distance from the Access DB

For 1066 wRC videos

1. • Extract frames
2. • Azure OCR produces text and corresponding bounding boxes which is written into JSON format
3. • Determine bounding box which lies entirely inside the distance region and the corresponding text (needs a script to loop over JSON files)
   1. • Frames are snipped to extract the distance region in each frame
4. • Use the snippets and the extracted text to train the EasyOCR model
5. • train + valid split for EasyOCR training
6. • training script + dir structure for VS to run on compute-canada