Skip to main content

Week 05

Sept 27 2022 (End of the day update)​

  1. I submitted the job using accelerate launch --config_file $SLURM_TMPDIR/sewer_ml/config/acc_config.yaml to run fastai_distributed.py.
    1. Here are errors that I got:
      1. Having a problem with wandb. AssertionError.
      2. ERROR:torch.distributed.elastic.multiprocessing.api:failed.
      3. Other jobs that started when this job was running failed when trying to connect to wandb. error_messages
  2. I resubmitted jobs for my sample train and valid data set, but still got the problem with wandb.
  3. I looked up Whisper to extract text from images. However, I could only file example code where they used on audio files and not images.
  4. I did a google search and found tesseract-ocr. Tesseract-ocr is trained to extract text from images. Tesseract_GIT and UB-Mannheim_Windows_Installer.
  5. I ran fastai with my sample train and valid data set on colab. I defined f1score_multi = FBetaMulti(1, average=None)because we want to be able to extract the score that only corresponding to ND and f2score_multi = FBetaMulti(2, average=None) because we can apply our custom weight to each label after the run.
  6. Problem: Wandb only show only one set of fbeta_score that is corresponding to f2score in the summary section. How do we get it to displace both f1score and f2score?

Sept 28 2022​

Meeting​

1. Extract text from images:​

  1. DocQuery -> scans an image -> you can ask questions has a dep on tessaract
  2. open-source SOTA OCR system --> tessaract, opencv
  3. openai --> whisper model: extract audio --> get the text from audio, extracted text has a timestamp --> (maybe?)(implicit assumption: when something is spoken, corresponding text appears at the same time shown in the video)
  4. There is python package pytessaract.
  5. The defect will still be present right Before and after the observation text. Need to decide to which one to keep and discard. Do Auto correlation to find the image that is similar with each other.
  6. Run accelerate (without wandb) to train model and save the model. Run wandb with predition script.

End of the day update:​

  1. I submitted the job for fastai_distributed without wandb. The job crashed because it Unable to find a valid cuDNN algorithm to run convolution. In addition, base on the Compute Canada Support, because we only request 24 cpu per node (p100l), we should set --num_cpu_threads_per_process=6 for 4 GPUs. Because the configure file did not have information on the num_cpu_threads_per_process, num_cpu_threads_per_process was set to 24 as default value when accelerate launch was called.
  2. I resubmitted the same job from point 1 and modify my accelerate launch line from accelerate launch --config_file {path to config} {script} {arg}... to accelerate launch --num_cpu_threads_per_process=6 --config_file {path to config} {script} {arg}.... However, I still got the same error.
  3. Sudhir believed that accelerate might not work on Slurm environment because when we called acceleratre config on google colab, the first question asked in which compute environment are you running, they only have two options: This machine (local computer) and AWS (Amazon SageMaker).

error message 1/2

error message 2/2

Questions to Deven:​

  • Can you open an issue to huggingface asking if accelerate working with Slurm environment?
  • Can we log F1 Multi and F2 Mulit to wandb when predicting the label of the image?

To-do List for tomorrow:​

  • Modify print the fastai_prediction to print f1 and f2 score.

  • Send email Jerry Weimer about if I can access the code? for NASSCO PACP.

  • Send email to Pipelogix to see what is the difference between pipelogix and Phoenix

  • Send email to Sacramento Area asking if they follow PACP standards

  • Read Joakin's people why he uses Gaussian blur to block the text.

    Sept 29 2022​

  • 720 X 576 sewer-ml image size

  • Smart image extract: Google OCR

  • Label detection - given image it will assign the label (dog, cat) without giving where the object are

  • text detection - given image extract text from image

  • Document text detection - give pdf file extract text from pdf file.

  • Google vs microsoft comparison:

  • ArcGIS model builder in

  • Model builder in FME to detecting grade (infer the information from video and match with GIS information) Have a sensor detect the height of the pipe, changing over time can detect corrosion. Detect CCTV pipe diameter via CCTV. Can we detect the change in pipe diameter via CCTV footage Detect the water level

  • Gaussian blur date and time, distance

  • rewire operation to grab image after the observation text appear.(so that we can a frame before the observation text appear).

  • Redo the image extract with the same name

  • Distance is not always increased because we do some zooming

  • Do spatial auto correlation function to find if we did not go far enough and still see the defect. (to get the clean image)

  • Try to with fix time to see (go back to two sec) to see it work well without out doing spatial correlation.

  • Detect camera movement by detect the text in the image. (to know if it moves forward and backward)

  • The faster the ocr engine the faster frame you can go

  • Only gaussian blur date and time and include the image without observation text.

List of questions to Sean:​

  • why counter is in distance. the value is different from video and access
  • how sheet number is being recorded? by paper or through a database software

List of questions to Compute Canada​

  • how many cpu does p100l have?
  • write accelerate again with wandb on compute canada.
  • train all train and valid folder with smaller batch size

List of questions to other:​

  • email to author that is on the leader board. Asking him about the model
  • email Joakim how he did gaussian blur
  • send email to deven and sudhir the paper of segmentation and water level
Last updated on by Viraj