Week 01

March 01

• Multi GPUs run using the narval cluster

  • copy shell file, py, notebook, settings.py, and script_runner.py to the narval cluster
  • modify shell file to work on narval cluster
  • request for interactive node to create the new config file
    • run tutorial notebook on the interactive node for num of processes = 2,3,4
    • [ ]compare the results to the cedar cluster's results
  • run tutorial py file via sbatch (number of batch, epoch, process, and requested gpus the same as runs submitted to the cedar cluster)
    • submit the job by requesting the same parameters as cedar cluster (number of cpu cores per node = 24 and 32)
    • submit the job by requesting a higher number of cpu per node (number of cpu cores per node = 48)
    • compare the results to the cedar cluster's results
  • run CCTV py file via sbatch (80 videos dataset)(number of batch, epoch, process, and requested gpus the same as runs submitted to the cedar cluster)
    • submit the job by requesting the same parameters as cedar cluster (number of cpu cores per node = 24 and 32)
    • submit the job by requesting a higher number of cpu per node (number of cpu cores per node = 48)
    • compare the results to the cedar cluster's results

• The below problem has been fixed by running wandb offline on narval cluster because narval cluster does not have an internet access.

  • In narval, problem with wandb.init()
    • wandb 0.13.10
    • protobuf 3.19.4
  • In narvel, problem with wandb.init()
    • wandb 0.13.10
    • protobuf 4.21.12
  • In cedar, multi_gpu env, there is no problem with wandb.init()
    • wandb 0.13.10
    • protobuf 4.21.12

• reorganize compute-canada/sewer-ml folder

• Question about rclone

  • divide the file in the folder that I want to copy into n folder (each folder has 5GB in size)
  • use time rclone copy to copy files from gdrive to cedar
  • Show this that the problem is not from the quota limit but the rclone token only valid for one hour. Asking how we can get rclone token that last more than 1 hour (like for days)
  • could not use rclone mount to mount gdrive on the cedar cluster

• problem with extract from azure json
  • previously we search for the label with 1 character first before look at 5, 4, 3, 2 character
  • upon discuss with pavan, he suggested changing to look at 5, 4, 3, 2, 1.
  • row 416, there is no BRO in the PACP code.
  • 

March 02

• Create a read me for collimator say how to get the physical model and the biorector 1_AI model.
• rename BIOREACTOR 1 py with bioreactor_1_py
• create a read me and cut and paste to description in collimator
• markdown that have the Name and Description of each submodel and model. Naming it the same style collimator. bioreactor_v3 to Biorector_1_with_integrator_delay_block (use abbreviation)
• if csv file comes from matlab, put matlab in front of the file name.

It is better to run on $SLURM_TMPDIR

• read file directly from $PROJECT
• What is speed performance different between cp file to slurm_tmpdir between reading directly from $PROJECT
• do the extract frames for 10 videos and copy the performance between cp file to slurm_tmpdir or read the file from $PROJECT
• train the model for specific material. (read the pipe sleuth)

March 03

Meeting with Lina for Waste Water Treatment Plant Hybrid Model Project

• The simulation is driven by influent and other parameter such ras flow...

  • Simbar, gpx simulation software? you can feed influent to sub process the run the simulation

• Optimization of the parameter in Waste Water Treatment

• How can they can integrate their input to perform optimization?

• OCR Azure (2.5 frames per second on colab)

• create a test cases for each express captured by labels extract function

• run azure on 10 COV videos on compute canada (get json files)

  • require python >= 3.8 to install azure cognitive vision packages

• run prediction on COV using DNV model training on 80 videos