Skip to main content

Week01

Dec 5​

  • make sure Q, temperature, Soluble, and particulate has to be greater than zero

  • Hard code index for Soluble and Particulate and replace it for every python script

  • Pavan: Get the input and out from biroeactor 1 and give them to Deven.

  • There is problem with So4 control because the KLa has to be in output through unit delay causing a sawtooth output for So..etc

  • There is possible to have two tap breaks next to each other. For an example, in SANMN01140_8, there was one abandoned and one active tap break that located in proximity to each other.

Dec 6​

Collimator​

  • Asking Brian about initialization script

  • Asking Brian about non-float csv

  • Need to check column name csv: KeyError: 'Thickener.Q_in.out_0'

  • I was able to get the output close to Matlab if I used use the integrator to delay the input of Kla (same concept as hydraulic delay) instead of using unit delay

CCTV​

  1. Location of access database csv
    1. master_access.csv contains list of defects found in each videos (all the videos from three folders)
    2. videoname_filename_vs_access_database.csv is used to match the actual .MP4 file name to the name input in access database
      1. MP4 file name = column header ['videofile_name']. Note all the csv files produced from Azure are saved under name found in this column.
        1. csv files produced from Azure are stored here
      2. name of the video input in access database = column header ['videoname_in_access]
  2. Location of image label csv
    1. create_image_labels_v2.ipynb is what I used to label each image with defect
      1. There are two ways to label image
        1. According to annotation captured from image
        2. According to access database (including continuous defect)(there is still bug that needed to be fixed)
    2. location of image label csv using annotation captured from image and OCR via Azure
      1. train_test_csv folder contains 10 set of training dataset and testing dataset.
        1. Two videos from Azure_33_39.csv file (image label containing all videos except SANMN00304_1.MP4) were set aside for testing.
        2. I have generated 10 set of training dataset and testing dataset from Azure_33_39.csv
        3. notebook used to split data to training and testing dataset
          1. Run the cells under Split the test dataset according to pipe material header
            1. All unique pipe materials have at least 6 videos. Therefore, we do not have to exclude any pipe material from doing random generating test dataset.
            2. 35 videos of AC pipe
            3. 16 videos of PVC pipe
            4. 9 videos of RCP pipe
            5. 8 videos of VCP pipe
            6. 6 videos of CP pipe
          2. We need to exclude videos that have rare defect (number of defect <=2 in access database) from getting picked as test dataset
            1. list of videos with rare labels:
              1. 'SANMN00304_1.MP4',
              2. 'SANMN04161_63.MP4',
              3. 'SANMN05599_57.MP4',
              4. 'SANMN04377_61.MP4',
              5. 'SANMN05502_58.MP4',
              6. 'SANMN04267_74.MP4',
              7. 'SANMN86600_4.MP4',
              8. 'SANMN00893_1.MP4',
              9. 'SANMN01000_5.MP4',
              10. 'SANMN00937_2.MP4',
              11. 'SANMN04254_42.MP4',
              12. 'SANMN00834_10.MP4',
              13. 'SANMN03805_41.MP4',
              14. 'SANMN05724_53.MP4',
              15. 'SANMN07555_38.MP4',
              16. 'SANMN08088_46.MP4',
              17. 'SANMN05099_48.MP4',
              18. 'SANMN08142_44.MP4',
              19. 'SANMN03959_68.MP4',
              20. 'SANMN00868_27.MP4'
        4. List of test dataset:
          1. ['SANMN04339_62.MP4', 'SANMN05955_57.MP4'],
          2. ['SANMN04167_64.MP4', 'SANMN01138_6.MP4'],
          3. ['SANMN01140_8.MP4', 'SANMN06083_74.MP4'],
          4. ['SANMN04050_82.MP4', 'SANMN08983_19.MP4'],
          5. ['SANMN01275_21.MP4', 'SANMN06099_58.MP4'],
          6. ['SANMN01275_21.MP4', 'SANMN00947_25.MP4'],
          7. ['SANMN04339_62.MP4', 'SANMN03959_67.MP4'],
          8. ['SANMN05390_52.MP4', 'SANMN06083_74.MP4'],
          9. ['SANMN05893_72.MP4', 'SANMN05283_51.MP4'],
          10. ['SANMN03745_18.MP4', 'SANMN00850_28.MP4']
        5. The rest of the videos that were not included in the test dataset were used in training
      2. continuous_defect folder contains image label csv created using access database

Dec 07​

  • Tie pytorch transform to fastai to sharpening in memory.

  • Possible trying sharpening to SewerML to see if the result improve

  • Ask if our name can be on the leader board for SewerML

  • Improve the document on the notebook

  • Ask Brian about non-float csv

  • share the result with Brian and ask him they have something similar to c-mex s function.

    • In matlab there is mechanism to break algebraic loop

  • set Submodel parameter

  • how to set end time when running simulation in notebook

  • need to exclude any image 1 m away from the start of continuous defect.

    • Can be fixed by including the start of continuous defect in the list of point defect. List of point defect is the input when drop_frame_within_x_dist_from_point_defect is called.

Dec 08​

  • The Anaerobic Digester influent values are different between Collimator and matlab outputs. However, the output for Anaerobic Digester effluent values are similar except for Salk.
    • Variables that are different:
      • Si (influent)
      • Ss (influent)
      • So (influent)
      • Sno
      • Snd
      • Salk (effluent)
  • The error in the Salk concentration of Anaerobic Digester effluent caused the difference in the value of Salk concentration of Dewatering influent and effluent and Storage Tank influent and effluent between Collimator and Matlab.

Dec 09​

  • The error with the Anaerobic Digester influent values because there is a bug when demuxing the output from the input port. Fixing the issue by demuxing the output from the blocks that is not an input port.
  • I can only run the simulation for an hour (about 2 end days for BSM2) before getting out-of-time error
  • I was able to implement sharpening transformer given by Deven successfully
  • Adding sharpening transformer to fastai script and submit 10 jobs that corresponding to 10 jobs created on Nov 30 (randomly set 2 videos for testing) with the new fastai script.
  • Editing create_image_labels_v2 notebook for continuous defect. (In progress). Link to issues to be fixed
Last updated on by Viraj