Skip to main content

External Drives

T7 (2TB SSD)​

NOTE The T7 has proven to crash the MSI Server and we don't know why. Thus, it has been moved to the Microcenter machine.

Directory structure 2 levels down within T7​

T7
├── Samsung Portable SSD SW for Android.txt
├── SamsungPortableSSD_Setup_Mac_1.0.pkg
├── SamsungPortableSSD_Setup_Win_1.0.exe
├── System Volume Information
│   ├── WPSettings.dat
│   └── IndexerVolumeGuid
├── cctv
│   ├── COV (merge with unionsine2)
│   ├── dnv (merge with unionsine2)
│   └── SD1 (unblurred frames)
├── downloaded_cctv_files (can be deleted)
│   └── dnv
└── delete_me_temp
    ├── unzip.sh (can delete)
    ├── fuse_zip_test (can delete)
    ├── cov_regular (contains a lot of pngs, don't know if we need to delete or not)
    ├── dnv (contains pngs for PACP, WRc videos may not want to delete it since the pngs are already unzipped)
    └── 01182019.zip (can delete)

  • The folders inside cctv/COV/Data should be moved to /media/gqc/unionsine2/cctv/cov/Data/COV_A on the msi-server since they are important for running the CCTV CV pipeline.

  • The cctv/dnv folder contains a sqlite database , blurred frame zips, unblurred frame zips, condition data, and a yaml file for the WRC_1066 dataset. My recommendation is to move the dnv folder to unionsine2. I could not find a copy of this on the msi server.

  • The cctv/SD1 folder contains SD1_700_extracted_frames, and SD1_B_extracted_frames folders which contain the unzipped unblurred frames for the 2023_05_26 and SD1_B datasets respectively. The zipped versions of the unblurred frames for 2023_05_26 exists on the msi-server over here: /home/gqc/CCTV/SD1/Data/2023_05_26/Extracted_Frames. The zipped versions of the unblurred frames for SD1_B exists on the msi-server at: /media/gqc/unionsine2/cctv/sd1/Data/SD1_B/extracted_frames. Apart from SD1_700_extracted_frames and SD1_B_extracted_frames folders, the SD1 folder also contains a folder called SD1_BackupDBs which contains some of the older databases that were run through the CCTV CV pipeline with SD1 data. We are no longer using these databases. Finally, there is a SD1_blurred_frames_unzipped folder which contains the unzipped blurred SD1 frames. A copy of these unzipped SD1 blurred frames exist in /home/gqc/CCTV/SD1/SD1_B_+_2023_05_26_blurred_frames. Considering that there are copies of the files and folders, it is safe to delete the cctv/SD1 folder.

  • The downloaded_cctv_files/dnv folder is empty and can be deleted.

  • All the folders and files in the delete_me_temp folder can be deleted except the cov_regular and dnv (this dnv folder is different from the one in cctv) folders. I have not found copies of the cov_regular and the dnv folder on the msi server.

Tree diagrams of some of the folders:

Inside the COV folder:

COV
└── Data
    ├── Video_Lists
    ├── Extracted_CSV
    ├── Extracted_Frames
    ├── transfer_data_fromvidlists.py
    ├── Video_DB
    ├── Condition_Data
    ├── create_csv_from_frames.py
    ├── transfer_frame_ids.py
    ├── Azure_JSON
    ├── Labelled_Images_CSV
    ├── script_to_update_OCR_json.py
    └── transfer_data_from_labelled_imgs.py

Inside the SD1 folder:

SD1
├── SD1_BackupDBs
├── SD1_700_extracted_frames
├── SD1_blurred_frames_unzipped
└── SD1_B_extracted_frames

WD_Black (5TB HDD)​

Directory structure 1 level down within WD_BLACK​

WD_BLACK
├── System Volume Information
├── $RECYCLE.BIN
├── VS-DRIVE-DOWNLOAD-DNV
├── .Trash-1000
├── sd1-may-9-23
├── calgary_pipe_breaks_models
├── ai3-old-data
├── ai3-gdrive-new-data
├── config.yaml
├── DNV
├── delete_me_test_rclone_folder
├── orsanco_db_dumps
├── videoplayback.mp4
├── videosPathsList.txt
├── converted
├── second_download
├── separated_old_videos
├── extracted-570
└── external_hd_hierarchy.md

Unionsine1 (10TB HDD)​

Directory structure 2 levels down within unionsine1​

unionsine1
├── config.yaml
├── DNV_Blurred_Frames
│   └── dnv_791_train.tar.gz
├── Procedures
│   ├── Buildings along the Flow path.docx
│   ├── Delineation of Watershed.docx
│   ├── DNV Flow directions using ArcGIS pro／SagaGIS.docx
│   ├── flow path_flowchart.docx
│   └── Summary of Ji and Qiuwen (2015) - A GIS-based subcatchments division approach for SWMM.docx
└── VS_Research
    ├── .fastai
    ├── 01 Flow path
    ├── 02 BRI
    ├── 03 Research topics
    ├── 04 COF for COV
    ├── 06 DNV
    ├── 07 NLP
    ├── 08 Climate Change
    ├── 09 HEC RAS
    ├── 10 Collimator
    ├── 11 FME-to-Python
    ├── ArcGISPro_30_182209.exe
    ├── CCTV
    ├── config.yaml
    ├── Eric's Notes
    ├── FME
    ├── Lit Review
    ├── loe_coe_app
    ├── pipe_breaks
    ├── videoplayback.mp4
    └── weights.zip

Inside the CCTV folder within VS_Research​

CCTV
├── accelerate
├── All_Utilities
├── Archive
├── CCTV_video_type_metadata.csv
├── COV
├── DNV
├── feature_engineer_model_output
├── level3.txt
├── METRO
├── models
├── model_output
├── ND_VS_Defect-no_internet.ipynb
├── ND_VS_Defect.ipynb
├── notebooks
├── plots
├── Prediction
├── prediction_csv
├── SD1
├── Sewer-ML
├── test.zip
└── test_dataloader

Inside the '10 Collimator' folder within VS_Research​

10 Collimator
├── AI Models from Deven
│   ├── soluble-particulate-t-InceptionTimePlus-01-11-23.pkl
│   ├── soluble-particulate-t-TSTPlus-01-11-23.pkl
│   └── soluble-particulate-t.pkl
├── ai-abstract-submission-instructions.pdf
├── BSM2 in collimator.docx
├── BSM2 in MATLAB.docx
├── BSM2_R2019b_working_version.zip
├── Collimator Q&A.docx
├── Lit Review
│   ├── 03.IWA.hydroinformatics.pdf
│   ├── Benchmark simulation model
│   │   ├── Benchmarking of control strategies for wastewater treatment.pdf
│   │   └── Jeppsson_et_al_2007-Benchmark_simulation_model_no_2.pdf
│   ├── BSM2 documents
│   │   ├── ASM_ADM_ASM interfaces for BSM2.pdf
│   │   ├── BSM2_agreement_LU.pdf
│   │   ├── BSM2_README.pdf
│   │   ├── Description_BSM2_20090101.pdf
│   │   ├── Rosen_&_Jeppsson_2006.pdf
│   │   ├── Rosen_et_al_054040011.pdf
│   │   └── Takacs et al_1991-A dynamic model of the clarification-thickening process.pdf
│   ├── Connecting a wastewater treatment plant to ML platform
│   │   ├── Wallin and Nordlander-2021-Challenges in connecting a wastewater treatment plant to a machine learning platform.pdf
│   │   └── Wallin and Nordlander_2021.docx
│   ├── Wastewater Treatment Plant Models.docx
│   └── WW treatment
│       ├── CLEAN Soil Air Water - 2011 - Khataee - Modeling of Biological Water and Wastewater Treatment Processes Using Artificial.pdf
│       └── Wallin and Nordlander-2021-Challenges in connecting a wastewater treatment plant to a machine learning platform.pdf
├── notebooks
│   ├── Compare_results_hybrid_model_with_physic_model.ipynb
│   └── Copy of Collimator to TDEngine.ipynb
└── output csv
    ├── Bioreactor1_AI_TSTPlus_BSM2_model_current_version_2023-01-13-12-03-50_for_1day.csv
    ├── BSM2_model_current_version_2022-12-08-17-36-13_for_2days.csv
    ├── collimator_cross_reference_2022-12-08.csv
    ├── matlab_output_2days.csv
    ├── matlab_output_cross_reference.csv
    └── old files
        ├── BSM2_model_current_version_2022-12-05-18-14-18_for1day.csv
        ├── BSM2_model_current_version_2022-12-07-12-30-00_for1day.csv
        ├── BSM2_model_current_version_2022-12-07-14-46-10_for2days.csv
        ├── BSM2_model_current_version_2022-12-08-15-58-43_for0_7day.csv
        ├── collimator_output.csv
        ├── collimator_output_column_name.csv
        ├── collimator_output_column_name_2022-12-05.csv
        ├── collimator_output_column_name_2022-12-07.csv
        └── matlab_output_primary_clarifier_2day.csv

Unionsine2 (10TB HDD)​

Directory structure 1 level down within unionsine2​

unionsine2
├── .HD3510icon.ico
├── cctv
├── msi-desktop-ubuntu
└── msi-server

Inside the CCTV folder​

cctv
├── cov
│   └── data
├── dnv
│   ├── data
│   └── db
└── sd1
    ├── .vscode
    ├── data
    ├── db
    └── test_label_images.py