Week 4

1. See the Action Items for CCTV project here.
2. See the Action Items for DeepVibe project here.

ToDo

• CCTV GIS+Prediction: Porting the latest code to our git repo and testing - ongoing

  • First nb complete 2 and 3 are in progress
  • Add writing to shape files component following the previous lightGBM notebook.
  • Need to save a trained model as currently we only save the optimal hyper-parameters.

• CCTV-Apps documentation update

  • Docs should be refreshed to reflect the latest status of the apps
  • Draw a mermaid diagram for the dataflow for each app.

• Document the run details on the binary model training on 'I&I'

  • Now need to copy the training images from the corresponding locations to the MSI (either from Google Drive or downloaded VS_Research folder)
  • Proposed set of updates are present here

• Document pipe-breaks

November 30, 2023

According to the project page, there are three repos related to pipe-breaks project.There are uncommitted changes in the MSI machine on these repos.

1. pipe-breaks

   1. Data Files (CSV and Model) Updated by New Runs:

      • Modified: outputs/dnv_cox_ph_partial_hazards.csv
      • Modified: outputs/dnv_output_dataframe_baseline_survival.csv
      • Modified: outputs/dnv_output_dataframe_filtered.csv
      • Modified: pkl/cox-ph-dnv.pkl

   2. Config Changes:

      • Modified: requirements.txt

        • 3 Python packages have been removed from the requirements. Some are Windows-specific installations, which could potentially fail in Linux.

          • pywin32==227
          • pywinpty==2.0.5

        • Additional Info: requirements.txt seems to contain more Python packages than needed by 'pipe-breaks.' It appears that the requirements.txt file may have been created by exporting a list of all Python packages present in the system.

2. loe-coe-app_deven-fork (master)

   danger

   This repo is connected to deven-gqc/loe-coe-app and not under github.com/gqc.
   From the commit history it seems the practise was to make the changes in the fork under Deven's account and merge the changes to the main repo under github.com/gqc

   1. Documentation changes:
      There seems to be a configuration change done on the Sphinx documentation engine configuration.
      Also there are changes in the documentation as well.

      • modified: docs/source/conf.py
      • modified: docs/source/index.rst
      • modified: docs/source/pipe_breaks.common.rst
      • modified: docs/source/pipe_breaks.rst
      • docs/source/Notes/ (New file)
      • docs/source/pipe_breaks.tl.rst (New file)

   2. Simple modifications:

      • modified: pipe_breaks/common/processing.py

        Imported scipy but not used anywhere.

      • modified: pipe_breaks/constants_gqc.py

        Parameter changes. Not sure what are the implications

      • modified: settings.ini

        Looks like this was copied from cctv-apps repo and modified.

   3. Needs additional review:

      • nbs/ (new)

        There are two new notebooks which is seems to be intended for transfer learning by looking at the names.
        

      • modified: app.py

        A DB connector and a streamlit widget is imported
        A boolean switch is added to the logic. Have to see the impact to the logic.
        do_transfer_learning = False Added a new page. New page does not do anything functional. (See the next file)
        

      • modified: setup.py

        A custom read_requirements function is added. Not sure about the reason.

      • pipe_breaks/streamlit_helper.py (New file)

        Seems like a streamlit page that is yet to be completed

   • • 21-07-2023 10-42-14.sh (New file)

       Can be a temporary run script.

       ```bash
       #!/bin/bash
       #SBATCH --nodes=1
       #SBATCH --gpus-per-node=v100l:4
       #SBATCH --ntasks=1
       #SBATCH --cpus-per-task=24
       #SBATCH --account=def-blence
       #SBATCH --mail-type=ALL
       #SBATCH --mail-user=vannary@mail.ubc.ca
       nvidia-smi
       python script_runner.py -s xgbse_gqc -u dnv
       ```

     • pipe_breaks/tl/loader.py (New file)

       A helper file for the transfer learning nbs

     • reqs.txt (New file)

       A requirements txt. Not sure what the intension is. Not called from any of the configuration files.

     • reqs2.txt (New file)

       Another requirements txt. Not sure what the intension is. Not called from any of the configuration files.

3. pipe-breaks-transfer-learning (master)

   • Just a new trained model file, xgbse-gqc2-dnv.pkl, is present as an uncommitted change.