Skip to main content

Week 4

ToDo​

  1. See the Action Items for CCTV project here.
  2. See the Action Items for DeepVibe project here.

September 29, 2023​

  1. Reproduce Vannary's latest results. Verify it runs end-to-end.

    1. Changes to the 2nd notebook are grouped together to the notebook section, Add X,Y long and lat
    2. Confirmed the output CSV is identical. The order of the output labels were different. Hence, I added sort in my code to have consistent results.
    3. Ran the cleaned up training nb. It is currently being run. It seems to search across different combinations of hyper-parameters of each of the different model types.

  2. I checked Vannary's 3rd notebook in colab and it has new updates. And according to Vannary's email she will develop this further based on the Monday's meeting with Dr. Lence.

    1. If we want to discard all other models except lightGBM, I can modify the latest nb for that purpose. The older lightGBM only training notebook vs. the lightGBM training in the latest have few known changes in the input features. There might be more differences too.

  3. Document the Vannary's replies on the emails thread on the weighted losses.

    1. Find them here

September 28, 2023​

  1. AI3 meeting with Jian and post-meeting discussion.
    1. The cumulative notes and the suggestions that arose in the discussion will be published to AI3 Action Items and Q&A pages.
    2. I will review them and add any missing pieces once published.

September 27, 2023​

  1. Refactoring my daily notes to,

    1. Moved todo's related to projects to Action Items pages under projects rather than my notes. Check the todos above.

  2. Documenting Vannary's latest replies on CCTV and creating action items based on that.

    1. GIS + Prediction Q&A
    2. training Q&A

  3. Install QGIS in my local computer

    • Dowloaded the QGIS 3.32.3 installer from https://www.qgis.org/en/site/forusers/download.html
    • Checked with Jake to see if there are other configurations that is needed to use QGIS.
    • Compared the shape files from Vananry's saved data vs. the files generated by the version I've cleaned.
    • However, Vannary had shared a new notebook on training. Need to do the same experiment with that after a cleanup.

  4. Modifying and running the new training notebook for CCTV which Vannary shared for defect predictions. It includes binary Random Forest classification/xgboost/lightgbm/catboost trainers. I'm working on gqc-utility-notebooks repo in my local WSL.

    • This notebook does not write back to shape files.

    • List of issues that needed to be fixed:

      1. outdated functions from a previous version of scikit-learn python library. Instead of changing the code to use the modified api from the updated library, I opted to downgrade the library version to match with colab to avoid divergence in the code.

      2. There are pkl files containing the model parameters which are loaded from hard-coded paths inside functions. https://drive.google.com/drive/u/0/folders/1-FxAPvJhCmVO_7W_yzdhvhMDqGHTbUW4 model parameter pkl Made a copy locally and modified the functions to point to the pkl directory. Path is loaded through a GLOBAL variable now.

      3. Expects additional features for training. These columns were not exported from the training data preparation notebook version we have. features

        • Diff between the training csv column headers that were used in the latest training script vs. the local version I have.
        • csv diff
        • Based on this I checked the Vananry's colab notebook training data preparation, it has been updated after we made a copy. diff
          • As Vannary's colab notebook development is still an ongoing process, these same issues will continue to come into play. We need to decide how to go about this
Last updated on by Pavan