Week 2

Aug 7th 2023

• Completing tasks from Friday discussion on pipe-breaks

• add notes on msi and hp env in project.docs.com

• Document dependencies adding in requirements.txt

• Vannary's email

  • format of models in 2 formats (.sav and .pkl) - why?

      (Van) I initially saved the model with .sav, as this is what I learned from my class. Deven, later on, changed it to .pkl

  • Requirements on compute-canada - pip freeze

      (Van) For pipe-breaks, we used the 3.7 Python version. Please see the attached text file for lists of libraries used to train the model.
        1. requirements.txt is a file I used to do pip install -r requirements.txt
        2. list_of_packages.txt is a file containing all the packages installed in the environment that I used to train the pipe break models.

• check the format of dates in tables

  • The input & output format of csv in pipe-breaks seems to be year/month/day concatenated.

Aug 8th 2023

• Find diff btw xgbse_gqc and xgbse_gqc_2
• Things done on MSI:
  • Created pipe-breaks-transfer-learning and installed all requirements from transfer learning repo
  • Changes form HP transfer learning were pushed and pulled into MSI.
  • Streamlit works with models generated by script_runner.py in pipe-breaks.
• Pushed documentation on pipe-breaks to projects docs site.
• Debugging XGBSE algorithm on pipe-breaks repository.
• Updating Docs
  • Document that we should not edit auto-generated docs or scripts.
  • Document about word2md.com

Aug 9th 2023

• Working on documentation for workflow and deploying comment extractor.

• Debugging XGBSE algorithm on pipe-breaks repository.

  • Fix below warning while running xgbse_gqc_2 on pipe-breaks repository.

      C:\git\gqc\ai\pipe_breaks\common\processing.py:592: SettingWithCopyWarning:
        A value is trying to be set on a copy of a slice from a DataFrame.
        Try using .loc[row_indexer,col_indexer] = value instead

Aug 10th 2023

• Jake renamed old gqc-utility-python repo to gqc-utility-notebooks and created new repo gqc-utility-python.
• All the docs sites have been updated with the latest links.
• gqc-utility-python has comment_extractor.py in it.
  • gqc-utility-python needs to have docs using Sphinx
• pipe-breaks
  • Change pipe-breaks data from .csv to SQLite (later)

Aug 11th 2023

• Understand why histograms are displayed in a particular way after running compare-results.py
  • Because of the bin interval being large. Here is the same plot with different bin intervals.
• Revert inputs / outputs file changes before pushing code changes to git for pipe-breaks
• Generalize hardcoded text like DNV, XGBSE in pipe-breaks > training and prediction with {utility} and {algorithm}
• Make corresponding code changes to loe-coe-app and resolve the errors
• Sync environments in pipe-breaks, loe-coe-app and pipe-breaks-transfer-learning.
• (low) Look into Streamlit display of plots and graphs, such that they are user-interaction friendly like React graphs.