Week 3

Oct 16th 2023

EBMUD
- Create script to run model predictions on batch of video frames
- Create CCTV project on JIRA (Jake did)
- Create an epic in JIRA for ebmud and add tasks
- Modify settings.py and code to make it run
- Committed the code to github and updated JIRA
Analyze sentimental analysis for AI3 and prompts we can provide.
Documentation
- No usage of GPU in the streamlit cctv-apps/utils/model_inference() function
- Monkey patching in the code present doesn't let the VSCode IntelliSense detect Patched functions and their definitions.
- Compute-MSI docs. Mostly README
  - Scripts with generic in name suggest Single or multi-gpu models usage

Modified Vannary's abstract for sewer paper and sent to Sudhir and Pavan. Included all the details discussed.
Pipe-breaks
- Commit the changes in HP windows to a dev branch. (created dev branch and pushed changes)
- Create survival curve for cox-ph using dnv or cov data.
  - Research requirements, files and data needed to get a survival curve
  - The input for survival curve needs to contain fname, probability of survival, time
  - Time period will be study_end_date + tets_end_date
  - Test to see if we can use Power BI or Excel for graphs
- Change display in console - top % value to be not 0.456 but 45.6%
- Might have to generate separate survival curve for broken pipes in test period

EBMUD
- Extract group3 frames and generate predictions using git/cctv-apps/AI-batch-prediction.py. This generates 2 CSvs - with and without threshold. We focus on the CSV after threshold.
- Generate CSV using pivot tables with columns - names, labels, Value
  - Using the threshold CSV that is generated above, we un-pivot columns other than fname and filter out the rows with Value 0.
  - Then we are left with frames with their defects (with value as 1).
  - We copy these rows after picking out ND and other labels of no interest and save as a CSV with suffix unpivoted
  - Copy that into cctv-apps/csv and modify the names column to add prefix as path to frames
  - For more information on how to do this, we followed 1st approach here: indeed.com
  - HACK: To quickly modify multiple rows with same text in VSCode, we can use file search along with regular expression option available in there
- Verify that we can retrieve defects querying based on defect labels - did through MS Excel
- Need to verify if the defects are actually present in the frames of the videos
- Sent Sudhir a PPT with snapshots from streamlit with predictions, for videos 11, 30, and 35.
CCTV
- Add PACP dictionary / CSV into cctv-apps
- (low) A streamlit page for AI-batch-prediction

EBMUD
- Modified the PPT with streamlit images to add all of them at threshold 0.1, with label description as title
- Sent to Sudhir
AI3
- Sent the AI3 latest set of questions. Will be updating on projects site too.
- Updated checklist on projects site for Sudhir to verify in the final round. Available under New computer setup section.
- Made synonym primary key in ai3-lookup-09132023
- Create a new DB called ai3_lookup_db with phrase, synonym (primary key), mean, phrase_count columns
- Remove - and _ in the keys
- Run mean generation script on old data and insert into ai3_lookup_db DB
- Verify what keys are missing in ai3_lookup_db DB and old DB
- Import ai3-lookup-09132023 into the ai3_lookup_db
- Verify all contents and test against data in subscriber.

AI3: mean creation
- create copy of ai3-lookup-09132023.db
- add phrase_count column in the DB
- add code to append new keys
- call it in mean gen notebook
- check if new keys are appended
- generate mean of new keys
- new_mean = old_mean * count, then increase count (if the value is not null)
- exclude team_frites folders for mean generation
- Add the pending tasks on JIRA
Issues in mean calculation:
- Root key adding doesn't work for keys with string values
- Root key refreshing when we switch from one nested dictionary to another isn't happening. The first root value is remaining constant when it has to be replaced by other root values.
- Check the DB for better understanding