Week 4

Sept 18th 2023

Jake fixed MSI issue - It works now
TBD for 09/28 - Update questions in Projects site (Questions in physical notebook)
Email Vannary about Google colab <-> GitHub syncing of notebooks.
- Vannary sent a document with instructions on how to do it.
- Need to test it out and update in general site.
Create AI3 tasks on JIRA, from my daily tasks and add locations of data / code in decsription.
QGIS - Cleanup and update notebooks on colab (Prepare data for binary prediction, lightGBM), Update documentation accordingly.
- Adding overview flowcharts of notebooks

AI3 - Training data prep using sample-for-training notebook editing
- Modified standardizing functions in utils to handle nested JSONs.
- Completed test run on all PAICP data (fire, EQ, weather) and saved training data as llama2-finetune-training-data.db.
- Committed changes on MSI in ai3-hackathon to git.
CCTV QGIS
- Adding comments to CCTV GIS + Prediction: 02 prepare_data_for_defect_prediction_version_2.ipynb
- Updated flowcharts accordingly and verified it has Vannary's changes.
- Mailed Vannary with questions regarding notebook 02.
Test repo for github - collab

AI3
- Cleaned documentation about data and flowcharts.
- Looking into finetuning LLAMA for already fintuned model.
CCTV QGIS
- Vannary replied with answers to yesterday's questions.
- Documented them under questions section under CCTV > QGIS
- TODO Draw a high level understanding of the CCTV DNV flow - from what we are getting, what features rae passed to LightGBM and the output
Documented about mounting drives on general site under Dev > linux.

Worked on cleaning AI3 finetuning-trained-LLAMA.ipynb notebook.
Discussed on CCTV QGIS features, training, prediction and understanding the output.

Worked on cleaning AI3 finetuning-trained-LLAMA.ipynb notebook.
Started a training job on 600 samples from PSIAP, that will run over the weekend.

danger

Observed that after training, model.save_model() will save the files in the same directory it read from, which means there is overwriting of old files. So, specifying model.save_model(out_path) should most likely resolve the issue.
Will verify this once the run is finished.