Week 3

Sept 11th 2023

• Test run finetune-llama2.ipynb on google collab using old DB data it was finetuned on.
  • Ran using T4 GPU for 1 epoch on 200 sample data from openai-db-train-2k-samples.db with batch size 6 and 16-bit floating point.
  • Issues faced - Mostly it was import issues, whcih were resolved using pip install. Other issue is the versions of libraries supported based on the GPU selected.
• Prepare weather data to fine tune llama2 on.
  • Test BARD token works or not for labelling the JSON data - doesn't work
  • Run sample-for-training.ipynb on the weather data JSON using BARD or OpenAI - Used OpenAI for 300 messages - data needs to be saved
• Update docs
  • to include about google colab - a100, v100, t4 gpus
  • AI3 docs - BARD > cuda vs meta usage with config.init_device
  • Document that a notebook that works in local environment, with libraries and dependencies installed for that hardware, might not work in google colab for example, because of the pre-installed versions of some libraries and the dependencies not supported for those library versions.

Sept 12th 2023

• Run sample-for-training.ipynb on weather data(300 rows), fire data(100 rows), and EQ data(100 rows).

  • The data is saved and uploaded to GDrive as llama2-train-test-samples.db with 500 rows in total

• Modify finetune_llama.ipynb to finetune trained model present on MSI.

  • Existing notebook loads LLAMA model from HF to finetune. Whereas we want to finetune the model that we already trained on NIST data, available on MSI.
  • created new colab n/b for this : finetune-trained-llama.ipynb

Sept 13th 2023

• Clean data - Standardize, sentencify and then label to create training DB.

  • Was able to calculate mean for EQ and weather correctly, need to work on fire data since it's nested.

  • Created a new DB ai3-lookup-09132023.db with all keys and phrases from new data PSIAP_AI3

  • Mean calculation:

    1. For nested JSONs, we reach the leaf dictionary and maintain a new dictionary 'temp_df' with the key:value pairs of format:

       key: {'sum': sum of previous values + new value, 'count': count of previous values + 1}

    2. For each such leaf node, we keep adding values and maintain as list mentioned above.

    3. Once done parsing that JSON message, we calculate mean of each key in the 'temp_df' dictionary as

       mean[key] = temp_df[key][sum] / temp_df[key][count]

    4. For keys with string values, we calculate the mode i.e. most frequent string. So, we might have to append all unique values to the dictionary and then calculate the mode. (on-hold right now)

    5. Exception: We need to find a way to handle unix times in the JSON, as it's hard to calculate mean for them. It throws exception if we are generating mean using df.describe, which excludes columns like dates. And if we pass NK with such dates (with current logic of swapping NK to test, the unix dates are also considered and replaced with NK).

• Tested running subscriber.py on new database ai3-lookup-09132023.db with subscriber (The DB at this point has None for dates and time keys along with fire keys which had only strings)

• Implemented Levenshtein distance calculation for Standardization. Moved the code to utils and updated subscriber.py accordingly.

• TODO

  • Need to merge old lookup DB with new lookup DB.
  • Need to handle nested JSON such that only keys with numeric values are added
  • Fix issue with Subscriber.py exception while standardizing NK keys with Unix dates, since publisher logic allows NK to have unix dates. Need to discuss on handling this.
  • Exception: Need to create mean column as type Numeric with default as NULL, and not None. Must check this while importing csv into DB.
  • Remove all_keys table and update flowchart with the same
  • OpenAI responses (3.5-turbo) are sometimes cut short and give only short sentences as replies (tracking on JIRA)

Sept 14th 2023

• Working on modifying sample-to-train.ipynb to handle nested JSON and create training DB for llama2.
  • Modified it to include standardization. Currently, it works for non-nested JSONs.
  • Had to halt since MSI doesn't turn on.
• Q_GIS - find files to discuss in the meeting (csv or shp files)
• Look at the last status of pipe-breaks before we switched to AI3.
• Work on documentation for AI3 and any thing else that's pending.

Sept 15th 2023

• MSI doesn't turn on - AI3 on halt.
• Continue working on AI3 documentation.
• Meeting on Q-GIS and LightGBM.
  • Updated the notes and docs on Projects > Internal projects > CCTV > GIS + Predictive model
• List the documents uncommitted on MSI and roadmap on what can be or can't be done for 2 weeks (Assuming MSI won't be available till then) - Done through email