Week 5

Sept 25th 2023

• Testing and documenting findings on the completed AI3 training job.
• Results from testing finetuned LLAMA2:
  • Got good results on EQ data, but not for fire data and weather data. The input and output samples are documented here: https://projects.gqc.com/internal/Hackathon/AI3/llama#results

Sept 26th 2023

• Document AI3 Phase 3 progress as a slide deck (Sent in the mail and located on local in phase_3 folder)
• Update documentation on AI3 as part of the process.
• Conversation with SK and PS about AI3 summary PPT - Multi-model approach usage has to be mentioned and explained.
• Notes for meeting on Thursday with AI3 team
  1. Start with outstanding questions (Go through them once again)
  2. Mention that we are using LLMs for common sense understanding of the data and that it is working well.
  3. We need to know how they are going to measure the accuracy using NK keys
     1. Is there a mechanism of using unique identifier for JSON objects, that will help in picking the objects with NK keys from entire JSON message and comparing that object (with actual value before NK) with predicted object (with NK replaced by a standard key)?
     2. Or are they just going to compare and find the difference between the actual JSON and Predicted JSON as a whole, without comparing each object? This means if there is any difference, then our prediction of NK keys is wrong.
  4. Feedback on our report

Sept 27th 2023

• Update AI3-Hackathon-Summary PPT with comments from yesterday.
• Sent the Progress-Summary-and-Questions PPT slide with questions and updates in mail.
• YOLO - Looking into improving the results
  • Need to label the images to train it on using tools like labelme.
  • [SK] Ask if we want to train YOLO using disaster city videos by extracting frames using ffmpeg and then label all those frames.
    • Update - Sudhir said to roll forward with this. Will be updating if we find anything else.

Sept 28th 2023

• Meeting with Jian Tao - Asked the questions from Progress-Summary-and-Questions PPT and documented all the notes and answers on projects sites under both Questions and TODO pages.
• Explored on tools that can be used for YOLO training data.
• Discussed with Sudhir about the meeting and documented more questions to ask and some more notes.

Sept 29th 2023

• YOLO finetuning research

  • List of resources found:

    1. https://medium.com/@yongsun.yoon/fine-tuning-yolov8-using-custom-dataset-generated-by-open-world-object-detector-5724e267645d
    2. https://learnopencv.com/train-yolov8-on-custom-dataset/
    3. https://encord.com/blog/automate-video-annotation-guide/#:~:text=In%20automated%20video%20annotation%20or,and%20end%20of%20a%20video.
    4. https://youtu.be/DMRlOWfRBKU?si=OKf95AbxtJ9Y5Aq3

  • Approach to fine-tuning:

    1. Download videos from the GDrive for PSIAP

    2. Extract frames from the videos using ffmpeg

    3. Annotate the frames with objects of intrest using any tool

       1. Manually using tools like LabelMe
       2. Automated using tools like Grounding DINO

    4. Save the annotations in YOLO supported formats, like JSON files for each frame.

    5. Prepare a YAML file with paths to above data and parameters like epochs etc.

    6. Run the command for training YOLO and provide data=abc.yaml in the command. Example is below:

         yolo task=detect mode=train model=...pt data=abc.yaml epochs=50 imgsz=640

    7. The trained model is saved in the runs/detect/train/weights folder. We will use the best.pt model.

• Updating AI3 questions from yesterday discussion with Sudhir.

• Get updates on all AI3 projects - Monday.