Week 1

Nov 11/01/2022

1. Major updates in the DNV streamlit app
   1. Only thing left to implement is the sharper logic for showing un-blurred images
2. Summarize work on the multi-variate time series front.
   1. Next steps
   2. Challenges
3. Push the work on the Elsevier API front again.

Nov 11/02/2022

Links explored for Azure Cognitive Series

1. How to apply Azure OCR API with Request library on local images?
2. How to send a local image instead of URL to Microsoft Cognitive Vision API(analyze an image) using Python?
3. Azure-Web-sites: How to cancel a deployment?

Nov 11/03/2022

Things to do for today

• See how Dr. Vered Shwartz does her work in common sense reasoning
  • Made a list of a few papers. Quite difficult for me to understand at this stage.
• Call Azure guy for support on Azure Cognitive Services
  • Made a call at 11:35 am EST (Nov 3, 22) -- call back in an hour -- needs time to go through the code
• Find the thesis of the nixtla guy -- any interesting work related to time series forecasting
  • Reached out to the guy on Slack -- waiting for his response
  • He has not graduated yet
• Visualize and plot results from the Prophet model in the notebook.
• See how to create custom dataloaders in pytorch lightning
• Register for the webinar
  • Watch the webinar
• Try to think and design a proposal for the webinar
• Revert to RoboFlow guy (Salo)
• Get Pavan setup with nbdev
• Discuss the revised pipeline for CCTV with Pavan and Sudhir.

Nov 11/04/2022

Tasklist

• SOCIAL CHEMISTRY 101: Learning to Reason about Social and Moral Norms
  • Summary for this paper can be found.
• Unsupervised Commonsense Question Answering with Self-Talk

Nov 11/07/2022

• Summary document for AI3 hackathon.
  • Sent to Sudhir and Steve.

Segmentation in a live video stream

• We have used pixellib in the past to do segmentation in a video.
• But only included pretrained models from the library.
• Trying our custom fastai model for binary flood segmentation does not work.
• A GitHub issue is opened here.

Other tasks for the day

• code review with Pavan
• EarlyStoppingCallback for Vannary → idea was discussed briefly.
  • She might need help with implementation.

Nov 11/08/2022

Tasks at hand include

• Update document on timeseries forecasting.
• Next steps for AI3 hackathon
• Next steps for all things NLP

Quick discussion with Sudhir

• Major focus is on earthquake this time.
• How will we capture (collect) training data?
  • Probably some government websites -- build a LightGBM model for predicting the grid having a earthquake.
• Take a look at the Ramesh Jain book and see how is knowledge extracted.

Nov 11/09/2022

Project	Task	Update Status
Time series forecasting	Multi variate multistep forecasting -- (TFT implementation from nixtla/neuralforecast)	An example from neuralforecast was tried. Need some help with creating the custom dataloader in PyTorch Lightning
NLP	Elsevier API	Some response from the Elsevier support people. Vannary would have to go to the campus to test out the code.
AI3 hackathon	Real time image segmentation	Test implementation of pixellib was tried. Did not work. Issue was created.
	Common sense reasoning
	Decision making
	Situational awareness
	Some litt review from Ramesh Jain
CCTV research and new directions	Increasing validation loss in the water level classification model	EarlyStopping – not yet implemented
	Accelerate and SLURM	Still no reply
	Utility for finding optimal batchsize	Put on hold

For question answering, semantic search and common sense reasoning

• Keep deepset and haystack in mind.
• Check models on huggingface for question answering
• Training the model on disaster response text dataset.
  • disaster_response_messages · Datasets at Hugging Face
  • Multilingual Disaster Response Messages | Kaggle
  • Machine Learning Datasets | Papers With Code
• Training the model on disaster response text/image dataset.
  • firojalam/multimodal_social_media: multimodal social media content (text, image) classification (github.com)
  • burakekim/MTL_homoscedastic_SRB: This repository contains the code for the paper "A MULTI-TASK DEEP LEARNING FRAMEWORK FOR BUILDING FOOTPRINT SEGMENTATION" (github.com)

Discussion with Sudhir

1. If we are planning to have real time video segmentation,
   1. Think of a wifi mesh,
   2. Direct inference from Jetson Nano, mounted on the first responders helmet
2. Revisit the work done on sentimarket
   1. Reuse the same workflow to mine tweets related to disasters and other situations.
   2. Find tweets made in California during a certain natural calamity

Nov 11/10/2022

From the source code of pixellib -- how to do video segmentation?

1. Will probably need Pavan's help on this one.
2. Capture each and every frame from the video
3. Segment each and every frame
4. Stitch every frame together, either with ffmpeg or opencv and write the output to disk.

Nov 11/11/2022

Holiday