Week 1

Apr 04/07/23

1. Discussion related to RFI
   1. Introduction of delay in time-series forecasting models
   2. GQC has expertise in physics based model using delays, etc
2. Keywords from Melissa
   1. Using those keywords to get papers out
   2. The script is running and we are getting all the papers out.

Concept of lag in time-series forecasting

info

Prompt: how would I introduce the concept of delay in Transformer model used for timeseries forecasting?

1. Response from ChatGPT:
   1. To introduce the concept of delay in a Transformer model used for timeseries forecasting, you can explain the following:
   2. In timeseries forecasting, the goal is to predict the value of a variable at some future time based on its past values. To achieve this, we need to feed the model a sequence of past values and the model learns to predict the next value in the sequence.
   3. However, when dealing with timeseries data, there is often a time delay between the past values and the future values we want to predict. For example, if we want to predict the temperature at noon tomorrow, we need to look at the temperature values from the past few hours, not just the temperature values at midnight.
   4. In a Transformer model, we can introduce the concept of delay by adding time embeddings to the input sequence. Time embeddings are vectors that represent the time interval between the current input value and the target value. By including time embeddings, the model can learn to capture the time delay and make accurate predictions.
   5. Additionally, we can also include lagged variables in the input sequence, which means including past values of the variable we want to predict as input features. This can further help the model learn the relationship between past and future values and improve the accuracy of the predictions.
   6. Overall, introducing the concept of delay in a Transformer model for timeseries forecasting involves incorporating time embeddings and lagged variables into the input sequence to capture the time delay and improve prediction accuracy.
2. Other hits from different libraries:
   1. darts
   2. tsai
   3. neuralforecast
   4. hierarchicalforecast
   5. statsforecast

Apr 04/06/23

1. CSDP
   1. Asset migration to the hydrotrek account
2. cctv-apps
   1. New release -- v0.0.1
   2. Bump version
   3. New CLI functionality
3. minimal-fastapi-template
   1. Made a minimal working template with fastapi
   2. Has a fake-data endpoint which returns fake-data
   3. The data is successfully received and seen on the frontend

nbdev migration

There are over 150 notebooks which need to put into version control from Google Colab. I have started a list of notebooks for which nb should go where.

Apr 04/05/23

Progress on the model training front

1. Tried to do the pre-training with the JSON data using the GPT-neo (2.7B model)
   1. The model is too large to fit on a Colab GPU
   2. Needs atleast 25GB RAM
   3. Possible alternatives
      1. Using distilled models
      2. Smaller versions of GPT-neo (1.3B)
      3. GPT-neo (125M)

Update on the RFI

GQC would be involved in →

1. software technologies or machine learning based on models used to implement sewer flow forecasting in operations, including scalability, level of effort to retrain models
2. Examples of successful or machine learning based models in conveyance and storage operations for sewer systems
3. Operational System Integration

Data acquisition and ML/forecasting inputs would be given by Ayyeka. So, we would be interacting with Ayyeka a lot.

Further clarification sent to Jake

I think GQC can contribute on the forecasting front as we have experience with recursive forecasting models.

In the Streamflow forecasting project, GQC created LSTM/Bi-LSTM based neural network models which could forecast n timesteps in the future. Besides that recently, GQC has also adapted to recent changes by utilizing models built with the Transformer architecture for forecasting.

On the sewer systems side, we have the pipe breaks app, which has a UI allowing us to choose from a variety of algorithms of survival analysis algorithms. The UI shows us survival curves and has the functionality to store the runs in a database.

Apr 04/04/23

Review the steps for the AI-3 hackathon

1. Deadline photo
   
2. Where did we end things?
   1. Chatbot creation
      1. Some exploration with langchain -- allows us to take a Language Model and then be turned into a ChatBot
      2. Can be potentially fine-tuned to make a ChatGPT as such for Disaster Response situations.
   2. Ingestion and fusion of IoT Data streams
      1. Needs work on the data standardization pipeline
      2. Need of dummy/mock JSON data to see how we would go about training the GPT-neo model
   3. Situational awareness
      1. Maybe explore the Twitter API again, get some tweets out
      2. Use the extracted text for sentiment analysis -- geo-coordinate extraction
      3. Derived knowledge analysis -- we have some reference links and papers for this already
3. What is new?
   1. Datasets made available by usnistgov. Can be found → here
   2. The dataset has Flood folder which contains data pushed by Sudhir.
   3. I looked at some GeoJSON files in the datasets, besides that I'm not sure where or should we proceed

Addressing the feedback from the judges

1. Addressing concern about data reliability from extracted tweets
   1. We would have a classifier which identify whether a tweet is made by a bot/fake.
   2. It is quite common that in cases of disaster situations, when place "X" is affected by a disaster, people all over the world write about it.
   3. This requires us to distinguish between the tweets made by people who are actually affected by the disaster vs the people who aren't.
   4. We wish to address this problem by using the additional metadata from the tweets.
      1. By verifying the geo-coordinates from the tweet against the coordinates of the actual disaster, we would be able to distinguish between the tweets.
   5. We are also planning to have rank (priority/severity) determining mechanism which would also us to filter tweets which need immediate/prompt attention. This model would depend on various factors like location, language, type, retweets, etc

Apr 04/03/23

Instructions for Vannary to run the app

1. Pull the latest code changes from GitHub
2. In the sidebar, go the Visualizer-2 app
3. Upload the csv for the images you wish to see in the carousel
4. Select the x,y,w,h accordingly to have the red rectangle align correctly with the distance in the frame.
5. After you have coordinates according to your liking, click on Insert to DB.