Attention LSTMs in Multimodal Models: A holistic approach to predicting COVID infection trends

In our search for a way to simultaneously predict all state-level COVID infection rates in the United States with COVID heat maps and domestic flight graphs, we propose novel methods of processing graph and image sequences with attention-based LSTM layers as well as evaluate the effectiveness of different multimodal fusion techniques.

COVID-19 and its infection trends are a hot topic in the AI space due to the seeming applicability of AI, which could help with case preventions and medical resource allocations. However, too often people focus on a predetermined area and scale, overlooking the fact that COVID doesn’t flow into or out of just one area but instead spreads everywhere simultaneously. Thus, we propose novel approaches to process the infection rates of all states in the United States as a whole while taking into account the interconnectivity between states through domestic flights. We introduce 1) the graph attention LSTM for analyzing domestic flights across states, 2) the image attention LSTM for understanding infection rates from United States COVID heat maps, and 3) the attention bottleneck mid fusion model that integrates graph and image attention LSTMs. Through tests that require simultaneous predictions of the infection rates in all US states, we compare the performances of our methods with those of classic sequence-processing methods and achieve significantly more accurate results.

Explore More!

Research Paper

Source Code

Project Webpage