A Late Fusion Approach with Multimodal Image-Text Data

This project explores late fusion methods in multimodal machine learning to improve emotion detection by combining image and text predictions, and evaluates their effectiveness compared to other fusion approaches using the MVSA-Single dataset.

As the number of users on social media platforms continues to rise, an increasing number of people are expressing their emotions and opinions. Before the popularity of multimodal machine learning, previous works included identifying texts and images separately to determine sentiment. However, these studies often overlooked features that other modalities could capture that are crucial to emotion. Recent advances in multimodal machine learning are now making essential and precise judgments in image recognition [1]. In this project, we explore a specific type of fusion called late fusion, demonstrating its ability to combine different modalities. In particular, we highlight two late fusion methods that combine the predictions of the image and text models. Additionally, we investigate a contrastive learning approach, which will help improve the visual embeddings, making them more discriminative in feature space. We compare the strengths and limitations of late fusion against other fusion approaches. Our findings are based on experiments conducted with the publicly available MVSA-Single dataset.

Explore More!

Research Paper