MULTIMODAL EMOTION DETECTION UNDER MISSING MODALITY CONDITIONS

Abstract

Multimodal emotion detection systems commonly integrate visual, audio, and textu-al information to improve recognition accuracy in human–computer interaction and effec-tive computing applications. While such approaches perform well under controlled condi-tions, they typically assume the constant availability of all modalities during inference. In real-world deployments, however, one or more modalities may be unavailable due to tech-nical, environmental, or privacy-related constraints. This study investigates the robustness of multimodal emotion detection under missing modality conditions. A modular framework is proposed in which visual, audio, and text modalities are processed independently using transformer-based emotion detection models, and their outputs are combined through a late fusion strategy. The system is evaluated on the IEMOCAP dataset by simulating various missing modality scenarios during inference while keeping the training process unchanged. Experimental results show that the complete multimodal configuration achieves the highest performance, while partial modality setups exhibit a gradual and predictable performance degradation rather than abrupt failure. Relative performance retention analysis indicates that a large proportion of the full-system performance is preserved even when operating with reduced modality availability. These findings demonstrate that late fusion-based mul-timodal systems can provide reliable emotion detection in practical settings where complete multimodal input cannot always be guaranteed.