RAGE-Fusion: Reliability-Aware Multimodal Emotion Fusion for Real-Time Interactive Interfaces

Abstract

Emotional responsive interactive interfaces can enhance the user experience; they can change the response and presentation depending on the affective condition of the user. But implementing these systems to practical application is still difficult since emotion cues are multimodal, noisy, and frequently absent (e.g. webcam turned off, poor audio, occlusions) and interactive systems also demand low latency and predictive behaviour so as to keep the user trusting them. The paper is about RAGE-Fusion, which is a reliability-conscious multimodal deep learning system used in emotion recognition and interface adaptation, which models text, audio, and visual information together. RAGE-Fusion is a cross-modal attention and pretrained modality encoder architecture that integrates the complementary affective information and a reliability-gated fusion mechanism, elaborating on the weighting of each modality in the case of missing or corrupted input based on an inferred quality improvement in robustness. In order to fit affect recognition to interactive limitations, we also introduce a multi-objective optimization plan, balancing the performance of emotion prediction, the inference latency, and prediction temporal consistency between conversational turns. Simulations of the MELD benchmark show that it steadily outperforms unimodal and baseline fusion baselines especially when there is modality drop and noise. Calibration and stability analysis are reported by us as well to facilitate a safe interface adaptation decision. The findings reveal that reliability-conscious fusion and interaction-based optimization are a viable basis in development of robust and real-time emotion-conscious interfaces.