Data Mining and Uncertainty-Aware with Missing Modalities for Multimodal Sentiment Analysis

1  Introduction

Multimodal Sentiment Analysis (MSA) aims to understand and integrate sentiment cues conveyed through multiple modalities. It has attracted considerable attention in recent studies [1–3]. However, in real-world applications, missing modalities often occur due to factors such as device malfunction, data corruption, or privacy restrictions, which can substantially degrade the performance of MSA models [4,5]. Current methods for missing-modality multimodal sentiment analysis are mainly divided into two categories: reconstruction-based recovery methods and robustness-enhanced methods. On the one hand, reconstruction-based approaches adopt feature completion techniques to restore missing modal information. For instance, TFRNet [6] leverages the Transformer architecture to repair incomplete multimodal sequences, while MPLMM [7] integrates accelerated learning strategies and prompt learning mechanisms to generate missing features for sentiment analysis and emotion recognition tasks. However, such explicit reconstruction inevitably introduces extra noise and considerable computational overhead, which greatly limits the practical deployment of relevant models. On the other hand, robustness-oriented methods avoid explicit reconstruction. For example, LNLN [8] takes text as the dominant modality and uses adversarial learning to enhance feature robustness. TF-Mamba [9] further proposes a text-enhanced Mamba framework to align complete and missing modalities with linear complexity. Overall, model enhancement paradigms reduce computation and reconstruction errors compared with recovery-based approaches. However, such robustness-enhancement methods still suffer from inherent limitations: they rely on a fixed dominant modality to resist noise and fail to adjust the contribution weights of individual modalities adaptively.

Despite their promising results, parts of existing methods fail to fully exploit the latent valid information in incomplete modalities due to the “lazy exploration” inherent in these models. This insufficient attention to critical sentiment cues can cause the model to favor incorrect categories, ultimately degrading predictive performance. Moreover, parts of existing methods may overlook the varying contributions across modalities to sentiment understanding. As illustrated on the left side of Fig. 1, the misleading information in certain modalities may confuse the sentiment cues from other modalities, leading to reduced accuracy and robustness in sentiment recognition. For modality contribution weighting, several approaches have been proposed, including sentiment score–based methods [10] and prediction outcome–based methods [11]. However, score-based models typically demand considerable computational resources for training and are highly sensitive to modality-specific data variations. As a result, they often struggle to generalize across different datasets and tend to exhibit unstable or unreliable performance, as illustrated on the right side of Fig. 1.

Figure 1: Problem description. The left side explains how modality dominance changes under low-quality sample conditions. The right side explains how missing data leads to unreliable model predictions.

Therefore, different from existing methods [7–11] and to thoroughly explore the information from the remaining modalities and balance the contribution of each modality during sentiment analysis under missing modality conditions, we propose the DUDF-MSA (Data Mining and Uncertainty-Aware with Missing Modalities for Multimodal Sentiment Analysis) method. Specifically, in DUDF-MSA, we first process the incomplete multimodal features Xts, Xas, and Xvs using modality-specific encoders. We adopt Bidirectional Encoder Representations from Transformers (BERT) to encode the text modality and Transformer encoders for audio and video modalities, yielding the corresponding feature representations Et, Ea, and Ev. To alleviate the “lazy” behavior exhibited in some recent models and enhance data mining from missing modalities, we propose the Common Feature Extraction (CFE) module, which learns common cross-modal features Fsh to focus on critical sentiment cues. Furthermore, we employ sentiment intensity-guided contrastive learning to capture fine-grained sentiment patterns.

Meanwhile, different with methods [8–11], which overlook the varying contributions across modality to sentiment understanding, or score-based methods, that are sensitive to modality-specific data variations, we design the Specific Feature Uncertainty-Aware Dynamic Adjustment (SFUA) scheme. It extracts modality-specific features and estimates each modality’s contribution by computing class uncertainty from feature probability distributions. That is, a lower energy score implies clearer sentiment orientation and higher adaptive weight, which reduces the disturbance caused by ambiguous cues from unreliable modalities during feature aggregation.

Finally, the common features Fsh from the CFE module and the adjusted modality-specific features from the SFUA scheme are jointly utilized to generate the hierarchical sentiment predictions at common, specific, and final levels. The auxiliary predictions at both the common and specific levels help optimize the performance of the corresponding modules. Experiments on three public datasets validate the superior performance of our method in MSA with missing data settings. The contributions of this work are summarized as follows:

1.   We introduce the Common Feature Extraction (CFE) mechanism to mitigate the model’s “lazy” behavior by thoroughly exploring the potential sentiment information within each modality with missing data.

2.   We propose a Specific Feature Uncertainty-Aware Dynamic Adjustment (SFUA) scheme that dynamically adjusts the weights of modality-specific features by quantifying each modality’s class uncertainty from its probability distribution.

3.   Experimental results on public English and Chinese datasets demonstrate that our DUDF-MSA achieves better classification accuracy and stronger robustness under various modality missing rates.

2  Related Work

2.1 Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) combines multiple modalities of information to enhance sentiment understanding. Current research on MSA [12–15] focuses on developing advanced fusion strategies and interaction mechanisms for better sentiment prediction accuracy. For instance, Han et al. [16] enhanced multimodal fusion by maximizing mutual information between unimodal pairs in a hierarchical manner. Zhang et al. [17] guided representation learning using sentiment-intensive cues. Despite progress, most studies assume full availability of all modalities, a condition that often fails due to missing data.

Recent studies [6–8,18] address the issue of missing modalities by learning joint multimodal representations or reconstructing incomplete information. For example, Li et al. [18] decompose modalities into sentiment-related and modality-specific components, reconstructing sentiment semantics. Zhang et al. [8] proposed a Transformer-based, language-dominant, noise-robust network to prioritize text features. However, these methods overlook the fact that the contributions of different modalities may vary under missing modality conditions and may introduce additional overhead due to reconstruction requirements. Therefore, we adopt a decision-level late fusion strategy. To fully exploit the sentiment information across modalities, we introduce the Common Feature Extraction (CFE) module to guide the model’s attention toward crucial sentiment cues in incomplete modalities by learning cross-modal common features. In addition, we design a Specific Feature Uncertainty-aware Dynamic Adjustment (SFUA) scheme to evaluate the contributions of different modalities and to alleviate the negative interference caused by confusing information from unreliable modalities.

2.2 Uncertainty Estimation and Multimodal Fusion

Multimodal fusion is a core theme in multimodal learning, focused on integrating modality features into joint representations for downstream tasks. It can be classified into early, intermediate, and late fusion. While neuroscience and machine learning studies suggest intermediate fusion may enhance representation learning [19,20], late fusion remains the most widely used approach due to its simplicity in interpretation and practical implementation.

However, the reliability of current fusion methods remains significantly unexplored, limiting their application in safety-critical fields such as financial risk assessment and medical diagnosis. The motivation behind uncertainty estimation is to indicate whether the predictions made by a machine learning model are prone to error. Numerous uncertainty estimation methods have been proposed over the past decades, including Bayesian Neural Networks (BNNs) and their variants, deep ensembles, predictive confidence, Dempster-Shafer Theory, and energy scores [21]. Predictive confidence typically refers to the consistency between the predicted class probabilities and empirical accuracy, often mentioned in classification tasks. The Dempster-Shafer Theory (DST) is an extension of Bayesian theory to subjective probabilities, providing a general framework for modeling cognitive uncertainty. Energy scores are a promising method for capturing out-of-distribution (OOD) uncertainty, which arises when a machine learning model encounters inputs that differ from its training data, leading to unreliable predictions. Recent studies have extensively addressed the OOD uncertainty problem [22]. Uncertainty-based multimodal fusion methods have demonstrated significant advantages across various tasks, including clustering, classification [23,24], regression [25], object detection, and semantic segmentation. In this paper, we explore energy scores due to their theoretical interpretability and effectiveness.

3  Method

In this paper, different from existing methods [7–11] and to thoroughly explore the information from the remaining modalities and balance the contribution of each modality during sentiment analysis under missing modality conditions, we propose the DUDF-MSA (Data Mining and Uncertainty-Aware with Missing Modalities for Multimodal Sentiment Analysis) method.

3.1 Framework Description

As illustrated in Fig. 2, in our framework, we first generate multimodal features with randomly missing data, Xts, Xas, and Xvs. After input preparation, different encoding methods are chosen for each modality: BERT for text; Transformer Encoder for video and audio, thus obtaining feature representations Et, Ea, and Ev. Based on these representations, to sufficiently exploit useful sentiment information from incomplete modalities, we introduce a Common Feature Extraction (CFE) mechanism; in this mechanism, features extracted across modalities Tc, Ac, and Vc are aggregated based on the attention mechanism, thus forming the common features Fsh which guide the model’s attention toward critical cues in missing modalities. Besides, contrastive learning guided by sentiment intensity is incorporated during training to capture more fine-grained sentiment cues.

Figure 2: Overview of the proposed framework (Xts, Xas, Xvs denote the input features from text, audio and visual modalities, respectively, where the gray regions indicate randomly missing parts. Fsh represents the common features, while Ets, Eas, Evs are the modality-specific features. L-common, L-specific, and L-task correspond to the losses associated with the common-level, specific-level and final-level predictions, respectively).

Meanwhile, we design the specific feature uncertainty-aware dynamic adjustment (SFUA) scheme. In this scheme, besides extracting modality-specific features including Ets, Eas, and Evs from text, audio, and visual modalities, respectively, it accurately evaluates the distinct contribution of each modality by calculating the class uncertainty of each modality based on the probability distribution of its features. A lower modality uncertainty, as represented by the energy score Energym, indicates a clearer sentiment orientation, and consequently, a higher adjustment weight for this modality. Afterwards, based on these adaptively weighted, our method can further effectively mitigate the negative impact of ambiguous sentiment cues from unreliable modalities during the following features aggregation stage. Finally, we combine the common features Fsh and the adjusted modality-specific features to generate the hierarchical sentiment predictions at common, specific, and final levels to obtain the analyzed multimodal sentiment result.

3.2 Encoding of Missing Multimodal Data

Building upon previous work, we use BERT [26], Librosa [27], and OpenFace [28] to separately encode the text, audio, and video source data. These preprocessed inputs are represented as sequences, denoted by Xm∈RTm×dm, where m∈{t,a,v} represents the modality type (t for text, a for audio, v for video), Tm is the sequence length, dm is the dimensionality of the modality vector.

To ensure the reproducibility of the missing data generation process, we explicitly present the detailed implementation details of missing modality simulation based on previous research [8,29], including missing rate configuration, random seed strategy, and missing value replacement schemes, which are specified as follows:

3.2.1 Missing Rate Configuration

The missing rate r ranges from 0% to 100% and is configured independently for each modality. Specifically, for each modality m∈{t,a,v}, we randomly select r×Tm feature positions from the sequence Xm for masking and removal. The missing rates of different modalities are independent of one another without mutual interference. This configuration conforms to real-world scenarios, where each modality may suffer from independent data loss (e.g., audio corruption caused by device failure and text missing due to data damage).

3.2.2 Random Seed Strategy

To guarantee experimental reproducibility, the random seed is fixed to 1111 throughout the missing data generation. Meanwhile, for fair comparison, we adopt three fixed random seeds (1111, 1112, 1113) for each missing rate setting and repeat the training process three times to obtain the average result. These seeds govern the random selection of masked positions in each modal sequence, ensuring that identical missing patterns can be reproduced under consistent experimental settings.

3.2.3 Missing Value Replacement Process

After determining the masked positions for each modality, we adopt modality-specific replacement strategies, thereby forming multimodal inputs with missing data Xms∈RTm×dm. The simplified procedure is illustrated in Algorithm 1:

After forming the multimodal inputs with missing data, we introduce Transformer Encoders for encoding the audio and video modalities, while the text modality is encoded using BERT [26]. The encoded representations of the sample’s text, video, and audio modalities are denoted as Et∈Rlt×dt, Ev∈Rlv×dv, Ea∈Rla×da, respectively. Here, lm represents the sequence length of each modality, and dm represents the corresponding feature vector dimension, where m∈{t,v,a}.

3.3 Common Feature Extraction

To fully explore the inherent sentiment information in incomplete modalities, we introduce a Common Feature Extraction (CFE) mechanism to guide the model’s attention to crucial sentiment cues by learning the common features across modalities. Simultaneously, during the training process, we employ contrastive learning guided by sentiment intensity to capture more fine-grained sentiment cues.

Notably, in our architecture, to avoid redundancy between common features and modality-specific features, the CFE module is designed to explicitly disentangle common cross-modal information and modality-specific information. CFE only captures modality-invariant sentiment patterns common across text, audio, and video, which are complementary to the modality-specific features in the subsequent SFUA module.

Based on this, the primary objective of this common feature extraction module can be described as to project information from different modalities into a shared representation space. Specifically, for the text modality, we select the [CLS] vector output by the BERT model, Etc∈Rl×dt, as its common vector representation. For the video and audio modalities, we choose the output vectors from the last layer of the Transformer encoder, Evc∈Rl×dv and Eac∈Rl×da, as their common vector representations. Next, through fully connected (FC) layers and ReLU activation functions, these three vectors are unified and converted into the same dimension to obtain Tc∈Rdc×1, Vc∈Rdc×1 and Ac∈Rdc×1.

Based on these information, in CFE, the formed common features which contain the crucial sentiment cues of missing modalities can be defined as follows:

Fsh=∑I∈{A,T,V}WI⋅Ic(1)

where Fsh denotes the common features, Ic is the feature representation of different modalities, and WI is the fusion weight corresponding to each modality. Specifically, we use a fully connected layer (MLP) to compute the fusion weights for each modality:

WI=softmax(MLP(Hc)), I∈{A,T,V},Hc={Ac,Tc,Vc}(2)

In this context, Hc is a set of features from three modalities. softmax(⋅) denotes the soft attention scheme. And MLP consists of two linear transformations and a ReLU activation function. In addition, to capture fine-grained sentiment information, we introduced contrastive learning guided by sentiment intensity during the training process.

As shown in Fig. 3, given samples i, j, and k, where the sentiment intensity differences from i to j and from i to k are 0.5 and 1.1, respectively, both (i, j) and (i, k) are initial negative sample pairs for i. However, the sentiment intensity difference between i and k is clearly much larger. Therefore, when calculating the contrastive loss, we assign a higher weight to the pair (i, k) to further separate i and k in the representation space compared to i and j. To achieve this, we design a weight function using the nonlinear function ∣tanh(x)∣, as shown below:

ω(i,j)={|tanh⁡(D(i,j)−2κ)|×1.5,(i,j)∈initial negative pairs|tanh⁡(D(i,j))|×1.5,(i,j)∈initial positive pairs(3)

Figure 3: Contrastive learning. The numbers in the figure represent different levels of sentiment intensity.

In this process, D(i,j) is the selection of initial positive and negative sample pairs. κ is the sentiment intensity difference threshold, to determine whether a sample j is classified as an initial positive or negative sample of i. This process is explained in the following equation:

{D(i,j)>κ,(i,j)∈initial negative pairsD(i,j)≤κ,(i,j)∈initial positive pairs(4)

and D(i,j) can be further described as:

D(i,j)=|yi′−yj′|,j∈B&j≠i(5)

In this form, yi′ and yj′ represent the sentiment intensity labels of samples i and j, respectively. B is a batch. Since the sentiment intensity ranges vary ([−3, 3] in MOSI/MOSEI and [−1, 1] in SIMS). To address the inconsistent label distribution across datasets, we adopt a strict linear normalization strategy to unify all sentiment intensity values into the interval of [−1, 1]. For MOSI and MOSEI with an original range of [−3, 3], the mapping is formulated as:

y′=y3(6)

This linear scaling only compresses the value scale while completely preserving the relative order and pairwise distance of sample labels. Since the transformation is bijective and reversible, no effective fine-grained sentiment information is lost during normalization. Furthermore, the normalized labels are only utilized for weighted contrastive learning, which relies on relative sentiment differences between samples; thus, the linear mapping will not introduce additional bias. The original raw labels are still adopted for the final regression prediction task. Based on the above settings, we use a unified mapping during the contrastive learning process to standardize dataset differences in label distribution. Finally, the contrastive loss we obtain is:

Lcl=−Ei∈B∑(a,p)∈Piδ(a,p)∑(a,q)∈Pi∪Niδ(a,q)(7)

In this equation, δ(a,p)=e[ω(i,j)∗sim(a,p)τ], ω(i,j) is the weight specified by Eq. (3). B is a batch. Pi and Ni are the positive and negative sample pairs for sample i, respectively. Pi and Ni are defined as follows:

Pi=Pinteri∪PintraiNi=Ninteri∪Nintrai(8)

where Pinteri and Ninteri the positive and negative sample pairs between modalities. Pintrai and Nintrai are the positive and negative sample pairs within the modality. The inter-modal positive and negative sample pair selection is as follows:

Pinteri={(Vci,Tcj), (Vci,Acj), (Tci,Acj)∪(Vci,Tcj), (Tci,Vcj), (Vci,Acj),(Aci,Vcj), (Tci,Acj), (Aci,Tcj)∣ (i,j)∈initial positive pairs}Ninteri={(Vci,Tck), (Tci,Vck), (Vci,Ack), (Aci,Vck), (Tci,Ack), (Aci,Tck)∣ (i,k)∈initial negative pairs}(9)

Besides, the intra-modal positive and negative sample pair selection can be defined as follows:

Pintrai={(Tci,Tcj),(Vci,Vcj),(Aci,Acj),∣(i,j)∈initial positive pairs}Nintrai={(Tci,Tck),(Vci,Vck),(Aci,Ack),∣(i,k)∈initial negative pairs}(10)

In above equation, Tci, Vci, Aci correspond to the representations of the three different modalities of sample i, with the remaining symbols having the same meaning.

3.4 Specific Feature Uncertainty-Aware Dynamic Adjustment

In addition to the common feature representation, which strengthens the mining of latent sentiment information, we design the modality-specific branch that captures unique emotional information of every single modality and introduce the specific feature uncertainty-aware dynamic adjustment (SFUA) scheme to adaptively assess each modality’s contribution by quantifying its class uncertainty based on the probability distribution of its features. Following this way, our method can further effectively mitigate the negative impact of ambiguous sentiment cues from unreliable modalities during the following feature aggregation stage.

Step 1: Specific feature extraction. Firstly, given a modality Em∈Rlm×dm,m∈{t,v,a}, we use global average pooling (GAP) along the sequence length to compress Em into Em1∈R1×dm. Then, two-step nonlinear transformations are applied to project Em1 into a new low-dimensional space:

Ems=σ2(W2σ1(W1Em1T)), m∈{t,v,a}(11)

In this context, W1∈R(dm/8)×dm, W2∈R(ds)×(dm/8), Ems∈Rds×1, the σ1 represents the ReLU function,the σ2 represents the Sigmoid function.

Step 2: Uncertainty-aware dynamic adjustment

Unlike traditional score-based methods, which may struggle to generalize across different datasets and tend to exhibit unstable or unreliable performance, we use an energy-based uncertainty estimation to calculate the class uncertainty of each modality and output the class energy scores. And according to these scores to form the adjustment weights stably. This process can be described in Eq. (12):

Energym=−log∑kKefm(k)x(m),m∈{t,a,v}(12)

In this equation, Energym is the energy score we calculate, x(m) the input single-modality feature, fm is the classifier we designed, and fm(k)xm is the output log of the classifier fm corresponding to the k-th class label. In this framework, we set up independent classifiers for each modality. Notably, each classifier does not require additional pre-training, which makes our model highly adaptive. We define the classification loss for each modality as follows:

Lspm=∑mℒCE(y,fm(x(m))),m∈{t,a,v}(13)

In this class, uncertainty measure for each modality, a lower energy score indicates a more concentrated probability distribution of the modality for the sample’s category, implying a clearer sentiment orientation, and the less confusing information exists in this modality. Conversely, a higher energy score reflects a more dispersed distribution, suggesting that the sample lacks distinct category attributes, and may exist with more confusing sentiment category cues.

As illustrated in Fig. 4a, the predicted class probability distribution of the text modality is more concentrated, indicating that its specific features possess a clearer sentiment orientation and contain less confusing information; therefore, a higher weight should be assigned to this modality. In contrast, in Fig. 4b, the predicted probability distribution for the text modality is less concentrated than that of the audio modality, revealing the presence of confusing information that may hinder sentiment prediction accuracy. Accordingly, a lower weight should be assigned to this modality.

Figure 4: Uncertainty explanation. Based on the concentration of the category probability distribution predicted by the model. (a,b) illustrate the predicted probability distributions for different samples, respectively.

Therefore, according to the calculated energy scores, we determine the final adjustment weights:

ωm=α×Energym,m∈{t,a,v}(14)

In this context, α<0 serves as a hyperparameter. Finally, the final modality-specific representations are derived according to the adjustment weights assigned to each modality:

Fspm=ωmEms,m∈{t,a,v}(15)

3.5 Post-Processing for Sentiment Prediction

In this part, the common and the adjusted specific features are integrated to obtain the final fused modality:

F∗=Concat(Fsh,Fspt,Fspa,Fspv)(16)

where Fsh is the common features obtained by CFE; (Fspt), (Fspa), and (Fspv) are the adjusted text, audio, and video modality-specific features formed by SFUA. Then, the combined features F∗ is employed to output the final sentiment results.

Moreover, to enhance the representational capability of different encoders and improve prediction accuracy, we employ a hierarchical prediction strategy comprising common-level, specific-level, and final predictions. And during the training stage, the outputs from the common-level and specific-level predictors are each supervised by the ground-truth sentiment labels. This auxiliary supervision helps guide the learning of individual modules, enabling them to achieve better representation quality and overall performance.

3.6 Overall Learning Objective

The entire network is trained end-to-end by integrating common features and modality-specific features. The two branches have different optimization objectives and non-shared parameters. Meanwhile, we introduce hierarchical prediction to provide separate supervision for the two types of features, which enhances the discriminability of different encoders and ensures no redundancy between common features and modality-specific features. We define the overall learning objective as a combination of multiple loss components corresponding to different sub-modules. Specifically, the overall training loss is formulated as:

Loverall=Lsh+Lspm+Ltask+γLcl(17)

In this context, Lsh and Lspm are the losses associated with the common features extraction and specific feature learning, Lsh is demonstrated as:

Lsh=1Nb∑i=0Nb|yi−y^i|(18)

where Nb is the number of samples in batch B, y^i is the true sentiment intensity and yi is the predicted sentiment intensity value. Furthermore, the Ltask and Lcl denote the output loss and the contrastive loss. Here, Ltask can be described as:

Ltask=1Nb∑i=0Nb|yi−y^i|(19)

4  Experiments and Analysis

4.1 Datasets and Metrics

We evaluated our method on three widely used MSA datasets: the English datasets MOSI [30] and MOSEI [31], and the Chinese dataset SIMS [32]. More detailed information about the datasets can be found in Appendix A.

To ensure a fair comparison and evaluation, we follow the same missing modality setting as LNLN [8], where training data is randomly dropped with a certain probability. During testing, the missing rate r is varied from 0 to 0.9 in increments of 0.1, notably, r = 1.0 is excluded as it removes all modalities, making the task ill-posed. Based on the results for each missing rate, we calculated the average value from the missing rate of 0 to 0.9 as the overall performance of the model under different noise levels.

Based on the research by Zhang et al. [8], we report the 5-level (Acc-5) and 7-level (Acc-7) accuracies for MOSI and MOSEI, as well as the 3-level (Acc-3) and 5-level (Acc-5) accuracies for SIMS. Additionally, all datasets report the 2-level accuracy (Acc-2), Mean Absolute Error (MAE), Pearson Correlation (Corr), and F1 score (F1).

For Acc-2 and F1 on MOSI and MOSEI, the results are provided for two settings: Negative vs. Positive (to the left of “/”) and Negative vs. Non-negative (to the right of “/”). Except for MAE, higher values indicate better performance.

Comprehensive details of the experimental setup are provided in Appendix B.

4.2 State-of-the-Art Comparison

We compare DUDF-MSA with seven typical multimodal sentiment analysis methods: MISA [33], Self-MM [29], CENET [34], TFR-Net [6], ALMT [17], GLGIS [35], and LNLN [8]. These baselines include traditional multimodal fusion approaches, modality completion and reconstruction methods, as well as recent robust models designed for missing and noisy scenarios. The proposed method combines common feature learning and uncertainty-aware dynamic weighting, which outperforms existing approaches and yields stronger robustness under various modality missing conditions. Detailed baseline settings and descriptions are provided in Appendix C.

As presented in Table 1, our method consistently outperforms existing approaches on most evaluation metrics across both the MOSI and MOSEI English datasets (note that lower MAE values are better). Specifically, on the MOSI dataset, when compared with GLGIS [35], which is our baseline, our method achieves improvements of 0.95% in multi-class accuracy (Acc-7) and 0.9% in the stability metric (Corr), while reducing MAE by 2.5%. On the MOSI dataset, when compared to the text-dominant LNLN [8], our approach further yields gains of 1.04% in Acc-7 and 1.16% in F1. On the MOSEI dataset, our method surpasses LNLN [8] by 1.2% on Acc-7, 1.81% on F1, while reducing MAE by 0.9%. These results demonstrate the robustness and effectiveness of our method in multimodal sentiment analysis, even under challenging conditions with missing data.

Moreover, as shown in Table 2, our method also achieves state-of-the-art performance on most metrics for the Chinese dataset SIMS. Compared to the static fusion model GLGIS [35], our method improves by 1.01% on Acc-5, 2.0% on Corr, and Moreover, compared to the text modality-dominated LNLN [8], our method improves by 1.62% on Acc-5 and 2.3% on Corr and the MAE is reduced by 1.3%. These results demonstrate that our model performs well in terms of accuracy and robustness across different types of datasets, proving the generalization capability of our model. Detailed experimental results for various missing rates are presented in Appendix D.

4.3 Robustness Comparison

To visually compare the robustness across different methods at various missing rates, Fig. 5 plots the performance curves of five state-of-the-art baseline methods: MISA [33], CENET [34], ALMT [17], GLGIS [35], and LNLN [8]. Fig. 5a–c show the F1 curves on MOSI, MOSEI, and SIMS, respectively, while Fig. 5d–f depict the corresponding MAE curves. It can be observed that our method consistently achieves superior performance across all datasets, maintaining higher F1 scores and lower MAE values compared with other approaches. These results further demonstrate the robustness of our method under varying missing rates (note that lower MAE values indicate better performance).

Figure 5: Performance curves at various missing rates ((a–c) show the F1 scores on MOSI, MOSEI, and SIMS, respectively, while (d–f) depict the corresponding MAE results on the same datasets).

4.4 Ablation Study

4.4.1 Effectiveness of Main Components

In our framework, the Common Features Extraction (CFE) and Specific Features Uncertainty-aware Dynamic Adjustment (SFUA) modules are the two key components responsible for constructing the common features and the adjusted specific features, respectively. To verify their effectiveness, we conducted ablation experiments on the MOSI and SIMS datasets by individually removing each module from our method. The corresponding results are reported in Table 3.

In this table, “w/o CFE” denotes the removal of the CFE module, and “w/o attention” represents the removal of the fusion weights generation scheme in CFE, as defined in Eq. (2); in this case, the features from each modality are aggregated averagely. Additionally, “w/o SFUA” denotes the removal of the specific feature uncertainty-aware dynamic adjustment (step 2) scheme in SFUA module, and “w/o spe” means removing specific features from the overall framework.

1. Effectiveness of CFE for Common Features

In the “w/o CFE” setting, removing the CFE, which is the common feature extractor, leads to performance degradation. Specifically:

On the MOSI dataset, the Acc-7 decreases by 0.71%, and the MAE increases by 1.6%. Moreover, on the SIMS dataset, the Acc-5 drops by 0.98%. The performance degradations demonstrate the crucial role of the CFE in sentiment accurate analysis by capturing inherent sentiment cues from missing modalities.

To further analyze the contribution of our CFE in enhancing the model’s data mining ability, we visualize the confusion matrices on the MOSI dataset under different missing rates (0%, 0.5%, and 0.9%). As illustrated in Fig. 6a–c, which are the results without CFE, with the missing rate increases, the model tends to bias its predictions toward specific incorrect categories. These increasingly concentrated predicted results indicate the “lazy” behavior of the model with missing data, where the model over relies on partial modalities and ignores inherent informative cues.

Figure 6: Confusion matrices of the MOSI dataset under different missing rates (The upper row illustrates the confusion matrices after removing the common features, while the lower row presents those of the complete model, both under varying missing rates). (a–c) illustrate the confusion matrices on the MOSI dataset without common features at missing rates of 0%, 0.5% and 0.9%, respectively. (d–f) illustrate the confusion matrices on the MOSI dataset with common features at missing rates of 0%, 0.5% and 0.9%, respectively.

Conversely, Fig. 6d–f display the confusion matrices of our entire method on the MOSI dataset for the same missing rates (0%, 0.5%, and 0.9%, respectively). Compared with the results without CFE, our model maintains more diverse and relatively balanced category distributions. Combined with the quantitative results, these findings confirm that the proposed CFE effectively improves the model’s mining of valid information under conditions of missing modality.

2. Effectiveness of SFUA for Specific Feature Fusion

To examine the contribution of SFUA module, we conduct two ablation settings by removing the specific features (“w/o Spe”) and the uncertainty-aware weight adjustment (Ste p2) scheme (“w/o SFUA”) from our framework, respectively.

As shown in Table 3, when remove the step 2 from SFUA (“w/o SFUA”), the Acc-7 on the MOSI dataset decreases by 0.48%, and the Acc-5 on the SIMS dataset drops by 0.57%. This demonstrates the importance of effectively measuring the contributions of each modalities, as the confusing or noisy class information in certain modalities can otherwise hinder the accuracy of the final prediction.

Furthermore, when the specific features are completely removed (“w/o Spe”), the Acc-7 on the MOSI dataset further decreases by 0.39%, and the Acc-5 on the SIMS dataset drops by 1.63%. These performance degradations highlight the necessity of molding modality-specific representations to overall sentiment prediction.

3. Effectiveness of the Combination

As exhibited in Table 3, jointly learning both common and the adjusted specific features achieves the best overall performance. This result confirm the complementary relationship between the common and specific components in our framework, demonstrating the coherent integration of Common Features Extraction (CEF) and Specific Feature Uncertainty-Aware Dynamic Adjustment (SFUA) enables more accurate multimodal sentiment prediction.

4.4.2 Effectiveness of Contrastive Learning and Hierarchical Prediction

During the training stage, contrastive learning is integrated into the CFE module to capture fine-grained sentiment information. At the same time, hierarchical prediction is utilised to compute the losses for the CFE, SFUA, and final prediction, allowing for separate and coordinated optimisation to achieve better performance.

To evaluate the effectiveness of these two designs, we conduct ablation experiments on the MOSI and SIMS datasets by removing contrastive learning and hierarchical prediction, respectively. As shown in Table 4, “w/o CL” denotes the removal of the contrastive learning mechanism; “w/o L-common” represents removing the loss from the CFE module, while “w/o L-specific” denotes removing the loss from the SFUA component.

Specifically, after removing contrastive learning (“w/o CL”), the Acc-7 on the MOSI dataset decreases by 0.9%, and the Acc-5 on the SIMS dataset drops by 1.87%. This notable degradation in multi-class accuracy highlights the importance of integrating contrastive learning to capture more fine-grained sentiment representations.

Furthermore, when the common-level loss is removed (“w/o L-common”), the Acc-7 on MOSI decreases by 0.76%, and the MAE increases by 1.0%. Similarly, removing the specific-level loss (“w/o L-specific”) results in a 0.93% decrease in Acc-7 and a 1.2% increase in MAE. These results verify that hierarchical prediction effectively enhances the training of both modules, promotes mutual feature refinement, and ultimately contributes to improving the overall model accuracy.

4.5 Sensitivity Analysis of Hyperparameter α

To determine the optimal hyperparameter α for the uncertainty-aware weighting strategy, we conduct a sensitivity analysis. Since α<0, we test a fine-grained set of values: −0.05, −0.1, −0.3, −0.5, −0.7, −0.9, and −1.0. All experiments are conducted under the same modality missing setting, and we report Acc-7/Acc-5 and MAE on the MOSI and SIMS datasets.

As shown in Table 5, our model achieves the best accuracy and the lowest MAE when α=−0.1. When α=−0.05, the weighting strength is insufficient, leading to slightly inferior performance compared with α=−0.1. As |α| increases (i.e., α≤−0.3), the performance declines continuously, indicating that excessive suppression causes the loss of useful information. The performance curve is stable around α=−0.1, which verifies that this value is robust and reasonable. Therefore, we uniformly adopt α=−0.1 for all experiments in this paper.

5  Conclusion

This paper proposes a novel framework, DUDF-MSA, to address the multimodal sentiment analysis (MSA) task under conditions of randomly missing data. In DUDF-MSA, to fully mine and exploit the inherent sentiment cues within each modality, we first introduce a Common Feature Extraction (CFE) mechanism that guides the model’s attention toward critical cues in incomplete modalities by learning modality-invariant common features. During this stage, contrastive learning is incorporated into the module training to enhance the extraction of fine-grained sentiment features. Subsequently, we design a Specific Feature Uncertainty-Aware Dynamic Adjustment (SFUA) scheme to accurately evaluate the contribution of each modality. SFUA quantifies the class uncertainty of modality-specific features based on their probability distributions and adaptively determines aggregation weights, thereby reducing the negative influence of ambiguous sentiment cues from unreliable modalities during feature aggregation. Finally, the common features and the adaptively weighted modality-specific features are jointly learned to predict sentiment intensity. Extensive experiments on three public benchmark datasets demonstrate that DUDF-MSA consistently outperforms state-of-the-art methods in handling varying degrees of data incompleteness.

Even with the promising performance of the proposed method, it still has certain limitations. First, the introduction of contrastive learning incurs extra computational cost and training overhead. Second, the hyperparameter α in the uncertainty weighting strategy is determined empirically, and the model exhibits obvious sensitivity to this parameter. For future work, we will conduct further research from three perspectives: developing a more efficient contrastive learning scheme to reduce computational consumption, designing an adaptive mechanism to automatically optimize the hyperparameter α and improve model robustness, and extending the current framework to handle more challenging missing cases, including full modality missing and hardware-caused data corruption.

Acknowledgement: Not applicable.

Funding Statement: This work is supported by the Key Research and Development Program of Zhejiang Province under Grand No. 2024C01026 and Key Research and Promotion Projects of Henan Province under Grand NO. 262102320010.

Author Contributions: Ying Cao contributed to methodology, validation, software, investigation, writing—original draft, supervision, and project administration. Penghui Zhao contributed to methodology, validation, software, writing—original draft, and visualization. Xinyu Qiao contributed to visualization and resources. Ningfan Zhan contributed to formal analysis and data curation. Xiaomei Zou contributed to data curation, resources, funding acquisition, validation, and visualization. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: The data used in this study are publicly available benchmark datasets: [MOSI] “The data that support the findings of this study are openly available in GitHub at https://github.com/A2Zadeh/CMU-MultimodalSDK”. [MOSEI] “The data that support the findings of this study are openly available in GitHub at https://github.com/A2Zadeh/CMU-MultimodalSDK”. [SIMS] “The data that support the findings of this study are openly available in GitHub at https://github.com/thuiar/MMSA”.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

Appendix A Dateset

The MOSI [30] dataset is a widely used dataset comprising three modalities (text, video, and audio). It is collected from 93 YouTube videos, where a speaker expresses their opinion about a movie. It is divided into a training set with 1284 samples, a validation set with 229 samples, and a test set with 686 samples. Each sample has an emotion score ranging from –3 (representing strongly negative emotions) to 3 (representing strongly positive emotions).

The MOSEI [31] dataset is a larger version of MOSI, comprising 22,856 annotated video segments spanning 250 distinct topics. The samples are divided into 16,326 segments for training, 1871 segments for validation, and 4659 segments for testing. Each segment is labelled with a score ranging from −3 (strongly negative) to 3 (strongly positive).

The SIMS [32] dataset is a Chinese multimodal sentiment dataset comprising 2281 video segments from various movies and TV shows. It is divided into 1368 samples for training, 456 samples for validation, and 457 samples for testing. Each sample has a multimodal label and three unimodal labels, with sentiment scores ranging from −1 (strongly negative) to +1 (strongly positive).

Appendix B Baselines

We compare DUDF-MSA against several MSA baselines. All experiments are conducted under the same dataset settings for fairness. We reproduce GLGIS and LNLN in our environment, while the results for other baselines are reported from [8]. Below are brief descriptions of them.

MISA [33] models the shared and specific features of different modalities to improve the robustness of sentiment prediction.

Self-MM [29] generates unimodal labels and performs multi-task training to capture both consistent and divergent representations.

CENET [34] is a cross-modal enhancement network that adaptively aligns and fuses multimodal signals through a cross-attention interaction mechanism.

TFRNet [6] is a transformer-based model that combines fusion-reconstruction strategies to improve sentiment prediction performance under incomplete modality conditions.

ALMT [17] leverages linguistic features to suppress irrelevant or conflicting information from visual and audio inputs, and learns complementary multimodal representations.

GLGIS [35] models the common and specific features of different modalities, while incorporating contrastive learning guided by sentiment intensity to improve the robustness of sentiment prediction.

LNLN [8] treats the language modality as the dominant modality and introduces dominant modality correction and dominant-modality-based multimodal learning to enhance robustness in noisy and missing modality scenarios.

Appendix C Implementation Details

The experiments on the MOSI and SIMS datasets were conducted using a single NVIDIA RTX 3080 Ti GPU, while those on the MOSEI dataset were performed using an NVIDIA RTX 4060 Ti. The training mode adapted was complete training, without any additional pre-training. Both the video and audio Transformer encoder consists of two layers. The AdamW optimizer was used for parameter optimization.

To ensure consistent and fair comparisons across all methods, each experiment was repeated three times with fixed random seeds of 1111, 1112, and 1113. The key hyperparameters used in our experiments are summarized in Table A1.

Appendix D Details of Robust Comparison

We conduct comprehensive experiments under different missing rate conditions to further assess model robustness and examine the impact of discarding different modalities. The detailed results on the MOSI, MOSEI, and SIMS datasets are presented in Tables A2–A4, respectively.

References

1. Liu Y, Yuan Z, Mao H, Liang Z, Yang W, Qiu Y, et al. Make acoustic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module. In: Proceedings of the 2022 International Conference on Multimodal Interaction; 2022 Nov 7–11; Bengaluru, India. p. 247–58. [Google Scholar]

2. Sun Z, Sarma P, Sethares W, Liang Y. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2020 Feb 7–12; New York, NY, USA. p. 8992–9. [Google Scholar]

3. Hu G, Lin TE, Zhao Y, Lu G, Wu Y, Li Y. UniMSE: towards unified multimodal sentiment analysis and emotion recognition. arXiv:2211.11256. 2022. [Google Scholar]

4. Aguilar G, Rozgić V, Wang W, Wang C. Multimodal and multi-view models for emotion recognition. arXiv:1906.10198. 2019. [Google Scholar]

5. Pham H, Liang PP, Manzini T, Morency LP, Póczos B. Found in translation: learning robust joint representations by cyclic translations between modalities. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2019 Jan 27–Feb 1; Honolulu, HI, USA. p. 6892–9. [Google Scholar]

6. Yuan Z, Li W, Xu H, Yu W. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In: Proceedings of the 29th ACM International Conference on Multimedia; 2021 Oct 20–24; Chengdu, China. p. 4400–7. [Google Scholar]

7. Guo Z, Jin T, Zhao Z. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (volume 1: Long Papers); 2024 Aug 11–16; Bangkok, Thailand. p. 1726–36. [Google Scholar]

8. Zhang H, Wang W, Yu T. Towards robust multimodal sentiment analysis with incomplete data. Adv Neural Inf Process Syst. 2024;37:55943–74. doi:10.52202/079017-1779. [Google Scholar] [CrossRef]

9. Li X, Cheng X, Miao D, Zhang X, Li Z. TF-Mamba: text-enhanced fusion mamba with missing modalities for robust multimodal sentiment analysis. In: Christodoulopoulos C, Chakraborty T, Rose C, Peng V, editors. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2025; 2025 Nov 4–9; Suzhou, China; 2025. p. 11252–67. [Google Scholar]

10. Golagana V, Row SV, Rao PS. Adaptive multimodal sentiment analysis: improving fusion accuracy with dynamic attention for missing modality. J Electr Syst. 2024;20:134. [Google Scholar]

11. Zeng J, Zhou J, Liu T. Mitigating inconsistencies in multimodal sentiment analysis under uncertain missing modalities. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; 2022 Dec 7–11; Abu Dhabi, United Arab Emirates. p. 2924–34. [Google Scholar]

12. Wu Y, Lin Z, Zhao Y, Qin B, Zhu LN. A text-centered shared-private framework via cross-modal prediction for multimodal sentiment analysis. In: Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021; 2021 Aug 1–6; Virtual. p. 4730–8. doi:10.18653/v1/2021.findings-acl.417. [Google Scholar] [CrossRef]

13. Yang J, Yu Y, Niu D, Guo W, Xu Y. Confede: contrastive feature decomposition for multimodal sentiment analysis. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2023 Jul 9–14; Toronto, ON, Canada. p. 7617–30. [Google Scholar]

14. Li X, Zhang H, Dong Z, Cheng X, Liu Y, Zhang X. Learning fine-grained representation with token-level alignment for multimodal sentiment analysis. Expert Syst Appl. 2025;269:126274. [Google Scholar]

15. Sun K, Tian M. Sequential fusion of text-close and text-far representations for multimodal sentiment analysis. In: Rambow O, Wanner L, Apidianaki M, Al-Khalifa H, Eugenio BD, Schockaert S, editors. Proceedings of the 31st International Conference on Computational Linguistics; 2025 Jan 19–24; Abu Dhabi, United Arab Emirates, p. 40–9. [Google Scholar]

16. Han W, Chen H, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis. arXiv:2109.00412. 2021. [Google Scholar]

17. Zhang H, Wang Y, Yin G, Liu K, Liu Y, Yu T. Learning language-guided adaptive hyper-modality representation for multimodal sentiment analysis. arXiv:2310.05804. 2023. [Google Scholar]

18. Li M, Yang D, Lei Y, Wang S, Wang S, Su L, et al. A unified self-distillation framework for multimodal sentiment analysis with uncertain missing modalities. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2024 Feb 20–27; Vancouver, BC, Canada. p. 10074–82. [Google Scholar]

19. Li L, Xu G, Shen X, Xu Z, Zhang Y, Zhang Z, et al. Senti-iFusion: an integrity-centered hierarchical fusion framework for multimodal sentiment analysis under uncertain modality missingness. arXiv:2511.16990. 2025. [Google Scholar]

20. Bao J, Qian J, Yan M, Yang J. Progressive representation learning for multimodal sentiment analysis with incomplete modalities. arXiv:2603.09111. 2026. [Google Scholar]

21. Liu W, Wang X, Owens J, Li Y. Energy-based out-of-distribution detection. Adv Neural Inf Process Syst. 2020;33:21464–75. [Google Scholar]

22. Ming Y, Fan Y, Li Y. Poem: out-of-distribution detection with posterior sampling. In: Proceedings of the International Conference on Machine Learning, PMLR; 2022 Jul 17–23; Baltimore, MD, USA. p. 15650–65. [Google Scholar]

23. Tellamekala MK, Amiriparian S, Schuller BW, André E, Giesbrecht T, Valstar M. COLD fusion: calibrated and ordinal latent distribution fusion for uncertainty-aware multimodal emotion recognition. IEEE Trans Pattern Anal Mach Intell. 2023;46(2):805–22. [Google Scholar]

24. Chen YT, Shi J, Ye Z, Mertz C, Ramanan D, Kong S. Multimodal object detection via probabilistic ensembling. In: European Conference on Computer Vision. Berlin/Heidelberg, Germany: Springer; 2022. p. 139–58. [Google Scholar]

25. Ma H, Han Z, Zhang C, Fu H, Zhou JT, Hu Q. Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. Adv Neural Inf Process Syst. 2021;34:6881–93. [Google Scholar]

26. Devlin J, Chang MW, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T, editors. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); 2019 Jun 2–7; Minneapolis, MN, USA, 4171–86. [Google Scholar]

27. McFee B, Raffel C, Liang D, Ellis DP, McVicar M, Battenberg E, et al. librosa: audio and music signal analysis in python. SciPy. 2015;2015:18–24. [Google Scholar]

28. Baltrusaitis T, Zadeh A, Lim YC, Morency LP. OpenFace 2.0: facial behavior analysis toolkit. In: Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); 2018 May 15–19; Xi’an, China. p. 59–66. [Google Scholar]

29. Yu W, Xu H, Yuan Z, Wu J. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence; 2021 Feb 2–9; Virtual. p. 10790–7. [Google Scholar]

30. Zadeh A, Zellers R, Pincus E, Morency LP. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv:1606.06259. 2016. [Google Scholar]

31. Zadeh AB, Liang PP, Poria S, Cambria E, Morency LP. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); 2018 Jul 15–20; Melbourne, Australia. p. 2236–46. [Google Scholar]

32. Yu W, Xu H, Meng F, Zhu Y, Ma Y, Wu J, et al. CH-SIMS: a Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 2020 Jul 5–10; Virtual. p. 3718–27. [Google Scholar]

33. Hazarika D, Zimmermann R, Poria S. Misa: modality-invariant and-specific representations for multimodal sentiment analysis. In: Proceedings of the 28th ACM International Conference on Multimedia; 2020 Oct 12–16; Seattle, WA, USA. p. 1122–31. [Google Scholar]

34. Tao H, Xie C, Wang J, Xin Z. CENet: a channel-enhanced spatiotemporal network with sufficient supervision information for recognizing industrial smoke emissions. IEEE Internet Things J. 2022;9(19):18749–59. [Google Scholar]

35. Yang Y, Dong X, Qiang Y. CLGSI: a multimodal sentiment analysis framework based on contrastive learning guided by sentiment intensity. In: Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2024; 2024 May 22–27; Mexico City, Mexico. p. 2099–110. [Google Scholar]