Open Access iconOpen Access



Multimodal Sentiment Analysis Using BiGRU and Attention-Based Hybrid Fusion Strategy

Zhizhong Liu*, Bin Zhou, Lingqiang Meng, Guangyu Huang

School of Computer and Control Engineering, Yantai University, Yantai, 264005, China

* Corresponding Author: Zhizhong Liu. Email: email

Intelligent Automation & Soft Computing 2023, 37(2), 1963-1981.


Recently, multimodal sentiment analysis has increasingly attracted attention with the popularity of complementary data streams, which has great potential to surpass unimodal sentiment analysis. One challenge of multimodal sentiment analysis is how to design an efficient multimodal feature fusion strategy. Unfortunately, existing work always considers feature-level fusion or decision-level fusion, and few research works focus on hybrid fusion strategies that contain feature-level fusion and decision-level fusion. To improve the performance of multimodal sentiment analysis, we present a novel multimodal sentiment analysis model using BiGRU and attention-based hybrid fusion strategy (BAHFS). Firstly, we apply BiGRU to learn the unimodal features of text, audio and video. Then we fuse the unimodal features into bimodal features using the bimodal attention fusion module. Next, BAHFS feeds the unimodal features and bimodal features into the trimodal attention fusion module and the trimodal concatenation fusion module simultaneously to get two sets of trimodal features. Finally, BAHFS makes a classification with the two sets of trimodal features respectively and gets the final analysis results with decision-level fusion. Based on the CMU-MOSI and CMU-MOSEI datasets, extensive experiments have been carried out to verify BAHFS’s superiority.


Cite This Article

Z. Liu, B. Zhou, L. Meng and G. Huang, "Multimodal sentiment analysis using bigru and attention-based hybrid fusion strategy," Intelligent Automation & Soft Computing, vol. 37, no.2, pp. 1963–1981, 2023.

cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 491


  • 264


  • 0


Share Link