Open Access iconOpen Access



Improving Targeted Multimodal Sentiment Classification with Semantic Description of Images

Jieyu An*, Wan Mohd Nazmee Wan Zainon, Zhang Hao

School of Computer Sciences, Universiti Sains Malaysia, Penang, 11800, Malaysia

* Corresponding Author: Jieyu An. Email: email

Computers, Materials & Continua 2023, 75(3), 5801-5815.


Targeted multimodal sentiment classification (TMSC) aims to identify the sentiment polarity of a target mentioned in a multimodal post. The majority of current studies on this task focus on mapping the image and the text to a high-dimensional space in order to obtain and fuse implicit representations, ignoring the rich semantic information contained in the images and not taking into account the contribution of the visual modality in the multimodal fusion representation, which can potentially influence the results of TMSC tasks. This paper proposes a general model for Improving Targeted Multimodal Sentiment Classification with Semantic Description of Images (ITMSC) as a way to tackle these issues and improve the accuracy of multimodal sentiment analysis. Specifically, the ITMSC model can automatically adjust the contribution of images in the fusion representation through the exploitation of semantic descriptions of images and text similarity relations. Further, we propose a target-based attention module to capture the target-text relevance, an image-based attention module to capture the image-text relevance, and a target-image matching module based on the former two modules to properly align the target with the image so that fine-grained semantic information can be extracted. Our experimental results demonstrate that our model achieves comparable performance with several state-of-the-art approaches on two multimodal sentiment datasets. Our findings indicate that incorporating semantic descriptions of images can enhance our understanding of multimodal content and lead to improved sentiment analysis performance.


Cite This Article

J. An, W. M. N. W. Zainon and Z. Hao, "Improving targeted multimodal sentiment classification with semantic description of images," Computers, Materials & Continua, vol. 75, no.3, pp. 5801–5815, 2023.

cc This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 650


  • 325


  • 0


Share Link