Open Access

ARTICLE

Fine-Grained Features for Image Captioning

Mengyue Shao¹, Jie Feng^1,*, Jie Wu¹, Haixiang Zhang¹, Yayu Zheng²

1 Zhejiang Sci-Tech University, Hangzhou, 310020, China
2 Zhejiang University of Technology, Hangzhou, 310020, China

* Corresponding Author: Jie Feng. Email:

Computers, Materials & Continua 2023, 75(3), 4697-4712. https://doi.org/10.32604/cmc.2023.036564

Received 04 October 2022; Accepted 13 February 2023; Issue published 29 April 2023

Abstract

Image captioning involves two different major modalities (image and sentence) that convert a given image into a language that adheres to visual semantics. Almost all methods first extract image features to reduce the difficulty of visual semantic embedding and then use the caption model to generate fluent sentences. The Convolutional Neural Network (CNN) is often used to extract image features in image captioning, and the use of object detection networks to extract region features has achieved great success. However, the region features retrieved by this method are object-level and do not pay attention to fine-grained details because of the detection model’s limitation. We offer an approach to address this issue that more properly generates captions by fusing fine-grained features and region features. First, we extract fine-grained features using a panoramic segmentation algorithm. Second, we suggest two fusion methods and contrast their fusion outcomes. An X-linear Attention Network (X-LAN) serves as the foundation for both fusion methods. According to experimental findings on the COCO dataset, the two-branch fusion approach is superior. It is important to note that on the COCO Karpathy test split, CIDEr is increased up to 134.3% in comparison to the baseline, highlighting the potency and viability of our method.

---

Keywords

---