Vol.39, No.1, 2021, pp.37-54, doi:10.32604/csse.2021.017230
Cyclic Autoencoder for Multimodal Data Alignment Using Custom Datasets
  • Zhenyu Tang1, Jin Liu1,*, Chao Yu1, Y. Ken Wang2
1 College of Information Engineering, Shanghai Maritime University, Shanghai, 200135, China
2 Division of Management and Education, University of Pittsburgh, Bradford, 16701, USA
* Corresponding Author: Jin Liu. Email:
Received 21 January 2021; Accepted 24 February 2021; Issue published 10 June 2021
The subtitle recognition under multimodal data fusion in this paper aims to recognize text lines from image and audio data. Most existing multimodal fusion methods tend to be associated with pre-fusion as well as post-fusion, which is not reasonable and difficult to interpret. We believe that fusing images and audio before the decision layer, i.e., intermediate fusion, to take advantage of the complementary multimodal data, will benefit text line recognition. To this end, we propose: (i) a novel cyclic autoencoder based on convolutional neural network. The feature dimensions of the two modal data are aligned under the premise of stabilizing the compressed image features, thus the high-dimensional features of different modal data are fused at the shallow level of the model. (ii) A residual attention mechanism that helps us improve the performance of the recognition. Regions of interest in the image are enhanced and regions of disinterest are weakened, thus we can extract the features of the text regions without further increasing the depth of the model (iii) a fully convolutional network for video subtitle recognition. We choose DenseNet-121 as the backbone network for feature extraction, which effectively enabling the recognition of video subtitles in complex backgrounds. The experiments are performed on our custom datasets, and the automatic and manual evaluation results show that our method reaches the state-of-the-art.
Deep learning; convolutional neural network; multimodal; text recognition
Cite This Article
Z. Tang, J. Liu, C. Yu and Y. K. Wang, "Cyclic autoencoder for multimodal data alignment using custom datasets," Computer Systems Science and Engineering, vol. 39, no.1, pp. 37–54, 2021.
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.