Open Access

ARTICLE

HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks

Yu Xue*, Ziming Yuan
School of Computer and Software, Nanjing University of Information Science & Technology, Nanjing, 210044, China
* Corresponding Author: Yu Xue. Email:

Journal on Internet of Things 2022, 4(1), 57-67. https://doi.org/10.32604/jiot.2022.025327

Received 15 November 2021; Accepted 15 February 2022; Issue published 17 May 2022

Abstract

The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. Most attention mechanisms are bound to the convolutional layer and use local or global contextual information to recalibrate the input. This is a popular attention strategy design method. Global contextual information helps the network to consider the overall distribution, while local contextual information is more general. The contextual information makes the network pay attention to the mean or maximum value of a particular receptive field. Different from the most attention mechanism, this article proposes a novel attention mechanism with the heuristic difference attention module (HDAM). HDAM's input recalibration is based on the difference between the local and global contextual information instead of the mean and maximum values. At the same time, to make different layers have a more suitable local receptive field sizes and increase the flexibility of the local receptive field design, we use genetic algorithm to heuristically produce local receptive fields. First, HDAM extracts the mean value of the global and local receptive fields as the corresponding contextual information. Then the difference between the global and local contextual information is calculated. Finally, HDAM uses this difference to recalibrate the input. In addition, we use the heuristic ability of genetic algorithm to search for the local receptive field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that HDAM can use fewer parameters than other attention mechanisms to achieve higher accuracy. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.

Keywords

Attention mechanism; convolutional neural network; genetic algorithm

Cite This Article

Y. Xue and Z. Yuan, "Hdam: heuristic difference attention module for convolutional neural networks," Journal on Internet of Things, vol. 4, no.1, pp. 57–67, 2022.



This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
  • 554

    View

  • 263

    Download

  • 0

    Like

Share Link

WeChat scan