Open Access

ARTICLE

Heuristic Weight Initialization for Transfer Learning in Classification Problems

Musulmon Lolaev¹, Anand Paul^2,*, Jeonghong Kim¹

1 The School of Computer Science and Engineering, Kyungpook National University, Dae-Hak Ro, Daegu, 41566, Republic of Korea
2 Department of Biostatistics and Data Science, LSU Health Sciences Center, New Orleans, LA 70112, USA

* Corresponding Author: Anand Paul. Email:

(This article belongs to the Special Issue: Artificial Intelligence Algorithms and Applications)

Computers, Materials & Continua 2025, 85(2), 4155-4171. https://doi.org/10.32604/cmc.2025.064758

Received 23 February 2025; Accepted 09 June 2025; Issue published 23 September 2025

Abstract

Transfer learning is the predominant method for adapting pre-trained models on another task to new domains while preserving their internal architectures and augmenting them with requisite layers in Deep Neural Network models. Training intricate pre-trained models on a sizable dataset requires significant resources to fine-tune hyperparameters carefully. Most existing initialization methods mainly focus on gradient flow-related problems, such as gradient vanishing or exploding, or other existing approaches that require extra models that do not consider our setting, which is more practical. To address these problems, we suggest employing gradient-free heuristic methods to initialize the weights of the final new-added fully connected layer in neural networks from a small set of training data with fewer classes. The approach relies on partitioning the output values from pre-trained models for a small set into two separate intervals determined by the targets. This process is framed as an optimization problem for each output neuron and class. The optimization selects the highest values as weights, considering their direction towards the respective classes. Furthermore, empirical 145 experiments involve a variety of neural network models tested across multiple benchmarks and domains, occasionally yielding accuracies comparable to those achieved with gradient descent methods by using only small subsets.

---

Keywords

---