TY - EJOU
AU - Vidyabharathi, D.
AU - Mohanraj, V.
TI - Hyperparameter Tuning for Deep Neural Networks Based Optimization Algorithm
T2 - Intelligent Automation \& Soft Computing
PY - 2023
VL - 36
IS - 3
SN - 2326-005X
AB - For training the present Neural Network (NN) models, the standard
technique is to utilize decaying Learning Rates (LR). While the majority of these
techniques commence with a large LR, they will decay multiple times over time.
Decaying has been proved to enhance generalization as well as optimization.
Other parameters, such as the networkâ€™s size, the number of hidden layers, dropouts to avoid overfitting, batch size, and so on, are solely based on heuristics. This
work has proposed Adaptive Teaching Learning Based (ATLB) Heuristic to identify
the optimal hyperparameters for diverse networks. Here we consider three architectures Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM),
Bidirectional Long Short Term Memory (BiLSTM) of Deep Neural Networks for
classification. The evaluation of the proposed ATLB is done through the various
learning rate schedulers Cyclical Learning Rate (CLR), Hyperbolic Tangent Decay
(HTD), and Toggle between Hyperbolic Tangent Decay and Triangular mode with
Restarts (T-HTR) techniques. Experimental results have shown the performance
improvement on the 20Newsgroup, Reuters Newswire and IMDB dataset.
KW - Deep learning; deep neural network (DNN); learning rates (LR); recurrent neural network (RNN); cyclical learning rate (CLR); hyperbolic tangent decay (HTD); toggle between hyperbolic tangent decay and triangular mode with restarts (T-HTR); teaching learning based optimization (TLBO)
DO - 10.32604/iasc.2023.032255