A New Correntropy-Based Conjugate Gradient Backpropagation Algorithm for Improving Training in Neural Networks

• Published in: IEEE transaction on neural networks and learning systems Vol. 29; no. 12; pp. 6252 - 6263
• Main Authors: Heravi, Ahmad Reza, Abed Hodtani, Ghosheh
• Format: Journal Article
• Language: English
• Published: United States IEEE 01-12-2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Artificial neural networks; Back propagation; Chaos theory; conjugate gradient (CG) descent; Conjugates; Convergence; correntropy; Information theory; mean square error (MSE) methods; Neural networks; Noise; Optimization; optimization methods; Robustness; Robustness (mathematics); Search methods; Signal processing algorithms; Training
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2018.2827778

Mean square error (MSE) is the most prominent criterion in training neural networks and has been employed in numerous learning problems. In this paper, we suggest a group of novel robust information theoretic backpropagation (BP) methods, as correntropy-based conjugate gradient BP (CCG-BP). CCG-BP algorithms converge faster than the common correntropy-based BP algorithms and have better performance than the common CG-BP algorithms based on MSE, especially in nonGaussian environments and in cases with impulsive noise or heavy-tailed distributions noise. In addition, a convergence analysis of this new type of method is particularly considered. Numerical results for several samples of function approximation, synthetic function estimation, and chaotic time series prediction illustrate that our new BP method is more robust than the MSE-based method in the sense of impulsive noise, especially when SNR is low.