On-line learning with minimal degradation in feedforward networks

• Published in: IEEE transactions on neural networks Vol. 6; no. 3; pp. 657 - 668
• Main Authors: de Angulo, V.R., Torras, C.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 1995; Institute of Electrical and Electronics Engineers
• Subjects: Adaptive systems; Applied sciences; Artificial intelligence; Backpropagation algorithms; Computer science; control theory; systems; Connectionism. Neural networks; Constraint optimization; Degradation; Encoding; Exact sciences and technology; Fault tolerance; Informatics; Intelligent networks; Interference; Learning and adaptive systems; Systems engineering and theory; Learning; Fault tolerance; On line; Constraint; Problem solving; Neural network; Feedforward; Algorithm; Constrained optimization
• ISSN: 1045-9227; 1941-0093
• DOI: 10.1109/72.377971

Dealing with nonstationary processes requires quick adaptation while at the same time avoiding catastrophic forgetting. A neural learning technique that satisfies these requirements, without sacrificing the benefits of distributed representations, is presented. It relies on a formalization of the problem as the minimization of the error over the previously learned input-output patterns, subject to the constraint of perfect encoding of the new pattern. Then this constrained optimization problem is transformed into an unconstrained one with hidden-unit activations as variables. This new formulation leads to an algorithm for solving the problem, which we call learning with minimal degradation (LMD). Some experimental comparisons of the performance of LMD with backpropagation are provided which, besides showing the advantages of using LMD, reveal the dependence of forgetting on the learning rate in backpropagation. We also explain why overtraining affects forgetting and fault tolerance, which are seen as related problems.< >