Regularization Algorithms for Learning that are Equivalent to Multilayer Networks

• Published in: Science (American Association for the Advancement of Science) Vol. 247; no. 4945; pp. 978 - 982
• Main Authors: Poggio, T., Girosi, F.
• Format: Journal Article
• Language: English
• Published: Washington, DC American Society for the Advancement of Science 23-02-1990; American Association for the Advancement of Science; The American Association for the Advancement of Science
• Subjects: A priori knowledge; Algorithms; Applied sciences; Approximation; Artificial intelligence; Computer science; control theory; systems; Coordinate systems; Exact sciences and technology; Greens function; Hypersurfaces; Learning; Machine learning; Mathematical functions; Mathematical vectors; Mathematics; Neural circuitry; Physiological aspects; Science; Learning; Regularization method; Approximation; Multilayer; N variable function; Neural network; Algorithm
• ISSN: 0036-8075; 1095-9203
• DOI: 10.1126/science.247.4945.978

Learning an input-output mapping from a set of examples, of the type that many neural networks have been constructed to perform, can be regarded as synthesizing an approximation of a multidimensional function (that is, solving the problem of hypersurface reconstruction). From this point of view, this form of learning is closely related to classical approximation techniques, such as generalized splines and regularization theory. A theory is reported that shows the equivalence between regularization and a class of three-layer networks called regularization networks or hyper basis functions. These networks are not only equivalent to generalized splines but are also closely related to the classical radial basis functions used for interpolation tasks and to several pattern recognition and neural network algorithms. They also have an interesting interpretation in terms of prototypes that are synthesized and optimally combined during the learning stage.