Early Stopping without a Validation Set

• Main Authors: Mahsereci, Maren, Balles, Lukas, Lassner, Christoph, Hennig, Philipp
• Format: Journal Article
• Language: English
• Published: 28-03-2017
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.1703.09580

Early stopping is a widely used technique to prevent poor generalization
performance when training an over-expressive model by means of gradient-based
optimization. To find a good point to halt the optimizer, a common practice is
to split the dataset into a training and a smaller validation set to obtain an
ongoing estimate of the generalization performance. We propose a novel early
stopping criterion based on fast-to-compute local statistics of the computed
gradients and entirely removes the need for a held-out validation set. Our
experiments show that this is a viable approach in the setting of least-squares
and logistic regression, as well as neural networks.