Bridging between soft and hard thresholding by scaling

• Main Author: Hagiwara, Katsuyuki
• Format: Journal Article
• Language: English
• Published: 02-02-2022
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2104.09703

In this article, we developed and analyzed a thresholding method in which
soft thresholding estimators are independently expanded by empirical scaling
values. The scaling values have a common hyper-parameter that is an order of
expansion of an ideal scaling value that achieves hard thresholding. We simply
call this estimator a scaled soft thresholding estimator. The scaled soft
thresholding is a general method that includes the soft thresholding and
non-negative garrote as special cases and gives an another derivation of
adaptive LASSO. We then derived the degree of freedom of the scaled soft
thresholding by means of the Stein's unbiased risk estimate and found that it
is decomposed into the degree of freedom of soft thresholding and the reminder
connecting to hard thresholding. In this meaning, the scaled soft thresholding
gives a natural bridge between soft and hard thresholding methods. Since the
degree of freedom represents the degree of over-fitting, this result implies
that there are two sources of over-fitting in the scaled soft thresholding. The
first source originated from soft thresholding is determined by the number of
un-removed coefficients and is a natural measure of the degree of over-fitting.
We analyzed the second source in a particular case of the scaled soft
thresholding by referring a known result for hard thresholding. We then found
that, in a sparse, large sample and non-parametric setting, the second source
is largely determined by coefficient estimates whose true values are zeros and
has an influence on over-fitting when threshold levels are around noise levels
in those coefficient estimates. In a simple numerical example, these
theoretical implications has well explained the behavior of the degree of
freedom. Moreover, based on the results here and some known facts, we explained
the behaviors of risks of soft, hard and scaled soft thresholding methods.