Learning overcomplete, low coherence dictionaries with linear inference

• Main Authors: Livezey, Jesse A, Bujan, Alejandro F, Sommer, Friedrich T
• Format: Journal Article
• Language: English
• Published: 10-06-2016
• Subjects: Computer Science - Learning
• DOI: 10.48550/arxiv.1606.03474

JMLR 20(174) 1-42 (2019) Finding overcomplete latent representations of data has applications in data
analysis, signal processing, machine learning, theoretical neuroscience and
many other fields. In an overcomplete representation, the number of latent
features exceeds the data dimensionality, which is useful when the data is
undersampled by the measurements (compressed sensing, information bottlenecks
in neural systems) or composed from multiple complete sets of linear features,
each spanning the data space. Independent Components Analysis (ICA) is a linear
technique for learning sparse latent representations, which typically has a
lower computational cost than sparse coding, its nonlinear, recurrent
counterpart. While well suited for finding complete representations, we show
that overcompleteness poses a challenge to existing ICA algorithms.
Specifically, the coherence control in existing ICA algorithms, necessary to
prevent the formation of duplicate dictionary features, is ill-suited in the
overcomplete case. We show that in this case several existing ICA algorithms
have undesirable global minima that maximize coherence. Further, by comparing
ICA algorithms on synthetic data and natural images to the computationally more
expensive sparse coding solution, we show that the coherence control biases the
exploration of the data manifold, sometimes yielding suboptimal solutions. We
provide a theoretical explanation of these failures and, based on the theory,
propose improved overcomplete ICA algorithms. All told, this study contributes
new insights into and methods for coherence control for linear ICA, some of
which are applicable to many other, potentially nonlinear, unsupervised
learning methods.